Rare species are increasingly recognized as crucial, yet vulnerable components of Earth’s ecosystems. This is also true for microbial communities, which are typically composed of a high number of relatively rare species. Recent studies have demonstrated that rare species can have an over-proportional role in biogeochemical cycles and may be a hidden driver of microbiome function. In this review, we provide an ecological overview of the rare microbial biosphere, including causes of rarity and the impacts of rare species on ecosystem functioning. We discuss how rare species can have a preponderant role for local biodiversity and species turnover with rarity potentially bound to phylogenetically conserved features. Rare microbes may therefore be overlooked keystone species regulating the functioning of host-associated, terrestrial and aquatic environments. We conclude this review with recommendations to guide scientists interested in investigating this rapidly emerging research area.