Rare Phyla, Such as CPR and DPANN, Shape Ecosystem-Level Microbial Community Structure Dissimilarities

Abstract

Rare microbial lineages, such as members of the candidate phyla radiation (CPR) bacteria and Diapherotrites, Parvarchaeota, Aenigmarchaeota, Nanoarchaeota, and Nanohaloarchaeota (DPANN) archaea, are increasingly recognized as key components of microbial communities in natural systems. Yet, their global distribution, biogeographic patterns, and broader role in shaping microbial community structure across diverse ecosystems remain poorly characterized. Here, we analyzed 2860 metagenomes spanning nine ecosystems using a curated reference database and a bias-aware taxonomic filtering approach to quantify the richness, relative abundance, and structural influence of low-abundance microbial taxa on community structure across a wide range of ecosystems. Our findings reveal that rare taxa, primarily CPR and DPANN, disproportionately shape microbial community dissimilarities across global ecosystems. We observed that the richness of these two groups, that drives community structure variation, increases with latitude, peaking in temperate regions, thereby contrasting classical latitudinal diversity patterns and suggesting unique biogeographic drivers. CPR and DPANN were predominantly enriched in free-living environments, particularly groundwater and soil, then in host-associated habitats, consistent with niche specialization shaped by environmental filtering and dispersal constraints. These findings challenge abundance-centric assumptions in microbial ecology and highlight the need to integrate low-abundance taxa into macroecological frameworks. Fully resolving their ecological functions, however, will require targeted experimental and multi-omics investigations.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00248-025-02595-0.

Introduction

The accelerating impacts of climate change are fundamentally restructuring the ecosystems around the world, driving shifts in species distributions, trophic interactions, and biogeochemical cycles [1, 2]. Within this context, microbial communities have emerged as essential regulators of ecosystem stability and resilience, mediating processes from nutrient cycling to energy flow [3, 4]. These communities are characterized by a “rare biosphere,” taxa with relative abundance consistently low across ecosystems (below 1%) but high diversity, that may act as reservoirs of ecological adaptability under environmental stress [5, 6]. While abundant taxa have long been emphasized as primary drivers and guarantors of community structure stability [7], emerging evidence suggests that low-abundance (rare) microorganisms, defined here as those with a mean relative abundance below 1% in a given sample, can disproportionately influence metabolic processes [8, 9], interspecies interactions [10], and cross-ecosystem dissimilarities [11, 12].

Among rare microbial taxa, the candidate phyla radiation (CPR) and DPANN (Diapherotrites, Parvarchaeota, Aenigmarchaeota, Nanoarchaeota, and Nanohaloarchaeota) archaea consistently exhibit low relative abundances (< 1%) across diverse environments [13, 14] and are phylogenetically distinct groups often overlooked in standard reference databases. These lineages are characterized by their metabolic streamlining, ecological ubiquity, and the persistent gap in understanding their role in structuring microbial communities at large spatial scales. A critical unresolved question is whether rare taxa, despite their low relative abundance representation, underpin microbial community dissimilarities across ecosystems. Groups such as the CPR bacteria and DPANN archaea exemplify this paradox: Their reduced genomes and streamlined metabolisms [15, 16] suggest niche specialization, yet their ubiquity across diverse environments with specific community structure [17], implies unrecognized roles [18] in maintaining both community and ecosystem resilience. For example, some of these low-abundance microorganisms may, independent of their numerical abundance, play a critical role in shaping community structure and ecosystem function by driving nutrient transformations or enabling key microbial interactions [8]. Traditional ecological frameworks, which prioritize abundance as a proxy for functional impact [5], fail to account for the cascading effects of rare biosphere shifts, particularly under climate stressors like ocean acidification and warming [7].

Despite growing recognition of the rare biosphere’s relevance, the extent to which low-abundance taxa drive microbial community dissimilarities across ecosystems remains an open empirical question. This pattern likely stems from mechanistic influences of obligate metabolic interdependence and niche-specific genomic adaptations (i.e., genes or operons that confer a fitness advantage in particular environmental contexts), which allow CPR and DPANN lineages to persist and thrive in specialized habitats such as extreme or resource-limited environments [5, 7]. Compounding this challenge, large-scale metagenomic studies integrating datasets from multiple sources are prone to methodological variability that can confound biological interpretation. Differences in sample collection, sequencing depth, equipment, and investigator-specific protocols may introduce technical biases that mimic ecological patterns. To address this apparent contradiction, we analyzed 2860 metagenomes from nine ecosystems using a custom-curated database that combines 12,804 metagenome-assembled genomes (MAGs) and NCBI RefSeq genomes [19, 20] enabling the detection of rare biosphere members, mainly CPR and DPANN, across diverse habitats (Fig. S1a–f). CPR and DPANN were selected as focal taxa due to their consistent low abundance, ecological distinctiveness, and underrepresented in standard reference databases. These lineages represent well-defined subsets of the rare biosphere with unique genomic features, including reduced metabolic capabilities and obligate symbioses, making them suitable models for investigating how rare taxa shape microbial community composition across environments. To improve detection and classification of these groups, we curated a custom database integrating 12,804 metagenome-assembled genomes (MAGs) and NCBI RefSeq genomes, enabling retrospective identification of CPR and DPANN lineages across ecosystems. To account for methodological heterogeneity across datasets, we incorporated technical metadata variables, including sequencing depth, PI identity, collector, and sampling equipment, as potential covariates. PI identity was used as a proxy for unreported laboratory-specific protocols (e.g., DNA extraction, library preparation, sequencing platform), which can influence community composition independently of ecological factors. The present study tackles three key gaps in microbial ecology: (1) As conventional databases often miss rare taxa like CPR and DPANN, our expanded database, totaling 159 phyla, including ASGARD archaea and Microgenomates, boosted candidate phyla detection by 90% (Fig. S1f), showing previously undetected contributions to richness. (2) To disentangle the relative contributions of technical, environmental, and biological factors to microbial community structure, we employed random forest models [21] using predictor variables such as sequencing depth, principal investigator identity, collector, and sampling equipment, while also quantifying the explanatory power of ecological metadata, including ecosystem type, latitude, and longitude, and individual taxon identity to evaluate whether biological structuring outweighs technical variability (Fig. S2a–b). (3) By connecting Similarity Percentage (SIMPER) analysis of dissimilarity contributions [22] to CPR and DPANN genomic traits [23, 24], we explored if their streamlined metabolisms drive ecosystem differentiation more than abundant taxa.

Methods

Data Collection

The present study used 2860 metagenomic samples filtered from 24,694 publicly available samples deposited in the MG-RAST repository (https://www.mg-rast.org/), downloaded in September 2023, comprising approximately 11 TB of sequence data. All samples retrieved, given the dynamic nature of public databases and to ensure reproducibility, were archived with the complete list of accession numbers and associated metadata, which are available in Supplementary Table S1 and our project repository. After initial filtering, we retained 7043 samples with complete metadata information in our sample pool. Additionally, we included 240 aquifer and sediment metagenomes samples from the NCBI/SRA repository (https://www.ncbi.nlm.nih.gov/sra) to benchmark our metagenomic annotation approach, given that CPR groups were first discovered in these ecosystems. Finally, manual curation was conducted to exclude samples with erroneous metadata such as geographically implausible locations and laboratory or experimental studies that could introduce biases in taxonomic representation. This curation led to 3467 complete metadata samples. To retrieve metagenomic data, we implemented a standardized computational workflow (Fig. S3). For MG-RAST samples, we collected files using the “0.299” code from the MG-RAST API pre-processing pipeline (http://api.metagenomics.anl.gov/download/), ensuring that all files (FASTA and FASTQ format) had undergone initial filtering, including the removal of putative spurious sequences [25]. For NCBI/SRA samples, the retrieval process utilized SRAToolKit (https://www.ncbi.nlm.nih.gov/sra/docs/toolkitsoft/), employing the prefetch function to download raw sequences, followed by conversion to FASTQ format and validation using the vdb-validation tool. We then performed cross-platform standardization, applying identical quality filters to samples from both platforms to ensure data consistency. Specifically, all sample metadata were manually curated for uniformity, and sequences were filtered for both quality and quantity, including minimum read length, minimum quality score, and the removal of low-quality reads. Quality control was performed using Fastp, for FASTQ files, to trim adapter sequences, trim low-quality edges (quality scores below 20), remove low quality reads (mean quality score below 20), enforce a minimum sequence length of 50 bases, and filtering out sequences with two or more ambiguous bases. Since FASTA format does not have quality scores, for FASTA files, we applied Prinseq [26] to perform the quality control, enforcing the same minimum sequence length, and filtering out sequences with two or more ambiguous bases. Additionally, we employed Bowtie2 [27] to filter out potential contaminants before taxonomic annotation and downstream analyses (see the entire workflow in Fig. S3). A detailed step-by-step protocol and source codes for data retrieval and processing are available in our online repository (https://github.com/MeirellesLab/metagenomic_public_database/).

Metagenomic Classification

Metagenomic datasets may present limitations for estimating diversity, particularly of rare taxa, we implemented a series of methodological steps to mitigate these biases and ensure a robust, high-quality dataset for comparative ecological analysis. This approach builds upon a growing number of metagenomic studies that have successfully recovered and analyzed rare microbial groups across various ecosystems, including groundwater, lake sediments, and host-associated environments [28–30]. These studies demonstrate that, while not without constraints, metagenomic approaches remain valuable tools for exploring the ecological patterns and potential roles of rare taxa. Thus, we first assigned taxonomy to metagenomic sequences, using Kraken2 [31]. To enhance the taxonomic resolution of recently identified microbial groups, we constructed a custom reference database incorporating genomes from NCBI RefSeq [19, 20, 23, 24, 32, 33].

The final custom database included genomes from both cultured and uncultured microbial taxa, totaling 159 phyla (Table S2). Specifically, it encompassed representatives from CPR, ASGARD archaea, DPANN, TACK, Microgenomates (OP11), and Parcubacteria (OD1) supergroups, covering more than 70 phyla. These genomes were manually curated and integrated into the reference database. Following quality control and decontamination, all metagenomes passing filtering thresholds were taxonomically classified using Kraken2 with default parameters. The kraken-biom script (https://github.com/smdabdoub/kraken-biom) was used to generate summarized classification reports. To ensure accurate taxonomic annotation, only the Kraken2-generated kraken-report files were retained for downstream analysis. To compile a unified abundance matrix across all samples, a custom script leveraging the kraken-biom tool and the biom-format package was employed to convert the individual kraken-report files into a matrix of absolute taxonomic abundances. The resulting matrix was then processed using a custom R script to normalize the data, converting absolute counts into relative abundances for each taxon in each sample. This normalization was done by dividing the raw read counts for each taxon by the total read count for each sample. Reads that could not be confidently assigned at the lower taxonomic level (genera/species) were designated as “Unclassified” at the highest available taxonomic rank (family). Based on rarefaction analysis (Fig. S4), which showed that taxonomic richness plateaued at this sequencing depth, samples with fewer than 1000 classified reads were removed, together with samples with less than half of the expected phylum richness (< 77 phyla) to ensure sufficient sequencing depth and taxonomic signal for reliable community structure analysis. This threshold, equivalent to 50% of the total 159 phyla in our custom database, was applied to exclude samples with artificially reduced taxonomic representation, such as those originating from microbial enrichments, experimental manipulations, or selective isolations. Such samples often exhibit atypical diversity patterns that can distort community structure analyses and undermine ecological comparability across ecosystems. Applying these criteria resulted in the removal of 17.5% (607) of the samples (337 due to insufficient read count; 270 due to low phylum richness), yielding a final dataset of 2860 samples for downstream analyses (Fig. S4).

Although a statistically significant positive correlation was observed between the relative abundance of candidate taxa and metagenome size (p < 0.00001), the effect size was negligible (R2 = 0.00014), indicating no biologically relevant association (Fig. S1b; Table S3). However, our custom database recovered 1.69% more sequences than the standard Kraken2 database, classifying an average of 132,374 additional sequences per sample, with a total of 780,743,986 sequences across all samples (Fig. S1c). This gain was particularly pronounced in plant host-associated and groundwater samples, where sequence classification increased by over 5% (Fig. S1d). Moreover, the custom reference database significantly increased detected richness by over 200% (Fig. S1e), primarily due to a 90% rise in candidate taxa detection (Fig. S1f). Throughout this study, we operationally define rare taxa as those exhibiting a mean relative abundance of less than 1% across samples, following previous studies [34–36].

Statistical Analysis

All statistical analyses and data visualizations were performed in R 3.4.1 [37], using base functions and specialized packages as detailed below.

Public Metagenomic Analysis and Bias Investigation

Annotating publicly available metagenomes introduces potential biases arising from variations in sample collection, sequencing protocols, and processing pipelines. Ignoring these biases may lead to misleading biological inferences. To assess the impact of technical variability on microbial community structure, we employed a series of random forest models, following established methods for soil microbiome studies [21]. Three models were applied to disentangle the relative contributions of technical, environmental, and biological factors to microbial community structure. The first model employed random forest regression [38] with phylum-level relative abundances as predictor variables and the first principal coordinate axis (PCoA1) derived from Bray–Curtis dissimilarities of the full community matrix as the response variable. This allowed us to evaluate which phyla contributed most strongly to the overall community structure, without circularity, as each phylum’s influence was assessed relative to a global ordination score rather than directly predicting community sums. Variable importance (VI) scores were averaged across 30 runs, and a threshold for variable elimination was determined using the caret package in R [39]. Model performance was assessed via out-of-bag error rates using 10,000 trees, default hyperparameters, and a one-third random subspace sampling strategy. The second model was a supervised random forest classifier trained to predict categorical metadata variables (i.e., principal investigator identity, ecosystem type, and geographic zone) based on microbial community composition. The input features consisted of relative abundances of the 12 most influential phyla, identified using a CART-derived threshold applied to the variable importance scores, which defined a point of diminishing returns in model performance. Separate classification models were built for each metadata variable to evaluate their individual predictability (Fig. S2a), enabling a direct comparison of biological versus metadata-driven structuring patterns. This classifier used 300,000 trees, default hyperparameters, and a square-root random subspace approach. The last model was an unsupervised random forest analysis with 300,000 trees, and all other hyperparameters default, and a one-third random subspace strategy to detect clustering patterns independent of predefined categories.

Analysis of Variance

We employed two-way permutational multivariate analysis of variance (PERMANOVA), using compositional data (taxon relative abundances), to assess structural differences in microbial communities between samples categorized with host-associated and free-living lifestyles across nine ecosystems, using Bray–Curtis dissimilarity matrices (Fig. 1b; Table S4), and performed pairwise comparisons between classification levels (Table S5). Samples were classified as free-living or host-associated based on detailed environmental metadata and manual curation. In the case of biofilms, we considered their ecological context: Biofilms on abiotic substrates (e.g., rocks in aquifers) were classified as free-living, whereas those associated with plant or animal hosts (e.g., coral surfaces) were classified as host-associated, reflecting their integration into holobiont systems. Complementary permutational univariate ANOVAs [40] evaluated richness and abundance patterns for the three microbial groups: (1) taxonomic groups confidently assigned based on high-confidence sequence classification (will be here referred as “Bonafide”), (2) CPR bacteria, and (3) DPANN archaeal lineages (Table S6). All permutational multivariate analyses of variance (PERMANOVA) were conducted with 4999 Monte Carlo permutations using Bray–Curtis distance matrices calculated through the vegan package [41] in R. Post hoc pairwise comparisons between significant factors were performed using the pairwise.adonis2() function [42] with p-values adjusted for multiple comparisons using the Benjamini–Hochberg false discovery rate (FDR) procedure [43]. Community patterns were visualized through non-metric multidimensional scaling (NMDS) ordination with stress values to indicate a good representation of multivariate space.

Fig. 1.

Global microbial community structure across nine distinct ecosystems. A Geographic distribution of 2860 metagenomic samples from microbial communities sampled across nine ecosystems worldwide, with point size scaled to sampling effort (latitude/longitude coordinates). Non-metric multidimensional scaling (NMDS) ordinations illustrating dissimilarities in community structure for B total microbial community; C taxonomically defined, culturable microbes (Bonafide); D candidate phyla radiation (CPR) uncultured, recently identified bacteria; and E DPANN superphylum (Diapherotrites, Parvarchaeota, Aenigmarchaeota, Nanoarchaeota, Nanohaloarchaeota). Distinct colors denote ecosystem types (consistent across panels A–E). Detailed sampling effort is provided in Table S1, with statistical details reported in Tables S4 and S5

Latitudinal Richness Gradient Analysis

To explore large-scale biogeographic trends in microbial diversity and minimize potential biases associated with uneven geographic sampling [44], we assessed the relationship between latitude and microbial richness, specifically of Bonafide, CPR, and DPANN phyla, using generalized additive models (GAMs). GAMs were selected for their capacity to accommodate nonlinear relationships and unbalanced datasets, as commonly encountered in macroecological analyses [45, 46]. We used a Gaussian family with identity link, selected smoothing terms via residual deviance comparisons and REML, and evaluated model assumptions through Q–Q plot diagnostics and deviance residuals. Despite the relatively minor explanatory contribution of latitude to overall community structure in our random forest analysis (Fig. S2a), latitude remains a key ecological gradient known to influence biodiversity patterns in macroecology [41] and microbial systems [47]. Our goal was to test whether rare taxa exhibit deviations from the classical latitudinal diversity gradient. Longitude was excluded due to its weaker ecological interpretability in shaping global biodiversity patterns compared to latitude (Table S7). Before analysis, variables were centered and scaled using the caret package [39, 48] to mitigate scale-related biases. Models were fitted using the mgcv package [49] in R 3.4.1, with smooth terms selected via restricted maximum likelihood (REML). All model diagnostics and visualization of smooth terms were generated using mgcViz package in R [50], and all other figures were generated using the R package ggplot2 [51]. The statistical significance of latitude effects was evaluated using ANOVA-like deviance tests, and pairwise comparisons of means were conducted with the emmeans package [52].

Microbial Community Differentiation Analysis

To identify taxa-driving microbial community dissimilarities between ecosystems, we performed a SIMPER (Similarity Percentage) analysis using the simper function in the vegan package (v2.6–10) [41]. This method decomposes Bray–Curtis dissimilarity (δ) between ecosystem pairs into taxon-specific contributions, enabling the identification of those taxa with the strongest influence on compositional differences. For clarity, we define based on SIMPER output metrics: (1) total dissimilarity (δ), the overall Bray–Curtis dissimilarity between two ecosystems; (2) cumulative contribution (C_i), the proportion of δ explained by the top i taxa when ranked by contribution (e.g., C₅ = 60% indicates the top five taxa collectively explain 60% of δ total dissimilarity); and (3) incremental contribution (Δ C_i), the unique contribution of the ith taxon, calculated as:

Δ C_i = C_i − C_{i −1}.

This framework isolates taxon-specific effects by distinguishing dominant drivers from marginal contributors. For each taxon–ecosystem pair, we computed three metrics. The first metric is the mean contribution to δ (%), which represents the average percentage contribution of the taxon across all relevant pairwise ecosystem comparisons. Next, we calculated the standard deviation (SD) of contribution to assess the variability in the taxon’s role, reflecting its context-dependency. Finally, we analyzed the cumulative contribution trajectory, which represents the rate at which cumulative δ is explained as taxa is added, thereby highlighting saturation points where adding more taxa yields diminishing returns. While these metrics help summarize the results of SIMPER, they do not fully capture the ecological consistency or differentiation power of taxa across ecosystems.

To quantify the contribution of each taxon to community structure differentiation across ecosystems, we used SIMPER-based decomposition of Bray–Curtis dissimilarities. We defined the ecosystem differentiation index (DIₖ) as the average contribution of a given taxon to all pairwise dissimilarities involving a specific ecosystem (k). Unlike model-derived metrics such as random forest variable importance, which reflect predictive gain across all samples, DIₖ is unsupervised and directly interpretable in dissimilarity space. It offers a taxon-centric view of compositional distinctiveness and allows us to identify lineages that disproportionately shape beta-diversity across environments.

Results

We assessed the richness, abundance, and structure of the 2860 metagenomic samples divided into the nine ecosystems (Fig. 1), using generalized additive models (GAMs) to model the variability of the metrics in the latitudinal gradient (Fig. 2), compared the abundance and richness of microbial groups and lifestyles across ecosystems (Fig. 3), and measured the contribution of each taxon to the communities’ dissimilarities through similarity percentage analysis (Fig. 4).

Fig. 2.

Latitudinal variation in microbial richness across microbial groups. A Candidate phyla radiation (CPR; uncultured, recently identified bacteria), B DPANN superphylum (Diapherotrites, Parvarchaeota, Aenigmarchaeota, Nanoarchaeota, Nanohaloarchaeota), and C taxonomic groups confidently assigned based on high-confidence sequence classification. (Bonafide taxon). Solid lines represent the mean fit from generalized additive models (GAMs), with Shaded areas indicating 95% confidence intervals. Statistical significance of the latitudinal gradient is denoted by *p < 0.05, **p < 0.001, and ***p < 0.0001, and ns (p > 0.05) for no significant difference

Fig. 3.

Microbial community composition and diversity across ecosystems and lifestyles. A–C Relative abundance and D–F richness of microbial communities in free-living and host-associated lifestyles. G–I Relative abundance and J–L richness of microbial groups (Bonafide, CPR, and DPANN) compared between ecosystems. Bar plots represent mean ± SE. Statistical significance is indicated by *p < 0.05, **p < 0.001, and ***p < 0.0001, and ns (p > 0.05) for no significant difference. Different letters indicate significant differences across ecosystems (p < 0.05) based on Tukey’s HSD post hoc test (shared letters indicate no difference)

Fig. 4.

Distribution of CPR and DPANN and their impact on community structure across ecosystems and lifestyles. A Relative abundance of each CPR (candidate phyla radiation) and DPANN (Diapherotrites, Parvarchaeota, Aenigmarchaeota, Nanohaloarchaeota, Nanoarchaeota) phylum across nine ecosystems, with individual samples shown as dots colored by ecosystem. B Median relative abundances indicating lifestyle preferences (yellow: free-living, blue: holobiont) for each phylum. C Heatmap of phylum ecosystem differentiation index (DI_k; Table S10), with darker colors highlighting potential important taxa, lighter colors less-important taxa, and empty squares absence in the habitat

Overall, we found that nine out of 11 taxa (82%) with contributions to community dissimilarities above 1% in all ecosystems are from the rare biosphere, including some candidate phyla with extremely low abundance. We observed that microbial community structure differed across ecosystems and lifestyles. Building on this, we assessed the richness and abundance of CPR and DPANN groups, elucidating the relationship between this low-abundance groups richness and latitude and their distinct distribution patterns across ecosystems.

Our results also indicate that technical artifacts, such as (a) principal investigator identity, (b) total sequencing depth, and (c) abiotic metadata variables (e.g., ecosystem type, latitude, longitude), exerted minimal influence on community composition compared to biological factors (Fig. S2a). Specifically, phylum-level taxonomic structure exhibited a strong positive correlation with relative abundance patterns (Fig. S2b), suggesting that technical biases did not significantly distort the observed microbial community structures. These findings confirm that metagenomic datasets retrieved from public repositories remain robust for microbial ecology analyses, provided that appropriate filtering and normalization procedures are applied.

Microbial Community Structure and Latitudinal Richness Variations

We observed that microbial community structures differed significantly between host-associated microbiomes and free-living environments (PERMANOVA: R2 = 0.078; F₁ = 69.094; p = 0.0002), indicating that host association imposes distinct ecological filters on microbial assemblages. Although the effect size may appear modest, it represents a meaningful signal in light of the high taxonomic richness and compositional redundancy characteristic of microbial communities, where a small subset of abundant taxa typically drives most dissimilarity patterns. Additionally, those communities differ across the nine ecosystems examined (PERMANOVA: R² = 0.288; F₈ = 21.899; p = 0.0002; Fig. 1b; Table S4), irrespective of latitude, reflecting ecosystem-specific microbial assemblages, likely shaped by niche differentiation across distinct environmental contexts. However, multivariate dispersion was not fully homogeneous across groups (Fig. S5), while this should be considered when interpreting the PERMANOVA results, the differences observed are unlikely to result exclusively from within-group dispersion and still reflect meaningful variation in group centroids, remaining ecologically informative. The differences in community structure between ecosystems were primarily driven by abundant taxa from the Bonafide group, taxonomically well-defined and culturable microbes that largely shape dominant community patterns (Fig. 1c; SIMPER Table S8 and rank in Table S9). However, 11 of the top 20 taxa contributing most to ecosystem dissimilarities belonged to the CPR group (SIMPER Tables S8 and S9), highlighting that despite their overall lower abundance and influence on total community structure (Fig. 1d, e), individual CPR taxa may exert disproportionate influence on cross-ecosystem variation. Indeed, the CPR group comprised a significant portion of the microbial composition, accounting for ~67.9% of the total microbial richness, but its relative abundance in the overall community was only 1.2%.

When evaluating whether microbial richness and abundance latitudinal gradient variability follow similar patterns to macro-organisms ecology, we found that latitude alone explained 4.5% of the CPR richness deviance (GAM: R² = 0.035; p = 0.0052) and 5.4% for DPANN richness deviance (GAM: R² = 0.043; p < 0.0001) but not for relative abundance, with temperate latitudes supporting twofold richer microbial communities than tropical regions (Fig. 2a, b; Table S7), regardless of lifestyle and ecosystem. This pattern was non-significant for Bonafide groups (Fig. 2c; Table S7).

Microbial Richness and Abundance Across Lifestyles and Ecosystems

Our analysis revealed that only CPR richness varied across lifestyles (holobiont and free-living; PERANOVA: F₁ = 4.223, p < 0.0382), with free-living communities showing an increased number of phyla (2.26%) compared with holobiont communities (Fig. 3a; Table S6). When evaluating the relative abundance across lifestyles, the abundance of Bonafide (taxonomic groups confidently assigned based on high-confidence sequence classification) taxa was marginally lower (0.38%) in free-living communities (PERANOVA: F₁ = 7.138, p = 0.0082). In contrast, both CPR (PERANOVA: F₁ = 6.923, p = 0.0110) and DPANN (PERANOVA: F₁ = 5.285, p = 0.0426) exhibited abundances between 1.3 and 1.5 times greater in the free-living microbiomes (Fig. 3b). The differences in relative abundance between lifestyles for CPR and DPANN were driven by their higher relative abundance in groundwater (PERANOVA: CPR: F₇ = 19.286, p = 0.0002; and DPANN: F₇ = 31.127, p = 0.0002; Fig. 3d), where they exhibited 1.7- and 1.8-fold greater relative abundance, respectively, compared to other ecosystems.

Upon closer examination, although rare, a few CPR and DPANN phyla had elevated abundance in some ecosystems. For instance, Ca. Rokubacteria are abundant in soil, Ca. Woesebacteria and Ca. division WW3 in aquifers, and Ca. Cloacimonetes in wastewater (see Fig. 4a). Notably, our analysis identified the human-associated ecosystem as the most distinct, displaying a low average abundance for most CPR and DPANN (Figs. 3d and 4a). Although our findings suggest that most CPR and DPANN have a higher abundance in free-living ecosystems (Fig. 3b), a few phyla, such as Ca. Saccharibacteria, Ca. Gracilibacteria, Ca. Heimdallarchaeota, and Ca. Parvarchaeota, were equally or more abundant in host-associated lifestyles (Fig. 4b).

Individual Mean Contribution to Ecosystem Dissimilarities

Our results indicate that microbial community differentiation across ecosystems is driven mostly by rare taxa, both by shifts of widespread taxa and distinct ecosystem-specific groups. We found that 21 of the 159 phyla had an ecosystem differentiation index (DIₖ) exceeding 1% in at least one ecosystem. We defined a 1% contribution threshold as the cutoff for identifying “contributionally important” taxa. This threshold is conservative and data-informed, since in a typical comparison involving ~ 160 taxa, an equal distribution of dissimilarity would yield an average contribution of ~ 0.63% per taxon. Thus, a 1% value represents a contribution at least 1.5 times greater than expected under a uniform null model, allowing us to identify taxa that play a disproportional role in compositional differentiation, even if they are rare in absolute abundance. Furthermore, 11 of these 21 phyla consistently surpassed this 1% contribution level across all studied ecosystems. Notably, eight of these 12 consistently contributing phyla belong to the rare biosphere (relative abundance < 1%), including some candidate phyla with extremely low abundances (relative abundance < 0.01%). These specific rare taxa displayed DIₖ values up to twice as high as most other taxa (Fig. 4c), implying that the rare biosphere strongly influences community dissimilarities across ecosystems. However, certain abundant groups, namely, Actinomycetota and Acidobacteriota, also demonstrated substantial influence, exhibited DIₖ values up to 10 times higher than most taxa across all ecosystems (Fig. S6c).

In contrast, some taxa exhibited environment-specific contributions—for example, Ca. Abawacabacteria exceeded 1% DIₖ in freshwater, plant–host, and sediment environments, while Ca. Absconditabacteria surpassed this threshold only in sediment and wastewater. Aside from these exceptions, all taxa contributing more than 1% to dissimilarity did so in at least six out of the nine ecosystems examined (Fig. 4c). This suggests that microbial divergence is governed both by compositional adjustments of widespread taxa, in higher degree, but also by the contribution of some ecosystem specific taxonomic groups, depending on the environment.

Discussion

Our research reveals a dual engine driving microbial community divergence across ecosystems, the quantitatively larger effect of abundance shifts among widespread taxa, and the qualitatively distinct impact of unique, ecosystem-specific groups. This pattern aligns with previous findings that microbial communities are shaped by dispersal limitations and environmental filtering, where generalist taxa persist across multiple ecosystems and adjust their relative abundance according to local conditions [11, 13]. The widespread occurrence of high-contribution taxa across distinct regions suggests that microbial community assembly is shaped less by physical dispersal constraints and more by habitat filtering, whereby environmental conditions select for taxa with specific ecological traits [53, 54]. On the other hand, microbial communities can be driven by specific environment specialized groups; for instance, some freshwater systems contain unique bacterial lineages not found in marine environments [55], highlighting the influence of habitat-specific factors on microbial community composition.

A key implication of these results is the high degree of redundancy across ecosystems. Most taxa exhibited low contributions to community dissimilarity, suggesting that many microbial species can fulfill similar ecological roles, leading to compositional shifts without necessarily altering ecosystem compositional identity, despite taxonomic variations [56]. The predominance of widespread, generalist taxa in driving dissimilarity further suggests that microbial ecosystems are structured by gradual compositional adjustments rather than abrupt taxonomic shifts, a pattern commonly observed in microbial succession across spatial gradients (e.g., among ecosystems or biome; [57]) and over time (e.g., seasonal or successional dynamics [58];) in response to environmental change. The observed shifts in relative abundance, rather than the turnover of entire taxonomic groups, highlight the plasticity of microbial communities in adapting to changing conditions. Although shifts in taxonomic composition are evident, it remains unclear whether these changes might point to significant functional differences or if functional redundancy ensures stability across ecosystems. Previous studies have shown that microbial communities can undergo taxonomic restructuring while maintaining metabolic functions [3]. To address this issue, future studies should integrate multiomics approaches, such as metatranscriptomics and metaproteomics, to determine whether compositional shifts, particularly among influential rare taxa like CPR and DPANN, translate into functional changes in ecosystem processes. Focusing on pathways related to nutrient acquisition, energy metabolism (considering their reduced genomes), and potential host-associated interactions would be particularly insightful. This will be crucial for understanding microbial communities’ resilience and ability to maintain ecosystem functions despite taxonomic variation.

We found an inverse latitudinal pattern in taxon richness, with temperate zones harboring higher CPR and DPANN phyla richness. This pattern is uncommon for macro-organisms but appears to be prevalent among microorganisms [59, 60]. One possible explanation is that temperate areas have lower stability and energy availability than tropical areas [61]. In addition, dispersal limitations and species selection [60] may also play a role in shaping this pattern. Another plausible explanation lies in host–pathogen interactions, as temperate regions may accommodate greater pathogen diversity [62] and host specificity [63, 64]. In this context, CPR and DPANN might be better suited to thrive in low-energy environments due to their metabolic capabilities and/or ecological strategies. Additionally, they may face more barriers and filters to dispersal, resulting in greater isolation and specialization, and potentially benefit from pathogen-host interactions if they act as symbionts, parasites, or pathogens within microbial communities. Yet, it is notorious that global metagenomic datasets reflect historical sampling priorities, resulting in uneven geographic representation across ecosystems. In particular, tropical and other under-explored regions, which remain undersampled, and may constrain the resolution and generalization of global-scale ecological inferences [65].

CPR and DPANN taxa may primarily exploit specialized ecological niches, potentially constraining their ability to compete for resources. In contrast, the dominance of taxa such as Proteobacteria [66] and Cyanobacteria [67] within marine ecosystems underscores their enhanced adaptability and a robust ability to compete for limited resources. This disparity in competitive dynamics might be the main factor contributing to the lower abundance of CPR and DPANN taxa, thus influencing the microbial diversity across marine ecosystems. Whilst CPRs had a higher richness in holobiont communities, their abundance was higher in free-living communities, driven by soil ecosystems, likely due to their ability to supply essential secondary metabolic components [68] and a broader symbiotic range [69].

Additionally, free-living communities, particularly those in soil and aquatic environments, [70, 71] are subjected to abiotic variations, such as temperature shifts, moisture variability, and changes in pH and salinity. These environments also experience frequent physical disturbances such as erosion, sediment resuspension, and runoff, as well as high rates of stochastic colonization and extinction events, which collectively reduce the temporal stability of community composition and function [72, 73]. In contrast, host-associated microbial communities often occupy buffered microenvironments, where the host provides physical protection, stable nutrient availability, and regulated physicochemical conditions, promoting a more stable environment for microbes [74, 75]. Such specificity can be problematic, given that most of the taxa with influences up to 200% higher than most of the taxa on the communities were rare (with relative abundances < 1%), making the ecosystems where they play a potential keystone role more susceptible to natural and anthropogenic disturbances.

In contrast, stable and isolated ecosystems such as aquifers provide environments with reduced disturbance and competition, allowing CPR and DPANN taxa to thrive. Their presence in nutrient-limited environments suggests they may possess unique metabolic capabilities that enable them to exploit scarce resources efficiently or engage in specialized metabolic pathways that provide advantages in challenging environments [68, 76]. However, their observed abundance in aquifers may also be biased by historical research focus, as they were first discovered in these environments, leading to intensified efforts to find them there compared to other ecosystems. The high mean contribution to community dissimilarity of some of these taxa across the environments suggests they can have a disproportionate importance to communities’ stability, possibly acting as keystone species.

Our study draws on a large and taxonomically diverse collection of publicly available metagenomes to investigate broad-scale patterns in microbial community structure, with particular attention to low-abundance taxa. At the same time, it underscores important limitations that highlight the need for robust macroecological frameworks in microbial ecology. Detecting ultra-small and host-associated lineages such as CPR and DPANN remains inherently challenging due to their reduced genomes, genomic fragmentation, and frequent host dependence [17, 77] factors that contribute to their under-representation in sequencing outputs and reference databases. To address these constraints, we filtered out samples with low phylum richness, expanded our custom reference database to include underrepresented lineages, and accounted for potential metadata-driven biases using random forest models. While PI identity and other technical metadata were included to evaluate potential methodological bias, we acknowledge that these variables may be confounded with ecological factors, such as habitat or geographic focus. As such, their importance in the model should not be interpreted causally but rather as a means to test for possible technical artifacts within a heterogeneous public dataset. Furthermore, while compositional dissimilarity metrics such as SIMPER and Bray–Curtis effectively characterize structural variation, they do not capture functional or transcriptional dynamics. These limitations underscore the need for integrative approaches combining metagenomics with functional, transcriptomic, and experimental data, alongside standardized sampling strategies to clarify the ecological roles of rare microbial taxa and their influence in microbial resilience.

Conclusion

Our analysis, grounded in a custom-curated metagenomic database and bias-aware filtering framework, reveals that low-abundance microbial taxa, particularly CPR and DPANN lineages, can exert a disproportionate influence on community structure dissimilarities across global ecosystems. While abundant taxa predictably shape broad compositional trends, we show that specific rare phyla contribute disproportionately to ecosystem-level differentiation, challenging abundance-centric paradigms in microbial ecology. The deviation from the classical latitudinal diversity gradient, once the richness of these lineages’ peaks in temperate rather than tropical regions, together with contrasting patterns across free-living and host-associated lifestyles, underscores their likely niche specialization driven by distinct environmental constraints, dispersal limitations, and biotic interactions. These findings highlight that members of the rare biosphere may play non-random, context-dependent roles in shaping microbial biogeography and ecosystem functioning. We contend that achieving a comprehensive understanding of microbial community assembly demands explicit consideration of low-abundance taxa alongside dominant groups. Yet, we recognize that fully elucidating the ecological significance of these lineages will require targeted functional validation through integrative multi-omics approaches and experimental investigations.

Supplementary Information

Below is the link to the electronic supplementary material.

Author Contribution

PMM conceptualized and designed the study, interpreted the data, and wrote the manuscript. CMF designed and created the analytical pipeline, analyzed and interpreted the data, and wrote the results. DBA collaborated to conceptualize and develop the contribution to community dissimilarities analysis. DBA, ABC, RM, APA, RFSA, PRG, AR, and AMT collaborated to conceptualize the analytical approach and pipeline. DBA, LB, FA, and PAV collected and filtered the data, created the bioinformatic pipeline, and organized all codes used in the study into a cohesive and streamlined pipeline. FHC, TB, RB, and PIPR reviewed the design and results. All authors equally contributed to the manuscript revision and final version.

Funding

This work was primarily supported by the Serrapilheira Institute (grant number Serra-1709–17818). CMF was supported in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior—Brazil (CAPES)—88887–468244-2019–00 and IN-TREE—National Council for Scientific and Technological Development—Brazil (CNPq)—114693/2022–6. PMM thanks the PROPESQ-UFBA (11268). FASB was supported by PIBIC/UFBA–FAPESB (4102/2019). We acknowledge the National Laboratory for Scientific Computing (LNCC/MCTI, Brazil) for providing the HPC resources of the Santos Dumont supercomputer, which enabled the development of the analysis and results reported within this article. TB is supported by Rede Nordeste de Biotecnologia (RENORBIO), Biotechnology Graduate Program. F. H. C. was supported by a Ramón y Cajal fellowship (RYC2022-037094-I) and by a Juan de la Cierva—Incorporación fellowship (Grant IJC2019-039859-I) from the Spanish Ministry of Science and Innovation. As a member of the ICM, F. H. C. had the institutional support of the “Severo Ochoa Centre of Excellence” accreditation (CEX2019-000928-S) funded by AEI / 10.13039/501100011033. F. H. C. was granted access to the Galician Supercomputing Center (CESGA) infrastructure where some of the data analysis was performed. The supercomputer FinisTerrae III and its permanent data storage system have been funded by the Spanish Ministry of Science and Innovation, the Galician Government and the European Regional Development Fund (ERDF).

Data Availability

All data and codes used in the present study are fully available in the MeirellesLab github at https://github.com/MeirellesLab/CPR_and_DPANN_global_overview_paper.

Declarations

Ethics Approval and Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Competing Interests

The authors declare no competing interests.