Identifying keystone species in microbial communities using deep learning

Abstract

Previous studies suggested that microbial communities can harbor keystone species whose removal can cause a dramatic shift in microbiome structure and functioning. Yet, an efficient method to systematically identify keystone species in microbial communities is still lacking. Here we propose a Data-driven Keystone species Identification (DKI) framework based on deep learning to resolve this challenge. Our key idea is to implicitly learn the assembly rules of microbial communities from a particular habitat by training a deep-learning model using microbiome samples collected from this habitat. The well-trained deep-learning model enables us to quantify the community-specific keystoneness of each species in any microbiome sample from this habitat by conducting a thought experiment on species removal. We systematically validated this DKI framework using synthetic data and applied DKI to analyze real data. We found that those taxa with high median keystoneness across different communities display strong community specificity. The presented DKI framework demonstrates the power of machine learning in tackling a fundamental problem in community ecology, paving the way for the data-driven management of complex microbial communities.

The notion of keystone species has its roots in food-web ecology^1,2. Since Paine coined it in describing results from his pioneering field experiments in 1969, the notion of keystone species has been widely applied in the ecological literature. Such a broad application (and often abuse) has generated considerable confusion about what precisely a keystone species is³. Here, we adopt the original definition by Paine, i.e., a keystone species is a species that has a disproportionately large effect on the stability of the community relative to its abundance^1,2. Existing methods to identify keystone species for macro ecosystems can be classified into two approaches: experimental manipulations and statistical comparisions⁴.

Previous studies also suggest that microbial communities harbor keystone species^5–9. Yet, the keystone species identification approaches developed for macro ecosystems are challenging to apply to large, complex microbial communities^5–9. For experimental manipulations, targeted removal of each species in a complex community is impossible with current antimicrobial techniques, not to mention the corresponding ethical concerns for host-associated microbial communities such as the human gut microbiome. As for statistical comparisons, finding two communities that differ by just one species is challenging, especially for complex host-associated microbial communities (e.g., the human gut microbiome) with very personalized compositions^10,11. Moreover, statistical comparisons can suffer from numerous confounding factors¹². To resolve the above limitations, one may consider directly inferring a population dynamics model to predict the temporal behavior of microbial communities and then identify keystone species through numerical simulations of targeted species removal. Yet, model misspecification and high-quality absolute abundance data requirement^13–15 for those dynamics inference methods limit their application for identifying keystone species in large, complex microbial communities.

A recent numerical study¹⁶ claimed that those highly connected (i.e., “hubs”) and high betweenness centrality species in the microbial correlation network are keystone species of microbial communities^6,17. Despite the popularity and interpretability of the correlation network approach, we think this claim is problematic for at least two reasons. First, edges in microbial correlation networks do not represent direct ecological interactions but just statistically significant co-occurrences or mutual exclusions of species. Second, the impact of a species’ removal naturally depends on the resident community. This underscores a fundamental challenge in the keystone species identification --- the community specificity, i.e., a species may be a keystone in one community but not necessarily a keystone in another community, which is completely ignored based on degree, betweenness or any topological indices in the correlation (or ecological) network.

So far, very few microbial species have been experimentally confirmed as keystones^5,18–20. An efficient method to systematically identify community-specific keystone species in complex microbial communities is still lacking^21–23. In fact, we even lack a widely accepted operational definition of keystoneness --- an index to quantify the role of a species to be a keystone. In this work, we first propose an operational definition of keystoneness for microbial species based on commonly available relative abundance data. Then we proposed a Data-driven Keystone species Identification (DKI) framework to compute the keystoneness. The DKI framework does not assume any particular ecological model, naturally avoiding the model misspecification issue. Moreover, the DKI framework quantifies the keystoneness of each species for each community (sample). Hence, it naturally considers the community specificity of keystoneness.

Results

An operational definition of keystoneness for microbial communities.

Consider a microbiome sample or microbial community $s=(\mathit{z},\mathit{p})$. Here, the species assemblage of the community $s$ is represented by a binary vector $\mathit{z}\in \{\mathrm{0,1}{\}}^N$ with the $i$-th entry $z_i=1$ (or $0$) if species-$i$ is present (or absent) in $s$. The microbial composition or taxonomic profile of this community is characterized by a compositional vector $\mathit{p}\in {\Delta }^N$ with the $i$-th entry $p_i$ representing the relative abundance of species-$i$ in $s$ and ${\Delta }^N$ is the probability simplex. Inspired by the keystoneness definition in macroecology²⁴, we defined the keystoneness of species in microbial communities as the product of two components: the impact component and the biomass component.

More specifically, we defined the structural keystoneness of species-$i$ in a community $s=(\mathit{z},\mathit{p})$ as

$$K_{\mathrm{s}}(i,s)\equiv d\left(\tilde{\mathit{p}},\stackrel{-}{\mathit{p}}\right)\left(1-p_i\right),$$ [1]

where the impact component $d(\tilde{\mathit{p}},\stackrel{-}{\mathit{p}})$ quantifies the structural impact of species-$i$’s removal on the community $s$, while the biomass component $(1-p_i)$ captures how disproportionate is this impact.

For the impact component, we quantified the impact of species-$i$’s removal on the structure of community $s$ as the dissimilarity between the taxonomic profiles $\tilde{\mathit{p}}$ and $\stackrel{-}{\mathit{p}}$, i.e., $d\left(\tilde{\mathit{p}},\stackrel{-}{\mathit{p}}\right)$. Here, $\tilde{\mathit{p}}$ is the new community composition after species-$i$’s removal. $\stackrel{-}{\mathit{p}}$ is the null composition, which is obtained by assuming that species-$i$’s removal will not affect any other species (see Methods).

Similarly, we define the functional keystoneness of species-$i$ in a community $s=(\mathit{z},\mathit{p})$ as

$$K_{\mathrm{f}}\left(i,s\right)\equiv d\left(\tilde{\mathit{f}},\stackrel{-}{\mathit{f}}\right)\left(1-p_i\right).$$ [2]

Here, the dissimilarity between the functional profiles $\tilde{\mathit{f}}$ and $\stackrel{-}{\mathit{f}}$, i.e., $d\left(\tilde{\mathit{f}},\stackrel{-}{\mathit{f}}\right)$, captures the impact of species-$i$’s removal on the function of community $s.\tilde{\mathit{f}}$ (or $\stackrel{-}{\mathit{f}}$) can be computed from $\tilde{\mathit{p}}$ (or $\stackrel{-}{\mathit{p}}$) and the genomic content network (GCN)²⁵, respectively (see Methods).

We emphasize that the structural (or functional) keystoneness defined here is community-specific, which is fundamentally different from those topological indices used in the food web and other ecological systems²⁶.

The DKI framework.

Consider a particular habitat (or meta-community) that harbors a pool of $N$ different microbial species, denoted as $\Omega =\{1,\cdots ,N\}$. Suppose we have a large set of microbiome samples $𝒮=\{1,\dots ,M\}$ collected from this habitat. A microbiome sample $s\in 𝒮$ can be viewed as a local community of the habitat. We assume the collected samples roughly represent the steady states of the local communities so that they can be used to learn the assembly rules of those communities.

The DKI framework consists of two phases. In the first phase (Fig.1a), we implicitly learn the assembly rules of microbial communities in this habitat using a deep-learning method with $𝒮$ as the training data. This is achieved by learning a map from the species assemblage $\mathit{z}$ of a sample $s=(\mathit{z},\mathit{p})$ to its taxonomic profile $\mathit{p}$, i.e., $\phi :\mathit{z}\mapsto \mathit{p}$. Various deep-learning methods, including Multi-Layer Perceptron²⁷ (MLP) or ResNet²⁸ can be used to learn such map without using any population dynamic model, but with a few reasonable assumptions (e.g., the universality of microbial dynamics, steady-state samples, no true multi-stability, and enough training samples. See SI Sec.1 for details) to ensure the problem is mathematically well-defined. Here, based on our previous work²⁹, we developed cNODE2 (composition Neural Ordinary Differential Equation version 2.0) to learn the map $\phi$ (see SI Sec.1 for details). Learning this map $\phi$ will enable us to predict what will happen to the taxonomic profile of a local community upon any species removal.

Figure 1: Workflow of the Data-driven Keystone species Identification (DKI) framework.

a, The species assemblage of a microbiome sample $s$ is represented by a binary vector $\mathit{z}\in \{\mathrm{0,1}{\}}^N$, where its $i$-th entry satisfies $z_i=1(z_i=0$) if species-$i$ is present (or absent) in this sample. The microbial composition of this sample is characterized by a vector $\mathit{p}\in {\Delta }^N$, where its $i$-th entry $p_i$ is the relative abundance of species-$i$ in this sample and ${\Delta }^N$ is the probability simplex. A deep learning model (cNODE2) is trained to learn the map: $\mathit{z}\in \{\mathrm{0,1}{\}}^N\mapsto \mathit{p}\in {\Delta }^N$. b, We conduct a thought removal experiment of species-$i$. In particular, for the community $s=(\mathit{z},\mathit{p})$ with species collection $\mathit{z}$ and microbial composition $\mathit{p}$, we remove species-$i$ from $\mathit{z}$ to form a new species collection $\tilde{\mathit{z}}=\mathit{z}\backslash i$. Then, for the new species collection $\tilde{\mathit{z}}$, we use cNODE2 to predict its new composition $\tilde{\mathit{p}}=\phi \left(\tilde{\mathit{z}}\right)$. To quantify the impact of species-$i$’s removal, we need to compare the new composition $\tilde{\mathit{p}}$ with a null composition $\stackrel{-}{\mathit{p}}$ in the absence of species-$i$ (obtained by assuming species-$i$’s removal will not affect other species’ abundances at all). The structural impact of species-$i$’s removal on the community $s=(\mathit{z},\mathit{p})$ can be defined as the distance or dissimilarity between $\tilde{\mathit{p}}$ and $\stackrel{-}{\mathit{p}}$, i.e., $d(\tilde{\mathit{p}},\stackrel{-}{\mathit{p}})$. Similarly, the functional impact of species-$i$’s removal on the community $s=(\mathit{z},\mathit{p})$ can be defined as the distance or dissimilarity between $\tilde{\mathit{f}}$ and $\stackrel{-}{\mathit{f}}$, i.e., $d(\tilde{\mathit{f}},\stackrel{-}{\mathit{f}})$. Here, the functional profile $\tilde{\mathit{f}}$ (or $\stackrel{-}{\mathit{f}}$) can be computed by multiplying $\tilde{\mathit{p}}$ (or $\stackrel{-}{\mathit{p}}$) with the incidence matrix of the genomic content network (GCN), respectively.

In the second phase (Fig.1b), to quantify the community-specific keystoneness of species-$i$ in a local community or microbiome sample $s$, we conduct a thought experiment of removing species-$i$ from $s$ and use cNODE2 to compute the impact of species-$i$’s removal on $s$. In particular, for $s=(\mathit{z},\mathit{p})$ with species collection $\mathit{z}$ and microbial composition $\mathit{p}$, we remove species $i$ from $\mathit{z}$ to form a new species collection $\tilde{\mathit{z}}=\mathit{z}\backslash i$. Then, for the new species collection $\tilde{\mathit{z}}$, we use cNODE2 to predict its composition $\tilde{\mathit{p}}=\phi \left(\tilde{\mathit{z}}\right)$. To quantify the impact of species-$i$’s removal, we need to compare the new composition $\tilde{\mathit{p}}$ with the null composition $\stackrel{-}{\mathit{p}}$ (with ${\stackrel{-}{p}}_i=0$ and ${\stackrel{-}{p}}_j=p_j/{\sum }_{k\ne i}{p}_k$ for $j\ne i$). In reality, the map $\phi$ cannot be perfectly learned, and the composition prediction always contains some error. To take this into account, we can compute the null composition $\stackrel{-}{\mathit{p}}$ by renormalizing the relative abundances of the remaining species from the predicted composition of the original community, i.e., ${\stackrel{-}{p}}_j=p_j^{\prime }/{\sum }_{k\ne i}{p}_k^{\prime }$, where ${\mathit{p}}^{\prime }=\left(p_k^{\prime }\right)=\phi \left(\mathit{z}\right)$. This way, the prediction errors in $\tilde{\mathit{p}}$ and $\stackrel{-}{\mathit{p}}$ will be canceled to some extent, and hence the predicted keystoneness will be more accurate (see Fig.S1). From the predicted $\tilde{\mathit{p}}$ and $\stackrel{-}{\mathit{p}}$, we can compute functional profiles $\tilde{\mathit{f}}$ and $\stackrel{-}{\mathit{f}}$, and then compute both the structural keystoneness and the functional keystoneness of species-$i$ in the community $s=(\mathit{z},\mathit{p})$.

Validation of DKI framework using synthetic dataset.

To demonstrate DKI’s performance in the keystoneness prediction, we generated synthetic data using the Generalized Lotka-Volterra (GLV) model with $N=100$ species in the species pool (meta-community). The initial species collection of each sample (local community) consists of 50 species randomly drawn from the species pool (see SI Sec.2). We characterized the population dynamics of the meta-community using two parameters: (1) The connectivity $C$ of the underlying ecological network (which encodes all the pairwise inter-species interactions), representing the probability that two species interact directly. (2) The characteristic interaction strength $\sigma$ represents the typical impact of one species over the per-capita growth rate of another species if they interact. To introduce keystone species to the local communities, inspired by a previous study²³, we amplified all the link weights (inter-species interactions) to ${\tilde{a}}_{ij}={\theta }_{ij}{a}_{ij}$, where ${\theta }_{ij}$ is randomly drawn from a log-normal distribution with mean 0 and standard deviation $\eta$. This will generate a few strong interactions, presumably leading to a few species with high keystoneness. Hereafter, we refer $\eta$ as the characteristic amplification coefficient.

We first trained cNODE2 to minimize the loss function defined as the mean Bray-Cutis dissimilarity between the true and predicted compositions for all samples (see SI Sec.1). Then, we evaluated DKI using all possible new species collection $\tilde{\mathit{z}}$ by removing each of present species in each sample. We systematically evaluated the performance of DKI in predicting the structural keystoneness $K_s$ using simulated data generated from the GLV model with different values for the parameter pair ($C,\eta$). We found that DKI can accurately predict the $K_s$ of each species over different samples for a wide range of $C$ or $\eta$ values (see Fig.2a–f). The Spearman correlation $\rho$ between the true $K_s$ (calculated from the simulated species removal process in the GLV model) and the predicted $K_s$ is around 0.97 with p-value<0.001 (two-sided t test). See SI Figs.S2–S3 and Sec.1.3 for a discussion of performance of DKI with other factors.

Figure 2: In silico validation of the DKI framework.

Results are obtained for the pools of $N=100$ species with Generalized Lotka-Volterra population dynamics. We generate 500 samples to validate the performance of DKI. The population dynamics is characterized by two parameters: the network connectivity $C>0$ and boosting strength $\eta >0$. Predicted structural keystoneness and true structural keystoneness with network connectivity $C=0.3$ (a), $C=0.5$ (b) and $C=0.7$ (c) or boosting strength $\eta =0.5$ (d), $\eta =1$ (e) and $\eta =1.2$ (f). Predicted functional keystoneness and true functional keystoneness with network connectivity $C=0.3$ (g), $C=0.5$ (h) and $C=0.7$ (i) or boosting strength $\eta =0.5$ (j), $\eta =1$ (k) and $\eta =1.2$ (l). For different network connectivities, characteristic interaction strength $\sigma =0.01$ and boosting strength $\eta =1$. For different boosting strengths, characteristic interaction strength $\sigma =0.01$ and network connectivity $C=0.4$. In each panel, we show the Spearman correlation ($\rho$) between the predicted and true keystoneness values, and the p-value. The p-values for the Spearman correlation coefficients ($\rho$) were obtained with a two-sided t test.

To calculate the functional keystoneness using the synthetic data, we generated a random GCN displaying nested structure (with Nestedness metric based on Overlap and Decreasing Fill³⁰ $\text{NODF}=0.31$) for 100 species and 500 genes. We found that DKI can also accurately predict the $K_f$ of each species over different samples for a wide range of $C$ or $\eta$ values (see Fig.2g–l). The Spearman correlation $\rho$ between the true $K_f$ (calculated from the simulated species removal process in the GLV model and the randomly generated GCN) and the predicted $K_f$ is around 0.96 with p-value<0.001.

We emphasize that each species’ structural (or functional) keystoneness is context-dependent or community-specific. Yet, existing methods, especially those based on topological indices of correlation (or ecological) networks constructed (or inferred) from a collection of samples, cannot offer community-specific keystoneness. Moreover, those topological measures do not correlate with each species’ structural (or functional) keystoneness. To demonstrate this point, we generated synthetic data (see SI Sec.1) and compared the structural keystoneness $K_{\mathrm{s}}$ (calculated from the simulated species removal process in the GLV model) with two classical topological indices, i.e., degree (the number of species connected with the species under consideration) and betweenness (the frequency of the species under consideration on the shortest paths connecting all pairs of other species) in the directed ecological network or the undirected correlation network. As shown in Fig.3 and Fig.S4, the two topological indices do not correlate with $K_{\mathrm{s}}$ at all, regardless of using the correlation network (Fig.3) or the ecological network (see Fig.S4 and SI Sec.3 for details).

Figure 3: Traditional topological indices calculated from the undirected correlation network do not correlate with structural keystoneness.

Synthetic samples (taxonomic profiles) were generated from the GLV model with $N=100$ species in the species pool. The initial species collection of each sample (local community) consists of 50 species randomly drawn from the species pool (see SI Sec.1). The structural keystoneness $K_{\mathrm{s}}$ of each species in each sample was calculated from the simulated species removal process in the GLV model. Two traditional topological indices (betweenness and degree) of each species were calculated from the correlation network of species abundances constructed using sparCC⁴³ with threshold 0.1. a-c, Structural keystoneness vs. betweenness. The ecological network connectivity $C=0.3$ (a), $C=0.5$ (b), $C=0.7$ (c). The characteristic interaction strength $\sigma =0.01$ and the boosting strength $\eta =1$. d-f, Structural keystoneness vs. betweenness with $\eta =0.5$ (d), $\eta =1.0$ (e) and $\eta =1.2$ (f). $C=0.4$ and $\sigma =0.01$. g-i, Structural keystoneness vs. degree. $C=0.3$ (g), $C=0.5$ (h), $C=0.7$ (i). $\sigma =0.01$ and $\eta =1$. j-l, Structural keystoneness vs. degree with $\eta =0.5$ (j), $\eta =1.0$ (k) and $\eta =1.2$ (l). $C=0.4$ and $\sigma =0.01$.

Validation of DKI using in vitro synthetic microbial community.

We then validated our DKI framework using data from an in vitro study of synthetic microbial communities comprising of up to eight soil bacterial species: Enterobacter aerogenes (Ea), Serratia marcescens (Sm), Pseudomonas citronellolis (Pci), Pseudomonas putida (Pp), Pseudomonas aurantiaca (Pa), Pseudomonas chlororaphis (Pch), Pseudomonas fluorescens (Pf), Pseudomonas veronii (Pv). Those communities involved 101 different species combinations: all eight solos, 28 duos, 56 trios, all eight septets, and one octet³¹. To validate the DKI, we used duos and trios (in total 42 species combinations). For each sample with species collection $\mathit{z}$, we examined whether there is a corresponding sample with species collection $\tilde{\mathit{z}}=\mathit{z}\backslash i$. In total 56 sample pairs were identified to test the prediction of DKI. For each sample pair ($\mathit{z},\tilde{\mathit{z}}$), we used the ground truth sample $\tilde{\mathit{z}}$ as the test set and the remaining samples were used to train the DKI. We found that the keystoneness of each species displays strong community-specificity for most of the species (Fig.4a). Then, we compared the predicted keystoneness of each species with its true keystoneness, finding that the keystoneness predicted by DKI is consistent with the true keystoneness (Spearman correlation $\rho =0.85$, p-value<0.001, Fig.4b). Importantly, the two topological indices (i.e., degree and betweenness) do not correlate with $K_{\mathrm{s}}$ at all (Fig.4c,d). To examine the sensitivity of each method in keystone identification, we considered the species with the highest true keystoneness as keystone species in each community, and the predicted keystone as the species with the highest predicted keystoneness or topological indices. We found that DKI yields the highest accuracy 0.85 (Fig.4e) than degree (accuracy 0.4, Fig.4f) and betweenness (accuracy 0.25, Fig.4g).

Figure 4: In vitro validation of cNODE in keystoneness prediction.

We analyzed data from a synthetic consortium of 8 soil species: Enterobacter aerogenes (Ea), Serratia marcescens (Sm), Pseudomonas citronellolis (Pci), Pseudomonas putida (Pp), Pseudomonas aurantiaca (Pa), Pseudomonas chlororaphis (Pch), Pseudomonas fluorescens (Pf), Pseudomonas veronii (Pv) to validate DKI. a, True keystoneness of each species in different communities. b-d: Spearman correlations between true keystoneness and DKI-predicted keystoneness (b), degree (c), and betweenness (d) were calculated. Shaded areas indicate 95% confidence intervals of the regression line. e-g: True keystone species in each sample (yellow square) and DKI-predicted keystone (green circle) (e), degree-predicted (f), and betweenness-predicted keystone species (g) were calculated. True (predicted) keystone species is considered as the species with the highest true (predicted) keystoneness. Two traditional topological indices (betweenness and degree) of each species were calculated from the correlation network of species abundances constructed using sparCC⁴³ with threshold 0.05. The p-values for the Spearman correlation coefficients ($\rho$) were obtained with a two-sided t test.

Keystone species in the human microbiome.

We applied the DKI framework to the human gut microbiome data collected in a curated metagenomic database³². We focused on the metagenomic data of stool samples of healthy adults aged between 18 to 65 and without antibiotics usage. In total, we have 2,815 samples involving 1,103 species. Species in RefSeq (NCBI Reference Sequence Database), are defined based on comprehensive, integrated, non-redundant, and well-annotated reference sequences, including genomic, transcript, and protein data.

We first trained cNODE2 by using all the 2,815 samples. Then for each of the 2,815 samples, we computed the structural keystoneness $K_s$ for each species present in the sample. For species present in at least 10% of the samples, we ranked them based on their median structural keystoneness: ${\text{median}(K}_s)$. Fig.5a,b show the top-20 and bottom 20 species, respectively. We found that those species with a higher ${\text{median}(K}_s)$, e.g., Prevotella copri, tend to have a larger variation of their $K_{\mathrm{s}}$ across different samples, suggesting a stronger community specificity (Fig.5a); while those species with a lower ${\text{median}(K}_s)$, e.g., Alistipes finegoldii, tend to have a smaller $K_s$ variation, suggesting a weaker community specificity (Fig.5b). In addition, those species with the highest keystoneness tend to have a much larger biomass component than the impact component (see Fig.S5).

Figure 5: Keystone species in the human gut microbiome.

We applied DKI to a large-scale human gut microbiome dataset collected in curatedMetagenomicData³². This dataset includes 2,815 fecal samples from healthy adults without antibiotics usage. In total, we have 1,103 species. a-b, Structural keystoneness distribution of the top-20 (a) and bottom-20 species (b) ranked by their median structural keystoneness. c, Spearman correlation between the median structural keystoneness and median deviation of keystoneness of each species. d-e, Functional keystoneness distribution of the top-20 (d) and bottom-20 species (e) ranked by their median functional keystoneness. f, Spearman correlation between the median functional keystoneness and median deviation of keystoneness of each species. In panels (a), (b), (d) and (e), the top/bottom-20 species were selected among species present in at least 10% of total samples. The p-values for the Spearman correlation coefficients ($\rho$) were obtained with a two-sided t test.

To systematically explore the community-specificity of those species’ structural keystoneness, we plotted their median keystoneness $\text{median}\left(K_s\right)$ vs. their median absolute deviation of structural keystoneness $\text{MAD}\left(K_s\right)$ over all samples. We found that $\text{MAD}\left(K_s\right)$ is highly correlated with $\text{median}\left(K_s\right)$ with Spearman correlation $\rho =0.87$, p-value<0.001 (Fig.5c). This result indicates that taxa with low median structural keystoneness are unlikely to be keystone taxa in any community. By contrast, taxa with high median keystoneness have high keystoneness (and hence are likely keystone taxa) in some communities, but they can also have small keystoneness in other communities. Similar results were also observed from the human oral microbiome and environmental microbiomes (see SI Secs.4–7 and Figs.S6–8 for details).

We noticed that the largest (structural) keystoneness value is still smaller than 0.2 (see Fig.5a), which is much lower than the upper bound of our keystoneness metric, i.e., 1. This can be explained by the fact that many complex microbial communities (including the human microbiome) typically have high functional redundancy³³, meaning that many phylogenetically unrelated species carry similar genes and perform similar functions. A high level of functional redundancy can be related to the reliability with which an ecosystem will continue to deliver services under moderate species loss. Such functional redundancy has been considered to underlie the stability and resilience of microbial communities. Therefore, a complex microbial community with high functional redundancy will not be so fragile that removing one species will cause a collapse in the services it provides (see Fig.S9).

To compute the functional keystoneness, we constructed a reference GCN (See SI Sec.4 for details). We found that, in general, a species’ functional keystoneness is smaller than its structural keystoneness (see Fig.S10). This is also closely related to the concept of functional redundancy²⁵. Similar to our results on structural keystoneness, we found that those species with a higher ${\text{median}(K}_f)$, e.g., Bifidobacterium bifidum, tend to have a larger variation of their $K_f$ across different samples, suggesting a stronger community specificity (Fig.5d), while those species with a lower $\text{median}\left(K_f\right)$, e.g., Roseburia hominis, tend to have a smaller $K_f$ variation, suggesting a weaker community specificity (Fig.5e). We found that $\text{median}\left(K_f\right)$ and $\text{MAD}\left(K_f\right)$ are strongly correlated: Spearman correlation $\rho =0.88$, p-value<0.001 (Fig.5f).

Based on the ranking of the median structural keystoneness, we found that among those top-ranking species, many of them have been identified as keystone species that carry unique functions and are essential for the maintaining host-microbe hemeostasis³⁴. For example, Bifidobacteria are keystone microorganisms in gut microbiota associated with early life³⁵; Prevotella copri is a keystone species of a healthy human intestinal mucosa³⁶; Faecalibacterium prausnitzii is a keystone species that produces butyrate and its reduced abundance has been associated with Crohn’s disease³⁷; Bifidobacterium longum is a minority species, but influences gut microbiota formation by breaking down complex carbohydrates and providing degradants for other bacterial groups to use³⁸; Ruminococcus bromii plays key roles in promoting the synergistic utilization of resistant starch (RS) by initiating degradation of insoluble RS particles^5,34 (Fig.5a).

Interestingly, some of the species with high median structural keystoneness also have high median functional keystoneness, e.g., Faecalibacterium prausnitzii and Prevotella copri (Fig.5d). Based on the high median functional keystoneness, we also identified some potential keystone species that have been reported to perform important functions. For example, Intestinimonas-like bacteria are important butyrate producers that utilize N-ε-fructosyllysine and lysine in formula-fed infants and adults³⁹.

Discussion

The concept of keystone species has been extensively investigated in ecology. Despite the considerable confusion and difference³, the operational definitions agree that a keystone species disproportionately affects its natural environment relative to its abundance. Systematically identifying keystone species in complex microbial communities is very challenging due to our limited knowledge of the population dynamics of those communities, as well as many logistical and ethical concerns regarding the manipulation of those communities. In this work, we propose a data-driven framework to systematically identify keystone species in complex microbial communities. This framework enables us to compute the structural and functional keystoneness of each species in a community for the first time. Our framework can be used to facilitate data-driven management of complex microbial communities.

We emphasize that the proposed framework is general enough and can be modified in many different ways. For example, instead of using cNODE2, one can use other deep learning models (e.g., Multi-Layer Perceptron (MLP)) to learn the map $\phi :\mathit{z}\mapsto \mathit{p}$ (see Fig.S11). Moreover, instead of using Bray-Curtis dissimilarity, one can use other dissimilarity or distance measures (e.g., the weighted UniFrac distance) to quantify the structural or functional impact of species’ removal (see Fig.S12). One can also use other formulas that combine the impact component and the biomass component to quantify the keystoneness²⁴. In addition, beyond studying the impact of a species’ removal on the community-level functional profile, we can also focus on its impact on any specific microbial function and hence quantify its sensitivity. For instance, one can calculate the relative abundances of a function before and after a species’ removal, respectively (see Fig.S13 and Methods).

Among all the five microbiome datasets analyzed in this work, we found that keystoneness value of in vitro synthetic microbial can be higher than 0.4. However, keystoneness of other four large real microbial communities is much lower than 1. This result could be interpreted as saying there are hardly any keystone species for the studied microbiome samples in those four datasets. Importantly, this conclusion does not imply that there are hardly any keystone species for any microbial community because of the strong community-specificity of keystoneness. Indeed, one may design a synthetic community with a few keystone species present. But for naturally observed microbial communities (regardless of host-associated or host-free), we believe the chance to find keystone species is quite low. In addition, the relative ranking of keystoneness might be more important, instead of absolute value in quantitatively identifying keystone species, e.g., the species with highest median keystoneness across communities are more likely to be keystone species.

We admit there are some caveats in our calculation/interpretation of functional keystoneness (see SI Sec.8). All the functions are potential functions encoded in the microbial genomes. They do not have to be active. To study the true functions, it is necessary to leverage metaproteomics data. It is known that the (protein-level) functional profiles, or the selective expression of proteins, might depend on the community itself. That is some species might express certain proteins to consume certain resources to avoid niche overlap with other species in the community. In other words, the protein-content network (PCN) is community-dependent, while the gene-content network (GCN) is independent of the community⁴⁰.

Methods

Keystoneness calculation.

Inspired by the keystoneness definition in macroecology²⁴, we defined the structural keystoneness of species in microbial communities as the product of two components: the impact component and the biomass component. The impact component is computed as the dissimilarity between the taxonomic profiles $\tilde{\mathit{p}}$ and $\stackrel{-}{\mathit{p}}$, i.e., $d\left(\tilde{\mathit{p}},\stackrel{-}{\mathit{p}}\right)$. Here, $\tilde{\mathit{p}}$ is the new community composition after species-$i$’s removal, while $\stackrel{-}{\mathit{p}}$ is the null composition computed by simply setting ${\stackrel{-}{p}}_i=0$, and renormalizing the relative abundances of the remaining species, i.e., ${\stackrel{-}{p}}_j=p_j/{\sum }_{k\ne i}{p}_k$ for $j\ne i$. Note that $d(\tilde{\mathit{p}},\stackrel{-}{\mathit{p}})$ can be any distance or dissimilarity measure, e.g., the Bray-Curtis dissimilarity. The biomass component is simply computed as $\left(1-p_i\right)$.

Note that our structural keystoneness $K_{\mathrm{s}}\left(i,s\right)=d\left(\tilde{\mathit{p}},\stackrel{-}{\mathit{p}}\right)\left(1-p_i\right)$ has clear lower and upper bounds. The lower bound of $K_{\mathrm{s}}\left(i,s\right)$ is 0, representing the case that species-$i$ is not interacting with any other species in the community, hence its removal will have zero impact on the community. This is of course an extreme case. The upper bound of $K_{\mathrm{s}}\left(i,s\right)$ is 1. This corresponds to another extreme case: species-$i$’s relative abundance $p_i$ is close to 0, and its removal will cause many other species to die out and only a few species survive, and the sum of those survived species’ initial relative abundances is also close to 0. Hence, mathematically, our quantitative metric describes the qualitative keystoneness (in terms of species extinction or system collapse). In fact, our keystoneness definition is more generic in the sense that as long as species-$i$ has a very low relative abundance and its removal will have a very large impact on the community composition (not necessarily leading to many species extinctions), then its keystoneness will be very high.

To compute the functional keystoneness $K_{\mathrm{f}}\left(i,s\right)=d\left(\tilde{\mathit{f}},\stackrel{-}{\mathit{f}}\right)\left(1-p_i\right)$, we need to compute the functional profiles $\tilde{\mathit{f}}$ and $\stackrel{-}{\mathit{f}}$ from their taxonomic profiles $\tilde{\mathit{p}}$ and $\stackrel{-}{\mathit{p}}$, and the genomic content network (GCN)²⁵. Here, the GCN is a weighted bipartite graph connecting the $N$ species to their genes (see SI Sec.4,5). Suppose there are in total $M$ genes in the metagenome of the $N$ species. The GCN can then be represented by an incidence matrix $\mathit{G}=\left(G_{ia}\right)\in {\mathbb{R}}^{N\times M}$, where a non-negative integer $G_{ia}$ indicates the copy number of gene-$a$ in the genome of species-$i$. The gene composition or functional profile $\mathit{f}\in {\Delta }^M$ of a microbiome sample with taxonomic profile $\mathit{p}$ can then be calculated as $\mathit{f}=c\mathit{p}\mathit{G},$ where $c={\left[{\sum }_{a=1}^M{\sum }_{i=1}^N{p}_i{G}_{ia}\right]}^{-1}$ is a normalization constant.

Function sensitivity.

Instead of studying the impact of a species’ removal on the community-level functional profile, we can also focus on the impact on any specific function and hence quantify the sensitivity of the function. This study provides insight into microbial metabolites and associated functional pathways in controlling host physiology³⁴. Here, we quantified the sensitivity of each specific function (i.e., metabolic pathway) in the human gut microbiome by calculating its abundance change, i.e., pathway-$a$’s abundance change caused by species-$i$’s removal in community $s$ is given by: ${{\delta }_a^{(s,i)}=|f}_a^{\left(s\right)}-f_a^{\left(s\right)}(z\backslash i)|/(f_a^{\left(s\right)}+ϵ)$, where $f_a^{\left(s\right)}$ and $f_a^{\left(s\right)}(z\backslash i)$ are the relative abundances of pathway-$a$ before and after species-$i$’s removal, respectively and $ϵ={10}^{-10}$. Then, we ranked the pathways based on their mean change over different species and samples/communities, i.e., $\stackrel{-}{{\delta }_a}={\sum }_{i=1}^N({\sum }_{j=1}^{M_i}{\delta }_a^{(s,i)}/M_i)/N$, where $M_i$ is the prevalence of species $i$.

Data availability.

Gut microbiome data was collected from curatedMetagenomicData⁴¹ database. Oral microbiome data is available at CNGB Sequence Archive (CNSA) of China National GeneBank DataBase (CNGBdb) (CNSA: CNP0000687 for the 4D-SZ cohort and CNP0001221 for the Yunnan cohort). Coral and soil microbiome data were collected from Qiita⁴² (IDs: 10895 and 2104). Source data supporting all our findings are provided with this publication as a Source Data file. Source data are provided at https://github.com/spxuw/DKI.