Structured community transitions explain the switching capacity of microbial systems

Significance

Explaining and predicting qualitative changes in the taxon membership (community) of microbial systems is synonymous for understanding qualitative changes in the environment. Yet, it remains unclear the extent to which the capacity of going back and forth among few different communities is partially conditioned by the current taxon membership or is fully an outcome of random environmental changes. Here, we introduce a conceptual, mathematical, and practical framework to demonstrate that the relatively high switching capacity of microbial systems can be explained by structured community transitions that increase the dependency on the current community. We corroborate our theory using temporal data of human and ocean microbiota, opening the opportunity to enhance our understanding of a wide array of microbial systems.

Abstract

Microbial systems appear to exhibit a relatively high switching capacity of moving back and forth among few dominant communities (taxon memberships). While this switching behavior has been mainly attributed to random environmental factors, it remains unclear the extent to which internal community dynamics affect the switching capacity of microbial systems. Here, we integrate ecological theory and empirical data to demonstrate that structured community transitions increase the dependency of future communities on the current taxon membership, enhancing the switching capacity of microbial systems. Following a structuralist approach, we propose that each community is feasible within a unique domain in environmental parameter space. Then, structured transitions between any two communities can happen with probability proportional to the size of their feasibility domains and inversely proportional to their distance in environmental parameter space—which can be treated as a special case of the gravity model. We detect two broad classes of systems with structured transitions: one class where switching capacity is high across a wide range of community sizes and another class where switching capacity is high only inside a narrow size range. We corroborate our theory using temporal data of gut and oral microbiota (belonging to class 1) as well as vaginal and ocean microbiota (belonging to class 2). These results reveal that the topology of feasibility domains in environmental parameter space is a relevant property to understand the changing behavior of microbial systems. This knowledge can be potentially used to understand the relevant community size at which internal dynamics can be operating in microbial systems.

The behavior and function of ecological systems are defined by the specific membership of interacting taxa present in a given place and time—known as community (1). In microbial systems, the communities can vary quickly, especially under fast-changing environments due to the short generation times of organisms, and their strong context dependencies (2–4). For example, considering a pool of n taxa, the membership (presence–absence) combinations that are possible to be observed grow exponentially with the pool size as $2^n$. Yet, it has been reported (5–7) that these systems tend to switch back and forth among few dominant communities. While changes in microbial systems have been mainly attributed to random environmental factors (8, 9), it remains unclear the extent to which internal community dynamics affect the switching capacity of microbial systems (5, 10). This knowledge is important, for instance, to accurately distinguish risky factors within a host or ecosystem that depend strongly on community dynamics across time (11). This has motivated a solid body of work documenting and studying microbial abundance and composition fluctuations over time (4–7, 9, 12–19).

From a dynamical systems perspective (20), the switching behavior of microbial systems can be described by their capacity to move back and forth among alternative stable states (21). Typically, alternative stable states are defined by situations involving multiple equilibria at a fixed set of environmental conditions (7, 22). Similarly, alternative stable states are also often related to bifurcations leading to sudden changes in the number of states with changing environmental conditions (20, 23). This implies that changes in environmental conditions can combine with multiple equilibria to produce complex switching behaviors in dynamical systems (24). Unfortunately, the virtual impossibility of measuring exact initial conditions of state variables and environmental parameters limits our capacity to separate the potential confounding factors acting on the emergence of alternative stable states (25). This has highlighted the possibility to provide a coarse-grained description of a system without an elaborate examination of the underlying quantitative dynamics (25–27). These limitations and opportunities involved in measuring and understanding nonlinear dynamical systems have motivated the statistical analysis of qualitative behaviors as a potential simplification (28, 29).

The structuralist approach has been instrumental in our qualitative understanding of complex living systems (30–33). The structuralist approach takes as valid that any quantitative change in the state of a living system is causally connected to a change in external conditions. This conceptual simplification enables the construction of a one-to-one mapping between environmental conditions in environmental parameter space and the quantitative states of a system. Additionally, the structuralist approach assumes that the qualitative effects of these external changes (and consequently quantitative changes) are constrained by an invariant structure of interactions among the components of the system. Thus, contrary to quantitative states, the diversity of qualitative states in a system does not form a continuum but is limited by a set of descriptive laws that restrict the possibilities of qualitative change. These restrictions introduce a discrete partition of feasibility domains in environmental parameter space associated with a finite collection of qualitative states. Therefore, one can interpret the probability of observing a particular qualitative state in a system proportional to the size of its feasibility domain in environmental parameter space. This suggests that systems subject to randomly changing environments may spend most of their time in a qualitative state with a large feasibility domain precisely because such a state is more probable (34–36). The structuralist approach does not replace the definition of alternative stable states but simply provides a parsimonious framework to overcome some of the limitations of studying the complex nature of living system by moving the analysis to the probability distribution of qualitative behaviors (25, 33). However, this approach does not enable the possibility to differentiate the probability distribution of quantitative stable states, nor if alternative stable states owe their origin to different initial conditions operating under the exact same environmental conditions.

Following the structuralist approach (Materials and Methods), we investigate whether the switching capacity of qualitative states in microbial systems is explained only by the size of feasibility domains in environmental parameter space (what we call unstructured transitions) or it is also necessary to include the topology of such feasibility domains in environmental parameter space (what we call structured transitions). Unstructured transitions, contrary to structured transitions, would be equivalent to cases where changes in qualitative states are not conditioned by the current taxon membership, but rather, only by environmental stochasticity. Formally, we introduce six basic assumptions to establish a statistical analysis of our structural approach on microbial dynamics: i) Any abiotic and biotic changes external to a focal system are considered part of the environmental parameter space. ii) There is a one-to-one mapping between parameters in environmental parameter space and a focal system’s abundance (quantitative state). iii) Many combinations of parameters in environmental parameter space (and consequently abundance compositions) will result in the same community (taxon membership: qualitative state). iv) The probability of observing a particular community is directly related to the size of its feasibility domain in environmental parameter space. v) Unstructured transitions between any two communities happen proportionally to the size of their feasibility domains and independent of their distance in environmental parameter space. vi) Structured transitions between any two communities happen proportionally to the size of their feasibility domains and inversely proportional to their distance in environmental parameter space—a special case of the gravity model (Materials and Methods) (37–39).

Put simply, picturing the finite partition of the environmental parameter space as a multicolored board, unstructured community transitions are equivalent to throwing a dart randomly over the entire board, where each color represents a different community and the colored area relative to the board size specifies the probability of observing such transitions. By definition, unstructured transitions do not depend on the current community of the system, external perturbations move the system to a possible community in proportion to the size of its feasibility domain in environmental parameter space. Alternatively, structured community transitions are equivalent to throwing a dart randomly over a restricted section of the multicolored board, where the section is relatively close to the current community’s feasibility domain. By definition, structured transitions depend on the current community of the system, external perturbations move the system to a possible community in proportion to the size of its feasibility domain and its location in environmental parameter space. These two processes generate distinct transition matrices under a Markov chain analysis (Materials and Methods), from which we can measure and compare the expected switching capacity generated in a system across different community sizes. We showcase the predictions of our theory by generating partitions of feasibility domains in environmental parameter space following a general (model-free) framework. Then, we corroborate our general predictions using temporal data of human gut, vaginal, and oral microbiota as well as ocean microbiota (5, 6, 12, 40). We discuss how this theory can be applied under ecological models to describe specific partitions of feasibility domains in environmental parameter space.

Theoretical Analysis

Constructing the Topology of Feasibility Domains.

The structuralist approach (31, 41) is rooted in the topology of feasibility domains (partitions) in environmental parameter space. These domains are associated with the realization of possible qualitative states of a system (Fig. 1A). To describe the feasibility domain $D({\mathcal{C}}_i)$ assigned to each qualitative state ${\mathcal{C}}_i$, it is necessary to introduce two parameters: $\mathbf{\Omega }=\{\mathrm{\Omega }({\mathcal{C}}_i)\}$ and $\mathit{d}=\{d({\mathcal{C}}_i,{\mathcal{C}}_j)\}$. The former is a $2^S$-element size vector denoting the volume (or size) of each domain $D({\mathcal{C}}_i)$. The latter is a $2^S\times 2^S$ matrix denoting the distance between centroids ${\mathit{\theta }}_i$ and ${\mathit{\theta }}_j$ of domains $D({\mathcal{C}}_i)$ and $D({\mathcal{C}}_j)$. To establish general predictions from this theory, we study the topology of feasibility domains from random distributions (more details below) and from ecological models (more details in Materials and Methods).

Fig. 1.

Illustration of concepts and measures to estimate the switching capacity of microbial systems. (A) Following a structuralist approach (30, 31), the black lines denote a hypothetical environmental parameter space: set of external parameters compatible with the realization of a given community formed from a given taxon (species) pool. The broken arrows depict the distance in environmental parameter space between two different communities. (B) The transition force between communities (i and j) is modeled as a special case of the gravity model (37–39) written as a function of the size of feasibility domains [(${\mathrm{\Omega }}_i$) and (${\mathrm{\Omega }}_j$)] and inversely related to the distance $d_{\mathit{ij}}$. Note that unstructured (resp. structured) community transitions correspond to the structural parameter $\beta =0$ (resp. $\beta >0$). The transition probability ($\mathbb{P}(j|i)$) is the transition force normalized by the sum of all the forces going out from community i. (C) The calculation of switching capacity is a function of the transition matrix ($H(T)$) and the stationary distribution ($\mathbf{s}$). Note that $T_{\mathit{ii}}$ corresponds to the probability of remaining in the community i. (D) Transition matrix (T) summarizing all the transition probabilities between each pair of possible communities. This transition matrix gives rise to the stationary distribution ($\mathbf{s}$) specifying the long-term frequency of each community. The darker the element, the higher the probability $T_{\mathit{ij}}$ and frequency $s_i$. For each type of transition, we show an illustrative example of a transition matrix (Left) and the corresponding evolution of probability distribution of states/communities (Right). Starting from an initial distribution (${\mu }_0$), unstructured transitions converge immediately to the long-term probability distribution (equilibrium: ${\mathit{s}}_{\mathrm{str}}(\beta )$) shown in Panel A. Instead, starting from the same initial distribution (${\mu }_0$), structured transitions gradually converge (i.e., require many steps) to their unique long-term probability distribution ${\mathit{s}}_{\mathrm{str}}(\beta )$. The larger the structural parameter $\beta$, the larger the distance between ${\mathit{s}}_{\mathrm{str}}(\beta )$ and ${\mathit{s}}_{\mathrm{uns}}$-pagination (SI Appendix, section S1).

To study the topology of feasibility domains from random distributions, we let $\{\mathrm{\Omega }({\mathcal{C}}_i)\}$ be independently drawn from a uniform distribution $U(0,1)$ and normalized such that all elements sum to $1$. Then, we make a list of centroid vectors $\{{\mathit{\theta }}_i\}$, where each has $2^S$ coordinates jointly sampled from a uniform distribution on the S-dimensional unit sphere (36). Finally, we calculate the pairwise distances as $\{d({\mathcal{C}}_i,{\mathcal{C}}_j)\}$. This implies that all the subsets ($2^S$ communities) of the pool $\mathcal{S}$ have a given feasibility domain in environmental parameter space. This random partition of feasibility domains corresponds to a mutually exclusive and exhaustive partition of environmental parameter space. The randomly generated distances assume that each feasibility domain can be located freely at any region within the environmental parameter space with $d({\mathcal{C}}_i,{\mathcal{C}}_i)=0$ (Fig. 1).

Probability of Community Transition.

Formally, a community ${\mathcal{C}}_i$ is a subset of interacting taxa from a constant taxa pool $\mathcal{S}$ of dimension $S=|\mathcal{S}|$ (i.e., ${\mathcal{C}}_i\subseteq \mathcal{S}$). These definitions assume that while there is a relatively constant but rare flux of taxa $\mathcal{S}$ to a given place, the current context defined in the environmental parameter space only allows the sustained growth of the taxa ${\mathcal{C}}_i$. Thus, community transitions happen when the parameters shift from being compatible with ${\mathcal{C}}_i$ to being compatible with ${\mathcal{C}}_j$. In turn, we define the current context in environmental parameter space as an effective vector of growth rates ($\mathit{\theta }$, Fig. 1A). We call this an effective value to denote the phenomenological net effect from all the abiotic and biotic factors not explicitly considered within the focal system. Thus, there could be cases when the effective vector (the current context) is inside the feasibility domain ($D({\mathcal{C}}_i)$) of ${\mathcal{C}}_i$, allowing the realization of such a community.

Next, to account for transition dynamics, we define the transition force of moving from ${\mathcal{C}}_i$ to ${\mathcal{C}}_j$, $\mathbb{F}({\mathcal{C}}_i\to {\mathcal{C}}_j)=f[\mathbf{\Omega },\mathit{d}]$, as the phenomenological interaction strength. In general, $f[·]$ is an arbitrary functional form, similar to the possibility of using or not higher-order interaction terms in Lotka–Volterra models (42, 43). Here, we write $f[·]$ as a special case of the gravity model (37–39) given by

$$\begin{array}{c}\hfill \mathbb{F}({\mathcal{C}}_i\to {\mathcal{C}}_j)=\mathrm{\Omega }({\mathcal{C}}_i)·\mathrm{\Omega }({\mathcal{C}}_j)·exp(-\beta ·d({\mathcal{C}}_i,{\mathcal{C}}_j)),\end{array}$$ [1]

where $\beta$ is a nonnegative structural parameter that penalizes distant community transitions described in the topology of the environmental parameter space. Specifically, we assume that transition forces from ${\mathcal{C}}_i$ to ${\mathcal{C}}_j$ are proportional to the size of each feasibility domain and decay monotonically with distance in environmental parameter space (Fig. 1B).

Then, the probability of observing a transition from community ${\mathcal{C}}_i$ to community ${\mathcal{C}}_j$ is a function of the topology of feasibility domains, formally written as

$$\begin{array}{c}\hfill \mathbb{P}({\mathcal{C}}_j|{\mathcal{C}}_i)=\frac{\mathbb{F}({\mathcal{C}}_i\to {\mathcal{C}}_j)}{\mathbb{F}({\mathcal{C}}_i\to ·)}=\frac{\mathbb{F}({\mathcal{C}}_i\to {\mathcal{C}}_j)}{{\sum }_{k=1}^{2^S}\mathbb{F}({\mathcal{C}}_i\to {\mathcal{C}}_k)}.\end{array}$$ [2]

This definition intuitively implies that the probability of such a transition is the relative ratio of the transition force normalized by the total forces acting on ${\mathcal{C}}_i$ given that $\sum _j\mathbb{P}({\mathcal{C}}_j|{\mathcal{C}}_i)=1$. Different from the transition force, the transition probability does not necessarily preserve the symmetry to exchange ${\mathcal{C}}_i$ and ${\mathcal{C}}_j$. Then, if $\beta =0$, we recover unstructured transitions

$$\begin{array}{c}\hfill {\mathbb{P}}^{\mathrm{uns}}({\mathcal{C}}_j|{\mathcal{C}}_i)=\frac{\mathrm{\Omega }({\mathcal{C}}_i)\mathrm{\Omega }({\mathcal{C}}_j)}{{\sum }_k\mathrm{\Omega }({\mathcal{C}}_i)\mathrm{\Omega }({\mathcal{C}}_k)}=\frac{\mathrm{\Omega }({\mathcal{C}}_j)}{{\sum }_k\mathrm{\Omega }({\mathcal{C}}_k)}=\mathrm{\Omega }({\mathcal{C}}_j),\end{array}$$ [3]

where the probability is independent of both the current community ${\mathcal{C}}_i$ and the topology of feasibility domains in the environmental parameter space. Likewise, if $\beta >0$, we recover structured transitions

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathbb{P}}^{\mathrm{str}}({\mathcal{C}}_j|{\mathcal{C}}_i)& =\frac{\mathrm{\Omega }({\mathcal{C}}_i)\mathrm{\Omega }({\mathcal{C}}_j)exp(-\beta d({\mathcal{C}}_i,{\mathcal{C}}_j))}{{\sum }_k\mathrm{\Omega }({\mathcal{C}}_i)\mathrm{\Omega }({\mathcal{C}}_k)exp(-\beta d({\mathcal{C}}_i,{\mathcal{C}}_k))}\hfill \\ \hfill & =\frac{\mathrm{\Omega }({\mathcal{C}}_j)exp(-\beta d({\mathcal{C}}_i,{\mathcal{C}}_j))}{{\sum }_k\mathrm{\Omega }({\mathcal{C}}_k)exp(-\beta d({\mathcal{C}}_i,{\mathcal{C}}_k))},\hfill \end{array}\end{array}$$ [4]

where the probability does depend on both the current community ${\mathcal{C}}_i$ and the topology of feasibility domains in the environmental parameter space.

It is important to note that unstructured ($\beta =0$) and structured ($\beta >0$) transitions have different stationary probability distributions (Eqs. 3–4). Furthermore, unstructured transitions need just one step to converge to their stationary probability distribution, that is, ${\mathit{s}}_{\mathrm{uns}}=\mathrm{\Omega }$, while structured transitions gradually converge to their unique distributions ${\mathit{s}}_{\mathrm{str}}(\beta )$ (Fig. 1D). The larger the structural parameter $\beta$, the larger the distance between ${\mathit{s}}_{\mathrm{str}}(\beta )$ and ${\mathit{s}}_{\mathrm{uns}}$ (SI Appendix, section S1). Therefore, it is expected that unstructured and structured transitions exhibit different dynamics characterized by their transition matrices, which can then translate into differences in switching capacities.

Measuring Switching Capacity.

Our definition of transition probability enables us to measure the switching capacity: the ability of a system to move back and forth between dominant communities. Formally, $\mathbb{P}({\mathcal{C}}_j|{\mathcal{C}}_i)$ defines a Markov process on a taxon pool $\mathcal{S}$ (44) and specifies the $(i,j)$-th element of a row-normalized Markov transition matrix T (Materials and Methods) (Fig. 1D). Therefore, $T_{\mathit{ii}}=\mathbb{P}({\mathcal{C}}_i|{\mathcal{C}}_i)$ is the probability that the system will stay at the current community ${\mathcal{C}}_i$, and its complement $1-T_{\mathit{ii}}=1-\mathbb{P}({\mathcal{C}}_i|{\mathcal{C}}_i)=\sum _{j\ne i}\mathbb{P}({\mathcal{C}}_j|{\mathcal{C}}_i)$ is the probability that the system will move to any other destination. Then, the switching capacity of the system can be expressed as

$$\begin{array}{l}H(T)=-{\displaystyle \sum _{i=1}^{2^s}}{s}_i[T_{ii}{\log }_2(T_{ii})\\ +(1-T_{ii}){\log }_2(1-T_{ii})]\in [0,1],\end{array}$$ [5]

which represents the uncertainty of transition of each community i (row i of T) weighted by its frequency of occurrence $s_i\in [0,1]$ (Fig. 1D). This frequency vector ($\mathbf{s}$) can be theoretically derived from the stationary distribution of Markov chains (44). Specifically, $T^{\text{T}}\mathbf{s}=\mathbf{s}$, where $\mathbf{s}$ corresponds to the eigenvector with eigenvalue $1$ from the transposed transition matrix $T^{\text{T}}$. Regardless of the pool size, the base 2 of the logarithm establishes that the maximum switching capacity $H(T)=1$ is attained when the system has equal probability of staying or leaving the most frequent (dominant) communities (i.e., $T_{\mathit{ii}}$ close to 0.5 and $s_i$ close to uniform).

Theoretical Results.

Following our theory, the switching capacity of a system is a function of the corresponding partitions of feasibility domains in environmental parameter space. These partitions determine the forces of community transitions, which are themselves a function of the structural parameter $\beta$ and the size of taxon pool S Eq. 1. To illustrate this dependence, we start with 200 random systems (following the methods above) formed by $S=5$ (32 possible communities). For each random system, we compute its corresponding switching capacity $H(T)$ Eq. 5 across different values of the structural parameter $\beta \in [0,8]$.

Fig. 2A shows the distribution of $H(T)$ conditioned on $\beta$. Switching capacity is generally stronger in structured transitions ($\beta >0$, blue symbols) than in unstructured ones ($\beta =0$, orange symbols). This is confirmed with two-sample Kolmogorov–Smirnov tests (P value $<0.001$). Specifically, as $\beta$ becomes larger, the switching capacity initially increases and then decreases (as well as the variance), with both the median and extreme values of $H(T)$ reaching their maximum at ${\beta }_{\mathrm{opt}}=2.50$ (broken vertical line). This implies that H becomes uninformative for large $\beta$ values. This is mainly because large $\beta$ values make even regions with small distances highly penalized, and $T_{\mathit{ij}}(i\ne j)$ becomes extremely small—allowing $T_{\mathit{ii}}$ to take most of the probability due to the normalization and consequently the switching level for each row of T decreases together with $H(T)$. This pattern is robust across a wide range of pool sizes (SI Appendix, section S2). Importantly, the value of the structural parameter that maximizes the switching capacity changes as a linear function of pool size approximately as ${\beta }_{\mathrm{opt}}\sim S/2$ (Fig. 2B). This indicates that the larger the community size, the larger the transition constraints needed to distinguish it from random (unstructured transitions).

Fig. 2.

Theoretical predictions. (A) Each point corresponds to one out of 200 randomly generated transition matrices (T) of a pool $\mathcal{S}$ with 5 taxa (see main text). The x-axis corresponds to different values of the structural parameter $\beta$ modulating the distance effect of transition probabilities according to Eq. 1. Note that $\beta =0$ (resp. $\beta >0$) corresponds to unstructured (resp. structured) community transitions. The y-axis corresponds to the computed switching capacity $H(T)$ (Eq. 5). For $\beta =0$, the median of $H(T)$ is $0.245$ with an IQR (interquartile range) of $[0.237,0.251]$. As a reference, for $\beta =2.50$ (closest to ${\beta }_{\mathrm{opt}}$ for $S=5$, broken vertical line), the median of $H(T)$ is 0.954 with an IQR of $[0.946,0.962]$. (B) ${\beta }_{\mathrm{opt}}$ changes monotonically as a function of community size S approximately as ${\beta }_{\mathrm{opt}}(S)=S/2$. (C) Each point corresponds to one out of 200 randomly generated transition matrices (T) of a pool $\mathcal{S}$ from 4 to 12 taxa (see main text). The Top gray points correspond to structured community transitions with $\beta ={\beta }_{\mathrm{opt}}(S)$. The Middle blue points correspond to structured community transitions with $\beta =2$. The Bottom orange points correspond to unstructured community transitions with $\beta =0$. In general, structured community transitions yield a stronger switching capacity $H(T)$ (y-axis), yet for fixed $\beta$ values, the maximum values of $H(T)$ decrease with pool size (x-axis).

Comparing the switching capacity across different pool sizes, Fig. 2C clearly distinguishes between structured and unstructured transitions. Structured transitions ($\beta >0$, bottom orange symbols) are represented by either a fixed (as a reference, we use $\beta =2$, middle blue symbols) or optimized $\beta ={\beta }_{\mathrm{opt}}(S)$ (top gray symbols) as a function of pool size. Overlaps between these two structured transition cases may occur only when S is small. Qualitative results are not sensitive to the choice of distributions when generating partitions of feasibility domains (SI Appendix, section S3). These results reveal that there are two broad classes of systems following structured transitions: one class where switching capacity is high across a wide range of community sizes structured transitions are a function of pool size, $\beta ={\beta }_{\mathrm{opt}}(S)$, and another class where switching capacity is high only inside a narrow size range (structured transitions are not a function of pool size, $\beta =C$).

Empirical Analysis

Measuring Switching Capacity from Time Series Data.

To corroborate our theory with data from natural microbial communities, we measured switching capacity from publicly available times series data of microbial systems from the human gut, vaginal, and oral microbiota as well as ocean microbiota (5, 6, 12, 40). In each dataset, we have 16S sequencing measurements of relative abundance for each detected operational taxonomic unit (OTU) from multiple individual donors (human microbiota) or locations (ocean microbiota) over time. The gut microbiota dataset contains time series from 80 donors, the longest time series (donor am) comprising 317 samples over effectively 204 d (12) and recording 43 unique microbial families. The vaginal microbiota dataset comprises time series from 32 donors, with samples collected twice a week for 16 wk (5). These data track more than 86 families for each donor. The oral microbiota dataset samples a single subject on a daily basis (6), recording in total 211 microbial families over effectively 273 d. The ocean microbiota dataset focuses on a consecutive 93-d sampling of coastal plankton in three different sampling sites (40), recording a maximum of 228 microbial families on each site. In SI Appendix, section S4, we illustrate the presence–absence data using a principal coordinate analysis, supporting previous observations that microbial communities tend to be characterized by few dominant communities.

To increase the detection level and reduce the dimensionality of the data, we aggregated OTUs by family. We confirmed that aggregating OTUs at different taxonomic levels yielded the same qualitative results (SI Appendix, section S5). To convert the original time series of taxon abundance into a binarized time series, first, we calculate the daily relative abundance of each taxon (in relation to the total abundance of taxa reported in each day), and then, we translated into daily presence–absence observations using ${10}^{-6}$ of relative abundance as a threshold. In SI Appendix, sections S6–S9, we use different sensitivity analyses to validate this procedure. For each donor and location, we compiled all the unique taxa detected across the donor’s entire time series. Because it is virtually impossible to track the exact taxon membership of microbial systems due to technical limitations, and practical interventions are typically targeted to a specific set of taxa, it is then useful to focus on a focal system and take everything external to this system as the environment (i.e., biotic and abiotic conditions). This approach is well aligned with our basic assumptions. Specifically, many combinations of parameters in environmental parameter space (and consequently abundance compositions) will result in the same community (taxon membership: qualitative state). Therefore, while the environment (external set of taxa not explicitly considered as part of a system) can be different, the observed communities from the focal system can be qualitatively the same. Thus, we sub-sampled from each donor’s unique taxon list to generate pools $\mathcal{S}$ with sizes ranging from 3 to 20 (Fig. 3A). Then, we used the time series of the set $\mathcal{S}$ and binarized it to denote presence (one) and absence (zero) (Fig. 3B). For each day, these binarized data specify a given community ${\mathcal{C}}_i\subseteq \mathcal{S}$ formed by the daily presence of taxa from the studied pool $\mathcal{S}$. Therefore, the binarized data provides an empirical time series of community changes, which we used to quantify community transitions and their switching capacities. We assumed that each observed community ${\mathcal{C}}_i$ is feasible, but not necessarily at equilibrium (e.g., can be under transient dynamics) (41).

Fig. 3.

Workflow of empirical analysis. (A) For a given taxon (species) pool of size (S), we randomly choose the studied (focal) taxon using compiled time series data (see main text). Then we compute the relative abundance of the In SI Appendix, sections S6–S9, we use different sensitivity analyses. (B) We binarize the time series data based on presence and absence of taxa, yielding a time series of community transitions. Additionally, as a null model, we randomized the data per day by randomly shuffling the presence of taxa while preserving the number of observed taxa per day. This null model is the equivalent of unstructured transitions established theoretically (Fig. 1). (C) We estimate the observed transition probability ($T_o=\{\mathbb{P}(j|i)\}$) and stationary distribution (${\mathbf{s}}_o=\{s_i\}$) from the frequency of occurrences. Note that the observed (structured) and randomized (unstructured) binarized time series generate different $\mathbb{P}(j|i)$ and $s_i$. We calculate the observed (structured: $H_o(T_o,{\mathbf{s}}_o$)) and randomized (unstructured: $H_r(T_r,{\mathbf{s}}_r$)) switching capacities using the corresponding transition matrices and stationary distributions. This analysis is performed separately for each empirical time series.

To quantify empirical transitions, we calculated the frequency ($f_{\mathit{ij}}$) a composition ${\mathcal{C}}_i$ is followed by another composition ${\mathcal{C}}_j$. Then, the observed transition matrix ($T_o$) and observed stationary distribution (${\mathbf{s}}_o$) are, respectively, given by

$$\begin{array}{c}\hfill T_o[i,j]=\mathbb{P}(j|i)=\frac{f_{\mathit{ij}}}{{\sum }_k{f}_{\mathit{ik}}}\text{and}{\mathbf{s}}_o[i]=\frac{{\sum }_j{f}_{\mathit{ij}}}{{\sum }_{l,k}{f}_{\mathit{lk}}}=\frac{{\sum }_j{f}_{\mathit{ij}}}{m},\end{array}$$ [6]

where $|k|=2^S$ and m is the length of the time series in the data. The stationary distribution can be derived directly from the transition matrix as we do in our theoretical analysis, giving similar results because $T_o$ and ${\mathbf{s}}_o$ are not independent. The observed switching capacity ($H_o(T_o,{\mathbf{s}}_o)$) can be estimated by combining Eq. 5 and Eq. 6. Last, to compare the observed (structured) switching capacity to a randomized (unstructured) case $H_r(T_r,{\mathbf{s}}_r)$, we randomized the original time series by randomly shuffling the presence of taxa while preserving the number of taxa per day. This process assumes that environmental conditions can affect equally the presence of each taxon independently of the community. We then estimate the randomized (unstructured) switching capacity $H_r(T_r,{\mathbf{s}}_r)$ following the same methodology established for $H_o(T_o,{\mathbf{s}}_o)$ as in Eq. 6. It is worth mentioning that empirical results are a function of the length and quality of the observed time series. However, our analysis is not focused on the specific values of statistics, rather it is focused on the general patterns in the distribution of switching capacities. Sensitivity analyses corroborate the robustness of our empirical results (SI Appendix, sections S6–S9).

Empirical Results.

Corroborating our theoretical predictions (shown in Fig. 2), Fig. 4 shows that the studied microbial systems display a stronger switching capacity (for pools with more than 4 taxa, i.e., $S>4$) than the randomized (unstructured) counterparts (i.e., $H_o(T_o,{\mathbf{s}}_o)>H_r(T_r,{\mathbf{s}}_r)$). In Fig. 4, each boxplot corresponds to the distribution of switching capacity (y-axis) generated by 200 randomly chosen family pools with given size (x-axis). This pattern is consistent across a wide range of pool sizes ($S\in [5,12]$) and statistically confirmed with two-sample Kolmogorov–Smirnov tests (P value $<0.001$). In particular, Fig. 4 shows the analysis for donor am (human gut microbiota), subject 5 (human vaginal microbiota), subject A (human oral microbiota), and location 37 (ocean microbiota). All the other donors and locations display similar patterns across a wide range of community sizes (SI Appendix, sections S10 and S11). In SI Appendix, sections S12 and S13, we present evidence that the inferred Markov process from time series data as well as the sampling procedure does not affect our conclusions.

Fig. 4.

Empirical analysis of switching capacity in microbial systems. The analysis is performed using empirical time series of sequenced human gut, vaginal, and oral microbiota as well as ocean microbiota. This analysis is similar to its theoretical counterpart shown in Fig. 2, corroborating our theory. Data were collected and made available in previous publications (5, 6, 12, 40). In each panel, each boxplot consists of 200 data points in total and corresponds to a different pool of size varied between 3 and 20 taxa. These pools are randomly chosen from the time series of 43 families observed in donor am (human gut microbiota, Panel A), from 86 families observed in subject 5 (human vaginal microbiota, Panel B), from 211 families observed in subject A (human oral microbiota, Panel C), and 228 families in location 37 (human ocean microbiota, Panel D). Results are qualitatively the same for different time series, inference, and sampling procedures (SI Appendix, sections S10–S13). The randomized (unstructured) analysis is generated using the surrogate time series such that we randomly shuffle the presence of taxa while keeping the number of observed taxa per day (see main text).

In agreement with our theory, empirical results reveal two broad classes of microbial systems. One class formed by the human gut and oral microbiota (Panels A and C) displays high values of switching capacity across a wide range of community sizes (more aligned to the $\beta ={\beta }_{\mathrm{opt}}(S)$ scenario in Fig. 2). This first class reveals that strong and structured community dynamics can operate at different levels of living matter organization. The second class formed by the human vaginal microbiota and ocean microbiota (Panels B and D) displays high values of switching capacity only inside a narrow range of pool sizes (more aligned to a fixed $\beta >0$ scenario in Fig. 2). This second class is subject to stronger seasonal or periodical environmental changes than the first class, and community transitions become more predictable (lower switching capacity) the higher the pool size. Additionally, we found that the relationship between switching capacities and the diagonal probabilities generated by transition matrices (i.e., the probabilities of remaining in a current state i) follow a nontrivial nonmonotonic pattern that is observed in both theoretical and empirical data (SI Appendix, section S14). These results suggest that different external pressures can give rise to different switching patterns and limit the scope of internal community dynamics in microbial systems.

Conclusions

Living systems are constantly evolving and adapting as a function of environmental changes (4, 32). Microbial systems tend to exhibit relatively stable communities, especially over short timescales (5, 6, 8, 40). This stability appears to be linked with a switching behavior among few dominant communities (5, 6, 8). Following a structuralist approach (30, 31), we have introduced a simplified framework to distinguish the interplay between internal community dynamics and external forces in shaping the switching capacity of microbial systems. We assumed that it is possible to construct an environmental parameter space, where such a space can be partitioned into feasibility domains associated with the realization of each possible community. Then, the probability of observing a given community is proportional to the size of its feasibility domain in environmental parameter space. We established that structured transitions between communities are inversely related to their distance in environmental parameter space. In other words, structured transitions limit the possible changes to nearby communities in environmental parameter space, but changes do not need to happen between similar taxon memberships (it all depends on the topology of feasibility domains in environmental parameter space). Likewise, unstructured transitions are independent of their distance in environmental parameter space, they are only dependent on the size of their feasibility domains. That is, unstructured transitions are not initially restricted, it all depends on environmental stochasticity. Thus, unstructured transitions can be considered as processes that are already at equilibrium (i.e., the long-term probability that the system will be in each state), whereas structured transitions are processes that are gradually moving toward an equilibrium (Materials and Methods). We calculated the switching capacity as the weighted capacity of going back and forth between the system’s dominant communities using measures anchored in information theory (45).

Theoretically, we predicted that structured transitions display a stronger switching capacity than unstructured transitions. Our theoretical results also revealed two broad classes of systems under structured transitions: one class where a high switching capacity happens across a wide range of community sizes and another class where high switching capacity happens only inside a narrow size range. Although we presented our theoretical predictions using partitions of feasibility domains in environmental parameter space following a general (model-free) framework, these partitions can be described by specific models in order to increase our understanding of the topology of feasibility domains in environmental parameter space. For example, leveraging on the scalability and tractability of the generalized Lotka–Volterra (gLV) model, one can construct well-defined partitions of feasibility domains (Materials and Methods). Additionally, from a mechanistic perspective, partitions of feasibility domains can incorporate the functional landscape of microbial consortia, not only their taxon membership (43, 46). Future work may benefit from delving deeper into such broader schemes of partitioning the environmental parameter space and their implications.

We corroborated all our theoretical predictions using time series data on human gut, vaginal, and oral microbiota as well as ocean microbiota. Community transitions and switching capacities were calculated directly from the observed time series. The unstructured community transitions were then generated by randomizing the presence of families in the time series while keeping the number of observed families per day, mimicking a daily effect of unrestricted environmental forces. Our results revealed that the estimated switching capacity was generally higher than the expected capacity under the null model (unstructured case). This implies that we cannot rule out the possibility that the switching capacity of microbial systems is affected by internal community dynamics, and not an outcome derived simply from environmental stochasticity. Specifically, we found that structured community transitions increase the dependency of future communities on the current taxon membership, enhancing the switching capacity of microbial systems. While unstructured transitions converge immediately to a long-term probability distribution of states, structured transitions converge gradually. Importantly, this differentiation of community transitions does not happen across all community sizes. Our empirical results revealed two broad classes of systems following structured transitions. Systems subject to strong periodicity, such as the human vaginal microbiota and ocean microbiota (5, 40) displayed a high switching capacity inside a narrow range of community sizes. Instead, systems subject to less periodicity, such as the human gut and oral microbiota (6, 12) displayed a high switching capacity across a wide range of community sizes. We speculate that these differences may arise as potential trade-offs between adaptability (size of feasibility domain in environmental parameter space) and evolvability (location of the feasibility domain in environmental parameter space) as a response to external perturbations acting on the size and membership of microbial systems (31)—revealing the importance of understanding the topology of feasibility domains in environmental parameter space. This knowledge can be potentially used to understand the relevant community size at which internal dynamics can be operating in microbial systems. Our work introduces a conceptual, mathematical, and practical framework that future work can potentially use and expand to detect anomalous community behaviors within hosts or ecosystems.

Materials and Methods

Structuralist Approach.

To guide the reader into the structuralist perspective, we provide a formal application of this perspective in ecology (36). Let us suppose a generalized gLV system of S populations, where the interaction coefficients are represented by the matrix $\mathbf{A}$ and the per capita growth rates are represented by the vector $\mathbf{r}$. Then, the dynamical system can be described as $\frac{\mathrm{d}\mathbf{N}}{\mathrm{d}\mathbf{t}}=\mathrm{diag}(\mathbf{N})(\mathbf{r}-\mathbf{A}N)$. In the structuralist perspective, the interaction matrix $\mathbf{A}$ can be considered the invariant internal structure, while the growth rate vector $\mathbf{r}$ can be considered as the effective factors (not explicitly considered in the system) acting on the intrinsic growth rate of each population. Then, the environmental parameter space is thus the S-dimensional space of the parameter $\mathbf{r}$. The linearity of the gLV model implies a one-to-one mapping between directions of $\mathbf{r}$ and the quantitative outcome of species abundances at equilibrium, i.e., ${\mathbf{N}}^{\ast }={\mathbf{A}}^{-1}\mathbf{r}$. The community $\mathcal{S}=\{1,2,\cdots ,S\}$ is feasible (${\mathbf{N}}^{\ast }>0$) as long as the effective vector is inside the feasibility domain: $\mathbf{r}\in \mathbf{r}:\mathbf{r}=\mathbf{A}{\mathbf{N}}^{\ast },\forall {\mathbf{N}}^{\ast }>0$. The size of the feasibility domain can be geometrically calculated (41). Moreover, any subset (community) of $\mathcal{S}$ has a feasibility domain (41) located in environmental parameter space (SI Appendix, section S15)

Gravity Model.

The gravity model can be seen as a class of models that describe the general idea of attraction forces between a pair of entities. It starts from Newton’s gravity law, where two bodies with mass $m_1$ and $m_2$ attract each other with a force of $F=G{m}_1{m}_2/r^2$. In general, the attraction force is proportional to the individual weights/sizes/importance and inversely proportional to their distances. The gravity model has been widely used in areas as diverse as transportation (37), disease transmission (38), and genetic inheritance (39), among others. In our work, we take the gravity model form $F=m_1{m}_2/exp(\beta d)$ to write the transition force between communities.

Markov Chain Analysis.

For both unstructured and structured community transitions, the future community at most can only depend on the current community. Thus, any switching process generated by these two types of transitions is a Markov process. Because we care about the $2^S$ possible communities, the Markov process is on finite space. Thus, the transition probability generated from the gravity model has the property that $T[i,j]>0\forall i,j\in \mathcal{S}$, implying that the Markov process is also irreducible. Having an irreducible and finite Markov chain guarantees the existence of a unique stationary distribution $\mathbf{s}$. This reveals that the switching capacity expressed in Eq. 5 as a function of T and $\mathbf{s}$ is a well-defined measure.

Supplementary Material

Appendix 01 (PDF)

Data, Materials, and Software Availability

Previously published data were used for this work (5, 6, 12, 40). The source code to produce the results is available on GitHub at https://github.com/MITEcology/Long_PNAS_2024 (47).