Statistical physics of community ecology: a cavity solution to MacArthur’s consumer resource model

Abstract

A central question in ecology is to understand the ecological processes that shape community structure. Niche-based theories have emphasized the important role played by competition for maintaining species diversity. Many of these insights have been derived using MacArthur’s consumer resource model (MCRM) or its generalizations. Most theoretical work on the MCRM has focused on small ecosystems with a few species and resources. However theoretical insights derived from small ecosystems many not scale up large ecosystems with many resources and species because large systems with many interacting components often display new emergent behaviors that cannot be understood or deduced from analyzing smaller systems. To address these shortcomings, we develop a statistical physics inspired cavity method to analyze MCRM when both the number of species and the number of resources is large. Unlike previous work in this limit, our theory addresses resource dynamics and resource depletion and demonstrates that species generically and consistently perturb their environments and significantly modify available ecological niches. We show how our cavity approach naturally generalizes niche theory to large ecosystems by accounting for the effect of collective phenomena on species invasion and ecological stability. Our theory suggests that such phenomena are a generic feature of large, natural ecosystems and must be taken into account when analyzing and interpreting community structure. It also highlights the important role that statistical-physics inspired approaches can play in furthering our understanding of ecology.

I. INTRODUCTION

One of the most stunning aspects of the natural world is the diversity of species present in most ecosystems. The community structure of ecosystems are shaped through a complex interplay of the externally supplied resources available in an ecosystem, competition for these resources, as well as stochasticity [1–4]. A fundamental problem in community ecology is to understand how these processes give rise to observed pattern of species abundances. A rich theoretical framework has been developed to address this problem. Niche-based theories have emphasized the role of competition for resources [2, 5–10], while neutral theory has highlighted the role of stochastic effects [4, 11–13], and several works have investigated the interplay between stochasticity and competition [14–18].

Many of these theoretical insights have been synthesized in what is commonly referred to as contemporary niche theory. Contemporary niche theory highlights the role played by equalizing mechanisms, processes that decrease fitness differences between organisms, and stabilizing mechanisms, processes that decrease competition for resources. These basic organizational schema have been successfully applied to understand community structure in a wide range of settings [1–3].

One of the simplest and most influential mathematical models for niche theory is MacArthur’s consumer resource model (MCRM) [2, 7, 8, 10]. Most analysis of MCRM – including those that inform contemporary niche theory and modern coexistence theory – have focused on small ecosystems with a few species and and few resources [2, 7, 8, 10]. However, it is unclear to what extent the theoretical insights derived from ecosystems with just a few species can be scaled up to diverse, natural ecosystems. One of the defining features of large complex systems is that they often display new “emergent behaviors” that cannot be understood or deduced from analyzing small systems with just a few parts [19–22]. For this reason, it is essential to directly analyze large ecosystems with many resources and species and ask how they differ from the few-species ecosystems that have been analyzed previously. Recently, several works suggest that large ecosystems can exhibit unexpected behaviors such as phase transitions, emergent community-level cohesion, and the analogues of critical points [15, 23–27]. This highlights the need for new theoretical frameworks for directly analyzing large, heterogeneous ecosystems.

Perhaps the most successful and ubiquitous approaches for analyzing large systems in statistical physics is mean field theory. We emphasize that what is meant by a mean field theories in statistical physics is distinct from the way it is commonly understood in ecology [28, 29]. Unlike most usages in ecology, mean field theories in physics account for not only the means of various quantities but also fluctuations around the mean. In this paper, whenever we use the term mean field theory, we will mean it in this broader statistical physics definition rather than the narrow usage common in ecology. Mean field models have long history in statistical physics and have played a central role in the study of phase transitions and collective emergent behaviors in physical systems [30, 31]. Most mean field theories in physics focus on homogenous systems with identical components and couplings. However, more sophisticated variants such as the cavity method can be used to analyze heterogeneous “disordered systems” [32]. Here, we develop a statistical physics inspired mean field theory, based on a generalization of the cavity method, and use it to analyze diverse ecosystems. In this paper, we will refer to this as the cavity method (CM).

Our methods are inspired by and build upon recent work showing the connection between community ecology the physics of disordered systems [15, 23, 24, 27, 33–38]. It is also closely related to the statistical mechanics of interacting socio-economic agents [39]. However, unlike these previous works our analysis explicitly incorporates resource dynamics, including resource heterogeneity and depletion. This allows us to naturally connect our results to contemporary niche theory and modern coexistence theory. One of the most striking aspects of our analysis is that we nd that modification of fitness and carrying capacity due to collective phenomena is a generic feature of all diverse ecosystems [40]. In diverse ecosystems, organisms can and do significantly reshape their environments by changing resource abundances and, importantly, depleting resources. Moreover, we show that many of the central theoretical quantities in our novel CM have natural ecological interpretations that generalize many classical quantities and results of niche theory to large ecosystems and quantify the effect of collective phenomena in shaping community structure.

II. MACARTHUR CONSUMER RESOURCE MODEL

In this work, we will analyze one of the canonical and most influential models in community ecology: MacArthur’s Consumer Resource Model (MCRM) [7, 8]. MCRM consists of S species or consumers with abundances N_i (i = 1 … S) that can consume one of M substitutable resources with abundances R_α (α = 1 … M). The consumer preferences of species i for resource α are encoded by a S × M matrix, c_iα.

In the MCRM, the growth rate g_i(R) of a species depends of the concentration of all the resources. To model the growth rate, following MacArthur, we assume that a species i have some minimum maintenance cost, m_i, that they must meet. The growth rate, gi(R), is proportional to amount of resources consumed, weighted by a quality factor $w_{\alpha }$, minus this maintenance cost

$$g_i\left(\text{R}\right)=\sum _{\alpha }{C}_{i\alpha }{w}_{\alpha }{R}_{\alpha }-m_i.$$ (1)

If g_i > 0, then this is also the growth rate of species i.

The resources can be considered as “biotic”, i.e. organisms which are themselves being consumed, this is the scenario the original MCRM was designed to describe. These resources have their own internal dynamics which, following MacArthur, we assume can be modeled using logistic growth. Furthermore, when a resource is consumed, it’s abundance is reduced. This ecological dynamics is captured by the following coupled, nonlinear differential equations

$$\begin{array}{l}\frac{d{N}_i}{dt}=N_i{g}_i\left(\text{R}\right)\\ \frac{d{R}_{\alpha }}{dt}=F_{\alpha }\left(R_{\alpha }\right)-\mathbf{\sum }_i{N}_i{c}_{i\alpha }{R}_{\alpha },\end{array}$$ (2)

where F_α (R_α) = R_α (K_α − R_α ) describes the resource dynamics in the absences of consumption and K_α is the carrying capacity of each resource α. Such dynamics are standard when modelling biotic resources. Note also that both the species and resource abundances N_i and R_α must be strictly non-negative. For our analysis of the MCRM, it will be useful to define an “effective resource capacity”

$$K_{\alpha }^{e\text{}f\text{}f}\left(\mathbf{\text{N}}\right)=K_{\alpha }-\sum _i{\mathbf{N}}_i{c}_{i\alpha }$$ (3)

that accounts for depletion of resources by consumers [23]. The MCRM can be rewritten in terms of $K_{\alpha }^{e\text{}f\text{}f}\left(\mathbf{N}\right)$ as

$$\begin{array}{l}\frac{d{N}_i}{dt}=N_i{g}_i\left(\text{R}\right),\\ \frac{d{R}_{\alpha }}{dt}=R_{\alpha }\left(K_{\alpha }^{e\text{}f\text{}f}\left(\mathbf{\text{N}}\right)-R_{\alpha }\right).\end{array}$$ (4)

A crucial property of these equations is that resources can be completely depleted from the environment. This will play an important role in what follows. Finally, we emphasize that these equations are identical those analyzed by MacArthur, Chesson, and others in deriving modern niche theory.

III. STATISTICAL MECHANICS APPROACH TO MACARTHUR’S CONSUMER RESOURCE MODEL

Previous approaches to analyzing the MCRM have largely been confined to small ecosystems with a few species and resources. Here, we consider the opposite limit of large, diverse ecosystems where both the number of species and number of resources is large, S, M ≫ 1. In this limit, the number of parameters needed to define the ecosystem dynamics becomes extremely large. To overcome this problem, we follow a long tradition in theoretical ecology pioneered by Robert May of looking at the case where the parameters are drawn from a random distribution [41]. This allows us to ask questions about the behavior of a generic, diverse ecosystem.

We consider the case where all the consumption coefficients c_iα , resource carrying capacities K_α , and maintenance costs m_i are drawn from a random distribution. Our analytic calculations depend only on the mean and variances of the probability distributions. Denoting the expectation value of a parameters x over a distribution by 〈x〉, then we denote the mean and variances of our parameters by:$〈c_{i\alpha }〉={\mu }_c/S,〈{\left(c_{i\alpha }-〈c_{i\alpha }〉\right)}^2〉={\sigma }_c^2/S,〈m_i〉=m,〈{\left(m_i-〈m〉\right)}^2〉={\sigma }_m^2,〈K_{\alpha }〉=K,$ and $〈{\left(K_{\alpha }-〈K_{\alpha }〉\right)}^2〉={\sigma }_K^2.$ We can also define a parameter γ = M/S that measures the ratio of resources to species. This description is consistent with all species being generalists, that is, feeding from many resources. An examples is a predator which can feed on many different species with each prey making up a small part of its diet. In such a scenario, it is reasonable that the flux from each resources decreases with the number of resources M, through the scaling of c_iα.

A. Invasion, ecological stability, and self-consistency

One of the cornerstones of community ecology is the idea of invasion [6, 42, 43]. In our analysis, we will ask under what circumstances a new species can invade an ecosystem. Denote the growth rate of species i when it tries to invade the ecosystem $g_i^{inv}$ . We will call this the invasion growth rate. Since we are interested in statistical properties, we will be primarily concerned with the mean and variances of the invasion growth rate averaged over all species i in the regional species pool: $〈g_i^{inv}〉=g$ and $〈{\left(g_i^{inv}\right)}^2〉-{〈g_i^{inv}〉}^2={\sigma }_g^2.$

The key idea that we will exploit in our analysis is the observation that as S and M get large, both the invasion growth rates, $g_i^{inv},$ and the effective carrying capacities, $K_{\alpha }^{ef\text{}f}$ are the sum of a large number of small terms. Each individual resource makes only a small contribution (of order 1/M) to the growth of any consumer, and every consumer makes an order 1/S contribution to the effective resource capacity. Thus, from the central limit theorem, the distribution of growth rates $g_i^{inv}$ and the distribution of effective resources $K_{\alpha }^{ef\text{}f}$ in the ecosystem can be well-approximated by a normal distribution. In the language of the cavity method of statistical physics, this corresponds to the replica symmetric solution. For future reference, denote the means and variance of the effective carrying capacity by $〈K_{\alpha }^{ef\text{}f}〉=K_{\alpha }^{ef\text{}f}$ and $〈{\left(K_{\alpha }^{ef\text{}f}\right)}^2〉-{〈K_{\alpha }^{ef\text{}f}〉}^2={\sigma }_{K^{e\text{}f\text{}f}}^2$ (see Figure 1).

FIG. 1. Analyzing MacArthur Consumer’s Resource Model for large, diverse ecosystems.

(Top) The growth rate for a species i is a sum of terms resulting from consuming M ≫ 1 resources. For this reason, from the central limit theorem, it can be well modeled by a (truncated) normally distributed variable. (Bottom) Each resource α is consumed by S ≫ 1 consumers. From the central limit theorem, the effective carrying capacity of the resource (i.e. the resource abundance at steady-state) can also be modeled using a (truncated) normal distribution. The truncation is due to the fact that neither species nor resource abundances can become negative. To derive our analytic cavity equations, we require selfconsistency for the means and variance of these distributions.

This suggests the following intuition for thinking about our ecosystem. Each species, i, has a invasion growth rate drawn from a normal distribution. In other words, we can think of $g_i^{inv}\approx g+{\sigma }_g{z}_i,$ where z_i is a standard normal variable. Similarly, each resource has an effective carrying capacity that is also drawn from a normal distribution, with $K_{\alpha }^{ef\text{}f}\approx K^{ef\text{}f}+{\sigma }_K{\tilde{z}}_{\alpha },$ with ${\tilde{z}}_{\alpha }$ standard, normal variable. In general, the means and variances $\left(g,K^{ef\text{}f},{\sigma }_g^2,{\sigma }_{K^{ef\text{}f}}^2\right)$ depend on the abundances of all other species and resources. Our statistical mechanics inspired mean field approach exploits this observation to self-consistently solve for the means and variances of the invasion growth rate and effective resource carrying capacity. In the physics literature, these is known as cavity method (CM). In general, this is a very subtle calculation but can be done using a generalized cavity equation (see below and in appendix).

In order to derive the CM self-consistency equations, we consider a system with S species and M resources and ask what happens when we add an additional species and resource to the system. We denote the abundances of the additional species and resource by N₀ and R₀ respectively. This two-step cavity where both a resource and species is removed is similar to the procedure employed to analyze the Hopfield model and compressed sensing [44, 45] and is necessary to correctly capture subtle correlations between resource and species dynamics. This approach is intimately related to classic works by MacArthur and Levins that analyzed ecological dynamics by asking if a new species could invade an ecosystem [6]. Whereas their analysis was applicable to small ecosystems with a few species, our analysis is valid for large, diverse ecosystems.

Since the number of species and resources in the original ecosystem is large (S, M ≫ 1), the addition of the new resource and consumer represent a small perturbation of the original system. For this reason, it is useful to define two susceptibilities, χ and ν that measure the sensitivity of an ecosystem to small perturbations. The resource susceptibility, χ, measures the average change in the mean resource abundance at steady-state if we slightly increase the supply of all the externally supplied resources. Denoting the steady-state value of a quantity

X by $\overline{X}$ , we can mathematically define χ, as

$$\text{χ=}\frac{1}{M}\sum _{\alpha }\frac{\partial {\overline{R}}_{\alpha }}{\partial K_{\alpha }}.$$ (5)

The average species-cost susceptibility, ν, measures the change in mean species abundances if we slightly decrease the minimum fitness cost (or equivalently increase the growth rate),

$$v=\frac{1}{S}\sum _i\frac{\partial {\overline{N}}_i}{\partial g_i}=-\frac{1}{S}\sum _i\frac{\partial {\overline{N}}_i}{\partial m_i}.$$ (6)

These susceptibilities characterize the sensitivity of an ecosystem to perturbations and are derivatives which can be directly measured in experiments. To estimate ν, one would alter the environment to put stress on the consumers, for instance in the context of microbes, one might force microbes to express a useless protein such as GFP as done in [? ] to reduce m_i. To estimate χ, one would alter the environment to vary the carrying capacity and measure how the resource abundance fixed point changes, this could be done in many ways for one example of an experiment varying the environment to impact resource abundances see [? ].

In terms of these quantities, one can derive a simple expression for the steady-state abundances of a newly added consumer and resource (see Appendix):

$$\begin{array}{l}{\overline{N}}_0=\frac{\max \left[0,g+{\sigma }_g{z}_0\right]}{\text{γ}{\sigma }_c^2\text{χ}},\\ {\overline{R}}_0=\frac{\max \left[0,K^{ef\text{}f}+{\sigma }_{K^{ef\text{}f}}^2{\tilde{z}}_0\right]}{1-{\sigma }_c^{2}v},\end{array}$$

where as above z₀ and ${\tilde{z}}_0$ are independent, unit normal variables. These equations have a beautiful and straightforward interpretation. A new species added to the system will have an invasion growth rate $g_0^{inv}=g+{\sigma }_g{z}_0,$ which is normally distributed. If the growth rate is negative, it will not be able to invade the system and go extinct. If its growth rate is positive when introduced in the ecosystem, then it survives with an abundance proportional to its invasion growth rate. We emphasize that this proportionality constant can differ significantly from what would be expected in a single-species ecosystem and depends on all the other resources and species present in the ecosystem through the susceptibility χ and the variance of the consumption coeffcients ${\sigma }_c^2$ . For this reason, the invasion growth rate of a species when it invades an ecosystem is positively correlated with its abundance. Similarly, the new resource is depleted if its effective carrying capacity is negative. Otherwise the steady-state abundance of the new resource is proportional to its effective carrying capacity. These equations are similar to the arguments of MacArthur and Levins on the necessary conditions for invasibility to large ecosystems [6]. They also generalize results for species abundances derived in [27] using the Lotka-Volterra equation and the results in [24, 38, 39] which ignored resource depletion and resource fluctuations.

B. Comparison with numerics

Unlike small ecosystems, we cannot analytically solve for the all the resource and species abundances. However, we can take a statistical approach that allows us to calculate statistical properties of species and resource abundances at steady-state. We also restrict our analysis to uninvadable steady-states, defined as a steady-state which cannot be invaded by any species. This, both simplifies the mathematics, and allows us to more directly relate our calculations to ecology.

Using (7) is it possible to derive self-consistency equations for the fraction of species in the regional species pool that survive, ϕ_N , the mean abundance of the species $〈N〉=1/S{\sum }_i{N}_i,$ and variance and second moment of surviving species abundances, $〈{\left(\delta N\right)}^2〉$ and $q_N=〈{\left(\delta N\right)}^2〉+{\left(N\right)}^2=1/S{\sum }_i{N}_i^2$ respectively. We can also calculate the analogous equations for resources: the fraction of resources with non-zero abundance,ϕ_R, the mean abundance of resources $〈R〉=1/M{\sum }_{\alpha }{R}_{\alpha },$ and variance and second moment of the resource abundances, $〈{\left(\delta R\right)}^2〉$ and $q_R=〈{\left(\delta R\right)}^2〉+{〈R〉}^2=1/M{\sum }_{\alpha }{R}_{\alpha }^2$. The equations are derived in Appendix C and can be solved numerically.

The validity of our derivation is dependent on the MCRM having replica symmetry in the correspondence between the closely related cavity and replica methods (see e.g. [? ]), and while we do not rigorously prove replica symmetry is not broken in this setting, we expect this assumption to be correct because there is a convex Lyapunov function (see e.g. [8]) and therefore just one fixed-point in the MCRM. Broken replica symmetry occurs when the landscape is complex, see [? ]. The other assumption is a large system size S, M ≫ 1. The theory is expected to be exact for infinite systems, but as we will now discuss, our simulations demonstrate that moderately large ecosystems are well predicted by the theory in the setting of Gaussian distributed consumption coefficients for which it was derived.

To check the accuracy of our CM, we compared our analytic predictions to numerical simulations (see Figure 2). We simulated (2) for two different choices of distributions for the c_iα. In the first set of simulation, the c_iα were binary random variables with c_iα = 1 with probability p and c_iα = 0 with probability 1 − p. The probability p can be viewed as the level of generalism in the regional species pool. As p → 0, all organisms in the community are specialist and consume a handful of resources. When p → 1, the community consists of generalists who can consume almost all resources. In the second set of simulations, we drew the consumption coeffcients from a Gaussian distribution with the same mean and variance as the corresponding Bernoulli distribution with probability p.

FIG. 2. Comparison of numerical simulations with theory.

Ecosystems were simulated with the consumption coefficients c_iα = 0, 1 drawn from a Bernoulli distribution with probability p (black lines) or a Gaussian distribution with same mean and variance as the binomial distribution (red line). Here S = M = 30, K = 1, σ_k = 1, m = 1, σ_m = .1, and 250 trials were used in these simulations: the error bars denote ±2 standard deviations. The K_a and m_i were drawn iid from a gamma distribution in the binomial plot to ensure nonnegativity of the parameters, and a Gaussian distribution in the Gaussian approximation plot. The numerical results were compared to theoretical predictions from the self-consistent cavity equations for ecologically stable steady-states (green circles). (A) Fraction of species that survive ϕ_N , (B) fraction of resources that are not depleted ϕ_R, (C) average abundance of all species 〈N〉, (D) average abundance of all resources 〈R〉 as a function of the probability p. (E) Mean-abundance of surviving species 〈N〉/ϕN and (F) mean-abundance of surviving resources 〈R〉/_R.

As shown in Fig. 2, our analytic results agree remarkably well with numerical simulations. The agreement between theory and numerics is nearly exact when c_iα are drawn from a Gaussian and shows qualitative agreement even when the consumption coeffcients c_iα are binary random variables. This is a result of the Gaussianity assumptions used to derive the cavity equations (see Appendix). The discrepancy between the binary case and Gaussian case stems from the fact that the for large S and M the c_iα are strictly positive for the binary case but generically contain some negative elements for Gaussian distributions. A negative c_iα implies that species i produces resource α at a fitness cost to itself. Thus, all simulations with Gaussian include a small fraction of public good producers that are accounted for in our theoretical calculations but are absent in the simulations with binary variables.

Despite these differences, for both choice of distributions the fraction of surviving species declines with increasing p. This is consistent with the basic idea of nichetheory that as p increases, there is increased competition resulting in greater competitive exclusion. In contrast, the mean abundances of surviving species and resources shows a non-monotonic behavior as a function of p in both numerical simulations and analytics (see appendix and Fig. 5 for additional simulation results).

FIG. 5. Comparing CRM theory vs simulation - another setting.

All parameters defining the species were drawn from a Gaussian distribution as in main text but with σ_m = 1 (a larger maintenance cost variance). Interestingly, we see a difierent behavior than in Fig 2 in that the average abundance 〈N〉 of all species is increasing with increasing niche overlap. Also, the number of surviving species is reduced by high niche overlap as it is in Fig 2, which makes sense since when species are very similar they will compete more directly for the same resources. The parameters used are: σ_m = 1; m = 1; σ_k = 1; K = 1, and simulations were run assuming an equal number of species and resources S = M = 30 (γ = 1). Additionally, μ_c = 1 was fixed and σ_c varied (note that the niche overlap ρ is determined by these two variables). The theoretical predictions from the cavity solution (blue), match well with the numerical solutions (red) for the CRM model averaged over 100 trials with and plotted with ±2 standard deviation error bars (dashed lines).

IV. GENERALIZING NICHE THEORY TO LARGE ECOSYSTEMS

The MacArthur consumer resource model has played a central role in the development of niche-based theories of community assembly [2, 7–10]. However, most of these analyses have focused on small ecosystems with just a few species and resources. Here, we discuss the ecological implications of our analysis for understanding community assembly in large ecosystems with many species and resources.

A. Relating MCRM parameters to ecology

We begin by relating the parameters of the MCRM to more ecologically meaningful quantities such as the niche overlap, fitness, zero net- growth isoclines (ZNGI), and impact vectors. In ecology, the niche overlap,ρ , measures how much two species compete for the same resources. The larger the niche overlap, the more species compete. For small ecosystems, the niche overlap is bounded between 0 and 1, with a niche overlap of zero meaning the species do not compete for resources and a niche overlap of one indicating the species have identical consumption profiles. In the context of the two species MacArthur resource model, the niche-overlap between species can be thought of as the percentage of variance explained if one performs a regression of the first consumer’s consumption vector against the consumption vector of the second species[1, 7, 8]. Using this observation, we can naturally extend the idea of niche overlap to entire ecosystems by defining an ecosystem-level niche overlap ρ in terms of the mean and variances of the consumption coeffcients $c_{i\alpha }$:

$$\rho =\frac{{\mu }_c^2}{{\mu }_c^2+{\sigma }_c^2}.$$ (8)

One useful way of thinking about ρ is that it measures the niche-overlap between two species randomly drawn from the regional species pool. It is easy to see that when ${\sigma }_c^2\ll {\mu }_c,$ all species have nearly identical consumption preferences and ρ → 1. In contrast when ${\sigma }_c^2\gg {\mu }_c,$ species will have very distinct consumer preferences and ρ → 0.

Another fundamental quantity in contemporary niche theory is the ecological fitness of an organism, $f_i={\sum }_{\alpha }{c}_{i\alpha }{K}_{\alpha }-m_i$ [1,8]This fitness is the initial growth rate of organism i in the absence of other species. In general, the actual growth rate of a species will differ significantly from the fitness if the resource abundances differ significantly from the resource carrying capacities K_α. For this reason, we will refer to this as the “naive” fitness.

We show in the appendix that it is also possible to relate our parameters directly to ZGNIs and generalized impact vectors.

B. Niche overlap and coexistence

One of the fundamental results of niche-based theories is that as the niche-overlap between species increases, coexistence become more and more diffcult [1]. The underlying reason for this is species that have similar consumer preference are more likely to compete with each other, resulting in competitive exclusion. Thus, increasing the niche-overlap in the community should decrease the fraction of species ϕ_N that can co-exist in a community. On the other hand, stabilizing mechanisms that decrease the tness differences between species should increase coexistence. We can parameterize the fitness differences in the community by the dimensionless quantity ${\sigma }_m/m$ equal to the standard deviation over the mean of the maintenance costs m_i over all species in the regional species pool. This choice of parameterization is in line with contemporary niche theory where fitness differences are dened as the difference in growth rates when species have identical consumption preferences [1]. Figure 3 shows ϕ_N as a function of the niche overlap ρ and ${\sigma }_m^2/m$. This choice of niche-overlap corresponds to varying the probability p for having a non-zero c_iα from 0:1 to 0:9 (see Fig. 2). As predicted by niche theories, increasing ρ leads to increased competition and a smaller ϕ_N. In constrast decreasing ${\sigma }_m/m_i$ at a fixed ρ, leads to a larger fraction of species surviving. Thus, in this regard large ecosystems behave quite similarly to predictions made by analyzing smaller models.

FIG. 3. Co-existence and diversity.

Here we use the same parameters as in the previous plot, apart from allowing σ_m to vary, and show the cavity prediction of the fraction of surviving species ϕ_N as a function of p and the standard deviation over mean σ_m/m of the maintenance costs m_i of species. In this regime increasing p leads to more similar species by increasing the niche overlap ρ as defined in (8) from ρ ≈ 77 to ρ ≈ 99 in the range shown above. As predicted by niche theories, increasing ρ leads to increased competition and a smaller ϕ_N while decreasing σ_m/m_i leads to larger fraction of species surviving at a fixed p.

C. Resource depletion, effective fitness, and carrying capacity

One important feature of our analysis, which we are able to analyze statistically with our CM approach, is the large-scale depletion of resources. As shown in Fig. 2, species can signiffcantly change the resource profile and deplete a large fraction of resources initially present in the environment. This collective behavior can change which species survive and thrive in an environment. One way to measure this effect and the reshaping of the resource profile is to measure the correlation between the naive fitness of an organism, $f_i={\sum }_{\alpha }{c}_{i\alpha }\max \left\{K_{\alpha },0\right\}-m_i,$ and its steady-state abundance in the ecosystem ${\overline{N}}_i.$ The fitness f_i measures the growth rate of organism i if it is introduced into an environment in the absence of other species. For this reason, we expect f_i to be highly predictive of ${\overline{N}}_i.$ when resource abundance profiles are not significantly perturbed by consumption. On the other hand, it is possible for correlation between f_i and ${\overline{N}}_i$ to decrease significantly.

Fig. 4 shows f_i versus ${\overline{N}}_i$ for numerical simulations where the c_iα drawn from a binomial distribution with p = 0:1 and σ_m/m = 0.1, as well as the case where parameters are Gaussian random variables with mean an variance matching the binomial setting. From the figure, it is clear there is a significant correlation between f_i and ${\overline{N}}_i.$ Organisms with higher fitness disproportionately survive in the ecosystem. However, a significant number of organisms that have a high naive fitness f_i can still go extinct in the ecosystem (black points). The difference between plots (A,B) and (C,D) is that in the former K_α and m_i are kept positive by ensuring they are drawn from a gamma distribution and the consumer preferences c_iα are always positive since they are binary (1 with probability p or 0 otherwise). In (C,D), each of these parameters is drawn from a Gaussian distribution, but with the same mean and variance as in (A,B). This allows K_α, m_i, and c_iα to be negative. We will be agnostic about the physical meaning of negative c_iα: it could for instance be related to a common good production by a species at an energetic cost to itself or simply that species i is preyed on by resource α. A more careful breakdown of relevant behaviors which could lead to the ecological engineering phenomena we will discuss such as poisons and common good production, and this is a direction of future work, but not considered in the present manuscript.

FIG. 4. Predicting the effective fttness and effective carrying capacity.

(A) Steady-state abundance ${\overline{N}}_i$ versus the fitness $f_i={\sum }_{\alpha }{c}_{i\alpha }\max \left\{K_{\alpha },0\right\}-m_i$ for each species i . Fitness is defined as the initial growth rate of species i in the environment in the absence of all other species. Points colored black are species with positive fitness that go extinct in the community $\left(f_i>0,{\overline{N}}_i=0\right)$. (B) Comparison of the steady state resource levels ${\overline{R}}_{\alpha }$ with their capacity K_α. The filled circles are generated from simulations with M = S = 30 resources and species, the data is generated from 50 separate trials. Parameters for simulations as in Figure 2 with p = and σm/m = 0.1. Black points indicate resources which have a positive capacity but go extinct in the community. The difference between plots for (A,B) and (C,D) is that in the former K_α and m_i are always positive because they are drawn from a gamma distribution and Bernoulli distribution respectively. In (C,D), each of these parameters is drawn from a Gaussian distribution with the same mean and variance as in (A,B). This means that a small fraction of the K_α, m_i, and c_iα are negative. Negative c_iα corresponds to production of resource α by species i at a fitness cost to itself (i.e. public good production). In (C,D), red points indicate species with negative fitness that can stably exist in the community $\left(f_i<0,\text{}{\overline{N}}_i>0\right)$ or resources with negative K_α that are produced by the ecosystem. Contours show theoretical predictions of our CM for correlation between${\overline{N}}_i$ and f_i as well as ${\overline{R}}_{\alpha }$ and K_α (see Appendix E for details). Each contour represents half a standard deviation of our theory.

We find that allowing these negative consumer preference, does not dramatically alter the model predictions in terms of number/abundance or species/resources: we demonstrate a relatively good agreement between models which allow (Gaussian) and don’t allow (binomial) negative consumer preferences in Fig. 2. However, we also find that the two cases lead to different ecological engineering phenomena when one observes individual species in the environment as in Fig. 4. Here, in panel D the red points correspond to resources α with a negative carrying capacity (K_α < 0) which end up in the environment due to “help” from a set of fit species (i₁, ...) because of the negative consumer preference $\left(c_{i_1\alpha },\mathrm{...}\right).$

In Fig. 4 C, red points indicate species with negative fitness that can stably exist in the community $\left(f_i<0,{\overline{N}}_i>0\right)$ due to the effect of negative consumer preferences, which can boost resources above their carrying capacity. Thus, we see species that cannot survive in the environment in the absence of other species can fixate (red points). Importantly, this emergent phenomenon is a collective property of the whole ecosystem and results from a complex interplay between organisms and environment and is an interesting direction for future study.

Additionally, Fig. 4 shows predictions from our CM for the correlation between f_i and ${\overline{N}}_i.$ Within our replicasymmetric ansatz, these correlations are described by normal distributions whose variances and covariances can be calculated using our self-consistent equations. The contour lines represent half a standard deviation spread of our normal distribution. Our theory qualitatively captures the shape of the correlation between f_i and ${\overline{N}}_i.$ We give explicit expression for these correlations as well as the mutual information between species abundances and naive fitness in Appendix E.

V. DISCUSSION

Niche-based theories have played a fundamental role in shaping our understanding of community assembly and community ecology. In this work, we use ideas and methods from statistical physics to analyze a canonical model in community ecology, MacArthur’s Consumer Resource Model (MCRM). Unlike previous works, our statistical physics inspired approach allows us to analyze large ecosystems with many species and resources. Our results suggest that organisms can significantly perturb their environments. The abundance of resources can be significantly altered and resource can even be completely depleted. We nd that such niche-construction is a generic feature of MCRM. This suggests that in complex ecosystems, organisms actively construct their environment. To quote Levins and Lewontin, “they are not the passive objects of external forces, but creators and modulators of these forces” [22]. These effects are even more dramatic when consumers can produce public goods at a fitness cost to themselves. In this case, species and resources that could not survive in isolation can fixate in the ecosystem.

To carry out our analysis, we developed a sophisticated theory based on the cavity method. One of the most striking things about our analysis is that many physical quantities that appear in the “cavity equations” have natural ecological interpretations in terms of invasion growth rates and effective carrying capacities. The underlying reason for this is that the cavity methods is based on asking how ecosystems are perturbed when a new species and a new resource are introduced into the ecosystem. Conceptually, this is very similar in spirit to many classical arguments in community ecology pioneered by Levins and MacArthur that ask whether a new species can invade [2, 6]. This naturally allows us to generalize many of the results from niche-based theories to large, diverse ecosystems. However, the price we pay for using our cavity approach is that we are limited to making statistical predictions.

An important question for future investigation is to ask how our results change if we make the model more realistic. In the MCRM, all species are assumed to have a linear, Type I functional response. It will be interesting to generalize our model to non-linear functional responses. We have also neglected the effects of environmental and demographic stochasticity. Stochasticity can induce phases transitions in ecosystems from a nichelike phase where competitive effects dominate community assembly to an ecologically neutral-like phase where stochasticity is the primary determinant of community structure [15, 23]. It will be interesting to see if the techniques developed here can be generalized to this more complicated setting. Finally, we have assumed that our population can be modeled as a well-mixed community. However, spatial effects can qualitatively change the behavior of cellular populations [46, 47] and are likely to play an important role in community assembly.

Appendix A: Basic Setup

We brie y summarize MacArthur’s classical consumer resource model (MCRM). Species i = 1 … S grows at a rate proportional to its utilization of resources, $R_{\alpha },\alpha =\mathrm{1...}M,$ in the environment. This is described by the equation:

$$\frac{1}{N_i}\frac{d{N}_i}{dt}=\sum _{\alpha }{c}_{i\alpha }{R}_{\alpha }-m_i+h_i,$$ (A1)

where w_α is the value of one unit of resource to a species (e.g. ATPs that can be extracted); c_iα is the rate at which species i consumes resource α and converts that into a “growth rate”, m_i; m_i is the minimum amount of resources that must be consumed in order to have a positive growth rate. We have also added a small perturbation h_i to the system that will do a linear expansion in. The original MCRM corresponds to the choice h_i = 0.

We define the growth rate to be

$$g_i\left(R\right)=\sum _{\alpha }{c}_{i\alpha }{W}_{\alpha }{R}_{\alpha }-m_i.$$ (A2)

In consumer resource model, resources satisfy their own dynamical equations:

$$\frac{d{R}_{\alpha }}{dt}=R_{\alpha }\left(K_{\alpha }+b_{\alpha }-R_{\alpha }\right)-\sum _j{c}_{j\alpha }{N}_j{R}_{\alpha },$$ (A3)

where the first term (with b_α = 0) describes the resource dynamics in the absence of any species and the second term models the consumption of resource by species in the environment, and b_α is small perturbation. The original MCRM corresponds to the choose b_α = 0. Furthermore, define the effective carrying capacity

$$K_{\alpha }^{e\text{}f\text{}f}=K_{\alpha }-\sum _j{c}_{j\alpha }{N}_j.$$ (A4)

We will consider the case when the consumer preferences c_iα are random variables that can be characterized by their means and variances. In particular,

$$〈c_{i\alpha }〉=\frac{{\mu }_c}{S},$$ (A5)

and

$$〈c_{i\alpha }{c}_{j\beta }〉=\frac{{\sigma }_c^2}{S}{\delta }_{ij}{\delta }_{\alpha \beta }+\frac{{\mu }_c^2}{S^2}\approx \frac{{\sigma }_c^2}{S}.$$ (A6)

To perform the cavity equations, it is useful to define several other quantities. Let us a define the fluctuating part of the consumer preferences d_iα as

$$c_{i\alpha }\equiv \frac{{\mu }_c}{S}+{\sigma }_c{d}_{i\alpha }.$$ (A7)

Then, we have that

$$〈d_{i\alpha }〉=0,$$ (A8)

and

$$〈d_{i\alpha }{d}_{j\beta }〉=\frac{{\delta }_{ij}{\delta }_{\alpha \beta }}{S}.$$ (A9)

We will also assume that the carrying capacities are drawn from a Gaussian distribution with

$$〈K_{\alpha }〉=K,$$ (A10)

and

$$〈\delta K_{\alpha }\delta K_{\beta }〉=〈\left(K_{\alpha }-K\right)\left(K_{\beta }-K\right)〉={\delta }_{\alpha \beta }{\sigma }_K^2.$$ (A11)

Finally, we assume that the minimum survival coeffcients are also drawn from Gaussian distribution with

$$〈m_i〉=m,$$ (A12)

and

$$〈\delta m_i\delta m_j〉=〈\left(m_i-m\right)\left(m_j-m\right)〉={\delta }_{ij}{\sigma }_m^2.$$ (A13)

For future reference, it will also be helpful to define the ratio

$$\text{γ=}\frac{M}{S},$$ (A14)

the average resource abundance,

$$〈R〉=\frac{1}{M}\sum _{\alpha }{R}_{\alpha },$$ (A15)

and the average species abundance

$$〈N〉=\frac{1}{S}\sum _j{N}_j.$$ (A16)

With these definitions, notice that we can rewrite (A1) as

$$\frac{1}{N_i}\frac{d{N}_i}{dt}=\mu \text{γ}〈R〉-m+\sigma \sum _{\alpha }{d}_{i\alpha }{R}_{\alpha }-\delta m_i,$$ (A17)

And rewrite (A3) as

$$\frac{1}{R_{\alpha }}\frac{d{R}_{\alpha }}{dt}=\left[K-{\mu }_c〈N〉\right]-R_{\alpha }-{\sigma }_c\sum _j{d}_{j\alpha }{N}_j+\delta K_{\alpha }.$$ (A18)

We can de ne the mean growth rate of the population

$$g={\mu }_c\text{γ}〈R〉-m$$ (A19)

and the mean e ective capacity of resources in the ecosystem to be

$$K^{e\text{}f\text{}f}=K-{\mu }_c〈N〉$$ (A20)

as in the main text. In terms of these quantities, we can rewrite these equations as

$$\begin{array}{l}\frac{1}{N_i}\frac{d{N}_i}{dt}=g+\sigma \sum _{\alpha }{d}_{i\alpha }{R}_{\alpha }-\delta m_i,\\ \frac{1}{R_{\alpha }}\frac{d{R}_{\alpha }}{dt}=K^{e\text{}f\text{}f}-R_{\alpha }-{\sigma }_c\sum _j{d}_{j\alpha }{N}_j+\delta K_{\alpha }.\end{array}$$ (A21)

The terms on the right hand sides of the equation above have a natural interpretation as the “fluctuating parts” of the growth rate and effective carrying capacity. In particular, we have rewritten the growth rate for species i as the sum of the mean growth rate g and a uctuating component $\delta g_i$ defined as

$$\delta g_i=\sigma \sum _{\alpha }{d}_{i\alpha }{R}_{\alpha }-\delta m_i.$$ (A22)

We have split the effective carrying capacity of resource α is divided into a mean K^eff and fluctuating component $\delta K_{\alpha }^{e\text{}f\text{}f}$ defined as

$$\delta K_{\alpha }^{e\text{}f\text{}f}=-\sum _j{d}_{j\alpha }{N}_j+\delta K_{\alpha }.$$ (A23)

Appendix B: Deriving the Species and Resource Distributions

To derive the cavity equations, we will relate a system with S species and M resources to a new system where we add an additional resource R₀ and and additional species N₀. Thus, the cavity equations relate a ecosystem with S + 1 and M + 1 resources to a ecosystem with S and M resources.

Then we can write equations for this new ecosystem (to leading order in S):

$$\frac{1}{N_i}\frac{d{N}_i}{dt}=g+{\sigma }_c\sum _{\alpha }{d}_{i\alpha }{R}_{\alpha }-\delta m_i+{\sigma }_c{d}_{i0}{R}_0,$$ (B1)

and

$$\frac{1}{R_{\alpha }}\frac{d{R}_{\alpha }}{dt}=K^{e\text{}f\text{}f}-R_{\alpha }-{\sigma }_c\sum _j{d}_{j\alpha }{N}_j+\delta K_{\alpha }-{\sigma }_c{d}_{0\alpha }{N}_0.$$ (B2)

We can also write down the corresponding equations for the new resource and species:

$$\frac{1}{N_0}\frac{d{N}_0}{dt}=g+{\sigma }_c\sum _{\alpha }{d}_{0\alpha }{R}_{\alpha }-\delta m_0+{\sigma }_c{d}_{00}{R}_0,$$ (B3)

and

$$\frac{1}{R_0}\frac{d{R}_0}{dt}=K^{e\text{}f\text{}f}-R_0-{\sigma }_c\sum _j{d}_{j0}{N}_j+\delta K_0-{\sigma }_c{d}_{0\alpha }{N}_0.$$ (B4)

We now focus on steady-state. Let us denote the steady-state value of a quantity X by $\overline{X}$ . Then, we can define some susceptibilities that are extremely useful for what follow:

$$\begin{gathered} {\chi }_{i\beta }^{\left(N\right)}=\frac{\partial {\overline{N}}_i}{\partial K_{\beta }} \\ {\chi }_{\alpha \beta }^{\left(R\right)}=\frac{\partial {\overline{R}}_{\alpha }}{\partial K_{\beta }} \end{gathered}$$ (B5)

and

$$\begin{array}{l}{v}_{ij}^{\left(N\right)}=\frac{\partial {\overline{N}}_i}{\partial m_j}\\ v_{\alpha j}^{\left(R\right)}=\frac{\partial {\overline{R}}_{\alpha }}{\partial m_j}.\end{array}$$ (B6)

Now we are in a position to perform the cavity calculation. Let us denote the steady-state value of a quantity X in the absence of the new resource and species as ${\overline{X}}_{/0}$. Then, since the addition of a resource and species represents a small perturbation (order 1/S), we can write:

$${\overline{N}}_i={\overline{N}}_{i/0}-{\sigma }_c\sum _{\beta }{\text{χ}}_{i\beta }^{\left(N\right)}{d}_{0\beta }{\overline{N}}_0-{\sigma }_c\sum _j{v}_{ij}^{\left(N\right)}{d}_{j0}{\overline{R}}_0,$$ (B7)

and

$${\overline{R}}_{\alpha }={\overline{R}}_{\alpha /0}-{\sigma }_c\sum _{\beta }{\text{χ}}_{\alpha \beta }^{\left(R\right)}{d}_{0\beta }{\overline{N}}_0-{\sigma }_c\sum _j{v}_{\alpha j}^{\left(R\right)}{d}_{j0}{\overline{R}}_0.$$ (B8)

We can now plug in these expressions into the steady-state equations for N₀ and R₀. This gives:

$$0={\overline{N}}_0\left[g+\sigma d_{00}{R}_0+\delta m_0+{\sigma }_c\sum _{\alpha }{d}_{0\alpha }{R}_{\alpha /0}-{\sigma }_c^2\sum _{\alpha \beta }{\text{χ}}_{\alpha \beta }^{\left(R\right)}{d}_{0\alpha }{d}_{0\beta }{N}_0-{\sigma }_c^2\sum _{\alpha ,j}{v}_{\alpha j}^{\left(R\right)}{d}_{0\alpha }{d}_{j0}{R}_0\right].$$ (B9)

If we now take leading order contributions to S in this expression, and take expectation value over expressions this reduces to

$$0={\overline{N}}_0\left[g+\sigma d_{00}{R}_0-\text{γ}{\sigma }^2\frac{1}{M}\sum _{\alpha }{\text{χ}}_{\alpha \alpha }^{\left(R\right)}{N}_0+\delta m_0+\sigma \sum _{\alpha }{d}_{0\alpha }{R}_{\alpha /0}\right].$$ (B10)

Notice that, to leading order in S, we can model the term $\delta m_0+\sigma {\sum }_{\alpha }{d}_{0\alpha }{R}_{\alpha /0}$, which is just the invasion growth rate minus the mean growth rate $g_0^{inv}-g$, as a Gaussian random eld with mean 0 and variance:

$${\sigma }_g^2={\sigma }_c^2\text{γ}\frac{1}{M}\sum _{\alpha }{\overline{R}}_{\alpha /0}^2+{\sigma }_m^2={\sigma }_c^2\text{γ}{q}_R+{\sigma }_m^2,$$ (B11)

and

$$q_R=\frac{1}{M}\sum _{\alpha }{R}_{\alpha /0}^2.$$ (B12)

If we let z_N be random field with mean zero and unit variance, and define the average suspectibility

$$\text{χ=}\frac{1}{M}\sum _{\alpha }{\text{χ}}_{\alpha \alpha }^{\left(R\right)}$$ (B13)

then we can write the equation for $N_0$ as

$$0={\overline{N}}_0\left[\mu \text{γ}〈R〉-m-\text{γ}{\sigma }^2\text{χ}{\overline{N}}_0+{\sigma }_g{z}_N\right].$$ (B14)

We can also derive a similar equation for R₀. This is given by

$$0=R_0\left[K^{e\text{}f\text{}f}-R_0-{\sigma }_c\sum _j{d}_{j0}{N}_{j/0}-{\sigma }_c^2\sum _{j\beta }{\text{χ}}_{i\beta }^{\left(N\right)}{d}_{j0}{d}_{0\beta }{\overline{N}}_0-{\sigma }_c^2\sum _{j,k}{v}_{jk}^{\left(N\right)}{d}_{j0}{d}_{k0}{\overline{R}}_0+\delta K_0-{\sigma }_c{d}_{00}{N}_0\right].$$ (B15)

Using the same logic as above, to leading order we have

$$0=R_0\left[K^{e\text{}f\text{}f}-R_0+{\sigma }_c^{2}v{\overline{R}}_0+{\sigma }_{K^{e\text{}f\text{}f}}{z}_R\right],$$ (B16)

where

$$v=\frac{1}{S}\sum _j{v}_{jj}^{\left(N\right)},$$ (B17)

and

$${\sigma }_{K^{e\text{}f\text{}f}}^2={\sigma }_K^2+{\sigma }_c^2{q}_N,$$ (B18)

with

$$q_N=\frac{1}{S}\sum _j{N}_{j/0}^2.$$ (B19)

We can solve these equations and get

$$N_0=\frac{\max \left[0,g+{\sigma }_g{z}_N\right]}{\text{γ}{\sigma }^2\text{χ}}$$ (B20)

$$R_0=\frac{\max \left[0,K^{e\text{}f\text{}f}+{\sigma }_{K^{e\text{}f\text{}f}}{z}_R\right]}{1-{\sigma }^{2}v}.$$ (B21)

Thus, the distributions for N and R are given by truncated Gaussians.

Appendix C: Self Consistency Equations

Let us now write some self-consistency equations in the replica symmetric phase. Let us define the number of non-zero species and resources as S* and M* respectively. Furthermore, de ne

$${\varphi }_S=\frac{S^{\ast }}{S},$$ (C1)

$${\varphi }_M=\frac{M^{\ast }}{M}.$$ (C2)

Our goal is, given some parameters $\left\{K,{\sigma }_K,m,{\sigma }_m,\mu ,\sigma ,S,M\right\}$, to find the values for $\left\{{\varphi }_S,{\varphi }_N,〈N〉,〈R〉,q_R,q_N,\text{χ,}v\right\}$. Since there are eight unknowns we will need eight equations. It will also be useful to define:

$${\Delta }_g=\frac{\mu \text{γ}〈R〉-m}{{\sigma }_g},$$ (C3)

$${\Delta }_{K^{e\text{}f\text{}f}}=\frac{K-\mu 〈N〉}{{\sigma }_{K^{e\text{}f\text{}f}}},$$ (C4)

and the function

$$w_j\left(\Delta \right)={\int }_{-\Delta }^{\infty }\frac{dz}{\sqrt{2\pi }}{e}^{\frac{-z^2}{2}}{\left(z+\Delta \right)}^j.$$ (C5)

First, let us write self-consistency equations for the susceptibilities. Taking derivatives with respect to m and K of (B21) and noting that the fraction of non-zero species and non-zero resources is _N and _R respectively gives

$$v=-\frac{{\varphi }_N}{\text{γ}{\sigma }_c^2\text{χ}},$$ (C6)

$$\text{χ=}\frac{{\varphi }_R}{1-{\sigma }_c^{2}v}.$$ (C7)

Notice now that if we define $y=\max \left[0,\frac{a}{b}+\frac{c}{b}z\right]$ with z a gaussian random variable we have that:

$$〈y^j〉={\left(\frac{b}{c}\right)}^j{\int }_{-\frac{b}{a}}^{\infty }dz\frac{1}{\sqrt{2\pi }}{e}^{\frac{-z^2}{2}}{\left(z+\frac{b}{a}\right)}^j.$$ (C8)

We can now use the fact that (B21) implies that the species distribution and resource distribution is given by a truncated Gaussian to write self consistency equations for the fraction of nonzero resources and species as well as the the moments of their abundances:

$${\varphi }_N=w_0\left({\Delta }_g\right),$$ (C9)

$${\varphi }_R=w_0\left({\Delta }_{K^{e\text{}f\text{}f}}\right),$$ (C10)

$$〈N〉=\left(\frac{{\sigma }_g}{\text{γ}{\sigma }_c^2\text{χ}}\right)w_1\left({\Delta }_g\right),$$ (C11)

$$〈R〉=\left(\frac{{\sigma }_{K^{e\text{}f\text{}f}}}{1-{\sigma }_c^{2}v}\right)w_1\left({\Delta }_{K^{e\text{}f\text{}f}}\right),$$ (C12)

$$q_N=〈N^2〉={\left(\frac{{\sigma }_g}{\text{γ}{\sigma }_c^2\text{χ}}\right)}^2{w}_2\left({\Delta }_g\right),$$ (C13)

$$q_R=〈R^2〉={\left(\frac{{\sigma }_{K^{e\text{}f\text{}f}}}{1-{\sigma }_c^{2}v}\right)}^2{w}_2\left({\Delta }_{K^{e\text{}f\text{}f}}\right).$$ (C14)

Together (C4), (C7), and (C14) define the 10 selfconsistency equations we need, along with the de nitions (B11) and (B18). We solve these mean field equations numerically using the sum of squared differences between the left and right sides of equations (C9-C14) as an energy function which we minimize using the basinhopping optimization algorithm from the scipy.optimize. The algorithm uses random perturbations, local minimization, and an accept or reject criterion to attempt to minimize function which may be non-convex. The parameters we used were a temperature of 1, a step size of 0:5, and 5 iterations or initializations. Note that the equations can also be solved iteratively, but we found these solutions were stable for a smaller set of parameter values using this approach.

Appendix D: Zero net-growth nullclines and generalized impact vectors

We can also easily relate our mean-field quantities to ZGNIs. Recall, that ZGNI’s delineate range of resource conditions in which a species maintains a positive growth rate [9, 10]. Each species i defines a ZGNI in the resource space $R_i^{Z\text{}G\text{}N\text{}I}$ defined by the equation $g_i\left(R_i^{Z\text{}G\text{}N\text{}I}\right)=0$ . Geometrically, we can view $R_i^{Z\text{}G\text{}N\text{}I}$ as a hyperplane in resource space whose dot product with the consumption coeffcients c_iα of species i equals m_i (see Eq. 1). If m_i ≫ 1, the ZGNI is well-approximated by the plane perpendicular to c_iα. We can calculate some statistical properties of these ZGNI. Notice that the mean value of each component is just

$$〈R_{i\alpha }^{Z\text{}G\text{}N\text{}I}〉=\frac{m}{\text{γ}{\mu }_c},$$ (D1)

and the expected value of the square is just

$$〈{\left(R_{i\alpha }^{Z\text{}G\text{}N\text{}I}\right)}^2〉=\frac{{\sigma }_m^2+m^2}{\text{γ}{\sigma }_c^2}.$$ (D2)

In Modern Niche Theory, another important quantity is the impact vector of a species i. The impact vector describes how resources are depleted by the addition of another individual. Here, we introduce the idea of generalized impact vectors that measure how the steady state concentration of a resource α changes due to the introduction of a species j Alternatively, we can consider a system without resource β and then ask how it’s addi-tion changes the species abundance of a species i. These de ne the generalized impact vectors (GIVs).

These are of course the leading order contribution (in S) to the cavity equations under the replica symmetric assumption, namely (B8) and (B7). Thus, the components of the two “generalized’ impact vectors are given by:

$${\overline{R}}_{\alpha }-{\overline{R}}_{\alpha /j}=-{\sigma }_c\sum _{\beta }{\text{χ}}_{\alpha \beta }^{\left(R\right)}{d}_{j\beta }{\overline{N}}_j,$$ (D3)

$${\overline{N}}_i-{\overline{N}}_{i/\beta }=-{\sigma }_c\sum _j{v}_{ij}^{\left(N\right)}{d}_{j\beta }{\overline{R}}_{\beta },$$ (D4)

Appendix E: Comparison of individual fitness to true growth rate and steady state abundance

We want to quantify how much the naive fitness (the growth rate of an organism without other the growth rate of an organism without other growth rate $g_i^{inv},$ which in turn is closely related to the steady state abundance of each of the species in the community. From the definition of the invasion growth rate:

$$g_i^{inv}=\sum _{\alpha }{c}_{i\alpha }{R}_{\alpha /i}-m_i,$$ (E1)

and

$$f_i=\sum _{\alpha }{c}_{i\alpha }{K}_{\alpha }-m_i,$$ (E2)

we can compute the level of correlation between the two using the CM. To begin, we compute the means of each of these distributions:

$$〈f_i〉=\sum _{\alpha }〈c_{i\alpha }〉〈K_{\alpha }〉-〈m_i〉=\text{γ}{\mu }_{c}K-m,$$ (E3)

and

$$〈g_i^{inv}〉=\sum _{\alpha }〈c_{i\alpha }〉〈R_{\alpha /i}〉-〈m_i〉=\text{γ}{\mu }_c〈R〉-m.$$ (E4)

The final equality in the preceding equation holds in the asymptotic limit of large S and M. Note that the con-sumer preferences c_iα of an individual species are independent of the resources steady state levels when that species i is not included in the community as in the previous equation. We can also compute the correlation between fluctuations from the mean naive fitness and the mean invasion growth rate: i.e. if a species has a higher or lower individual fitness, if a species has a higher or lower individual in the community? To understand this correlation, we define $\delta f_i=f_i-〈f_i〉$ and $\delta g_i^{inv}=g_i^{inv}-〈g_i^{inv}〉$ and compute:

$$〈\delta f_i\delta g_i^{inv}〉=\begin{array}{c}〈\left(\sum _{\alpha }\left[{\sigma }_c{d}_{i\alpha }{K}_{\alpha }+\frac{{\mu }_c}{S}\left(K_{\alpha }-〈K_{\alpha }〉\right)\right]+\delta m_i\right)\\ .\hspace{0.33em}\hspace{0.33em}\left(\sum _{\alpha }\left[{\sigma }_c{d}_{i\alpha }{R}_{\alpha /i}+\frac{{\mu }_c}{S}\left(R_{\alpha /i}-〈R_{\alpha /i}〉\right)\right]+\delta m_i\right)\end{array}$$

In the large S limit, the important terms remaining in the average above are:

$$〈\delta f_i\delta g_i^{inv}〉=\text{γ}\frac{{\sigma }_c^2}{M}\sum _{\alpha }{R}_{\alpha /i}{K}_{\alpha }+{\sigma }_m^2,$$ (E5)

thus in the asymptotic limit we can write the correlation between the two forms of fitness as:

$$〈\delta f_i\delta g_i^{inv}〉=\text{γ}{\sigma }_c^2〈R_{\alpha /i}{K}_{\alpha }〉+{\sigma }_m^2.$$ (E6)

To compute the correlation between carrying capacity and resource level we modify (B21) from our capacity calculation, which yields such a relationship:

$$R_{\alpha }\left(K_{\alpha }\right)=\frac{\max \left[0,K_{\alpha }-{\mu }_c〈N〉+{\sigma }_c\sqrt{q_N}{z}_N\right]}{1-{\sigma }_c^{2}v}.$$ (E7)

Using this relationship and letting k be drawn from the same distribution as K_α, where

$$R\left(k\right)=\frac{\max \left[0,k-{\mu }_c〈N〉+{\sigma }_c\sqrt{q_N}{z}_N\right]}{1-{\sigma }_c^{2}v},$$ (E8)

we compute

$${〈kR\left(k\right)〉}_{k,z_N}.$$ (E9)

The full form of this integral is thus:

$${〈\int dkkR\left(k\right)e^{-\frac{{\left(k-K\right)}^2}{2{\sigma }_K^2}}〉}_{z_N}.$$ (E10)

By rewriting this as

$${〈\int dk\left(k-K\right)R\left(k\right)e^{-\frac{{\left(k-K\right)}^2}{2{\sigma }_K^2}}+\int dkK\text{}R\left(k\right)e^{-\frac{{\left(k-K\right)}^2}{2{\sigma }_K^2}}〉}_{z_N},$$ (E11)

it may be simplified via integration by parts on the first term, yielding: $\begin{array}{l}{〈kR\left(k\right)〉}_{k,z_N}={\sigma }_K^2〈R\text{'}\left(k\right)〉+K〈R\left(k\right)〉\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.33em}\hspace{0.33em}\hspace{0.17em}\text{}=\frac{1}{1-{\sigma }^{2}v}\left({\sigma }_K^2{w}_0\left({\Delta }_{K^{e\text{}f\text{}f}}\right)+K{w}_1\left({\Delta }_{K^{e\text{}f\text{}f}}\right)\right),\end{array}$ where ${\Delta }_{K^{e\text{}f\text{}f}}=\frac{K-〈N〉}{{\sigma }_{K^{e\text{}f\text{}f}}},$ and

$$w_j\left(\Delta \right)={\int }_{-\Delta }^{\infty }\frac{dz}{\sqrt{2\pi }}{e}^{\frac{-z^2}{2}}{\left(z+\Delta \right)}^j.$$ (E12)

Note, we can also compute Pearson’s correlation coeffcient for these two tness metrics:

$$c_{\rho }=\frac{〈\delta f\delta g^{inv}〉}{\sqrt{〈{\left(\delta f\right)}^2〉〈{\left(\delta g^{inv}\right)}^2〉}}$$ (E13)

$$\begin{array}{l}=\frac{\text{γ}{\sigma }_c^2{〈kR\left(k\right)〉}_{k,z_N}+{\sigma }_m^2}{\sqrt{\left({\sigma }_m^2+\text{γ}{\sigma }_c^2\left({\sigma }_K^2+K^2\right)\right)\left(\text{γ}{\sigma }_c^2〈R^2〉+{\sigma }_m^2\right)}}\\ =\frac{\frac{\text{γ}{\sigma }_c^2}{1-{\sigma }_c^{2}v}\left({\sigma }_K^2{w}_0\left({\Delta }_{K^{e\text{}f\text{}f}}\right)+K{w}_1\left({\Delta }_{K^{e\text{}f\text{}f}}\right)\right)+{\sigma }_m^2}{\sqrt{\left({\sigma }_m^2+\text{γ}{\sigma }_c^2\left({\sigma }_K^2+K^2\right)\right)\left(\text{γ}{\sigma }_c^2〈R^2〉+{\sigma }_m^2\right)}}.\end{array}$$ (E14)

Using (C10) and (C12) we can write the preceding expression as:

$$\frac{\frac{\text{γ}{\sigma }_c^2}{1-{\sigma }_c^{2}v}\left({\sigma }_K^2{\varphi }_R+K\left(\frac{1-{\sigma }_c^{2}v}{{\sigma }_{K^{e\text{}f\text{}f}}}\right)〈R〉\right)+{\sigma }_m^2}{\sqrt{\left({\sigma }_m^2+\text{γ}{\sigma }_c^2\left({\sigma }_K^2+K^2\right)\right)\left(\text{γ}{\sigma }_c^2〈R^2〉+{\sigma }_m^2\right)}}.$$ (E15)

1. Abundance vs naive tness

Note that given the relationship that N is a scaled version of g^inv where all negative values are truncated to zero (B21), it follows that we can compute the correlation between ${\tilde{N}}_i$ and f_i where ${\tilde{N}}_i=\frac{g_i^{inv}}{\text{γ}{\sigma }_c^2\text{χ}}.$ Where we let $N_i={\tilde{N}}_i$ if ${\tilde{N}}_i>0$ and N_i = 0 otherwise. To better understand that the correlation between the abundance and fitness of a species, we compute the correlation between $\tilde{N}$ and f:

$$〈\delta \tilde{N}\delta f〉=\frac{1}{\text{γ}{\sigma }_c^2\text{χ}}\left(\frac{\text{γ}{\sigma }_c^2}{1-{\sigma }_c^{2}v}\left({\sigma }_K^2{\varphi }_R+K\left(\frac{1-{\sigma }_c^{2}v}{{\sigma }_{K^{e\text{}f\text{}f}}}\right)〈R〉\right)+{\sigma }_m^2\right),$$ (E16)

$$〈{\left(\delta \tilde{N}\right)}^2〉=\frac{\left(\text{γ}{\sigma }_c^2〈R^2〉+{\sigma }_m^2\right)}{{\left(\text{γ}{\sigma }_c^2\text{χ}\right)}^2},$$ (E17)

and

$$〈{\left(\delta f\right)}^2〉={\sigma }_m^2+\text{γ}{\sigma }_c^2\left({\sigma }_K^2+K^2\right).$$ (E18)

Using these covariances, along with the means:

$$〈f〉=\text{γ}{\mu }_{c}K-m,$$ (E19)

$$〈\tilde{N}〉=\frac{\text{γ}{\mu }_c〈R〉-m}{\text{γ}{\sigma }_c^2\text{χ}},$$ (E20)

we are able to generate theoretical predictions for the distribution of f, N. See Fig 4 where the theoretical plot of f, $\tilde{N}$ is compared with values of fittness f and abundance N for all species in a network over many realizations.

2. Resource capacity versus resource abundance

In the same Fig 4, we additionally plot a theoretical prediction overlayed with numerics of how the resource capacities K_α are related to the resource abundances R_α.

If we define ${\tilde{R}}_{\alpha }=\frac{K_{\alpha }^{e\text{}f\text{}f}}{1-{\sigma }_c^{2}v},$ , which is equal to the prediction of R_α when the resources abundance is positive, and use the fact

$$\delta K_{\alpha }^{e\text{}f\text{}f}=-\sum _j{d}_{j\alpha }{N}_j+\delta K_{\alpha },$$ (E21)

then we may combine these two relations to yield:

$$\delta {\tilde{R}}_{\alpha }=\frac{1}{1-{\sigma }_c^{2}v}\left(-\sum _j{d}_{j\alpha }{N}_{j/\alpha }+\delta K_{\alpha }\right).$$ (E22)

We can thus compute the correlations:

$$\frac{1}{M}\sum _{\alpha }\delta K_{\alpha }\delta K_{\alpha }\to {\sigma }_K^2,$$ (E23)

$$\frac{1}{M}\sum _{\alpha }\delta K_{\alpha }\delta {\tilde{R}}_{\alpha }\to \frac{{\sigma }_K^2}{1-{\sigma }_c^{2}v},$$ (E24)

$$\frac{1}{M}\sum _{\alpha }\delta {\tilde{R}}_{\alpha }\delta {\tilde{R}}_{\alpha }\to \frac{{\sigma }_c^2{q}_N+{\sigma }_K^2}{{\left(1-{\sigma }_c^{2}v\right)}^2}.$$ (E25)

Also the means are easy to compute:

$$〈K_{\alpha }〉\to K,$$ (E26)

$$〈{\tilde{R}}_{\alpha }〉\to \frac{K-\mu 〈N〉}{1-{\sigma }_c^{2}v}.$$ (E27)

This allows us to make theoretical predictions for how the resource abundances are correlated to resource capacities.

3. Mutual information between individual fitness and true growth rate

The mutual information between two Gaussian variables x and z is simply (note the means of these random variables do not contribute so we will assume them are zero mean random variables):

$$I\left(x,z\right)=\frac{1}{2}\ln \left(\frac{{\sigma }_x^2{\sigma }_z^2}{{\sigma }_x^2{\sigma }_z^2-{\sigma }_{xz}^2}\right).$$ (E28)

Thus,

$$I\left(f,g^{inv}\right)=-\frac{1}{2}\ln \left(1-c_{\rho }^2\right).$$ (E29)

This gives us a theoretical prediction using the predicted form for the correlation coeficient (E15).

Appendix F: Additional simulations and notes

We discussed how the theoretical curves were generated in Appendix C. The numerical simulations were performed by solving the corresponding ODEs (4) and integrating numerically until time 50, 000 with 1000 steps. Although it is not always needed, we improved the accuracy by additionally including a small amount of migration noise which we lowered linearly to a negligible roundoff error over the course of the integration to help ensure that a species that was favored to survive would not go extinct.

We also ran simulations in other regimes, such as the one shown in Fig 5 where we consider fixing μ_c = 1 while varying σ_c to study the setting when we are less interested in comparing specialists to generalists and more interested in the effect of niche overlap and how a high overlap in the generating distribution can reduce the number of surviving species.