Matrix models for quantifying competitive intransitivity

Abstract

Assessing the relative importance of intransitive competition networks in nature has been difficult because it requires a large number of pairwise competition experiments linked to observed field abundances of interacting species. Here we introduce metrics and statistical tests for evaluating the contribution of intransitivity to community structure using two kinds of data: competition matrices derived from the outcomes of pairwise experimental studies (C matrices) and species abundance matrices. We use C matrices to develop patch transition matrices (P) that predict community structure in a simple Markov chain model. We propose a randomization test to evaluate the degree of intransitivity from these P matrices in combination with empirical or simulated C matrices. Benchmark tests revealed that the methods could correctly detect intransitive competition networks, even in the absence of direct measures of pairwise competitive strength. These tests represent the first tools for estimating the degree of intransitivity in competitive networks from observational datasets. They can be applied to both spatio-temporal data sampled in homogeneous environments or across environmental gradients, and to experimental measures of pairwise interactions. To illustrate the methods, we analyzed empirical data matrices on the colonization of slug carrion by necrophagous flies and their parasitoids.

INTRODUCTION

Species differ in their competitive ability, and these differences may translate to observed inequalities in species’ relative abundances within multi-species assemblages (Meserve et al. 1996, Levine and Rees 2002). Ecologists have devoted much effort inferring competitive processes from observed patterns of species abundances and morphology, and from changes in the temporal and spatial distribution of species (Gotelli and Graves 1996, Chesson 2000). Such data are often summarized in a sample matrix A, in which rows represent species i, columns represent different sites j (or multiple sampling times at a single site), and entries are the abundance or incidence of species i at site (or time) j. Classic assembly rules models (Diamond 1975), derived from the principle of competitive exclusion (Gause 1934), predict that differences in competitive abilities should cause non-random patterns of species occurrences among sites and generate inequalities in species abundances within sites. Competitively inferior species are predicted to occur less frequently and at lower abundance, and an important and largely unresolved question is how such species can persist in a community over long time periods (Fox 2013).

Many theoretical models of competitive interactions assume that species can be ranked unequivocally (A>B>C…>Z) according to their competitive strength or resource utilization efficiency (e.g. Tilman 1988). However, intransitive competitive networks (Gilpin 1975) can generate loops in the hierarchy of competitive strength (e.g. the rock-scissors-paper game, in which A>B>C>A). Theoretical studies have shown that competitive intransitivity can moderate the effects of competition, allowing weak competitors to coexist with strong ones (Huisman et al. 2001, Kerr et al. 2002, Laird and Schwamp 2006, 2009, Reichenbach et al. 2007). The degree of intransitivity in the strength of competition may change depending on environmental heterogeneity (Allesina and Levine 2011), successional stage (Worm and Karez 2002), or the presence of consumers (Paine 1984).

Despite the conceptual simplicity of intransitive competitive hierarchies, the empirical estimation of the strength of competition and the frequency of competitive intransitivities in nature has proven difficult. Estimation is possible for small assemblages, because researchers can perform separate competition experiments for every unique pair of species (Grace et al. 1993; Shipley 1993; Keddy and Shipley 1989). However, because there are m(m-1)/2 such pairs, it quickly becomes impractical to test all such pairs for even a moderately-sized community. Perhaps as a consequence of this limitation, intransitivity as a driver of community structure (e.g. species diversity) has mainly been studied in a conceptual (e.g. Bowker et al. 2010, Bowker and Maestre 2012) and mathematical (e.g. May and Leonard 1975, Laird and Schwamp 2006, 2009, Rojas-Echenique and Allesina 2011) framework; existing models have rarely been applied to empirical data (but see Grace et al. 1993; Soliveres et al. 2011).

Theoretical work on the impact of intransitive competition hierarchies begins with the assumption that the structure and strength of the competitive networks are already known, but in real communities such competitive hierarchies are generally unknown (but see Miller and Werner 1987, Keddy and Shipley 1989 for examples). Instead, ecologists try to infer quantitative interactions from observed patterns of species incidences, abundances, or co-occurrences (Ovaskainen et al. 2010, Ulrich and Gotelli 2010). Such inferences are challenging, because replicated samples from real assemblages often exhibit multiple contrasting patterns of species co-occurrence that may be caused by multiple mechanisms, including biotic interactions such as competition and facilitation, as well as abiotic responses reflecting niche conservatism and habitat filtering (Gotelli and Ulrich 2012, Ulrich et al. 2012).

Laird and Schamp (2006, 2008, 2009) initiated approaches to inferring the existence and strength of competitive intransitivity from patterns of species occurrences among replicated patches. Such a meta-community approach (Leibold et al. 2004, Holyoak et al. 2005) is attractive because it uses information on species occurrences from different patches, and thus integrates over habitat differences and environmental gradients that might influence occurrence probabilities. For a set of m species, Laird and Schamp (2006) coded competitive superiority and inferiority in a matrix of competitive outcomes. For m species, this m × m matrix defines the outcome of the interaction between the species pair in row i and column j. If the row species wins, the entry is 1, and if the column species wins, the entry is 0. The diagonal entries of such a matrix are undefined. The distribution of the 1s and 0s in the outcome matrix potentially contains information on the degree of competitive intransitivity.

Specifically, Laird and Schamp (2006) interpreted the degree of nestedness (the ordered decrease of 1s in the outcome matrix after ordering according to row and column totals; Ulrich et al. 2009) as a measure of transitivity, such that a perfectly nested matrix is also perfectly transitive. However, competitive interactions rarely can be described by deterministic dichotomous outcomes (win or lose) in nature, where competitive exclusion is uncommon even at equilibrium (Tilman 1994, Chesson 2000). Therefore, we need more realistic methods that allow for probabilistic outcomes of species interactions estimated from species abundance data, which are widely available.

In this paper, we propose metrics and statistical tests for evaluating the contribution of intransitivity and other patterns of competitive interaction to community structure in real communities. We develop a framework for the analysis of two kinds of data: competition matrices derived from the outcomes of pairwise experimental studies (hereafter C matrices) and species abundance matrices (hereafter A matrices) derived from field samples that are replicated in time or space. Our approach is based upon the construction of patch transition matrices (P), such as those used in Markov chain models. We propose a randomization test to evaluate the degree of intransitivity from these P matrices in combination with empirical or simulated C matrices. In the first part, we relate empirically-derived competition matrices C to an explicit colonization-interaction model to obtain patch transition matrices P. In the second part, we use a ‘reverse engineering’ approach to infer the structure of the competition (C) and the transition (P) matrices from an empirical (temporal or spatial) A matrix. There is no unique solution to this problem because a large number of different competition matrices (C) can generate the same patch transition matrix (P) that will reproduce the A matrix. However, by simulating a large set of stochastically created C matrices, the set of matrices that provide the best fit to an empirical A matrix can be analyzed with respect to their transitivity patterns. Finally, we develop and test different indices to quantify the degree of intransitivity from the underlying C and P matrices for different types of data. To illustrate our methods, we analyze empirical data matrices on the colonization of slug carrion by necrophagous flies and their parasitoids (Ulrich 1999).

A MATRIX ALGEBRA APPROACH TO INTRANSITIVITY

Defining the pattern

Traditionally, researchers have organized experimentally estimated measures of pairwise competitive strengths among m species as a square m × m matrix C in which an entry c_ij is the probability that species i replaces species j in a competitive interaction (Fig. 1). In such a matrix, the diagonal elements are set to 1.0, and therefore, neither the row nor column entries sum to unity. However, the complementary matrix elements c_ij and c_ji always sum to unity (Fig. 1) because, for any pair of species, c_ij = (1.0 − c_ji). These C matrices are often transformed to dichotomous 0/1 data (i.e. species i completely replaces species j or vice versa [c_ij = 1 or c_ji = 1]) in models aimed to test the degree of intransitivity in a given community (Laird and Schamp 2006).

Figure 1.

Four competitive strength matrices (C1 to C4) and the corresponding column stochastic transition patch matrices (P1 to P4) generated from Equations 8 and 9. Entries in the competitive strength matrices designate the probability that species A wins in pairwise competition against species B. Entries in the patch transition matrices (which are column stochastic) designate the probability that a patch occupied by species A is converted to a patch occupied by species B.

It is difficult, however, to relate the entries in the C matrix directly to the relative abundances of species in an assemblage at equilibrium because relative abundances will depend simultaneously on the outcomes of all possible pairwise interactions (Engel and Wetzin 2008). The C matrix itself is not column stochastic (column sums do not sum to 1.0), so it cannot be used to estimate a vector of relative abundances in a Markov chain model without further information (for instance on resource use; see Allesina and Levine 2011). To translate competitive strengths coded in the C matrix to species abundances, we need an additional patch matrix (P) that is column stochastic. The P matrix specifies the probability of transition from one species occupancy state to another, given the underlying competitive strengths in the C matrix.

Transforming a competition matrix (C) to a transition matrix (P)

A simple Markov chain model that predicts relative abundances is Horn’s (1975) classic patch transition model. In this model, an m × m patch transition matrix P describes the probability p_ij of a transition from a patch occupied by species i to a patch occupied by species j in a single time step. Here we adapt this model to predict relative abundances from competitive coefficients coded in C. Although the transitions in Horn’s (1975) model are determined by the outcome of species interactions, their precise relationship to the elements of the C matrix is not clear. Horn’s (1975) general model can also describe turnover that is neutral or that reflects facilitation (McAuliffe 1988),

Assuming that the probabilistic outcome of species interactions are fully described by the entries of C, we want to compute the patch transition matrix P in terms of the competition matrix C. Here are the additional assumptions in our model:

1. There are many homogeneous patches, each of which can be colonized and occupied by individuals of a set of m species;

2. All species produce a large number of potential propagules, so there is a ‘propagule rain’, and colonization is never limited by dispersal limitation;

3. Only a single species can occupy one patch at a time;

4. In a single time-step, a species occupying a patch either retains its occupancy or is replaced by a different species;

5. During a single time-step, each resident species in a patch will engage in a pairwise competitive encounter with all remaining (m − 1) species that do not occupy that patch. The (m − 1) invading species may all interact with the resident species and each one can potentially replace it, depending on the probabilistic outcome of competition between the resident and the invader. The order in which these encounters with the resident species occur is not important and the probability that the resident species will engage in a pairwise competition with other (m − 1) remaining species is uniformly distributed and equals 1/(m − 1).

6. The competition in a given patch stops when an invading species wins over the resident and over the rest of invaders or when the original resident defeats all of the potential invaders.

A model satisfying the above assumption might describe a sessile resident species of plant or invertebrate that competes with the mobile propagules of all potentially invading species. With this model, we can use simple probability rules to convert the entries in the C matrix into a P matrix. We denote the probabilities in the P matrix that species i is replaced by species j by p_ij. For example, consider an assemblage of m = 3 species {1, 2, 3}, with the m × m competition matrix C. The off-diagonal entry c₁₂ in the C matrix specifies the probability that species 1 wins over species 2. The probability p₁₁ that species 1 is not replaced by species 2 or 3 is given by

$$p_{11}=p(1>2)\wedge p(1>3)=c_{12}{c}_{13}$$ (1)

Diagonal entries for p₂₂ and p₃₃ are calculated the same way.

This result can be achieved differently: with probability 1/2, species 1 meets species 2, wins over it with probability c₁₂, then species 1 meets species 3 and with probability c₁₃ wins; or species 1 meets first species 3 with probability 1/2 and wins over it with probability c₁₃ and then it meets species 2 winning over it with probability c₁₂. Hence the total probability is

$$p_{11}=\frac{1}{2}{c}_{12}{c}_{13}+\frac{1}{2}{c}_{13}{c}_{12}=c_{12}{c}_{13}$$ (2)

The same logic can be applied to calculate the off-diagonal elements of the P matrix. With probability 1/2, species 2 meets species 1 and loses with it with probability c₁₂ or with probability 1/2, species 2 meets species 3, wins over it with probability c₂₃ and then loses with probability c₁₂ with 2. Hence

$$p_{12}=\frac{1}{2}{c}_{12}+\frac{1}{2}{c}_{12}{c}_{23}$$ (3)

In general the probability p_ij, with 1 ≤ i ≠ j ≤ 3, is calculated as

$$p_{ij}=\frac{1}{2}{c}_{ij}+\frac{1}{2}{c}_{ij}{c}_{jk}$$ (4)

and, for any 1 ≤ i ≤ 3,

$$p_{ii}=c_{ij}{c}_{ik}$$ (5)

where j,k ≠ i.

To see the pattern, consider m = 4. For 1 ≤ i ≠ j ≤ 4 we have

$$p_{ij}=\frac{1}{3}{c}_{ij}+\frac{1}{3}\left(\frac{1}{2}{c}_{ij}{c}_{jk}+\frac{1}{2}{c}_{ij}{c}_{jk}{c}_{jl}\right)+\frac{1}{3}\left(\frac{1}{2}{c}_{ij}{c}_{jl}+\frac{1}{2}{c}_{ij}{c}_{jl}{c}_{jk}\right)$$ (6)

where k ≠ i, j, l and l ≠ i, j, k, and for 1 ≤ i ≤ 4

$$p_{ii}=c_{ij}{c}_{ik}{c}_{il}$$ (7)

where j,k,l ≠ i.

To generate the formula for p_ij, i, j = 1, …,m, for an arbitrary m, we need the following notation: given a set A = {a₁, …, a_n} of species with the corresponding competition matrix C, let P_A[a_j → a_i] denote the probability that species a_j is replaced by a_i, i, j = 1, …, m.

Within this notation: for 1 ≤ i ≠ j ≤ m

$$p_{ij}=P(1,\dots ,m)[j\to i]=\frac{1}{m-1}{c}_{ij}+\frac{1}{m-1}{\Sigma }_{k=1,k\ne i,j}^m{c}_{jk}P(1,\dots ,k-1,k+1,m)[j\to i]$$ (8)

and for 1 ≤ i ≤ m

$$p_{ii}={\Pi }_{j=1,j\ne i}^m{c}_{ij}=\frac{1}{m-1}{\Sigma }_{k=1,k\ne i,j}^m{c}_{ik}P(1,\dots ,k-1,k+1,m)[i\to i]$$ (9)

Equations 8 and 9 generate the required transition matrix P for an arbitrary number m of species in terms of competitive strength matrices for sets consisting of (m − 1) species.

The fact that the transition probability for two species (eq. 8) contains terms that include other species means that a fully transitive competitive strength matrix C is not necessarily transitive with respect to the transition matrix P (Fig. 1). A fully transitive C matrix translates into a transitive P matrix only if competitive strengths in C are either constant or increase in each row from left to right (Fig 1, C2, C3). This feature is equivalent to a fully quantitatively nested pattern of competitive strength (Ulrich et al. 2009). If this ordering is broken, a transitive C matrix translates always into an intransitive P matrix (Fig. 1, C4). Thus, it is important to quantify intransitivity in both the P matrix and in the underlying C matrix.

We note that the right eigenvector of the simple Markov chain model predicts the relative abundances of all species at equilibrium. This model implies that whether or not the P matrix is dominated by transitive or intransitive chains, the more pairwise interactions in which a particular species wins, the greater its abundance at equilibrium (Allesina and Levine 2011). Therefore pronounced differentiation in species abundances should be correlated with a predominance of transitive species interactions while highly even abundance distributions of potentially competing species indicate the dominance of intransitive loops.

Estimating Transitivity Patterns With Three Data Structures

1. Temporal data

To estimate the degree of intransitivity in a given community, we need first to estimate the transition matrix P from an observed distribution of species abundances or occurrences. Depending on the data there are three different scenarios. The first and most obvious approach relies on appropriate time series data. If A_t is the vector of relative abundances of m species at time t, then PA_t=A_t+1 if P is column stochastic. If data are available from at least t+1 time steps, the single abundance vectors of each step can be converted into two matrices N_1,t, which runs from generation 1 to t, and N_2,t+1, which runs from generation 2 to t+1. Combining these two matrices yields:

$${\mathbf{PN}}_{1,t}={\mathbf{N}}_{2,t+1}$$ (10)

and

$$\mathbf{P}={\mathbf{N}}_{2,t+1}{{\mathbf{N}}_{1,t}}^{\mathrm{T}}{\left({\mathbf{N}}_{1,t}{{\mathbf{N}}_{1,t}}^{\mathrm{T}}\right)}^{-1}$$ (11)

where ^T denotes the transpose. This approach allows for the estimation of the P matrix from an A matrix of consecutive temporal censuses of an assemblage. Although a unique solution for P usually exists, in many cases the variability in species abundances not caused by competitive effects will return P matrices that are not column stochastic and that do not allow for an assessment of competitive interactions. Therefore we used a ‘reverse engineering’ approach to find those C and P matrices that best mimicked observed abundances. For this task, we generated a large number (n = 100,000) of randomly assembled C matrices, in which each entry above the diagonal in the C matrix was chosen from a random uniform [0,1] distribution. Using eq. 8 and eq. 9, we transformed the randomly assembled C matrices into P transition matrices to predict the N_2,t+1 matrices from our Markov chain model. We used average rank order correlations between respective columns in the predicted and observed N_2,t+1 matrices to assess goodness of fit, and selected those P and C matrices that generated the best fit to the observed vector of relative abundances (A_t). A worked example of this approach is presented in Fig. 2.

Figure 2.

Worked example of the benchmark testing procedure, which consists of four steps. (1) translation of a competition matrix C_test into a patch transition matrix P_test. (2) Calculation of the column vectors of the U_test matrix that simulate the abundance distribution of species (5 rows) among sites (7 columns) by randomly filling the U_test matrix proportion to the values of the right eigenvector EV of the P_test matrix. (3) Calculation of the dot products between each U_test column vector and the matrix P_temp matrices (obtained from C_temp) generates a predicted species × sites matrix; the average Spearman rank correlation between the respective sites serves as a measure of goodness of fit. We repeat this procedure at least 10,000 times and retain the best fitting matrix C_pred. (4) Finally, we generate the associated matrices of the numbers of intransitivities, and calculate for both C_test and C_pred the degrees of transitivity (TrC and TrP), the proportions of species engaged in intransitive species pairs (fC and fP). Except for first step translation from the C to the P matrices, the same benchmark procedure applies when starting directly from a P matrix.

2. Spatial and environmental data

The second approach is based on spatial abundance data for m species collected at i = 1 to n sites for which environmental variables are available. Assume a number of homogeneous patches. If observed species abundance distributions were determined only by negative species interactions, we could make a time-space substitution and interpret the vector A_i of the abundance distribution of m species at site i as representing the outcome of a single time step of a Markov chain triggered by the m×m matrix P. At equilibrium, we can assume that the abundance distribution A_i at site i (scaled to unity) approximates the right stable state vector (eigenvector) U (CU=ΛU), with Λ as the diagonal matrix of the largest eigenvalue λ. Thus it holds approximately

$${\mathbf{PA}}_{\mathrm{i}}\approx \lambda {\mathbf{A}}_{\mathrm{i}}$$ (12)

We note that this eigenvector exists only if the matrix P defines an ergodic process, i.e., a process that converges to an equilibrium condition. But if P defines a periodic competitive hierarchy, no stable state vector exists, and species abundance distributions do not reach equilibrium. For instance, the intransitive competitive chain A>B>C>A is periodic and does not achieve an abundance equilibrium. Although such a system does not reach a point equilibrium, we could assume the distribution of abundances is continuous, in which case we can approximate P from a large number of steps n in (P)ⁿA_i ≈ A_i.

In the case of n study sites, the single column vectors A_i form the m×n matrix U that contains the abundance distributions of the m species among the n sites. These can be viewed as individual approximations to the stable state vector U_i. This is equivalent to

$$\mathbf{PU}=\mathbf{U}$$ (13)

Therefore, the problem of identifying the patch transition matrix P is reduced to the problem of solving P in eq. 13. For this task, we decompose the variance in PU into a part explained by U and a part contained in an m×n matrix E to get the model

$$\mathbf{PU}=\mathbf{U}+\mathbf{E}$$ (14)

We next incorporate environmental data to estimate the matrix E from a multiple regression analysis (see Ovaskainen et al. 2010 for a related approach). These data are contained in a n × h matrix H, with n sites (rows) and h environmental variables (columns) measured at each site. The occurrence vector B_i of a species i at the n sites predicted by H is given by

$${\mathbf{B}}_j={\mathbf{HX}}_j$$ (15)

where X is the vector of regression parameters. Computed for all species, the regression model yields a predicted n×m matrix of species abundances that is identical to U^T=HX, and that provides an estimate of that part of E that is not explained by competition [(HX)^T=E]. This unexplained part can be plugged into eq. 14 to yield

$$\mathbf{PU}=\mathbf{U}+{\mathbf{X}}^{\mathrm{T}}{\mathbf{H}}^{\mathrm{T}}$$ (16)

Therefore

$$\mathbf{P}=\mathbf{I}+{\mathbf{X}}^{\mathrm{T}}{\mathbf{H}}^{\mathrm{T}}{\mathbf{U}}^{\mathrm{T}}{\left({\mathbf{UU}}^{\mathrm{T}}\right)}^{-1}$$ (17)

with I being the m×m identity matrix. Eq. 17 provides testable predictions about the competitive structure of the focal community. As with time series data, we would use the ‘reverse engineering’ approach and compare predicted and observed environmental terms X^TH^T (eq. 16) to find those C and P matrices that best mimic the observed abundance distributions.

3. Spatial data

Without additional environmental information for a precise definition of the minimum condition for E, eq. 14 has no unequivocal non-trivial solution (Kryszewski unpubl.). Therefore we use again the ‘reversed engineering’ approach and assume that P predicts U best if the average rank correlations of all equivalent columns between predicted (PU) and observed (U) are above a predefined level r_min. Below, we use r_min =0.95. We define this case as the minimum state of E = PU-U. As before, we create a large number of randomly assembled C matrices, transform them into P matrices, and calculate PU. The respective rank order correlations between equivalent matrix columns in the PU and U abundance matrices serve as a measure of goodness of fit. The best performing C and P matrices are then candidates for the competitive and transition matrices that generated the distribution of abundance in U (Fig. 2).

An important limitation of this method is that the final competition matrix does not necessarily coincide with a P matrix that produces the observed pattern of species abundances. If strong environmental heterogeneity and environmental filtering (Webb et al. 2002) influence species abundances, the matrix E will contain more variance than under the minimum condition. If environmental heterogeneity dominates the pattern of species abundances, P might underestimate the true strength of competition. Conversely, in cases where heterogeneity enhances disparities in abundance, P might overestimate the strength of competition. Both scenarios could bias the assessment of the importance of intransitivity. Thus a proper assessment of competitive hierarchies should be based on homogeneous set of relatively similar study sites for which environmental variables do not contribute greatly to the variance contained in E.

Eq. 14 highlights an important property of the transition matrix P. If all abundance distributions A_i in the n sites of U have the same ranking (all combinations of rank correlations among sites = 1.0), the P matrix generates for all representations of A_i a constant abundance hierarchy. This occurs if the C matrix is highly (or even maximally) transitive. Conversely, the lower the average rank correlations within U, the greater the degree of intransitivity in P.

Quantifying transitivity in C matrices

Laird and Schwamp (2006) proposed several useful metrics to quantify the degree of intransitivity in discrete (0/1) C matrices. However, these indices do not work with C matrices that have probability entries 0 < p < 1. A simple measure of the degree of transitivity in probabilistic C matrices is the count N of species pairs for which p_ij < p_ji after the matrix has been sorted to maximize the number of matrix elements with p > 0.5 in the upper right triangle (Petraitis 1979). Although for each individual species pair p_ij = (1 − p_ij), the number of entries that end up above the diagonal after sorting should reflect the number of transitive chains in the matrix. The normalized version of this transitivity count (TrC) is a quantitative measure of the degree of transitivity in C:

$$TrC=\frac{2N(p_{ij}<p_{j,i})}{m(m-1)}(\mathrm{i}\ne \mathrm{j})$$ (18)

Quantifying transitivity in P matrices

To quantify transitivity in the transition matrix P, we propose the normalized count of the number of reversals in the decreasing order of probabilities for each column as an appropriate metric (Fig. 1):

$$TrP=1-\frac{4N(p_{ij}<p_{k,j})}{m(m-1)(m-2)}(\mathrm{i}<\mathrm{k}\text{and}i,k\ne \mathrm{j})$$ (19)

where i and j run from 1 to m and k from i+1 to m (m is the total number of species in the studied community). Theoretically, this measure runs from 0.0 (completely intransitive) to 1.0 (fully transitive). However, because the entries in the P matrix are calculated from the probabilities in the C matrix, they are not independent of one another, and the matrix ordering of species according to the largest eigenvector imposes constraints on the lower boundary. In a simulation study of randomly filled C and P matrices, we found a rapid asymptotic convergence of the lower boundary towards TrP = 0.5 with increasing species richness, with a minimum observed value TrC_min = 0.25 (cf. Petraitis 1979).

For both the C and the P matrices, another possible measure of intransitivity is f_intr, the fraction of species in the assemblage that participate in intransitive loops. However, this metric always tends to include a large number of species—even for moderately intransitive communities— and therefore cannot discriminate well among transitive and intransitive matrices. In nearly all test matrices (see below) for which TrP and TrC scores were less than 0.7, all species took part in at least one intransitive relationship. We therefore used f_intr as an auxiliary metric in cases of high, but imperfect, transitivity.

METHODS

Before a new randomization method is applied to ecological data, it should be subject to benchmark testing with artificial data sets to assess its precision and statistical performance (Gotelli and Ulrich 2012). For each of the three approaches described above, we first generated 200 competitive strength matrices C_test (with m taken from a uniform random distribution: 5≤m≤50). Each matrix had a different predefined degree of intransitivity I (0.7≤I≤1) implying that no (I = 1) or (nearly) all (I < 0.7) species were part of at least one intransitive loop. This specified degree of intransitivity serves as the true or known parameter, so the set of matrices can be used to test the accuracy of the procedures and metrics and their ability to correctly detect intransitivity. For the environmental and abundance approaches, we transformed these artificial C_test matrices into their respective P_test transition matrices (eq. 1 and 2) and calculated for each transition matrix the respective right eigenvector V_test. This eigenvector served as an estimate of the equilibrium distribution of the relative abundances of the species (Fig. 2).

Proportional to this distribution, we randomly sampled individuals and assigned them to the elements of a matrix of m species and n sites (n again was sampled from a uniform random distribution: 5≤n≤50) until, at each site, all m species were placed. This procedure yields for each site a normal approximation of the equilibrium abundance distribution and defines the required abundance matrix U_test. For the environmental data approach, we also generated three environmental variables, each with values sampled from a random uniform [0,1] distribution, that served as an additional source of variability for species abundances.

For the time series approach, we generated m < n ≤ 50 sequential abundance vectors by multiplying the species abundance distribution at time step t with P_test to obtain the abundance distribution at time step t+1. To introduce additional variability, all species abundances at each time step were multiplied by a uniform random number arbitrarily ranging between 0.8 and 1.2. Thus, approximately 80% of the variance in a species relative abundance was determined by competitive interactions and approximately 20% was random.

In the next step, we simulated for each C_test matrix 10×m×n, but not less than 100,000, random C_t matrices with entries of the upper triangular matrix sampled from a uniform random [0,1] distribution and translated each into a corresponding P matrix. To estimate the goodness of fit, we calculated for each of the random matrices the average rank order correlation between the columns of the U_test matrices (N_2,m+1 in the case of the time series approach) and the respective columns of the predicted (U_pred and predicted N_2,m+1) matrices (Fig. 2). We retained for each C_test matrix and its corresponding P_test matrix the 100 best-fitting matrices C_pred and P_pred. We used this set of best-fitting matrices to compare estimated and true patterns of intransitivity and to identify the species with the strongest intransitivities in the C_test and P_test matrices. We also used these best-fitting matrices to compare the degree of transitivity TrC that occurred in intransitive loops.

The calculation of the total probability for all p_ij of the transition matrix P from the entries of the competition matrix C needs the evaluation of all combinations of c_ik (k ≠ i,j) according to the recursive eq. 8. This becomes computationally challenging at higher species richness. A good approximation of p_ij uses the fact that the calculation of each p_ij (eq.8) involves multiplications of all combinations of c_ik within each row i. Therefore we might reduce eq. 8 to a geometric series using the geometric average $\stackrel{‒}{x}$ of the respective c_ik values. This leads to

$$p_{ij}\approx \frac{1}{m-1}{c}_{ij}{\Sigma }_{k=0}^{m-2}{\stackrel{‒}{x}}^k$$ (20)

For our test matrices the average relative error introduced by this approximation for p_ij was always less than 3% (not shown). Below we use this approximation to efficiently estimate P from C.

Case study

To test our three methods, we used data from a controlled field experiment on the colonization of slug carrion (Arion ater, Arionidae) by eight species of necrophagous Diptera and five species of oligophagous primary hymenopteran parasitoids of the Dipteran genus Megaselius (Phoridae) in a temperate German beech forest (Table 1; detailed description in Ulrich 1999). A total of 99 dead slugs were each assigned to one of nine weight classes (2-3, 3-4,…,9-10, 12-19 g fresh weight) and exposed for 30 days in the field in polystyrol boxes, which allowed flies and parasitoids access, but excluded larger arthropod and vertebrate scavengers. In the same beech forest, the density of Hymenoptera parasitoids associated with the Diptera in this experiment was monitored for seven years (1981-1987; full description in Ulrich 2001). We used average densities (individuals × m⁻²) for the first (spring) and the second (summer) generation of these parasitoids.

Table 1.

Numbers of correctly identified test matrices of a predefined degree of transitivity (TrP and TrC) in benchmark tests based on spatial, time series, and environmental data. N denotes the total number of simulated matrices in each category (from a total of 200 test matrices) and ‘identified’ gives the number of tests for which the upper 95% confidence limit of the 100 best-performing reverse-engineered P or C matrices included the value 1.0 (predefined TrC, TrP = 1) or excluded the value 1.0 (predefined TrC, TrP < 1).

Degree of transitivity	Spatial			Time			Environment
	N	Identified	% identified	N	Identified	% identified	N	Identified	% identified
	P matrices
TrP=1	30	18	60	20	9	45	25	9	36
0.95≤TrP<1	43	40	93	27	20	74	33	31	94
TrP<0.95	127	127	100	153	144	94	142	142	100
	C matrices
TrC=1	35	17	49	40	20	50	33	20	61
0.95≤TrC<1	16	13	81	18	13	72	20	18	90
TrC<0.95	149	131	88	142	114	80	148	130	88

Additionally, we used data from a leaf-litter manipulation experiment conducted in 1986 to assess the influence of environmental variability on competitive strength (full description in Ulrich 2001). In this experiment, leaf litter in two experimental plots was either totally or partially removed and on two other plots was supplemented two- or five-fold (Ulrich 2001). Collectively, these data provide all the information necessary to calculate the three proposed methods for the same set of species within a single habitat. To estimate transitivity, we used the summed biomasses of each of the Diptera species, parasitism rates for the parasitoids in the slug experiment, and the abundances for the parasitoids in the time series data set. The complete raw data are provided in the associated Dryad storage data set (doi:xxx).

To assess the degree of community-wide negative species associations, we used the incidence based C-score (Stone and Roberts 1990) and its abundance-based equivalent, CA (Ulrich and Gotelli 2010). Both metrics need to be compared with a null model that provides an appropriate random expectation (Gotelli and Ulrich 2012, Ulrich and Gotelli 2013). For the Diptera and hymenopteran parasitoids, the individual slugs represent sites, which differed in size and desiccation rate. The different Diptera and Hymenoptera species also differed in their local abundances and incidences, which probably reflects differences in colonization potential.

To account for these typical sources of heterogeneity among sites and among species, we used the fixed abundances null model IT of Ulrich and Gotelli (2010), which was developed for the randomization of abundance matrices. The IT algorithm assigns individuals randomly to matrix cells with probabilities proportional to observed row and column abundance totals until, for each row and column, total abundances are reached. Null expectations and standard deviations of the IT null distributions were in all cases based on 200 randomizations for each matrix. All analyses were conducted with the FORTRAN software application Turnover (Ulrich 2011, Ulrich and Gotelli 2012). Because null distributions appeared to be approximately normal, we used the standardized effect sizes (SES, calculated from the observed score x and the mean μ and standard deviation σ of the null distribution as: SES = (x-μ)/σ) to infer statistical significance For a two-tailed 95% confidence interval, “random” SES scores should range between ±1.96. Source code for all computer programs is available from WU upon request.

RESULTS

Benchmark testing

We found highly significant correlations between simulated and predicted degrees of transitivity, regardless of the approach used to recover competitive interactions from abundance data (Fig. 3). Our ‘reverse engineering’ algorithm performed best for P matrices in combination with the spatial and environmental data (Figs. 3A, B). In these analyses, the regression of estimated versus true transitivity explained over 94% of the variance found in the data. Our methods were less successful at estimating pairwise competitive strength, and the respective regressions explained only between 51% (abundance data, Fig. 3F) and 60% (environmental data, Fig. 3D) of the variance.

Figure 3.

Simulated (TrP_test, TRC_test) and predicted (TrP_pred, TrC_pred) degrees of transitivity of competitive strength C and transition P matrices for the abundance method (A, B), the time series method (C, D), and the environmental correlation method (E, F). Each point represents a the scores from a different simulated matrix (n= 200 matrices). Regression lines: A: r² = 0.94; B: r² = 0.51; C: r² = 0.44; D: r² = 0.53; E: r² = 0.94; F: r² = 0.60. All P(r²>0) < 0.01.

Despite variability in the prediction of the precise degree of transitivity, all three approaches were able to identify at least moderate degrees of intransitivity in test matrices (Table 1). For P matrices, each of our three approaches correctly recovered more than 94% (time series approach) of the moderately to highly intransitive test matrices, with TrP < 0.95 (Table 1). For C matrices, at least 80% (times series) of them were correctly identified. Of the weakly intransitive matrices (0.95 < TrP < 1.0) between 74% (time series) and 94% (environmental data) were identified as being intransitive by the P matrices, and between 72% (time series) and 90% (environmental data) of them were identified as being intransitive by the C matrices.

These methods were less successful in identifying perfectly transitive matrices (Table 1). For P matrices, between 36% (environmental data) and 60% (abundance data) of the upper 95% confidence limits of the TrP distributions of the 100 best-performing matrices included the value of 1.0 (full transitivity). For C matrices, between 49% (abundance data) and 61% (environmental data) were correctly identified as transitive. In all of the fully transitive test matrices, the predicted transitivity scores of the best-performing engineered P and C matrices was > 0.95 (not shown). Therefore, values of TrP or TrC > 0.95 might serve as a strong indicator of full transitivity.

Case study

The eight dipteran and five hymenopteran species differed markedly in abundances, biomass, and parasitism rates (Table 2). Abundances of Diptera and Hymenoptera ranged between 0.2 and 52 individuals per slug, which corresponds to 19 individuals of the least abundant species and 5186 individuals of the most abundant species. Irrespective of carrion weight class, species incidences and abundances of both taxa tended to be significantly segregated (Table 3), suggesting a predominance of negative species interactions.

Table 2.

Average densities (D, individuals per slug), dry biomasses (B, mg dry weight per slug), and parasitism rates p of eight necrophagous Diptera and five primary parasitoids of Megaselia ruficornis and M. pulicaria. Errors denote plus or minus one standard deviation. Average rank and range of ranks give the mean and the range of species ranks in the competitive hierarchies of the ten carrion weight classes.

Species	D	B	Average rank	Range of ranks
Diptera
Conicera schnittmani	51.9±80.4	0.030±0.051	2.2	1-6
Fannia immuntica	1.0±2.8	0.004±0.012	4.6	2-8
Gymnophora arcuata	0.6±1.8	0.001±0.004	3.8	2-8
Limosina spec	23.2±34.1	0.014±0.020	4.7	1-8
Megaselia pulicaria	6.7±13.9	0.007±0.015	5.3	2-8
Megaselia ruficornis	8.2±9.3	0.009±0.010	5.0	1-8
Panorpa spec	0.4±0.9	0.008±0.021	5.8	4-8
Psychoda spec	3.3±7.3	0.003±0.007	4.6	2-8
Hymenoptera
Aspilota A	3.4±5.8	0.22±0.30	1.6	1-2
Aspilota B	3.4±8.1	0.18±0.28	2.0	1-3
Aspilota C	0.5±1.4	0.04±0.16	4.0	3-5
Aspilota D	0.2±1.1	0.01±0.07	4.9	4-5
Orthostigma spec	2.7±4.4	0.17±0.28	2.5	1-4

Table 3.

Matrix-wide measures of negative species associations based on standard effect sizes (SES) of the incidence based C-score and the abundance based CA score for species × slug carrion matrices of 10 carion weight classes (in g). TrP and TrC are the average transitivity metrics based on the 100 best fit transition and competition matrices (eqs. 11 and 12). P_P(1) and P_C(1) give the probabilities that the distribution of TrP and TrC based on the 100 best-fitting P an C matrices include the fully transitive pattern of TrP = 1 and TrC = 1.

Snail weight class	SES C-score	SES CA	TrP	PP (1.0)	TrC	PC (1.0)
Diptera
2	0.89	2.68	1.00	>0.50	1.00	>0.50
3	5.36	5.98	0.98	0.01	0.93	0.31
4	5.84	6.36	0.95	0.01	0.96	0.21
5	6.98	8.13	0.98	0.12	0.96	>0.50
6	4.64	6.72	0.99	0.09	1.00	>0.50
7	7.78	12.55	0.99	0.10	1.00	>0.50
8	4.32	8.67	1.00	0.13	1.00	>0.50
9	7.52	7.50	0.98	0.01	1.00	>0.50
10	8.60	13.41	1.00	>0.50	1.00	>0.50
12	4.86	15.19	0.98	0.01	0.96	0.41
All	13.73	20.24	0.99	0.08	0.96	>0.50
Hymenoptera
2	2.41	1.58	0.98	>0.50	1.00	>0.50
3	1.79	2.79	0.98	>0.50	1.00	>0.50
4	3.99	2.01	0.97	0.35	0.98	0.31
5	1.79	1.38	1.00	>0.50	1.00	>0.50
6	1.43	1.41	1.00	>0.50	1.00	>0.50
7	3.41	2.89	1.00	>0.50	1.00	>0.50
8	6.51	3.31	0.98	0.03	0.97	0.02
9	3.04	3.19	0.98	0.13	1.00	0.14
10	3.75	4.01	0.95	0.14	1.00	0.15
12	4.43	3.17	0.93	0.01	0.87	0.02
All	5.71	5.11	0.98	0.10	0.98	0.09

The average predicted degree of transitivity of the Diptera community was TrP = 0.99 and TrC = 0.99, respectively (Table 2). The level of transitivity did not change with carrion weight class (r = 0.13, P > 0.1) or with the degree of negative species association (r = −0.12, P > 0.1). Five of the predicted TrP scores and all predicted TrC scores were not significantly different from 1.0, a score which indicates full transitivity (Table 3). Thus competitive hierarchies within the fly communities appeared to be fully, or at least nearly fully, transitive. In accordance with our probabilistic interpretation of pairwise species interactions, the degree of transitivity differed between the pairwise competitive strength (C) and the transition (P) matrices with the latter showing a stronger pattern of intransitivity (Table. 3).

For the Hymenoptera, there was a trend towards increasing intransitivity (TrP) at higher slug weight (r = −0.82, P < 0.01). The confidence limits of TrP of the 8g and 12g carrion weight classes did not encompass 1.0 (Table 3). TrP was also negatively correlated with the degree of species segregation (r = −0.78, P < 0.01). This trend was not obvious for TrC (Table 3). There was a larger proportion of intransitive interactions of the parasitoids in comparison with their dipteran hosts (Table 1). Both dipteran species had on average 2.7% and 1.6% intransitive interactions in the P matrices, and 0.3% and 0.6% in the C matrices (Table 2), while their parasitoids had on average 3.4% (P matrices) and 1.9% intransitive interactions (C matrices (Table 2).

Detailed comparisons of the competitive hierarchies of flies and parasitoid wasps (Table 2), revealed a reordering of species competitive strength between the different carrion weight classes. The average coefficient of correlation between all 45 combinations of predicted species rank orders of competitive strength was r = 0.11 for Diptera and r = 0.76 for Hymenoptera. Predicted rank order of competitive strength of both taxa was for all ten weight classes significantly (P < 0.05) correlated with the respective observed relative abundance distributions, which differed between weight classes (cf. associated Dryad data base).

Both time series (spring and autumn generations) and the data from the leaf manipulation experiment pointed to fully transitive competitive relationships among the five parasitoid species (Table 3). Again, we observed differences in the competitive hierarchy between generations and between the time series and the leaf litter experiment.

DISCUSSION

A central question in community ecology is how diversity is maintained among a set of competing species. Together with niche differentiation and neutral theory (e.g. Tilman 1994, Chesson 2000; Hubbell 2001), empirical (Bowker et al. 2010), mathematical (Laird and Schwamp 2006), and theoretical (Paine 1984) research has pointed to intransitivity in competition networks (Gilpin 1975) as a key mechanism for the maintenance of diversity in natural communities (but see Shipley 1993). However, assessing the relative importance of intransitive competition networks in the field has been very difficult because it requires a large number of pairwise competition experiments linked to observed abundances of the interacting species in the field, something extremely rare to find (Silvertown and Dale 1991, Allesina and Levine 2011). This is likely the reason why empirical research focused on this topic is very scarce, despite the fact that it was introduced over 40 years ago by Gilpin (1975).

The approach introduced here overcomes this problem by estimating competition hierarchies— and their associated degree of intransitivity—among the interacting species from their observed abundances in the field rather than from direct measurements of interaction coefficients from pairwise competition experiments. We developed three methods for reconstructing pairwise competitive strength matrices C using intermediate patch transition matrices P. Because C matrices code all pairwise interactions between the species involved in a competitive network, they are preferable from a theoretical point of view. However, there are so many entries in a typical C matrix, that it may be difficult to estimate them from pairwise competition experiments. P matrices can be more easily estimated from repeated samples of an assemblage of potential competitors, such as successional series. However, if conditions change through time (as in classic succession models; Connell and Slatyer 1977), the P matrix entries will be affected by both species interactions and abiotic conditions in each time step (Zaplata et al. 2013).

As revealed by our benchmark testing, the methods introduced here successfully identify candidate competition matrices that predict abundance distributions that are very similar to the observed ones. Our approach recovers intransitive hierarchies (Fig. 3), and intransitive test matrices always had predicted transitivity values (TrP and TrC) less than 0.95. Thus, we propose this 0.95 value as a rule of thumb to separate communities with a strong transitive hierarchy in their competitive networks from those showing some degree of intransitivity. Environmental heterogeneity can override these patterns (Fig. 3), but a pattern of consistent abundance hierarchies among sites is always a strong indicator of a high degree of competitive transitivity. However, the converse is not true: an invariant abundance hierarchy among sites does not necessarily imply a perfect transitive hierarchy.

Our approach characterizes transitive hierarchies only in terms of simple pairwise species interactions. Complex multi-species interactions and indirect effects may alter the limiting resources and the outcome of pairwise interactions. For example, shading by large plants in drylands may limit light, alter nutrient availability and ameliorate water stress (Moro et al. 1997; Maestre et al. 2003), and these environmental changes can modify the competitive outcome between neighboring species (Soliveres et al. 2011). Additional species might also introduce indirect positive interactions (Levine 1999). In a fully pairwise transitive competition network (A > B, A > C, B > C), species A might enhance the performance of C by suppressing B, and thus causing a competitive loop (Levine 1999). Of particular importance when studying intransitivity are additive competitive effects. For instance in pairwise contests, species A outcompetes B and C, but jointly B and C might outcompete A. Our approach is a first step towards disentangling the possible interactions in multispecies competitive situations, and provides testable hypotheses on the bivariate competitive interactions. These might be verified in subsequent controlled experiments.

A common but unrealistic assumption of many intransitivity analyses conducted to date is that only negative, competitive interactions are important (e.g. Laird and Schamps 2006, Allesina and Levine 2011). However, positive (or facilitative) interactions are ubiquitous, not only among plants (Callaway 2007), but also among many other taxa (e.g. Kawai and Tokeshi 2007, Fugère et al. 2012). Facilitative interactions can increase the degree of intransitivity in a given community in two ways: i) by increasing the number of species that can colonize a given site (Lortie et al. 2004), because there is a greater chance of finding an intransitive network in a large assemblage (Laird and Schamp 2008), and ii) by increasing the heterogeneity in the spatial distribution of resources, which can permit coexistence of weak competitors (Allesina and Levine 2011). Although we can consider these positive interactions with our method by applying it to contrasting microsites (e.g., nurse plant vs open areas; Soliveres et al. 2011), future studies should explicitly include positive interactions directly into pairwise competitive strength (C) or transition probability (P) matrices.

Case study

Our case study revealed differences in the structure of competitive hierarchies of communities of necrophagous flies and their hymenopterous parasitoids. For the Diptera, the most probable pattern was complete transitivity. The hymenopteran parasitoids were characterized by transitive hierarchies at lower carrion weight classes and a tendency towards intransitive loops at higher weight classes (Table 2). Interestingly, the average coefficient of correlation in abundance ranks of the flies among the 10 replicates per carrion weight class was only r = 0.30 ± 0.13 (P < 0.05) and the relative abundances of the hymenopteran parasitoids were even less correlated (r = 0.17 ± 0.09 (P > 0.05). We conclude that stable abundance distributions among sites are not a necessary prerequisite for competitive transitivity. Interestingly, Ulrich (1999) did not observe an increased number of parasitoid species with increasing host density. Therefore, biases introduced by possible effects of matrix sizes (more species) cannot explain the observed differences. A possible trigger might have been the number of dipteran hosts available, which increased linearly with carrion weight (Ulrich 1999). At low density, host availability is a limiting factor and priority effects – but not the survival of host larvae – should be decisive to determine competitive hierarchy. At higher host density, priority effects might be less important, and other factors such as mutualism could become more important.

Predicted competitive hierarchies in Diptera and Hymenoptera were not stable among experimental treatments (Table 1) and across generations of Hymenoptera (Table 3). Communities of necrophagous insects are not usually in equilibrium, and are driven by food inputs (Beaver 1977, Hanski 1987). Priority effects at a suitable carcass largely determine the survival of larvae and the abundance of patches (Ulrich 1999) and therefore control the competitive hierarchy in a carrion patch. As a consequence, hierarchies vary among patches, which may enhance the regional coexistence of species (Allen et al. 1993). Thus, the detection of fully transitive relationships in combination with varying competitive rank order might be an indicator of both non-equilibrium conditions and priority effects. This interpretation is corroborated by the fact that intransitive hierarchies appeared only in the case of Hymenoptera colonizing the largest carrion weight classes with sufficient numbers of host larvae to reduce priority effects in favor of other forms of competition, including differential larval mortality and predation (Peschke et al. 1987).

SUMMARY

To the best of our knowledge, our tests represent the first tools for estimating the degree of intransitivity in competitive networks from observational datasets, widely available in the literature for a vast range of organisms and environments. Benchmark tests with artificial matrices revealed that these metrics could correctly detect intransitive competition networks, even in the absence of direct measures of pairwise competitive strength. These methods can be applied to replicated temporal and spatial data sampled in homogeneous environments or across environmental gradients, and can be applied to experimental measurements of pairwise interactions when they are available.

Table 4.

Time series and environmental data measurs the degree of transitivity in the hymenopteran parasitoids of Megaselia flies. TrP and TrC are the average transitivity metrics based on the 100 best-fitting transition and competition matrices (eqs. 11 and 12). P_p(1) and P_C(1) give the probabilities that the distribution of TrP and TrC based on the 100 best fitting P an C matrices includes the fully transitive pattern of TrP = 1 and TrC = 1.

Variable	TrP	Pp(1.0)	TrC	Pc(1.0)
First Generation	0.963	0.01	1.000	0.50
Second Generation	0.998	0.50	1.000	0.50
Both Generations	1.000	0.50	1.000	0.50
Environment	1.000	0.50	1.000	0.50