Diversity in biology: definitions, quantification and models

Abstract

Diversity indices are useful single-number metrics for characterizing a complex distribution of a set of attributes across a population of interest. The utility of these different metrics or sets of metrics depend on the context and application, and whether a predictive mechanistic model exists. In this topical review, we first summarize the relevant mathematical principles underlying heterogeneity in a large population before outlining the various definitions of ‘diversity’ and providing examples of scientific topics in which its quantification plays an important role. We then review how diversity has been a ubiquitous concept across multiple fields including ecology, immunology, cellular barcoding studies, and socioeconomic studies. Since many of these applications involve sampling of populations, we also review how diversity in small samples is related to the diversity in the entire population. Features that arise in each of these applications are highlighted.

1. Introduction

Diversity is a frequently used concept across a broad spectrum of scientific disciplines, ranging from biology [1-5] and ecology [6-11], over investment and portfolio theory [12-16], to linguistics [17, 18] and sociology [19-24]. In each of these disciplines, diversity is a measure of the range and distribution of certain features within a given population. It is considered a key attribute that can be dynamically varying, influenced by intra-population interactions, and modified by environmental factors. The concept of diversity, variety, or heterogeneity can be applied to any population. The evolution of the population can also be highly correlated with its diversity. Some examples of biological population dynamics occurring at different scales are shown in figure 1. At first sight, diversity seems to be an intuitively simple concept, but since certain population attributes require a full distribution function to quantify, it can be rather complex and difficult to capture using a single metric [3, 4, 25, 26]. We could for example think of a community with a total of four species, with one of the species dominating the total population. Consider a second community that consists of two equally common species. Which one of the two communities exhibits a higher diversity? The first one, because it harbors a larger number of species? Or the second one, because a sample is more likely to contain two species? This example shows that diversity is intrinsically linked to the total number of extant species (richness) and how the population is distributed throughout the species (evenness), and thus cannot be captured by a single number [3]. As a result, there are numerous different diversity indices and associated concepts used in different applications [3, 4, 25-29]. Nonetheless, diversity measures are important for assessing the current condition of ecosystems, to quantify the influence of environmental factors on different species, and in the context of conservation planning [2, 5, 9, 10, 29-31]. In addition, the concept of diversity is important for the quantitative description of wealth distributions and, more generally, to identify mechanisms leading to variations in societies [32-36]. In a broader sense, diversity indices may be helpful for the design of robust energy distribution systems [37] or even to assemble well-performing teams [23]. Thus we see that, despite the ambiguity in the definition of diversity, the concept is very relevant to many different disciplines and applications.

Figure 1.

Examples of complex, multicomponent populations in which diversity may be a meaningful quantitative concept. (a) Diversity in island ecology. A large number of species may migrate onto an island. Organisms can proliferate and die, leading to a specific time-dependent pattern of species diversity on the island. (b) Microbes are ingested and form a community in the gut by proliferating, competing, and dying. They can also be cleared from the gut. (c) Naive T cell generation in vertebrates. Naive T cells develop in the thymus. Each T cell expresses only one type of T cell receptor (TCR). Naive T cells can proliferate and die in the peripheral blood. The possible number of T cell receptors that can be expressed is enormous > 10¹⁵, but only perhaps 10⁶−10⁸ different TCRs usually exist in an organism. The diversity of the T cell receptor repertoire is an important determinant of the organism ’s response to antigens. (a) Island biodiversity. (b) Gut microbiota. (c) T cell production.

In this topical review, we start by summarizing the basic concepts from information theory which are necessary for a quantitative treatment of diversity. We continue with describing aspects of populations and diversity that are common to many applications in biology. In the next section, we present the common mathematical descriptions of diversity in terms of both number and species counts. Moreover, in most applications, only a small sample of a population is available. Thus, we place particular emphasis on the effects of sampling on diversity measures in section 5. In section 6 and subsections within, we survey a number of biological systems in which concepts of diversity play a key role in understanding the dynamics of the population. These include ecological populations, stem cell barcoding experiments, immunology, cancer, and societal wealth distributions. Each of these systems carry their unique attributes and thus require specific diversity measures. Finally, in section 7 we summarize the advantages and disadvantages of some common diversity measures and conclude with a discussion of possible future applications of concepts of diversity.

2. Mathematical concepts

2.1. Entropy, relative entropy, KL divergence, KS statistic, mutual information and all that

We first provide a summary of the fundamental mathematical structures that arise in the analysis of populations in which one naturally seeks to quantitatively compare distributions or frequencies of subpopulations. These mathematical notions invariably involve ideas from information theory such as entropy and mutual information which have a rich history and deep connections to thermodynamics, coding theory, cryptography, inference, and communication [38]. To review the necessary information-theoretic concepts, we consider a discrete random variable X which takes on values from the set {x₁, x₂, …, x_N} with probability P_k = Pr (X = x_k) such that

$$\sum _{k=1}^N{P}_k=1,$$ (1)

where the sum is taken over all possible values x_k. This probability mass function may represent the relative frequency that the attribute X takes on the value x_k in a large population. In the case of species diversity, we may interpret P_k as the proportion or frequency of species with X = x_k in a certain population. The entropy, or ‘Shannon entropy’ is defined by

$$H(X)=-\sum _{k=1}^N{P}_k\log P_k$$ (2)

and can be thought of as the expected uncertainty or surprise $-\mathbb{E}[\log P(X)]$.

The continuous limit of Shannon entropy, or differential Shannon entropy has also been defined, but care must be taken if X carries physical dimensions. If the probability of X taking on values in the interval [x, x + dx] is denoted by P(x) dx, the differential Shannon entropy is

$$H(X)=-\int P(x)\log P(x)\mathrm{d}x.$$ (3)

These expressions are synonymous with the ‘Shannon index’ of species diversity with some freedom in the choice of the base of the logarithm. Without any constraints on the distributions other than being compactly supported, the form of P_k or P(x) that maximizes H(X) is a uniform distribution. With additional constraints there are classes of distributions that maximize the Shannon index. For example, for a fixed mean and variance on an unbounded domain, the Shannon index- or entropy-maximizing distribution is Gaussian. Within Gaussian distributions, the Shannon index increases logarithmically with the variance. In fact, within a specific class of distributions, the Shannon index is larger for flatter distributions [39, 40]. As such, the Shannon index has been used as a measure of diversity [41].

The problem with the differential entropy of equation (3) is that P(x) carries dimensions X⁻¹, because the cumulative distribution function $P(X⩽x)={\int }_{-\infty }^{x}P(x^{\prime })\mathrm{d}{x}^{\prime }$ has to be dimensionless. Therefore, the argument of the logarithm in equation (3) is not dimensionless as required. To avoid such an issue, one can define a point-density function P₀(x) according to [39]

$$\underset{N\to \infty }{\text{lim}}\frac{\#\text{points}\in [a,b]}{N}\equiv {\int }_a^b{P}_0(x)\mathrm{d}x.$$ (4)

Given that the limit is well-behaved, we can express the difference between two adjacent points x_k+1 and x_k in terms of

$$\underset{N\to \infty }{\text{lim}}[N(x_{k+1}-x_k)]=P_0^{-1}(x_k).$$ (5)

We now consider the continuum limit of the discrete Shannon entropy as defined in equation (2), and set

$$P_k=P(x_k)(x_{k+1}-x_k)=P(x_k)[N{P}_0(x_k){]}^{-1}.$$ (6)

In this way, it is possible to derive a continuous Shannon entropy

$$\begin{gathered} \underset{N\to \infty }{\text{lim}}{H}_N(X)=-\int P(x)\log \left(\frac{P(x)}{N{P}_0(x)}\right)\mathrm{d}x-\log (N) \\ =-\int P(x)\log \left(\frac{P(x)}{P_0(x)}\right)\mathrm{d}x \end{gathered}$$ (7)

that is invariant under parameter changes and whose logarithm depends on the dimensionless quantity P(x)/P₀(x). We subtracted log(N) in equation (7) to obtain a finite H_N(X).

To characterize the diversity between two communities, we consider two discrete random variables X and Y with the corresponding joint probability mass function P_X,Y(x_k, y_ℓ) = Pr(X = x_k, Y = y_ℓ). Given the joint distribution P_X,Y(x_k, y_ℓ), we can compute the marginal distributions $P_X(x_k)={\sum }_{\ell }{P}_{X,Y}(x_k,y_{\ell })$ and $P_Y(y_{\ell })={\sum }_k{P}_{X,Y}(x_k,y_{\ell })$ by summing over the complementary variable. These definitions enable us to define the joint entropy

$$H(X,Y)=-\sum _{k,\ell }{P}_{X,Y}(x_k,y_{\ell })\log P_{X,Y}(x_k,y_{\ell }),$$ (8)

which may be also written as $-\mathbb{E}[\log P_{X,Y}]$. Moreover, the conditional entropy

$$H(X\mid Y)=-\sum _{k,\ell }{P}_{X,Y}(x_k,y_{\ell })\log \left(\frac{P_{X,Y}(x_k,y_{\ell })}{P_X(x_k)}\right)$$ (9)

$$=-\sum _{k,\ell }{P}_{X,Y}(x_k,y_{\ell })\log P_{X\mid Y}(x_k\mid y_{\ell })$$ (10)

describes the expected uncertainty in the random variable Y given X. It can be also expressed as $-\mathbb{E}[\log P_{X\mid Y}]$ where P_X∣Y is the conditional probability mass function. For independent random variables X and Y, we find that H(Y∣X) = H(Y) and H(X∣Y) = H(X).

While the Shannon index is a measure of the absolute entropy of a distribution, the relative entropy or Kullback–Leibler (KL) divergence

$$D_{\text{KL}}(P\Vert Q)=\sum _{k}P(x_k)\log \left(\frac{P(x_k)}{Q(x_k)}\right)={\mathbb{E}}_p[\log P(x_k)-\log Q(x_k)],$$ (11)

quantifies the distance between two probability mass functions P and Q. In the case of continuous distributions P(x) and Q(x), we obtain D_KL(P∥Q) = ∫ P(x) log(P(x)/Q(x)) dx.

The KL divergence is the relative entropy of P with respect to the reference distribution Q. Note that the limiting Shannon entropy is simply the KL divergence between the distribution P(x) and the associated invariant measure P₀(x). Usually, P is an experimental or observed distribution and Q is a model that represents P. Furthermore, the KL divergence is nonnegative and equals zero if and only if P = Q [38]. It is not symmetric, D_KL(P∥Q) ≠ D_KL(Q∥P), and is thus not a metric. In addition, a special case of the KL divergence is the ‘mutual information’

$$\begin{gathered} I(X;Y)=D_{\text{KL}}(P_{X,Y}\Vert P_X{P}_Y) \\ =\sum _{k,\ell }{P}_{X,Y}(x_k,y_{\ell })\log \left(\frac{P_{X,Y}(x_k,y_{\ell })}{P_X(x_k)P_Y(y_{\ell })}\right). \end{gathered}$$ (12)

Note that I(X; Y) = I(Y; X) is symmetric and quantifies how much knowing one variable reduces the uncertainty in the other. If X and Y are completely independent, I(X, Y) = 0. According to equation (12) and the definitions of joint and conditional entropy in equations (10) and (8), the mutual information can be written in terms of marginal, conditional, and joint entropies [38]:

$$\begin{gathered} I(X;Y)=H(X)-H(X\mid Y)=H(Y)-H(Y\mid X) \\ =H(X)+H(Y)-H(X,Y). \end{gathered}$$ (13)

A symmetric version of the KL divergence is provided by the Jensen–Shannon divergence [42]

$$\text{JSD}(P\Vert Q)=\frac{1}{2}{D}_{\text{KL}}(P\Vert M)+\frac{1}{2}{D}_{\text{KL}}(Q\Vert M),$$ (14)

where M = (P + Q)/2 defines the mean distribution of P and Q. These divergences can be extended to include multiple and higher-dimensional distributions. The square-root of the Jensen–Shannon divergence is a distance metric between two distributions.

Another useful distance metric is the Kolmogorov–Smirnov (KS) distance, which is defined as

$$D_{\text{KS}}=\underset{x}{\max }\mid G(x)-F(x)\mid ,$$ (15)

where F(x) is a cumulative reference distribution and G(x) is an empirical distribution function. The distribution G(x) is based on different samples with cumulative distribution function that can be F(x) or another distribution to be tested against F(x). The KS metric is the maximum distance between the two cumulative distributions F(x) and G(x). We outline in section 6.6 that the KS metric is related to the Hoover index which is used to quantify diversity, or inequity, in wealth or income distributions relative to a uniform distribution.

3. Commonly used measures of diversity

The notions of entropy and information are naturally related to the spread of a distribution P(x), and can be subsumed into a general metric for quantifying diversity. Usually, a population is measured and can be thought of as one realization of an underlying distribution. Consider a realization n = {n₁, n₂, … , n_R} describing the number n_i of entities of a discrete and distinguishable group/species/type (1 ⩽ i ⩽ R). The total population is $N={\sum }_{i=1}^R{n}_i$. This given realization constitutes a ‘distribution’ across all possible types. Thus, any realization is completely described by a set of R numbers. Diversity measures are reduced representations of the distribution. An example would be a single parameter which captures the spread of the distribution of realizations {n_i}. This is not different than, for example, defining a Gaussian distribution by its mean and standard deviation. Realizations {n_i}, however, usually are not described by specific functions that can be defined by one or two parameters such as Gaussians. However, many different diversity indices can be unified into a single formula called ‘Hill numbers’ of order q [43-45]:

$${}^{q}D={\left(\sum _{i=1}^R{f}_i^q\right)}^{1∕(1-q)},$$ (16)

where f_i ≡ n_i/N is the relative abundance of types i. This general formula represents different classes of ‘diversity indices’ for different values of q. It is also useful because one can consistently define an effective proportional abundance

$$f_{\text{eff}}≔1{∕}^{q}D={\left(\sum _{i=1}^R{f}_i^q\right)}^{1∕(q-1)}$$ (17)

that corresponds to an average abundance with increasing weighting towards the larger-population species as q increases [45, 46].

Note the similarity of this definition to the standard mathematical p-norm

$$\Vert \mathbf{f}{\Vert }_p≔{\left(\sum _{i=1}^R{f}_i^p\right)}^{1∕p},$$ (18)

except that the exponent is 1/p instead of 1/(1 − q). Another diversity measure is provided by the Renyi index [47]

$${}^{q}H=\log {}^{q}D=\frac{1}{1-q}\log \left(\sum _{i=1}^R{f}_i^q\right),$$ (19)

which is a generalization of the Shannon entropy defined in equation (2). The order q describes the sensitivity ^qD and ^qH to common and rare types [48]. Below, we provide an overview of the most commonly used indices which result from the generalized diversity ^qD for different values of q:

3.1. Richness

In the limit of q → 0⁺, the probabilities $f_i^q$ are equal to unity and ⁰D is simply the total number of types in the population, or the ‘richness’ R. The richness is often used in quantifying the diversity of T cells and species counts in ecology [3] and represents a metric that weights the smallest subpopulations the most.

3.2. Shannon index

For q = 1 − ε in the limit ϵ → 0⁺, the generalized diversity as defined by equation (16) becomes

$$\begin{gathered} {}^{1}D=\underset{\epsilon \to 0^{+}}{\text{lim}}{\left(\sum _{i=1}^R{f}_i^{1-\epsilon }\right)}^{1∕\epsilon }=\underset{\epsilon \to 0^{+}}{\text{lim}}{\left(\sum _{i=1}^R{f}_i{e}^{-\epsilon \text{ln}{f}_i}\right)}^{1∕\epsilon } \\ =\underset{\epsilon \to 0^{+}}{\text{lim}}{\left(\sum _{i=1}^R{f}_i(1-\epsilon \text{ln}{f}_i+O({\epsilon }^2))\right)}^{1∕\epsilon } \\ =\underset{\epsilon \to 0^{+}}{\text{lim}}{\left(1-\epsilon \sum _{i=1}^R{f}_i\text{ln}{f}_i\right)}^{1∕\epsilon } \\ =\text{exp}\left[-\sum _{i=1}^R{f}_i\text{ln}{f}_i\right], \end{gathered}$$ (20)

which is the exponential of the Shannon index

$$\text{Sh}≔\text{ln}\left(\underset{q\to 1^{-}}{\text{lim}}{}^{q}D\right)=-\sum _{i=1}^R{f}_i\log f_i$$ (21)

that parallels the Shannon entropy defined in equations (2) and (10). This index is also sometimes called the Shannon–Wiener index (H) and can be defined using any logarithmic base. Usually measured values are Sh ~ $\mathcal{O}$(1). Qualitatively, e^Sh can be thought of as a rule of thumb for the number of effective species in a population.

3.3. Evenness

Evenness is another class of diversity indices often invoked in ecological and sociological studies. One definition (‘Shannon’s equitability’) is based on simply normalizing the Shannon diversity by the maximum Shannon diversity that arises if every species is equally likely [49]:

$$J_{\mathrm{E}}≔\frac{\text{Sh}}{{\text{Sh}}_{\max }}=\frac{\text{Sh}}{\text{ln}R}.$$ (22)

3.4. Simpson’s index with replacement

When q = 2, we find

$${}^{2}D=1∕\left(\sum _{i=1}^R{f}_i^2\right).$$ (23)

Simpson’s diversity index is defined as

$$S_{\mathrm{r}}=1{∕}^{2}D=\sum _{i=1}^R{f}_i^2=\sum _{i=1}^R{\left(\frac{n_i}{N}\right)}^2,$$ (24)

which carries the interpretation that upon drawing an entity from a given population the same type is selected twice.

3.5. Simpson’s index without replacement.

A related index that cannot be directly constructed from ^qD is Simpson’s index without replacement:

$$S=\sum _{i=1}^R\frac{n_i(n_i-1)}{N(N-1)}.$$ (25)

Here, when an entity is drawn, it is not replaced before the second entity is drawn. The differences between S_r and S are significant only for systems with small numbers of entities n_i for all types i.

3.6. Berger–Parker diversity index

In the q → ∞ limit, we find

$$\begin{gathered} {}^{\infty }D=\underset{q\to \infty }{\text{lim}}{\left(\sum _{i=1}^R{f}_i^q\right)}^{1∕(1-q)} \\ =\underset{q\to \infty }{\text{lim}}{f}_{\max }^{-\frac{1}{1-1∕q}}{\left[\sum _{i=1}^R{\left(\frac{f_i}{f_{\max }}\right)}^q\right]}^{1∕(1-q)} \\ =f_{\max }^{-1} \end{gathered}$$ (26)

where f_max = max_{i∈{1,…,R}}(f_i). The Berger–Parker diversity index

$$1{∕}^{\infty }D≔f_{\max }$$ (27)

is defined as the maximum abundance in the set {f_i}, i.e. the abundance of the most common species. It is equivalent to the optimal solution of an ∞-norm of f= n/N.

4. Clone count representation

An alternative way of quantifying a population is through the species abundance distribution or ‘clone counts’ defined by

$$c_k≔\sum _{i=1}^{R}1(n_i,k)\in {\mathbb{Z}}^{+},$$ (28)

where the discrete indicator function $1(n,k)=1$ if n = k and zero otherwise. The sum is usually taken over all species for which n_i ⩾ 1. Clone counts can also be defined over only a certain special subset of species. Clone counts, or species abundance distributions, in the language of computational mathematics, can be thought of as the measure of the level-sets [50] of the discrete function n_i, or, in the language of condensed matter physics, the density of states if n_i are thought of as energies of states i [51]. The clone counts also satisfy

$$N=\sum _{k=1}^{\infty }k{c}_k\text{and}R=\sum _{k=1}^{\infty }{c}_k,$$ (29)

where N and R are the discrete total population and the total number of species (richness) present.

Clone counts are commonly used in the theory of nucleation and self-assembly [52-54], where all particles are identical and c_k represents the number of clusters of size k. They are equivalent to ‘species abundance distributions’ or sometimes ambiguously described as ‘clone size distributions.’ Clone counts have been recently used to quantify populations in barcoding studies [55] described below.

Clone counts do not depend on the specific labeling of the different types i and do not contain any identity information. However, since the common diversity indices are only a summary of the vector {n_i} and also do not retain species identity information, ^qD can be written in terms of c_k rather than n_i:

$${}^{q}D={\left[\sum _{k=1}^{\infty }{c}_k{\left(\frac{k}{N}\right)}^q\right]}^{1∕(1-q)},$$ (30)

which leads to corresponding expressions at specific values of q, e.g. ⁰D = R,

$$\begin{gathered} {}^{1}D=\text{exp}\left[-\sum _{k=1}^{\infty }{c}_k\left(\frac{k}{N}\right)\text{ln}\left(\frac{k}{N}\right)\right]\text{and} \\ 1{∕}^{2}D=\sum _{i=1}^{\infty }{c}_k{\left(\frac{k}{N}\right)}^2. \end{gathered}$$ (31)

While the definitions of ^qD are well-defined when the discrete species are delineated, for more granular or continuous traits, the delineation of different species will affect the values of n_i and c_k. Figure 2 shows population counts ordered by a continuous trait x. By defining the discrete species i according to different binning windows over x, we find different sets of number and clone counts. Thus, measures of diversity can be highly dependent on the resolution and definition of traits and species.

Figure 2.

Number counts and clone counts vary depending on the definition and thresholding of discrete species. This consideration arises in designing experimental measurements.

5. Sampling

In most applications, including all the ones we will discuss below, the entire population is not accessible for identification and measurement. In an ecology, all animals of the population cannot be tracked. In blood samples, only a small fraction of the cell types in the whole organism is drawn for identification/sequencing. Thus, inferring the diversity in the entire system from the diversity in the sample is a key problem encountered across many fields.

There are numerous ways to randomly sample a population. One approach is to draw one individual, record its attributes, return it back into the system, and allow it to well-mix or equilibrate before again randomly drawing the next individual. This process can be repeated M times. To indicate this type of sampling, we use the subscript 1 × M in the corresponding distributions and expectation values. Similar sampling approaches are used in the ‘mark-release-recapture’ experiments to estimate population size [56], survival, and dispersal of mosquitos [57]. For a given configuration {n_i} and total population size N [58], the probability that the configuration {m_i} is drawn after M samples is simply

$$P_{1\times \mathrm{M}}(\mathbf{m}\mid \mathbf{n},M,N)=\left(\begin{array}{c}\hfill M\hfill \\ \hfill m_1,m_2,\dots ,m_R\hfill \end{array}\right)\prod _{j=1}^R{f}_j^{m_j},$$ (32)

where f_j ≡ n_j/_N is the relative population of species i, $N\equiv {\sum }_{i=1}^R$ n_i is the total population and $M\equiv {\sum }_{i=1}^R$ m_i is the total number of samples.

We can now use P_1×M to compute the statistics of how the system diversity is reflected in the diversity in the samples. For example, the mean population in the sample in terms of n_i is ${\mathbb{E}}_{1\times M}[m_i]\equiv {\sum }_{\mathbf{m}}\mathbf{m}{P}_{1\times \mathrm{M}}(\mathbf{m}\mid \mathbf{n},M,N)$. The lowest moments of the populations in the sample are

$$\begin{gathered} {\mathbb{E}}_{1\times M}[m_i]=M{f}_i=n_i\frac{M}{N}, \\ {\mathbb{E}}_{1\times M}[m_i{m}_j]=f_i{f}_{j}M(M-1)+f_{i}M1(i,j). \end{gathered}$$ (33)

An alternative random sampling protocol is to draw a fraction α ≡ M/N < 1 of the entire population once. This type of sampling arises in biopsies such as laboratory blood tests. To be able to distinguish between this sampling protocol and the previous one, we now use the notation M × 1. In this case the combinatorial probability of a specific sample configuration, given n, N, and M is

$$P_{\mathrm{M}\times 1}(\mathbf{m}\mid \mathbf{n},M,N)=\prod _{j=1}^R\frac{\left(\begin{array}{c}\hfill n_j\hfill \\ \hfill m_j\hfill \end{array}\right)}{\left(\begin{array}{c}\hfill N\hfill \\ \hfill M\hfill \end{array}\right)}1\left(M,\sum _{i=1}^R{m}_i\right),$$ (34)

where the discrete indicator function enforces the constraint between m_i and the sampled population M. In this single-draw sampling scenario, we use the Fourier decomposition $1(x,y)\equiv {\int }_0^{2\pi }\frac{\mathrm{d}q}{2\pi }{e}^{iq(x-y)}$ to find

$${\mathbb{E}}_{M\times 1}[m_i]=n_i\frac{M}{N}=n_i\sigma ,$$ (35)

$$\begin{gathered} {\mathbb{E}}_{M\times 1}[m_i{m}_j]=n_i{n}_j\frac{M}{N}\frac{M-1}{N-1} \\ +1(i,j)n_i\frac{M}{N}\left(\frac{N-M}{N-1}\right). \end{gathered}$$ (36)

Results using p_1×M and P_M×1 rely on perfectly random sampling, where certain clones/species are not more likely sampled or captured than others. The moments $\mathbb{E}[m_i{m}_j]$ can be directly used to evaluate the expected Simpson’s diversities, S_r (with replacement) and S (without replacement) defined by equations (24) and (25), in the corresponding sample. In the case of 1×M sampling, we find

$$\begin{gathered} {\mathbb{E}}_{1\times \mathrm{M}}[S_{\mathrm{r}}]={\mathbb{E}}_{1\times \mathrm{M}}\left[\sum _i{\left(\frac{m_i}{M}\right)}^2\right] \\ =\frac{M(M-1)}{M^2}\sum _i{f}_i^2+\frac{1}{M}\sum _i{f}_i \\ =S_{\mathrm{r}}\left(1-\frac{1}{M}\right)+\frac{1}{M}, \end{gathered}$$ (37)

and

$$\begin{gathered} {\mathbb{E}}_{1\times \mathrm{M}}[S]={\mathbb{E}}_{1\times \mathrm{M}}\left[\sum _i\frac{m_i}{M}\frac{m_i-1}{M-1}\right] \\ =\sum _i\frac{{\mathbb{E}}_{1\times \mathrm{M}}[m_i^2]}{M(M-1)}-\sum _i\frac{{\mathbb{E}}_{1\times \mathrm{M}}[m_i]}{M(M-1)} \\ =\sum _i{f}_i^2\equiv S \end{gathered}$$ (38)

while for M×1 sampling, we find

$$\begin{gathered} {\mathbb{E}}_{\mathrm{M}\times 1}[S_{\mathrm{r}}]={\mathbb{E}}_{\mathrm{M}\times 1}\left[\sum _i{\left(\frac{m_i}{M}\right)}^2\right] \\ =S_{\mathrm{r}}\frac{M-1}{M-\sigma }+\frac{1-\sigma }{M-\sigma } \end{gathered}$$ (39)

and

$$\begin{gathered} {\mathbb{E}}_{\mathrm{M}\times 1}[S]={\mathbb{E}}_{\mathrm{M}\times 1}\left[\sum _i\frac{m_i}{M}\frac{m_i-1}{M-1}\right] \\ =\left[\sum _i\frac{{\mathbb{E}}_{\mathrm{M}\times 1}[m_i^2]}{M(M-1)}-\sum _i\frac{{\mathbb{E}}_{\mathrm{M}\times 1}[m_i]}{M(M-1)}\right] \\ =S. \end{gathered}$$ (40)

Note that for both types of random sampling, we find that the expected Simpson’s diversity (without replacement) in the samples are equal to the Simpson’s diversity in the full system. In general, the expectations do not commute and $\mathbb{E}[S]\ne S(\mathbb{E}[m_i])$.

Effects of sampling on clone counts c_k can be similarly calculated by averaging the definition for the sampled clone count

$$b_k≔\sum _{i=1}1(m_i,k)\in Z^{+}$$ (41)

over the sampling probabilities P_M×1(m∣n, M, N) or P_1×M(m∣n, M, N). For clone counts, the calculations of moments of sampled quantities b_k are more involved, and explicitly noncommutative $\mathbb{E}[b_k]\ne {\sum }_{i}1(\mathbb{E}[m_i],k)$. One advantage of working in the b_k representation is that diversity indices such as the expected sampled richness are difficult to extract from $\mathbb{E}[m_i]$ but is simply $\mathbb{E}[R^{\mathrm{s}}]={\sum }_k\mathbb{E}[b_k]$. Some related results are given in [59, 60].

The above results provide expected diversities in the sample assuming full knowledge of {n_i} in the system. They represent solutions to the forward problem, the so-called mean ‘rarefaction’ in ecology. However, the problem of interest is usually the inverse problem, or extrapolation in ecology. In the simplest case, we wish to infer the expected diversity (or {n_i} and c_k) in the system from a given configuration {m_i} or clone count b_k. Extrapolation is a much harder problem and is the subject of many research papers [61-65].

One may wish to use the observed sample diversity ^qD(M) to approximate the population diversity ^qD(N). For any q, the underestimation of ^qD(N) using ^qD(M) decreases as the sample size M increases. The deviation of ^qD(M) from ^qD(N) is smaller for larger q, as higher-order Hill numbers are more heavily weighted by large species, which are less sensitive to subsampling.

Chao and others have shown that for q ⩾ 1 and in the N → ∞ limit nearly unbiased approximations can be obtained and when q ⩾ 2, these unbiased estimates are very insensitive to sample size M [59, 60]. Using clone counts in a sample of population M, Chao et al [66] obtained for q = 1 (in terms of Shannon’s index):

$$\begin{gathered} \mathrm{S}\widehat{\mathrm{h}}=\sum _{k=1}^{M-1}\frac{1}{k}\sum _{1⩽m_i⩽M-k}\frac{m_i}{M}\frac{\left(\begin{array}{c}\hfill M-m_i\hfill \\ \hfill k\hfill \end{array}\right)}{\left(\begin{array}{c}\hfill M-1\hfill \\ \hfill k\hfill \end{array}\right)} \\ -\frac{d_1}{M(1-A{)}^{M-1}}\left\{\log A+\sum _{r=1}^{M-1}\frac{1}{r}(1-A{)}^r\right\}, \end{gathered}$$ (42)

where A = 2d₂/[(M − 1)d₁ + 2d₂].

For q ⩾ 2, Gotelli and Chao [59] obtained

$${}^q\widehat{D}={\left[\sum _{m_i⩾q}\frac{m_i^{(q)}}{M^{(q)}}\right]}^{1∕(1-q)}$$ (43)

where x^(j) = x(x − 1)…(x − j + 1). For example, ${}^2\widehat{D}=M(M-1)∕{\sum }_{m_i⩾2}{m}_i(m_i-1)$, the inverse of Simpson’s index without replacement (equations (23) and (25)).

The ill-conditioning of the inverse problems is particularly severe for the richness ⁰D. The general formula for an estimate of the system richness is

$${}^0\widehat{D}=R(M)+{\widehat{d}}_0,$$ (44)

and reduces to the unseen species problem for determining d₀ [67, 68]. Since the sample size M and the richness R in the system are uncorrelated, rigorously, one must use information contained in the species fractions f_i or the clone counts c_k in the full system [69, 70]. However, a popular estimate for the system richness R(N) is the ‘Chao1’ estimator [59, 71]

$$\text{Chao1}:\widehat{R}(N)=R(M)+\frac{d_1^2}{2d_2},$$ (45)

which is actually a lower bound and gives reliable estimates for systems of size only up to approximately double or triple the sample size M. The uncertainty of the Chao1 estimator has also been derived via a variance that is also a function of d₁ and d₂ [72]. The ‘Chao2’ estimator gives the system richness as a function of measured incidence [59]

$$\text{Chao2}:\widehat{R}(N)=R(M)+\frac{q_1^2}{2q_2},$$ (46)

where q₁, q₂ are the number of species found in 1 or 2 samples out of many (as in the 1 × M sampling method). Shen et al [73] derived another estimate

$$\widehat{R}(N)=R(M)+d_0\left[1-{\left(1-\frac{d_1}{M{d}_0+d_1}\right)}^{N-M}\right],$$ (47)

which is only reliable if the sample size M is more than half of the system size N. Many of these estimators have been coded into analysis software such as R and iNEXT [74].

Regardless of the estimator, the major limitation is an insufficient sample size M ⪡ N. Models predicting species abundances as a function of system size can help bridge this gap. For example a log-normal relationship for the clone count c_k [75] has been used to find agreeable results [76, 77]. In general, models can be extremely useful analyzing the effects of sampling, particularly when a Bayesian prior is desired.

We have outlined the basic mathematical frameworks for quantifying diversity that have utility across applications in different disciplines. The above summary of sampling assumes a well-mixed population, precluding any spatial dependence of the distribution of individual species. Spatially dependent sampling has been proposed for the origin of relationships between the number of species detected and the total area occupied by the population (see below).

6. Fields in which diversity play a key role

Below, we summarize a few modern applications in which diversity is important. By no means exhaustive, the following are simply examples of specific systems in modern biology that reflect the authors’ intellectual biases.

6.1. Ecology, paradox of the plankton

The classic problem in the context of biological diversity is dubbed the paradox of the plankton and was originally discussed in a paper of the same title [78]. It describes diverse populations of plankton in environments of limited number of resources or nutrients. Sampled populations of plankton exhibit a large number of species even in low nutrient conditions during which one expects strong competition for resources. This observation runs counter to the competitive exclusion principle arising in many settings [79].

Perhaps the most common application of diversity arises in biological population studies, specifically in ecology [6-11]. Possible areas of application include the monitoring of ecosystems and the development of efficient species conservation strategies [2, 5, 9, 10, 29-31]. Multiple overlapping and nebulous definitions of ecological diversity have been advanced [3, 4, 25-29]. Early work by Fisher [6] introduced a logarithmic series model to mathematically describe empirical species diversity data. Here, the diversity index referred to a free parameter in the corresponding model. In a later study, MacArthur defined species diversity based on the size of the sampled area [80]. In the ecological setting, multiple layers of subpopulations are an important feature of populations. These subpopulations may be delineated by another property of the individual species, such as size, weight, behavioral attributes, etc. Subpopulations can also be distinguished through their spatial distribution or occupation of different habitats. Whittaker [81, 82] qualitatively defined four types of diversity (point, alpha, beta, and gamma) conditioned on habitat or spatial distribution of the subpopulations [82]. Fundamentally, these differences arise from different methods of sampling, leading to different Hill numbers ^qD. We summarize a few often-used descriptions below:

• ‘Point diversity’ refers to samples taken at a single point or ‘microhabitat.’ This quantity is usually operationally measured by trapping organisms at one or more specific points.

• ‘Alpha diversity’ is defined as the diversity within an individual location or specific area. In general, one can define a Hill number derived from measurements at a specific location as ^qD_α, while the index α ≡ ⁰D_α is the richness encountered within a defined area or specific location. A few subtle variations in the definition of the index α exist, mostly related to the sampling process [45, 46]. For example, in relation to beta diversity (discussed below), alpha diversity is the mean of the specific-location diversities across all locations within a larger landscape.

• ‘Gamma diversity’ is the diversity index ^qD_γ determined from the entire dataset, the total landscape, or entire ecosystem. The index γ ≡ ^qD_γ usually denotes the total number of different species or clones at the largest scale. Note that the mean or sum of the alpha diversities is in most cases not equal to the gamma diversity. The nonlinearity of the Hill numbers as well as the intersection or exclusion of species amongst the different sites suggests a need for indices that connect alpha and gamma diversities.

• ‘Beta diversity’ was devised to describe the difference in diversity between two habitats or between two different levels of ecosystems. While the different levels of diversity are designed to the spatial aspects of diversity, different habitats overlap, leading to some amount of arbitrariness in determining, or sampling of the β-diversity. Moreover, beta diversity was initially described in different ways [45, 81, 82], leading to confusion about its mathematical definition and use [45, 46, 48]. One possible definition is Whittaker’s [81] multiplicative law ^qD_γ ≡ ^qD_α ^qD_β where here, α is defined as the mean of the diversities across all micro-habitats. Whittaker’s definition describes beta diversity ^qD_β = ^qD_γ/^qD_α as a measure to quantify the diversity in the total population relative to the mean diversity across all micro-habitats [45]. In the limit of q → 1⁻, we obtain the Shannon diversity relationship Sh_γ = Sh_α + Sh_β according to equation (21). Another definition of β is given by Lande’s [83] additive law γ ≡ α + β according to which diversity indices are measured in the same units. One concept associated with β in terms of the additive partitioning is ‘species turnover’ quantifying the difference in richness between the entire and the local population. As an example, consider two distinguishable or spatially separate habitats A and B. If A contains species {a, b, c, d, e} and B contains {b, c, f, g}, we find β_A,B = 5 associated with the set {a, d, e, f, g}. The laws of Whittaker and Lande sparked debates about how to properly define beta diversity, and led to the distinction between multiplicative and additive diversity measures [45, 46, 48].

• ‘Delta, epsilon, omega diversity’ are other hierarchical definitions of diversities proposed by Whittaker [82]. Delta diversity is analogous to beta diversity but defined at the larger among-landscape scale, while epsilon diversity corresponds to gamma diversity, but at the regional scale that contains many landscapes. Omega diversity is measured at the biosphere scale, and thus characterizes the diversity of all ecosystems [84].

• ‘Zeta diversity’ was introduced by Hui and McGeoch [85], and is defined by a set of ζ indices that mathematically describe the species numbers between different partitions of a certain habitat. Specifically, ζ_i is the mean number of species shared by i partitions. In particular, ζ₁ is the mean richness across all sites. For example, between two samples A and B or sets of data, the average number of species is ζ₁ := (R_A + R_B)/2, while the intersection is ζ₂ := A ∩ B. Generalizations to multiple samples can be defined using a series of zeta diversity indices ζ_i.

• Many other indices have been defined for different applications. The Jaccard index [45, 81, 85, 86] is defined as J(A, B) = ∣A ∩ B∣/∣A ∪ B∣, and is a general measure for quantifying the similarity in richness between two sets of populations A and B. Margalef’s index [87] and Menhinick’s index [88] are relative richness measures given by R/ ln N and $R∕\sqrt{N}$, respectively. Other indices include the Bray–Curtis dissimilarity [89], the Berger–Parker diversity index [90] as defined in equation (27), Fager’s index [91], Keefe and Bergersen’s index [92], McIntosh’s index [93], and Patil and Taillie’s index [94].

A myriad of different definitions of diversity indices arise from specific cases of the Hill numbers and consideration of different spatial scales of ecosystems. There is potential to further unify these definitions in a more systematic way using mathematical norms and more general mathematical structures of spatial dispersal of particles.

6.2. Area-species law and Island biodiversity

A particularly consistent, albeit qualitative feature observed in ecology is the species-area relationship (SAR) which relates the measured number of species (richness) with the relevant area. These areas can represent distinct habitats, such as mountain tops, or islands. For the latter, much work has been done in the subfield of island biodiversity.

The SAR is usually expressed as a power-law relationship between the number of species (or richness) R and the habitat/island area:

$$R=c{A}^z,$$ (48)

where c is a constant prefactor and z is an exponent. On a log–log plot, log R = log c + z log A defines a line with slope z. An example of the area-species law for species counts of long-horned beetles in the Florida Keys is shown in figure 3, yielding a slope z = 0.29. An alternative species-area relationship is e^R = cA^z [95], which is a straight line on a semi-log plot.

Figure 3.

Plot of ln R versus ln A with area A measured in terms of km². Species counts of long-horned beetles in the Florida Keys are plotted against the island size [98]. The linear regression line yields a slope of z = 0.29. Usually, fits of the species-area exponent z yield a small number.

The classic book by MacArthur and Wilson [96] and many subsequent analyses have promoted and extensively analyzed the SAR idea. In MacArthur and Wilson’s neutral equilibrium theory, immigration to and death on an island are monotonically decreasing and increasing functions of the number of species already on the island, respectively. Usually, measured values of the exponent fall in the range z ~ 0.1−0.4. Field work has also found relationships between the parameters c and z and system-specific attributes such as the island distance to the mainland, habitat type, etc [96, 97]. Nonetheless, reasonable predictions based on equation (48) are ubiquitous across many ecological examples.

Mechanistic origins of the robustness of the SAR have been proposed [99-101]. Different models for species populations n_i or clone counts c_k were surveyed and the corresponding species-area laws were derived by He and Legendre [100]. Spatial clustering of species and the averaging of random measurements was shown to robustly generate a power-law species-area curve [100, 101], highlighting the fundamental importance of sampling.

6.3. Gut microbiome

Another ecological system that has recently received much attention is the human microbiome, especially in the gut. The gut bacterial ecosystem is important for health and can impact cardiovascular disease, diabetes, neuropsychiatric diseases, inflammatory bowel disease (IBD), digestive and metabolic function to the point that fecal transplantation (bacteriotherapy) has become an effective treatment for recurrent C. difficile colitis infections [103]. This type of infection often occurs after antibiotics disrupt the gut microbiome. Transplants have also shown to be effective in treating slow-transit constipation [104].

Recent efforts to collect and curate gut microbiome data have included NIH’s Human Microbiome Project (HMP) [105, 106] and the European Metagenomics of the Human Intestinal Tract (MetaHIT) [107-109], as well as the integration of the data in [110]. Each dataset contains sequence data from samples from different body regions of hundreds of individuals, both healthy and diseased.

Bacterial species are usually determined by sequencing of the 16S ribosomal RNA (rRNA), a component of prokaryotic ribosomes that contain hypervariable regions that are species-specific. However, closely related taxa can have very similar sequences, making separation imperfect [111]. Nonetheless, with numerous public databases [102, 112-114], estimates of species abundances in samples are readily available. In the gut, there are usually on the order of 10³ bacterial species, with Bacteroidetes and Firmicutes being the dominant phyla [115, 116]. Indeed, lower gut diversity is seen to be associated with conditions such as Crohn’s disease [115]. For example, the frequency distribution of bacterial species in healthy and irritable bowel syndrome patients are shown in figure 4. The quantification of diversity of human microbiome is an essential step in ongoing research and the diversity indices have been applied to microbiome data, including α-diversity and β-diversity across the microbiome from different anatomical regions and different patients. As with island biodiversity, the gut microbiome can be modeled as birth-death-immigration (BDI) process.

Figure 4.

Frequencies of approximately 200 species of bacteria distributed across about a dozen phyla. (a) Group 1 depicts the relative abundance distribution for healthy individuals while (b) Group 2 shows the pattern for irritable bowel syndrome (IBD) patients. The differences in abundance patterns are apparent and have been quantified using the Shannon index for each individual plotted in (c). From Park et al [102]. (a) Group 1. (b) Group 2.

6.4. Barcoding experiments

Besides taxonomy of gut bacteria, the accurate identification of animal and plant species from samples is an essential task in ecology. In the early 2000’s a DNA barcoding method was developed to read relatively short DNA regions specific to certain species [120, 121]. These barcodes are usually found in mitochondrial DNA and often derived from a region in the cytochrome oxidase gene [120]. By sequencing samples and comparing with a sequence database such as The Barcode of Life Data System [122, 123], one can infer the number of species present within a sample. Detecting specific species within samples using DNA barcoding and DNA libraries have been used in many applications including identification of birds [121], identification of flowering plants [124], detecting contaminants [125], and tracking plant composition in processed foodstuffs [126].

Recently, a number of barcoding or tagging protocols [127-129] have been developed to genetically label a large population of cells to study how they differentiate and proliferate, especially in the context of hematopoiesis [117, 118, 130, 131] and cancer progression [132-134].

A novel approach used to investigate hematopoiesis exploits in situ barcodes [130]. Mice were engineered with an enzyme (Sleeping Beauty Transposase) that randomly moves DNA sequences (transposons) to different parts of the genome. The transposase is designed to be controllable by doxycycline, an antibiotic that can be used to switch on or off gene regulation. When the transposase is briefly activated, transposons within cell are randomly rearranged within a brief period. Since the genome length ⪢ transposon length, the new locations of the transposons will be distinct across the founder cells. After switching off the transposase, proliferation of founder cells will impart, except for rare DNA replication events, the same genomic sequence to its daughter cells. These collections of cells constitute a multiclonal population that proliferates and differentiates.

Analysis of the clonal population within differentiated cell pools show that granulocytes derive from stem cells at particular time points during the life of the mouse [130]. Comparing clonal abundance structure within different cell lineages showed that clones originally predominant in the lymphoid lineages eventually arise in myeloid cells, indicating that multipotent progenitor cells continually produce cells of both lineages. These conclusions arise after statistical analysis of the clone (defined by their transposon sites) abundance distribution within different groups of cells.

In another recent series of studies on hematopoiesis, stem cells (HSCs) were extracted from rhesus macaques and infected with a lentiviral vector. The lentivirus integrates its genome randomly in the genome of the HSCs. Since the lentivirus genome is much shorter than that of mammalian cells, nearly every successful infection results in a new viral integration site (VIS) or clone. The infected stem cells are autologously transplanted into the animal and some of them resume differentiation into progenitor cells that transiently proliferate and further differentiate. Descendant cells carry the same genetic sequence, including the lentivirus integration locations, or the viral integration sites (VIS). Another approach is to use libraries of synthesized DNA/RNA as tags. Here, the different sequences, rather than their integration sites, serve as the distinguishing feature. This process avoids the need to determine VISs.

In all the above approaches, each successive generation of cells will acquire the same tag, VIS or specific DNA barcode sequence, as their parent and ultimately the founder HSC. Compared to the Sleeping Beauty Transposon protocol, the VIS or barcoding experiments require an additional viral transfection step. Nonetheless, these VIS and barcoding experiments are equally effective in dissecting the differentiation process and quantifying lineage bias with age. For example, the variation (in time) of the abundances of a clone across different lineages indicates the level of fate switching of a stem cell [117, 135].

These experiments also enabled observation of biological mechanisms on a finer scale compared to traditional studies, allowing inference of parameters that are difficult to measure directly such as the initial HSC differentiation rate and the proliferative potential (number of generations) accessible to progenitor cells [55, 136].

After sampling, PCR amplification, and sequencing (each process exhibiting their specific errors), the relative species populations and clone counts within defined cell types can be quantified. Figure 6(a) shows frequencies of barcode i as a function of sampling times t_j in rhesus macaque. The fraction of each clone is depicted by the vertical distance between two neighboring curves. Here, it is important to note that the ‘diversity’ is a measure of the distribution of clone ID (barcodes) instead of lineages (cell types). In figure 6(b), we plot three different and rescaled diversity indices associated with the data in (a). The sampled richness is initially low at month 3 when barcoded clones have not fully differentiated and emerged in the peripheral blood. The sampled richness then peaks at month 9 before stabilizing after month 29. Simpson’s diversity seems to continue to increase after month 29 which may indicate more unevenness and coarsening (fewer clones dominating the total population). Shannon’s index is shown to decrease slightly, suggesting a decrease in the effective number of barcodes.

Figure 6.

(a) The fractional populations of the largest clones (barcodes) detected in granulocyte blood samples from rhesus macaque. Relative populations are described by the distances between neighboring curves. (b) Diversity indices derived from the data in (a). The Simpson’s index and Shannon diversity are rescaled to fit on the same plot.

Sun et al [130] and Kim et al [117] also used simple clustering algorithms that identified similar clones according to their activity patterns across time. They identified distinct groups of clones that are featured by different time points of contribution to hematopoiesis. Koelle et al [135] calculated Shannon diversity to ensure comparability between animals, different cell types, and across time.

The employment of neutral barcodes to study blood cell populations is statistically insensitive to spatial partitioning (different tissues in the organism). Nonetheless, small sampling M ⪡ N makes inference difficult. Thus, mechanistic simplifications and mathematical models have been used to quantify clonal evolution. Assuming a multispecies birth-death-immigration process (figure 7) Dessalles et al [137] found explicit steady-state distribution functions for n_i (log series) and c_k (Poisson) for constant r and μ, as well as formulae for the expected Shannon’s and Simpson’s diversities. Goyal et al [55] derived a master equation for the evolution of $\mathbb{E}[c_k]$ and then extended the solution to expected clone counts in the progenitor cell and sampled cell pools. By comparing results to the expected clone count in the sample at steady-state, they were able to infer kinetic parameters of the differentiation process. Biasco et al [139] proposed two candidate stochastic models for n_i and used Bayesian Information Criterion (BIC) to assess the likelihood of each.

Figure 7.

A simple multispecies birth-death-immigration (BDI) process [55, 136-138]. A constant source (i.e. stem cells with slow dynamics) generated by 16 cells, each of a different clone, undergo asymmetric differentiation with rate α to produce differentiated cells that can undergo birth or death with rates r(N) and μ(N) that may depend on the total population in the differentiated pool. In this example, the differentiated population contains N = 30 cells, R = 9 different clones (barcodes), thus leaving c₀ = 7 unseen species.

6.5. Cells of the adaptive immune system

Another intra-organism system for which diversity is often quantified is the adaptive immune system in vertebrates. The simplest immune subsystem consists of lymphoid cells (e.g. B and T cells) and tissues. B and T cells originate from common lymphoid progenitors (CLPs) that differentiate from HSCs in the bone marrow. B cells develop from CLPs in multiple stages in the bone marrow and spleen while T cells are formed from CLPs in the thymus. During T cell development in the thymus, T cell receptors (TCRs) are generated by random recombination of the associated receptor gene. TCRs are heterodimeric proteins that usually consist of an alpha chain and a beta chain. After a specific genetic sequence–corresponding to a specific amino acid sequence–is chosen and then selected for, the naive T cell is exported from the thymus into peripheral tissue (such as circulating blood and lymph nodes) where they can further proliferate or interact with antigens presented on the surface of antigen-presenting cells (APCs) and become activated. Naive T cells (those that have not previously strongly interacted with an antigen) can be activated through association of the surface T cell receptors (TCRs) with antigens presented by major histocompatibility complex (MHC) molecules on the surface of APCs. Similarly, naive B cells are generated in the bone marrow. The B cell receptors (BCRs) are comprised of heavy and light chains and an antigen-binding region, which is generated by the same recombination processes as TCRs. B cells are subsequently activated within tissues by binding to an antigen via their B-cell receptors (BCRs).

The mechanism responsible for creating very diverse repertoires of both BCRs and TCRs is V(D)J recombination [140]. In developing B cells, this mechanism involves the random recombination of diversity (D) and joining (J) gene segments of the heavy chain (DJ recombination). In the following step, a variable (V) gene segment joins the previously formed DJ complex to create a VDJ segment. In light chains, D segments are missing and therefore only VJ segments are generated. During T cell development and TCR generation, gene segments of the alpha chain and beta chain, the VJ and VDJ segments, respectively, also undergo random recombination. In the case of the beta chain, one of two different D regions of thymocytes recombine with one of six different joining J regions first, followed by rearrangement of the variable V region connecting it to the now-combined DJ segment. Due to the missing D segments in alpha chains, only VJ recombination is taking place. The recombination and joining processes in B cells and T cells involve many different genetic deletions and insertions that result in many different BCR and TCR protein sequences and a very large theoretical total number of possible clones with R ≳ 10¹⁴–10¹⁵ [141, 142].

In the end, each T or B cell expresses only one TCR or BCR type (an ‘immunotype’ or ‘clonotype’). TCR sequences are preserved during proliferation, while BCR sequences can further evolve [143]. Since the space of antigens (the different amino acid sequences, or epitopes, presented by MHCs) is large, a large number of different TCR and BCR sequences should be present in the organism in order to mount an effective response to a wide range of infections. However, before T cell export from the thymus, a complex selection process occurs [144]. Positive selection eliminates T cells that interact too weakly with MHC molecules. Subsequently, negative selection eliminates those T cells and TCRs that bind too strongly to epitopes. Cells that escape negative selection may lead to autoimmune disease as they react to self-proteins. Thus, the total number of different distinct immunoclones realized in an organism (the richness) defines its T cell repertoire and is estimated to range from 10⁶−10⁸ [145], with the lower range describing mice and the higher range an estimate for humans. B cell richness in man is estimated to be 10⁸−10⁹ [146, 147]. These values are much lower than the theoretical repertoire size R ≳ 10¹⁴−10¹⁵. TCR and BCR diversity is an important factor in health. For example, TCR diversity it has been shown to influence the tumor microenvironment and survival in lymphoma [148].

Although specific TCR sequences i can be determined, and their populations n_i measured and estimated, the TCR identities vary significantly across individuals (private sequences) so clone counts are usually studied. Figure 8(a) shows T cell clone counts b_k sampled from mice [142] that exhibit a biphasic power-law behavior. Figure 8(b) shows preliminary clone counts for six individuals, three HIV-negative patients and three HIV-infected patients [151].

Figure 8.

Examples of recently published clone count data. (a) Clone counts derived from a small sample (10⁵ sequences) of T cells [142]. Note the broad distribution described by a biphasic power-law curve. Ignoring the largest clones, power-law fits for each regime yield slopes of – 1.13 and – 1.76. However, one should be cautious describing sampled TCR (and BCR) clone counts using power laws as they hold typically for far less than two decades. (b) Human TCR clone counts for three HIV-infected (red) and three uninfected (black) individuals show qualitative differences between the distributions (unpublished). Other data from mice and humans, under different conditions and in different cell types, have been recently published [149, 150].

Quantifying T cell diversity is confounded by a number of technical limitations. Usually, the complete T cell repertoire in an animal cannot be directly measured. Rather, as in most other applications, small samples of the entire population are usually drawn. In sampling from animals, the fraction of cells drawn and sequenced is perhaps only M/N ~ 10⁻⁵−10⁻². Thus, clones that have small populations may be missed in the sample. Besides sampling, sequencing requires PCR amplification of the sample, leading to PCR bias, especially in the larger-sized clones [150]. Finally, as in many other applications, there are multiple subclasses of the T cell population. Naive T cells that are activated by antigens develop into memory T cells that carry the same TCR and that can further proliferate. Thus, it is difficult to separate the clone counts of different subpopulations such as naive or memory T cells [150].

Many mathematical models for the development and maintenance of the immune systems have been developed [136, 137, 141, 144, 152, 153]. For the multiclonal naive T cell population, rudimentary insights can also be gleaned from a birth-death-immigration process, much as in the modeling of hematopoiesis. Here, the thymus mediates the immigration of a large number of clones, which undergo homeostatic proliferation and death in the periphery. Immigration rates can be different for different clones, depending on the likelihood of specific recombination patterns which may be inferred from probabilistic models of VDJ recombination [154, 155].

Proliferation in the periphery depends on interactions between self-peptides with T cell receptors and is thus clone-dependent. Recently, it has been shown that TCR-dependent thymic output and proliferation rates (a nonneutral BDI model) influence the measured clone count patterns [156]. These processes form and maintain a diverse T cell receptor repertoire, which is usually characterized by its richness. Unlike the barcode abundances in arising during hematopoiesis, the neutral BDI processes are not able to capture the shapes of the measured TCR clone counts.

It is also known that T cell residence times depend on interactions between tissues and T cell receptors. Thus, different clones of T cells are expected to be differentially spatially distributed in the body. Hence, diversity metrics should be defined within and between habitats, much like that in ecology.

Finally, it is known that T cell richness decreases with age [157-160]. Qualitatively, a loss of diversity has been predicted within the multispecies BDI process by assuming a decreasing thymic output rate with age. Even when the thymus is completely shut down, the diversity of the T cell repertoire slowly decreases as successive clones go extinct and the clone abundance distribution coarsens. In humans, since the overall T cell population is primarily maintained by proliferation rather than thymic immigration [161], the reduction in diversity is fortunately a slow process.

6.6. Societal applications of diversity: wealth distributions

Metrics associated with diversity have been naturally applied in human social contexts [19-21, 162], including physical, cultural, educational [24, 32], and economic settings. For example, the distribution of wealth is the chief metric in many economic and political studies. As with all applications, data collection, sampling, and delineating differences in attributes are main research challenges.

Wealth or income, unlike species, are essentially continuous and ordered quantities, and can be described by many indices designed by economists to measure different wealth attributes of a population. Distinct from cellular or ecological contexts, socio-economic diversity is also often discussed in terms of ‘inequality,’ ‘evenness,’ or ‘polarization.’ Diversity or ‘inequality’ indices in the socioeconomic setting usually invoke a number of additional assumptions

• Individual identities are irrelevant: this is analogous to barcoding studies of a singular cell type in which the barcode identity is not important.

• Size and total wealth invariance: the diversity is invariant to the total population size. Only proportions of the total population that are associated with a proportion of the total wealth are relevant.

• Dalton principle: any inequality index should increase if any amount of wealth is transferred from an entity to one with higher existing wealth.

Mathematically, one starts by ordering the wealth or income of a population of N entities w₁ ⩽ w₂ … ⩽ w_i ⩽ w_i+1, … ⩽ w_N. For large N, the rescaled wealth distribution w(f) ≡ W_fN is a function of the relative fraction of the total population f = n/N ∈ [0, 1]. Furthermore, we can define a normalized wealth distribution or density

$$\tilde{w}(f)=\frac{w(f)}{W_{\mathrm{T}}},W_T=\sum _{i=1}^N{w}_i\approx {\int }_0^{1}w(f^{\prime })\mathrm{d}{f}^{\prime },$$ (49)

and the corresponding cumulative distribution

$$W_i=\frac{1}{W_T}\sum _{j=1}^i{w}_j$$ (50)

or

$$W(f)={\int }_0^f\tilde{w}(f^{\prime })\mathrm{d}{f}^{\prime }\equiv \frac{1}{W_{\mathrm{T}}}{\int }_0^{t}w(f^{\prime })\mathrm{d}{f}^{\prime }.$$ (51)

The functions W (f) are known as ‘Lorenz-consistent’ if they satisfy the above assumptions [33]. Four representative Lorenz consistent raw wealth distributions are shown in figure 9(a) as functions of the individual index. In figure 9(b), we plot the continuous cumulative rescaled wealth distribution W (f) as a function of the relative population fraction f corresponding to the wealth distributions shown in figure 9(a). From any ordered distribution, we can define a so-called ‘Lorenz curve’ that illustrates many indices graphically. The Lorenz curve is defined as the cumulative wealth of all individuals of a relative index f = n/N and lower.

Figure 9.

(a) Ordering of all N = 100 individuals in increasing wealth or income. The hypothetical wealth distributions plotted are w_i = 3 (equal wealth, black curve), w_i = 10 + (i − 1)/2 (linear distribution, red), W_i = 5 + e^i/5−15 − e^−14.8 (green), and w_i = 14.5 + 50/(101 − i) (blue). The latter three represent distributions with some amount of inequity. (b) These inequalities can be visually quantified by their corresponding Lorenz curves, plotted as the relative fraction of the population f. The Lorenz curve for a perfectly uniform wealth distribution is given by the straight diagonal line. The area between the diagonal equality line and any other Lorenz curve can be used to visualize the Gini coefficient of the associated wealth distribution. The Gini coefficient, Gini = A/(A + B), is calculated by dividing the difference in areas between the equality line and the Lorenz curve in question (A) by the total area (A + B = 1/2) under the equality curves. The ‘Robin Hood’ index is defined as the maximum difference between the equality line and a given Lorenz curve, and is indicated by arrow for the red and green Lorenz curves.

Many indices can be visualized by the Lorenz curves. For example, the Gini index [163, 164] for the red distribution (linear wealth) in figure 9(a) is calculated by the area of the red shaded region (A) divided by the area under the equality curve (A + B = 1/2): Gini = A/(A + B) = 2A. In a society where every person receives the same income, the Gini index equals zero. However, if the total wealth is concentrated in only one out of N entities, Gini = 1 – 2/N. This motivates one to define the Gini index for discrete cumulative wealth values W_i according to

$$\text{Gini}=1-\frac{2}{N}\sum _{i=1}^N{W}_i,$$ (52)

while the ‘Hoover’ or ‘Robin Hood’ index defined by [34, 165, 166]

$$H=\underset{f}{\max }\{\mid f-W(f)\mid \}$$ (53)

is the Legendre transform at f^*, the fraction of individuals corresponding to dW(f)/df∣_f=f^* = 1. For the two Lorenz curves in figure 9(b), the Robin Hood index is indicated by the two corresponding arrows.

The Robin Hood index happens to be a specific case of the Kolmogorov–Smirnoff statistic as defined in equation (15) for two cumulative distributions. For convex functions W(f) such that W(0) = 0, W (1) = 1, the index H corresponds to the fraction of the total wealth that needs to be distributed in order to achieve uniform wealth. This can be seen by considering the wealth w_i up to an index n^* such that W_i ⩽ N⁻¹ for all i ⩽ n^*. The total wealth that needs to be redistributed to obtain equal wealth fractions N⁻¹ for every individual is

$$H=\sum _{i=1}^{n^{\ast }}\left(\frac{1}{N}-w_i\right)=\frac{n^{\ast }}{N}-W_{n^{\ast }}\approx f^{\ast }-W(f^{\ast }).$$ (54)

Another possibility is to sum over all entities w_i according to

$$\begin{gathered} H=\frac{1}{2}\sum _{i=1}^N\mid \frac{1}{N}-w_i\mid \\ \approx \frac{1}{2}{\int }_0^1\mid 1-w(f)\mid \mathrm{d}f \\ =\frac{1}{2}\left[{\int }_0^{f^{\ast }}(1-w(f))\mathrm{d}f+{\int }_{f^{\ast }}^1(w(f)-1)\mathrm{d}f\right] \\ =f^{\ast }-W(f^{\ast }). \end{gathered}$$ (55)

The specific, local redistribution is not specified but it would be intriguing to cast it in the language of optimal transport and Wasserstein distances [167]. This way, one might also define costs to wealth redistribution.

It is also possible to quantify inequity according to the Theil index [168-170]

$$T=\frac{1}{N}\sum _{i=1}^N\frac{w_i}{\mathbb{E}[w]}\log \left(\frac{w_i}{\mathbb{E}[w]}\right),$$ (56)

which corresponds to a relative entropy as defined in equation (11). In this case, the entropy of the distribution of W_i is measured with respect to the expectation value $\mathbb{E}[w]=N^{-1}{\sum }_{i=1}^N{w}_i$. If ${\sum }_{i=1}^N{w}_i=1$, we may interpret w_i as the probability of finding an individual in income class i, and $\mathbb{E}[w]=N^{-1}$ corresponds to the relative share of equally distributed wealth. Naturally, many others measures for inequality have been defined my numerous authors focussing on specific socioeconomic areas [171].

However, typical inequality indices do not convey any judgment, belief system, or behavioral propensity on measured inequity and thus may not capture typical social concepts. In an effort to better quantify concepts such as inequity or ‘polarization’ [172], a sociologists have proposed a number of polarization indices that are argued to be more directly correlated with social tension and unrest. For example, Esteban and Ray [35, 36] developed a measure of polarization to account for clusters within which individuals are more similar in an attribute x (such as wealth) than they are between clusters. While there may be many ways to define polarization, imposing a few reasonable features and constraints can narrow down the allowable forms. First, they assume an ‘identity-alienation framework’ in which an individual also identifies with his own distribution f (x) at value x. An effective ‘antagonism’ of an individual with attribute x towards those with attribute y is defined as T[f(x), d] where a simple form for the distance is d = ∣x – y∣. The polarization P is then assumed to take the form

$$P[f]=\int \int T[f(x),\mid x-y\mid ]f(x)f(y)\mathrm{d}x\mathrm{d}y.$$ (57)

By imposing axioms that the polarization (i) cannot increase if the distribution is squeezed (compressed towards its peak), (ii) must increase if two non-overlapping distributions are moved farther apart, and (iii) the polarization should be invariant to scalings of the total population. Using these constraints, the polarization can be more explicitly defined as

$$P[f]=\int \int f^{1+\alpha }(x)f(y)\mid x-y\mid \mathrm{d}x\mathrm{d}y,$$ (58)

where 1/4 ⩽ α ⩽ 1 [36] (Esteban and Ray [35] and Kawada, Nakamura, and Sunada [173] find 0 ⩽ α < 1.6 using slightly different assumptions). The parameter α describes the amount of ‘polarization sensitivity.’ It measures identification of a population with its distribution and distinguishes polarization from other standard inequity measures such as the Gini index (when α = 0 [35]) or Simpson’s index. Also, note that when α = 0, the form of P[f] resembles the total potential energy of a system of particles which is distributed according to f (x) and exhibits an interaction energy ∣x – y∣. The discrete analogue of equation (58) is $P[f]\propto {\sum }_{i,j}{f}_i^{1+\alpha }{f}_i\mid x_i-x_j\mid$, for which the individuals i, j can be generalized to groups. In empirical studies, the Esteban and Ray polarization measure is given by

$$P_{\text{ER}}[f]\propto \sum _{i,j}{\pi }_i^{1+\alpha }{\pi }_j\mid {\mu }_i-{\mu }_j\mid ,$$ (59)

where

$${\pi }_i={\int }_{x_{i-1}}^{x_i}f(x)\mathrm{d}x\text{and}{\mu }_i=\frac{1}{{\pi }_i}{\int }_{x_{i-1}}^{x_i}xf(x)\mathrm{d}x,$$ (60)

are the relative frequency and the mean of the wealth in group i, respectively [174]. D’Ambrosio and Wolff suggested replacing the difference of mean wealths in equation (59) by the Kolmogorov measure of variation distance [174, 175]

$${\text{Kov}}_{ij}=\frac{1}{2}\int \mid f_i(y)-f_j(y)\mid \mathrm{d}y,$$ (61)

to obtain

$$P_{\text{DW}}[f]\propto \sum _{i,j}{\pi }_i^{1+\alpha }{\pi }_j{\text{Kov}}_{ij}.$$ (62)

Additional indices have been proposed, including a class of polarizations by Tsui and Wang [176] of the form

$$P_{\text{TW}}(\mathbf{x})=\frac{1}{N}\sum _{i=1}^N\psi (d_i),d_i=\mid \frac{x_i-m(\mathbf{x})}{m(\mathbf{x})}\mid ,$$ (63)

where ψ is a smooth function of the rescaled distance d_i. The median income m(x) is computed from the individual incomes x_i (1 ⩽ i ⩽ N).

Many of these polarization metrics can in fact be expressed in terms of the Gini coefficient. For example, the Foster–Wolfson polarization index is defined as [177]

$$P_{\text{FW}}(\mathbf{x})=({\text{Gini}}_B-{\text{Gini}}_W)(\mu (\mathbf{x})∕m(\mathbf{x})),$$ (64)

where μ(x) is the corresponding mean income, and the subscript indices B and W denote the between and within group Gini coefficients. According to the definition of P_FW (x), inequity differs from polarization in the following way: the Gini index as the sum of Gini_B and Gini_W quantifies the unequal distribution of wealth in a society whereas polarization is measured in terms of the difference of Gini_B and Gini_W. Thus, an increase in within-group inequality leads to a larger total inequality, but to a lower polarization. A more refined understanding of socioeconomic diversity will need to consider multiple classes of attributes, including possible geographic or spatial distributions.

The described polarization measures are relevant not only in the context of wealth distributions, but they are also able to provide important insights in other sociological phenomena associated with the notion of diversity. As one example, quantitative measures of polarization are applicable to examine factors that influence the cohesiveness of groups [23]. In addition, diversity measures may help to identify mechanisms which lead to inequality among different social groups in our education system [24]. In this context, the social entropy theory aims to quantitatively compare diversity across social systems such as societies, organizations, and individual groups [19, 20, 178].

7. Summary and discussion

Quantifying the diversity of a given population in terms of a single measure such as richness does not fully describe the underlying distribution of species or other properties. Various diversity measures have been developed and tailored to specific applications in different fields including ecology, biology, and economics. Mathematically, one can describe populations in terms of species numbers n_i (number of entities of type i) or clone counts c_k (number of species of size k). Hill numbers ^qD provide a framework to unify some common diversity indices that are based on a species-number description. Hill numbers with large values of q put more weight on common species whereas small values of q yield measures that are more sensitive to rarer species. This implies that measures such as richness (q = 0) and evenness (q = 1) are more prone to sampling effects than Simpson’s diversity index (q = 2) or Hill numbers with q > 2 [180]. In table 1, we summarize some common diversity measures, their applications, and advantages and disadvantages.

Table 1.

Summary of fundamental, commonly used diversity measures. The variable q indicates the order of the corresponding Hill number ^qD as defined in equation (16).

Measure	Interpretation	Application	Advantages	Disadvantages
Species number ni	Number of entities of type i	Evolutionary and population models	Straightforward interpretation in models	Keeping track of species identity may be unrealistic
Species abundance (clone count) ck.	Number of species of size k	Models of self-assembly/nucleation [52-54]; characterization of population in barcoding studies [55]	Directly related to richness, useful when clone identity is not important	No clone identity information, insensitive to exchange of populations between clones
Richness (0D)	Total number of distinguishable species	Conservation planning; assessment of ecosystems [3]	Straightforward mathematical definition and interpretation	Maximally affected by small sampling; all species are treated equally
Evenness (1D)	Uniformity of relative abundances of species in a population	Characterization of ecosystems and inequity in societies; Theil index [3, 168-170]	Straightforward mathematical definition and interpretation; similar to entropy	Affected by sampling
Simpson’s diversity (2D)	Probability that two randomly drawn entities are of the same species	Characterization of cell populations [147, 179]	Less affected by sampling	More intricate mathematical definition & Less interpretability
qD, q > 2	N/A	Characterization of more frequent species in a population; Berger–Parker index [90]	Significantly less affected by sampling	No intuitive interpretation
Lorenz curve	Cumulative relative wealth	Economics, wealth distributions	Fundamental mathematical object	No identity information (like ordered clone counts)
Gini index	Deviation of Lorenz curves from absolute equality	Population-level wealth inequality	Easily understood	No identity information, Values are subjectively interpreted
Hoover/Robin Hood index	KS statistic between Lorenz curve and equality line	Population-level wealth inequality	Easily understood	No identity information, values are subjectively interpreted

In conclusion, we have provided an overview of the most relevant measures of diversity and their information-theoretic counterparts. We then summarized common applications of diversity indices in biological and ecological systems. Despite the ambiguity in the definitions and the variety of different diversity measures [3, 4, 25-29], the concept is of great importance for the monitoring of ecosystems and in the context of conservation planning [2, 5, 9, 10, 29-31].

We also described the importance of a quantitative treatment of diversity for experiments in the study of the gut microbiome, stem cell barcoding, and the adaptive immune system. Finally, we discussed examples of the application of diversity measures in human social systems including the characterization of wealth distributions in societies and measures of political or cultural polarization. Scientific conclusion in these fields, and in ecology, are particularly sensitive to sampling and measurements. However, accurate measurements [181], meaningful classification, spatial resolution [101], and informative sampling protocols [69, 76] remain elusive across almost all fields. Sometimes, as shown for example in figure 6(b), different measures even lead to contradictory conclusions [182]. There is no golden rule in choosing a unique metric for a specific situation, as the sampling effects also depend on the underlying unknown clone-count distribution [180]. It is recommended that one considers different metrics and cross-checks their values while bearing in mind how sampling effects may impact diversity measures differently.

Figure 5.

(a) Protocol for Viral Integration site (VIS) barcoding studies of hematopoiesis in rhesus macaque [55, 117, 118]. Here, ‘barcodes’ are defined by the random integration sites of a lentiviral vector. (b) Xenograft barcode experiments using mice [119] in which a library of barcodes was used to tag leukemia-propagating cells before direct transplantation into mice.