Evolutionarily stable strategies in stable and periodically fluctuating populations: The Rosenzweig–MacArthur predator–prey model

Significance

Many evolutionary studies of ecological systems assume, explicitly or implicitly, ecologically stable population dynamics. Ecological analyses typically assume, on the other hand, no evolution. We study a model (using predator–prey dynamics as an example) combining ecology and evolution within the same framework. For this purpose, we use the evolutionarily stable strategies (ESSs) framework, emphasizing that evolutionary change, in general, will occur as a result of mutant strategies being able to invade a population. The significance of our contribution is to derive mathematical conditions for the existence of an ESS in a periodically limit-cycle ecological system.

Abstract

An evolutionarily stable strategy (ESS) is an evolutionary strategy that, if adapted by a population, cannot be invaded by any deviating (mutant) strategy. The concept of ESS has been extensively studied and widely applied in ecology and evolutionary biology [M. Smith, On Evolution (1972)] but typically on the assumption that the system is ecologically stable. With reference to a Rosenzweig–MacArthur predator–prey model [M. Rosenzweig, R. MacArthur, Am. Nat. 97, 209–223 (1963)], we derive the mathematical conditions for the existence of an ESS when the ecological dynamics have asymptotically stable limit points as well as limit cycles. By extending the framework of Reed and Stenseth [J. Reed, N. C. Stenseth, J. Theoret. Biol. 108, 491–508 (1984)], we find that ESSs occur at values of the evolutionary strategies that are local optima of certain functions of the model parameters. These functions are identified and shown to have a similar form for both stable and fluctuating populations. We illustrate these results with a concrete example.

1. Introduction

Natural systems exhibit both ecological and evolutionary dynamics. Nevertheless, many studies focus only on either the ecological or the evolutionary dynamics, assuming the other nonchanging and fixed. There are many excellent exceptions (see, e.g., refs. 1 and 2). Here, we study the ecological and evolutionary dynamics in both ecologically stable as well as periodically fluctuating populations of predator and prey, using the Rosenzweig–MacArthur predator–prey model (3) (see also refs. 4–9). Our overall aim is to understand under what conditions evolutionarily stable strategies (ESSs) will exist. Specifically, we focus on evolutionary stability in stable as well as periodically fluctuating ecological systems.

The concept ESS was originally coined by Maynard Smith (10) (see also Smith and Price) (11), as an extension of Hamilton’s (12) “unbeatable strategy.” Maynard Smith (10) defined it as a strategy that, once adopted by a majority of the members of a population, cannot be overturned by any alternative strategy that is initially rare. It is essentially a Nash equilibrium that is evolutionary stable (13). The concept of ESS was used initially in behavioral ecology and sociobiology but Lawlor and Smith (14)—see also Roughgarden (15) and Nowak (16)—extended its use to ecological dynamics of competing populations.

Many ecological systems are characterized by fluctuating dynamics, some of which have periodic fluctuations: the famous lemming and small rodent cycles (see, e.g., refs. 17–19) and the North American hare–lynx cycle (see, e.g., ref. 20). The predator–prey models of the kind presented by Rosenzweig and MacArthur (17) are therefore of broad ecological interest. Several studies have analyzed these—and related—models from an ecoevolutionary perspective (5, 21–26). However, no one has addressed this from an ESS perspective, particularly for the limit-cycle case. This we do in the present contribution.

Models for predatory–prey interactions have played a major role within the field of ecology since the pioneering contribution by Lotka (27, 28) and Volterra (29) (see, e.g., ref. 30). Depending on the choice of parameters, such models are known to accommodate equilibrium populations that are constant or fluctuating (including limit-cycle dynamics).

The biological contribution of this paper will be the theoretical question regarding the existence of ESS in periodically fluctuating populations exemplified by Rosenzweig–MacArthur predator–prey systems. Specifically, we establish sufficient conditions for the existence of ESSs in such models. We consider populations that become stable (constant) over time and periodically fluctuating populations or systems exhibiting stable limit-cycle dynamics. Extending the approach of Reed and Stenseth in ref. 31, we show that ESSs occur at values of the evolutionary strategies that are local optima of certain functions of the model parameters. We identify these functions and show that they have a similar form for both stable and fluctuating populations. These results are illustrated with a concrete example.

The main mathematical contribution is that we prove Lyapunov stability through an adaptation of classical linearization arguments, variation of parameter formulas, and Floquet-type analysis. The main finding is the case of limit cycles, where we establish stability of two-dimensional limit cycles in a four-dimensional system of ordinary differential equations. Within the approach of Reed and Stenseth (31) to evolutionary games and ESS, we then show that these stability results lead to sufficient conditions for ESSs for both types of equilibrium solutions. These sufficient conditions could be of independent interest in game theory.

In section 2, we state the model and results about equilibrium solutions. Then, in section 3, we combine ecology and evolution: evolutionary strategies, an extended ecological model, and the concept of ESS are introduced. The strategies are model parameters that can be changed over time by the populations to ensure a more beneficial evolution, and the extended model includes mutant populations for predators and prey satisfying the same model equations at different (i.e., mutant) values of the strategies. An ESS is introduced as an extension of the definition of Reed and Stenseth (31) to include periodically fluctuating populations—it is a choice of evolutionary strategies for which mutant populations cannot survive in the long run. Evolution can be seen as a competition between different species and be modeled as a game (13). An ESS is then a Nash equilibrium (32), in the sense that no population can improve their own survivability acting alone. A game can only move away from an ESS when the original populations change their strategies. An ESS is therefore a type of locally optimal evolutionary strategy for the original populations.

When time becomes large, the populations will approach limit populations, a (stable) equilibrium solution of the model. Examples of equilibrium solutions are constants (limit points) and periodic solutions (limit cycles). In sections 4 and 5, we study such solutions for the extended ecological model. We first observe that equilibrium solutions of the two-population model remain equilibrium solutions in the extended model when the mutant populations are zero. Then, we establish conditions to guarantee the (Lyapunov) stability of these solutions. Stability here means that solutions starting near an equilibrium solution will over time converge to the equilibrium solution. These results extend earlier results for models with two populations. By taking into account the dependence on the strategies in our extended model, we use the stability results and our definition of ESS to give conditions that guarantee ESS. In section 4, we give the results for equilibrium points (constant populations in the limit), and in section 5, we give the results for limit cycles (periodic limit populations). ESSs are shown to occur at values of the evolutionary strategies that are local optima of certain functions of the model parameters. These results are the main contribution of our paper. In section 4, we also discuss examples of functional dependencies that lead to an ESS: loosely speaking, if we take the predation rate to be an increasing bilinear function of the evolutionary strategies, then we find that the rate constants of prey growth and predator mortality have to be decreasing convex functions. We give a concrete example where the latter two rate constants are quadratic functions of the evolutionary strategies.

2. The Ecology of Predator–Prey Models

Let $x_1$ and $y_1$ denote the population sizes of prey and predator, respectively. We will assume that the dynamics follow a logistic Gause-type model:

$$\frac{d}{dt}\left(\begin{array}{c}\hfill x_1\hfill \\ \hfill y_1\hfill \end{array}\right)\hspace{0.17em}=\hspace{0.17em}\left(\begin{array}{c}\hfill x_1\left(f_1\left(x_1\right)-y_1{\phi }_{11}\left(x_1\right)\right)\hfill \\ \hfill y_1\left(-c_1+k{x}_1{\phi }_{11}\left(x_1\right)\right)\hfill \end{array}\right)\hspace{0.17em}=\hspace{0.17em}{F}_1\left(x_1,y_1\right).$$ [1]

Here, the rate of predation is $x_1{y}_1{\phi }_{11}\left(x_1\right)$, where ${\phi }_{11}$ is given by

$${\phi }_{11}\left(x\right)=\frac{a}{b_{11}+x}$$ [2]

for positive constants $a$ and $b_{11}$. The quantity ${\phi }_{11}\left(x\right)x$ is referred to in ecological literature as the functional response curve, with $a$ being the saturation point and $b$ the half-saturation constant in the sense that $x{\phi }_{11}\left(x\right){|}_{x=b_{11}}=a/2$ (33, 34). Furthermore, the predator mortality rate is given by $c_1{y}_1$ for $c_1>0$, and the prey growth rate $x_1{f}_1\left(x_1\right)$ by the commonly used logistic model, where

$$f_1\left(x\right)=r_1\left(1-\frac{x}{K}\right),$$ [3]

$r_1>0$ is the rate constant and $K>0$ the carrying capacity of the prey population. Note that $f_1$ is decreasing, with $f_1\left(0\right)=r_1$ and $f_1\left(K\right)=0$.

This model goes into a long tradition of predator–prey models of the form

$$\frac{dx}{dt}=xF\left(x,y\right),\hspace{1em}\frac{dy}{dt}=yG\left(x,y\right),$$ [4]

first identified by Kolmogorov (35) in 1936 (see also ref. 36). The system in Eq. 1 has been intensively studied in the literature (see, e.g., refs. 37–39 and the references therein).

For this system, the first quadrant is an invariant region (i.e., the populations can never become negative). Under some reasonable assumptions, the system has three limit points and up to one limit cycle. These results along with stability results are given in the next theorem (also see Fig. 1).

Fig. 1.

The phase diagram of Eq. 1 with $a=2.8$, $b_{11}=0.7$, $c_1=1.35$, $r_1=3.5$, $k=1.5$, and $K=1.4$, which corresponds to case (iii) in Theorem 2.1. (A) The full phase diagram. Dashed curves as well as the coordinate axes are isoclines. The brown points give the stationary points. The red curve is the limit cycle. (B) A close-up of the full diagram in A. If $K>b_{11}+2\widehat{x}$, then $\left(\widehat{x},ŷ\right)$ is unstable and there exists exactly one limit cycle in the first quadrant in the $\left(x_1,y_1\right)$ plane, which is an (asymptotically) stable limit cycle.

Theorem 2.1 (37)

Assume that $x_1\left(0\right)=x_0\in \left(0,K\right)$ and $y_1\left(0\right)=y_0\in \left(0,\infty \right)$.

• (a)Solutions $\left(x_1\left(t\right),y_1\left(t\right)\right)$ of Eq. 1 are positive and bounded.
• (b)Define $\sigma =ak/c_1$ and $\widehat{x}=b_{11}/\left(\sigma -1\right)$ if $\sigma >1$. Then:
• (i)If $\sigma \le 1$ or $K\le \widehat{x}$, then the critical point $\left(K,0\right)$ is asymptotically stable and

$$\underset{t\to \infty }{lim}{x}_1\left(t\right)=K,\hspace{1em}\text{and}\hspace{1em}\underset{t\to \infty }{lim}{y}_1\left(t\right)=0.$$ [5]

• (ii)If $\widehat{x}<K\le b_{11}+2\widehat{x}$, then the critical point $\left(\widehat{x},ŷ\right)$, with $ŷ=\left(r_1/a\right)\left(1-\widehat{x}/K\right)\left(b_{11}+\widehat{x}\right)$, is asymptotically stable and

$$\underset{t\to \infty }{lim}{x}_1\left(t\right)=\widehat{x},\hspace{1em}\text{and}\hspace{1em}\underset{t\to \infty }{lim}{y}_1\left(t\right)=ŷ.$$ [6]

• (iii)If $K>b_{11}+2\widehat{x}$, then $\left(\widehat{x},ŷ\right)$ is unstable and there exists exactly one limit cycle in the first quadrant in the $\left(x_1,y_1\right)$ plane, which is an (asymptotically) stable limit cycle.

Further properties can be found in SI Appendix.

3. Evolution: The Extended Ecological Model

Evolutionarily, there is a “conflict” or “arms race” between the predator and the prey regarding predation: the prey “wants” to evolve to avoid being caught by the predator, whereas the predator “wants” to be able to catch prey as efficiently as possible, even when the prey is at low abundance. However, this will come at some costs: we assume that for the prey, a decreased predation rate will lead to a reduced growth rate, whereas for the predator, an increased predation rate may lead to an increased mortality rate. A natural way to analyze this situation is through (differential) game theory and concepts like Nash equilibriums and ESS; we refer to refs. 13, 14, 40, and 41 for more information. We will follow the simplified approach of Reed and Stenseth (31), an approach explicitly emphasizing that a new mutant will have to establish itself through competition, even though rare initially.

Let ${\alpha }_1$ and ${\beta }_1$ represent the strategies of prey and predator, respectively. All model parameters are assumed to depend on ${\alpha }_1$ and ${\beta }_1$, i.e., different mutations correspond to different values of the various constants in Eqs. 1–3. The idea is that over time, the populations can modify their own strategies in order to increase their chances of survival.

We now make some assumptions about how the different model parameters depend on ${\alpha }_1$ and ${\beta }_1$. We know from, e.g., studies of the hare–lynx cycle (42) that the predator and the prey mutually “disagree” on the value of the half-saturation constant; for instance, large $b$ and hence low predation is beneficial for the hare, while the opposite is the case for the lynx. We therefore take

$$b_{11}=b\left({\alpha }_1,{\beta }_1\right).$$

To simplify, we then assume that $K$ and $k$ are constants, while

$$r_1=r\left({\alpha }_1\right)\hspace{1em}\hspace{1em}\text{and}\hspace{1em}\hspace{1em}{c}_1=c\left({\beta }_1\right).$$

We assume the following reasonable constraints/tradeoffs among the parameters under evolution:

$$\frac{d{r}_1}{d{\alpha }_1}\le 0,\hspace{1em}\frac{d{c}_1}{d{\beta }_1}\le 0,\hspace{1em}\frac{\partial b_{11}}{\partial {\alpha }_1}>0,\hspace{1em}\text{and}\hspace{1em}\frac{\partial b_{11}}{\partial {\beta }_1}>0.$$ [7]

The interpretation is the following: an increase in $b_{11}$ (which means smaller ${\phi }_{11}$) should result in both a decrease in $r_1$ and $c_1$. This means that a lower predation rate, which is beneficial for the prey, comes at the cost of lower prey growth rates. Lower predation rates are bad for the predators and are compensated by lower mortality rates. Thus with $X_1=\left(\begin{array}{c}\hfill x_1\hfill \\ \hfill y_1\hfill \end{array}\right)$, Eq. 1 can be written as

$$\frac{d}{dt}{X}_1=F_1\left(X_1,{\alpha }_1,{\beta }_1\right).$$ [8]

In this paper, the strategies (i.e., the ecological parameters under evolution) do not depend on $X_1$ or time $t$.

In the next step, we extend the model to include mutants. Let $x_1$ and $y_1$ denote the original prey and predator populations, respectively, and $x_2$ and $y_2$ denote the corresponding mutant populations. The extended mode then takes the form

$$\frac{d{x}_1}{dt}=x_1\left(f_1\left(x_1+x_2\right)-y_1{\phi }_{11}\left(x_1+x_2\right)-y_2{\phi }_{12}\left(x_1+x_2\right)\right),$$ [9a]

$$\frac{d{y}_1}{dt}=y_1\left(-c_1+k{x}_1{\phi }_{11}\left(x_1+x_2\right)+k{x}_2{\phi }_{21}\left(x_1+x_2\right)\right),$$ [9b]

$$\frac{d{x}_2}{dt}=x_2\left(f_2\left(x_1+x_2\right)-y_1{\phi }_{21}\left(x_1+x_2\right)-y_2{\phi }_{22}\left(x_1+x_2\right)\right),$$ [9c]

$$\frac{d{y}_2}{dt}=y_2\left(-c_2+k{x}_1{\phi }_{12}\left(x_1+x_2\right)+k{x}_2{\phi }_{22}\left(x_1+x_2\right)\right),$$ [9d]

where for $i,j=\mathrm{1,2}$,

$${\phi }_{ij}\left(x\right)=\frac{a}{b_{ij}+x},$$ [10]

$$f_i\left(x\right)=r_i\left(1-\frac{x}{K}\right),$$ [11]

and $c_i,k,a,b_{ij},r_i,K$ are positive constants. Note that the rate of predation of $y_j$ upon $x_i$ is $x_i{y}_j{\phi }_{ij}\left(x_1+x_2\right)$. Also note that

$$\begin{array}{ll}\hfill {\phi }_{ij}^{\prime }\left(x\right)& =-\frac{1}{a}{\phi }_{ij}^2\left(x\right),\hfill \\ \hfill {\phi }_{ij}\left(x\right)-{\phi }_{kl}\left(x\right)& =\frac{1}{a}\left(b_{kl}-b_{ij}\right){\phi }_{ij}\left(x\right){\phi }_{kl}\left(x\right),\hfill \end{array}$$

$0\le {\phi }_{ij}\left(x\right)\le a/b_{ij}$ and $0\le x{\phi }_{ij}\left(x\right)\le a$ when $x\ge 0$, and $0\le {\left(x{\phi }_{ij}\left(x\right)\right)}^{\prime }\le a/b_{ij}$.

We now introduce evolutionary strategies $\left({\alpha }_1,{\beta }_1\right)$, $\left({\alpha }_2,{\beta }_2\right)$ for $\left(x_1,y_1\right)$, $\left(x_2,y_2\right)$, respectively. As before, we assume for $i=\mathrm{1,2}$ that

$$r_i=r\left({\alpha }_i\right),\hspace{1em}{c}_i=c\left({\beta }_i\right),\hspace{1em}\text{and}\hspace{1em}{b}_{ij}=b\left({\alpha }_i,{\beta }_j\right),$$ [12]

and require that

$$\frac{d{r}_i}{d{\alpha }_i}\le 0,\hspace{1em}\frac{d{c}_i}{d{\beta }_i}\le 0,\hspace{1em}\frac{\partial b_{ij}}{\partial {\alpha }_i}>0,\hspace{1em}\text{and}\hspace{1em}\frac{\partial b_{ij}}{\partial {\beta }_j}>0.$$ [13]

Note that the dependence in $i,j$ is through ${\alpha }_i$ and ${\beta }_j$ only and that the mutants and original populations satisfy the same model equations, only for different values of the strategies. With $X_2=\left(\begin{array}{c}\hfill x_2\hfill \\ \hfill y_2\hfill \end{array}\right)$, Eq. 9 can then be rewritten as follows:

$$\frac{d}{dt}\left(\begin{array}{c}\hfill X_1\hfill \\ \hfill X_2\hfill \end{array}\right)=M\left(X_1,X_2,{\alpha }_1,{\beta }_1,{\alpha }_2,{\beta }_2\right),$$ [14]

where $M$ denotes the right-hand side of Eq. 9.

Remark 3.1:

• (i)If ${\widehat{X}}_1$ is an equilibrium point (stable or not) for Eq. 8, then $\left({\widehat{X}}_1,0\right)$ is an equilibrium point for Eq. 14 and

$$M\left({\widehat{X}}_1,0,{\alpha }_1,{\beta }_1,{\alpha }_2,{\beta }_2\right)=\left(\begin{array}{c}\hfill F_1\left({\widehat{X}}_1,{\alpha }_1,{\beta }_1\right)\hfill \\ \hfill 0\hfill \end{array}\right).$$

• (ii)If $\gamma \left(t\right)$ is a periodic solution (a limit cycle) for Eq. 8, then $\left(\gamma \left(t\right),0\right)$ is a periodic solution for Eq. 14 and

$$M\left(\gamma \left(t\right),0,{\alpha }_1,{\beta }_1,{\alpha }_2,{\beta }_2\right)=\left(\begin{array}{c}\hfill F_1\left(\gamma \left(t\right),{\alpha }_1,{\beta }_1\right)\hfill \\ \hfill 0\hfill \end{array}\right).$$

• (iii)If $\left({\alpha }_1,{\beta }_1\right)=\left({\alpha }_2,{\beta }_2\right)$, then by Eq. 12, $r_i=r\left({\alpha }_1\right)$, $c_i=c\left({\beta }_1\right)$, and $b_{ij}=b\left({\alpha }_1,{\beta }_1\right)$ for $i,j=\mathrm{1,2}$, and hence $\left(x_1+x_2,y_1+y_2\right)$ is a solution to (the nonlinear) Eq. 8. This is consistent since now there are no mutants, and the populations should then be determined by Eq. 8 alone.

An ESS corresponds to the situation where the original population $\left(x_1,y_1\right)$ cannot be invaded by mutants $\left(x_2,y_2\right)$ when only small evolutionary deviations are allowed.

Definition 3.2:

Assume we are given an asymptotically stable equilibrium point (limit cycle) ${\widehat{X}}_1$ to Eq. 8 with corresponding constant strategy $\left(\widehat{\alpha },\widehat{\beta }\right)$. Then $\left(\widehat{\alpha },\widehat{\beta }\right)$ is called an ESS if $\left(X_1,X_2\right)=\left({\widehat{X}}_1,0\right)$ is an asymptotically stable equilibrium point (limit cycle) to Eq. 14 for all constant strategies $\left(\widehat{\alpha },\widehat{\beta },{\alpha }_2,{\beta }_2\right)\hspace{0.17em}/\hspace{0.17em}=\hspace{0.17em}\left(\widehat{\alpha },\widehat{\beta },\widehat{\alpha },\widehat{\beta }\right)$ sufficiently close to $\left(\widehat{\alpha },\widehat{\beta },\widehat{\alpha },\widehat{\beta }\right)$.

An ESS has the Nash equilibrium-like property that if $\left(\widehat{\alpha },\widehat{\beta }\right)$ does not change, mutants with strategy $\left({\alpha }_2,{\beta }_2\right)\hspace{0.17em}/\hspace{0.17em}=\hspace{0.17em}\left(\widehat{\alpha },\widehat{\beta }\right)$ will never survive in the long run.

Remark 3.3:

The ESS, as defined in Definition 3.2, is stable against mutations in one or both populations at the same time. Single population mutations are realized simply by letting the other mutant population be zero over time.

The issue of time scales for the ecological and evolutionary processes is important here (see, e.g., Carroll et al.) (43). The approach of Reed and Stenseth (31) emphasizes the ecological interaction between the common wild-type strategy (with its corresponding phenotype) and the rare mutant strategy (with its corresponding phenotype). If the mutant strategy cannot (ecologically) invade the wild-type ecological system, there will be evolutionary stability (here in the form of ESS). If the mutant strategy can invade the wild-type ecological system, it will eventually change into a new one. The result will depend upon many features of the different populations—issues that are beyond our current discussion but are essentially addressed by Dercole et al. (44), Dercole and Rinaldi (45), Doebeli (46), and Cortez and Weitz (26). This all refers to ecological processes and time scales. The overall question being addressed in this contribution is under which conditions will we have evolutionary stability in ecologically varying (periodically fluctuating) systems.

The evolutionary processes and time scales enter in relation to how frequent new mutant strategies appear in the two populations (through the process of mutation and/or invasions)—issues outside of our current discussion. The discussion by Khibnik and Kondrashov (47) are relevant here.

4. Equilibrium Points, Stability, and ESS

SI Appendix, Eq. 1 has three equilibrium points: $\left(\mathrm{0,0}\right)$, $\left(K,0\right)$, and $\left(\widehat{x},ŷ\right)$, where

$${\phi }_{11}\left(\widehat{x}\right)\widehat{x}=\frac{c_1}{k},$$ [15a]

$$ŷ=\frac{f_1\left(\widehat{x}\right)}{{\phi }_{11}\left(\widehat{x}\right)}=\frac{k}{c_1}{f}_1\left(\widehat{x}\right)\widehat{x}.$$ [15b]

Theorem 2.1 states that the equilibrium point $\left(\widehat{x},ŷ\right)$ is asymptotically stable provided $\sigma =\frac{ak}{c_1}>1$ and $\widehat{x}<K<b_{11}+2\widehat{x}$, making the eigenvalues ${\lambda }_1,{\lambda }_2$ for Eq. 1 negative. These eigenvalues are also eigenvalues of the full system Eq. 9.

We show in SI Appendix that the eigenvalues ${\lambda }_3$ and ${\lambda }_4$ for the Jacobi matrix $dM$ of the full system Eq. 9 are

$${\lambda }_3={\widehat{f}}_2-{\widehat{\phi }}_{21}ŷ,$$ [16a]

$${\lambda }_4=-c_2+c_1\frac{{\widehat{\phi }}_{12}}{{\widehat{\phi }}_{11}}.$$ [16b]

When ${\lambda }_1,\dots ,{\lambda }_4$ are negative at $\left(\widehat{x},ŷ,\mathrm{0,0}\right)$, then this point is an asymptotically stable equilibrium point for the full system Eq. 14. We summarize this discussion in the following result.

Lemma 4.1

The equilibrium point $\left(\widehat{x},ŷ,\mathrm{0,0}\right)$ is asymptotically stable if $\sigma =ak/c_1>1$, $\widehat{x}<K<b_{11}+2\widehat{x}$,

$$r_2\left(b_{21}+\widehat{x}\right)<r_1\left(b_{11}+\widehat{x}\right),\hspace{1em}\text{and}\hspace{1em}\widehat{x}\left(ak-c_2\right)<c_2{b}_{12}.$$

See SI Appendix, Lemma 9.5 and below in SI Appendix for a full proof. From this result and our definition of an ESS, we find the following conditions that guarantee an ESS when the system has a stable equilibrium point as the global attractor or omega limit point. In other words, at this ESS the original populations are nearly constant after some time.

Theorem 4.2 (Conditions for ESS—Equilibrium Point Case)

Assume Eq. 12. A constant evolutionary strategy $\left(\widehat{\alpha },\widehat{\beta }\right)$ is an ESS for Eq. 14 if

• (i)$\sigma =\frac{ak}{c\left(\widehat{\beta }\right)}>1$,
• (ii)$\widehat{x}<K<b\left(\widehat{\alpha },\widehat{\beta }\right)+2\widehat{x}$,
• (iii)the function $e_1\left({\alpha }_2\right)=r\left({\alpha }_2\right)\left(1-\frac{\widehat{x}}{K}\right)\left(b\left({\alpha }_2,\widehat{\beta }\right)+\widehat{x}\right)$ has a strict local maximum at the point ${\alpha }_2=\widehat{\alpha }$,
• (iv)the function $e_2\left({\beta }_2\right)=c\left({\beta }_2\right)b\left(\widehat{\alpha },{\beta }_2\right){\left(ka-c\left({\beta }_2\right)\right)}^{-1}$ has a strict local minimum at the point ${\beta }_2=\widehat{\beta }$.

Proof:

Note that by the definition of $\widehat{x}$, the second inequality in Lemma 4.1 can be written as $c_1{b}_{11}/\left(ak-c_1\right)<c_2{b}_{12}/\left(ak-c_2\right)$ when $ak-c_2>0$. The result then follows from Eq. 12, the definition of an ESS, and Lemma 4.1.

Now, we consider an example where $b$, $r$, and $c$ are given as bilinear, quadratic, and rational functions, respectively, of $\alpha$ and $\beta$. In this case, we use Theorem 4.2 to find the values of the ESS. We also check that Eq. 13 is satisfied near these ESSs.

Example 4.3:

Assume $b\left(\alpha ,\beta \right)=B\alpha \beta$, $c\left(\beta \right)=ak\left(C_1{\left(\beta -{\beta }_0\right)}^2+C_2\right)/\left(1+C_1{\left(\beta -{\beta }_0\right)}^2+C_2\right)$, and $r\left(\alpha \right)=R{\left(\alpha -{\alpha }_0\right)}^2$ for positive constants $B,C_1,C_2,R,{\alpha }_0,{\beta }_0$. Then, the conditions of Theorem 4.2 are satisfied for $\left(\widehat{\alpha },\widehat{\beta }\right)$, where

$$\begin{array}{ll}\hfill \widehat{\alpha }& =\frac{ak-c\left(\widehat{\beta }\right)}{3ak-c\left(\widehat{\beta }\right)}{\alpha }_0,\hfill \\ \hfill \widehat{\beta }& =\frac{1}{3}\left(2{\beta }_0+{\left({\beta }_0^2-\frac{3C_2}{C_1}\right)}^{1/2}\right),\hfill \end{array}$$

under the conditions

$$\begin{array}{ll}\hfill {\beta }_0^2& >\frac{3C_2}{C_1},\hfill \\ \hfill 1<\frac{K}{B\widehat{\alpha }\widehat{\beta }}\left(\frac{ak}{c\left(\widehat{\beta }\right)}-1\right)& <1+\frac{ak}{c\left(\widehat{\beta }\right)}.\hfill \end{array}$$

Note that $\widehat{\alpha }\in \left(0,{\alpha }_0\right)$ and $\widehat{\beta }\in \left(0,{\beta }_0\right)$. We conclude that $\left(\widehat{\alpha },\widehat{\beta }\right)$ is an ESS for Eq. 14. From a direct calculation, it now follows that

$$\frac{dr}{d\alpha }\left(\widehat{\alpha }\right)<0,\hspace{1em}\frac{dc}{d\beta }\left(\widehat{\beta }\right)<0,\hspace{1em}\frac{d^{2}r}{d{\alpha }^2}\left(\widehat{\alpha }\right)>0,\text{and}\hspace{1em}\frac{d^{2}c}{d{\beta }^2}\left(\widehat{\beta }\right)>0.$$

In other words, the natural relations Eq. 13 hold at the ESS, and the functions $r$ and $c$ are convex decreasing at the ESS (Figs. 2 and 3).

Fig. 2.

The function $r\left(\alpha \right)$.

Fig. 3.

The function $c\left(\beta \right)$. Note that $C_2\to 0$ implies $\widehat{\beta }\to {\beta }_0$.

Note that we need $C_2>0$. If $C_2=0$, then $\widehat{\beta }={\beta }_0$, $c\left(\widehat{\beta }\right)=0$, and $\frac{dc}{d\beta }\left(\widehat{\beta }\right)=0$. However, this is unrealistic since the predator mortality rate constant $c$ must be strictly positive. Eq. 13 is also violated. See SI Appendix, Remark 9.7 for further computations on this example.

Biologically, this example may be interpreted as follows. An ESS will exist when the evolutionary strategies ${\alpha }_1$ and ${\beta }_1$ are linked, respectively, to the logistic growth rate constant $r$ for the prey and the mortality rate constant $c$ for the predator, in a decreasing convex fashion, while at the same time, the predation half-loading constant $b$ is bilinearly increasing.

5. Limit Cycles, Stability, and ESS

We now find conditions for when there exist (asymptotically) stable limit-cycle solutions of Eq. 9 with the two last components $x_2$ and $y_2$ equal to 0. Thereafter, we will look at the parameter dependent system Eq. 14 and identify ESSs. When $x_2$ and $y_2$ are identically 0, our model reduces to Eq. 1. To prove the existence of a limit cycle inside the first quadrant, both the position and the stability of the three equilibrium points $\left(\mathrm{0,0}\right)$, $\left(K,0\right)$, and $\left(\widehat{x},ŷ\right)$ play an essential role. If there exists a limit cycle inside the first quadrant, it must surround at least one equilibrium point. In our case it must surround $\left(\widehat{x},ŷ\right)$, which lies in the first quadrant by Theorem 2.1 if we assume that ${\sigma }_1=\frac{ak}{c_1}>1$. By the same theorem, it follows that $\left(\widehat{x},ŷ\right)$ is a stable node or spiral if $K<b_{11}+2\widehat{x}$ and all phase paths inside the first quadrant end up in $\left(\widehat{x},ŷ\right)$. In this case there cannot exist a limit cycle. If, on the other hand, $K>b_{11}+2\widehat{x}$, then there exists at least one limit cycle surrounding $\left(\widehat{x},ŷ\right)$. To establish the existence of an asymptotically stable limit cycle is much harder than establishing the existence of an asymptotically stable equilibrium point, but the main ideas behind are quite similar.

In ref. 37, it has been shown that every limit cycle surrounding $\left(\widehat{x},ŷ\right)$ is asymptotically stable. As an immediate consequence it then follows that there is at most one limit cycle in the first quadrant, since an asymptotically stable limit cycle cannot be surrounded by another asymptotically stable limit cycle. The key ingredient in the argument is the following theorem, which is taken from ref. 48 (Corollary, p. 216) and adjusted to our notation and assumptions.

Theorem 5.1

Let $\gamma \left(t\right)=\left(x\left(t\right),y\left(t\right)\right)$ be a periodic solution of Eq. 1 of period $T$, and assume that the right-hand side of Eq. 1, $F_1$, is continuously differentiable. Then, $\gamma \left(t\right)$ is an (asymptotically) stable limit cycle if

$${\int }_0^T\nabla \cdot F_1\left(\gamma \left(s\right)\right)ds<0.$$ [17]

Remark 5.2:

The condition in Eq. 17 in Theorem 5.1 plays a similar role for limit cycles in ${\mathbb{R}}^2$ as checking the eigenvalues to determine the stability of equilibrium points. It is derived by a topological argument that makes it possible to compute the sign of the generalized eigenvalues for the limit cycle of Eq. 1.

In our case, this condition, which has been checked in ref. 37, reads,

$$\begin{array}{ll}\hfill {\int }_0^T& x\left(s\right)\left(f_1^{\prime }\left(x\left(s\right)\right)-y\left(s\right){\phi }_{11}^{\prime }\left(x\left(s\right)\right)\right)ds\hfill \\ \hfill & ={\int }_0^{T}x\left(s\right)\left(f_1^{\prime }\left(x\left(s\right)\right)+\frac{1}{a}{\phi }_{11}\left(x\left(s\right)\right)f_1\left(x\left(s\right)\right)\right)ds\hfill \\ \hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-{\int }_0^T\frac{1}{a}{\phi }_{11}\left(x\left(s\right)\right)x^{\prime }\left(s\right)ds\hfill \\ \hfill & ={\int }_0^{T}x\left(s\right)\left(f_1^{\prime }\left(x\left(s\right)\right)+\frac{1}{a}{\phi }_{11}\left(x\left(s\right)\right)f_1\left(x\left(s\right)\right)\right)ds<0.\hfill \end{array}$$ [18]

Here, we used that ${\phi }_{11}^{\prime }=-\frac{{\phi }_{11}^2}{a}$ and $xy{\phi }_{11}\left(x\right)=x{f}_1\left(x\right)-x^{\prime }$. Moreover, the last integral in the second line equals $\mathrm{\Phi }\left(x\left(T\right)\right)-\mathrm{\Phi }\left(x\left(0\right)\right)$ for a $\mathrm{\Phi }$ satisfying ${\mathrm{\Phi }}^{\prime }={\phi }_{11}$ and hence is 0 by the periodicity of $x$.

Based on the discussion in this section for Eq. 1, we are going to identify conditions under which the solution $\mathrm{\Gamma }\left(t\right)=\left(x\left(t\right),y\left(t\right),\mathrm{0,0}\right)$ is an asymptotically stable limit cycle for Eq. 9. To be more precise, we show strong Lyapunov stability for $\mathrm{\Gamma }\left(t\right)$ by looking at it as a perturbation of the asymptotically stable limit cycle $\gamma \left(t\right)$ for Eq. 1. We follow the same strategy as is used for proving the stability of equilibrium points for a nonlinear system, i.e., one studies first the stability of the linearized system and then uses the variation of parameter formula and Floquet-type arguments to conclude for the nonlinear system.

We have the following result.

Lemma 5.3

Assume $\sigma =ak/c_1>1$ and $b_{11}+2\widehat{x}<K$. Then, the limit cycle $\mathrm{\Gamma }\left(t\right)=\left(x\left(t\right),y\left(t\right),\mathrm{0,0}\right)$ will be asymptotically stable if

$$\begin{array}{ll}\hfill & {\int }_0^T\frac{1}{b_{21}+x\left(s\right)}\left(1-\frac{x\left(s\right)}{K}\right)\left(r_2\left(b_{21}+x\left(s\right)\right)\right.\hfill \\ \hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\left.-r_1\left(b_{11}+x\left(s\right)\right)\right)ds<0,\hfill \end{array}$$

and

$${\int }_0^T\frac{ak-c_2}{b_{12}+x\left(s\right)}\left(x\left(s\right)-\frac{c_2{b}_{12}}{ak-c_2}\right)ds<0.$$

The proof is given in SI Appendix. We then show for which choices of parameters we can have an ESS. By Definition 3.2 and the discussion in this section, we have the following conditions.

Theorem 5.4 (Conditions for ESS—The Limit-Cycle Case)

Assume Eq. 12. A constant strategy $\left(\widehat{\alpha },\widehat{\beta }\right)$ is an ESS for Eq. 14 if

• (i)$\sigma =\frac{ak}{c\left(\widehat{\beta }\right)}>1$,
• (ii)$b\left(\widehat{\alpha },\widehat{\beta }\right)+2\widehat{x}<K$,
• (iii)the function

$$\begin{array}{ll}\hfill e_1\left({\alpha }_2\right)=\frac{1}{T}{\int }_0^T& \frac{1}{b\left({\alpha }_2,\widehat{\beta }\right)+x\left(s\right)}\left(1-\frac{x\left(s\right)}{K}\right)\hfill \\ \hfill & \times \left(r\left({\alpha }_2\right)\left(b\left({\alpha }_2,\widehat{\beta }\right)+x\left(s\right)\right)\right.\hfill \\ \hfill & \hspace{1em}\hspace{1em}\left.-r\left(\widehat{\alpha }\right)\left(b\left(\widehat{\alpha },\widehat{\beta }\right)+x\left(s\right)\right)\right)ds\hfill \end{array}$$

has a strict local maximum at the point ${\alpha }_2=\widehat{\alpha }$, and at the same time

• (iv)the function

$$e_2\left({\beta }_2\right)=\frac{1}{T}{\int }_0^T\frac{ak-c\left({\beta }_2\right)}{b\left(\widehat{\alpha },{\beta }_2\right)+x\left(s\right)}\left(x\left(s\right)-\frac{c\left({\beta }_2\right)b\left(\widehat{\alpha },{\beta }_2\right)}{ak-c\left({\beta }_2\right)}\right)ds$$

has a strict local maximum at the point ${\beta }_2=\widehat{\beta }$.

Proof:

The proof follows from Lemma 5.3 and the fact that $e_1\left(\widehat{\alpha }\right)=0=e_1\left(\widehat{\beta }\right)$ by definition.

With this result, we have given conditions that will ensure the existence of an ESS in a periodically fluctuating population (exhibiting a limit cycle). This is an important contribution of this paper. With empirically derived functions $r\left(\alpha \right),c\left(\beta \right),b\left(\alpha ,\beta \right)$, we can now give the ESS values of $\alpha$ and $\beta$. It is, however, beyond the current contribution to do so.

6. Red Queen Type of Continued Evolution or Stasis

Our analysis is related to the issue of Red Queen type of continued evolutionary evolution of stasis (cf. Van Valen) (49); see also Schaffer and Rosenzweig (50) and Rosenzweig et al. (51). In Example 4.3, we have assumed that both the predator and the prey have equal influence on the determination of the overall half-saturation parameter $b$. For this system, we find that evolution will lead to an ESS without evolutionarily fluctuating dynamics. If the two interacting species (the predator and prey) had asymmetric influence on the overall half-saturation constant, this might not be the case [see, e.g., Nordbotten and Stenseth (52)]. A further analysis of this would indeed be worthwhile. The contributions by Dercole and coworkers (44, 45), Doebeli (46), and Cortez and Weitz (26) are all important stepping stones in such further analysis.

7. Discussion

We establish conditions that guarantee stability of equilibrium solutions of the extended ecological model. This model has four equations, and, for periodic solutions, such a result appears to be previously undescribed. We prove full Lyapunov stability of the periodic solutions. The proof relies on an adaptation of classical linearization arguments, variation of parameter formulas, as well as Floquet-type analysis. The approach of Reed and Stenseth in ref. 31 to evolutionary games and ESS is extended to include fluctuating populations. ESSs are then shown to occur at values of the evolutionary strategies that are local optima of certain functions of the model parameters. We identify these functions and express them in a similar way for both stable and fluctuating populations. A concrete example to illustrate our results is given. Mathematically, these results are obtained from the abovementioned stability results, and our characterizations of ESSs seem to be previously undescribed and could be of independent interest in game theory.

The concept of ESSs was developed with a stable ecological setting in mind. However, typically ecological systems vary in time, often with more or less periodic fluctuations. Predator–prey systems are such examples. If we are to link ecology and evolution, we must allow for varying ecological population fluctuations. With this contribution, we have extended the ESS concept to be applicable for periodically fluctuating ecological systems.

The application of ESS to ecology is similar to the “adaptive dynamics” approach in that it models the time evolution of populations. See Dieckmann et al. (53); see also Kang and Fewell (54). The “adaptive dynamics” approach focuses on how strategies evolved under changing ecological conditions. The ESS approach, on the other hand, focuses on finding the unbeatable fixed strategy within a population. To our knowledge, it has not been shown before that such unbeatable ESS strategies do exist when the ecological system exhibits a periodically fluctuating dynamics. With this contribution, we derive the mathematical conditions for such fixed ESSs to exist when the ecological system exhibits limit cycle-type dynamics.

By extending the approach presented by Reed and Stenseth (31), it furthermore becomes clear that evolution occurs through the ecological process of a variant type being able to invade an ecological system (or population) when it is at its stable state—be this a stable equilibrium or a limit cycle. Whenever such ecological invasion occurs (i.e., the wild type is not an ESS), evolution will occur—and the ecology of the system changes. This emphasizes the interlinkages between ecology and evolution: it is a matter of realizing both that “nothing in biology makes sense except in the light of evolution” (55) as well as “very little in evolution makes sense except in the light of ecology” (56).

Data Availability.

There are no data underlying this work.

Change History

November 18, 2021: The SI Appendix has been updated to coincide with a formal Correction.