Optimal annual routines: behaviour in the context of physiology and ecology

Abstract

Organisms in a seasonal environment often schedule activities in a regular way over the year. If we assume that such annual routines have been shaped by natural selection then life-history theory should provide a basis for explaining them. We argue that many life-history trade-offs are mediated by underlying physiological variables that act on various time scales. The dynamics of these variables often preclude considering one period of the year in isolation. In order to capture the essence of annual routines, and many life-history traits, a detailed model of changes in physiological state over the annual cycle is required. We outline a modelling approach based on suitable physiological and ecological state variables that can capture this underlying biology, and describe how models based on this approach can be used to generate a range of insights and predictions.

1. Introduction

Many organisms live in an environment in which important variables such as day length, temperature and food availability change over the year. There may also be significant variation in pathogens or predators (e.g. Nelson et al. 2002; Altizer et al. 2006). For example, Lank et al. (2003) suggest that the timing of autumn migration in sandpipers is influenced by the need to travel south before the southerly migration of peregrine falcons. Even in the tropics, where day length is constant (and temperature may be reasonably constant), there may be strong seasonal changes (Komdeur & Daan 2005, Shine & Brown 2008).

Organisms in a seasonal environment may follow an annual routine, i.e. they schedule activities in a regular way over the year. There are many interesting questions related to these annual routines, including the following:

• Why do some organisms breed once a year whereas others breed several times a year and yet others do not breed in some years?

• Should an organism have a definite breeding season or maintain reproductive function so that it can breed whenever appropriate conditions arise?

• What determines whether it is worthwhile for an organism to migrate?

• What determines whether it is worthwhile for an organism to hibernate?

• Should organisms fund breeding from stored resources (capital breeding) or from resources obtained while reproducing (income breeding)?

• Why should there be a correlation between breeding date and number of offspring produced?

• Why is there a trend for birds to lay larger clutches with increasing latitude?

• Do we expect energy expenditure to be higher in winter or summer?

• How do we expect immune function to vary over the annual cycle?

If we assume that annual routines have been shaped by natural selection then life-history theory should provide a basis for explaining them and answering these questions. Life-history theory attempts to understand the evolutionary processes that account for the relationship between the environment and the way in which an organism schedules its activities. Most models in this area only consider one decision a year and usually model decisions at an abstract level (e.g. avoid predators, grow, reproduce) without reference to the underlying physiological mechanisms that achieve these ends. As they stand, such models are not adequate to analyse the details of an organism's annual routine. Since they only consider a single decision each year, they are obviously unable to analyse the details of how activities are organized over the year. We also have a fundamental objection to such models. As McNamara & Houston (1996) point out, many life-history trade-offs are mediated by an organism's physiology; for further discussion see Zera et al. (1998), Zera & Harshman (2001) and Speakman (2008). Traditional models that only census the population once a year usually do not even include physiological state (other than age). But even if they were to do so, the yearly time scale would mean that crucial changes in physiology could not be represented. In order to capture the essence of annual routines, and many life-history traits, we argue that a detailed model of changes in state over the annual cycle is required.

In this paper, we argue that a full understanding of behaviour requires knowledge of both the physiological processes that mediate life-history trade-offs and the ecological setting in which the organism evolved. Our aim is not only to highlight the importance of these factors, but also to outline a modelling approach based on suitable state variables that can capture the underlying biology. We begin (in §§2 and 3) by reviewing physiological and environmental state variables and the time scales on which they change. In §4, we consider various biological contexts to illustrate the choice of state variables that might be appropriate in a formal model. One of our contentions is that it is often impossible to consider one time of year in isolation owing to the time scale over which physiological effects last. This issue is discussed in §5. Having argued for a holistic approach, §§6 and 7 present a framework that can be used to model optimal routines over the entire annual cycle. The sorts of predictions that the models can generate are given in §8. Finally, in §9 we discuss general issues and suggest possible further applications of the framework.

2. Physiological state variables

Activities have consequences, but the time scale over which these consequences manifest themselves vary from immediate, to relatively short term to long term. Effects that are not immediate are often mediated through physiological state variables. The following examples of state variables and the time scales on which they change give an idea of the diversity and pervasiveness of state-dependent effects.

(a) Energy reserves

If an animal leaves cover to look for food, one immediate consequence might be that it is killed by a predator. If it is not killed then it obtains energy from the food it finds, and as a consequence its energy reserves increase. Energy reserves are an example of a physiological state variable. The increase in energy reserves can affect several aspects of performance and behaviour in the future. For example, the resultant increase in body mass may make a small bird less able to escape from predators (Hedenstrom 1992). The extra fat may insulate the animal. Since the animal is more likely to survive a lean period, it may consequently change its behaviour, reducing the risks that it is prepared to take while foraging (Witter & Cuthill 1993; Welton & Houston 2001). The resulting reduced energy gain will ultimately negate the original energy gain. Thus for a small animal (e.g. shrew, small bird) the consequences of energy gain are likely to be short term; although an increase in current energy reserves improves the animal's chances of surviving for the next few days, if it does survive then reserves at this future time will more or less be independent of current reserves (McNamara & Houston 1982). In contrast, a significant increase in reserves in a large animal may have longer-term consequences, especially if behaviour does not change to counter its effect. For example, a female polar bear (Ursus maritimus) in her den with cubs goes for months without food. During this time mass may drop by 40% (Atkinson & Ramsay 1995). An increase in her reserves at the beginning of this period is likely to affect her reserves significantly at the end of the period.

Even for a small animal, the effect of a change in reserves may not be short term if other state variables are influenced by the change. For example, if increased energy allows an animal to increase its position in a dominance hierarchy, a short term increase in energy can have a long-term effect on social standing. In addition, some differences in reserves may persist owing to the reinforcing behaviour of population members (e.g. Rands et al. 2003).

Some animals hoard food items in the environment, in which case the amount of food in the hoard is a relevant state variable. Models of these systems typically incorporate two energy variables, fat level and level of stored food (e.g. McNamara et al. 1990; Brodin & Clark 1997; Pravosudov & Lucas 2000).

(b) Protein and nutrient stores

There is more to foraging than gaining energy. In many contexts, energy and protein may be important. For example, Piersma & Jukema (2002) found that in autumn the mass gain of golden plovers (Pluvialis apricaria) consists of fat, whereas in spring protein is deposited. Noren & Mangel (2004) model the effects of body mass and protein mass on survival during the postweaning fast of northern elephant seal (Mirounga angustirostris Gill) pups. Simpson & Raubenheimer (2000) review the influence of the need for carbohydrate and protein on the foraging behaviour of locusts. Female birds may be limited by the calcium that is required for the production of eggs, e.g. Tilgar et al. (2002). For the importance of protein and calcium in mammalian reproduction, see Speakman (2008).

(c) Feather quality

Feather quality is an example of a physiological state variable with longer-term effects than energy reserves for a small bird. The way in which behaviour at one time of year can have consequences for this state variable at other times of the year is illustrated by the work of Nilsson & Svensson (1996). They removed the clutch from some pairs of blue tits (Parus caeruleus). Most of the females laid a new clutch of eggs, and as a result the pair fed their young later in the season than would have been the case without the removal of their clutch. Nilsson & Svensson found that birds from these pairs sometimes had to expend more energy on thermoregulation in the winter after the breeding season than birds from pairs whose clutch was not removed. Nilsson & Svensson also found that females from the experimental group laid their eggs later in the next breeding season than females from the control group. Nilsson & Svensson suggest that these long-term effects of the experimental manipulation may be the result of the late breeding following the clutch removal interfering with the moult that occurs after breeding. Since the rate of moult in starlings (Sturnus vulgaris) influences the quality of the new feathers (Dawson et al. 2000), the manipulation may have resulted in the new feathers being of poorer quality than usual, with thermoregulatory costs in winter being raised because the new feathers provided less insulation.

(d) State of reproductive system

Many organisms do not maintain a fully functional reproductive system throughout the year (Murton & Westwood 1977; Clarke 1981). For such organisms, the state of the reproductive system is an obvious state variable. It is particularly important to include such a variable when it takes a substantial time to become ready to breed or there are considerable costs associated with maintaining reproductive condition.

There are complex interactions between the reproductive system and the environment that involve state variables other than the state of reproductive system (see Hahn & MacDougall-Shackleton 2008; Paul et al. 2008; Wikelski et al. 2008). Many studies of behaviour over the annual cycle have shown that cues such as short days trigger a particular phase of the life cycle (e.g. Gwinner 1986). This presumably involves some long-term change in some state variable(s).

(e) Hard work and damage

If instead of being killed by a predator the animal is injured as a result of an attack, the consequences may be long lasting. In this case, we might regard some measure of the level of damage as a state variable. This damage may be reduced as a result of repair, but the process may be a slow one, and some aspects may be irreparable.

Breeding often involves a prolonged period of hard work with long-term consequences. For example, Daan et al. (1996) found that increasing the level of effort during reproduction in the kestrel (Falco tinnunculus) resulted in an increase in mortality in the following winter. Similar effects have been found in other species. Wernham & Bryant (1998) studied the reproductive behaviour of puffins (Fratercula arctica). The young in the experimental group were given extra fish. Their parents reduced the number of feeding visits, but owing to the extra fish, the chicks increased in mass. There was no effect of the treatment on the parents' lipid reserves or the probability that the parents returned to breed, but parents in experimental group had a higher success in fledging their young in the year after the treatment. So there is a long-term effect that does not appear to be mediated by reserves.

Energy reserves change too rapidly in response to expenditure to be an explanation for the effect observed by Daan et al. (1996), and do not seem to be directly implicated in the results of Wernham & Bryant (1998). Thus, there could be a physiological state variable associated with body condition that is driven down as a result of hard work and takes a long time to raise once hard work stops. In the case studied by Daan et al. (1996), the variable increases the chances of death when it is low. A variable that reflects the ability of the immune system to fight disease might be a suitable candidate.

Alonso-Alvarez et al. (2004) manipulated brood size in the zebra finch (Taeniopygia guttata) and found that an increase in reproductive effort was associated with a decrease in antioxidant defences. This presumably could affect future levels of oxidative damage—another state variable with possible long-term consequences.

Other activities, such as escape from a predator, can involve a short period of work that is more intense than parental effort. The length of time that a level of energy expenditure can be maintained, and the consequences for the animal afterwards, depend on a variety of physiological state variables. These variables act on a huge range of time scales. At the shortest, intense activity depletes creatine phosphate very quickly. Muscle glycogen takes longer to deplete and fat reserves still longer. Of the various deleterious effects, lactic acid build up is cleared within hours, microscopic muscle tears heal in a time scale of days, while the immune suppression and any resultant increase in parasite load that can accompany very long periods of hard work can act on time scales of months.

(f) Ability and other long-term variables

Differences between individuals in a population are represented by different values of their state variables. Some differences, such as in foraging ability or size, might be either temporary or persist over the lifetime of organisms.

Daunt et al. (2006) investigate the relationship between foraging during winter and the timing of breeding in the shag (Phalacrocorax aristotelis). They found that birds that devoted less time to foraging in late winter subsequently breed earlier than birds that foraged for longer. Daunt et al. explain this in terms of differences in foraging ability—good foragers can get into condition without having to devote much time to foraging.

Gender is an obvious difference between individuals and is an essential state variable in some models. In addition to morphological differences related to reproduction, males and females may differ in morphological traits such as body size. They may also differ in behaviour. For example, male and female birds may have different responses to cues (Ball & Ketterson 2008) or different patterns of migration (e.g. Ketterson & Nolan 1979; Kjellen 1992; Mueller et al. 2000).

3. Environmental state variables

In addition to the internal state variables that we have discussed, animals can be characterized by various external or environmental variables.

(a) Extended phenotype

Position in a dominance hierarchy or quality of territory are examples of state variables that apply to particular individuals in a population and can be thought of a part of their extended phenotype. Daan et al. (1990) show that the territories of kestrels differ in terms of the yield from hunting. These differences are able to predict individual differences in clutch size and lay date. Part (2001a,b) shows that territory quality in the wheatear (Oenanthe oenanthe) is related to vegetation height. In oystercatchers (Haematopus ostralegus) that breed in coastal areas, Ens et al. (1992) found that territories are either of good quality (adjacent to the feeding area) or poor quality (far from the feeding area).

(b) Abiotic variables that are relevant for the whole population

As we have said above, variables such as food abundance, temperature and day length may change over the year. All of these variables can be important state variables for members of a population. Furthermore, there may be interannual differences in seasonal effects. For example, in the cliff swallow (Petrochelidon pyrrhonota) years may differ in the probability of cold weather at the start of the breeding season (Brown & Brown 2000). In such cases, year type (e.g. whether it is a bad year for early breeding) could be a state variable.

(c) Biotic variables that are relevant for the whole population

For many animals the number and type of conspecifics present are state variables that are important determinants of life history.

4. Incorporating physiology and ecology into models

The physiological variables that mediate life-history trade-offs, the time scale at which they change, and the interaction between physiology and ecology are crucial components of models that try to understand activities in adaptive terms. We illustrate this with a series of examples.

(a) Migration

As a first example, assume that we are interested in whether or not a species should migrate. We might start to address this question by looking at how food availability changes over the year at various locations. (To keep the problem simple we could start with just two locations.) We could then ask whether staying in one location throughout the year gives the animal a higher energetic gain than moving between locations. We could include the energetic cost of moving between locations, but there are other costs, e.g. the risk of predation, that might be relevant.

A satisfactory account of whether an animal should migrate would need to allow an animal to change its level of energy reserves (either on its body or in the environment) over the year. This could be important for two reasons. An animal that does not migrate might be able to improve its chances of survival during winter by storing energy and an animal that migrates might need to build up fuel before its journey. The associated changes in foraging behaviour are likely to influence the animal's level of predation. In addition, changes in the amount of energy stored on the body can be expected to change the animal's rate of energy expenditure and its risk of predation.

Piersma (1997) suggests that long-distance migrants use habitats where parasites are rare, and so can afford to have low investment in immunocompetence and hence high growth rates (see also Mendes et al. 2005; Buehler & Piersma 2008). Parasite load may be a very important state variable for many animals, although less considered in formal life-history models. Since parasites can persist, avoiding parasites in one part of the annual cycle may have knock on consequences for other parts of the cycle and even for later years.

The most efficient way of storing energy in the body is in the form of fat. But protein stores in muscle can act as an energy source. They are also a potential source of amino acid for protein synthesis. For example, during the acute phase response of an infection, animals reduce food intake and become more reliant on amino acid stores in muscle, leading to muscle wasting. In our migration model we might therefore distinguish between storage of fat and protein stores, particularly if considering the response to infection over the annual cycle.

Many migrant birds have large changes in the size of flight muscles over the annual cycle. Organ size may also vary—see Piersma & Lindström (1997) for a review. For example, basal metabolic rate (BMR) and digestive organs such as the gut, liver and gizzard change over the year in the knot (Calidris canutus; Piersma et al. 1995; Piersma et al. 1996; see also Buehler & Piersma 2008). It seems reasonably clear why many of these changes occur. Large flight muscles are needed during the migration phase, but maintaining them on the wintering grounds is costly. Conversely, a larger gizzard enables the knot to process hard-shell molluscs at a higher rate, and consequently gizzard size can be crucial in terms of over winter survival. A large gizzard and gut are, however, costly to transport during the migration phase (van Gils et al. 2003). Knots therefore have a relatively small gizzard during autumn migration, but then increase gizzard size on arrival at the wintering grounds. The changes in gizzard size over the year can be understood in terms of the associated costs and benefits (van Gils et al. 2003). Any model that incorporated fat reserves, flight muscle mass and gizzard size as state variables would need to spell out how levels of these variables affect foraging ability, the ability to withstand periods of low food, predation risk and migration ability. In addition, the resources needed to increase size of muscle or gizzard would need to be specified, as would any benefits of catabolism of these organs in terms the release of usable energy or amino acids.

As Barta et al. (2008) emphasize, there is a considerable variation in the moult strategies of migratory birds, with some species moulting post breeding, others moulting on the wintering grounds, with other patterns also possible. To model moult within the context of the annual cycle, feather condition is a necessary state variable. Barta et al. include the condition of flight primaries and energy reserves, as well as location, as state variables in their annual routine model, and are able to predict how food availability should influence the pattern of moult and migration.

(b) Timing of breeding

The timing of breeding provides another example in which behaviour depends on both ecology and physiology. Lack (1950) saw food for young birds as the principal factor that determines the timing of breeding:

…the breeding season has been adjusted by natural selection to that season of the year when the food which the bird is adapted to collecting is sufficiently abundant for it to raise a family.

This view is simple and appealing, but it ignores the fact that the value of young (as indicated by their probability of surviving to independence) may decline as the date at which the young are born increases (Daan et al. 1989). This finding invites the question ‘why do not all members of a population breed as early as possible?’ A possible limitation on how early breeding may occur in the year is the fact that females need time to get into the right condition to produce young (Perrins 1970). For example, a female bird needs to store nutrients and calcium in order to produce eggs. (For a discussion of the costs of egg production and incubation see Nager et al. (2001) and Visser & Lessells (2001)). The time required to produce eggs results in a trade-off: delaying breeding has the advantage that a larger clutch can be laid but incurs a penalty because the young hatch later in the season. Drent & Daan (1980) argued that individual differences in quality could result in ‘good’ females laying large clutches early in the season and ‘poor’ females laying small clutches later in the season. (Empirical work on the effects of time and quality on seasonal differences in reproductive success is reviewed by Verhulst & Nilsson (2008)). The trade-off between quality and breeding date is modelled by Rowe et al. (1994). It is important to note that their framework assumes a relationship between the value of young and their birth date. The framework also assumes the initial value of the state of females. It is thus unable to explain the initial distribution of female states and the time at which resources for reproduction start to be built up. In contrast, the framework for modelling annual routines that we present below handles both these aspects.

Thomas et al. (2001) compare two populations of blue tits (Parus caeruleus) that differ in terms of the relationship between the timing of breeding and the peak in the availability of food. In the population in which the timing of peak offspring's need for food matches the peak in food, parents work at approximately 3–4 times BMR, whereas if there is a mismatch in timing, parents are forced to work at levels that cannot safely be sustained. Thus the correct timing of egg laying is crucial. However, there is a lag between egg laying and the period of peak offspring need in this species. Thus parents make their decision on the timing of laying before the timing of peak food is known for certain, and must use cues in deciding when to lay. Cues that could potentially give information about future food abundance include day length, temperature and the amount of foliage. Thus in modelling the optimal timing of egg laying, both physiological state variables representing resource levels and/or condition, and ecological state variables, such as day length and current temperature, could be included in a formal model. A bird's laying strategy would then be a rule that specifies when to lay as a function of all of these state variables and time of year. Under such a rule, time of egg laying would not be fixed, but responsive to these variables. An optimal strategy would then be an optimal plastic response. The degree of responsiveness predicted and the cues on which decisions are based (e.g. the extent to which behaviour depends on day length as opposed to food) would, however, depend on correlations in the environment in which the population evolved. In particular, seasonal trends, the variability of conditions and the correlation between current and future conditions, determine the degree to which a bird should respond to current conditions. These correlations are key components of any model to predict the optimal plastic response.

The plastic response that a bird uses to time egg laying determines the way that the bird responds to climate change, e.g. Nussey et al. (2005). Visser et al. (1998) studied the timing of reproduction in the great tit (Parus major) at a site in the Netherlands. They found that peak abundance of caterpillars is now earlier in the year, but birds do not lay their eggs earlier. The inference is that with climate change the cues that used to predict the best time to breed no longer do so.

(c) Biennial breeding

If the environment is not very seasonal, then breeding may be observed at any time of the year. For example, in the case of the Australian sea lion (Neophoca cinerea), breeding takes longer than a year and breeding is not synchronized across populations (Higgins 1993; Gales et al. 1994).

If the environment is seasonal, reproduction tends to occur at specific time of year, but members of a population may not breed every year. In some species this is because the breeding cycle is longer than a year. For example, the cycle of a successful King penguin (Aptenodytes patagonicus) is longer than 14 months (Olsson 1996). Parents feed chicks until the start of winter. During winter the availability of food is low and the chicks fast, but they are fed again in the spring (Olsson 1996). At South Georgia, Olsson found that some pairs started to breed in early summer whereas other pairs started later in summer. Start date was determined by previous success. If a breeding attempt is successful, the pair finish at about midsummer and then attempt to breed again. Such late attempts typically fail, and the parents are then able to try again at the start of the next summer (Olsson 1996).

Some species living in a seasonal environment tend to alternate, breeding every other year (known as biennial breeding) even though the breeding cycle lasts less than 1 year. In these cases, changes in the state of the animal seem to be responsible for the alternation. Energy reserves are an obvious state variable that might mediate these effects. Species that depend upon stored energy to fuel reproduction are known as capital breeders. Many snake species are capital breeders (Seigel & Ford 1987; Shine 2003), as are most phocids (e.g. Schulz & Bowen 2005). In capital breeders, we can expect reproduction to be initiated when energy reserves exceed some critical level. If, having bred once, a female cannot get to this critical level by the breeding season in the next year then she defers breeding until the following year. If there is a cost (in terms of energy or survival) associated with reproduction, the threshold for breeding is likely to be high and as a result animals do not reach it the year after breeding. Bull & Shine (1979) note that species that miss a reproductive opportunity tend to have high costs associated with reproduction.

The black-browed albatross (Diomedea melanophrys) and the grey-headed albatross (Diomedea chrysostoma) have breeding cycles that are shorter than 1 year, and could in principle breed every year. These species are similar in terms of body mass and breeding site, but differ in their pattern of breeding. The black-browed albatross breeds every year, whereas the grey-headed albatross does not breed in the year following a successful breeding attempt (Prince et al. 1994; Tickell 2000). If an attempt is not successful, the pair may breed in the following year but the later the time at which the failure occurs, the lower is their probability of breeding next year. These data suggest that loss of condition during breeding does not prevent annual breeding in the black-browed albatross but does in the grey-headed albatross. At present, it is not clear what constitutes condition, but it seems unlikely to be just energy reserves.

Jouventin & Dobson (2002) use data from 12 species of albatross to investigate the importance of physiology (body size) and ecology (foraging over large distances). They show that large body size and large foraging distances are both associated with biennial breeding. Jouventin & Dobson (2002) explain this in terms of a correlation between reproductive rate and body size. We would prefer an account that related body size and behaviour to changes in crucial state variables. In our approach any correlation would emerge rather than being assumed.

(d) Year-to-year variation in environmental conditions

There are species in which failure to breed seems to be the result of a low availability of resources, and non-annual breeding can be attributed to year-to-year variations in the availability of food. For example, Ural owls (Strix uralensis) feed on voles, and the abundance of voles follows a 3 year cycle. Many more owls breed in years of high vole abundance than in years of low vole abundance (Pietiainen 1989). In addition, clutch size is larger when vole abundance is high.

For species such as the Ural owl, it is not realistic to assume that seasonal trends are the same in each year. To allow for inter-annual variation, it is necessary to introduce state variables that represent the current state of the environment into models. To build a model we specify a range of year types, their frequency of occurrence, and the correlations between conditions in the first year and the next year. As a simple case, the overall food availability might be good, intermediate or bad, and current food availability would be the appropriate environmental state variable. If the example concerned breeding in the tropics, year type might be the time at which the rainy season starts. If the example concerned breeding in temperate latitudes, year type might specify the time of peak food abundance. In a more complex variant of this, we might hypothesize that the timing of peak food abundance depended in some way on the profile of temperature in the preceding weeks. We would then describe the change of temperature during spring as a stochastic process. In this scenario, current temperature would be a state variable in the model.

Once environmental state variables have been defined, we need to specify what an organism ‘knows’ about its environment. In other words, what variables can be used as the basis for its decisions? These crucially affect the organism's ability to predict future conditions. For example, consider possible predictors of future food availability. We can distinguish several possible ways in which decisions are made. At one extreme, decisions might be based only on physiological state variables and time of year. In the case of an animal that stores fat on its body, current fat reserves might provide information about past food availability and hence (given environmental correlations) future availability. At an intermediate level, decisions might also be based on the current availability of food. Food availability, together with current fat levels might give a better predictor about future food than fat levels alone. Finally, at the other extreme the animal might also know the current year type (e.g. whether it is an El Niño year) and base its decision on this information as well. The cues that are used by a species could be predicted on the basis of the form of year-to-year variation (cf. Wingfield et al. 1992).

We have suggested that in capital breeders, we can expect reproduction to be initiated when energy reserves exceed some critical level. The water python (Liasis fuscus) feeds on dusky rats and the abundance of rats fluctuates from year to year. More female pythons breed in years of high rat abundance (Shine & Madsen 1997). This is the same sort of pattern that is seen in Ural Owls. Madsen & Shine (1999) measured the body length and mass of female pythons. Body condition (mass relative to length) was higher in reproducing than non-reproducing females in all the 8 years of study. Furthermore, the critical level of condition above which a female reproduces changes from year to year. In years when rats are rare, females will reproduce in a much lower condition than is required in a year when rats are abundant. In this example, a physiological state variable (body condition) and an environmental state variable (current rat abundance) are both important, and interact to determine whether a female reproduces. The observed effect is not intuitively obvious, and highlights the need for a formal model to understand the circumstances in which it is predicted.

5. Considering effects in isolation

Science often makes progress by isolating certain effects. In this section, we discuss attempts to consider a time of year in isolation from the rest of the year. We are concerned with when such reductionism is useful and its limitations. We focus on a time interval, designating the start of this interval as time 0 and the end of the interval as time T. For simplicity, we assume that no reproduction occurs during the interval, so as to avoid the complication of having to consider the fate of any young.

(a) Coupling through initial and final state

A strategy is a rule specifying, for every possible time within the focal time interval and every possible state of an animal, what action to take in that state. The strategy employed determines the probability the organism survives until time T and its state at this time given it does survive. Since the value of being in a particular state at time T depends on the organism's biology after T, the optimal strategy during the interval depends on the biology after T (McNamara & Houston 1986).

The optimal strategy over the focal time interval does not directly depend on the past. In particular it is not affected by the animal's biology before time 0. We need, however, to distinguish a strategy from the behaviour observed under the strategy. Behaviour depends on both the strategy employed and the state of the animal. Although the optimal strategy during the focal interval does not depend on biology before time 0, the state of animal does. Thus, observed behaviour during the focal interval is potentially influenced by events before the interval.

We illustrate these concepts in two contexts, one involving the level of energy reserves over winter and the other involving behaviour during a migratory journey.

(b) Fat levels over winter

By increasing its fat level, a small bird increases its store of energy and hence improves its chances of avoiding starvation during times in which it is unable to forage (night and periods of bad weather). It is probable, however, that an increase in fat (all else being equal) will increase a bird's probability of being killed by a predator, either as a result of a direct increase in the probability of being killed if a predator attacks or as a consequence of spending more time foraging to maintain reserves at a high level (metabolic rate increases with fat level; McNamara & Houston 1990). Thus many small birds face a trade-off between the danger of starvation and that of predation.

A number of models, motivated by this trade-off, have been used to predict the optimal foraging strategy and target fat reserves of a small bird during winter (Lima 1986; McNamara & Houston 1990; Houston & McNamara 1993; Clark & Dukas 2000; Pravosudov & Lucas 2001; Welton et al. 2002). In the models, the level of fat reserves is the only physiological state variable. It is assumed that at each time interval of a day, the bird has a choice of foraging options that differ in net energy gain and predation risk with high gain being associated with high risk. For a given choice of food option, either metabolic expenditure or predation risk, or both, increase with increasing fat reserves. Behaviour is considered over a hypothetical winter that ends at a time T that is before the bird starts to breed in spring. The optimal strategy over winter maximizes the expected reproductive value of the bird at this time.

For these models, the optimal strategy on a given day in winter specifies how a bird's choice of food option should depend on its current energy reserves and time of day. Near the end of winter this strategy reflects both the importance of surviving until time T and how valuable are high energy reserves at this time. Value depends on the biology of the bird after the end of winter. If high reserves are essential for successful reproduction or migration immediately after winter ends, near the end of winter a bird with low reserves will risk death from predation in order to substantially increase reserves. In contrast, if the bird has plenty of opportunities to get food after time T before reproduction or migration, high reserves at T are not particularly valuable, and behaviour close to the end of winter will be focused mainly on survival until T.

We have argued that the effect of increased energy reserves for a small bird may be short term. This is certainly true in the above models. Consequently, the optimal strategy on a day well before the end of winter is essentially independent of the value of high energy reserves at the end of winter. Thus, at least in these models, optimal behaviour well before the end of winter is uncoupled from the biology of the bird after winter ends.

The optimal strategy of a bird over winter is not affected by the bird's biology in the preceding autumn. However, the bird's energy reserves, and hence behaviour, at the start of winter will reflect this biology. Nevertheless, because the effects of reserves are short term, provided that sufficient time has elapsed since the start of winter, subsequent reserves and behaviour will be independent of reserves at the start of winter (given the bird is still alive). Thus, at least in these models, optimal behaviour well after the start of winter is uncoupled from the biology of the bird before winter starts.

These conclusions about uncoupling depend crucially on the short-term effects of energy reserves. We would not expect them to hold for a larger animal whose reserves change on a longer time scale. They would also not hold if other condition variables with long time scales were important. For example, Houston & McNamara (1999) include both energy reserves (which can vary on a short time scale) and immunocompetence (which varies on a long time scale) as state variables in a model. In their model, increasing reserves of a ‘virtual’ animal in midwinter reduces subsequent foraging, resulting in improved immunocompetence in spring and hence earlier breeding.

(c) The migratory journey

On a migratory journey, birds may alternate between periods of flight and periods at stopover sites where they build up their fuel levels. Previous theoretical work on optimal migration has considered the journey in isolation from the rest of the year. Working with overall speed of the journey as a currency, some models find the optimal flight speed (e.g. Alerstam 1991; Hedenstrom & Alerstam 1995) whereas others find the optimal fuel load on departure (e.g. Lindström & Alerstam 1992; Hedenstrom & Alerstam 1997). For a review, see Hedenstro¨m (2008).

Problems associated with treating migration in isolation arise in the context of the time at which a bird arrives on the breeding grounds. Usually males arrive at the breeding grounds before females. In juncos, males achieve this by wintering nearer to the breeding grounds than the females (Ketterson & Nolan 1976). Males are bigger and so are better able to survive in the harsher conditions where they spend the winter. Females are smaller and go further south to spend the winter in regions that are less harsh. Although this pattern makes intuitive sense, it does not provide a complete account. Without a model of the annual cycle, it is not clear why males do not go as far south for the winter as females but leave the wintering grounds before them. Furthermore, the value to a male arriving at a given time also depends on when other males arrive. This frequency dependence means that arrival times need to be derived from a game-theoretic model (see Kokko 1999). The frequency dependence also makes it difficult to consider a migratory journey in isolation because the arrival time of males will depend on the time at which they leave the wintering grounds and hence will depend on conditions and behaviour during winter.

The work of Norris et al. (2004b) shows that there may be knock-on effects from one time of year to another, again demonstrating that it can be difficult to consider a single migratory journey in isolation. Norris et al. used stable hydrogen isotopes to study the relationship between reproductive effort and moult in the American redstart (Setophaga ruticilla). Some male birds moult on the breeding grounds, whereas others moulted once they had moved south of the breeding area on their migratory journey to the wintering area. The probability of the latter behaviour was high if the male's reproductive effort was high. In addition, males that moulted south of the breeding area produced feathers with less orange–red coloration. Norris et al. conclude that a high level of reproductive effort can have consequences for both the degree to which moult overlaps with migration and the colour of the new feathers. In turn, feather colour may have an effect on breeding success in the following year. The reproductive success of male redstarts depends on both age and arrival date, with males that arrive earlier being more likely to mate (Lozano et al. 1996). Norris et al. (2004a) show that the quality of a male's wintering habitat influences the date at which the breeding area is reached, and this date influences the success of the breeding attempt. This example shows that reproductive behaviour can influence various aspects of subsequent performance.

6. A framework for modelling annual routines

Models that concentrate on part of the year suffer from various limitations. They typically assume that one activity, rather than some other activity is performed at this time. In addition, the state of an animal at the beginning of the period is assumed. As we have seen, it is not reasonable to assume that activities can be considered in isolation. Activities such as breeding preclude some other activities (e.g. migrating) and can have long-term effects on the organism's condition. For example, the effects of hard work can act at a range of time scales, from short-term changes in energy reserves and other variables, to long-term changes as a result of processes such as oxidative damage, affecting the ability of the organism to perform activities at various times in the future.

A few models have been constructed for activities over the whole year. Nevertheless, most of them have kept the timing of key events (e.g. breeding) fixed. They therefore constrain an animal's pattern of activity.

We need an approach that avoids all these limitations. An adequate approach should allow an animal to perform any of its activities at any time. In doing so it allows the animal to make the best use of all activities over the year, taking into account how these activities interact. In other words we need a holistic approach.

There are difficulties with carrying out such an approach. Performing an activity now may prevent another activity being performed, so that we need to consider when to perform this other activity as well. Furthermore, changes in state as a result of performing an activity can influence possible future options. An additional complication arises because the time of breeding is not specified in advance. As a consequence it is necessary to specify how the value of young depends on the time of year at which they are produced. A related problem concerns the state of the young at independence. If either the state or the behaviour of parents, or the time of year, affects the state of the young at independence, then the dependence of reproductive value of these young on their state must be specified.

We now present the framework for modelling optimal annual routines that was introduced by McNamara et al. (1998) and Houston & McNamara (1999). It can deal with all the above complications. In this framework, the timing of events such as reproduction is not assumed in advance; in fact the animal is free to breed at any time and consequently have multiple breeding attempts within a year. It is also free to overlap activities such as breeding and moult. The framework not only takes into account the time of production of young, but also their state at independence. As such it is capable of handling intergenerational effects such as maternal effects and the trade-off between number of young and their quality (McNamara & Houston 1996, 2006). This framework has the following ingredients.

(a) The environment

The standard approach in behavioural ecology is to predict an organism's behaviour from its environment, but very few models consider a seasonal environment. In our framework we include seasonal trends. Although seasonal factors such as pathogens or predators could co-evolve, we do not model this here. Instead we take seasonal trends in them to be a given part of the environment. There can be frequency- and density-dependent effects in the model so that, for example, we might allow food to depend on the number of animals in the population. Thus not all aspects of the environment are given, some are the results of conditions created by the activities of population members, and emerge from the models.

(b) State variables

Life-history trade-offs are often mediated by changes in state variables. We have attempted to show the importance of a variety of state variables. Depending on context, at least one of the following variables might be relevant and included in a model: fat reserves, muscle mass, other nutrient stores, level of immune function, parasite load, level of injury, level of oxidative damage, feather or fur condition, organ size, and geographical location.

(c) Actions and their consequences

The definition of an action is very broad. Possible actions include behaviours such as ‘initiate breeding’, ‘hibernate’ or ‘migrate’. They also can refer to physiological decisions. Examples include the decisions that determine muscle size, fat levels or size of digestive organs. The amount of a particular hormone to release can also be considered as an action. In some species, changing sex is a possible action—see Charnov (1982) for an account of the theory. The actions that are available to an animal at a decision epoch may depend on both its state and time of year. Sometimes an animal must choose a combination of more basic actions at a decision epoch. For example, it might have to decide whether to start breeding and simultaneously decide how hard to forage.

A model must specify the set of all possible actions and the short-term consequences of each action. For every decision epoch t and state at this decision epoch, the model specifies how the animal's action (or combination of actions) at time t determine the animal's state (or probability distribution of states) at decision epoch t+1. It is this determination of state dynamics that we refer to as the short-term consequences. One specific short-term consequence that is specified is the probability of survival until time t+1.

Note that in this formulation there is no mention of costs and benefits, just consequences. The consequences form the basis of life-history trade-offs within our modelling framework. If required, the fitness costs and benefits of actions can be inferred once an optimal strategy has been found, so that cost and benefits are outputs of the model not inputs (Houston & McNamara 1999).

In the formulation an action can have long-term consequences because its effect on state persists. These long-term consequences emerge from the state dynamics, rather than being assumed. For example, consider an animal that makes a daily decision about whether to defend its territory if this territory is challenged. Territorial defence has a mortality risk, but once the territory is lost it is difficult to obtain a new territory. Failing to defend the territory means that the animal survives, but loses its territory. For this animal one of its state variables will be binary; whether it has a territory or not. Suppose that on day t the animal has a territory and is challenged. If it does not defend the territory, with probability 1 its state on day t+1 is ‘no territory’. If it defends the territory its state on day t+1 is ‘territory’ with probability sp and is no territory with probability s(1-p). Here, s is the probability the animal survives the defence and p is the probability the defence is successful given that it survives. In this example, given that territories are difficult to obtain once lost, whether the animal has a territory in 1 year's time may depend strongly on its decision now. But this dependence need not be spelled out. It emerges from the state dynamics and a specification of whether the animal defends on future days.

Within the formalism we are describing, decisions about the allocation of energy or resources within the body count as actions. Examples where such decisions are important include allocation of resources to immune function and to repair of damage.

Disposable soma models of ageing (e.g. Abrams & Ludwig 1995; Cichon 1997; Cichon & Kozlowski 2000; Kirkwood & Austad 2000; Mangel 2001) are state-based models where the state represents some aspect of condition of the body and one action is the level of repair.

(d) Strategies

A strategy specifies, for each state and decision epoch during the year, the action (or combination of actions) chosen at this time. Note that the animal is free to allow its choice to depend on both state and time of year. However, under a given strategy the action chosen in a given state at a given time is the same each year. This does not mean that the animal chooses the same action on, say, 1st February, each year. This is because the animal is likely to be in different states at this time in different years. We discuss this further below in the context of behavioural routines.

The number of possible strategies is typically huge. For example, suppose that at each week of the year an animal has to decide between two actions. Then, even without any state variable there are 2⁵²=4.5×10¹⁵ strategies. For more complex state-dependent models the number of strategies is even bigger, and typically exceeds the number of atoms in the observable Universe.

(e) Modelling dependent young

In the modelling approach that we are describing, the state variable of an individual includes information about whether the individual has dependent young. If there are dependent young, the state of the individual also includes relevant information on the states of young. On becoming independent the individual's state variable changes to reflect the fact that dependent young are no longer present.

7. Fitness and optimal strategies

The ingredients above form the basis of the modelling approach. But to find an optimal strategy we need to specify two other entities. First, we need to specify how the reproductive value of young at independence is determined. Then we need to define an appropriate measure of fitness that is maximized. We now highlight two approaches to supplying these missing ingredients.

(a) Fixed background

Natural populations are subject to frequency- and density-dependent factors. This means there is a feedback between individuals within a population and the environment that they experience. For example, the number of offspring that a female produces might depend on the food abundance in the environment. But the food abundance is in part determined by numbers of offspring produced, since the more that are produced the greater the competition for food. Feedback also occurs owing to competition for other resources such as breeding sites or mates. Mortality risks may also be frequency- or density dependent. For example, the optimal level of immune defence will depend on the infection rate, which will depend on the level of immune defence of other population members.

A model that incorporates the sorts of feedback mentioned above has potential disadvantages. It necessarily includes much detail, such as the precise form of density dependence assumed. It may be hard to find evolutionarily stable strategies owing to the feedback. Furthermore, results obtained can be difficult to analyse and understand. Thus, when feedback is not the main focus of the investigation or central to it, a simpler approach may be warranted. To motivate one approach we note that even when there is feedback, at evolutionary stability each individual in the population maximizes fitness in the environment created by the actions of all population members. Thus, one approach is to take the environment as given, and focus on a single individual to find the best strategy for this individual within this environment. We now describe how to implement this simple approach.

A function for reproductive value at independence is assumed. Usually the form of this function will be motivated by data from study populations. For example, motivated by observations on bird populations (Daan et al. 1989), we might assume that the reproductive value of offspring at independence decreases with the time of independence during spring and summer. Of course, we would then still need to specify what the reproductive value would be if young were to become independent at time of year when breeding does not occur in real populations. This highlights a general difficulty with models; it is necessary to specify the consequences of every action, even if that action is never taken in practice. If the state of offspring could potentially differ at dependence, then the models would need to assume some function for the dependence of reproductive value on both state and time of independence.

The fitness criterion taken in this simple case would usually be the expected total reproductive value (at independence) of all offspring produced during the animal's lifetime (lifetime reproductive success, LRS). For a justification of this measure see Taylor et al. (1974) or McNamara (1993). We also outline the justification below.

We emphasize that fitness is a measure assigned to a strategy not an individual. A strategy is a rule for choosing actions as a function of state and time of year. Even when following the same rule, different individuals will be in different states (owing to, for example, good and bad luck foraging), will consequently behave differently, and will leave different numbers of offspring. Fitness is the average lifetime reproductive success of an individual using this rule.

For most problems, it is not practical to evaluate the fitness of every strategy since there are too many. Nevertheless there exist techniques that can find the strategy that maximizes fitness without exhaustive search. In the electronic supplementary material, we outline such a technique based on dynamic programming.

(b) Population feedback

The feedback between the individual level and the population level is a crucial ingredient in various problems. For example, if the model is concerned with mating and paternity, then modelling the whole population automatically ensures that all offspring have exactly one mother and one father. This consistency condition can be central to model predictions (Houston & McNamara 2002, 2005; Houston et al. 2005; Kokko et al. 2006). The dependence of clutch size on latitude is another issue where feedback is liable to be crucial. In any model in which the seasonality in the food supply changes, other aspects, such as the mean food supply, are also likely to change. The change in these other aspect cannot be arbitrary, since as seasonality is changed the overall food must change so that the population size is constant (J. M. McNamara & M. Wikelski unpublished data). To ensure this, we need to be explicit about the density dependence and hence the feedback in the population. Other examples in which frequency- and density dependence are likely to be important in modelling annual routines of migration have already been mentioned.

To incorporate feedback we need to build a model that explicitly considers the whole population. In particular, we need to explicitly model what happens to young after independence, allowing them to grow and mature and so become part of the adult population. Of course newly independent offspring will tend to have different state variables to older individuals. For example, if size is a state variable then young individuals will tend to be smaller at independence than older individuals. If foraging experience is a state variable, then young individuals will tend to be poorer foragers. However, in the approach we advocate, after independence offspring are treated just as other population member in the model, except their state at independence will reflect the fact that they are young/small/poor foragers, etc.

Once we explicitly model the whole population we need to seek an evolutionarily stable strategy (ESS) rather than an optimum. The principle behind an ESS in a life-history context is described by Metz et al. (1992). The idea of an ESS is that it is a strategy, which if adopted by almost all population members, cannot be invaded by a mutant strategy. More formally, we can describe the concept of an ESS using the language of game theory as follows. A strategy π is referred to as the resident strategy if almost all population members follow this strategy. Consider a rare mutant strategy π′ within a population in which almost all members follow strategy π. The fitness of this mutant strategy is a suitably defined rate at which mutant numbers are increasing over time. Since mutant fitness will depend on both mutant and resident strategies we denote it by W(π′,π). (Note that in this notation W(π,π) is the rate of increase of a mutant strategy that is the same as the resident strategy, and so is just the rate of growth of the whole population, which is zero owing to the density dependence.) A sufficient condition for a strategy π^* to be an ESS is then that

$$W({\pi }^{\prime },{\pi }^{*})<W({\pi }^{*},{\pi }^{*})\text{for}\text{all}{\pi }^{\prime }\ne {\pi }^{*}.$$ (7.1)

This states that when the resident strategy is π^* any rare mutant strategy different from π^* has a lower rate of increase then the resident population, and thus cannot invade. This condition can be restated as follows. Define the best response to the resident strategy π to be the mutant strategy B(π) with the greatest fitness within this resident population. That is

$$W({\pi }^{\prime },\pi )\le W(B(\pi ),\pi )\text{for}\text{all}{\pi }^{\prime }.$$ (7.2)

For simplicity, assume such a strategy is unique. Then a sufficient condition for π^* to be an ESS is that it is a best response to itself, i.e.

$$B({\pi }^{*})={\pi }^{*}.$$ (7.3)

We outline below how to solve this equation numerically in the case where environmental conditions faced by population members are the same at the same date on successive years. For this to hold, although there can be seasonal effects, there are no differences in the environment from year to year; either as a result of external factors or internal factors such as population size. In particular, we assume that under every resident strategy, the population dynamics are stable. We first describe how to determine the best response B(π) to a resident strategy π in this setting.

To determine environmental conditions when the resident strategy is π, we begin with a large population composed entirely of individuals following this strategy. It is assumed that all descendants of these initial population members also follow strategy π. We then follow the population forwards in time, taking into account density- and frequency-dependent effects. Eventually the population settles down so that for each time of year, the number of population members in each state at this time is the same on successive years. When this is achieved the environmental conditions faced by any individual is the same at the same date on successive years. (Of course the state of the individual can change from year to year.)

Having found how environmental conditions in the resident π population depend on time of year, we are concerned with the fitness of mutant strategies under these given conditions. The idea here is that while mutant numbers are rare, they do not alter the environmental conditions created by residents. The fitness of a mutant strategy is the asymptotic rate of increase in the number of mutants (in formal terms it is the maximum eigenvalue of the projection matrix under this strategy). However, in order to seek the mutant strategy with the greatest fitness, we do not explicitly evaluate the fitness of each mutant strategy; when there are vast numbers of possible strategies this is not feasible. Instead, we employ a technique based on dynamic programming that finds the (usually unique) strategy that maximizes the number of descendants left far into the future by an individual (see McNamara 1991; Houston & McNamara 1999). This strategy is then the mutant strategy maximizing fitness (McNamara 1991). In other words it is the strategy B(π).

Given a procedure for finding best responses, to find a solution of equation (7.3) we might initially proceed as follows. We start with an initial strategy π₀; we then find strategy π₁=B(π₀); then strategy π₂=B(π₁), and so on, at each stage finding the best response to the previous strategy. Hopefully, this sequence of strategies converges to a limiting strategy π^*. If so then we have identified a solution to equation (7.3). Unfortunately, the sequence of strategies obtained under this crude procedure does not always converge. If this is the case then it is necessary to modify the procedure using various tricks to ensure convergence (McNamara et al. 1997). Fortunately these tricks work although they may be computationally intensive.

With strong density or frequency dependence in a model there is always the possibility that there may be more than one ESS. Some insight into whether this is so can be gained by performing the above iteration for a range of different initial strategies π₀. If all end up at the same ESS, it suggests that the ESS is unique.

Earlier, we have considered two approaches to modelling life histories. In one the environment is a given input and LRS is taken to be the fitness measure. The other involves a full population analysis with feedback. The two approaches are linked in the following way. Consider the situation with feedback and a full population analysis. Suppose that the population is at evolutionary stability and is also at a density-dependent equilibrium. Then it can be shown that members of the resident population are following a strategy that maximizes their LRS (Taylor et al. 1974; McNamara 1993).

(c) Fluctuating environments

So far we have described the situation where there are seasonal changes, but environmental conditions are the same at the same time of year in successive years. However, in many circumstances conditions fluctuate from year to year. Fluctuations may be due to weather, changes in pathogen levels or other fluctuating external factors. Fluctuations may also be due to internal factors such as the changes in population density resulting from unstable population dynamics. When there are year-to-year changes it is necessary to include state variables that record conditions each year. In particular, it may be necessary to have current population density and other population characteristics as state variables.

When there are population fluctuations, the appropriate fitness measure is the dominant Lypunov exponent of the stochastic population projection matrix (Metz et al. 1992). Unfortunately, this fitness measure is difficult to work with. The fitness of a strategy can be estimated by simulating population growth (e.g. Caswell 2001). But such an approach is very limited, and has no hope of being of use when the set of possible strategies is large, as would be the case for annual routine models. The standard approaches based on dynamic programming are no longer directly applicable, although a technique for finding optimal strategies without the need to calculate the fitness of each individual strategy does exist (McNamara 1997). However, this technique is only feasible when there are very few states.

8. Model outputs

As mentioned, a strategy is a contingent rule specifying the dependence of behaviour on state and time. The behaviour at a given time of year depends on both the strategy and the state of the organism at this time. To illustrate the distinction between a strategy and the behaviour observed under the strategy, consider a capital breeder whose relevant state variable is its energy reserves. For definiteness, we assume that reproduction is rapid for this organism, but that reserves are depleted during reproduction. Suppose that the strategy for this organism is never to initiate breeding during the period between October and the end of January; outside this period breeding is initiated once energy reserves exceed a critical threshold that depends on time of year. For example, it might be that the threshold is high in February and September, but lower in April. From this description alone it is impossible to tell when breeding will occur. For that, we also need to specify the dynamics of energy reserves under the strategy. For example, if the organism can build up reserves quickly all year round, then it will breed several times between February and September. In contrast, if the build up in reserves is very slow then it may not be able to breed every year, but tend to breed biennially. Note that under this biennial breeding the organism's strategy is the same each year, but its state tends to alternate between high and low values in successive years (McNamara et al. 2004).

To find the behaviour under a strategy we usually proceed in two stages as follows. First a large (infinite) population is projected forwards under the strategy, taking into account density-dependent effects. The projection is continued for several years until the transient effects of the initial population density and distribution of states disappear. Note that this is a computational procedure based on an exact calculation of densities and the distribution of population members over states, not a simulation based on following individuals (Houston & McNamara 1999). The computation gives information on equilibrium population density, the distribution of states of population members at this equilibrium, and the proportion of individuals performing each action at each time of year. Thus for example, it might tell us that in any given year only half the population breeds.

The above procedure does not follow individuals over time and so cannot tell us about how the behaviour of an individual at one time is correlated with its behaviour at other times. For example, it could be that half the population is breeding each year because all organisms are biennial breeders, or because half of population members breed every year and the other half never breed. Consequently, a simulation that follows individuals is also performed. To do so, the above computational procedure is used to find the distribution of birth dates and states in the population. A birth cohort (of say 100 000 individuals) is then chosen from the distribution. Cohort members are then followed over time until all are dead, recording for each individual all breeding attempts, young produced, surviving young, foraging behaviour, migratory journeys, etc.

Possible features of routines that can be investigated by these computational procedures are now outlined.

(a) Breeding routines

Do we expect behaviour to be free running or entrained to the annual cycle? If breeding is entrained, what is the pattern? How many breeding attempts are there per year? Is there a tendency to skip a year after a successful breeding attempt? Are there odd patterns, for example breeding twice every second year as observed by Doody et al. (2003)?

(b) Changes in state and behaviour over the year

The outputs of state-based models include the way in which state and behaviour vary over the annual cycle. Model outputs might include fat load, muscle mass, foraging intensity, location, parasite load and immune function (e.g. Nelson 2004; Martin et al. 2008) over the year.

(c) Sources and timing of mortality

How do levels of mortality from the various possible sources vary over the year? For example, under what circumstances does most of the mortality occur during breeding or shortly after it, as opposed to over the winter? It might be tempting to identify those times of year with high levels of mortality as bottlenecks during which the environmental conditions are most severe. It needs to be remembered, however, that state variables can have long-term effects so that high mortality at one time of year may be just a reflection of what has happened at a previous time of year (Buehler & Piersma 2008).

(d) Degree in overlap of activities

Our model allows for the possibility of overlap of activities such as breeding and moult, and can thus be used to determine when it is optimal to overlap these activities. More generally, we can investigate whether we expect to observe distinct life-cycle stages (cf. Wingfield 2008).

(e) Lifespan and ageing

Typically, we do not include age in our models because it is not necessarily an informative state variable. Many state variables may be correlated with age, but it is often better to work with these variables rather than age itself. For example, in many species the age at first reproduction is variable because the decision to reproduce depends on state, which is not perfectly correlated with age (McNamara & Houston 1996). Models that analyse the decision to start reproduction solely on the basis of age are severely limited.

Although age is not usually a state variable in our models, we nevertheless obtain age-related effects as outputs. In addition to predicting the distribution of the age at first reproduction, models that have appropriate physiological state variables can also predict the pattern of reproduction over the life of an organism and its mortality schedule.

(f) Knock-on, carry over, etc.

Our approach provides a natural framework for understanding the way in which behaviour at one time of year can have consequences at later times, and hence allows us to identify when it is possible to look at a time of year in isolation. This is illustrated in the case of a migratory bird by McNamara et al. (1998). They investigate the influence of migration to the breeding grounds in spring on the timing of the autumn migration. To do so, they compare the output of a full annual routine model, with that of a modified model in which a bird has to survive until spring on the wintering grounds, but then does not need to migrate. Another possible approach to understanding interactions over the year is to change conditions at one time of year to see if they influence behaviour at another time of year.

In addition to knock-on effects, we can investigate correlations in behaviour round the year. For example, whether those individuals that breed early also migrate early, etc. Various explanations for these correlations are possible. Stochasticity may mean that some individuals are lucky in the outcomes that they obtain over the year even though individuals and habitats are the same. At the other extreme, there may be permanent differences in individual abilities. Between these extremes, there may be differences in habitats or non-permanent differences in individual state that may generate correlations. State-based models of annual routines can be used to explore the extent to which various assumptions can generate correlations.

(g) Virtual experiments

An appealing aspect of the approach is that the models that it generates can be used to mimic empirical investigations. We now illustrate the use of virtual experiments in several contexts, bringing out the distinction between two ways of analysing the effects of a change.

1. The strategy does not evolve. In this case, the optimal strategy for a particular environment is found and then the consequences of following this strategy under an experimental manipulation or environmental change are investigated. The basic assumption here is that there has not been the opportunity for the strategy to evolve in response to the change.

2. Evolution of new strategy. In contrast to the first case, in the second case a new optimal strategy is found for the novel circumstances.

(i) Supplementary feeding

A common experimental manipulation involves providing animals with extra food; see Boutin (1990) for a review. Houston & McNamara (1999) attempt to mimic this manipulation in a virtual experiment. Their annual routine model considers the number and timing of reproductive attempts of a hypothetical animal during the annual cycle. The animal lives in an environment in which the food supply varies over the year, but there is no year-to-year variation. Energy reserves are one of the state variables of the model animal. The optimal strategy specifies whether or not to reproduce for every level of energy reserves (and values of the other state variables) and every time of year. Under this strategy, the animal never reproduces when reserves are low. Reproduction only starts if reserves (and condition) are sufficiently high during a crucial ‘breeding window’ in spring. In order to mimic the effect of giving the animal extra food at a particular time during the winter, the energy reserves of the model animal are increased by a given amount at this time. The animal is then assumed to follow the original optimal strategy. As a consequence of the manipulation, reserves later on during the breeding window tend to be higher for the manipulated animal than for an unmanipulated animal. Condition also tends to be higher since the manipulated animal can afford to work less hard in getting food. The effect of the manipulation is to increase the likelihood of breeding that year. It also results in earlier breeding within the breeding window.

In the environment modelled by Houston & McNamara there is no experimenter giving animals extra food. If instead it were common for the animal to receive supplementary feeding, breeding would be easier for the animal and its optimal strategy would be different. In other words, it is assuming that the animal is not adapted to receiving supplementary feeding when it responds to this manipulation. Although animals are usually not adapted to supplementary feeding, in their natural environments food availability fluctuates. Food availability is not the same at a given time in successive years, and food availability now is likely to be positively correlated with future food availability. This aspect of the biology could be incorporated into a modification to the model of Houston & McNamara by including another state variable representing food availability. Behaviour would then be allowed to depend on the value of this state variable as well as energy reserves, condition and time of year. The extra food given in the manipulation in winter would alter the animal's estimate of current food availability, affecting the future value of this state variable and the future relationship between behaviour and energy reserves. The predictions of this modified model would therefore be different to the predictions of the original model.

(ii) Catch up growth

In many species, a period in which food is scarce or absent is followed by a period of faster than usual growth. This is known as compensatory growth or catch up growth; for reviews see Metcalfe & Monaghan (2001) and Ali et al. (2003). Mangel & Munch (2005) investigate this phenomenon in a model with two state variables, size and damage. The animal's level of activity determines the animal's energy intake and expenditure. The rate of mortality is determined by the two state variables and the level of activity. The amount of damage that is repaired is not optimized; instead the energy that is devoted to repair is a given function of size and the level of damage. What is optimized is the level of activity. The optimal strategy specifies the optimal activity level as a function of size, damage and time. This optimal strategy is then used to predict the trajectory of growth (and damage) following a period of food deprivation. The model of Mangel & Munch does not incorporate periods of food deprivation. Such an approach is appropriate if food shortage is taken to be a rare event that the animal has not evolved to cope with. If shortages were common, growth should be investigated in an environment with periods of food shortage (cf. Lindström et al. 2005). Growth trajectories would then be optimal given an anticipation of future food shortages, and the response after a food shortage would depend on how probable one period of shortage was likely to be soon followed by another (cf. McNamara et al. 2005).

(iii) Changing food availability at stopover sites during migration

Weber et al. (1999) analyse the effect of a change in the availability of food at sites where migrating birds stop to gain energy. In contrast to the annual routine models that we present in this paper, Weber et al. consider a single journey of a bird on a migration from its wintering site to its breeding site. Between these sites there are a given number of sites where the bird can stop to refuel. These sites are known as stopover sites. The bird's state is its location (i.e. which site it is at) and its level of fuel. If fuel falls to zero the bird dies from starvation. The bird's reproductive success depends on the date at which it reaches the breeding site and its level of fuel when it arrives. The bird can decide on its foraging intensity while at a site and the level of fuel at which it leaves the site. A high foraging intensity results in fuel being gained at a high rate, but incurs a cost in terms of a high risk of being killed by a predator. Weber et al. assess the consequences of a change in the availability of food at a stopover site for a bird's reproductive success, comparing the effect of keeping the strategy fixed with the effect of allowing the new optimal strategy to evolve. If the birds are able to evolve the new strategy, then they decide not to use a stopover site if its level of food availability has been reduced. If the birds follow a strategy that was optimal for the original conditions, then the loss in fitness depends on the form of the strategy. If behaviour depends just on initial level of fuel, the effect differs from the case in which behaviour depends on fuel load, location and time. Weber et al. did not consider a third case in which behaviour depends on food availability as well. Including food availability as a state variable would allow greater flexibility in response to environmental changes. Although this work provides a clear illustration of the difference between the two ways in which animals may respond to a change in their environment, it has several limitations.

1. It is based on a single migratory journey rather than a full model of the animal cycle.

2. Since the whole of the animal's annual routine is not represented, it is necessary to assume a particular form for the terminal reward, i.e. the equation that specifies reproductive success as a function of fuel load at arrival and date of arrival.

3. A change in the quality of one of the stopover sites is assumed to have no effect on the terminal reward.

As these examples demonstrate, the predicted effects depend on model details. In particular, it is important whether the experimental procedure is itself regarded as being a part of the natural environment of the animal and is hence explicitly included in the model. The choice of state variable can also be crucial to predictions. Including extra environmental states can increase the degree of plasticity of a model animal.

9. Discussion

We have advocated a view of annual routines in which trade-offs are given a physiological basis. Environmental conditions and behaviour interact to determine how physiological variables change over time. The levels of physiological variables affect the ability of an organism to survive and reproduce. We have described how optimal behaviour can be found. A general strength of our approach is that it considers the whole year as an entity. In our formalism, we allow for the possibility that the behaviour of organisms and their number affect the environmental conditions experienced by population members. Our approach thus links behaviour to both physiology and ecology.

Jacobs & Wingfield (2000) and Wingfield (2008) outline a framework for annual routines based on finite state machines. This framework involves transitions from one life-history stage to another over the year. We regard this approach as an informed description that might suggest general ‘laws’ that characterize the diversity of annual routines. In contrast, we are using models based on optimization to try to predict annual routines. State variables are a central component of these models. Early models did not include enough detail about the state variables to introduce inertia into the system. For example, animals could initiate breeding at any time of year, regardless of their state. But a realistic model for many species would need to include changes in the state of the reproductive organs before reproduction can occur. Once variables such as testes size are included, our models have the potential to make predictions about when sequences go just in one direction, as for Wingfield's stages, or can be reversed, as for Wingfield's sub-stages. We expect that future work along these lines could thus unify our approach and that of Wingfield.

(a) Implementing a strategy: mechanisms and constraints

An animal's behaviour over the year is controlled by physiological processes. Incorporating mechanisms in models makes it necessary to consider realistic and important details that are often ignored by standard life-history theory. Our approach based on state variables is a way of representing this underlying machinery and capturing some of the fundamental constraints such as conservation of energy and those imposed by nutrient limitations.

In finding optimal behaviour, our approach allows for flexibility in that the action taken in one state does not constrain the action taken in another state. The validity of this approach depends to some extent on the choice of state variables that are used in a model. It has been claimed, however, that mechanisms impose a more fundamental limitation on flexibility, and that mechanisms limit the range of phenotypes that are possible. For example, there may be constraints resulting from the multiple effects of hormones (e.g. Ketterson & Nolan 2000; Ketterson et al. 2005). There may also be phylogenetic constraints (e.g. Hahn & MacDougall-Shackleton 2008). Within our formalism, the dynamics of state variables automatically imposes constraints but we view the control of this physiological machinery as being completely flexible. To invoke hormones as a constraint is to add some inflexibility in the control of the machinery that can be achieved.

It is easy to invoke hormones as unavoidable constraint—Gould (1997) asks whether the masculine genitalia of the female spotted hyena (Crocuta crocuta) are a consequence of the high level of testosterone in females that has resulted in females being larger than males. Gould suggests that the genitalia are a spandrel, i.e. an unavoidable side effect (see Gould & Lewontin 1979). In this case, the genitalia are taken to be a consequence of female dominance (masculine genitalia as an automatic result of hormones). Muller & Wrangham (2002) say that Gould overstates the case for constraint. They point out that sexual differentiation in mammals is ‘notable for its plasticity’ (p 6). Furthermore, female spotted hyenas have normal female sexual behaviour. It thus seems strange to attribute the female genitalia to inevitable effects of a hormone when its effects on brain and behaviour do not seem to be inevitable.

Ricklefs & Wikelski (2002) argue that variation in life histories lies along an axis that can be characterized as a slow–fast continuum. At the slow end, organisms have slow development, low reproductive rate, and long lifespan and at the fast end organisms have fast development, high reproductive rate and short lifespan. They suggest that the absence of different combinations of these variables is evidence for a constraint on possible life histories, and propose that hormonal mechanisms are the basis of the constraint. We are not convinced that such strong constraints are necessary to produce the slow–fast continuum. Given that the machinery will automatically impose constraints in our sense, it seems possible that selection could be responsible for the pattern of variation.

(b) Future directions

In the introduction we outlined various issues to do with annual routines. All can be investigated using our approach. We now look in more detail at two issues that we think are particularly exciting directions for future research.

(i) Immune function over the annual cycle

Immune function varies over the annual cycle (see Nelson et al. 2002; Martin et al. 2008). In part, this may reflect the fact that different pathogens are present at different times of year and require different parts of the immune system to be activated. It is also likely to reflect how energetic demands change with time of year (Altizer et al. 2006; Buehler & Piersma 2008). Here, we outline how the effects of energy and other nutrient demands may be analysed using annual routine models.

We might begin by ignoring the distinction between innate and adaptive immunity, concentrating instead on the level of constitutive immune function (the standing defence) and the strength of a response to infection should a pathogen manage to establish itself despite the constitutive defence (Janeway 2004). Daily decision epochs might be appropriate. In this basic model, the state variables could be the level of stored resource in the body, whether or not an animal is infected and the age of dependent young. Each day the animal would choose a foraging action together with an immune defence action. The foraging option could specify, for example, how much time to spend searching for food. The greater the time the more food is found but the greater the predation risk and energy expenditure. If the animal is not caring for young, the food that is found would be eaten by the focal animal, if caring the animal would first feed young (say a fixed amount) and then eat the remainder. The immune defence action would specify the level of resources allocated to immune function. If the animal is not infected, resources would be allocated to constitutive defence. The greater the investment the smaller the chance the animal is infected next day. If infected the investment would be to an acute phase response. The probability of death from the infection would decrease with the amount invested. The probability of death could also depend on food intake and energy expenditure during the course of the infection.

The above model could be used to investigate various issues, for example, how the level of immune defence of both types varies with time of year, how stored resources vary with time of year, and how mortality from each source (starvation, predation, disease) varies with time of year. All these outputs would be dependent on environmental factors, for example, how the level and virulence of pathogens changes with time of year and how seasonal is the food supply. In addition, the energy required in energetically costly phases such as reproduction and migration could be crucial.

More realistic models of immune function might split resources into fat and protein (in muscle). Models could also handle a two-way classification of immune function into constitutive defence versus response and innate versus adaptive immunity (Schmid-Hempel & Ebert 2003).

(ii) The pace of life

Latitude influences life-history traits in various organisms (e.g. Iverson et al. 1993; Heibo et al. 2005). In birds, tropical species tend to have smaller clutch sizes (Martin 1995) and mature more slowly (Klaasen 1994) than temperate species. These differences have been viewed as part of a general trend towards a slower pace of life in the tropics, associated with a lower metabolic rate (Wikelski et al. 2003). McNamara et al. (2004) and Barta et al. (2006) considered the effect of latitude, as represented by seasonality, on optimal annual routines of non-migratory birds. In general they found that there were more reproductive attempts per year as seasonality increased. This pattern is suggestive, but the models assume a fixed clutch size. To investigate the dependence of clutch size on latitude a new model, in which a bird has the choice of clutch size, is required. In such a model, it is important to include density dependence so that at whatever latitude, the population is at a density-dependent equilibrium. Preliminary results from a model with clutch size decisions and density dependence do indeed predict a strong increase in clutch size as seasonality increases (McNamara, J. M. et al. unpublished data). In agreement with the data, this model also predicts a decrease in the age of first breeding and in care times as seasonality increases.

Although the model of McNamara et al. (unpublished data) includes clutch size and maturation time, it does not capture some other aspects of the pace-of-life continuum. In particular, it cannot incorporate the rate at which the body ages. In order to make predictions about the effect of latitude on the rate of senescence, it is necessary to include a state variable that represents the extent to which the body has worn out. It is also necessary to include a decision about the extent to which resources are used to repair the body. In other words, we need to add the basis of disposable soma models to the clutch size model.

Given suitable trade-offs mediated by physiology, outputs of a seasonal disposable soma model could include metabolic rate while active and resting. Such models also have the possibility of predicting how growth and metabolic rates are linked and the whole question of how these and other fundamental attributes scale with body size. These are exciting possibilities, but the real difficulty is in specifying the correct physiology to capture the trade-offs.

Supplementary Material

Calculating the strategy maximizing lifetime reproductive success

This appendix shows how the strategy that maximizes lifetime reproductive success may be found using the technique of dynamic programming.