Cognition and the evolution of camouflage

Abstract

Camouflage is one of the most widespread forms of anti-predator defence and prevents prey individuals from being detected or correctly recognized by would-be predators. Over the past decade, there has been a resurgence of interest in both the evolution of prey camouflage patterns, and in understanding animal cognition in a more ecological context. However, these fields rarely collide, and the role of cognition in the evolution of camouflage is poorly understood. Here, we review what we currently know about the role of both predator and prey cognition in the evolution of prey camouflage, outline why cognition may be an important selective pressure driving the evolution of camouflage and consider how studying the cognitive processes of animals may prove to be a useful tool to study the evolution of camouflage, and vice versa. In doing so, we highlight that we still have a lot to learn about the role of cognition in the evolution of camouflage and identify a number of avenues for future research.

1. A quick guide to camouflage

Camouflage is a common and taxonomically widespread adaptation that many prey species have adopted in order to reduce the likelihood of being either detected or recognized by would-be predators [1]. Prey animals use a number of different forms of camouflage to avoid being eaten (box 1), but perhaps the clearest distinction is between masquerade and crypsis. Although there are many forms of crypsis, they all function to make prey difficult to detect or recognize as discrete objects [1]. In contrast, the appearance of masquerading prey ensures that predators mistake them for inedible objects (e.g. twigs, leaves, stones and bird-droppings) after they have been detected [2,3]. It is important to note that masquerade and crypsis are not mutually exclusive and that some species can benefit from both strategies simultaneously. For example, caterpillars that resemble twigs can initially remain undetected when viewed against a background of other twigs, and once they are detected as discreet objects they can be mistaken for twigs [7]. Until relatively recently, most work on camouflage has investigated the form and function of cryptic patterning [4–6,8–11]. Understandably, therefore, research has focused on how camouflage patterns exploit the sensory processes of predators (but see [12–16]). However, work on masquerade has highlighted how camouflage can also be designed to exploit the cognitive processes of predators, opening up new questions about how other camouflage strategies may also work at a cognitive level [17,18].

Box 1. Brief definitions of common forms of camouflage.

• Masquerade. Masquerading animals resemble inedible objects (models) commonly found in the local environment. Predators detect masquerading prey as discreet objects but misclassify them as the inedible items that they resemble.

• Background matching. Background matching animals resemble the general colour and pattern of the background. Predators find these prey difficult to detect because they ‘blend in’ to the background.

• Disruptive coloration. Animals using disruptive coloration possess (often high-contrast) markings at the margins of their bodies. Predators find these prey difficult to detect because the markings break up the body outline (a salient cue that predators often use to find and identify prey).

• Surface disruption. This involves the use of non-marginal markings to create ‘false edges' that are more salient than the true body form.

• Distractive coloration. Animals using distractive coloration possess small, isolated and conspicuous markings that are often not close to body margins. Predators are thought to find these prey difficult to detect because the markings draw predators' attention away from the salient body outline.

• Countershading. In countershaded prey, the body surface closest to the ambient light source (often the dorsal surface unless the species rests upside-down) is darker in colour than the body surface furthest from the light source. There are a number of ways in which countershading could reduce detection. For example, without countershading, surfaces closer to the light source appear lighter than those further away: countershading reduces this ‘self-shadowing’.

• (Nb. These definitions are intended to give a brief overview of different forms of camouflage. More detailed definitions can be found in the following papers: [1–6].)

The aim of our review is to consider the role that cognition plays in the evolution of camouflage strategies. This approach has been invaluable in the study of aposematism (where prey advertise their defences with conspicuous warning coloration) and mimicry (where defended or undefended species possess visual signals similar to those of a sympatric defended species) [19–21], and it is perhaps surprising that it has yet to be systematically applied to the study of camouflage. We will define cognition in the broadest terms: including how animals perceive, learn, classify and make decisions [22–24]. We will not only consider the cognitive abilities of predators, but also the cognitive abilities of the prey themselves since the behaviour of both cryptic and masquerading prey can influence the degree to which they benefit from their visual appearance [25–29]. We will limit our discussion to camouflage in the visual domain, since most research into the function and evolution of camouflage focuses on this. However, the principles can be broadly applied to camouflage in other sensory modalities [30]. We will begin by discussing the evidence demonstrating that predator cognition is an important selective agent driving the evolution of masquerade and move on to argue that there is reason to believe that it may also play an important role in the evolution of crypsis. We will then discuss the idea that prey cognition is also likely to influence the evolution of camouflage; and finally, we will highlight how studying the cognitive processes of animals could prove to be a useful tool to study the evolution of camouflage, and vice versa.

2. Predator cognition and the evolution of masquerade

It has long been believed that predators do not simply fail to detect masquerading prey, but rather mistake them for the inedible models (e.g. twigs and leaves) that they resemble post-detection [2]. This implicitly assumes that the cognitive processes of predators drive the evolution of masquerade: predators learn that models are inedible and generalize their learned avoidance of models to masquerading prey [2,3]. This has recently been confirmed empirically in a series of experiments in which naive domestic chicks acted as predators of caterpillars (early thorn moth, Selenia dentaria, and brimstone moth, Opisthograptis luteolata) that resemble the twigs of their host plants (hawthorn, Crataegus spp.). In these experiments, domestic chicks (Gallus gallus domesticus) with previous experience with twigs were slower to attack twig-resembling caterpillars than chicks with either no experience with twigs, or chicks with experience with twigs whose appearance had been manipulated so that they no longer resembled the caterpillars. Since the caterpillars were presented on a white background in an otherwise empty experimental arena, there was no possibility that they could be benefitting from crypsis. The chicks with experience of unmanipulated twigs learned that twigs were inedible, and so took longer to attack the caterpillars because they mistook them for inedible twigs [16,31–34]. Thus masquerading prey avoid predation by exploiting the cognitive processes of predators.

Further consideration of predator cognition allows us to better understand how masquerade evolves. It has been suggested that masquerading species evolved from cryptic ancestors, either by successive gradual increases in the resemblance to an inedible object or by a relatively large mutation resulting in imperfect masquerade followed by gradual improvement of the resemblance [2]. A recent comparative morphological analysis of the wing patterns of Kallima butterflies provides some support for the former: their leaf-like wing patterns evolved in a gradual manner from ancestors that did not resemble leaves [35]. However, at present, it is unclear whether this evolutionary trajectory is common for masqueraders, or whether predation is the selective force driving the evolution of these leaf-like wing patterns in this lineage [3]. Interestingly, both of the suggested evolutionary pathways rely on the same assumptions, that (i) prey can benefit from imperfect masquerade; and (ii) the benefit of masquerade increases as masqueraders become more similar to their models. And studies of predator cognition provide strong support for these assumptions. Predators generalize their learned avoidance of inedible objects sufficiently widely to be fooled by imperfect masqueraders that possess qualities that predators could use to differentiate them from their inedible models [33]. Importantly, however, imperfect masqueraders do not fool predators to the same extent as individuals that more accurately resemble inedible models [33]. Moreover, given the opportunity, predators can learn to discriminate between models and masqueraders, and they do this much faster when the masqueraders are imperfect compared to when they more closely resemble the model [28]. It is thus at least feasible that selection favoured the initial imperfect masquerading mutants over non-masquerading conspecifics, and subsequently, the gradual improvement of the resemblance over time.

Understanding predator cognition is also likely to be the key to explaining much of the diversity observed in the appearance of masquerading organisms. Many species of masqueraders are polymorphic or polyphenetic, or show high levels of intraspecific variation in their appearance [2,3]. For example, individual Nemoria arizonaria larva can resemble either oak twigs or oak catkins [36], and larvae of the American peppered moth, Biston betularia cognataria, resemble birch twigs when found on birch trees, Betula nigra, and willow twigs when found on willow trees, Salix babylonica [37]. This variability effectively allows a single masquerading species to resemble a number of distinct models [2,3,7], thereby increasing the total number of model items that they resemble. This is likely to influence predators' decisions about whether or not to include masquerading prey in their diet, because attempting to discriminate between models and masqueraders becomes less economically attractive as the density of models increases relative to that of masqueraders. Indeed, experiments have shown that predators are both less motivated to search for masqueraders, and more likely to misclassify them when models are common in comparison to masqueraders [29]. Intriguingly, individuals that differ in their appearance may also choose to live in different microhabitats (see prey cognition section below). In theory, this could ultimately lead to speciation. Therefore, predator cognition may drive both intraspecific and interspecific differences in the appearance of masquerading prey.

Clearly, predator cognition has played an important role in the evolution of masquerade. However, it is important to note that our understanding of how masquerading prey exploit the cognitive processes of predators is still very much in its infancy. Most experiments investigating the evolution of masquerade have focused on determining whether predators misclassify a particular masquerader as the inedible model it resembles [16,29,31–34], and/or what aspects of the masquerader's appearance or environment influences the probability of this happening [29,32,33]. Only a small number of experiments have investigated how predators learn to discriminate between masqueraders and their inedible models [28], and in these experiments, the masquerading prey were the only source of food available to predators. We know very little about how predators make decisions about when to invest in searching for and/or learning about masquerading prey when there are other sources of food available (as would be the case in many natural systems). While it seems reasonable to assume that predators make adaptive decisions that allow them to optimally invest in searching and learning [38,39], in the absence of experimental evidence in support of this, it is impossible to identify the full range of selection pressures driving the appearance of masquerading prey. Studying how predators learn about masquerading prey and make decisions about when to include them in their diets, promises to be an interesting avenue for future research.

3. Predator cognition and the evolution of crypsis

In prey animals, the primary function of crypsis is to reduce the chances of being detected (and consequently eaten) by visually hunting predators [1]. It is thus not surprising that the vast majority of studies in this field seek to explain how cryptic patterns work in terms of predator vision [4–6,8–11]. This has been a fruitful area of research and has led to the identification of a number of distinct forms of crypsis based on the mechanisms employed to hinder initial detection (box 1). This in turn, has allowed us to more accurately identify the specific selection pressures acting on different types of cryptic prey and has given us a better understanding of how predator vision works. However, this body of work focuses almost exclusively on how different forms of crypsis exploit the sensory process of predators and fails to consider the possibility that they may also influence attentional and cognitive processes. This could be an important oversight, as without a complete understanding of the function of different forms of crypsis, it is impossible to understand the selection pressures acting on cryptic prey, or why particular species possess the patterns they do.

There is good reason to think that the appearance of cryptic prey will affect how predators attend to, and learn about, them. A number of experiments have demonstrated that predators' abilities to detect cryptic prey can improve over successive encounters [12–15]. When predators initially find prey in a novel cryptic context, they must learn to discriminate between prey items and the background [12]. Once learning is complete, predators can form short-term search images over successive encounters with a single familiar prey type by selectively attending to salient prey features [14]. This increases their ability to detect prey of the encountered phenotype, but can reduce their ability to detect prey with other phenotypes ([13], see box 2 for further explanation). Importantly, we know that conspicuousness can influence both the speed of discrimination learning, and the degree to which search images decrease detection time. Predators learn discriminations more quickly when prey are conspicuous than when they are cryptic [40–42]; and although predators form search images for both conspicuous and cryptic prey, this only leads to a reduction in detection time when prey are cryptic because conspicuous prey are already easy to detect [14]. Thus, we might expect that as the detectability of cryptic prey decreases, predators should learn to discriminate prey and backgrounds more slowly and benefit more from search image formation. There may also be more subtle effects, with predators being better able to learn about or form search images for some forms of crypsis than others: perhaps because they have particular features that predators can learn to attend to. If this is true, then some forms of crypsis may be better at reducing initial detection, while others may be better at inhibiting learning or search image formation. However, at present, this is little more than speculation. While there is some evidence that the speed at which detection rates improve over time may be influenced by the form of crypsis that prey use [17,18], it is unclear whether or not this is due to differences in the speed of discrimination learning, search image formation or both. Consequently, we need to know much more about how different forms of camouflage influence learning and attention if we are to fully appreciate the selection pressures acting on cryptic prey.

Box 2. Hypothetical example explaining the distinction between discrimination learning and search image formation.

Imagine a scenario in which two groups of insectivorous birds are given a series of 10 trials. In each trial, they receive 20 sequential presentations of a single camouflaged moth. Individuals in one group receive a single species of moth and individuals in the other group received two visually distinct (but equally camouflaged) species of moth. Birds in both groups would learn to discriminate moths from the background: the total time to attack all 20 moths would decrease across trials until a stable asymptote was reached, and the asymptote would be lower when a single species was available then when two species were available (figure 1a). However, only birds given a single species of moth would form a search image once learning is complete: detection time would decrease across presentations in trials in which birds were performing at asymptotic levels (e.g. during trial 10) as these birds began to selectively attend to the salient visual features of the single prey type (figure 1b).

Figure 1.

(a) The mean total time taken to attack all 20 moths in each trial; (b) the mean time taken to attack each of the 20 moths presented in trial 10. Solid lines represent birds given a single species of moth, and dashed lines represent birds given two visually distinct species of moth.

Furthermore, when performing at asymptote, birds given a single camouflaged moth species would show increased detection times between the end of one trial and the beginning of the next (data not shown), since their search image would be lost during the interval between trials. In contrast, birds given two species of moth would not show an increase in detection time, since they would have no search image to lose.

We also know very little about how search images are formed, and what factors influence their formation. In laboratory experiments, predators tend to form search images for the most abundant cryptic prey type available, perhaps because they encounter these prey more often [13]. However, in these experiments all prey items have the same nutritional value, so the possibility remains that predators can make optimal decisions to form search images for the most rewarding prey in the environment. If this is the case, the profitability of prey may influence search image formation, and consequently highly nutritious prey may be under strong selection to evolve patterns that inhibit search image formation. The idea that predators could optimize search image formation also raises interesting questions about what specific features of prey predators form search images for. One possibility is that this is determined by which features of the prey pattern are most visually salient to the predator, and if this is the case, the features of prey that predators use are likely to be relatively fixed. Alternatively, predators may be able to use features more flexibly, selectively attending to the features that provide them with the optimal pay-off. For example, imagine a situation in which two prey species have different forms of crypsis but share a feature that is not particularly salient to predators. If predators find these species in isolation, it may benefit them to use the most visually salient feature in their search images; but when the two prey species are sympatric, predators may now benefit from using the less visually salient feature that the two species share, because it allows predators to find both types rather than just one. Therefore, understanding how predators learn about, and form search images for, cryptic prey may help us to understand what type of crypsis will most benefit a particular species, and how the composition of prey communities is likely to influence the relative benefits of different forms of camouflage.

4. Prey behaviour and cognition, and the evolution of camouflage

It is not just the cognitive abilities of predators that influence the evolution of camouflage: prey behaviour is also important. This is evident from widespread observations that prey are able to behaviourally increase their degree of concealment. Some prey species can alter their appearance, or switch between defensive strategies to better match their substrate, and this is often under behavioural control (e.g. [43–47]). This suggests that prey have evolved behavioural adaptations that allow them to assess how well hidden they are on their current substrate and make decisions to induce a change in their appearance. Research in this area has focused on what cues might elicit these changes, and what the benefits of colour change may be, and much less is known about how prey make decisions about when and how to change their appearance. Intriguingly, there are some data to suggest that the majid crab, Tiarinia cornigera, uses more algae to decorate its shell when in the presence of predatory fish [48], while hermit crabs (Pagurus bernhardus) may avoid swapping their current shell for a more cryptic one when predation risk is perceived to be high—perhaps to avoid the costs of being temporarily shell-less [43]. Therefore, changes in appearance do not seem to be solely based on information that animals have about their own appearance and that of the substrate they find themselves on. While these findings could be explained by assuming that crabs are using innate context-dependent rules of thumb to guide their behaviour, the possibility remains that they could be using cognitive processes to make adaptive decisions about when to change their appearance, and how much to invest in camouflage.

Other prey that are unable to change how they look can reduce their predation risk by restricting themselves to substrates that they readily match [49] or selecting microhabitats that are more concealing [25,28,29]. Again, the mechanisms underlying these kinds of decisions remain largely unexplored. However, they too appear to involve the integration of different sources of information. This can involve combining information from different sensory modalities about a specific location to select a resting spot (e.g. [27]), but can also be more complex than that. Masquerading early thorn caterpillars (S. dentaria) trade-off the potential benefits of protection from predation with foraging opportunity, but do so in a context-dependent manner. When caterpillars are given the choice between two twigs, one that provides foraging opportunities and the other that maximizes anti-predator defence, they select the one that increases the benefit of masquerade in daylight, when risks from visually hunting predators are high [28]. However, if predation risk is reduced by giving the same choice in the dark, or hunger is increased through food deprivation prior to testing, the caterpillars show a stronger preference for the branch offering the best feeding opportunities. They are able to adjust their decisions according to their physiological state and environmental cues that inform them about probable predation risk (see also [29]). Again such behaviour could be attributed to the use of either innate rules of thumb or more cognitive processes; and if the latter is true, then prey cognition may be an important selective agent driving the evolution of the appearance and behaviour of camouflaged prey.

It is important to know what processes are involved in evaluating and adjusting the level of concealment in order to identify the form of camouflage prey use, and determine how this enhances survival. For example, in laboratory studies, where prey are likely to encounter a more restricted set of options compared with those they have adapted to in the wild, prey may not make the same decisions as they would under more natural situations. This could potentially lead to the form of camouflage that the animal is using being incorrectly identified, or the benefits of a particular form of camouflage being underestimated. If the degree of concealment can be modified, it is important to know what information prey are using and how they are making their decisions, in order to know how effective the defensive coloration is, particularly if the prey's natural habitat is spatially or temporally variable. Selecting different microhabitats may also affect predator search strategies, which could influence the degree of protection afforded by the coloration. The efficacy of cryptic coloration or masquerade will depend upon the choices that both prey, and predators, make.

Furthermore, it is important to realize that the morphological and behavioural adaptations of prey are likely to coevolve [29,50,51], thus the evolution of these cannot be studied in isolation. For example, prey's microhabitat selection strategies probably evolved in response to the evolution of camouflage. However, the evolution of these behavioural adaptations is also likely to have altered selection on prey's visual appearance. Behaviours that allow prey to be better concealed on a particular background could lead to relaxed selection from predation on the prey's visual appearance, allowing further selection on prey coloration in relation to functions such as thermoregulation [52]. Thus prey could maintain predation rates at levels similar to those before the behavioural adaptation(s) evolved while being better able to thermoregulate. Therefore, the prey's behaviour will not only affect the efficacy of their camouflage, but also impact on other aspects of their ecology and life-history strategies; we must understand prey behaviour in order to identify what constitutes the optimal form or level of camouflage for a particular species.

5. Cognition as a tool for studying camouflage and vice versa

Above we have focused on how the cognitive processes of predators and prey may be important selective agents driving the evolution of prey camouflage. However, this is not the only reason for studying the cognitive processes of predators. Predator learning experiments could also be used as a tool for testing the mechanisms via which different forms of camouflage inhibit detection. Despite the tendency to define different types of camouflage by the mechanisms through which they prevent detection, studies investigating the mechanisms underlying some camouflage types (e.g. surface disruption and distractive markings) are limited in number, have produced equivocal results, or simply imply mechanistic explanations post hoc rather than testing them directly [4,11].

Experiments seeking to test mechanistic explanations tend to design a number of camouflaged prey types that should differ in their detectability in predictable ways if the mechanism being tested holds. They then subject these prey to predation by avian or human predators, and monitor either detection times or survival or mortality rates at regular intervals for a set period of time [4–6,8–11]. For example, animals using disruptive coloration possess markings at the margins of their bodies, and these have long been thought to make prey difficult to detect by breaking up the body outline: a salient cue that predators often use to find and identify prey. Experiments have consistently provided support for this idea by demonstrating that prey with disruptive markings placed at body margins survive better than those without these markings or those with similar markings positioned away from the body margins [8–10]. While these studies (and others like them) support the idea that disruptive coloration works by breaking up prey outlines, these experiments do not directly test whether disruptive coloration influences predators' abilities to detect prey edges. Thus, while the current findings may be highly consistent with particular mechanistic explanations and provide important support for them, they can never be critical tests of them.

This is where studies of predator cognition could prove complimentary to current approaches. For example, if disruptive coloration inhibits edge detection, then it would make it harder for predators to make discriminations that rely on the detection of prey edges. Consequently, it would be more difficult for predators to learn to discriminate between cryptic prey items of different shapes or sizes when they are disruptively coloured compared to when they are background matching (even if disruptive and background matching prey were designed to be equally difficult to detect). Furthermore, this approach may be particularly useful in testing mechanistic explanations that have proved tricky to pin down using more established experimental approaches. A prime example of this is surface disruption where non-marginal markings positioned away from the body outline are thought to create ‘false edges’ that are more salient than the true body form [4]. This explanation has proved difficult to test empirically. However, experiments comparing the speed at which predators learn discriminations based on differences in true prey edges with the speed at which they make discriminations based on differences in the false edges, could provide a critical test for this explanation.

Thus studying predator cognition may allow us to better understand the function and evolution of camouflage. But importantly, studying how predators behave when faced with camouflaged prey may also allow us to better understand predator cognition. Above we discussed the idea that predators develop search images for cryptic prey, and that these (and rates of discrimination learning) could be influenced by prey value and community structure. We also discussed how predators may make adaptive decisions about when to learn about masquerading prey and when to include them in their diets. Performing experiments to test these ideas will not only allow us to determine the extent to which predator cognition influences the evolution of camouflage prey, but will also allow us to better understand selective attention, discrimination learning and adaptive decision-making in predatory species.

Similarly, studying the anti-predator behaviour of camouflaged prey may enable us to better understand their cognitive abilities (if indeed cognitive processes direct prey's anti-predator behaviour). Studies of camouflage have revealed that prey's decisions to change colour or location are based not only on what they know about the substrate, but also on other factors, such as predation risk (e.g. [28,29,43]). This raises the possibility that there are other cues that are important in prey decisions that are yet to be explored and identified. For example, knowledge about the frequency and distribution of different microhabitats in the environment could determine whether or not prey make a colour change or look for a more suitable location. However, more crucially, we are yet to understand how these decisions are made. Perhaps the mechanisms involve non-cognitive innate ‘rules of thumb’ that have been selected over evolutionary time because they give a selective advantage to a particular type of colour pattern [28,29,50]. Alternatively, prey may be using sources of information in a more flexible cognitive manner, allowing them to better adapt to their current circumstances. For example, some species may be able to visually assess the efficacy of their camouflage and change their behaviour to further increase their chances of survival (e.g. [43,50,53]). Experiments that determine whether these changes are based upon rules of thumb or are modified with experience will therefore allow us to better understand what prey know about themselves and their environments. Moreover, selection for behavioural rules of thumb could be a direct consequence of the evolution of camouflage, and the ability to change colour may be one reason that cephalopods seem to possess relatively complex cognitive abilities. Therefore, understanding how cognitive processes enhance the function of camouflage patterns could also provide important insights into the selection pressures acting on cognition and the brain.

Authors' contributions

Both authors contributed to all aspects of this work.

Competing interests

We have no competing interests.

Funding

The development of these ideas was supported by a BBSRC project grant no. BB/L017709/1.