The current and future state of animal coloration research

Abstract

Animal colour patterns are a model system for understanding evolution because they are unusually accessible for study and experimental manipulation. This is possible because their functions are readily identifiable. In this final paper of the symposium we provide a diagram of the processes affecting colour patterns and use this to summarize their functions and put the other papers in a broad context. This allows us to identify significant ‘holes’ in the field that only become obvious when we see the processes affecting colour patterns, and their interactions, as a whole. We make suggestions about new directions of research that will enhance our understanding of both the evolution of colour patterns and visual signalling but also illuminate how the evolution of multiple interacting traits works.

This article is part of the themed issue ‘Animal coloration: production, perception, function and application’.

1. Introduction

Animal colour patterns are unusually accessible for study and experimental manipulation, and consequently make a superb model system for understanding evolution because they consist of multiple components that have measurable multiple interacting functions. Colour patterns are used during interactions with social partners, potential mates, predators and prey, serve in thermoregulation and environmental shielding, and their interaction within development is also accessible. Because colour patterns interact with the environment in predictable and environment-specific ways, they provide an unparalleled opportunity for predicting evolution and testing the predictions. This has not escaped the notice of researchers and there has been an enormous growth in research in the area in the past 10 years, as summarized by the papers in this issue. However, the growth has been uneven, probably because the network of functional interactions and processes affecting colour patterns has not been appreciated as a whole. The purpose of this paper is to provide an overview of the processes affecting colour patterns (figure 1), use this to point out important processes that are relatively neglected, and make a few suggestions about ways to fill some of these gaps.

Figure 1.

The functional (solid arrows) and evolutionary (dashed arrows) interactions among colour patterns, colour pattern–based behaviour, and the state of the receiver. The evolutionary link between fitness and reflectance is direct and the link to irradiance is through evolution of microhabitat choice. There are also evolutionary links to all of the other processes (sensory drive [1]). Note that this diagram makes no distinction between physical, neurophysiological or perceptual mechanisms and does not assume that they occur at the same level of organization or complexity. All these mechanisms occur during colour pattern function and hence all affect the mode and rate of colour pattern evolution. One could argue about the order of events between reception and decision-making, or even whether the processes occur in serial (as shown) or parallel. This diagram is intended more as a way of identifying understudied areas than an exact representation of the interactions. Understudied areas with respect to colour patterns are marked with asterisks.

For brevity we will refer to colour patterns as the geometric pattern produced by a set of radiances [2] of light originating from the animal or plant (the signaller) and received by another individual (the receiver) in the presence of other radiance (the visual background, signal exploiters such as predators with lures, and other noise [3]). This radiance originates either through the interaction between environmental irradiance (ambient light) and signaller surface reflectance, or directly from bioluminescence[2]. Colour patterns are part of a network of interacting factors that include the environment as well as the signallers and their receivers, their properties and their behaviour (figure 1). A consideration of this network illuminates areas that are relatively neglected. We will consider each component of the network of interactions to draw attention to areas that are in particular need of research.

2. Irradiance and radiance

Except for bioluminescent organisms, irradiance (ambient light) is the ultimate source of colour pattern signals on both land and water, and irradiance spectra influence both the visibility and visual contrast of colour patterns [2–7]. Marine and other aquatic habitats have a much greater diversity of irradiance spectra than land, which may both bias and constrain evolution of colour patterns [5–7]. Although much is known about variation in irradiance spectra [5,6], new discoveries are still being made, for example horizontal asymmetry in aquatic irradiance [8]. Unfortunately little is known about the distribution and abundance of irradiance spectra in nature; large-scale surveys are needed of irradiance spectra variation among and within habitats. For example, there is new evidence for radiance spectra gradients or ‘signatures’ that are specific to and characteristic of different terrestrial habitats (D.-E. Nillson, personal communication 2016). If we can find general patterns of variation we should be able to make testable predictions about the evolution and distribution of colours and colour patterns in communities and ecosystems.

3. Reflectance and radiance

A colour patch reflectance spectrum describes which wavelengths of light (originating in irradiance) are transmitted towards the viewer as radiance. The radiance off each component (patch) of a colour pattern is the light moving from the patch to the viewer's eye as a light beam. For some patches (with structural colours) the reflectance spectrum depends jointly upon the angle of illumination and angle of view. The radiance spectrum depends upon the interaction between the irradiance spectrum and the reflectance spectra of each patch [2–7,9]. In contrast, bioluminescent organisms create radiance directly without depending upon ambient light. If they luminesce in strong enough ambient light, their net radiance may be altered if irradiance is strong enough to add reflected light to the internally generated light. For all species, the geometry and radiance spectra from all the patches in a colour pattern (animal or background) defines the fundamental physical properties of the colour pattern and will influence both its visibility and visual contrast.

There are now large numbers of papers on reflectance spectra, including humans [10,11]. New discoveries in animal communication have led to work on fluorescence [12] and polarization [13–15], which can significantly enhance simultaneous communication and camouflage. Also expanding is work on colour change [16], and relationships among coloration, colour change and thermoregulation [17,18]. There is rapid growth in our understanding of the physical and chemical basis of coloration and the interactions between pigments and structures resulting in reflectance patterns [19]. As data on more and more species are available, we look forward to large-scale comparative studies on each of these topics in the future; these will illuminate how colour patterns and colour-related behaviour can evolve and co-evolve. Curiously, given the massive advances in genomics, the study of genetics and development of colours, particularly structural colours, has barely started [20]. There is a massive potential for using genomics to work out constraints and biases in the rates and directions of evolution of colours. Combining this with knowledge of reflectance and irradiance spectra would allow us to predict evolution of coloration for both structure and function.

4. Environmental transmission

The radiance from a colour pattern has to pass through air or water before arriving at the viewer's eyes. Along that path the radiance spectra of each patch can be altered due to wavelength-specific absorption or scattering [3,6,7,13] and therefore the transmission medium affects detectability and visual contrast. Absorption and scattering by the transmission media also affects irradiance, particularly in water [6]. Scattering and reflection by particles within the light path create veiling light [7] that mixes with the colour pattern radiance, altering the radiance spectra as well as blurring colour pattern geometry by mixing light beams from different patches. We see these effects in fog or dusty air but veiling light effects are far more common in aquatic environments, especially when there are abundant particles [6,7]. These phenomena can also cause interesting and important geometric effects on the light environment, with significant consequences for visibility of colour patterns [6,8,21]. As for irradiance, there are no large-scale comparative studies or documentation of geographical and habitat-specific distributions of different kinds of medium transmission properties. Such information would be valuable in predicting the direction of evolution of colour patterns in time and space.

5. Reception

Eyes must receive, focus and convert light beams to neural impulses, and at the same time code information on luminance, colour, polarization, geometry and movement. All of this must be performed with small sets of wavelength-specific photoreceptors, which impose information constraints but also make understanding vision easier than systems with multiple channels [7,22]. Light beams arrive from colour patterns, visual backgrounds (including colour patterns which are not of interest) and veiling light. Both image formation and aiming of the eye help to maximize reception of the colour pattern of interest, although this will be compromised by veiling light (for example [23]) and by motion faster than the response time of some photoreceptors [22,24,25]. Research on animal eye evolution and eye properties relative to the light environment is very active [7,26,27]. There is some evidence for specific sets of photoreceptors repeatedly evolving in specific light environments, although the evidence for this is far stronger in aquatic than terrestrial environments, possibly because there is less variation in light environments on land than in water [27]. However, as more surveys are done on both light environments and animal photoreceptors, we may find general predictable relationships between eyes and environments. Of great interest in predicting evolution are models that optimize sets of photoreceptors and photoreceptor properties for particular visual environments and visual tasks [27]. For example, how many photoreceptor classes are needed, and does this increase with the complexity of the visual background, the visual task or both? Are all photoreceptor types used for colour or are different sets used for different purposes (as in mantis shrimp [28])? With a lot more comparative data we will be able to address questions about coevolution of vision and colour patterns including whether or not there has been coevolution, convergence or just diversification.

6. Retinal processing

Immediately after transduction of light into neural impulses, the signals are processed in the retina of vertebrates and invertebrates, and this is where the fundamental coding of luminance, colour, polarization, geometry and movement takes place [7,29]. There are some general scaling principles that apply to most animals and most senses, such as Weber's law [30] and colour constancy [31]. Unfortunately, the detailed mechanisms of coding (for example opponency) are only known for a few species [22,25–27], and some species may use completely different methods (e.g. [28]). We do not know how much variation there is among species or whether it varies with environment, habitat or phylogeny, nor do we really know how coding affects the evolution of colour patterns.

7. Detection

Before colour-based behaviour can occur, the colour pattern signal must be detected. Although shown in figure 1 between retinal processing and higher-brain processing, detection processes could occur at any level. If we regard the colour pattern of interest to the receiver as the signal and all other visual inputs as noise, then detection is a joint function of the signal and the noise, and detection becomes essentially a problem in statistical inference within signal detection theory (figure 2, [22,32]). There are four outcomes when input is assessed, both in signal detection and statistics (statistics is essentially signal detection): (i) signal detected, (ii) signal present but not detected (missed detection or type II error), (iii) signal not present and not detected, and (iv) signal not present but mistakenly detected (false alarm or type I error). For example, if a colour pattern represents a random sample of the visual background with respect to its patch size, colour, luminance and shape distributions, it is cryptic [4], the signal/noise ratio (S/N) will be well below 1, the viewer will probably accept the null hypothesis (signal not present) and look elsewhere for a signal. This statistical definition is likely to be true even if the animal moves or has greater three-dimensional depth than the background; in those cases a pattern with a stationary S/N < 1 will be relatively more difficult to detect than one with S/N > 1. In terms of signal detection theory, cryptic signallers evolve colour patterns that favour missed detections. Note that false alarms also occur, and if these are frequent enough, prey species could evolve ‘detected’ false positives as an alternative to classical crypsis. This can lead to masquerade [33,34], such as resembling an easily detected bird dropping, stick or leaf but not identifying it as edible.

Figure 2.

Signal detection. X represents the signal channel output value and f the frequency of occurrence of that value. X can be as simple as intensity or be a multivariate function of multiple properties of a colour pattern. X_a and X_b are mean values of two visual stimuli and N are the visual noise (background, channel and receptor noise). Note that N do not have to be equal as shown in this figure. X_b is the signal of interest and X_a is either the visual background or a second colour pattern, depending upon whether we consider detection or discrimination. The signal/noise ratio of detection or discrimination is (X_b − X_a)/N = S/N; the larger the S/N the easier the detection. X_t is a threshold value above which the detection system confirms detection and below which it accepts the null hypothesis of no detection. The use of X_t results in two forms of errors: I (false alarm) and II (missed detection), and systems should evolve to minimize both kinds of errors, depending upon their relative costs [32]. The detection criterion can be a threshold (X_t), but can also be an increasing (dashed line K) or decreasing function of X, as in mate choice.

One possibility that does not seem to have been examined is the effect of the degree of variation in pattern component salience (N in figure 2) as opposed to the mean values for animal and background (X_a and X_b in figure 2). It is possible that cryptic animals may have different within-pattern variances than their background (the two N in figure 2 would have different magnitudes, not equal as shown) so even if the animal-background signal difference (X_b − X_a) were large, if the variance associated with the animal were larger than that of the background, causing much overlap with the background variation, the animal might only be detectable using very different retinal or brain processing than if both N were similar. How often does the variation in the signal converge on that of the background in cryptic animals, and diverge in aposematic species and in social communication components of coloration?

An ideal signal detection mechanism tries to minimize both type I and II errors [32]. If the costs or consequences of both kinds of errors are equal this is most simply accomplished by setting a threshold (X_t) midway between the mean values of the background noise (X_a) and the signal (X_b), see figure 2. If the error costs are unequal (C_a ≠ C_b) and the two stimuli are unequal in frequency (P_a ≠ P_b ≠ 0.5) then the detection criterion (X_t, figure 2) needs to be shifted by P_bC_a/P_aC_b [32]. An alternative way of adjusting costs is to have the detection criterion be an increasing (dashed line K in figure 2) or decreasing function of X, as is often the case in mate preference functions. Thresholds could have heritabilities less than one and could easily be learned or otherwise influenced by prior experience [22,35]. Studies examining conditions favouring different kinds of colour pattern detection thresholds or functions have yet to be done. The optimum foraging literature does explore costs and frequencies but has not related it to colour patterns. Is there any evidence that particular detection functions and criteria evolve under specific visual conditions?

Merilaita et al. [22] do an admirable job of discussing signal/noise ratios relevant to camouflage, an area that is rapidly expanding and yielding new insights into animal coloration as well as visual mechanisms. New insights from computer vision and pattern recognition are also starting to be incorporated into colour pattern detection studies, showing that object detection depends upon a combination of colour and pattern parameters [22,36,37]. Duarte et al. [16] expand our understanding of camouflage via colour change, which allows additional avenues for hiding from predators or potential prey. One of the more interesting insights is that if visual mechanisms use ‘shortcuts’ (simplified visual processing) to extract visual information, then these can be exploited in camouflage [22]. In contrast to camouflage, active communication requires colour patterns that are easy to detect (salient), and conspicuousness properties have been well explored in mimicry, aposematic coloration and brood parasitism [36,38,39]. But little is known, even in humans, about the detection of moving objects, or detection and tracking of moving patterns during attacks or active social behaviour (e.g. [40]).

8. Higher-level (brain) processing

Once a signal is detected it needs to be classified into commonly occurring or predictable entities or groups of entities, making up the raw materials for perception. An analogous process in computer vision is image segmentation, where an image is divided up into distinct entities and then the entities or building blocks are processed further to make inferences about image content. There is some literature on this from psychology and animal behaviour, but it has not been thoroughly integrated into studies of colour patterns. For example, how and how often do animals, like humans, group colours into categories? Has this evolved in order to minimize the effects of minor differences in colour owing to feather, hair or integument wear or ontogenetic accidents [41], which would make classification and decision-making more complex? If receptor noise evolved to increase just enough so that minor differences in colour were confused, this would remove the need for a lot of neural processing and avoid the expense of maintaining more complex classifying neural networks [41]. What causes a given level of receptor or other noise tolerance to evolve in order to minimize image processing, detection and classification? How often is signal noise–based grouping a means of minimizing processing effort? How often has this energetically cheap and computationally simple grouping mechanism evolved to be used in decision-making? Noise might actually be useful at appropriate magnitudes. We clearly need to know more about how the building blocks of perception are acquired, and whether neural expenses can be reduced by efficient use of noise.

Once the entities and groups in a scene are collected, how are they combined? What primitive features are extracted from a detected object and assembled into an internal representation of the three-dimensional object [42,43], and how are these used in colour patterns [22]? For example, experiments with humans detecting camouflaged individuals strongly suggest the importance of edge detection in predators and anti-predator mechanisms to break it [44]. How do they vary according to the type of signal and its function? What is the effect of different receivers with different visual systems, and do different receivers use different cues [45]? Even within a colour pattern the location of different elements (patches) relative to each other can affect various levels of processing from attention through feature assembly to perception, as well as the speed of any of these processes [46]. For example, one way that disruptive coloration may work is that, at least in humans, making use of different features on the same object is easier and faster than for two or more objects [46,47], and disruptive coloration gives the appearance of multiple salient objects. The mechanisms allowing assembly of geometric and other properties of objects into perceptual objects, and the effect of pattern geometry on attention, may give us completely new insights into the design of colour patterns used in both camouflage and successful communication.

9. Perception

Once signals or stimuli are classified they need to be interpreted and recognized, or perceived, as being relevant to guiding future behaviour. Although perception is the subject of a large literature, both on humans and animals [11,48], it is only just beginning to be seriously integrated into studies of colour patterns, but still rarely done with wild animals. Considerations of perception are most advanced in studies of aposematic coloration and mimicry [30,36,39]. For example, stimulus generalization, peak shift and selective attention effects can lead to different outcomes and predictions than if these phenomena are implicitly or explicitly assumed to be absent [49,50]. Related to selective attention are mechanisms that attract and hold gaze and bias subsequent gaze directions (e.g. [22,51]); these could be taken advantage of in both camouflage and communication, but have not been examined in this way. There are constraints as well as biases to perception; for example, signal complexity may require more memory [52]. Processing multiple channels and other sensory modes may affect the rate of evolution of every component [53]. Any of these phenomena can potentially bias or constrain the direction of colour pattern evolution. For example a perceptual bias can affect the direction of sexual selection [54] and then feedback to the evolution of colours and perception (sensory drive, [1]). The importance of perceptual mechanisms in colour patterns used in learning, social behaviour, prey selection and pollination is likely to be a highly fruitful line of investigation; there are huge gaps in our knowledge of animal cognition with respect to coloration [35]. And because most of our knowledge of perception comes from laboratory animals or humans, we do not know how generalizable this is to wild animals in their natural habitats, or even whether or not the perceptual experiments are relevant to the animal's evolved ecology. Perception and decision-making relevant to natural ecology is essentially unknown and needs more attention. Once data are available, large-scale comparative studies might reveal whether or not suites of perceptual mechanisms repeatedly evolve under specific natural conditions.

10. Discrimination

The perceptual world comprises a set of entities, and decisions are based upon discriminating among them. For example, behaviour can be based upon discrimination among good and bad potential mates, good and bad potential prey, or models and mimics. Discrimination does not simply depend upon the degree of differences among perceived entities (or image segmentation), but also depends upon prior experience and learning, both sensory and cognitive states of the individual, prior expectations about differences (from experience or evolution) and receiver psychology [11,55–57]. We know very little about how these factors interact and bias the direction of colour pattern evolution. In the future, large-scale comparative studies could be used to reconstruct patterns of coevolution among groups of traits and how the network of interactions among them evolves. It would be especially interesting to relate this to interacting coloration functions to see if interacting functions predict coevolution of traits. Signal detection theory (figure 2) applies to discrimination as well as detection, and, like detection, it is largely unknown what factors set the position or slope of the critical discrimination function, nor whether it should be a threshold or a gradient. Except possibly in mimicry systems [36,39], little is known about what factors aid or prevent discrimination. What are the effects of prior experience on discrimination (e.g. [58])? Do particular visual environments [46,59], sets of alternative visual targets [46] and sets of different viewers [60] affect how discrimination proceeds and the form and position of the discriminant function? How does the discriminant function or criterion change with the colour pattern viewer's internal state or its receiver psychology? How does the criterion change if discrimination is simultaneous or sequential? What are the long-term implications of these phenomena? These questions are likely to be very fruitful lines of research.

11. Decision-making

Once the sensory system and the brain perform discrimination, the discriminated object or objects form the basis for decision-making. There is a very large literature on decision-making relative to colour patterns, including avoiding predators (camouflage, aposematism, mimicry, etc.), hiding from or luring prey, communicating to conspecifics and heterospecifics, habitat choice and behaviour relative to thermoregulation, and all papers in this issue form an excellent review of our understanding of various aspects of colour-based decision-making. Decision-making may not only involve behaviour resulting from discriminations, but also may directly affect the position and slope of the discriminant function, and affect further decisions. If this occurs, sequential decisions and behaviours will not be independent, and may further bias the direction of colour pattern evolution. We also know little about the role of multimodal signals in either perception or decision-making [45,61,62].

12. Colour-based behaviour

Decision-making occurs in the brain and is then followed by behaviour (motor patterns), which is how we deduce the outcome of decision-making by animals. Consequently, most studies of colour implicitly combine decision-making and colour-based behaviour, but this is rarely made explicit. Most of the papers in this special issue summarize the latest data and ideas about colour-based behaviour. Advances in neurobiology and neuroethology will be needed before we understand the relationships between decision-making and behaviour (colour-based or otherwise) resulting from decision-making.

13. Consequences and fitness

Colour-based behaviour has both immediate and evolutionary consequences. These consequences range from altering the receiver's state and psychology, which affects further behaviour, to fitness, which will bias and constrain the evolution of colour patterns (figure 1, dashed lines). The greatest effort in colour pattern research is concentrated on this subject, and it is well covered by the papers in this special issue, except for the effects of varying predator communities. What is the effect on fitness of variable predator communities or the relative abundance of alternative prey? For example, selection may be relaxed or even change direction if alternatives are abundant or the predator community changes (e.g. [63,64])? Other interesting and under-explored questions from these papers include: What are the direct fitness effects of colour change? Are different predators with different vision likely to favour different kinds of coloration? If more than one kind of predator is likely to be encountered in an individual's lifetime, would differences in predators' vision favour mechanisms such as colour change or displaying different parts of the body that allow switching camouflage strategies depending on which predator is the immediate threat? Are there phylogenetic patterns in colour change abilities [16]? How do trade-offs between different colour pattern functions (e.g. sexual selection versus crypsis or warning coloration, or signalling versus thermoregulation) work, and how often do they evolve [16,18,37,65,66]? Why are some cases of mimicry far from perfect and other cases superb [36,39]? How often are systems optimized for most efficient detection within particular tasks [11]? What is the relationship between signal function, signal diversity, signal function and species diversity [67]? How do signals originate [11]? Do species recognition and gender recognition processes favour different kinds of colour patterns than mate choice and other social processes [38,66]? Does this affect speciation and extinction rates [68], and is this related to the cost of defence? Clearly there is a lot still to do relating colour pattern characteristics to function and fitness.

When colour patterns are used in communication, the behaviour and fitness of both the signaller and receiver are affected, and there is usually alternation of each member between signaller and receiver roles [3,69]. This favours efficient transfer of information. Most considerations of signalling consider noise and the design of signal and receiver properties. However, even with perfect design, transmission will not be effective if the receiver's attention is not captured and held long enough for signal-based decisions to be made. Before anything can happen the signaller needs to attract the receiver's attention and alert the receiver that a signal is about to be sent; the receiver needs to be looking in the correct direction and be attentive to the incoming signal. The signaller then needs to hold the receiver's attention long enough for assessment and behavioural choice. All this requires the signal/noise ratio to be high enough for attention, assessment and stimulation to increase the likelihood of successful communication (for example [60]). We do not know how often pattern geometry has evolved for attention as well as assessment. For example, do small-scale within-pattern differences in pattern geometry attract attention to specific parts of the pattern and affect acquisition of adjacent parts of the pattern [46,51], making communication faster and more accurate? Alternatively does geometry combined with movement make pattern acquisition and visual tracking less accurate, as in dazzle camouflage [4,22,70,71], or even change the average colour during fast movement [24]? In some cases visual illusions may be used to hold the receiver's attention and to alter perception about the signal [72–74]. There is almost nothing known about the effects of attention on animal colour pattern function and evolution, except for simple alerting signals (e.g. [75]). Alerting signals tend to involve motion rather than pattern, but motion is also an under-studied aspect of colour patterns.

Figures 1–2 emphasize the mechanisms affecting the function of colour patterns, this is a matter of ‘hardware’, signal design and signal efficacy, and provides many opportunities to predict and understand the evolution of colour patterns. However, the mechanisms are insufficient by themselves; information needs to be carried between the colour pattern and viewer [76]. It is the information that forms the basis of decision-making (‘software’); thus colour patterns must contain both design (efficacy) and content (strategy [3,55]). The study of the content of colour pattern signals and signalling strategy is well developed, especially in the context of mate choice and signal honesty [10,66,77]. Colour patterns consist of multiple components that vary in luminance, colour, polarization, shape and motion, and hence could communicate multiple channels of information simultaneously. This, and the general phenomenon of multimodal signalling, is just beginning to be appreciated [78], and should give significant new insights into the development of signal complexity as well as understanding the evolution of colour patterns [62].

In conclusion, although there is rapidly expanding knowledge and understanding about the function and evolution of colour patterns, the rapid growth has been concentrated in some areas while others have been neglected (figure 1). Even greater progress should be made if we integrate all aspects of colour pattern function into the evolutionary ecology of coloration.

Authors' contributions

The paper was planned jointly by J.A.E. and J.M., first draft written by J.A.E. and both J.A.E. and J.M. revised and worked up the final version.

Competing interests

We have no competing interests.

Funding

We received no funding for this study.