What artifice can and cannot tell us about animal behavior

Abstract

Artifice—the manipulation of social and environmental stimuli—is fundamental to research in animal behavior. State-of-the-art techniques have been developed to generate and present complex visual stimuli. These techniques have unique strengths and limitations. However, many of the issues with synthetic animation and virtual reality are common to playback experiments in general, including those using unmanipulated video or auditory stimuli. Playback experiments, in turn, fall into the broader category of experiments that artificially manipulate the array of stimuli experienced by a subject. We argue that the challenges of designing and interpreting experiments using virtual reality or synthetic animations are largely comparable to those of studies using older technologies or addressing other modalities, and that technology alone is unlikely to solve these challenges. We suggest that appropriate experimental designs are the key to validating behavioral responses to artificial stimuli and to interpreting all studies using artifice, including those that present complex visual displays.

Overview

Contemporary techniques for presenting complex visual stimuli (Chouinard-Thuly et al. 2017) are the vanguard of an experimental tradition that goes back at least to ter Pelkwijk and Tinbergen’s (1937) use of wax dummies to study aggression in sticklebacks. Artifice, defined here as the controlled manipulation of social or environmental stimuli, is the heart of experimental animal behavior. Artifice encompasses not only artificial social partners, but also artificially contrived social situations and environments, like pairwise choice tests for potential live partners isolated by physical barriers. In this issue, Chouinard-Thuly et al. (2017) detail the benefits and challenges of producing synthetic and interactive stimuli. The goal of this paper is to place these into the broader context of benefits and challenges that face experiments in animal behavior. The animations and virtual reality discussed elsewhere in this special column are kinds of stimuli used in playback experiments. Playback experiments, in turn, are a subset of techniques that involve artifacts. Many of the key issues facing animations and virtual reality are ubiquitous in the experimental analysis of behavior.

Playback

Animations and virtual reality environments are playback stimuli. Playback, the broadcasting of recorded or synthetic stimuli to elicit receiver responses, is a powerful tool with three main advantages over using live signalers. First, playback presentation provides control over stimuli, allowing experimenters to quantitatively manipulate and systematically vary specific aspects of signal variation while holding others constant (McGregor 1992). Second, playback allows one to create stimuli that are unavailable in nature, notably through de novo digital synthesis (Chouinard-Thuly et al. 2017). This is particularly useful when we want to test receiver responses to ancestral or novel signals (e.g., Phelps et al. 2001) or when we want to characterize how receiver responses align with multivariate signal variation (e.g., Fisher et al. 2009).

The third main advantage is consistency. Live signalers will inevitably vary in motivation, and their signaling behavior will inevitably interact with that of the subject. Playback controls these factors, allowing the experimenter to present and manipulate cues independent of identifiable confounding variables associated with signaler morphology and behavior, and has the potential to increase subject response repeatability (McGregor 1992; Kodric-Brown and Nicoletto 1997). Further, the repeatability of playback stimuli sequences allows us to track changes in behavioral responses over ontogeny, as the same subjects can be tested for response to the exact same stimuli at different life stages (Seyfarth and Cheney 1986; Mateo and Holmes 1997; Rosenthal 1999).

For synthetic animations and virtual reality to reap these advantages, it might be useful to consider a long-established technique in animal behavior, playback of acoustic stimuli (e.g., Marler and Tamura 1962; Bell 1974; Rand and Ryan 1981). Acoustic playbacks can be used to elicit context-appropriate responses, like aggressive displays or proceptive behaviors, in all major groups of animals known to signal in the auditory modality (King 2015; Rosenthal forthcoming 2016). Acoustic playback has always been an aspirational model for what can and cannot be accomplished in the visual modality, and video playback in vertebrates was first developed by birdsong researchers (Evans and Marler 1991). There are 2 main advantages of acoustic over video playback: they are easier to manipulate, and they retain more faithful representations of the natural stimuli they mimic due to the lower dimensionality of acoustic signals (Rosenthal and Ryan 2000).

Ease of Manipulation

In its simplest form, acoustic playback consists of calls or songs recorded in the field or laboratory and played back through commercially available, high dynamic range speakers in order to assay responses from a receiver. This is analogous to recording video of an animal performing a behavior, editing it, and playing it back on a monitor. What distinguishes acoustic playback, however, is the relative ease with which signals can be analyzed, manipulated, and synthesized. If a sound is assumed to originate from a point source, we can describe it as a function of frequency and intensity over time. Recorded signals can be broken into constituent parts through spectral decomposition techniques like Fourier and wavelet analyses (Beeman 1998). The same analytical approaches can be used to create synthetic stimuli that are mimetic to the extent that subjects fail to distinguish between a synthetic stimulus and a natural recording. For example, for túngara frogs, a synthetic signal described by 12 quantitative parameters is indistinguishable from recordings of natural calls to females (Ryan and Rand 1990, 1995). Likewise, mathematical decomposition of calls allows the experimenter to manipulate stimuli in diverse ways including “cutting and pasting” signal components (e.g., Clemens et al. 2014) or altering spectral and temporal properties. The modular nature of acoustic signals even allows experimenters to synthesize putative ancestral calls, thereby testing hypotheses about the coevolutionary history of signals and mating preferences (Ryan and Rand 1995).

Playback in the visual modality, which encompasses artificial stimuli ranging from video recordings of live exemplars to synthetic computer animations (Rosenthal 1999; Chouinard-Thuly et al. 2017), offers similar opportunities to manipulate stimuli but is inherently more challenging. In its technically simplest form, video playback uses recordings of live stimuli which are edited to vary in some attribute and then played back to test subjects. In the context of mate choice, for example, a subject might be tested for preference between 2 videos, presented either sequentially or simultaneously in a dichotomous choice arena, of the same courter performing different behaviors (e.g., Rosenthal et al. 1996). This allows the experimenter some control over variation in both the signal variable of interest and potentially confounding covariates.

Signal Fidelity

The technology used for recording, manipulating, and broadcasting stimuli is better for reproducing acoustic signals than it is for reproducing visual signals. Most of the longstanding criticisms of video playback methods revolve around the issue of signal fidelity (for more comprehensive reviews see D’Eath 1998; Fleishman et al. 1998; Fleishman and Endler 2000). Many of these issues still apply to contemporary techniques in the visual modality (Chouinard-Thuly et al. 2017).

In some important ways, video technology has caught up with acoustics. High-resolution monitors, high-speed video, and high monitor refresh rates have largely eliminated earlier concerns about the temporal and spatial resolution of stimuli (Baldauf et al. 2008; Chouinard-Thuly et al. 2017). Hyperspectral imaging, or just point-sampling of reflectance or radiance, allows us to quantify continuous color spectra (reviewed in Stevens et al. 2007), and multi-camera recording allows us to record 3-dimensional representations of visual scenes (Peters and Evans 2007). Signal fidelity becomes limited with the output technology. We first discuss 2 peculiarities of video, its 2-dimensional nature and the way it represents color, before turning to deficiencies that video playback shares with other experimental approaches in visual behavior.

Virtual reality holds promise for addressing the first limitation of video, the representation of a 3-dimensional scene on a 2-dimensional cue. Two-dimensional playback fails to mimic cues of depth reviewed in (D’Eath 1998; Zeil 2000). So-called pictoral cues to depth such as angular size, occlusion, and texture gradients can be incorporated into recorded video or mimicked automatically by animation software (Peters and Evans 2007). However, non-pictoral cues cannot be simulated in a 2-dimensional scene. These include binocular disparity cues by which depth is judged by the difference in retinal image between eyes, motion parallax, and accommodation or focusing cues (Zeil 2000). Further, stimuli can become distorted at close range, an issue that can partially be addressed by not allowing the subject to come into close contact with the video screen (D’Eath 1998). Virtual reality systems, however, have the potential to provide non-pictoral cues and correct for distortion in real time (Stowers et al. 2014).

The biggest deficiency of video relative to acoustic playback lies in the latter’s ability to represent a continuous frequency spectrum, even for ultra- and infrasonic stimuli (Pye and Langbauer 1998; Bradbury and Vehrencamp 2011). With video, by contrast, the color spectrum is down-sampled into 3 values that address human color perception. Video monitors therefore do not adequately mimic natural colors for non-human animals (Fleishman and Endler 2000; Chouinard-Thuly et al. 2017). It is possible in principle to adjust the white balance (the relative weight of the 3 color outputs) to the spectral sensitivities of other animals, but matching these sensitivities accurately is compromised since photoreceptor sensitivity can shift over ontogeny and seasonally (e.g., Cronly-Dillon and Sharma 1968; Spady et al. 2006; Temple et al. 2008). Further, commonly used animals will often have at least one photoreceptor whose sensitivity lies outside the spectral range of the monitor (Fleishman et al. 1998), notably in the ultraviolet (Cuthill et al. 2000; Fleishman and Endler 2000).

It should be noted that the failure to reproduce the ultraviolet portion of the visual spectrum is not a problem peculiar to video playback. Many, perhaps most, experiments using live exemplars or 3 dimensional models use a UV-deficient lighting environment or UV-absorbent materials that effectively abolish ultraviolet signals.

The same is true of polarization signals. Recent studies have shown that polarization sensitivity is important to the visual ecology of fish (Brady et al. 2015), cephalopods (Mäthger et al. 2009), and stomatopod crustaceans (Gagnon et al. 2015). With liquid crystal display monitors, all emitted light is linearly polarized while other monitor types such as cathode ray tube (CRT) produce virtually no polarized light (Chouinard-Thuly et al. 2017). The inability to manipulate polarized light is a potentially serious limitation of video stimuli, but again one that has been rarely considered in other types of manipulative experiments.

Finally, the background against which a stimulus is viewed is crucial for whether it is detected and how it is perceived by the subject. Most acoustic playbacks in the laboratory control for background–signal interactions by presenting a signal against a background of silence, but a “silent” visual background of black makes some signalers much more conspicuous than others. Since visual conspicuousness is defined relative to background, no such standard is possible for visual stimuli (Rosenthal 1999; Chouinard-Thuly et al. 2017). The spectral and spatiotemporal properties of visual backgrounds are of primary importance to receivers responding to visual stimuli (Uy and Endler 2004; Rosenthal 2007), and it is critical that researchers incorporate ecologically relevant background variation into their study designs.

Artifice is useful in animal behavior only to the extent that we can use it to draw conclusions about how animals would interact with natural stimuli. Signal fidelity poses the critical concern that a technique may introduce unanticipated confounding variables that may mislead our interpretation about what subjects attend to (D’Eath 1998; Fleishman et al. 1998; McGregor 2000).

Interactivity

An important feature of artifice is whether or not it is interactive—that is, whether the behavior of a model changes in response to the behavior of the subject. Playback can be interactive or not, as can the experiments using live exemplars (e.g., using 1-way mirrors). Most playback experiments are non-interactive; stimulus presentation does not depend on subject behavior. This can be advantageous, because it controls for receiver behavior and temporal variation in stimulus behavior (Kodric-Brown and Nicoletto 1997). However, courtship and agonistic displays, the most common signals simulated in video playback, are almost always interactive, with signalers responding dynamically with receiver responses. In some cases simulating such interactivity is useful for eliciting biologically meaningful subject responses (Ord et al. 2002; Chouinard-Thuly et al. 2017). In the case of courtship, the chooser interacts with the courter and often modifies its behavior according to the chooser’s response. It is possible to generate real-time interaction between stimulus and subjects in 2-dimensional space. For example, Butkowski et al. (2011) used data from tracking software to determine the position and behavior of a courting stimulus in real time. For longer-range interactions where a signaler broadcasts visual signals from one spot, such 2-dimensional interactions may mimic the natural dynamics of communication, for example, threat displays in Jacky dragon lizards (Ord et al. 2002).

Many of the behaviors simulated by playback, however, involve interactions in 3 dimensions, and as such playback may be inadequate to mimic complex interactive behaviors. For example, male sheepshead swordtail fish often prevent females from swimming away by blocking her exit path (Powell DL and Rosenthal GG, unpublished data). An entity that is trapped in 2 dimensions like a video playback stimulus cannot perform such 3 dimensional maneuvers. It should be noted that the same limitation is present when live signalers are blocked from directly interacting with subjects.

These kinds of interactions present a further problem when we want to interpret data from dichotomous choice tests, a very common design in experimental animal behavior. Here, it may be difficult to disentangle a subject moving away from stimulus A from a subject moving toward stimulus B. We may minimize this problem in part, by employing a Y-maze design in which moving away from one stimulus does not necessitate moving toward the other and by refining behavioral assays to characterize both aversive and proceptive responses to stimuli (Rosenthal forthcoming, in 2016). Some of these issues can also be circumvented by single-choice designs, whereby subjects are presented with a single stimulus at a time (Wagner 1998). However, single-choice assays tend to reduce choosiness, that is, the extent to which subjects respond differently to different stimuli. Single-choice assays may or may not be appropriate depending on whether subjects are likely to encounter signals simultaneously or sequentially in nature (Dougherty and Shuker 2015).

Robotic stimuli can provide 3-dimensional interaction. Robots have been used to explore several classes of behavioral responses e.g., differential responses to displays according to receiver sex in lizards (Martins et al. 2005), male courtship in birds (Patricelli et al. 2006; Patricelli and Krakauer 2009), female preference for color pattern (Phamduy et al. 2014) and schooling behavior in fish (Butail et al. 2013). However, the use of interactive robots presents a host of considerations not relevant for other playback experiments, notably incidental sensory cues that arise as a constraint of the mechanics of presentation, a response to any of which may complicate interpretation of results. For example, the inflation and deflation of artificial vocal sacs used in many anuran behavior studies produces mechanical sounds to which receivers may respond (Taylor et al. 2008; Klein et al. 2012).

Interactivity, or the lack thereof, presents a difficult problem in the experimental study of animal communication. Even so-called open-field interactions may introduce artifacts arising from spatial confinement or features of the habitat. One approach is to measure subject behavior across multiple contexts. For example, Kingston et al. (2003) used open field and dichotomous choices test with live stimuli, accompanied by dichotomous choice tests using color-filtered monochrome animated stimuli to show that females pygmy swordtails prefer blue morph males over the more aggressive gold morph males. Animated stimuli allowed them to decouple behavior from color.

Pseudoreplication

While workers have long articulated the problem of pseudoreplication (Hurlbert 1984; McGregor et al. 1992), many studies using synthetic, natural, and live stimuli continue to use limited numbers of exemplars to draw conclusions about responses to general properties of signals. It is worth mentioning issues specific to artifice here. Manipulated complex stimuli, whether digitally altered video or hand-painted dummies, are expensive and time-consuming to produce. Therefore, researchers often produce a handful at most of stimuli, and commit pseudoreplication by assuming that a given pair of artificial stimuli will elicit the same response as another similarly manipulated pair. One of the most attractive aspects of synthetic stimuli is that they help avoid pseudoreplication by allowing experimenters to frame hypotheses around defined stimulus properties, for example the population sample mean and standard deviation of a call parameter. With complex visual stimuli like animations, it can be challenge to measure all relevant parameters. This means that efforts to represent mean values for some parameters are often compromised by pseudoreplication with respect to others. For example, many of the animations we have generated (e.g., Rosenthal et al. 2002; Fisher et al. 2009; Verzijden and Rosenthal 2011) are compromised by the use of individual photographs for body texture and the use of single motion paths for rotoscoping stimulus display. This does nothing to eliminate pseudoreplication, unless many different animations are constructed each using unique texture files and motion paths.

Validating Experimental Designs Using Artifice

The hypotheses we test with artifice are usually ones that we want to extend to fitness consequences in natural stimuli. It is therefore necessary to demonstrate that subjects respond to artifice in ways that are directly comparable to how they behave in nature. If we have access to wild populations of the study organism, we can directly compare responses in experimental settings to responses to natural stimulus variation in the wild. Short of this, we should compare responses to artificial stimuli to responses to live stimuli. As Fleishman and Endler (2000) point out such comparison should be between stimuli of the same kind. That is, it is valid only to compare a specific class of synthetic stimulus to a live stimulus of the same type (i.e., an animation of a blue-morph male vs. a live blue-morph male).

It is also necessary to carefully consider whether the behaviors being assayed are reliable proxies for natural social behaviors associated with the relevant phenomenon, whether it be mate choice, aggression, or antipredator response. Measuring behavior across multiple contexts can increase confidence in the interpretation of subject responses to artifice (Kingston et al. 2003), as can testing appropriate negative controls such as subject responses to playback background in the absence of a stimulus. Furthermore, measures of attention such as gaze tracking (Yorzinski et al. 2013) represent a promising avenue for validating assayed behaviors. Ideally, we could measure behaviors or physiological responses specific to the process of interest, for example, proceptive behaviors like lordosis for mate choice. Such ground-truthing of methods is often overlooked or at least over-generalized from one taxon to another. When possible, we should directly compare behavioral proxies to natural behaviors. For example, the commonly used proxy of mate choice in small freshwater fishes, association time, can be directly validated by actual mate choice outcomes (Walling et al. 2009), but generalizing the utility of this proxy from one population or species to related groups may not be straightforward.

Where possible, experimental results should be validated using multivariate measures in wild populations. For the case of mate choice, this might include parentage analysis whereby responses to stimuli can be compared with actual mating decisions as quantified by the genetic makeup of offspring (Culumber et al. 2014). Another, behavioral approach might be to conduct social-network analyses of interactions involving proceptive or aversive behaviors of receivers as a function of variation in signalers (Wey et al. 2008). Analysis of agonistic interactions or responses to referential signals in the wild can further validate video stimuli designed to test such signals. The efficacy of synthetic stimuli is supported if they elicit responses that are congruent with responses in the wild to natural stimulus variation.

Synthesis and Future Directions

New technologies in animal behavior increase the scope and complexity of the stimuli we can represent and therefore the questions we can ask. With increasing complexity, however, comes increased risk that our experiments elucidate responses that don’t address the questions we ask. With proper design of stimuli and experiments, techniques like synthetic animations and virtual reality are powerful tools. For contemporary technologies with visual stimuli, the “best practices” paper by Chouinard-Thuly et al. (2017) addresses the important points that we need to be aware of with respect to signal fidelity with complex visual stimuli. If we cannot correct for these “known unknowns”, our hypotheses and interpretations need to be carefully circumscribed. For example, video playback cannot easily address hypotheses regarding color signaling for most organisms, but is nevertheless useful for testing responses to spatiotemporal patterns, given appropriate caveats about possibly confounding effects of distorted color representation (Fleishman and Endler 2000).

Complex visual manipulations, like any artifice, can address many hypotheses not tractable using other methods provided our hypotheses are sufficiently focused to accommodate its technical limitations (Oliveira et al. 2000). This of course is an important consideration for any study, just as the validation of stimulus efficacy as well as suitability of assayed behaviors as proxies for natural responses to stimuli are necessary for reliable interpretation of results. Technology alone cannot make our studies more interpretable. The major technical improvements to video output have removed limitations of spatial and temporal acuity, but we still have no way to present signals on video in a way that approaches the fidelity of acoustic signals. The increased complexity of animations and virtual reality has provided quantitative rigor, and mitigated pseudoreplication in some respects, but has at the same time raised a host of new problems that we must now address. By deploying an array of methodologies for assaying subject responses, and by appropriately framing and limiting our hypotheses, we can use this ever-expanding toolkit to gain unprecedented insight into animal communication.

We are especially indebted to Laura Chouinard-Thuly, Stefanie Gierszewski, and Klaudia Witte for organizing the symposium and putting together this special column. Pablo Delclós, Chris Holland, Mateo Garcia, Elizabeth Marchio, and 3 anonymous reviewers provided valuable comments that greatly improved this manuscript.

Funding

Manuscript preparation was supported by the National Science Foundation [1354172 to G.G.R. and [DGE 1252521 to D.L.P.].