Randomized or fixed order for studies of behavioral syndromes?

Abstract

There is a growing interest among behavioral ecologists in behavioral syndromes and animal personality. Studies of behavioral syndromes repeatedly measure the same individuals to quantify within-individual consistency and between-individual variation in behavior. Often these studies measure behavior in different contexts or in different behavioral assays to determine whether individual differences in behavior in one context are related to behavior in other contexts, that is, a behavioral syndrome. For studies of behavioral syndromes, there is not universal agreement about whether it is preferable to randomize the order of different assays or to administer them in a fixed order. Here, I articulate the advantages and disadvantages of testing in a randomized or fixed order and offer some recommendations according to the goals and power of the experiment. In general, studies using within-subjects designs that are primarily interested in mean-level differences between treatments should randomize the order that individuals experience different treatments. Under certain conditions, studies of behavioral syndromes should also administer the assays in a randomized order, but only if the study is sufficiently powerful to statistically account for carryover and period effects. If the experimenter is interested in behavioral syndromes that are caused by carryovers, it is often preferable to test in a fixed order. If the experimenter wants to guard against carryovers, but the experiment is not sufficiently powerful to account for carryover and period effects, then a compromise is to test in a fixed order, but to test individuals in the context that is most likely to affect subsequent behavior last.

INTRODUCTION

Studies of behavioral syndromes examine correlations between-individual differences in behavior in different contexts or situations (Sih et al. 2004). Often such experiments involve confronting animals with different challenges such as predation risk (e.g., Hedrick 2000) or a rival conspecific (e.g., Sluyter et al. 1995) or a novel object (e.g., Verbeek et al. 1994) in separate behavioral assays. However, prior experiences such as sensing a predator or winning a fight can have relatively enduring effects on subsequent behavior. Therefore, if individuals are measured for their behavioral reactions to a predator and then presented with a conspecific, for example, then behavior toward the conspecific might be influenced by recent experience with a predator, that is, a “carryover effect” (Diaz-Uriarte 2002). For studies of behavioral syndromes, there is not universal agreement about whether it is preferable to randomize the order of testing in different contexts or to administer assays in a fixed order (Table 1, Logue et al. 2009; Dochtermann 2010). Here, I articulate the advantages and disadvantages of measuring behavior in different contexts in a randomized or fixed order and offer some recommendations according to the goals and power of the experiment.

Table 1 .

Short survey of recent studies on behavioral syndromes published in Animal Behaviour and Behavioral Ecology ^a

Reference	Species	#assays	Assays	n	Fixed or random?	Test carryover?	Test period?	Time between assaysb
Adriaenssens and Johnsson (2011)	Salmo trutta, brown trout	3	Foraging in absence and presence of intruder, dominance	72	Fixed			5–24 h
Brodin (2009)	Lestes congener, damselfly	6	Activity when hungry, activity when fed, boldness, activity in presence of salamander predator, activity in presence of dragonfly predator, foraging	22	Fixed			unk
Chapman et al. (2011)	Myrmica ruginodis, Myrmica rubra, Myrmica ants	4	Novel environment, heterospecific encounter, response to alarm pheromone, social tendency	50	Fixed			24 h
Colleter and Brown (2011)	Melanotaenia duboulayi, rainbowfish	3	Aggression, boldness, activity	28	Fixed			2 weeks
David et al. (2011)	Taeniopygia guttata, zebra finch	5	Activity, neophobia, exploratory tendencies, risk-taking behavior, obstinacy	41	Fixed			unk
Duckworth (2006)	Sialia mexicana, bluebirds	3	Aggression toward bluebird, tree swallow, house finch	14	Random	Yes	No	<24 h
Gabriel and Black (2010)	Cyanocitta stelleri, Stellar’s jays	5	Risk-taking in presence of model predator, risk-taking at a trap, exploration of a novel feeder, exploration beyond territory, trapping success	20–114	Haphazardc	No	No	unk
Garamszegi et al. (2009)	Ficedula albicollis, collared flycatcher	3	Exploratory, aggressiveness, risk-taking	18–24	Fixed			unk
Herborn et al. (2010)	Cyanistes caeruleus, blue tits	4	Exploration in captivity, neophobia in captivity, exploration in wild, neophobia in wild	78, 115	Fixed			24h captivity, unk wild
Johnson and Sih (2007)	Dolomedes triton, fishing spider	4	Boldness control, foraging and boldness, courtship and boldness, parental care and boldness	60	Fixed	No	No	>4 days
Logue et al. (2009)	Gromphadorhina portentosa, hissing cockroach	5	Righting, saline, foraging, intruder female	70	Random	Yes	No	7 days
Mafli et al. (2011)	Urotestudo boettgeri, Hermann’s tortoise	4	Aggressiveness, fearfulness toward humans, exploratory behavior, activity	25	Haphazard	No	No	unk
Martin and Reale (2008)	Tamias striatus, eastern chipmunks	3	Handling test, hole-board test under cover, hole-board test in the open	24	Random	Yes	Tested for date	>10 days
Minderman et al. (2009)	Sternus vulgaris, starling	2	Exploratory behavior, environmental sensitivity	24	Fixed			Immediately
Moretz et al. (2007)	Danio rerio, zebrafish	4	Social behavior (shoaling), activity, predator approaches, feeding	84	Fixed			Immediately to 18
Nomakuchi et al. (2009)	Gasterosteus aculeatus, 3-spine sticklebacks	2	Exploratory behavior, joining performance	20/pop	Fixed			24 h
Piyapong et al. (2010)	Poecilia reticulata, guppy	2	Boldness in the presence of conspecifics of the same or different sex	31, 32	Random	No	No	24 h
Pruitt and Riechert (2009)	Anelosimus studiosus, comb-footed spider	5	Interindividual distance, prey attack, superfluous killing, antipredatory, exploratory-boldness	36, 80	Fixed			unk, 7 days
Pruitt et al. (2008)	A. studiosus, social spider	6	Interindividual distance, superfluous killing, prey attack sequence, antipredator, exploration and boldness, group feeding	30, 46	fixed			>24 h
Pruitt et al. (2011)	Anelosimus species, social spiders	3	Interindividual distance test, boldness, prey attack sequence	39–914/pop	Fixed			24 h
Salonen and Peuhkuri (2006)	Thymallus thymallus, European grayling	3	Aggression before threat, threat, aggression after threat	24/pop	Fixed			Immediately
Schuerch and Heg (2010)	Neolamprologus pulcher, cichlid	3	Propensity to explore, boldness toward novel object, aggression	14–21	Fixed			~10 min
Smith and Blumstein (2010)	P. reticulata, guppy	4	General activity, sociability, boldness, exploration	unk	Random	Yes	No	24 h
Snekser et al. (2009)	Stegastes leucostictus, damselfish	2	Aggression toward a male and courtship toward a female	19–20	Random	No	No	<24 h
Wilson and Godin (2009)	Lepomis macrochirus, bluegill sunfish	3	Exploratory behavior, response to novel object, boldness under predation risk	22–47	Fixed			24 h

^aPublications were identified in a search for “behav* syndrome” (topic) in Web of Knowledge in April 2012, restricted to the journals Animal Behaviour and Behavioral Ecology. Observational studies that did not experimentally measure individuals in different assays were excluded. The list is meant to be representative of recent studies, not exhaustive.

^bTime between measures in different assays.

^cSometimes fixed, sometimes random.

TESTING IN A RANDOMIZED ORDER

Randomization is a central tenet of experimental design. When a study aims to test the hypothesis that a treatment influences behavior, it might either assign different individuals to different treatments (parallel design, sensu Diaz-Uriarte 2002), or use a within-subjects design (crossover, sensu Diaz-Uriarte 2002) and apply all treatments to each subject in a randomized order. For example, Sih et al. (2003) tested the hypothesis that salamander larvae reduce levels of activity under predation risk by measuring the activity of the same individual salamanders in 2 conditions—in the presence and absence of chemical cues of a fish predator—and the order in which the 2 treatments was applied to each individual was randomized. For within-subjects designs that are interested in mean-level differences between treatments, it is important to randomize the order of testing because if there is a carryover of one condition on to the other, then the carryover might obscure the treatment effect. For example, if exposure to predator cues influences subsequent behavior and if individuals had always been tested in the presence of predator cues first (fixed order), then Sih et al. (2003) might not have detected an effect of predator cues on activity. However, because the order was randomized, that is, some individuals were tested in the presence of cues first and others were tested in the presence of cues last, then the authors were arguably in a better position to detect a true difference between treatments after statistically accounting for carryover effects (Diaz-Uriarte 2002). In general, when comparing average behavior across treatments with a within-subjects experimental design, the order in which treatments are applied should be randomized.

Unlike studies designed to test whether there are mean-level differences between treatments, studies of behavioral syndromes are primarily interested in the rank order stability of individual differences (differential stability; Stamps and Groothuis 2010) across 2 or more contexts. Such studies often measure the same individuals in several different behavioral assays that are designed to assess diverse behaviors ranging from “exploratory behavior” to “boldness” to “aggressiveness” (Table 1). In general, most recent studies on behavioral syndromes administer assays in a fixed rather than in a randomized order (Table 1). However, some authors have suggested that studies of behavioral syndromes should randomize the order of the different behavioral assays (Logue et al. 2009; Dochtermann 2010).

The rationale for randomizing the order is that the experimenter can statistically test and therefore correct for carryover effects. In the salamander larvae example discussed above (Sih et al. 2003), if there had been a carryover of exposure to predator cues on activity, then individuals might differ in activity in the absence of cues due to the order in which they were tested rather than because of intrinsic individual differences (Figure 1a). To test if there is a carryover effect, the experimenter could compare activity in the absence of cues between individuals that were tested in the presence of cues first versus in the presence of cues last. If there is a carryover effect, then including “order” in the statistical analysis should correct for it and could allow the experimenter to test for a behavioral syndrome using the order-corrected data. Importantly, testing in a random order does not, by itself, take care of carryover effects; only by accounting for them statistically will the experimenter reap the benefits of testing in a randomized order (Diaz-Uriarte 2002).

Figure 1 .

Potential sources of error caused by a carryover effect. Assume there is a carryover from assay 1 (e.g., exposure to predation risk) to assay 2 (e.g., confrontation by an opponent); individuals decrease behavior after assay 1. (a) Some individuals are tested in assay 1 before assay 2, whereas others are tested in assay 2 before assay 1 (randomized order). In that case, individual differences in behavior in assay 2 could reflect a carryover effect. (b) Individuals are tested in assay 1 before assay 2 (fixed order). If some individuals (very sensitive) experience more of a carryover than other individuals (I × E interaction), then individual differences in behavior in assay 2 could reflect individual differences in the effect of the carryover (sensitivity).

However, few studies measure individuals in just 2 different behavioral assays (average = 3.6 assays, Table 1) and as the number of assays increases, the number of individuals tested in any particular order decreases, making it harder to detect and therefore correct for a carryover effect. For example, with n = 40 individuals (typical for most recent studies, Table 1) tested in 4 contexts in a completely randomized order, only ~1–2 different individuals will be tested in any particular order, allowing only very large carryover effects to be detected. Therefore, it might not be surprising that few recent studies that test in a randomize order actually statistically test and correct for carryover effects (Table 1). Although it is appealing to think that we can control for carryover effects by testing in a randomized order, the reality is that it is only profitable when the experiment is sufficiently powered to correct for them. For other experimental designs that are more powerful for detecting carryover effects, see Dochtermann (2010).

Another less-appreciated issue is that there might also be period effects with repeated testing of the same individuals (Diaz-Uriarte 2002). For example, levels of behavior might generally change with age, or with seasonal changes, or with increasing body size (growth). Unlike testing in a fixed order, where carryover and period effects are confounded, testing in a randomized order opens the possibility of statistically teasing apart period and carryover effects, as well as their interaction, but this is rarely done (Table 1) and is only possible with certain experimental designs (Diaz-Uriarte 2002).

One possible solution is to take multiple measures on each individual in each assay in a randomized order, thereby effectively “washing out” the carryover effect for each individual. However, this is often not feasible because in addition to worrying about carryover effects, the experimenter often has to consider other constraints, such as the possibility of habituation with repeated testing and the potentially confounding influences of age and experience. More importantly, this approach increases within-individual variance, thereby making it harder to detect behavioral syndromes.

Another possible solution is to give the subjects plenty of time (e.g., days rather than hours) to recover to baseline between tests in order to minimize the carryover. However, this option is not feasible when there are relatively long-lasting effects on experience in one context on subsequent behavior, for example, 1-trial learning about predators (Magurran 1990), or the winner effect, which can last from hours to weeks (Hsu et al. 2006). Most recent studies allow less than 24h to recover between assays (Table 1). Moreover, assuming that animals can return to a nonshifting baseline between assays, this approach requires longer recovery times between assays, which could increase period effects.

TESTING IN A FIXED ORDER

The alternative to testing in a randomized order is to test in a fixed order, that is, all subjects are tested in assay 1, then assay 2, and then assay 3. The rationale for testing in a fixed order is that the experimenter suspects that there might be carryovers; to control for them, individuals are tested in the same order. The advantage of this approach is that individuals arrive at each behavioral assay with the same experience.

However, with a fixed-order experimental design, the experimenter cannot statistically correct for a carryover effect. Therefore the experimenter must assume that if there is a carryover effect, it is similar for all individuals, or if individuals differ in the carryover effect, the variation among individuals in the carryover effect is small relative to the main effect of the carryover. For example, the experimenter might assume that each individual’s activity is similarly affected by recent experience with predators (no I × E interaction, Nussey et al. 2007; Dingemanse et al. 2009; Martin et al. 2011). If this is the case, then rank order differences between individuals (if they exist) will be maintained across contexts, even if there are mean-level differences between treatments. However, if there is substantial variation among individuals in the magnitude of the carryover effect, that could generate increased error variance that could obscure a behavioral syndrome, leading to what could be perceived as a “false negative” (Figure 1b).

The disadvantage of testing in a fixed order is that “behavioral measurements would be influenced by the presentation order, not the experimental situation, and these carryover effects could heighten or diminish potential behavioral correlations” (Logue et al. 2009; Dochtermann 2010). That is, when individuals are tested in a fixed order and if the subjects have not yet recovered from the first assay when they are tested in the second assay, then behavior in the second assay could reflect the enduring effects of the first assay rather than response to the second assay, and this might generate a spurious correlation between behavior in assays 1 and 2. That is, a short-term carryover across contexts could cause a behavioral correlation between behavior in these 2 contexts that is dependent on the order and timing of experience, but that does not exist otherwise. In that case, the observed syndrome might be interpreted as a “false positive”.

The concern that testing a fixed order can generate spurious correlations assumes that we are not interested in short-term carryovers across contexts. Although this might be true for some studies, even relatively short-term carryovers lasting minutes to hours could be ecologically important (Sih and Bell 2008). For example, a short-term increase in testosterone in response to a fight could carryover to influence parenting behavior minutes to hours later (Ketterson and Nolan 1999). Indeed, the whole point of behavioral syndromes is that in nature, behavior in different contexts is not independent over ecological or evolutionary timescales. Such nonindependence can create carryovers across contexts or situations that result in limited plasticity, and even short-term carryovers can have fitness consequences (Sih et al. 2004).

Therefore although a carryover effect could generate a behavioral syndrome when assays are administered in a fixed order, the researcher might actually be interested in short-term carryovers. If a behavioral correlation is detected, then it would be important to report the recovery time between assays and to discuss the results accordingly (e.g., “behavior in the presence of a predator was correlated with foraging behavior 2h later”). However, if the experimenter wants to guard against syndromes that could be generated by short-term carryovers, then the experimenter might prefer to test in a randomized order.

RECOMMENDATIONS

Ultimately the decision about whether to test in a fixed or randomized order depends on the question of interest. If the experimenter is primarily interested in mean-level differences in behavior between contexts, then they should test in a randomized order, with the caveat that a within-subjects design might not be advisable if there is a long-lasting carryover. Similarly, a randomized order is preferable if the experimenter is explicitly not interested in short-term carryovers, with the caveat that the experimenter will only be able to reap the benefits of testing in a randomized order if the experiment is powerful enough to correct for them. Therefore, current standards for publishing studies of behavioral syndromes might need to be re-evaluated, that is, to require statistical tests for order and period effects and on larger sample sizes (Dochtermann 2010).

In contrast, if the experimenter is primarily interested in individual differences and is equally interested in both short- and long-term carryovers, then a fixed-order experimental design is often preferable. Even if the experimenter wants to guard against carryovers, then a fixed-order design can be suitable if the experimenter has a priori reasons to think that one assay is more likely to generate a carryover than others. In that case, they might test in a fixed order, and to administer the assay with the longest lasting carryover last. For example, if exposure to a predator has a longer-lasting carryover than exposure to a conspecific (as assessed, for example, by prolonged elevated levels of stress hormones after exposure to a predator, but relatively quick recovery after exposure to a conspecific; Bell et al. 2007), then behavior in the presence of a predator could be measured after behavior in the presence of a conspecific. In that case, the experimenter would not have to worry about the carryover effect and could estimate both mean-level differences in behavior between contexts, as well correlations across contexts.

In conclusion, when behavioral assays are carried out in a randomized order, then carryovers might generate or obscure individual variation and behavioral syndromes because randomizing the order of assays increases within-individual error variance. Although the ideal experiment might measure individuals many times in many assays in a fully randomized order, there are both statistical and biological reasons why this is not always advisable. Although testing in a fixed order introduces the possibility of carryover effects, if the experimenter is interested in short-term carryovers, this is not necessarily a problem. A compromise solution is to determine which behavioral assay is most likely to generate a carryover effect and to test individuals in that context last.