What’s wrong with my experiment?: The impact of hidden variables on neuropsychopharmacology research

Abstract

The field of neuropsychopharmacology relies on behavioral assays to quantify behavioral processes related to mental illness and substance use disorders. Although these assays have been highly informative, sometimes laboratories have unpublished datasets from experiments that “didn’t work”. Often this is because expected outcomes were not observed in positive or negative control groups. While this can be due to experimenter error, an important alternative is that under-appreciated environmental factors can have a major impact on results. “Hidden variables” such as circadian cycles, husbandry, and social environments are often omitted in methods sections, even though there is a strong body of literature documenting their impact on physiological and behavioral outcomes. Applying this knowledge in a more critical manner could provide behavioral neuroscientists with tools to develop better testing methods, improve the external validity of behavioral techniques, and make better comparisons of experimental data across institutions. Here we review the potential impact of “hidden variables” that are commonly overlooked such as light-dark cycles, transport stress, cage ventilation, and social housing structure. While some of these conditions may not be under direct control of investigators, it does not diminish the potential impact of these variables on experimental results. We provide recommendations to investigators on which variables to report in publications and how to address “hidden variables” that impact their experimental results.

Introduction

A major objective in the field of neuropsychopharmacology is to delineate underlying mechanisms of behavioral phenotypes related to mental illnesses [1] and substance use disorders [2] with the ultimate goal of identifying new treatments. This approach has been successful in research on cancer [3] and cardiovascular disease [4]. The development of new tools for studying brain function in rodents has revolutionized how the brain can be studied and manipulated. Rodent studies pair these tools with behavioral assays that model facets of human behavior such as cognition, stress responses, or motivated behaviors. Many assays have undergone extensive development for face and pharmacological validity. However, sometimes behavioral assays do not work as expected. This can occur when establishing an assay for the first time or when a lab moves to a new institution. Sometimes behavioral results can change for no apparent reason. The question of “why didn’t my behavior experiment work?” is likely familiar to many readers. The troubleshooting process can last weeks or even months. Sometimes experimental outcomes can be affected by factors that are not considered when designing an experiment. The time of day observations are performed, how animals are housed in the vivarium, or the social environment of those animals can dramatically alter the outcomes of rodent behavioral experiments. However, some of these variables are often not reported in publications, so investigators may be unaware of how these “hidden variables” affect their work. An additional complication is that in many institutions, investigators have little control over vivarium conditions that could directly or indirectly impact their results. Here we review how light cycles, transportation stress, cage ventilation, and the social environment impact measures of brain function and behavior relevant to mental illnesses and substance use disorders (Table 1). In some cases, investigators will have the freedom to optimize these variables for their research. However, in other situations (such as vivarium conditions) investigators may have less control over “hidden variables”. The empirical results reviewed below may help provide compelling arguments for change. Alternatively, in some cases low-cost practical steps can be taken to adapt to research conditions that cannot be altered. We suggest that improved reporting of these variables in publications could improve the external validity and repeatability, in much the same way that better validation and reporting of antibodies improves protein analyses [5].

Table 1.

Hidden variables in neuropsychopharamcology research, their impact on study outcomes, and suggested solutions.

Problems	Impact	Solutions
Light/dark phase Circadian effects	Physiological effects [8–10, 13, 18, 21–23] Drugs interaction effects [13–17, 20] Learning tasks [17, 25, 26] Pain perception [24]	Testing behavior during dark phase [10] Keeping all types of lighting dim during testing, including red light [11, 12]
Transportation Stress	Immune function [29] Reproduction [40, 41] Development [33–35]	Breed animals in lab or avoid shipping during puberty Consider “two-hit” model if shipping pregnant animals [45] Allow time for animals to acclimate before testing [31, 32]
Home Cage Environment	Cold Stress [51, 53] Hormone levels [52, 55–57], Behavior [60–64]	If forced air caging is unavoidable, add extra bedding or enrichment for thermoregulation [66, 68] Avoid behavioral testing on cage change days [60, 61] Intermix experimental groups that are housed on separate levels [63]
Social Environment	Hormone levels [76, 77] Aggressive behaviors [73–75, 78] Evolutionary history [71, 72, 91–93]	Record social ranking within cage [64, 82, 87, 88, 90] House animals based on species specific behavior Use alternative strategies to house animals after surgery recovery

Light cycles and circadian effects

Problem

In a laboratory, the question of when to conduct experiments is often viewed as a logistical one driven by personal preferences and/or feasibility. However, the time of day experiments are conducted can exert a major impact on results. Rodents are typically housed on a 12-hr light/dark cycle, with lights out occurring in the evening to accommodate animal care staff. Under this schedule, most experiments conducted during regular work hours occur when many rodent species would normally be sleeping [6], as most mouse and rat strains are descended from species that are most active during dusk and dawn [7]. Of special relevance for animal models of substance use disorders, pharmacokinetic functions such as drug absorption, distribution, metabolism, and excretion have circadian variability which can affect the toxicity and efficacy of pharmacological compounds [8]. Circadian rhythms in hormones such as glucocorticoids [8] and gene expression within the brain [9, 10] can strongly modulate behavioral responses to drugs or behavioral challenges.

Systemic amphetamine administration has a much higher toxicity in male rats during the inactive light phase compared to the active dark phase [13]. Similarly, oral self-administration of ethanol in mice does not achieve clinically relevant blood ethanol concentrations unless access is restricted to the beginning of the active dark phase [14]. This is because if ethanol is available ad libitum, mice consume the same amount of alcohol spread over a 24 hr period which results in consistently low blood ethanol concentrations. These time-of-day effects can modulate the behavioral effects of pharmacological manipulations across a wide range of contexts [15]. In mice, behavioral sensitization to cocaine is stronger at the beginning of the light phase versus the end [16]. Time of day effects on self-administration, conditioned place preference, and locomotor sensitization have been reported for a wide range of psychoactive substances[17]. Similarly, orexin-A is more efficient at initiating food-intake when administered during the light phase, an effect that is attenuated when administration occurs at the onset of the dark period [18]. This effect is likely modulated by effects of orexin on arousal [19], which are stronger in the inactive light phase when animals asleep. In contrast, acute effects of antidepressants on locomotor activity were maximal during the light phase, suggesting that the circadian activity of neurotransmitter systems may modulate the function of antidepressants [20]. The immune system also exhibits circadian rhythms [21, 22]. In rats, immune responses to pathogens are strongest during the dark active phase [23]. Therefore, an immune challenge performed during the inactive light phase may underestimate inflammatory responses. Other affective processes such as pain perception are modulated by circadian rhythms. In a mouse model of neuropathic pain, the effects of the analgesic gabapentin were strongest at the end of the dark phase when expression of its target molecule (a calcium channel subunit) peaked in the spinal cord [24]. Acquisition of many hippocampal-dependent learning tasks such as water maze [25] and contextual fear conditioning [26] are faster during the inactive light phase in rodents. In contrast, recall of cued fear conditioning is stronger early in the light phase [26]. Any task that involves learning could be affected by the time of day during which tests are performed, therefore time of day when experiments are conducted should be carefully chosen. Moreover, recent data suggest that traditional methods for performing behavioral studies in the dark under red light should be revisited [11, 12].

Solutions

For many experiments, it will be optimal to conduct studies during the first half of the dark phase, when rodents are more active. A common solution for performing behavioral observations in the dark phase is to shift light cycles to the afternoon and to use red lighting so that humans can work in dark rooms. Rats and mice do not have L cones in the retina, so it has long been assumed that wavelengths greater than 700 nm were undetectable. Recent data challenge this assumption and show that red light can generate retinal responses [11, 12], impact circadian rhythms, and possibly sleep. This creates a conundrum for investigators seeking to perform experiments in the dark phase, as laboratory staff need to see to be able to handle animals in the dark phase. These studies found that utilizing dim, diffuse red background light (as opposed to headlamps) can reduce animal exposure to variable light intensities and can minimize the impact of red light exposure. Using infrared light sources (which are not detected by the visual system) and cameras allows investigators to observe behavior without the use of red light. Planning experiments that optimize appropriate circadian timing and lighting conditions for experiments may help increase the translational potential of these studies. Accurate reporting of these variables in publications is also essential so that studies can be compared [10].

Transportation stress

Problem

A big expense for rodent research programs is the “per diem” cost for maintaining cages. For many, it is cost-effective to have animals shipped from a vendor rather than bred onsite. The process of shipping exposes animals to vibration, altered humidity, and disrupted circadian rhythms [27, 28]. This multimodal stressor exposure can decrease body weight, alter immune function, and increase cortisol levels in several mouse and rat strains [29]. Even transporting animals within a facility can expose animals to abrupt changes in accelerative forces [30]. Some of these effects are temporary and can be mitigated with a short acclimation period [31, 32]. However, shipping can induce enduring stress responses that can impact the results of experiments. The age at which animals are shipped has emerged as a critical variable that moderates the impact of this stressor on brain function and behavior. Rats and mice are commonly shipped as young adults, but investigators should use caution to avoid shipping animals during pubertal development. Both male and female C57BL/6J mice shipped at 6 weeks of age had disrupted sexual behavior when tested at 3 weeks later [33]. The disruptive effects of shipping were absent if mice were shipped at 3 or 12 weeks of age. In general, adolescent development is a period of heightened sensitivity to different forms of stress [34, 35]. Although the exact mechanism through which shipping stress impacts brain function and behavior is unclear, a lipopolysaccharide (LPS) challenge has similar behavioral effects in female mice between 4–6 weeks of age but not at 3 weeks of age or after 8 weeks of age [36]. If immune activation is a key mechanism of shipping stress on behavior, then shipping is likely to affect other affective and cognitive processes. Exposure to an LPS challenge during adolescence also affects performance in forced swim [37] and tests of cognitive function [38].

A second developmental window that shipping stress can impact is the prenatal period [39]. A common practice in developmental studies is to ship dams while pregnant, which allows experiments to begin quickly. Several lines of evidence suggest that shipping stress can impact both the dams and the offspring. Inbred DA/HAN rat dams that are shipped during their pregnancy show a decrease in maternal care [40]. Sprague-Dawley rat pups born to dams that were shipped at gestational day 9 (G9) were more susceptible to seizures than pups from dams that were shipped at G16 or non-shipped controls [41]. Stress during this critical developmental timepoint may increase sensitivity to stress later in life [42]. One possible mechanism is through altered maternal behavior, which can then change the sociality of their offspring once they mature [43].

Finally, shipping stress could also have an important impact on investigators studying the biology of aging. To reduce expense, aged animals are frequently purchased directly from commercial suppliers [44], but it is unclear if shipping stress has an exaggerated impact on brain function or behavior compared to young adults.

Solutions

A pragmatic solution for many investigators will be to ship animals either before puberty begins or after this developmental period ends. For studies examining the effects of early life stress on the development of offspring, it may be reasonable to consider these approaches as “two-hit” models that involve both the prenatal shipping stress exposure alongside the planned experimental stressor. Both prenatal or pubertal windows are periods of significant plasticity in brain structure and function [45]. Thus, stressors during these times can lead to enduring changes in neural circuits that control behavior. There is strong support for the translational relevance of two-hit stress models, as women exposed to one or more childhood adversities have higher risks for stress-induced depression and anxiety disorders in adulthood [46].

Home-cage environment

Problems

Investigators typically have a large degree of control over the conditions in which neural or behavioral data are collected. Investigators choose the size of a testing apparatus, lighting conditions, and sound levels. In contrast, the location where animals are housed in a vivarium is often largely defined by animal facilities or local animal care and use committees. For example, filtered cage covers and individually ventilated cages have been implemented to protect animals from pathogens and reduce costs [47]. Even though investigators often do not make the choices on how their animals are housed, these variables can have important impacts on experimental results [48]. Indeed, cage ventilation and bedding have important impacts on behavior and brain function, which could contribute to variable results across laboratories and institutions.

Individually ventilated cage systems (IVC), also known as forced-air systems provide air flow to each cage individually and allow for a highly efficient housing system. While this system is hygienic and convenient for animal care staff, the high rate of airflow is not a condition that laboratory rodents have evolved to live with. Although some studies report no differences in behavioral assays between IVC versus open cage systems [49], other studies have found more substantial differences [50]. For example, an IVC system can induce cold stress in single-housed mice as quantified by nonshivering thermogenesis that is supported by brown adipose tissue [51]. Mice housed on an IVC system also were found to have suppressed cell-mediated immune function and increased stress-induced corticosterone responses compared to mice housed in static cages [52]. In light of these findings, it is not surprising that when given a choice between a cage with a high (60 air changes per hour) forced air and one with no forced air, mice showed a preference for the unventilated cage [53].

Bedding is another variable that is often overlooked but can alter experimental results. Common forms of bedding include aspen chips, shredded paper, or corncob. It may be a surprise to some investigators that many rodents consume their bedding, an effect that is exaggerated when food is restricted [54]. Thus, a rodent’s diet is not limited to the food provided. Corncob bedding in particular is readily consumed by rodents, even when rodent chow is provided ad libitum. This is significant because corncobs contain estrogen-like compounds called tetrahydrofuran-diols (THF-diols) that do not bind directly to estrogen receptors but modulate estrogen signaling [55]. The effects of corncob bedding on estrogen-dependent behaviors can be substantial. In one study of California mice, aromatase inhibitors increased aggression if mice were kept on a shredded paper bedding but decreased aggression if mice were kept on corncob bedding [56]. Mice housed on corncob bedding also had reduced estrogen receptor alpha immunoreactivity in the limbic system and hypothalamus compared to mice housed on shredded paper bedding. In female California mice, corncob bedding reduced but did not eliminate effects of social defeat stress in a social interaction test [57]. Other forms of wood-chip bedding can produce dust and endotoxins [58] that may enhance inflammatory responses [59]. Investigators should be mindful of how the routine husbandry of changing cages can impact results. Standard cage change protocols where bedding is entirely replaced induces significant stress responses in rats [60] and mice [61].

Finally, the position of cages on a rack can influence both physiological variables and behavior. For example, male mice that were housed on the top shelves of a rack (elevated from the floor) were more reactive (escape behavior, urination, defecation) when handled by an experimenter than mice that were housed in the middle or bottom shelves [62]. Similarly, rats housed on higher shelves exhibited more anxiety-like behaviors in open field and black/white box tests [63]. This increased reactivity could be modulated by increased aggressive interactions among cagemates, which is also elevated in cages kept on higher shelves of a rack [64]. The mechanisms driving these shelf effects are unclear, but could be related to light levels, which are significantly brighter at higher shelves than lower shelves [65].

Solutions

While investigators may have limited control over vivarium infrastructure, careful consideration of enrichment provided in the home cage may help mitigate some of the unintended side-effects of IVC systems. The aversion to forced air caging was eliminated when a nest box or additional enrichment for nest building was added to cages [53]. Increasing the depth of bedding may also be an effective strategy as cages containing more bedding are preferred by mice [66] and reduce baseline corticosterone levels [67]. Improving species-appropriate access to environmental enrichment does not increase phenotypic variability in many research designs [68]. In addition, cage changes should be avoided leading up to behavioral testing days and experimental groups should be intermixed on the same racks and shelves to control for light level differences.

Social environment

Problem

The social environment has profound effects on immune function, stress responses, and the perception of pain [69, 70]. These physiological effects in turn modulate behavior and impact study outcomes. For both mice and rats, the standard approach is to house individuals in same-sex groups, even though there are important species and sex differences that modulate the impact of social housing conditions. Both male and female rats adapt well to pair housing, which is not surprising as most rat strains are derived from species that under natural conditions live in groups [71]. In contrast, the most commonly studied mouse strains are derived from Mus musculus, a species in which males defend distinct territories and females move across territories [72]. Under laboratory conditions, females from most strains of mice adapt well to group housing. However, in many lines of mice, group housing of males results in aggressive interactions. This effect is pronounced in male BALB-C [73] and CD-1 mice [74] but is also present in widely studied C57BL/6 lines [75]. Differences in social status between males can generate phenotypic variability that is more substantial than the estrous cycle in females [76]. For example, the most dominant male mouse in a cage can have elevated testosterone levels [77] while more subordinate individuals have elevated corticosterone levels and a higher probability of injuries [78], which can induce immune responses. Differences in social status can generate variability in experimental outcomes. For example, effects of chronic stress on drug seeking behavior were stronger in subordinate mice compared to dominants after chronic stress exposure [79]. Thus, if unaccounted for, social status may generate variability in experimental outcomes and reduce statistical power of studies.

Solutions

Several strategies can be used to either reduce aggressive interactions or integrate social status into a research program. Aggressive interactions can be reduced by housing animals with same-sex siblings, transferring nesting material during cage changes, or reducing cage changes with spot cleaning [80]. Environmental enrichment can have variable effects on aggressive behavior and its implementation must take into consideration the needs of each sex and species [68, 81, 82]. Moreover, it should be used in conjunction with behavioral observations to confirm whether aggression decreases [82]. While single housing for male mice can produce a short-term anxiolytic phenotype [83], it is not an effective long-term solution. Prolonged single housing is associated with increased depression- and anxiety-like behaviors [84], spatial memory deficits [85], and impairment of wound healing [86]. Alternatively, investigators can determine social rankings within a cage and control for this variability statistically. One good method is the tube test, which determines social rank by quantifying the extent to which an individual mouse forces an opponent backward in a narrow tube [87, 88]. These rankings are often maintained by aggressive interactions that may not be visible by routine checks for visible wounds [64]. Since social stress is an important risk factor for many forms of mental illness and substance abuse, comparing dominant and subordinate individuals provides a unique approach for assessing stress-induced differences in brain function and behavior [89]. In particular, the medial prefrontal cortex has been found to play a critical role in modulating social structure in both rodents [90] and primates [85]. Thus, integrating measures of social rank into mouse studies can both help control for variation in social status and provide additional insights into how social status affects behavioral outcomes.

Although domesticated mouse and rat lines are used for the majority of rodent studies, there are many other rodent species with different social systems that are well suited for behavioral pharmacology studies. The Syrian hamster is a solitary species [91] in which single housing does not appear to be a stressor [92]. In contrast, prairie voles generally live in family groups [93] and are especially sensitive to social isolation [94]. Similarly, effects of social contact on immune function are stronger in monogamous Peromyscus californicus compared to polygynous P. leucopus [95]. These species differences in social systems provide opportunities to study behaviors like pair-bonding [96], female aggression [97], and male parental behavior [98, 99] that are difficult or impossible to study in traditional rodent models. Experimental results that generalize across rodent species may be more likely to translate to human clinical conditions. Overall, we recommend that investigators consider how the social environment which their animals derive from will impact experimental results.

Conclusions

In this review, we highlighted how experimental results can be impacted by hidden variables that often receive less attention during the design of experiments (Fig. 1). These variables can play a key role in producing unexpected results, such as when control groups exhibit a stress-like phenotype in a behavioral assay such as elevated plus maze. We propose that considering the impact of circadian rhythms, transportation, vivarium housing, or social environments can provide a productive strategy for making sense of unexpected results. An important point is that there is no one “right” set of conditions for all investigators. However, for each investigator some conditions will be more optimal than others. A goal of this review is to help investigators chose those conditions (Table 1).

Fig. 1. There are many factors that can work together to impact an animal’s behavior.

Transportation, the social environment in the cage, the circadian rhythm, and home cage environment are some of the most common laboratory variables impacting the individual. One or several of these confounds can alter neural activity, parental behavior, social behavior, and behavioral testing to create inconsistent data.

One obstacle is that important methodological details are often not included in publications, making it impossible for investigators to directly compare experimental conditions. In the 2000’s there was a similar problem with the use of antibodies in neurobiological research, in which many publications did not even include catalog numbers. A solution to this problem was the development of research resource identification numbers (RRIDs), which provide unique identifiers to antibodies, model organisms, and tools [100]. Fortunately, existing reporting guidelines can be implemented to help bring the level of rigor for behavioral experiments to a similar level. For example, the ARRIVE (Animal Research: Reporting of In Vivo Experiments) guidelines were created to improve transparency and improve methodological reporting in in vivo experiments [101]. More recently, guidelines have been developed to improve the transparency of the reporting associated with specific animal models and housing environments [68, 102]. These guidelines include checklists that provide a guide for researchers to report key information in publications. These reporting guidelines are similar to the metadata required for the submission of gene sequencing data (eg BIOPROJECTS) or imaging data to the NIMH Data Archive. More complete reporting of experimental conditions could help increase the external validity of research. Improving the transparency of methodological details used in neuropharmacological studies will allow for better comparisons to be made across studies and determine the extent to which experimental manipulations are robust across different conditions. Thus, when evaluating “what went wrong with my experiment?”, we recommend that investigators consider and report the potential impact of the hidden variables on their experimental results.

Author contributions

HBS, ACK, and BCT wrote and co-wrote the paper.

Competing interests

The authors declare no competing interests.