The Elephant in the Room: Recognition and Documentation of Personnel Practices That Confound Reproducibility

Abstract

The ability to apply findings from animal studies efficiently and effectively is predicated on an understanding of biology and pathobiology, how that biology relates to the human systems being modeled, and how the studies are conducted and reported. This overview discusses various factors in research within the animal environment (referred to as extrinsic factors) that the NIH now expects to be documented to foster replicability in science and expand interpretations of study outcomes. Specifically, an important extrinsic factor in research with animals is that of individual personnel who perform handling practices, participate in research interactions, and share an overall presence in the housing facility with animals, all of which can confound reproducibility efforts in biomedical science. An improved understanding of the influences and behaviors of animal research personnel on animal responses is critical with regard to research results and the interpretation of data collected from animal models of biomedical disease.

Introduction

The NIH Advisory Committee to the Director Working Group (ACD WG) on ‘Enhancing Rigor, Transparency, and Translatability in Animal Research’ completed its charge in 2021 to provide recommendations for how the NIH could improve the value and reproducibility of animal-based research.⁸⁰ This focused federal effort to enhance and restructure aspects of animal research complemented a substantial number of reports, guidance, and reviews on research reproducibility from scientists^{3,17,20,41,44,48,65,66} and affiliated organizations, including the Institute for Laboratory Animal Research (ILAR),³² the National Center for the Replacement, Refinement, and Reduction of Animals in Research (NC3Rs),^36,56 NIH,¹⁰ the Federation of American Societies for Experimental Biology (FASEB),¹⁵ and the National Academies of Science, Engineering, and Medicine (NASEM).⁵¹

The recommendations of the ACD WG report were presented in 5 essential themes for advancing the quality and reproducibility of animal research through improved: 1) rigor of study design and data analysis; 2) reporting and research practices; 3) statistical modeling for appropriate selection of animals; 4) documentation of methods and results reporting; and 5) metrics of success (or effectiveness). Theme 4 emphasized the nuanced yet critical research variables (extrinsic factors) that directly affect animals and encouraged the NIH to alert research teams to these factors in the animal environment that can affect data outcomes and to emphasize an inclusive description of these factors in any published work.

The ACD WG report defines extrinsic factors as “housing, husbandry, handling, feed, water, bedding, enrichment, caging type, light cycles, etc. that have a direct impact on the experience of the research animal during the course of experimental phases.”⁸⁰ The ACD WG also specifically recommended in Theme 4 that organizations should support work to better understand, monitor, record, and report important extrinsic factors related to animal care.⁸⁰ By extension, a critical extrinsic factor is the people who interact with the research animals. The ACD WG report did not explicitly mention animal research personnel as contributing to irreproducibility; however, hindsight suggests that the ACD WG report would have benefited from the highlight of how individual practices of animal research personnel can directly influence experimental outcomes.

This overview highlights the ‘elephant in the room,’ or the metaphorical idiom that brings to light an acknowledgment that animal research personnel themselves confound reproducibility efforts, even if every other aspect of a study is controlled. An improved understanding of the influences and behaviors of personnel on animal responses is critical with regard to research outcomes and general interpretation of data collected from animal models of disease. This review intends to increase awareness of the inherent influence of personnel on study outcomes and to provide suggestions for mitigating personnel-related variability when possible.

Extrinsic (Environmental) Factors in Animal Research

Environmental and husbandry-related factors that influence animals in research have been well recognized in guidance published by ILAR and FASEB.^15,32 FASEB recommendations on enhancing research reproducibility indicate that “the organization, daily operation, environment, and staffing of an animal facility can affect the outcomes of experiments” and encourage stakeholders “to identify, understand, and promote the adoption of evidence-based husbandry practices” with subsequent “reporting of animal care practices and study-specific variations” as essential to enhanced research reproducibility.¹⁵ Stakeholders in promoting reproducibility of animal studies include the investigators and research staff who perform the experimental work, the animal facility staff who perform husbandry work, the veterinary staff, and the regulatory oversight personnel. Together, these groups provide unique perspectives on animal science and medicine, participate in experimental procedures, determine the appropriateness of the animal model to the question or hypothesis under study, and aid in minimizing adverse effects on and promoting the positive experiences of the animals involved.

Most contemporary animal research facilities are equipped with temperature- and humidity-controlled housing rooms and caging systems that minimize potential confounding variables from the environment. The facilities are overseen and maintained by trained personnel who adhere to well-conceived and regulated operating procedures.³² Providing consistency in both the microenvironment (the animal’s primary enclosure) and the macroenvironment (the physical conditions surrounding the microenvironment, including the room lighting, air quality, etc.) is beneficial to animal well-being and the quality of research data obtained from the animals.^13,26,30,43 Recommendation 4.3 from the ACD WG encouraged the recording and reporting of extrinsic factors, including aspects of animal handling, housing, husbandry, transportation to housing facilities, and ambient environmental parameters (for example, temperature, light, humidity, noise).^55,59,78

Recent publications in research animal science provide overviews of influential extrinsic factors like housing room light cycles,²⁴ social and behavioral factors,⁷⁹ and the animal’s microbiota,¹⁶ which is strongly influenced by feed type and nutritional composition. Drinking water is rarely considered in the experimental design, but recent evidence indicates that the source, microbial and chemical contaminants, and purification methods can potentially result in experimental variability.^5,22,38 In addition, a growing area of study in research animal sciences is the evaluation of enrichment substrates that are intended to promote species-specific behaviors.⁶⁰ Bedding and enrichment items can affect behavior and physiologic parameters in any species and should be disclosed in protocols and publications to aid with reproducibility efforts.^37,76 Even more granular impacts of the housing room environment, like cage placement on racks, opacity of cages, and relative distance from human activity within the room can affect rodent behaviors and responses; therefore, these factors should be taken into consideration in the planning and reporting of research.⁹ Sources of unanticipated noise and vibration within animal facilities also warrant consideration for their potential adverse effects on animal health.^63,64,73 With awareness of the influences of the animal environment in research settings, one can better appreciate the challenges of capturing and documenting with precision the details of the myriad extrinsic factors.

Influences of Personnel on Research Animals: Handling, Presence, and Odors

Multilaboratory studies investigating the confounding effects of research environments have found that despite rigorous standardization of housing conditions and study protocols, significant variations in outcomes occur across laboratories.^11,77 A lack of standardization of research methods (for example, with respect to timing and invasiveness of procedures, or specifics of housing conditions) and differences in types and brands of research equipment also contribute to irreproducible research.^20,34,42 In these multilaboratory studies, the individuals employed across labs were different people and the execution of procedural steps was dependent on their individual practices.^7,47 The reality of inherent variability in labs was reinforced in 2018 by the following blog post: “…differences between laboratories are unavoidable – the animals are different, the people interacting with the animals, the animals’ microbiome, their sensory perceptions and experiences, all of which may affect the animals’ phenotype and thus the outcome of the study.”⁸¹

A recent study attempted to control for animal and personnel variability across 6 laboratories by using the same sex (male) and strain (C57BL/6J) of mice from 6 different breeding sites and the same female experimenter to compare a standardized and a heterogenous study design.³⁵ Multiple outcome variables, both physical and behavioral, were assessed in mice at all locations, yet despite intense efforts toward experimental controls and consistency, the results revealed that phenotypic variability across labs was nonetheless high. The authors concluded that experimenter (personnel) influence was likely part of the variability (despite having the same person conduct all procedures in all 6 labs) and highlighted that the aspects of housing and husbandry that varied between laboratories (for example, cage ventilation, cage types, environmental enrichment, and animal care) must have influenced outcomes.³⁵ Indeed, these varied housing and husbandry factors are included in the definition of influential extrinsic factors by the ACD WG, reinforcing once more that despite attempted standardization of all obvious ‘knowns’ in an experiment, many unrecognized and undocumented factors continue to confound scientific reproducibility.

In truth, should all other extrinsic factors and variables remain consistent, an animal facility or research lab cannot be staffed such that the same individuals will interact with the animals in the same manner at the same time every day of the week: animal research personnel cannot be standardized. Nonetheless, aiming toward consistent personnel practices (like time of room entry, time of intervention, and approach to handling) may aid in minimizing inconsistent responses to personnel by research animals.

One of the most common research interactions with animal species is between animal handler and animal at times of cage change, experimental manipulation, and physical examination. These types of interactions, while integral to research practices, are also a source of stress for the animals.⁴ For example, active transfer of mice into a cage leads to a greater increase in plasma corticosterone levels than compared with passive transfer without handling; as well, it has been shown that rats respond differently depending upon cage-change techniques.^46,62 Approaches to and training in animal handling can influence both animal welfare and data and may contribute to unexplained variation in research findings and reproducibility in certain mouse strains and between sexes.⁵² The method of grasping the tail base to restrain and move mice continues to be used routinely, despite evidence that it induces stress²⁷; therefore, animal research personnel should consider incorporating less aversive, or even nonaversive, and low-stress handling methods into animal research practices to facilitate positive human-animal interactions and to prioritize animal welfare.

Gentle handling of rodents has been shown to benefit both behavior and interactions with experimenters.^8,21,50 Use of inanimate objects to transfer rodents between cages and surfaces (for example, tunnel handling) or gentle scooping and lifting of mice with palms of hands (also known as cupping) without direct physical restraint are refined methods of handling that can reduce anxiety-like behaviors and improve animal welfare, with greater interest in human interactions across a variety of experimental protocols.²⁷ Concerns regarding the implementation of nonaversive and tunnel handling methods by animal research personnel have included the potential for increased time in handling and cost in the purchase of handling equipment.^28,53 However, a more rigorous evaluation of these issues exposed research benefits of improved pup production and reduced loss of litters indicative that reduced handling stress provided substantial benefits to breeding operations with only a minor increase in time for gentle handling.³¹ Establishing positive personnel interactions with rats is best accomplished in younger pups or newly acquired young animals by manual ‘tickling’ to habituate animals to human interactions. During this acclimation phase, personnel can expose rats to interactions that simulate rough-and-tumble play by young rat pups.³⁹ Tickling has been shown to efficiently and practicably reduce rats’ fearfulness of humans and improve animal welfare by reliably creating positive affective states.⁸

Specific pathologic effects of routine manipulations and handling are largely undocumented, and researchers may be unaware of adverse health concerns associated with common manipulations of mice that require restraint, such as injections, blood sampling, and tumor monitoring. One study reported a postmortem assessment of 1,000 mice used in research, with 864 having been heavily manipulated and 136 being handled only for routine husbandry procedures. Mice with heavy manipulation were defined as requiring necessary and repetitive physical restraint or routine injections and handling throughout experiments. Osteoarticular lesions were found in about 7% (61 mice) of heavily manipulated mice, and in a single unmanipulated mouse, demonstrating a highly significant association with repeated restraint, forced handling of mice and the presence of traumatic lesions.² In this regard, the fractures and dislocations noted in this study present substantial and unexpected confounding factors, including unaddressed pain and distress in animal research subjects.² Training animal handlers to use refined and gentle techniques is critical to improve animal welfare and research outcomes and has been cited as the primary area for welfare improvements in a survey of research animal veterinarians.^2,45

In addition to handling methods, sex-associated olfactory stimuli (particularly from male humans) induce a stress response in mice and rats, leading to blunted indicators of pain behavior and increased indicators of anxiety.^6,67 The effect of male personnel on rodents was established by olfactory exposure of mice to shirts worn by men, exposure of bedding from intact and unfamiliar male mammals, and presentation of human cell secretions (for example, armpit sweat) from men.⁶⁷ Mice demonstrate a preference for the scent of female personnel and experience more stress from male personnel.¹⁹ With the realization that rodents are adept at odor detection, personnel should be discouraged from wearing heavily scented products (for example, cologne and body lotions) within animal areas. Overall, the ability of research animals to differentiate the sex of human experimenters can have measurable effects on behavioral and/or biologic responses. The presence of familiar personnel, with whom animals have interacted positively, can alleviate anxiety-like behaviors and increase consistency in results from animal tests.⁷⁴ This finding supports having a consistent group of personnel working in the facility areas and interacting with the animals throughout studies. The caveat to striving for consistent animal research personnel is that one should recognize that the simple presence of observers/experimenters influences expression of normal behavior in a wide variety of species. From a search of the keywords “observer influence on research animals” in NIH’s National Library of Medicine database, the hits included >36,000 articles on this topic (accessed 4 March 2024) in species as diverse as baboons and rhesus macaques, sheep and pigs, cats, and dogs, species of birds, poultry and rabbits.^57,58

To Err Is Human: When Animal Work Itself Is the Source of Distraction

Adherence to scientific protocols, or following steps in a planned experiment, is akin to following a recipe. In this analogy, one can envision that in following instructions to make, say, chocolate chip cookies, some bakers might be inclined to level off the flour and baking soda to the exact metric measurements, while others might estimate or round off the ingredients, adding in a bit of extra sugar, vanilla, or a handful of supplemental chocolate. The recipe might suggest a range of oven temperatures and baking times to achieve the desired outcome, introducing variability. Should the baker become interrupted or distracted from the timer, inattention to the recipe may lead to unintended outcomes (like a forgotten or improper ingredient, or a burnt cookie). Other nuanced variables, like sources of ingredients, supplies, equipment, and attention to detailed instructions, will result in diverse (inconsistent) appearance of the baked product; however, despite all potential nuances, the final cookies should be generally recognizable and identified by taste and overall appearance as chocolate chip.

This cookie recipe analogy illustrates the scientific concept of accepting generalization over reproducibility. These two terms are often used interchangeably but in fact are distinct: methodologic reproducibility requires the complete and transparent reporting of information required for exact replication of protocols and methods,²⁰ while generalization refers to the application of the results to other contexts or populations, or the ability to apply a specific result more broadly across settings, systems, or other conditions.^15,80 Achieving exactly the same taste and appearance of a chocolate chip cookie across numerous kitchens and by any number of bakers would be impossible, requiring infallible methodologic reproducibility and standardization.^20,59

Even with exact replication of methodology, equipment, and materials, confounding effects of individual human behaviors would still result in imprecise replications of effort. Just as occurs in research settings, the same is true for any discipline in which human beings are involved in the outcomes; as the adage states, ‘to err is human.’^12,33 The term ‘err’ in the context of this overview is meant to imply inconsistency, not necessarily mistakes with or disregard for animals. This extension of the famous quote clarifies that “personality, gender, motivation and other constitutional factors…give rise to variation, which in turn begets uncertainty and unpredictability.”¹²

In the contemporary workplace, inconsistent behaviors have been linked to human factors (for example, fatigue, stress, and inadequate knowledge) and environmental and organizational factors (ineffective management, distractions, and poor teamwork).¹ Distractions, leading to loss of detailed focus on the task(s) at hand, can be heightened due to the use of headphones and cellphones by persons within animal rooms or by interruptions from conversations with coworkers in animal spaces. It is recommended that research facility expectations include outlined efforts to minimize distractions for personnel when working with animals.

Careers that involve work with animals are rewarding and provide positive interactions and experiences for human personnel.⁷⁵ However, persons working globally in animal research can experience compassion fatigue as a result of caring for excessive numbers of animals; unfortunately, the experience of compassion fatigue was exacerbated by the COVID-19 pandemic.^23,40,54,72 Compassion fatigue in veterinary medicine is defined as an occupational hazard in which empathy, compassion, and caring for others are at the core of practice and can wane over time, with consequences that should not be underestimated.^68,69 Signs of compassion fatigue can include difficulty sleeping (which can lead to distraction and increased errors in the workplace), chronic ailments, lack of empathy, depression, and burnout.^23,68,71 Research also suggests that the extent of compassion fatigue and cognitive dissonance among animal welfare professionals likely varies with the nature of the job.^14,29 Effective coping mechanisms to improve the mental health of animal care workers can include social support, breaks for physical activity, balancing personal time away from work, and self-care; as such, research institutions should seek to provide necessary support and resources for animal research personnel to address compassion fatigue and mitigate behaviors that might lead to errors and inconsistencies during interactions with animals.^{49,54,61,70,71,75}

Final Considerations

The recognition of the importance of extrinsic factors in research has emerged as a discipline called therioepistemology.¹⁸ Given that human conditions of disease are not standardized as they are studied, the study of animal conditions should not be expected to be identical for every investigation. Although the recognition and documentation of extrinsic factors may be perceived as burdensome, much of the data regarding daily environmental parameters and husbandry efforts are available in existing animal program records (for example, AAALAC program descriptions and daily housing room checklists). These helpful data points can be provided to research teams by those involved in the delivery of animal care, including veterinary staff, facility and operations personnel, and compliance and regulatory oversight teams within the institutional animal care program.

Animal housing, handling, and husbandry will never be standardized fully across institutions, because uncontrollable factors (for example, the need to reassign animal care personnel to different rooms or facilities or the hiring of new animal researchers, the inherent diversity in building and facility ages, the nuances of HVAC systems, and the weather/seasonal changes, caging, equipment, and region-specific vendors) will always be present. The NIH ACD WG advised that specific subcategories of the ARRIVE guidelines be disclosed when providing details in animal study designs, grant submissions, and progress reports. Specific vivarium factors are outlined in Item #8 (Experimental Animals) of the “ARRIVE Essential 10” and in Items #15 (Housing and Husbandry) and #16 (Animal Care and Monitoring) of the “ARRIVE Recommended set.”^25,56 With consistent access to measurements and details of environmental factors, investigators can retain important experimental information for their data files, with the hope that more data will inform better decisions and lead to improved findings. In particular, if deviations from predicted research outcomes occur, these can be explored, addressed, and reported in the literature.

In response to the 2021 WG recommendations⁸⁰ and the 2022 NIH Extrinsic Factors Workshop,⁵⁵ the NIH intends to fund studies of extrinsic factors, as announced by the NIH Council of Councils in Fall 2023. These federal resources will address knowledge gaps and further clarify the impact of extrinsic and environmental factors on research. Despite limitations of methodologic reproducibility in the face of individual human behaviors and other nuanced factors, research involving animals as models of disease continues to be an essential component of biomedical discovery and prize-winning medical advances for both human and animal health. Given the inherent variability in how studies are designed and executed by personnel who work directly with animals, generalization (over reproducibility) is likely the most achievable expectation for application of biomedical discoveries.

Conflict of Interest

The author has no conflict of interest to declare. The opinions expressed are solely mine and not those of my employer.

Funding

The writing of this overview was not supported by internal or external funding sources.