Abstract
In June 2015, the National Institutes of Health (NIH) released a Guide notice (NOT-OD-15-102) that highlighted the expectation of the NIH that the possible role of sex as a biologic variable be factored into research design, analyses, and reporting of vertebrate animal and human studies. Anticipating these guidelines, the NIH Office of Research on Women’s Health, in October 2014, convened key stakeholders to discuss methods and techniques for integrating sex as a biologic variable in preclinical research. The workshop focused on practical methods, experimental design, and approaches to statistical analyses in the use of both male and female animals, cells, and tissues in preclinical research. Workshop participants also considered gender as a modifier of biology. This article builds on the workshop and is meant as a guide to preclinical investigators as they consider methods and techniques for inclusion of both sexes in preclinical research and is not intended to prescribe exhaustive/specific approaches for compliance with the new NIH policy.—Miller, L. R., Marks, C., Becker, J. B., Hurn, P. D., Chen, W.-J., Woodruff, T., McCarthy, M. M., Sohrabji, F., Schiebinger, L., Wetherington, C. L., Makris, S., Arnold, A. P., Einstein, G., Miller, V. M., Sandberg, K., Maier, S., Cornelison, T. L., Clayton, J. A. Considering sex as a biological variable in preclinical research.
Keywords: gender, methods, sex influences, sex differences
Sex, defined as being XY or XX, is a construct derived from chromosomal complement, is associated with biologic functions (1), and is an important biologic variable in preclinical research. Recognizing this fact is vital because preclinical data inform the premise and design of clinical studies. Historical reliance on male vertebrate animals (e.g., rats and mice) in preclinical research (2, 3) has resulted in the generation of incomplete data available to guide clinical trials that include female participants. This is particularly problematic in view of current knowledge that sex affects health status, including disease presentation, pathophysiology, and therapeutic response. The National Institutes of Health (NIH) have re-emphasized the importance of rigor and transparency to reproducibility, including appropriate accounting for the potential influence of sex on experimental outcomes in preclinical research (4). NIH has underscored the need to study males and females in animal, tissue, and cell studies (4). In June 2015, the NIH released a Guide notice, “Consideration of Sex as a Biologic Variable in NIH-funded Research,” which sets forth the expectation that sex will be factored into research design, analyses, and reporting in studies of vertebrate animals and humans (5).
In anticipation of NIH guidelines and as part of an ongoing conversation with investigators, the NIH Office of Research on Women’s Health convened key stakeholders in October 2014 to discuss methods and techniques for integrating sex as a biologic variable in preclinical research. The workshop had a dual focus on sharing research results as well as lessons learned and promoting discussion on the importance of considering sex as a biologic variable in preclinical research. The workshop included 4 scientific sessions:
Session 1: The concept of including male and female subjects in studies.
Session 2: What is the impact of including or not including sex as a basic biologic variable?
Session 3: Practical methods to integrate the biologic variable sex intro research projects.
Session 4: Cultivating a culture of “sex matters” across multiple disciplines.
Several opportunities for immediate action were noted and are expounded upon in this publication, including decreasing variability via appropriate experimental design, employing factorial design and other methods to analyze multiple variables, and collecting and reporting sex-aggregated data. The workshop video cast and summary may be found at: http://orwh.od.nih.gov/sexinscience/researchtrainingresources/methodstechniquesbiovar.asp. This paper builds on key elements that were derived from the workshop and serves as a primer to help orient investigators to approaches for considering sex as a biologic variable in preclinical research.
IMPORTANCE OF INCLUDING MALE AND FEMALE ANIMALS IN RESEARCH
The concept that both sexes should be considered in preclinical research is not new. A report published in 2001 by the Institute of Medicine Committee, Understanding the Biology of Sex and Gender Differences, recommended: 1) inclusion of sex as a basic variable in biomedical research 2), use of human-relevant research models to assess sex-related differences at levels of organization that ranged from the molecular to the whole organism, and 3) consideration of the entire lifespan of the organism (1). The biologic basis for this recommendation is incontrovertible: sex is established genetically at conception, sexual differentiation ensues, and intrinsic existence and extrinsic interactions of an organism are mediated by sex throughout life. Nevertheless, recommendations to include both female and male animals in preclinical research have been incompletely embraced by the research community, as evidenced by the continuing general exclusion of female animals from preclinical research. Furthermore, general guidance for such implementation has not been uniformly developed or applied (3).
POTENTIAL WAYS TO CONSIDER SEX AS A VARIABLE IN PRECLINICAL RESEARCH
Consider sex influences
NIH policy on consideration of sex as a biologic variable is a component of an NIH initiative to enhance the reproducibility of preclinical research via rigor and transparency and is grounded in the guiding principle of studying both sexes in biomedical research. NIH expects that sex as a biologic variable will be accounted for in design, analysis, and reporting of research in vertebrate animals and humans. In particular, this approach serves to expand the foundation of knowledge about male and female biology as well as to enhance the understanding of the applicability of research findings to males and females. The policy does not require use of a specific experimental design or a defined statistical analytic approach (6); rather, it provides the flexibility for development of appropriate design and analyses on the basis of the research question and the scientific context.
In considering how sex may influence the biologic process under study, one might begin by considering the translational context and clinical relevance by asking the following:
Is it known that the disease process or event applies to only males or females?
Is there evidence of male/female differences in humans in the incidence or prevalence, presentation, treatment response, or morbidity or mortality of the condition or disease of interest?
If there is no difference reported in the literature, is this because it has not been studied or reported?
How might sex influence the processes, pathways, and/or phenomena under study?
Incorporation of sex as a biologic variable can enhance research in several ways. In the simplest way, it should lead to better reporting the sex of animals and cells used in research, which would at least improve the chance to appreciate which sex was studied. A second way would be the study of outcome measures (e.g., effects of treatments) separately in each sex so that treatment effects would be more broadly known and could be applied to clinical studies of each sex. The third way is to compare outcome measures in females and males directly—and statistically—to establish whether there is a sex difference in treatment. Finding significant sex differences in treatment variables has the advantage that well-known sex-biasing factors (hormones, sex chromosomes, and environments) immediately become interesting candidates for factors that modulate or condition effects of treatment.
Studies of animal models can be particularly informative compared with studies of humans, because the sex variable can be broken down into its constituent parts, which are individual sex-biasing variables that affect physiology and disease. There are 2 variables that are constitutively different between males and females: the sex chromosome complement (XX vs. XY) and gonadal hormones (ovarian vs. testicular secretions). In animal studies, each of these factors can often be manipulated independently of the others to determine the specific effects of each. There are some conditions in humans that result from altered expression of X and Y chromosomes; however, differentiating effects of chromosomes and hormones in humans is difficult, and, thus, animal studies provide an important window into the role of each separate sex-biasing factor. When a sex-biasing factor, such as estradiol or an X gene that is always expressed higher in XX than in XY individuals, is found to reduce a disease process, the sex-biasing factor itself and the downstream gene pathways that it influences become a possible therapeutic or drug target that can alleviate disease.
Consider the role of sex chromosomes
Female and male cells differ in their complement of sex chromosomes (XX vs. XY), which causes an inherent sexual imbalance of expression of X and Y genes in virtually all cells of the body. Although this imbalance was historically considered to have little effect in creating distinctions between males and females in physiology and disease, in recent years, sex chromosome effects have been shown to be surprisingly large in several mouse models of disease (7–10). Mouse models exist that are suitable for the detection of effects of XX vs. XY sex chromosome complement, independent of their role in causing differences in gonadal hormone levels between males and females (11, 12). Studies of these models have already provided evidence that both X- and Y-encoded genes and mechanisms can protect from disease in mice. The next step is to discover the specific X or Y genes that are protective and to understand the gene pathways that they regulate as a strategy for uncovering novel mechanisms that could be enhanced to ameliorate disease in both sexes. Because of the novelty of sex chromosome mechanisms, much remains to be learned.
Consider the role of sex hormones
Sex hormones—androgens (testosterone and its metabolite, dihydrotestosterone), pregnanes (progesterone and allopregnanolone), and estrogens (estradiol, estriol, and estrone)—are generally considered to be the steroidal hormones that are produced by the gonads, adrenal glands, and certain tissues, such as the liver or brain. The term sex hormone is nearly always synonymous with sex steroid or gonadal steroid. The glycoprotein hormones—luteinizing hormone, follicle-stimulating hormone, and gonadotropin-releasing hormone—are produced by the anterior pituitary gland and are usually not regarded as sex hormones, although they play major roles in the reproduction and development of secondary sex characteristics in both males and females (13).
Sex hormones are crucial for the development and function of the body as well as for the regulation of sexual differentiation, secondary sex characteristics, and sexual behavior patterns (13). Although estradiol, an estrogen, and progesterone, a pregnane, are generally considered to be female’ sex hormones and testosterone the ale’ sex hormone, all 3 sex steroids are present in both males and females, albeit at different levels in the 2 sexes. Production of sex steroids also varies across the lifespan and reproductive life stage.
Sex hormone receptors exist throughout the body, which suggests that hormones affect a myriad of body tissues directly. Estrogens most often form complexes with their receptors and various transcription factors, thereby interacting directly with the genome and influencing a broad range of cellular events (14). Studies of males and females and the role of sex steroids are most often undertaken by using experimental animal models. One way to begin to elucidate the role of sex hormones is to remove the gonads of both male and female adult animals and then perform comparisons. Another approach is to provide exogenous hormones to gonadectomized animals (15). If a difference between males and females persists in the absence of gonadal hormones, one would then consider whether this distinction is attributable to developmental or chromosomal effects (16).
Consider the role of sex and gender assumptions in study design
Gender refers to the behavioral norms and, in the case of humans, attitudes that influence individual action, expectations, and experiences. For both humans and other animals, gender is shaped by biology, environment, and experience. Researchers’ gender assumptions may play an important role in the design of animal studies. Conceptions of gender may influence how investigators construct biologic hypotheses and interpret outcomes. For example, assumptions about the role of sex hormones or prevalence of a disease state in males or females may influence choices concerning which sex to test. Erroneous assumptions in participant selection can have cascading consequences for the study as a whole. For example, researchers may assume that testosterone is a male hormone, even though androgens also have natural effects in females. Such assumptions may lead to a decision to test the relationship between testosterone and a chosen variable only in male participants. In this case, potentially important information is ignored about the contributions of androgens and androgen receptors in females. Alternatively, effects of aromatization of testosterone to estrogen in males has effects on bone, but other tissue-specific conversions may have implications in the heart and brain. Moreover, therapies that might involve administration of androgens to either sex might benefit from being considered in a larger context than the idea that androgens can only make females more like males, especially if the response of females to androgens differs from that of males. Furthermore, assumption that breast cancer is a female disease has led to the development of female-only animal models, which limits the opportunity to better understand breast cancer in males (17, 18).
IMPLEMENTING EXPERIMENTAL DESIGN
The purpose of preclinical research—using animal model systems, tissues, or cells—is to model and characterize human disease processes, uncover pathogenic pathways, identify and/or interrogate potential therapeutic targets, and test therapeutic strategies/interventions. However, investigators have often been unable to reproduce promising preclinical research results, with a recent report (19) estimating that roughly 53% of irreproducible preclinical results were caused by root factors categorized as study design and data analysis and reporting, which resulted in considerable loss of economic impact on biomedical research. A plethora of variables (e.g., sex of experimental material, age, housing type, genes, environment, health status, time of day, sex of experimenter, circadian or diurnal hormone cycles in males and females, noise level, order of testing of multiple dependent variables, laboratory protocol error, etc.) can induce variability in outcome of studies (20). Hence, critical aspects of experimental design for animal research, such as sex of participant, time of day, room temperature, test article, administration method, etc., should be systematically addressed and controlled to avoid generating faulty results that lead to erroneous conclusions. In this section, incorporating both sexes and controlling for sex will be addressed by focusing on ways to employ experimental and statistical control in preclinical study designs.
Experimental hypotheses may address the direct comparison of males vs. females to detect a difference between them, they may address the influence of an intervention on each sex, or they may address the interaction of sex with an intervention (i.e., Does the intervention work equally well in males and females or does being male or female influence the manner in which the intervention works?).
Explore effects in males and in females
To begin investigation of the potential influence of sex as a biologic variable when little is known about the influence of sex in the context of a particular animal model and research question, males and females can be incorporated into experiments with a modest increase in the total number of animals (21). For example, if an investigator randomly assigned 8 male rats to the control group and 8 male rats to the treatment group, he/she might consider using 10 male rats and 10 female rats, randomly assigning them in comparable numbers to control (5 males and 5 females) and treatment (5 males and 5 females) groups.
The investigator might want to explore the potential effectiveness of a treatment in both males and females. As a first step, an investigator may start by examining results for each sex—results analyzed separately for males and for females. For example, the investigator might begin by assessing overall measures of response (e.g., peak response, area under the curve, clearance) in males and females (22). Such an approach may provide an indication of a sex difference; however, if no obvious sex difference is detected, one cannot conclude that there are no sex differences because the study was not powered a priori to detect sex differences or different regulatory processes may result in similarities in outcomes (compensation) in both sexes. Discussion of the limitations of study design should be included with interpretation of study results. Further studies designed and powered to detect sex differences would be needed. This approach is sometimes used in toxicology studies and in settings in which animals are not readily available in sufficient numbers (e.g., nonhuman primates).
This approach—using both males and females without much increase in total number of animals—can show a treatment effect when there is little sex difference in the effect of treatment; however, when the treatment is effective only in one sex or has opposite effects in the 2 sexes, this approach can prevent discovery of treatment effects because it is underpowered. This approach, however, does give an initial assessment of the effect size of both variables, sex and treatment, and provides a foundation for a power analysis to determine what group sizes would be needed to demonstrate statistically significant effects of each variable or their interaction.
Accounting for sex as a variable via factorial design
Males and females could be incorporated into a factorial design that allows the concurrent examination of both a participant variable, such as sex and another variable, and also an assessment of the interaction of sex with another independent variable on the outcome measure. In a factorial design, statistical analyses will present the effects of each independent variable (sex, other variable) on the outcome measurement regardless of the impact of the other variable as a main effect and will present the interaction of sex and the other variable. An interaction is detected if one independent variable alters the effect of the other.
For example, in an experimental design with a 2-level factor for treatment [ethanol at 5 mg/kg or no ethanol (0 mg/kg control solution)] and a 2-level factor for sex (male, female), one can envision a factorial design in which 8 animals are randomly placed into each of 4 groups/conditions (male-ethanol, male-control, female-ethanol, female-control) for a total of 32 animals in the experiment. With 16 male animals and 16 female animals, a test of the outcome between sexes may be sufficiently robust to detect a difference, if one exists, on the sex factor depending on variability in each sex and the effect size: this would be a main effect of sex. Similarly, if 16 animals receive ethanol and 16 animals receive the control solution, a test of the outcome may be sufficiently robust to detect a difference for treatment as the factor: this would be a main effect of treatment.
However, the true value of the factorial design involves the ability, via application of the 2-way ANOVA, to determine the extent to which the outcome is altered by being male or female and receiving the ethanol or control solution; this is a test of the interaction, which is achievable by use of factorial design (see Fig. 1). Statistical test of the interaction will show how the means of the 4 groups differ and whether being male or female changes the effect of ethanol vs. control conditions.
Figure 1.
Schematic of a 2×2 factorial design showing test of the main effects of sex and drug treatment. Various possible outcomes are shown. The top line of outcomes shows a finding of significant main effect only of drug, a significant main effect only of sex, and significant main effects of both without interaction. The bottom line shows 3 different outcomes that are all significant interactions of sex and drug, which means that the effect of drug depends on sex and, reciprocally, the effect of sex depends on the level of drug. DV, dependent variable.
The potential return on investment with a factorial design is attractive, because: 1) variability as a result of each factor is parsed out before assessment of the interaction; 2) conclusions about outcome can be refined to a specific level (e.g., drug dose if multiple drug doses are included) to either male or female participants, thus eliminating the need to perform the same study with the same multiple levels in the future; 3) the number of animals may potentially be reduced in future studies as a result of the statistical parceling of error variance; and 4) interpretation of outcome is clear because of powerful control over variance.
SEX-BASED ANALYSIS AND REPORTING AS A CONSIDERATION IN PRECLINICAL INVESTIGATIONS
Sex-disaggregated data analysis and characterization of the effect of a treatment on a selected outcome measure separately in each sex can enhance understanding of underlying mechanisms in males and females. In contrast, analyzing aggregate data from males and females combined may lead to false conclusions—for example, when a response is in opposite directions in males and females. Similarly, if a result is qualitatively different in males and females (e.g., different areas of the brain affected in males vs. females), analyzing data in aggregate may lead to erroneous conclusions for both sexes. Furthermore, analyzing results in one sex, but generalizing results to both males and females, may result in erroneous conclusions.
Analyses are based on/driven by the research question and study design. It is important to be transparent about the limitations of study designs. For example, presentation of descriptive analyses of findings in males and females (e.g., outcome measure, mean, median, mode, range, and SD in males and females) should not be interpreted as proving or disproving sex differences in studies that have not been designed to detect such differences. Nevertheless, given the importance of sex-specific data, even when comparative analysis cannot be performed as a result of design limitations, sex-disaggregated data should be reported (e.g., sex of research material, results in males, and results in females) when possible. It is equally important that limitations of study designs be acknowledged, that findings from one sex not be applied to the other without testing, and that data are interpreted appropriately in the context of the potential influence of sex as a basic biologic variable. Sex influences that are detected in preliminary studies and analyses may provide useful insights into our understanding and inform future investigations. A summary of considerations for the research process has been provided (Table 1).
TABLE 1.
Summary of ways to consider sex as a variable in preclinical research
| Consideration | Description | Reference |
|---|---|---|
| Consider sex chromosomes | XX vs. XY | 6 |
| 11 | ||
| 12 | ||
| 13 | ||
| Consider sex hormones | Gonadal hormones (ovarian vs. testicular secretions): androgens, estrogens, pregnanes | 14 |
| 15 | ||
| 16 | ||
| Consider sex and gender assumptions in design | Design of animal study | 6 |
| Choice of animal model | 16 | |
| Choice of animals to study (male, female, both) | 20 | |
| Analysis and reporting | Sex of research material | 2 |
| Sex-disaggregated data | 22 | |
| Report by sex in publications |
CONCLUSIONS
Sex is an important biologic variable. Preclinical research studies that incorporate both sexes are crucial to recognizing the applicability of study findings and to informing the translation of research from basic scientific discovery to drug development and testing of therapeutics. Studying both sexes in preclinical research is good science. Including both sexes in preclinical studies and experimental designs that appropriately account for sex as a biologic variable promotes understanding of experimental outcomes for males and females. Translation of such results to clinical testing advances us one step closer to evidence-based appropriate treatments for both men and women.
Supplementary Material
This article includes supplemental data. Please visit http://www.fasebj.org to obtain this information.
ACKNOWLEDGMENTS
The Office of Research on Women’s Health thanks all speakers who presented at the Methods and Techniques workshop and those who have contributed to this manuscript. The views expressed in this manuscript are those of the authors and do not necessarily reflect the views or policies of the authors affiliated organizations. The authors declare no conflicts of interest.
Glossary
- NIH
U.S. National Institutes of Health
Footnotes
This article includes supplemental data. Please visit http://www.fasebj.org to obtain this information.
AUTHOR CONTRIBUTIONS
L. R. Miller, C. Marks, J. B. Becker, P. D. Hurn, W.-J. Chen, T. Woodruff, M. M. McCarthy, F. Sohrabji, L. Schiebinger, C. L. Wetherington, S. Makris, A. P. Arnold, G. Einstein, V. M. Miller, K. Sandberg, S. Maier, and T. L. Cornelison wrote the manuscript; L. R. Miller, C. Marks, A. P. Arnold, and J. A. Clayton edited the manuscript; S. Makris and A.P. Arnold designed the figure; and L. R. Miller, W.-J, Chen, and S. Maier drafted the supplemental material.
REFERENCES
- 1.National Research Council (2001) Exploring the Biological Contributions to Human Health: Does Sex Matter? National Academies Press, Washington, DC, USA: [PubMed] [Google Scholar]
- 2.Landis S. C., Amara S. G., Asadullah K., Austin C. P., Blumenstein R., Bradley E. W., Crystal R. G., Darnell R. B., Ferrante R. J., Fillit H., Finkelstein R., Fisher M., Gendelman H. E., Golub R. M., Goudreau J. L., Gross R. A., Gubitz A. K., Hesterlee S. E., Howells D. W., Huguenard J., Kelner K., Koroshetz W., Krainc D., Lazic S. E., Levine M. S., Macleod M. R., McCall J. M., Moxley R. T. III, Narasimhan K., Noble L. J., Perrin S., Porter J. D., Steward O., Unger E., Utz U., Silberberg S. D. (2012) A call for transparent reporting to optimize the predictive value of preclinical research. Nature , 187–191 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Zucker I., Beery A. K. (2010) Males still dominate animal studies. Nature , 690 [DOI] [PubMed] [Google Scholar]
- 4.Collins F. S., Tabak L. A. (2014) Policy: NIH plans to enhance reproducibility. Nature , 612–613 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. National Institutes of Health. (2015). Consideration of sex as a biological variable in NIH-funded research. Accessed June 30, 2016, at: http://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-102.html.
- 6.Clayton J. A. (2016) Studying both sexes: a guiding principle for biomedicine. FASEB J. , 519–524 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Arnold A. P., Chen X., Link J. C., Itoh Y., Reue K. (2013) Cell-autonomous sex determination outside of the gonad. Dev. Dyn. , 371–379 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Case L. K., Wall E. H., Osmanski E. E., Dragon J. A., Saligrama N., Zachary J. F., Lemos B., Blankenhorn E. P., Teuscher C. (2015) Copy number variation in Y chromosome multicopy genes is linked to a paternal parent-of-origin effect on CNS autoimmune disease in female offspring. Genome Biol. , 28 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Du L., Bayir H., Lai Y., Zhang X., Kochanek P. M., Watkins S. C., Graham S. H., Clark R. S. (2004) Innate gender-based proclivity in response to cytotoxicity and programmed cell death pathway. J. Biol. Chem. , 38563–38570 [DOI] [PubMed] [Google Scholar]
- 10.Li J., Chen X., McClusky R., Ruiz-Sundstrom M., Itoh Y., Umar S., Arnold A. P., Eghbali M. (2014) The number of X chromosomes influences protection from cardiac ischaemia/reperfusion injury in mice: one X is better than two. Cardiovasc. Res. , 375–384 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Arnold A. P. (2014) Conceptual frameworks and mouse models for studying sex differences in physiology and disease: why compensation changes the game. Exp. Neurol. , 2–9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Cox K. H., Bonthuis P. J., Rissman E. F. (2014) Mouse model systems to study sex chromosome genes and behavior: relevance to humans. Front. Neuroendocrinol. , 405–419 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Ruiz-Cortes Z. T. (2012) Gonadal Sex Steriods: Production, Action, and Interactions in Mammals, Steroids - From Physiology to Clinical Medicine, InTech, Rijeka, Croatia [Google Scholar]
- 14.McEwen B. S., Alves S. E. (1999) Estrogen actions in the central nervous system. Endocr. Rev. , 279–307 [DOI] [PubMed] [Google Scholar]
- 15.McCarthy M. M., Arnold A. P., Ball G. F., Blaustein J. D., De Vries G. J. (2012) Sex differences in the brain: the not so inconvenient truth. J. Neurosci. , 2241–2247 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Becker J. B., Arnold A. P., Berkley K. J., Blaustein J. D., Eckel L. A., Hampson E., Herman J. P., Marts S., Sadee W., Steiner M., Taylor J., Young E. (2005) Strategies and methods for research on sex differences in brain and behavior. Endocrinology , 1650–1673 [DOI] [PubMed] [Google Scholar]
- 17.Francia G., Man S., Lee C. J., Lee C. R., Xu P., Mossoba M. E., Emmenegger U., Medin J. A., Kerbel R. S. (2009) Comparative impact of trastuzumab and cyclophosphamide on HER-2-positive human breast cancer xenografts. Clin. Cancer Res. , 6358–6366 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Francia G., Cruz-Munoz W., Man S., Xu P., Kerbel R. S. (2011) Mouse models of advanced spontaneous metastasis for experimental therapeutics. Nat. Rev. Cancer , 135–141 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Freedman L. P., Cockburn I. M., Simcoe T. S. (2015) The economics of reproducibility in preclinical research. PLoS Biol. , e1002165 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Bailoo J. D., Reichlin T. S., Würbel H. (2014) Refinement of experimental design and conduct in laboratory animal research. ILAR J. , 383–391 [DOI] [PubMed] [Google Scholar]
- 21.McCarthy M. M., Arnold A. P. (2011) Reframing sexual differentiation of the brain. Nat. Neurosci. , 677–683 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Howell D. C. (2007) Statistical Methods for Psychology, 6th ed., Wadsworth, Belmont, CA, USA [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.