Bias Detection: Study Identifies Instruments for Evaluating Animal Studies

No good scientist wants to produce, or be accused of producing, a poorly conducted study. Even so, toxicologists conducting animal studies are not widely expected to document the steps they take to ensure internal validity—that is, to prevent their results from being skewed due to a methodological issue (a concept known as risk of bias). This is likely to change soon, and a team of researchers at the University of California, San Francisco (UCSF) has taken a first step toward a solution by conducting a systematic literature review that identifies 30 instruments for evaluating risk of bias in animal research.¹

“Risk of bias” refers specifically to the introduction of systematic errors as a result of the way a study was conducted; it is unrelated to prejudice or manipulation of results to achieve a desired outcome. Other aspects of study quality include external validity (the extent to which results can be generalized) and reporting quality (descriptions of the experiment design, conduct, and analysis).² Tools to assess these other aspects of study quality are familiar to toxicologists,^3,4,5 but the concept of risk of bias is relatively new in environmental health, says Kris Thayer, director of the National Toxicology Program’s Office of Health Assessment and Translation (OHAT), who was not involved in the new review.

Thayer’s group is leading efforts to incorporate systematic review methodology into OHAT evaluations.⁶ She says it’s important, when conducting these evaluations, to clearly define which aspects of study quality are being considered and to present them as discrete elements. She lauds the new review for providing a starting point for determining how to evaluate the risk of bias in animal studies.

The review presents an inventory of different approaches for assessing existing study-quality tools and summarizes what is known about specific factors as sources of systematic bias. “Although there is a well-developed and empirically based literature on how to evaluate the risk of bias of randomized controlled clinical trials, less is known about how to do this for animal studies,” wrote the team of UCSF researchers, who were led by Lisa Bero, an expert in systematic review methods and evidence-based health care.

Bero and her coauthors combed through 45 years of MEDLINE publications and found 3,731 articles describing instruments for assessing risk of bias in animal studies. They identified 30 instruments that assess anywhere from 2 to 25 criteria associated with the risk of bias, methodological, or reporting criteria.

Rather than pinpointing a single “best instrument,” the authors attempt to identify the core set of procedures important for evaluating risk of bias in animal studies, Bero says. These procedures include some process of random allocation to experimental groups; a blinded assessment of outcome; statistical tools to be sure that studies aren’t underpowered; recording both inclusion and exclusion criteria to show that the animals were appropriate for the study; and ensuring that all collected data were reported.

Figure 1.

Bias, or the introduction of errors into experiments, may be one explanation for equivocal research findings.

© Getty Images

The authors acknowledge that they may not have identified all published assessment instruments because they searched only for English-language publications using just one database. “However,” they wrote, “both our consultation with a librarian and the large pool of studies identified through the electronic search suggests that it was comprehensive.”¹

Study coauthor Tracey Woodruff of UCSF’s Department of Obstetrics, Gynecology and Reproductive Sciences and Program on Reproductive Health and the Environment, calls the review “a foundational piece of a broader method that is going to greatly improve our ability to evaluate environmental chemicals.” Woodruff’s group is working to establish how the systematic reviews initially developed for use in the decades-old field of evidence-based medicine can be applied to environmental health and has developed a strategy known as the Navigation Guide.⁷ Woodruff’s overarching goal is to encourage medical practitioners to educate their patients about hazardous environmental exposures, particularly ones that may affect their reproductive health.⁸

Malcolm R. Macleod, a professor at the University of Edinburgh’s Centre for Clinical Brain Sciences, who was not involved in the study, calls the review a small but “really important” contribution toward reducing the risk of bias, says. Macleod’s interest in the subject sprang from his work on clinical trials in which drug candidates recommended by animal stroke studies ended up failing in humans. His subsequent analysis⁹ demonstrated how study quality and publication bias in those animal studies led to a major overstatement of the drugs’ efficacy.

His publications and others document that lack of randomization or blinding, failure to specify inclusion/exclusion criteria or use of comorbid animals, and lack of statistical power in animal studies have inflated the estimates of the effects of pharmaceutical interventions for a number of human maladies.^{10,11,12,13,14} They inspired high-profile pleas to establish better reporting criteria that incorporates risk-of-bias assessments.^15,16