The Use of Systematic Reviews and Reporting Guidelines to Advance the Implementation of the 3Rs

Abstract

In 1959, Russell and Burch published The Principles of Humane Experimental Technique, which included concrete advice on factors that they considered would govern progress in the implementation of these principles (enunciated as the 3Rs [Replacement, Reduction, and Refinement in animal-based studies]). One challenge to the implementation of the 3Rs was identified as information retrieval. Here, we further explore this challenge—the need for ‘research on research’—and the role that systematic reviews and reporting guidelines can play in implementation of the 3Rs. First, we examine the 2-fold nature of the challenge of information retrieval: 1) the identification of relevant publications spread throughout a large population of nonrelevant publications and 2) the incomplete reporting of relevant details within those publications. Second, we evaluate how systematic reviews and reporting guidelines can be used generally to address this challenge. Third, we assess the explicit reporting of the 3Rs in a cohort of preclinical animal systematic reviews. Our results show that Reduction methods are the most commonly reported by authors of systematic reviews but that, in general, reporting on how findings relate to the 3Rs is limited at best. Although systematic reviews are excellent tools for resolving the challenge of information retrieval, their utility for making progress in implementation of the 3Rs may be limited unless authors improve their reporting of these principles.

“We now have far too much information as a species to digest as individuals.”¹¹⁶

Since Russell and Burch published The Principles of Humane Experimental Technique, the 3Rs (replacement, reduction, refinement) have become the guiding principles for the ethical use of animals in science.⁶ In 1959, Russell and Burch deplored the long delay in the application of existing knowledge to improve experimental techniques. Currently, some 55 y later, a gap still remains between the guiding principles of the 3Rs and their application in practice.^{5,74,78,79,127,131} To help close this gap, we revisit Russell and Burch's advice regarding the “factors which govern its (the 3Rs) progress.”¹¹⁶

Russell and Burch acknowledged that an important factor for progress in the implementation of the 3Rs is the necessity for ‘research on research,’¹¹⁶ a general concept that continues to be advocated by contemporary authors, irrespective of any interest in the 3Rs.¹⁹ This factor was deemed to be so important that Russell and Burch suggested that “[a]s science continues to expand, this [research on research] will be seen more and more clearly as the only way to save it from grinding to a standstill.”¹¹⁷ Although Russell and Burch viewed interspecialist communication as a key factor governing progress in implementation of the 3Rs; the necessity for research on research stems from a wider problem: information retrieval. Here, we explore the challenges of information retrieval, the need for research on research, and the role that systematic reviews and reporting guidelines can play in closing the gap between the 3Rs principles and their implementation in practice.

Information Retrieval

The problem of information retrieval can be divided into 2 parts: 1) the identification of relevant publications spread throughout a large population of nonrelevant publications and 2) the incomplete reporting of relevant details within those publications.

Identification of relevant publications.

When searching for information on any subject only a finite proportion of information may be found because individual publications are spread throughout a wide range of periodicals. Russell and Burch recognized the challenges of information retrieval saying, “…there will be many such [publications] in a few periodicals and few [publications] in a great many [periodicals]”¹¹⁷. This challenge is also known as Bradford's Law of Scattering.^14,94 Finding information by searching a few periodicals with many relevant publications is easy, but when the relevant information becomes spread through a widening range of periodicals, the task becomes more difficult.

Information retrieval is also made difficult by the sheer quantity of information; as Russell and Burch noted in 1959,¹¹⁷ when the total number of citations indexed in PubMed (from 1907 to 1959), was 1,475,600, of which 65,785 (4%) were related to animals (Figure 1). From 1960 to 2012, the total number of citations indexed in PubMed was 20,786,790, of which 5,451,234 (26%) were related to animals (Figure 1). Therefore the retrieval of information scattered among an ever-increasing number of publications is a daunting task for any investigator attempting to summarize all the relevant information on a given topic.

Figure 1.

Total citations in PubMed and total animal citations in PubMed by Year 1907 – 2012. The total number of references indexed PubMed by year for the years 1907 to 2012 (noncumulative). Blue bars represent the total number of references in PubMed in a given year. The red shading overlying the blue bars represents the total number of animal-related references indexed in PubMed in a given year. A search of all indexed material in PubMed was conducted on 12 December 2013 by using the search string (“1900/01/01”[PDAT]: “2012/12/31”[PDAT]). The ‘results by year file’ (.csv) was downloaded from Pubmed, and the values for the total number of citations by year were entered into an Excel (Microsoft, Redmond, WA) file. For each year from 1907 to 2012, a search was conducted (for example, (“1959/01/01”[PDAT]: “1959/12/31”[PDAT])), and a validated search filter for animal-related research²⁸ was applied and the number entered into an Excel file. The quoted material is from reference 118.

Incomplete reporting of relevant details.

The information retrieval challenge becomes more formidable given the incomplete descriptions of methodology often found in publications.¹¹⁷ Russell and Burch described the challenge by using a quote from Maurice Visscher, “…methodology is usually relegated to a place of smaller type and sharply abbreviated importance in journal publication of research. Numerous essential details are customarily omitted, either because they are considered to be common knowledge or simply for lack of space.”¹³⁷ In recent years, increasing evidence has highlighted the prevalence of incomplete methodology reporting in preclinical animal research. For example, one cross-sectional study of the biomedical animal literature found that methodological details such as randomization and blinding often were not included in research publications.⁶¹ Another review of highly cited (at least 500 times) biomedical animal studies published in high-impact–factor journals found a similar pattern of missing methodologic details.⁴⁴ Unfortunately, missing details are not limited to methodology but sometimes include such basic information as the hypothesis or objective.⁶¹

Synthesis of information.

To resolve the problem of information retrieval, Russell and Burch recommended the use of a form of synthesis: “One approach to this special problem [information retrieval] is to concentrate some attention on the particular branch of synthesis which takes the form of general methodological study.”¹¹⁷ For Russell and Burch, a “general methodological study” was a book review, such as The Principles of Humane Experimental Technique. Here, we depart from Russell and Burch's recommended approach and suggest 2 complementary approaches to the synthesis of 3Rs information (which have evolved in the clinical sciences field since the publication of the Principles of Humane Experimental Technique): the use of systematic reviews and reporting guidelines.

Systematic Reviews and Reporting Guidelines

Systematic reviews.

A systematic review has been defined as “…a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyze and summarize the results of the included studies.”⁹¹

In general, systematic reviews are conducted by a multidisciplinary team. These teams may include the funder, subject-matter experts (for example, preclinical stroke researchers), systematic-review experts, librarians with expertise in using unbiased search strategies, knowledge users (for example, clinical stroke researchers), and statisticians.^93,105 Such multidisciplinary teams allow for what Russell and Burch called “interspecialist communication,”¹¹⁸ which they saw as fundamental for progress in the 3Rs.

Systematic reviews differ from traditional narrative reviews in several important ways. Narrative reviews are usually not framed around a specific question, whereas the question must be framed explicitly in a systematic review.⁴⁶ In human health research, this framing often involves the use of a structured approach with 5 elements used to frame the question. These elements, known by the acronym PICOS,⁷² include: P, the patient population or disease being studied (for example, healthy men and women aged 16 to 65 y⁵⁴); I, the intervention (for example, killed or live, attenuated vaccines administered via any route⁵⁴); C, the comparator group (for example, placebo or no intervention⁵⁴); O, the outcome or endpoint (for example, number and seriousness of cases of symptomatic influenza and of adverse effects⁵⁴); and S, the study design chosen (for example, randomized control trials⁵⁴). Use of the PICOS acronym facilitates the development of a clearly stated set of objectives, which allows review authors to develop a search strategy and set criteria for studies to be included and excluded in the review. In addition, the PICOS acronym is easily adaptable to different fields and can be used to frame questions that include other important elements, such as Context¹⁰⁶ (PICOCS) in the field of social sciences or Exposure³⁰ (PECO) in the field of etiology.

Narrative reviews and primary studies do not use systematic search strategies but instead rely on simple availability of periodicals and author preference to identify relevant publications. Such an approach cannot overcome the problem of information retrieval and may lead to a biased selection of publications because authors may fail to identify all those that are relevant. For instance, a systematic review of clinical trials of the drug aprotinin, which is used in cardiac surgery, found that the median percentage of previous trials cited per publication was only 20%.³³ This low level resulted in the completion of redundant studies long after the effectiveness of aprotinin in reducing the need for perioperative transfusion had been established.³³ The use of systematic methods—in particular, the search strategy and screening methods used to identify all relevant publications^52,70,138—is the chief way in which systematic reviews overcome the problem of information retrieval. Furthermore, reporting the methods used for the search strategy and the criteria used to include or exclude studies ensures that the entire process is transparent. This transparency allows readers to critique the methods used to identify and select relevant studies instead of pitting one expert opinion against another.¹²⁵

The ambition to “include a library of trial overviews, which will be updated when new data become available” was first articulated in 1986.¹⁸ Updating clinical systematic reviews is still a major component of the reviews themselves, although the methods for keeping reviews up to date are not as well-established as are the methods for conducting systematic reviews.²⁷ Updating reviews, in contrast to authoring new ones, is a concept that can save time and wasted effort as it ensures that the synthesized knowledge maintains the greatest currency for potential users.

Meta-analysis of data.

Findings from studies included in a systematic review are often analyzed by using techniques such as meta-analysis. The importance of analyzing the results from the individual studies, rather than the authors’ opinions regarding their data, has been demonstrated in retrospective cumulative meta-analyses of clinical trials.^58,68 For instance, retrospective analysis of all clinical trials that looked at the efficacy of streptokinase for acute myocardial infarction and the efficacy of tranexamic acid for blood transfusion showed that effectiveness of the treatments had been established years before trials were stopped.^58,68 In these and other cases, small trials and incomplete literature reviews prior to starting new trials led to the false belief that efficacy had not been established definitively.⁴⁰ Similarly, retrospective cumulative meta-analyses of animal studies have found that animal experiments are unnecessarily replicated; for example, in establishing the efficacy of tissue plasminogen in stroke models.¹²² The use of systematic reviews with prospective cumulative meta-analysis would prevent unnecessary replication by highlighting when a question has already been answered definitively (that is, when the addition of data from additional studies do not affect the cumulative findings), ensuring that all important findings are considered, as well as contributing to the reduction of redundant and unnecessary animal use.

Limitations of systematic reviews and meta-analyses.

One identified limitation of systematic reviews is publication bias:¹¹⁵ the tendency for positive findings to be published more often than are negative findings. Techniques are available that allow systematic reviews to assess the likelihood that publication bias has skewed the overall results of the review.⁴⁶ For example, evaluation of a systematic review of animal models for acute ischemic stroke revealed that publication bias caused a statistically supported 30% over-exaggeration of the treatment effect.¹²⁴ Publication bias represents an additional challenge to the primary problem of information retrieval (that of identifying relevant publications), but the use of systematic search strategies and statistical tools can at least estimate the extent of the publication bias and can even estimate the number of experiments and animals used in the nonpublished studies.¹²⁴ Another complication, similar to publication bias, is selective outcome reporting, where authors select for publication only a subset of the analyses they performed (for example, the significant ones).^20,134 Unlike publication bias, selective outcome reporting is a challenge to the secondary problem of information retrieval (that of incomplete reporting). Without preregistered protocols, as there are in clinical trials, the direct assessment of either publication bias or selective outcome reporting is difficult if not impossible.^20,62,63

Another limitation of systematic reviews and meta-analyses is the accuracy of the reporting of the primary studies used in the synthesis. Systematic reviews formally assess the validity of the included studies through tools such as a risk of bias assessment.⁴⁶ Such assessments help authors to objectively and transparently appraise whether findings are biased. For instance, in a systematic review of animal studies for the efficacy of NXY059 in ischemia, the authors found that primary studies that did not use randomization or concealment of treatment group allocation had significantly larger effect sizes than did those that used these methods.⁸³ Because systematic reviews use formal approaches to the synthesis and analysis of both the methods and outcomes of included studies, they are able to detect biases such as those described earlier, whereas traditional narrative reviews cannot. However, many primary animal studies do not fully report the methods used,⁶¹ as the secondary challenge to information retrieval addresses.

Reporting guidelines.

In both preclinical and clinical research, poor reporting has been identified as a systemic source of waste.^1,40,89 In response to concerns about the quality of reporting in randomized control trials involving humans, the CONSORT (CONsolidated Standards Of Reporting Trials) was developed.¹¹⁹ The CONSORT statement itself was not developed as a quality assessment instrument but was designed to improve the reporting of items related to the internal and external validity of trials.⁸⁹ However, the statement has proven to be an effective tool to improve the quality of reporting in trials.^56,90,135 Early in the development of reporting guidelines for randomized control trials, separate guidelines groups met and developed a common set of recommendations that became the CONSORT statement;^132,111 this one common set of recommendations facilitated their adoption. The development and assessment of reporting guidelines in the clinical field is now well-established,⁹² whereas preclinical reporting guidelines are a recent phenomenon and one that needs urgent attention.

The accuracy of reporting preclinical research has come under scrutiny as questions about the reproducibility of research have been raised.^10,108 The United States National Institute of Neurologic Disorders and Stroke held a workshop and recommended that key methodologic items, such as sample size calculations; method of randomization of animals; blinding to treatment; and data handling, should all be reported.⁶⁷ Similarly, motivated by the irreproducibility of research, the Nature publishing group has instituted a checklist³ for all animal-based papers, including requirements for sample size calculations; inclusion and exclusion criteria; methods of animal randomization; blinding to treatment; information on species, strain, sex, and age of animals; and ethics compliance statement.⁹⁶ Many of these reporting items relate to the design of animal experiments, which was identified as a problem more than 2 decades ago³⁴ and is now receiving increased attention.^21,35,37,45 In addition, reporting guidelines for animal studies have been developed, for example, the Animal Research: Reporting In Vivo Experiments⁶⁰ (ARRIVE) guidelines, Gold Standard Publication Checklist⁴⁸ (GSPC), and Description of Animal Research in Scientific Publications⁵³ (DARSP) recommendations, as well as area-specific guidelines such as the REFLECT statement for randomized control trials in livestock and food safety.¹¹⁸

Although the ARRIVE guidelines include a specific reporting element on the 3Rs and were developed by the National Centre for the 3Rs in the United Kingdom, they were developed to ensure that “data from animal experiments could be fully evaluated and utilised.”⁹⁵ Currently the ARRIVE guidelines are endorsed by more than 300 journals, funders, universities, and learned societies. However, reporting guidelines are not a panacea: improved training of authors, editors, and reviewers is required, as is the endorsement and implementation of the guidelines by journals and peer reviewers. The advent of digital publishing and the availability of space to report more methodologic detail should assist in removing the barriers to good reporting of research. The important next step is to evaluate the effect that these guidelines have on the reporting of animal studies to determine whether these statements help to achieve progress in the implementation of the 3Rs.^4,120

Summary of tools to address information retrieval.

The combined use of systematic reviews and reporting guidelines likely will help to achieve progress in implementation of the 3Rs by providing a structured way to address the problem of information retrieval of animal studies. Specifically, the use of systematic search strategies, often conducted by or in consultation with an expert in these techniques, combined with validated search filters and a well-defined question (for example, PICO) make systematic reviews an excellent adaptable tool to identify all relevant studies in the growing deluge of information confronting investigators.

In addition, systematic reviews provide authors with the means to assess the validity of the studies and to synthesize their results through techniques such as meta-analysis. As with the problem of publication bias,^24,62 for which the remedy is the publication of all results, it is better to preempt the formidable problem of retrieving missing information by including all relevant details in publications. Reporting guidelines, like systematic reviews, have proven effective in the clinical field to ensure adequate reporting of experimental details and are being adopted by investigators, journals, and funders.

We have argued in general terms that systematic reviews will help achieve progress in the 3Rs by ameliorating the problem of information retrieval as described by Russell and Burch. Because systematic reviews provide a formal method to evaluate multiple primary studies’ methods and outcomes (for example, through risk of bias assessment and meta-analysis), they also may identify opportunities for implementation of the 3Rs. To examine this possibility, we reviewed available systematic reviews of preclinical animal studies to identify the frequency of reporting on the 3Rs.

The 3Rs in Systematic Reviews and Reporting Guidelines

Introduction.

In 1990, at the 7th annual meeting of the Johns Hopkins University Center for Alternatives to Animal Testing, 30 persons from 13 countries active in animal issues or alternatives in research agreed that it was time to institute a World Congress on Alternatives and Animal Use in the Life Sciences.⁴² The first World Congress was held in Baltimore in 1994, and since then, scientists, veterinarians, policy makers, animal protectionists, and other interested parties have come together every 2 to 3 y to discuss progress in implementation of the 3Rs. In 1999, participants at the 3rd World Congress endorsed A Declaration of Bologna³¹, reaffirming the principles put forward by Russell and Burch in 1959: “Humane science is a prerequisite for good science and is best achieved in relation to laboratory animal procedures by the vigorous promotion and application of the Three Rs.” In 2011, participants at the 8th World Congress on Alternatives and Animal Use in the Life Sciences adopted the Montréal Declaration: on the Synthesis of Evidence to Advance the 3Rs Principles in Science. The declaration called for a “…change in the culture of planning, executing, reporting, reviewing, and translating animal research.”⁷¹ In particular, the Montréal Declaration called for the use of systematic reviews and reporting guidelines to be implemented in animal studies in a similar vein as had been done for clinical research decades earlier.

Currently, only the ARRIVE guidelines, implemented for primary animal studies, recommends that authors report “…any implications of your experimental methods or findings for the replacement, refinement or reduction (the 3Rs) of the use of animals in research.”⁶⁰ We decided to evaluate whether the 3Rs are being implemented through reporting in systematic reviews of preclinical animal research. Specifically, our objectives were to determine how often the 3Rs are explicitly reported in published systematic reviews of preclinical animal studies.

Methods.

One previous systematic review of systematic reviews and meta-analyses of in vivo animal experiments, carried out to inform human health, searched for materials through 2005.¹⁰⁴ In 2011, this earlier review¹⁰⁵ was extended to include studies through 2010.⁶⁵ The combined studies identified 185 systematic reviews without meta-analyses, 59 systematic reviews with meta-analyses, and 22 meta-analyses that did not include a systematic review. We selected the 59 systematic reviews that included a meta-analysis for our analysis (references 8, 12, 13, 23, 52, 25, 26, 32, 36, 41, 55, 57, 69, 73, 76, 80–82, 84, 97, 103, 107, 113, 116, 129, 130, and 141–143, which were included in the 2006 review,105 and references 2, 7, 9, 11, 22, 28, 38, 39, 47, 59, 64, 66, 75, 77, 83, 85–88, 98, 99, 101, 102, 110, 121, 133, 139, 140, 144, and 145, which were in the 2011 publication⁶⁶). We used standard definitions of the 3Rs from the Canadian Council on Animal Care¹⁷ to develop a 4-item data extraction checklist (Figure 2) to evaluate reporting of the 3Rs. Our approach was to read the 3Rs as they were reported and not to read the 3Rs into the papers for the authors. Each item was operationalized such that we scored a result of Yes, No, or Partial for each (Figure 2).

Figure 2.

Data extraction checklist to evaluate reporting of 3Rs.

For both Yes and Partial scores, authors had to have explicitly described an effect regarding animals that fell within the definition of the 3Rs, but authors did not need to have explicitly used the term replacement, reduction, or refinement (for example, “alternative” or “fewer animals” was sufficient). Yes and Partial scores were differentiated by whether the discussion of the item was in reference to the results of the systematic review (Yes) or not (Partial). When a paper had information to support both a Yes and a Partial score for an item, we scored the item as Yes only. The year of publication and journal name were extracted also.

Two authors (MTA and NF) independently pilot-tested the data extraction checklist on 5 randomly selected included studies and refined the criteria. Disagreements were resolved by discussion between the extractors (MTA and NF); if no agreement could be reached, then the third author (GG) acted as the decider. The same 2 authors (MTA and NF) independently extracted data from the 59 articles and one author (MTA) also compiled the supporting text for decisions that scored Yes or Partial (Table 1). For analysis, we counted the total number of Yes, No, and Partial scores for each item and evaluated differences descriptively.

Table 1.

Supporting text from references that met the criteria for Yes or Partial

Reference	Sub-item name	Text for Yes	Text for Partial
26	Reduction	Strains such as SHR are inbred and require only a few animals to achieve statistical significance.
70	Reduction		Because the use of large numbers of animals necessary for a survival study may not be justified ethically, the effect of an intervention on mortality may require extrapolation from results of studies that may not be primarily designed to establish survival differences. Therefore, insufficient numbers of animals are often employed. The sample size limitations associated with extrapolating results of survival from small experimental studies have, until now, been considered an inevitable limitation of such designs.
81	Reduction	As the number of drugs analyzed in this way increases, multiple regression modeling should allow identification of those factors which have greatest impact on estimates of effect size. In turn, this may allow identification of a subset of variables which are sufficient to describe the properties of a given drug, potentially reducing the number of experiments needed to characterize that drug.
82	Reduction	1. Based on our observations, to have 80% power to estimate the percentage improvement in outcome after treatment to the nearest 10% would require 115 animals per group. Such sample sizes may seem large, but the use of smaller cohorts represents a false economy and results are likely to be misleading; ultimately more animals would be required.
		2. Using this approach, we now have data for 3092 animals from 208 individual comparisons. With the addition of further data, multiple regression modeling should allow identification of those factors which have greatest impact on estimates of effect size. In turn, this may allow identification of a subset of variables which are sufficient to describe the properties of an administered drug, potentially reducing the number of experiments needed to characterize that drug.
103	3Rs		Medical Research Council found that most people support their use provided that there are benefits to human health care, no alternative exists, and no unnecessary suffering occurs.
114	3Rs		In a recent editorial, Smith promoted the 3 Rs of animal research first suggested by William Russell: replacement, reduction, and refinement. On methodological grounds, animal experimentation would better contribute to human health care if we promoted registration, randomisation, and systematic reviews.
	Reduction	In his book The Principles of Humane Experimental Technique, William Russell proposed the principle of reduction—that is, the use of methods to “reduce the number of animals needed to obtain information of a given amount and precision.” Meta-analyses of the results of previous animal experiments would increase the precision of estimates of treatment effects and therefore reduce the number of animals needed in future experiments.	However, animal researchers are encouraged to reduce the number of experimental animals to a minimum. Indeed, the need to use the minimum number of animals to obtain valid results is embodied in the Animals (Scientific Procedures) Act 1986 and European legislation.

Results.

Overall, most of the systematic reviews (53 of 59, 90%) did not explicitly report on any aspect of the 4 items. Of the 6 (10%) systematic reviews that did report on at least one item, 2 reviews reported on the 3Rs in general, 5 reported on reduction, and none reported on replacement or refinement (Table 2). The number of systematic reviews of animal-based studies each year increased, but no discernable trend for the reporting of the 3Rs was visible. However, only 1 of the 6 systematic reviews that reported on the 3Rs was from the most recent (2006 to 2010) cohort of studies.

Table 2.

Systematic reviews that reported on 3Rs

Reference	Year	3Rs	Replacement	Reduction	Refinement
26	1994	No	No	Yes	No
70	2003	No	No	Partial	No
81	2005	No	No	Yes	No
82	2005	No	No	Yes	No
103	2007	Partial	No	No	No
114	2002	Partial	No	Yes	No

Discussion.

We began this review by looking to Russell and Burch's advice on factors that would govern progress in the implementation of 3Rs principles. We argued that systematic reviews and reporting guidelines are ideal tools for tackling the problem of information retrieval, which Russell and Burch saw as central to progress in humane science. The advantages of systematic reviews are their use of structured methodology and, in particular, the use of systematic search strategies to identify all relevant information for the focus question. Reporting guidelines, when implemented and enforced, help to ensure that all the relevant information that is necessary to critically appraise a research report is available. These 2 tools—reporting guidelines and systematic reviews—complement one another by ensuring that all information is present in the primary publication and that all relevant publications are identified for the synthesis. We also argued that systematic reviews may be ideal tools for identifying opportunities to implement the 3Rs because they formally evaluate both the methods and outcomes of multiple studies. However, the majority of the systematic reviews that we examined did not take the extra step to explicitly report their findings in the context of the 3Rs. Because systematic reviews are general tools, there is no guarantee that authors will either interpret their findings in the context of the 3Rs or report on 3Rs explicitly. Our results show that reporting on the 3Rs by systematic review authors is limited at best and, although very few of the systematic reviews explicitly discussed the 3Rs, reduction was discussed by some authors. Despite this lack of explicit 3Rs reporting, we believe that systematic reviews are an excellent tool to address the challenge of information retrieval and thus aid in progress of the implementation of the 3Rs. If primary research reports are missing critical information, then the full use of the animals in those reports has not been achieved and may lead to unnecessary or misguided subsequent work. Similarly, if reviews of treatments do not assess all relevant papers in a systematic manner, then research may continue long after questions have been definitely answered, thereby resulting in the unnecessary use of animals. The use of cumulative meta-analysis in systematic reviews in particular has potential to avoid unnecessary testing by determining when a question has been answered definitely. One limitation of systematic reviews and reporting guidelines is that they cannot identify studies that are simply unpublished. Publication bias potentially harms both animals in the research, whose euthanasia never becomes part of the scientific discourse, and the scientific community at large, which is presented with a biased set of research findings.⁶²

Reduction opportunities were reported in 5 of the 6 systematic reviews that reported on any type of the 3Rs, and 4 of these were in relation to their results. Reduction opportunities that were identified included: 1) combining studies data in meta-analysis or other techniques, a strategy that increases the precision of estimates for treatment effects and thus reduces the number of animals needed in future studies;^69,113 2) the use of sample-size calculations to correctly detect effect sizes, a measure that may increase the number of animals per experiment in the short term but that reduces the overall number of animals used in the long term;⁸¹ 3) the identification of factors that had the greatest impact on estimates of effect size;^80,81 and 4) the identification of the animal strains that required fewer animals to detect statistical significance.²⁶ The greater number of reduction opportunities reported relative to those for replacement and refinement suggests that systematic reviews and meta-analyses may be particularly useful for identifying areas for reduction.

According to our reporting criteria, there were no examples of replacement or refinement opportunities among the 59 reviews we evaluated. However, our review evaluated a limited subsample of systematic reviews with meta-analyses from a larger pool of systematic reviews without meta-analyses, a feature that may have biased our results. In addition, we relied on 2 systematic reviews^65,104 that were focused on systematic reviews of in vivo animal studies to inform human health and thus will have missed systematic reviews from other fields (for example, veterinary medicine,^43,100 conservation,¹²⁸ and in vitro studies¹¹⁴). Our criteria also required authors to explicitly report about the 3Rs, but there is currently no reporting requirement for systematic review authors to do so.^104,123 The 2 reviewers for this paper (MTA and NF) found that some of the systematic reviews did report on methods and outcomes that could be related to the 3Rs, but they were not discussed in that context. For example, when multiple species were compared and one species was found to be a better model for clinical translation,⁷⁶ the potential replacement or reduction benefits were not mentioned (therefore the description did not meet our criteria). Another limitation of our review is that the systematic reviews are relatively old in a field that is rapidly developing and improving its methods.^15,65

Another approach to identify 3Rs-related findings in systematic reviews is to purposely review systematic reviews to identify the methods and outcomes that could assist 3Rs objectives—even if they are not explicitly reported as such. This shifts the focus from examining how the 3Rs are reported, as we did, to how the 3Rs can be interpreted as the reports are written. This approach may be a useful first step to identify the common elements within existing systematic reviews that can be used to achieve 3Rs objectives.

In the systematic reviews that we evaluated, the 3Rs were not part of the questions being addressed. If the 3Rs were considered at the initial planning stage of a systematic review, when the research question is framed, it may be incorporated more easily into the review or even made the focus. However, we recognize that expecting authors of systematic reviews to explicitly interpret their findings in the context of the 3Rs may be arguable. The ARRIVE guidelines for primary animal studies already include a reporting item for interpreting results in the context of the 3Rs, and we believe that a similar criterion could be adopted for the reporting of preclinical systematic reviews. There would be some advantages to discussing the 3Rs in systematic reviews compared with primary studies, due to the nature of the reports. First, because systematic reviews evaluate many studies and assess heterogeneity in design and the animal models, they are a more advantageous venue for identifying 3Rs opportunities than are primary studies. Although the systematic reviews in our evaluation were not designed to assess the 3Rs, some authors identified clear opportunities to achieve reduction. Second, it is more likely that authors have both the space and prerogative to explore broader issues and implications in a review article than in a primary study, where the reporting requirement may be seen as burdensome. Third, a key challenge for the achieving the 3Rs is information retrieval of 3Rs-relevant literature itself. When systematic review authors do identify a 3Rs opportunity, using the terminology of the 3Rs and including appropriate keywords in the publication would greatly facilitate the identification of 3Rs opportunities to other researchers.

Conducting systematic reviews of animal studies is still a relatively new endeavor.⁴⁹ In the preclinical field, the CAMARADES¹⁶ and SYRCLE¹⁰⁹ groups are leading the way in conducting systematic reviews of animal studies and developing tools and methods.^{50,51,123,136,138} Recently, at the 21st Cochrane Colloquium in Québec City, a consensus was reached to establish a Preclinical Animal Study Methods Group jointly with the Cochrane Collaboration.¹¹² In addition, the NC3Rs in the United Kingdom is actively promoting the use of systematic reviews in animal studies, providing funding opportunities, and acting as a leader in developing reporting guidelines.⁹⁵ To address the need for the development, promotion, and support of research reporting guidelines (not solely for animal-based studies) an umbrella organization, the Equator Network,¹²⁶ was formed, and its website is a valuable source of information for reporting guidance.²⁹

Progress in implementation of the 3Rs principles in practice is dependent on many factors. In this review, we looked at the problem of information retrieval and identified currently available, complementary tools for solving it. Systematic reviews and reporting guidelines both hold great promise for progress in implementation of the 3Rs. We showed that, currently, the explicit reporting of the 3Rs is limited and that additional research is needed to assess the implicit reporting of the 3Rs in systematic reviews. In particular, systematic reviews may be a source for reduction opportunities, although the path to implementation will require focused effort. For persons using animals in science, these tools present an opportunity to improve the quality of their research, use limited resources more effectively, and contribute to progress in the implementation of the 3Rs, as Russell and Burch advised more than 50 y ago.