A practical guide to preclinical systematic review and meta-analysis

1. Introduction

Preclinical systematic reviews (SRs) and meta-analyses (MAs) are important research activities to address the translational challenges of pain research. Systematic reviews provide empirical evidence to gain knowledge, inform future research agendas, and grant applications concurrent to developing researchers' professional skills.

Systematic reviews are an effective approach to consolidating high-volume, rapidly accruing, and often conflicting research on a specific topic. Designed to address a specific research question, SRs use predefined methods to identify, select, and critically appraise all available and relevant literature to answer that question in an unbiased manner.¹⁸ This structured approach distinguishes SRs from narrative reviews. Where appropriate an MA can follow, whereby quantitative data are extracted and statistical techniques are used to summarise the outputs. Together, a SR and a MA can be conducted to assess the quality of experimental design, conduct, analysis and reporting and the reliability of the available and relevant data.⁴⁵

Through decades of innovation by the Cochrane Collaboration and others, SRs and MAs now lie at the centre of clinical evidence. The information provided has fundamentally revolutionised clinical medicine at all levels, from informing policy and funding decisions to determining optimal treatments for individual patients. Before a clinical research project or funding application, it is best practise to conduct an SR to ascertain what is already known and to identify knowledge gaps.

In the preclinical setting, SRs are relatively novel, partly because of inherent complexities and resource requirements for processing the large number and diverse preclinical publications, paradoxically a strong justification for SRs because they provide the means to synthesise evidence from heterogeneous studies. In some fields, they are gaining popularity (eg, stroke³⁷), and feasibility is improving with technical advances, for example, online review software, machine learning, and text mining.³ However, it is important to highlight that not all SRs require machine learning expertise: research questions can be defined based upon capacity, SR software are free and widely accessible, and large-scale SRs constantly seek help from interested researchers who can learn as they participate.

The aim of this review is to highlight the exciting possibilities a preclinical SR can bring to your research toolkit, demonstrate the importance of preclinical SRs in generating empirical evidence to aid robust experimental design, inform research strategy, and support funding applications. We provide guidance and signpost resources to conduct a preclinical SR.

2. Importance of preclinical systematic reviews

Preclinical SRs offer a framework by which the range and quality of the evidence can be assessed, to improve study design,¹ rigour,^12,13,27,43 and reporting.^8,41 They summarise the knowledge into an easy-to-understand format in conjunction with identifying gaps in the knowledge base thereby providing the justification for raising funding for new studies.^6,9,36,42

To address translational challenges, SRs can inform robust experimental design. Experimental bias is a consequence of poor internal validity leading a researcher to incorrectly attribute an observed effect to an intervention.²⁶ Internal validity is comprised of mitigating a range of biases: selection, performance, detection, and attrition bias,⁵⁷ and quality assessments provide structured insight into whether the existing data are at risk of bias.^28,30,34 Concomitantly, SRs can also be used to inform study design, eg, optimal animal model and outcome measure. A MA can also be used to model the impact of publication bias (culture to publish novel, positive results, not neutral or negative data⁵²) and the consequential magnitude of overestimation of effects.⁴⁷

Systematic reviews make use of available data, prevent the unnecessary duplication of experiments, and offer the means to support scientific and technological developments that replace, reduce, or refine the use of animals in research (eg, as demonstrated by de Vries et al.¹⁷).

Finally, SRs can be used to inform clinical trial design and establish whether there is evidence to justify a clinical trial.^29,46,58 Retrospective preclinical SRs for interventions that failed in clinical trials have demonstrated that prospective SRs of the animal literature would have concluded that there was insufficient evidence of effect to justify progressing into clinical development (reviewed by Pound and Ritskes-Hoitinga⁴⁰).

In summary, SRs provide the empirical evidence for improving study design, methods, and analysis to produce unbiased results and increase usability, accessibility, and reproducibility thereby increasing value and reducing research waste.^7,23,32 It is also important to note the challenges, for example, crediting research between primary researchers and research synthesisers, and the limitations that persist, for example, SRs and MAs cannot overcome deficiency in evidence, nor do they correct biases (reviewed by Gurevitch et al.²⁴).

3. The practical challenges of systematic reviews

There are several unique challenges for the conduct of preclinical SRs including, high-volume and rapidly accruing data and wide variation in the design, conduct, analysis, and reporting of preclinical studies. The methods for SRs are well developed, but are resource-intensive⁵⁵ and time-consuming.⁴⁹ This problem is exacerbated by the exponentially increasing number of publications.⁴ In preclinical neuropathic pain research, the number of articles retrieved by a systematic search rose from 6506 in 2012 to 12,614 in 2015.¹³ Comparatively, only 129 articles were identified in an SR of neuropathic pain clinical trials.²¹

Widely accessible methods and resources to improve the feasibility of preclinical SRs and MAs have been developed by the Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies (CAMARADES), University of Edinburgh, United Kingdom, and the Systematic Review Center for Laboratory Animal Experimentation (SYRCLE), Radboud University, NL. Both groups offer guidance and support to researchers.

The Systematic Review & Meta-analysis Facility (SyRF) is a free, fully integrated online platform for performing preclinical SRs. The SyRF includes a secure screening database, data repository, and analysis application. Educational resources are also available. In conjunction, Learn to SyRF is a platform researchers can use to create project-specific training courses enabling reviewers to learn, practice, and demonstrate reviewing skills before contributing to a review.

4. The review stages

Before starting a SR, we recommend engaging with training resources to familiarise with SR methodology. In addition to the learning resources available through the CAMARADES′ and SYRCLE's websites, there are several comprehensive reviews,^35,45,56 SYRCLE's starting guide, and a recent Pain Research Forum webinar.⁵⁰ Generic online courses include Systematic Review Methods (open source) and Cochrane Interactive Learning (varied subscriptions for access). You should also consider whether you have the necessary time and human resources available and how you will involve external experts including statisticians, librarians, and collaborators with existing SR experience.

The stages of an SR are described below (corresponding to Figs. 1 and 2). Decisions required at the screening, annotation, and outcome data extraction stages can be subjective, therefore, to minimise bias and human error should be performed by 2 independent reviewers and disagreements reconciled by a third independent reviewer.

Figure 1.

“Getting Started” Infographic. This is a “how to guide” and describes each stage of the SR and MA processes and the resources for learning and development as well as performing a review. MAs, meta-analyses; SRs, systematic reviews.

Figure 2.

An example of the systematic review workflow and the platforms that can be used at each stage.

4.1. Protocol development and registration (mandatory)

This is the most important step, and the preparatory time spent here will not be wasted. The protocol provides methodological transparency and reduces the risk of introducing bias. It defines the research question and the methods you plan to use including the search strategy; inclusion and exclusion criteria; data to be extracted; risk of bias/quality assessment, which based on reporting of methodological criteria allows reviewers to determine whether a study is at low, high, or unclear risk of bias, for example, CAMARADES checklist,³⁴ SYRCLE Risk of Bias Tool,³⁰ and GRADE adapted for preclinical SRs²⁸; data synthesis; and statistical analysis plan.¹⁵ Other reporting biases including financial and academic conflicts of interest can also be assessed. Registering the protocol, for example, on the Open Science Framework Registries, PROSPERO or the SYRF Protocol Registry, allows others to locate reviews in progress and enables future replication. Some journals (eg, Pain Reports and BMJ Open Science) publish protocols and the associated peer review can significantly improve your SR.⁵¹

4.2. Search strategy

The search strategy is informed by your research question. We recommend consulting a librarian or bibliographic database expert for help because this can be a complex task. An SR aims to capture all the relevant literature specific to your research question. Electronic databases of preclinical research include PubMed, Ovid Embase and Web of Science, and SYRCLE have developed animal search filters for the databases.^16,31,33 We do not recommend using Google Scholar because its algorithms are not transparent and searches are not easily reproduced.²⁵ It is necessary to construct individual search strategies for each database because databases differ in their coverage of journals and how articles are indexed. A narrow search strategy will risk missing relevant studies, too broad and you will add many irrelevant studies and consequently time to the screening process.

4.3. Study selection: screening for inclusion

This is the assessment of search results against your prespecified inclusion criteria. There are 2 phases: (1) title and abstract and (2) full-text screening. Full-text screening can be combined with the annotation and data extraction stage. A PRISMA flow diagram should be produced to report the number of records identified, included, and excluded, and the reasons for exclusions.³⁸

4.4. Annotation and data extraction

Annotation questions about study quality, risk of bias, and study design should be specific and objective, limiting the need for reviewer judgement. Avoid temptation to extract data not pertinent to the research question that will not be analysed. There are several tools to manually extract outcome data presented in graphs, eg, WebPlotDigitizer and the inbuilt Adobe measuring tool.

4.5. Analysis

The analysis of an SR can use qualitative⁵³ or quantitative, with MA⁵⁶ and without MA⁵ or mixed techniques. A narrative summary can be used to synthesise study design and risk of bias information. Vesterinen et al.⁵⁶ provide comprehensive guidance for conducting a preclinical MA. A MA can be used to combine the outcome data of individual studies to estimate the overall intervention effect. A stratified MA or meta-regression can be used to investigate and quantify potential sources of heterogeneity, for example, study design characteristics and how they influence outcomes. The presence and magnitude of publication bias can also be estimated using statistical methods such as funnel plots; see Refs. 54 and 56 for further reading.

4.6. Reporting

Sena et al.⁴⁵ provide guidelines for the reporting of preclinical SRs. All aspects of the review process should be reported in adherence to the protocol with explanation for any deviations. It is also helpful to refer to the AMSTAR 2 critical appraisal tool for assessing the methodological quality of SRs.⁴⁸ Finally, to ensure transparency and sustainability of the SR, it is encouraged to make the data and analysis code available by uploading to a repository, eg, Open Science Framework or Figshare. In doing so, you are making it possible for others to perform secondary analyses thereby increasing the reach of your work.

5. Improving feasibility: the design of the review

Embarking on a preclinical SR can be daunting due to the complex, resource-intensive, and time-consuming processes.^13,20,51 Systematic reviews within the pain field to date have sought to answer broad research questions.^13,20,51 These large reviews have provided understanding of the range and quality of a field, however, there are more feasible possibilities ie, conducting smaller reviews, for time poor researchers and students.

5.1. Research question

The research question should be narrow, clearly defined, and answerable. Limiting the scope to a specific population, intervention, comparator, and/or outcome measure are ways to improve feasibility. Performing trial searches will assist you to determine possible workload and hone your research question. Other limitations may be added, for example, publication date; however, this adds a source of bias and must be justified.

5.2. Inclusion criteria for meta-analyses

A MA is not always appropriate; a systematic search, screen, and annotation (study characteristics and risk of bias) can inform a narrative summary and prospective animal studies. If outcome data are required, it is possible to reduce the data extraction burden by having inclusion criteria for the MA. For example, Federico et al.²⁰ calculated effect sizes based upon the availability of time-course data. Similarly, this could be achieved by only including studies at low risk of bias. Such decisions need to be justified and stated a priori within the protocol.

5.3. Efficient resource allocation

Multiple reviewers can be used to expedite the screening, annotation, and outcome data extraction stages (in parallel on SyRF). If you aim to conduct SRs for student projects, a single research question composed of several subquestions can be addressed. Students can collaborate during the protocol development, search, screen, and data extraction stages, thereby contributing to each stage of the project. Students will be able to demonstrate independent working by using the data pertinent to a subquestion and perform and report independent analyses in their examination submissions.

6. Improving feasibility: contribution to reviews

Systematic reviewers are regularly looking for contributors and it is worth contacting the authors of registered SRs to offer your assistance; it is a very efficient way to gain experience. Importantly, it provides collaborative opportunities for researchers with limited resources, eg, in low- and middle-income countries. As part of the recent IASP Cannabinoid Task Force, the preclinical SR recruited a crowd of reviewers to assist with the screening and data extraction phases.⁵¹ CAMARADES are currently recruiting a crowd to help them build a systematic and continually updated summary of COVID19 evidence.¹⁴ These reviews demonstrate it is possible to share the workload across a crowd, recruited locally or globally, although we recommend conducting training for reviewers to ensure quality.

7. Improving feasibility: automation tools for evidence synthesis

Several groups are taking advantage of emerging technologies to modernise the conventional review process and create living SRs. Systematic reviews are not often updated²² and not incorporating the most recent data risks making an SR at risk of inaccuracy.^11,49 Living SRs are SRs that are continually updated, incorporating relevant new evidence as it becomes available.¹⁹ Living SRs will ensure that decisions are dynamic and based upon the full body of evidence. Technological developments are continually being made to reduce the time and human effort required for SRs and automation tools can be used without making the review living (Fig. 3).

Figure 3.

Automation technologies that are being developed for the different stages of the review process. Machine learning and text mining have improved the feasibility and efficiency of the early stages of the process^{2, 3} and tool development continues to ensure that the full potential of preclinical SRs are realised. Technological developments for the latter stages are in their infancy. However, we are currently developing a machine-assisted approach to extracting data from graphs that aims to reduce time and improve accuracy,¹⁰ a feature that will soon be integrated into SyRF. SRs, systematic reviews; SyRF, Systematic Review & Meta-analysis Facility.

8. The future

Performing a SR enables researchers to hone their critical analysis skills and gain an in-depth understanding of the field. Like the vision proposed by Nakagawa et al.,³⁹ we envisage a new community for pain research that comprises of primary researchers who perform research synthesis with support from systematic reviewers, librarians, and statisticians. Primary researchers will use SRs to generate hypothesis and inform future research design. Evidence synthesis will also be recognised as an end goal of the research process. Research will be designed, conducted, analysed, and reported accordingly (eg, for eligibility in prospective MAs as described by Seidler et al.⁴⁴), thereby mitigating biases and reducing research waste. Conducting SRs will lead to improvements in education, practice, and communication of pain research and improve the predictive validity of animal research, reduce research waste, and improve pain outcomes for patients.

Conflict of interest statement

The authors have no conflicts of interest to declare.

Supplemental video content

A video abstract associated with this article can be found at http://links.lww.com/PAIN/B114.