How to Conduct a Systematic Review and Meta-Analysis: A Guide for Clinicians

Abstract

Evidence-based practice relies on using research evidence to guide clinical decision-making. However, staying current with all published research can be challenging. Many clinicians use review articles that apply predefined methods to locate, identify, and summarize all available evidence on a topic to guide clinical decision-making. This paper discusses the role of review articles, including narrative, scoping, and systematic reviews, to synthesize existing evidence and generate new knowledge. It provides a step-by-step guide to conducting a systematic review and meta-analysis, covering key steps such as formulating a research question, selecting studies, evaluating evidence quality, and reporting results. This paper is intended as a resource for clinicians looking to learn how to conduct systematic reviews and advance evidence-based practice in the field.

Introduction

Evidence-based practice is the integration of scientific evidence with clinical expertise and patient values. Using evidence from high-quality research to guide clinical practice ensures better and safer patient outcomes and more efficient health care.^1–3 Whereas evidence-based practice is a foundational element of modern health care, locating, appraising, and incorporating research evidence in practice is difficult and time consuming. Staying current with the plethora of published research evidence is impossible. For example, a rapid search on PubMed reveals that approximately 30,000 articles containing the words “respiratory care” were indexed in 2022, which is 2.5 times higher than in 2012, and this trend is not unique to respiratory care.⁴ Several research articles suggest that primary care physicians need to read 19 articles every day and spend 627.5 hours per month to “keep up with the literature.”⁵ These numbers suggest that clinicians, in their limited time, must be selective in which articles they read to ensure they remain up to date. Certain methodologies exist that aim to group similar studies together to facilitate easy access to evidence for busy clinicians. These groupings are referred to as knowledge synthesis and are more commonly known as review articles.

Knowledge synthesis is an umbrella term for different types of literature reviews that use transparent and reproducible methods to gather and synthesize studies within the larger body of knowledge on the topic. A discussion about the type, strengths, and weaknesses of every type of knowledge synthesis (eg, practice guidelines, realist reviews, etc) is beyond the scope of this paper.^6–9 In this paper, we first briefly discuss the differences between a narrative and scoping review while refocusing the discussion on systematic reviews. Finally, we provide an overview for clinicians regarding how to conduct a systematic review (and meta-analysis when appropriate) according to its key methodological steps. Systematic reviews are a category of papers published in Respiratory Care. Our goal for this paper is not to replace key existing guidelines^10,11 but rather to provide clinicians with an accompanying document that acts as a primer for those interested in learning more about systematic review methodology.

Types of Reviews

Narrative, scoping, and systematic reviews are commonly published in Respiratory Care, and each review type serves a specific purpose and involves different methods for collecting and synthesizing evidence.

Narrative Reviews

In a narrative review, researchers aim to provide an extensive description and interpretation of previous publications on a chosen topic.^12,13 It is defined as a scholarly report of a body of literature that includes interpretation and critique.⁹ Narrative reviews provide a flexible approach to exploring existing debates on a topic, analyzing and interpreting the literature to deepen the understanding of a certain research area.^12,13 Experts are often sought to write narrative reviews due to their deep knowledge and understanding of a topic. This expertise allows them to provide a comprehensive overview and integrate various perspectives and ideas related to the topic. The topics covered in narrative reviews often span multiple areas and require a nuanced understanding of the subject matter. Recent examples in Respiratory Care include narrative reviews of respiratory drive and silent hypoxemia,¹⁴ high-frequency jet ventilation,¹⁵ and consensus methodology in respiratory therapy.¹⁶ Papers arising from journal conferences, honor lectures, and New Horizons are other examples of narrative reviews.^17–19

Whereas narrative reviews can be informative, they can include elements of bias in the results if not transparently conducted. Narrative reviews often do not include a section describing the methods used in the review. Researchers conducting narrative reviews may (or may not) use systematic search methods with fixed inclusion and exclusion criteria. Often, researchers (as the experts) select key articles to use for their paper, which could introduce bias (eg, selection bias, author bias) and might undermine the validity of their claims. The omission of certain steps and potential bias indicates that the results from narrative reviews may be less comprehensive and reliable.

Scoping Reviews

Scoping reviews are narrative reviews that follow a formal, structured methodological process using transparent methods and are reported using standardized guidelines.^10,20–22 The decision to conduct a scoping review depends on the nature of the research question and the aims of the project. There are 6 main reasons for researchers to undertake a scoping review: (1) identify main concepts and definitions on a topic, (2) determine the primary dimensions of a concept, (3) uncover the types of evidence available on a topic, (4) determine gaps in the literature, (5) take a preliminary step toward conducting a systematic review, and/or (6) better understand how research has been carried out on a topic.^20,23

Researchers conducting scoping reviews do not aim to produce a critically appraised and synthesized answer to a specific question. Rather, their aim is to systematically map and discuss the breadth and depth of the literature on a topic.²⁰ Because of this, researchers often do not conduct an assessment of methodological limitations or risk of bias of the evidence included in the scoping review, which some perceive to bias the results of the scoping review. Given that an assessment of bias is not conducted, the implications for practice (from a clinical or policy-making perspective) that arise from a scoping review are different compared to those of a systematic review.²⁰ Examples of scoping reviews in respiratory therapy include mapping what is known about ultrasound training for respiratory therapists²⁴ or mapping what is known about telerehabilitation for respiratory tract disease.²⁵

Systematic Reviews

Systematic reviews are the most common knowledge synthesis published in the literature.²⁶ The Institute of Medicine (now the National Academy of Medicine) defines a systematic review as “a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar studies.”²⁷ Key features of systematic reviews are the detailed and comprehensive plan and the search strategy (developed a priori), with the goal of reducing bias by identifying, appraising, and synthesizing all relevant studies on a particular topic. Systematic reviews are best to answer questions about effectiveness (ie, whether complex interventions work) and usually use findings from randomized controlled trials (RCTs) to answer these questions.²⁸ However, RCTs sometimes do not exist for a particular topic, so many researchers also include other study designs (eg, observational designs) in systematic reviews to answer more complex research questions.

Systematic reviews are considered higher-quality evidence for clinical decision-making because (if rigorously done) they provide less biased evidence than narrative reviews due to their stricter methodology.²⁹ The main strength of systematic reviews is their rigorous and clearly defined process to identify, critically appraise, and distill all of the individual studies on a topic to provide recommendations to inform practice. Additionally, explicit methods to grade the literature help limit potential bias. Another strength of the systematic review is that it aims to answer a narrowly focused question with a clearly defined population, intervention, comparison, and outcomes. The focus on a well-defined question helps readers to rapidly determine if the results of the review apply to the needs of their patients in their context.

Guidelines regarding how to conduct and report systematic reviews are well documented and freely available to guide researchers to ensure systematic reviews are conducted rigorously, reported transparently, with minimal bias, and reproducible. For freely accessible guidelines, the Cochrane Handbook¹¹ and Joanna Briggs Institute Manual for Evidence Synthesis¹⁰ are detailed and authoritative on how to conduct systematic reviews. In addition to these, researchers should consult the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist to report systematic reviews.³⁰ Many journals require that the PRISMA checklist accompanies the manuscript submission.

Despite many strengths, systematic reviews have some weaknesses. Whereas the narrowly focused question of a systematic review is viewed as a strength, it can also be viewed as a weakness. If a research question is too narrow, it may not be of interest to readers or useful in clinical decision-making.^13,31,32 Researchers should consider the relevance and usefulness of their research questions to ensure that their results can inform decision-making and clinical practice.³³ Additionally, because systematic reviews are frequently cited, some researchers conduct them less rigorously and hope it will be published just because it is labeled as a systematic review.^34,35 Finally, systematic reviews are highly resource- and time-intensive projects, which take months to years to properly complete (excluding the time for publication).³⁶ Table 1 includes the major differences between narrative, scoping, and systematic reviews.

Table 1.

A Comparison of Narrative, Scoping, and Systematic Reviews

How to Conduct a Systematic Review

Preparing for the Systematic Review

After determining that a systematic review is the appropriate knowledge synthesis for the purpose of the researchers' project, the first step is to gather the appropriate team members. Systematic reviews are conducted by (often large) teams. Published literature suggests the mean number of members involved in conducting a systematic review is 5 (± 3).³⁶ Whereas the number of team members and their expertise vary, it is suggested that a review team comprises individuals with experience conducting systematic reviews, content experts, information specialists (ie, a medical librarian), and statisticians (if warranted by the research question).^37–39 A proper team can ensure a high-quality systematic review.³⁷

Beginning the Systematic Review

Figure 1 represents the life cycle of a systematic review project.⁴⁰ As with all research, the most critical step is to develop a clear and well-articulated research question. In systematic reviews, the question guides many aspects of the review process, including determining eligibility criteria, searching the literature, extracting data from included studies, structuring the syntheses, and presenting findings.⁴¹ In systematic reviews, the question should state the patient group of interest, the intervention being investigated, the control or comparator group, and the outcomes of interest. Many researchers use the PICO mnemonic (an acronym for population, intervention, comparison[s], and outcome) to guide their question design. Table 2 provides examples of PICO questions from published systematic reviews. Whereas PICO is the most common mnemonic for systematic reviews, other approaches exist to help researchers structure their review questions.⁴²

Fig. 1.

The review process. Modified from Reference 40. PICO = population, intervention, comparison(s), and outcome; MeSH = medical subject headings list of the National Library of Medicine; PRISMA = Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Table 2.

Population, Intervention, Comparison(s), and Outcome Examples

Preliminary Research and Idea Validation

As the research team begins to develop and refine the research question, they should do a preliminary search of the literature to determine what has already been published, ensure the validity of the proposed idea, avoid duplication of previously published systematic reviews, and ensure enough studies have been published to conduct an analysis (a minimum of 2 similar studies are needed to conduct a meta-analysis).^11,43 Doing this preliminary search sometimes influences how the eventual systematic review PICO question will be framed. Developing the PICO question and doing preliminary searches of the literature are an iterative process.

Creating and Registering a Protocol

Once the research question is finalized, the research team creates a protocol. A protocol is a description of the proposed systematic review, including methods, the rationale for the review, and steps that will be taken to eliminate bias while conducting the review. After creating a protocol, researchers should register it in an online database of systematic review protocols (eg, PROSPERO, the Open Science Framework). Registering the work promotes transparency, helps reduce the potential for bias, and serves to avoid unintended duplication of reviews.^44,45 Whereas registering the protocol is usually not mandatory for publication,⁴⁶ anecdotally, journal editors are emphasizing the necessity of registering a protocol.

Creating and Conducting the Systematic Search of the Literature

Collaborating with an information specialist to develop and run the search strategy is highly recommended. Whereas they are not content experts, they are trained to analyze complex questions and formulate comprehensive search strategies. They are skilled in using controlled vocabularies and Boolean operators (eg, AND/OR functions in databases) and fully understand the benefits and nuances of multiple databases.⁴⁷ Researchers may decide to build their own search strategies using existing resources.⁴⁸ However, having an information specialist as co-author is associated with more reproducible searches and higher search quality, which is a hallmark of a good systematic review.⁴⁹

The systematic review search strategy is built on the research question. Each aspect of the PICO mnemonic builds a portion of the search strategy, which often contains 3 of the 4 research question components to keep the search narrow but not overly narrow where researchers might miss relevant articles. Figure 2 depicts a visual representation of a search strategy using the PICO format, and Online Supplement 1 (See related supplementary materials at http://www.rcjournal.com) is a full MEDLINE search from a published systematic review.⁵⁰

Fig. 2.

An example of visual representation of a search strategy using the population, intervention, comparison(s), and outcome format. From Reference 50, with permission. MeSH = medical subject headings list of the National Library of Medicine.

The goal of the systematic search is to identify all available studies that might fit the review's inclusion criteria. The search must be done in a minimum of 3 databases. The Cochrane Handbook states that it is mandatory to search CENTRAL, MEDLINE, and Embase but suggests searching other specialized databases depending on the nature of the research question. For example, the CINAHL database has extensive coverage of nursing and allied health, including nursing and rehabilitation journals not covered by MEDLINE (particularly relevant for respiratory therapists); and PsycINFO is an excellent resource for research on psychological, social, behavioral, and mental health questions.⁵¹ An unintended consequence of searching multiple databases is researchers obtain duplicate studies, which can require significant time for the research team to deduplicate.

After searching the databases, researchers need to note the date the searches are run and how many papers are retrieved from each database.⁵² It is crucial that researchers keep meticulous notes about these searches as the PRISMA reporting guidelines require authors to include the full search strategy (Online Supplement 1) for at least one database in the paper's appendixes.³⁰ Also, searches might need to be rerun and updated if there is a significant delay between when it was run and when the article is submitted for publication.⁵³

Screening Articles

Once the search is conducted, the data set is deduplicated using systematic review software programs (eg, Rayyan, Covidence)⁵⁴ or reference management software (eg, EndNote, Zotero).^55,56 A minimum of 2 team members should independently screen all titles and abstracts of the papers to determine if they meet the inclusion criteria aligned with the original research question.⁵⁷ Screening articles can be done manually or with the software programs used to deduplicate. In our experience conducting knowledge syntheses, we also advocate for those involved in screening the articles to conduct a calibration exercise to ensure each team member applies the criteria similarly and to minimize discrepancies. Specifically, both team members take the same subset of papers from the data set (eg, 5%) and apply the inclusion/exclusion criteria to see if their decisions match and assess their decisions with a metric of quality (eg, > 90% agreement or an acceptable inter-rater reliability score).⁵⁸ This process is repeated for the full-text screening, and a third team member would be available to mediate any disagreements between the 2 team members.

Data Extraction and Analysis

Once the team has determined the final papers that meet their inclusion criteria, at least 2 team members should independently extract data from the individual studies to reduce bias and minimize the potential for errors.⁵⁹ To do so, the team members should create and pilot the data extraction form on a subset of papers using a calibration exercise. The content of the data extraction form will be based on the research questions and can be analyzed qualitatively (eg, in a narrative form) or quantitatively (using a meta-analysis).

Risk of Bias Assessment (Quality Assessment or Critical Appraisal)

Part of the data extraction process is also to determine the quality of each included paper within the systematic review. Just because a paper is published in a journal does not mean it is of high quality.⁶⁰ Individual studies that meet the inclusion criteria may include biases in their results. These biases are a result of inherent methodological and analytical flaws that raise questions about the validity of the study's findings.¹⁰ Many tools have been developed to assess the risk of bias, and the one researchers choose depends on the papers included in the systematic review. No matter which tool is used, researchers must provide a rationale as to why they chose that particular risk of bias tool. There are several commonly used risk of bias tools.⁶¹

If the researchers' systematic review only includes data from RCTs, the Cochrane collaboration advises researchers to use the risk of bias in randomized trials (RoB 2 tool),^62,63 which assesses each RCT using a simple judgment of low risk (−), high risk (+), or unclear risk (?) across different types of bias.⁶⁴ (Online Supplement 2 includes definitions for different types of biases; see related supplementary materials at http://www.rcjournal.com.) Figure 3 provides an example of how the RoB 2 tool conveys the risk of bias in a published paper.⁵⁰

Fig. 3.

An example of risk of bias assessment for randomized clinical trials. From Reference 50, with permission.

In some systematic review topics, it might not be possible to only include RCTs because either RCTs have not been published, are not feasible, or the topic would be unethical to conduct an RCT (eg, randomizing a group to vaping vs not vaping). In this case, researchers rely on non-randomized, primary studies and use other tools to assess the risk of bias. The 2 most common tools are the Risk of Bias in Non-randomized Studies-of Interventions (ROBINS-I) and the Newcastle-Ottawa scale.

The ROBINS-I is a tool developed to assess the risk of bias in the results of non-randomized studies of interventions.⁶⁵ The types of papers that can be evaluated using this tool are quantitative studies estimating the effectiveness (harm or benefit) of an intervention that did not use randomization to allocate units.⁶⁵ These are observational studies such as cohort studies and case-controlled studies.⁶⁵

The Newcastle-Ottawa scale is another tool developed to assess the risk of bias in the results of non-randomized studies included in systematic reviews.⁶⁶ Using the tool, each study is judged on 8 items, categorized into 3 groups: the selection of the study groups, the comparability of the groups, and the ascertainment of either the exposure or outcome of interest for case-control or cohort studies, respectively.

The Cochrane collaboration prefers the ROBINS-I tool, but it is not mandatory because this tool is time consuming and requires sufficient knowledge and experience to complete. They suggest the Newcastle-Ottawa scale as an alternative option.⁶⁷ Published research suggests that both the ROBINS-I and the Newcastle-Ottawa scale provide the same level of reliability evidence, but the Newcastle-Ottawa scale is more frequently applied in the literature.⁶⁸

The GRADE System

The grading of recommendations, assessment, development, and evaluations (GRADE) framework is used if the intent of the systematic review is to develop clinical practice guidelines (CPGs).⁶⁹ However, it is important to recognize that many systematic reviews stand alone without creating recommendations. CPGs are almost always sponsored by professional societies rather than smaller research teams. The American Association for Respiratory Care,⁷⁰ American Thoracic Society,⁷¹ American College of Chest Physicians,⁷² and the Society of Critical Care Medicine⁷³ each have processes in place to develop CPGs. Although GRADE is most used, other approaches for developing CPGs include Convergence of Opinion on Recommendations and Evidence and RAND/UCLA Appropriateness Method.^74–76

GRADE uses a summary of findings from a systematic review to make recommendations.^77,78 Whereas the previously discussed risk of bias assessment evaluates the quality and limitations of a particular study, researchers use the GRADE system to evaluate each outcome across different studies included.⁷⁸ The GRADE system is based on the principle that the quality of evidence can vary and that the strength of the recommendations should be based on the best available evidence. The key concepts in the GRADE system include:

1. Summary of findings: A table summarizing the evidence that informs the recommendations. It includes a summary of the main outcomes such as quality assessment, number of patients, effect, quality, and importance.

2. Quality of evidence: The quality of evidence refers to the degree of confidence that the evidence supports the findings of a study. The GRADE system classifies the quality of evidence using 4 levels: high, moderate, low, and very low (Table 3). These ratings depend on multiple factors (eg, the study design, the sample size, the population under study, the risk of bias, and the consistency and precision of the results). For example, RCTs always start as high-level evidence, and observational studies begin as low level. However, based on criteria specified by GRADE, the level of evidence can be upgraded or downgraded.

3. Strength of recommendation: The strength of recommendation refers to the level of certainty about the benefits and harms of an intervention or policy. Recommendations can be strong or conditional (weak), depending on the quality of the evidence and the balance between the benefits and harms of the intervention.

Table 3.

Quality of Evidence and Definitions Used Within the Grading of Recommendations, Assessment, Development, and Evaluations System

An example summary of findings from a systematic review is shown in Table 4.⁵⁰ Note that this is from a systematic review and not a CPG per se. Focusing on the outcome of “ICU stay,” 5 RCTs were included and had high risk of bias, so the authors graded this outcome as a low certainty of evidence. This suggests that more research is required to confidently determine that the intervention affects the outcome. Goldet et al⁷⁸ provide a comprehensive and clear explanation of the GRADE rating system, making it an accessible resource to learn the system.

Table 4.

Grading of Recommendations, Assessment, Development, and Evaluations on an Example⁵⁰

Meta-Analysis

A meta-analysis can be performed after completing a systematic review. A meta-analysis is a statistical method to combine the results of similar independent studies synthesized in a particular systematic review to provide more precise estimates of the effects of health care treatments or risk factors for disease or other outcomes than those derived from the individual studies included within a review.^9,79,80 A meta-analysis is a quantitative summary of results of a systematic review. A meta-analysis cannot stand alone without a preexisting systematic review. Grading individual research studies allows studies with the most rigorous methodology to carry more weight in the final review than those studies that lack important methodological elements.^79,81

Not all systematic reviews contain meta-analysis nor should they. Researchers need a minimum of 2 similar studies to perform a meta-analysis. Conducting a meta-analysis requires that the same measure or outcome be measured in the same way at the same time intervals^9,82 and the outcomes be quantitative (eg, mortality, intubation rate, hospital stay). Meta-analyses are most commonly used for combining data from RCTs. Observational studies can also be included in meta-analyses, but this is controversial as observational studies are more prone to bias, and this can significantly impact the outcome of a meta-analysis.^81,82 The overall result of a meta-analysis is presented as a forest plot, as illustrated in Figure 4.⁵⁰

Fig. 4.

Explanations on an example of meta-analysis forest plot. From Reference 50, with permission.

Presenting Systematic Review (and Meta-Analysis) Results

It is important to present the results of a systematic review and meta-analysis in a clear, transparent, and unbiased way. This helps readers understand the main findings and assess reliability and relevance.

PRISMA Flow Diagram

The PRISMA flow diagram allows readers to have a general idea of the process of the entire systematic review project.⁸³ A high-quality PRISMA flow diagram contains full items and counts for all 4 stages (identification, screening, eligibility, and inclusion).^30,83 Figure 5 is an example.⁵⁰ Online Supplement 3 (See related supplementary materials at http://www.rcjournal.com) includes a blank example flow diagram.^30,84

Fig. 5.

An example of Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowchart. From Reference 50, with permission.

Characteristics of the Studies, Results of Risk of Bias Assessment, and GRADE

Researchers then describe the studies included in the review (ie, their design, sample size, population, intervention, and outcomes). After the description, provide a summary of the risk of bias (ie, quality assessment) of the included studies and a summary of the overall results of the quality assessment. A GRADE assessment is needed if the systematic review is intended to lead to CPGs.

Summary of Main Findings

The final aspect of the results section provides a summary of the main findings of the systematic review. The usual sequence of presenting the main findings in systematic reviews is to report the primary outcome first, then the secondary outcomes, and so on. This is because the primary outcome is usually the main focus of the review and is considered the most important in decision-making. The secondary outcomes provide additional information and provide context for the primary outcome.

The Discussion of a Systematic Review (and Meta-Analysis)

Do Not Just Restate the Results Section

The discussion section is not merely a place to restate the results section.⁸⁵ Generally, in the discussion section, researchers will summarize the main results of the review and aim to integrate those findings within the larger context of the body of knowledge. Researchers can discuss contradictory evidence or surprising evidence. Research should state the clinical relevance of the findings and consider the implications of the review. What are the practical and clinical implications of the review for readers (eg, clinicians and policymakers)? Acknowledge the limitations (and strengths) of the review and propose areas for future research.

Using PRISMA as a Guide in Designing the Project and Writing

Use the PRISMA checklist as a framework to structure their manuscript writing in preparation for publication. Using the checklist ensures that you have included every major section of a systematic review in your manuscript. If sections of the checklist do not apply, acknowledge those missing sections and provide a rationale for why it is not applicable. Do not completely omit a section.

Summary

Evidence-based practice relies on research to guide clinical decision-making. However, staying current with all the published research can be challenging for busy clinicians. Many clinicians rely on different knowledge syntheses to guide clinical decision-making. In narrative reviews, researchers aim to provide an extensive description and interpretation of previous publications on a chosen topic without using clear and explicit guidelines. In scoping reviews, researchers use systematic methods to map the breadth and depth of a topic but often don't critically appraise and grade the included literature. In systematic reviews, researchers use a strict methodology to synthesize all relevant studies on a topic and critically appraise the evidence to provide recommendations to inform practice. Systematic reviews are considered higher-quality evidence because they provide (if rigorously done) less biased evidence than narrative reviews. Properly conducting a systematic review is a resource- and time-intensive project and requires attention to many interrelated details.