Systematic Reviews in Medical Education

Foundations

In medicine, physicians are charged to use the best available evidence to make decisions about the benefits and harms of clinical interventions. To this end, physicians have come to rely on systematic reviews to provide evidence in a synthesized form.¹ In medical education, practitioners are similarly encouraged to utilize systematic reviews to facilitate the practice of evidence-based education,^2,3 such that systematic reviews are the most published form of knowledge synthesis in medical education, with over 300 published in the last 20 years.⁴

The Institute of Medicine defines a systematic review as “a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar studies.”¹ Systematic reviews can also include a meta-analysis to demonstrate evidence through statistical power by combining results from individual studies. Taking root in the positivist tradition (ie, an absolute truth can be determined from scientific experiments), systematic reviews have traditionally focused on answering questions about effectiveness (eg, whether an intervention works) and have relied heavily on synthesizing findings of randomized control trials.^5,6 However, over time, the scope of systematic reviews has expanded to include multiple study designs and to answer a variety of research question types. For example, in the context of medical education, the following are illustrative questions that could be answered by a systematic review:

• ▪Does this training work? Does it cause harm? (Intervention effectiveness)
• ▪What are the costs of implementing simulation? (Intervention costs)
• ▪What is the prevalence or incidence of burnout? (Prevalence, incidence)
• ▪How accurate is this assessment tool? (Test accuracy)
• ▪Are learners who are exposed to bullying at risk for suicidality? (Causation, risk factors)

In Box 1, The Case of Dr Smith, a program director considers the use of a systematic review to answer an important question for the program. Each article in this JGME special series on conducting and reading reviews will consider this same question through a different review methodology and philosophical underpinning.⁷

Box 1 The Case of Dr Smith⁷

Given Dr Smith's interest in the effectiveness of interprofessional education (IPE) interventions, she decides to conduct a systematic review because it will help her answer the question of whether IPE works. It will also enable her to determine the outcomes that IPE produces.

To get started, Dr Smith would be advised to carefully examine the large and varied existing literature on IPE. While she has decided to conduct a systematic review, depending on her exploration of the literature, she may decide that it is warranted to move beyond determining the general effectiveness of IPE and to pose more targeted study aims. For example, she may aim to determine the effectiveness of a particular approach to IPE vs another form (eg, IPE longitudinal initiatives vs those of a shorter duration) if there are ample numbers of studies covering different forms of IPE. Alternatively, she may decide to focus her systematic review on studies that report only high-level outcomes, such as those at the behavioral patient outcomes level. To refine her study aims, Dr Smith would be encouraged to speak with stakeholders in the field to understand which interventions are most salient. In this way, Dr Smith will ensure that not only is she able to determine the effectiveness of IPE, but also will be able to provide evidence that is useful to the medical education community.

Strengths and Weaknesses

Like all knowledge syntheses, systematic reviews have strengths and weaknesses that should be considered. A major strength of systematic reviews is that, through a rigorous and clearly defined process, they aim to identify, critically appraise, and distill all of the individual studies on a topic to provide readers with recommendations to inform their practice.⁸ Another strength of a systematic review is that it aims to answer a focused, specific question, with a clearly defined population, intervention/exposure, comparison, and outcomes. This focus on a well-defined question concentrates all aspects of the conduct of the review and helps readers readily determine whether the review applies to their situation and context.⁹

Clear guidance exists for conducting and reporting systematic reviews, which can facilitate the systematic review process, decrease bias, increase research transparency, and enable reproducibility.¹⁰ For example, researchers can access the freely available Cochrane Handbook¹¹ or the JBI Manual for Evidence Synthesis,¹² which provides detailed instructions and considerations for conducting reviews. Specific to the context of medical education, there are resources, such as a Best Evidence Medical Education (BEME) guide on conducting systematic reviews,¹³ an article by Cook and West that describes a stepwise approach,¹⁴ and 2 “Twelve Tips” articles.^15,16 For transparency in the reporting of systematic reviews, researchers are referred to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)¹⁷ and its recent extension specific to searches.¹⁸ Those conducting systematic reviews in medical education would also benefit from reviewing the STructured apprOach to the Reporting In healthcare education of Evidence Synthesis (STORIES; see Box 2).¹⁹

Box 2 Key Resources

Publication Guidelines/Standards/Manuals

• ▪Aromataris E, Munn Z, eds. JBI Manual for Evidence Synthesis. JBI, 2020. Available from https://synthesismanual.jbi.global. https://doi.org/10.46658/JBIMES-20-01
• ▪Gordon M, Gibbs T. STORIES statement: publication standards for healthcare education evidence synthesis. BMC Med. 2014;12(1):143. doi:10.1186/s12916-014-0143-0
• ▪Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, eds. Cochrane Handbook for Systematic Reviews of Interventions. Cochrane, 2020. Available from www.training.cochrane.org/handbook
• ▪Page MJ, Moher D, Bossuyt PM, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160. doi:10.1136/bmj.n160
• ▪Gordon M, Grafton-Clarke C, Hill E, Gurbutt D, Patricio M, Daniel M. Twelve tips for undertaking a focused systematic review in medical education. Med Teach. 2019;41(11):1232-1238. doi:10.1080/0142159X.2018.1513642

Medical Education Focused Resources

• ▪Bearman M, Dawson P. Qualitative synthesis and systematic review in health professions education. Med Educ. 2013;47(3):252-260. doi:10.1111/medu.12092
• ▪Best Evidence in Medical Education Collaboration. https://www.bemecollaboration.org/
• ▪Cook DA, West CP. Conducting systematic reviews in medical education: a stepwise approach. Med Educ. 2012;46(10):943-952. doi:10.1111/j.1365-2923.2012.04328.x
• ▪Choi AR, Cheng DL, Greenberg PB. Twelve tips for medical students to conduct a systematic review. Med Teach. 2019;41(4):471-475. doi:10.1080/0142159X.2018.1426847
• ▪Gordon M, Vaz Carneiro A, Patricio M, Gibbs T. Missed opportunities in health care education evidence synthesis. Med Educ. 2014;48(6):644-645. doi:10.1111/medu.12478
• ▪Gordon M, Grafton-Clarke C, Hill E, Gurbutt D, Patricio M, Daniel M. Twelve tips for undertaking a focused systematic review in medical education. Med Teach. 2019;41(11):1232-1238. doi:10.1080/0142159X.2018.1513642
• ▪Maggio LA, Tannery NH, Kanter SL. Reproducibility of literature search reporting in medical education reviews. Acad Med. 2011;86(8):1049-1054. doi:10.1097/ACM.0b013e31822221e7
• ▪Maggio LA, Thomas A, Chen HC, et al. Examining the readiness of best evidence in medical education guides for integration into educational practice: a meta-synthesis. Perspect Med Educ. 2018;7(5):292-301. doi:10.1007/s40037-018-0450-9
• ▪Sharma R, Gordon M, Dharamsi S, Gibbs T. Systematic reviews in medical education: a practical approach: AMEE guide 94. Med Teach. 2015;37(2):108-124. doi:10.3109/0142159X.2014.970996
• ▪Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. doi:10.1136/bmj.j4008

While a systematic review's targeted question can be considered a strength overall, this narrow focus may not allow for the comprehensive overview of a topic sought by most educators. For example, a reader may be interested in learning more about feedback, but find that a systematic review on the topic focuses narrowly on specific types of feedback or a very focused population. In these cases readers may do well to search for a narrative review or scoping review. Readers should also recognize that systematic reviews do not answer questions about why and how an intervention does or does not work, which is more the domain of a realist review.⁹

To conduct a systematic review, authors need to analyze primary studies. In some cases topics are discussed in the literature, but their coverage is limited to commentaries, perspectives, or a few preliminary studies. Medical education populations and settings are usually diverse and difficult to tightly define. Interventions may also include subtle or obvious differences among studies, and “control” groups are actually difficult to control in the medical education realm. Thus, systematic review searches with well-defined criteria may find few papers to synthesize. Therefore, it is not feasible to conduct a systematic review until there is a significant body of evidence (ie, number of relevant articles) available, which can mean that a systematic review might not be published until the topic is quite established in the literature. In these cases, researchers might consider an alternate knowledge synthesis type, such as a scoping review, to provide an overview of the topic and a map of the current literature (see Box 3).²⁰

Box 3 Sample Medical Education Systematic Review Articles

• ▪Birden H, Glass N, Wilson I, Harrison M, Usherwood T, Nass D. Teaching professionalism in medical education: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 25. Med Teach. 2013;35(7):e1252-e1266. doi:10.3109/0142159X.2013.789132
• ▪Maggio LA, Aakre CA, Del Fiol G, Shellum J, Cook DA. Impact of clinicians' use of electronic knowledge resources on clinical and learning outcomes: systematic review and meta-analysis. J Med Internet Res. 2019;21(7):e13315. doi:10.2196/13315
• ▪Sadowski B, Cantrell S, Barelski A, O'Malley PG, Hartzell JD. Leadership training in graduate medical education: a systematic review. J Grad Med Educ. 2018;10(2):134-148. doi:10.4300/JGME-D-17-00194.1
• ▪Skjevik EP, Boudreau JD, Ringberg U, et al. Group mentorship for undergraduate medical students—a systematic review. Perspect Med Educ. 2020;9(5):272-280. doi:10.1007/s40037-020-00610-3

Lastly, like other knowledge synthesis types, systematic reviews are a major undertaking and are highly resource- and time-intensive.¹¹ Researchers must accept that systematic reviews take time to conduct: researchers estimate it takes from 6 months to 2 years,²¹ in addition to the time for the publication process.²² This can affect the timeliness of access to distilled evidence and has implications for researchers with promotion or graduation deadlines.

Hallmarks of a Quality Systematic Review

Systematic reviews should exhibit multiple hallmarks of quality. As a baseline, we encourage readers to consider the elements put forward by PRISMA (eg, including a research question, reporting a complete search strategy, using multiple coders).¹⁷ In addition to PRISMA, we recommend several hallmarks of quality specific to systematic reviews conducted in medical education.

As health professions education cuts across multiple fields (eg, education, psychology, sociology), a hallmark of a high-quality systematic review is that it includes a search of multidisciplinary and education-focused databases (eg, Web of Science, ERIC [Education Resources Information Center], PsycINFO), which helps to ensure the inclusion of relevant studies published outside of journals focused on biomedicine. Additionally, when appropriate for the topic, the review should provide details of teaching strategies, discussions of theory use included in the studies, and information about the educational context of the intervention.¹⁸ The systematic review should also contain an appraisal and description of the quality of the included studies. In medical education, quality indices for quantitative studies include the Medical Education Research Study Quality Instrument (MERSQI) and the Newcastle-Ottawa Scale–Education (NOS-E).²³

Process

Below we provide a brief overview of key steps in conducting a systematic review; however, we strongly advise that researchers consult other resources, such as the Cochrane Handbook,¹¹ for further details and a full rationale for each step. Additionally, we encourage researchers to familiarize themselves early with the relevant reporting guidelines (eg, PRISMA or STORIES) to facilitate robust reporting and to ensure that they meet reporting requirements for their targeted journal.

Step 1: Formulate the Research Question

Researchers may consider approaches such as PICO (Population, Intervention, Comparison, Outcome)²⁴ or SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type)²⁵ to guide question design. For example, using SPIDER, a research question could be: What are residents' (sample) experiences (evaluation) of burnout (phenomenon of interest) as explored through qualitative (research type) interviews (design)? Dedicating time upfront to formulating a strong research question will pay dividends, as the research question informs almost all aspects of the review.

Step 2: Assemble the Research Team

Strategically assemble a team with the necessary expertise to answer the research question. Recruitment may include subject matter experts, information professionals, and statisticians, if conducting a meta-analysis. We also suggest considering the inclusion of stakeholders (eg, learners, faculty). Discuss upfront the time commitments necessary to fully participate in a systematic review to ensure team members are comfortable with the overall research plan and time line. Systematic reviews conducted by a solo researcher are strongly discouraged.

Step 3: Register a Protocol of the Systematic Review

The team should register a protocol of the review with a systematic review register (eg, PROSPERO) to identify similar efforts²⁶ and to satisfy journal editors who increasingly require registration.

Step 4: Design and Execute Search Strategies

We encourage researchers to collaborate with an information professional to design a comprehensive search strategy that includes multiple databases and combines hand searching of key journals and the grey literature. Keep meticulous records of all searches and note the search dates. Keep in mind that it may be necessary to update searches if there is a significant lag time between the initial search and submission to a journal.

Step 5: Screen for Inclusion and Exclusion

Using inclusion criteria formulated in alignment with the research question, at least 2 team members should independently screen all titles and abstracts of citations to determine if they meet the inclusion criteria. For those meriting full-text review, 2 team members should also independently screen the full-text articles for inclusion. In both screenings, the presence of 2 independent raters helps to minimize bias. The availability of a third researcher to serve as a tiebreaker can also be useful.

Step 6: Analyze and Synthesize the Data

To extract data from the primary studies, the researchers should create and pilot a data extraction tool to be applied to the full text of included studies by a minimum of 2 team members. Depending on the review's study question, extracted data can be analyzed qualitatively or quantitatively, including through meta-analyses. Meta-analysis of quantitative studies is possible only if the data from the individual studies are homogenous; if data are too heterogeneous, as is often found in medical education, it is not possible to accurately perform a meta-analysis. In this step, the authors should also critically appraise the included studies using such tools as the MERSQI and NOS-E.²³

Step 7: Write Up the Review

All authors should participate in the drafting and revision of the review manuscript. When writing, ensure that the manuscript meets the reporting guidelines (eg, PRISMA, STORIES) of the target journal. Also, to increase the usability of the review, ensure that readers are provided a cohesive discussion of the findings in relation to the existing literature. To facilitate this step, keep in mind what will be valuable to stakeholders attempting to integrate the findings into their educational practices.

Conclusion

Undertaking a systematic review is an intensive process designed to produce a rigorous product that provides readers a synthesis of the evidence to answer a specific research question. While systematic reviews have both strengths and weaknesses, they are commonly conducted in the context of medical education and broadly accepted by journals in the field. Researchers considering conducting systematic reviews are advised to consult the variety of guidance available before beginning their research.