The scoping review: A flexible, inclusive, and iterative approach to knowledge synthesis

INTRODUCTION

The body of medical education research has exploded in recent years. A push for a communal effort among education researchers to answer the “big questions” and support evidence‐based approaches to education has resulted in a significant rise in the number of medical education publications. ¹ , ² , ³ Sifting through this expanding body of work can present a daunting task. For example, PubMed, which is generally considered one of the primary databases for health sciences literature, contains over 30 million citations and counting. ⁴ Additional education‐specific and other searchable databases contain billions more citations through which to sort. ⁵

For medical educators balancing clinical work with teaching and research, finding efficient ways to manage a rapidly expanding volume of literature has become increasingly difficult. In addition to the challenges of time constraints and the sheer size of available databases, knowledge syntheses in medical education are fraught with challenge due to the breadth and complexity of the field. Out of this milieu, collaborations such as the Best Evidence in Medical Education (BEME) that works to publish high‐quality systematic reviews have emerged to address the increasing need for efficient yet comprehensive assessment and synthesis of the literature. ⁶

The health educator's toolbox for knowledge synthesis includes a variety of methods, ranging from traditional health professions education systematic reviews to newer methodologies such as realist and scoping reviews. ⁷ Although scoping reviews have become an increasingly popular method, concerns have been raised about the rigor, merit, and appropriate application of this approach. ⁸ Herein, we describe scoping review methodology, the study questions to which this method is optimally suited, a rigorous approach for conducting them, and common pitfalls to avoid.

WHY AND WHEN TO PERFORM THIS METHODOLOGY

Scoping studies are particularly well suited to complex topics, where the literature base is broad and not yet comprehensively reviewed. The intent is to rapidly map key concepts corresponding to a particular research domain, including the primary sources and types of evidence currently available. ⁹ Arksey and O’Malley ¹⁰ provide four goals with which scoping review methodology appropriately aligns: to investigate the extent, range, and nature of research activity; to determine the value of performing more in‐depth or focused systematic reviews; to summarize and disseminate research findings; and to identify existing gaps in the literature.

The inclusive, flexible, and iterative nature of scoping reviews distinguishes them from other forms of knowledge synthesis. ¹¹ In contrast to traditional health professions education systematic reviews, scoping reviews do not adhere to strict methodological rules nor necessitate assessment of quality of evidence. ¹⁰ Whereas systematic reviews typically involve a well‐defined question and preidentification of inclusion criteria, scoping studies tend to examine broader topics; include a variety of study designs; and allow for evolution in the population (P), intervention (I), comparator (C), and outcomes (O) under study. ¹⁰ In contrast to narrative or literature reviews, scoping reviews require authors to perform analytical reinterpretation of the literature. ¹² , ¹³ Accordingly, the final write‐up of the scoping review is also flexible; its structure, content, and length can be adapted to the volume and type of literature reviewed. ¹¹ The scoping review also uniquely entails “charting” of the literature, whereby the authors generate a “map” reflective of the primary studies, corresponding to the review question(s). ¹⁰ , ¹² Because the map generally reflects researchers who may represent different disciplines examining the topic in question from different lenses, it is often multilayered. ¹¹ , ¹⁴ Finally, unlike other review methodologies, the scoping review process is iterative, allowing those conducting this type of review to deal with themes that are noted in the literature on a whole. This aspect of scoping reviews corresponds to a more constructivist approach and makes scoping reviews more aligned with other knowledge syntheses that seek to amalgamate large swaths of literature instead of deductively narrowing down a larger body of literature to a singular answer.

HOW TO PERFORM THIS METHODOLOGY

Scoping reviews must be conducted in a rigorous and transparent manner (i.e., the approach to searching for and synthesizing the evidence should be “systematic”). ¹⁵ They should be documented with sufficient detail to enable them to be replicated by others. While scoping reviews differ from other forms of systematic reviews because they do not have a rigid, preset protocol, some recommend that an a priori protocol should still be created and made publicly available. ¹⁶ A number of key steps must be followed to properly perform a scoping review. Table 1 provides an illustrative example. Table 2 highlights common pitfalls encountered with this technique.

TABLE 1.

Illustrative example of a scoping review

Steps	Workplace‐based Assessment Data in Emergency Medicine: A Scoping Review of the Literature 19
Step 1: Identify the research question	Developed the bounds of the review in collaboration with the research team. Study question: “What are the primary considerations when collecting, aggregating, and reporting WBA data for the diagnosis and support of trainees?”
Step 2: Identify the relevant studies	Searched six databases without language or date restrictions. Searched using set terms and published the search strategy. Utilized an experienced medical librarian.
Step 3: Select the studies to be included in the review	All study designs were considered for inclusion. Inclusion criteria: All studies highlighting procedures addressing the collection, aggregation, analysis, or report generation of WBAs for further downstream educational decision making. Two reviewers screened all abstracts with discrepancies resolved by consensus. Full texts were screened for inclusion by three authors.
Step 4: Chart the data	A data extraction tool was created by the research team, informed by prior research and refined through discussion. Outcomes from empirical studies were also classified using the Kirkpatrick framework. The tool was piloted and refined based on four studies.
Step 5: Collate, summarize, and report the results	Quantitative data were extracted. Qualitative thematic analyses were performed using an inductive method. The list of themes was iteratively expanded during the extraction process and all prior analyses were updated accordingly. A summary of the main themes was selected and discussed with the team for feedback. Validity evidence was determined for each study. Figures and tables were assembled to best summarize the charted data.
Step 6: Consult with key stakeholders	The themes and findings were reviewed with three education researchers with domain‐relevant expertise. Expert consultations were performed via one‐on‐one video conferencing.

Abbreviation: WBA, workplace‐based assessment.

TABLE 2.

Common pitfalls encountered in scoping reviews

Based on prior literature on scoping reviews, we have identified some common problems encountered by authors performing a scoping review.

Pitfall 1: Omitting the expert or stakeholder consultation phase

Many guidance papers for scoping reviews suggest that consulting stakeholders or experts can be of high yield for the last step of a scoping review. The rationale for this step is to ensure that your mapping resonates with those most knowledgeable about and impacted by the subject domain. Arksey and O’Malley 10 highlight that consulting experts in the domain (e.g., those who have published in this area) will help you to identify any missing literature within your analysis. Levac et al. 12 suggest that the usage of stakeholder consultation may provide similar help​ and should be a required component.

Pitfall 2: Failing to update your search

Depending on their size, scoping reviews can take a number of months to complete. Sometimes by the time you have completed your scoping review, new evidence has emerged in the field. It is best practice to quickly repeat your search at the end as you are mapping the literature for the time frame that has elapsed since your original search. Since you already have your inclusion/exclusion criteria and extraction forms streamlined by this stage, adding a few more papers to update analyses requires limited additional effort.

Pitfall 3: Poor visual representation of the final data

Pham et al. 18 found that less than one‐third of scoping reviews used graphics to represent their data. Consider moving beyond just tabular representations of your findings. Visual aids may help better explain concepts and trends than overly lengthy tables.

Pitfall 4: Not considering all your end‐users

While the primary intent of scoping reviews is often to map a field and identify gaps for scholars within a domain to advance research or innovation, medical education practitioners may also desire a concise summary of takeaways from the paper. If possible, consider suggesting policy or practice‐oriented recommendations. Consider including a variety of end‐users in the consultation stage to facilitate this.

Pitfall 5: Lack of a quality assessment

Pham et al. 18 found that only 22% of scoping reviews reported a quality assessment. Although this is controversial, it is important for authors leading scoping reviews to consider whether the final list of papers may be filtered by some sort of quality assessment tool to provide readers with insights about the quality of the literature. While not all scoping reviews will have the same requirement for determining quality of the literature within the field, it can be helpful to map the state of the literature in an area to determine what types of studies are needed next within a field. However, as Thomas et al. 8 point out, the heterogeneity of the types of scholarship that may occur in a health professions or medical education search may make it difficult to make firm assessments of quality for the various types of literature. Tools like the Medical Education Research Study Quality Instrument (MERSQI), the Newcastle‐Ottawa scale, or the visual approach to risk of bias assessment from the BEME guide may be used to complete quality assessments of health professions education scholarship. 20 , 21

Abbreviations: BEME, Best Evidence in Medical Education.

The first step comprises identifying the research question. ¹⁰ Researchers should formulate one overarching question in the area of interest. For scoping reviews, initial searches should be broad and inclusive. Investigators can narrow the inclusion criteria after they have a better sense of the data. Ultimately, the research team must clearly articulate the scope of their inquiry. ¹² The researchers should define terms, because these will be used to inform their search strategy. Minimally, the target population, overarching concept, and outcomes of interest should be articulated to clarify the focus of the review. ¹² Researchers should consider the purpose and goals of the review when articulating their research question, to ensure that the study has meaningful and relevant implications for educational policy, practice, or research. Prior to proceeding to the next step, reviewers must conduct a pilot search and iteratively refine their question and inclusion criteria, to ensure both the viability and the feasibility of the review. Scoping reviews aim to “map” the literature and are unlikely to add value if the number of primary papers is too small. If the number of primary papers is too large, the research team must consider their capacity (e.g., available time, budget, resources, and personnel) to successfully conduct the review. When limiting the scope, researchers need to provide a rationale and justification for their decisions as well as acknowledge the potential limitations with regard to scope and applicability. ¹² Those new to the field should be wary of this phase of the study since it can prevent wasting time on a question where a scoping review is simply not feasible (e.g., there is no literature to synthesize or the literature is simply too vast and unwieldy).

The second step involves identifying relevant studies. ¹⁰ In this stage, reviewers must determine their search strategy. The strategy should be exhaustive and rigorous. Engaging a medical librarian to assist with the search can improve the quality of the search. ¹² Once the search has been created, pilot the search strategy and ensure that it captures the key articles in the field of interest. The investigators should consider time span and whether the search will include all articles since database inception or only those within a certain time period. Any time limitations should have a clear rationale (e.g., Twitter emerged in 2006). In addition to common medical education research databases (e.g., PubMed, PsychINFO, CINAHL, ERIC, EMBASE), investigators should consider hand searching reference lists of relevant articles and reviews, key journals, conference abstracts, and online journals (e.g., MedEdPublish, MedEdPORTAL) as well as engaging with experts to assess for potential missed articles. Investigators should consider using a review reference manager (e.g., Covidence [Melbourne, Australia], DistillerSR [Ontario, Canada], Rayyan [Doha, Qatar]) to facilitate tracking and storage of articles.

The third step is study selection. ¹⁰ Two investigators should independently screen all abstracts and full texts, with disagreements resolved by discussion or involvement of a third person. Authors less familiar with the rigor required of systematic reviews often negate this step and charge through a structured review on their own. However, having at least two investigators screening is critical to minimize bias and error. The screening investigators should meet at the outset for calibration. Since coding behavior changes both between and within individuals over time, screeners should plan to reconvene several times to ensure consistency. Study selection is an iterative process that often involves post hoc modifications to the inclusion and exclusion criteria. Investigators should engage in this process in a reflexive manner, which may require repeating steps and components of the search as the reviewers gain familiarity with and understanding of the literature. During this stage, investigators should track the number of studies screened, assessed for eligibility, and included in the review as well as the reasons for exclusion at each stage. This should ideally be presented as a flow chart in the form of a modified PRISMA diagram. ¹⁷ Measures of inter‐rater reliability should be considered whenever feasible, but the evolving nature of the inclusion criteria in scoping reviews can make measurement of kappa statistics challenging beyond the initial title and abstract screening.

The fourth step involves charting the data. ¹⁰ During this step, investigators should utilize a “narrative” or “descriptive‐analytic” model when approaching the data collection. ¹⁰ Typically, at least two independent reviewers will be needed for this stage. Early on, reviewers should meet to determine whether their approach to data extraction is consistent with the research question and purpose. Reviewers should use a data extraction form to facilitate extraction and sorting. The data extraction will also likely be iterative, and researchers should continually update the data extraction form. When charting the data, researchers should focus on synthesizing and interpreting the data to identify themes. Once data extraction is complete, the researchers will need to present the data in a more narrative format to contextualize the findings within the study design and setting, so it is important to keep this in mind when selecting the data for extraction.

The fifth step consists of collating, summarizing, and reporting the results. ¹⁰ Data will then need to be charted and thematically organized. Charting of the data often consists of basic numerical analyses, such as grouping by geographic region, population, and time period. This can help to identify trends in research efforts and where there are significant gaps. Next, researchers should organize the information through thematic analysis. Arksey and O’Malley ¹⁰ recommend using the data table as a starting point, combined with researcher discussion using the “descriptive‐analytic” model to determine the final themes. This often shares similarities with qualitative content analytic techniques. Reviewers should utilize a clear and consistent reporting structure to reduce biases and better allow others to replicate the findings. While Arksey and O’Malley ¹⁰ espouse that scoping reviews should not include quality analyses, these can be important components of mapping and contextualizing the current literature. To understand the data and future directions for research, one must first understand the quality of said data; however, this point remains controversial. ⁷ , ¹⁴ In fact, Pham et al. ¹⁸ reported that only 22% of scoping reviews reported a quality assessment. When presenting the data, researchers should utilize tables and figures to demonstrate the main data while the text should serve to supplement and enhance, but not duplicate, the table. Researchers should also ensure that the final outcome or end‐product aligns with the purpose of the intended study. Finally, researchers must consider and present the findings in light of the broader context of research, policy, and practice.

The final step is consultation. Arksey and O’Malley listed consultation as a highly encouraged but optional sixth step while Levac et al. ¹⁰ , ¹² proposed that it should be a required component. We highly recommend this sixth step, because it often significantly enhances the quality of the work. This step may provide additional information, including references and resources. This may also provide valuable perspectives, meaning, and applicability to the study findings. Consultation is typically performed immediately after stage five and should involve multiple stakeholders, including experts in the field, as well as users and recipients of the interventions. Researchers should describe how they will collect the data (e.g., interviews, focus groups, surveys) and how the data will be analyzed, reported, and integrated into the overall study outcome. Investigators may also want to utilize this stage to identify dissemination strategies and also to get a sense from target audience members about the implications of the findings for the field at large. The experts consulted can often provide thoughtful insights into the relevance and broader implications the findings and help investigators better engage with scholarly conversation around the topic.

MARKERS OF RIGOR

Various groups have worked to define the structure and content of methodologically sound scoping reviews. Arksey and O’Malley ¹⁰ initially defined a six‐step approach in 2007. In 2010, Levac and colleagues ¹² expanded this to provide additional details and rigor. In 2018, Tricco et al. ¹⁷ created the Preferred Reporting Items for Systematic reviews and Meta‐Analyses extension for Scoping Reviews (PRISMA‐ScR) to provide clear reporting guidelines. In Table 3, we provide a summary of the major frameworks for scoping reviews.

TABLE 3.

Summary of the major frameworks for scoping reviews

Steps	Arksey and O’Malley 10	Levac et al. 12	PRISMA‐ScR 17
Step 1: Identify the research question	Consider which facets of the review question are particularly important. Maintain a broader approach early on.	Clearly articulate the research question. Consider the concept, target population, and health outcomes of interest when determining the research question. Consider the purpose and rationale/goal for the scoping study when developing the research question.	Describe the rationale for the review in the context of what is already known. Explain why a scoping review is the appropriate approach. Provide an explicit statement of the questions and objectives being addressed.
Step 2: Identify the relevant studies	Create a plan for the search, including databases, search terms, time span, and language. Consider time, budget, and personnel.	The research question and purpose should guide decisions regarding the scope of the study. Assemble a team with sufficient content and methodological expertise. Justify decisions and acknowledge limitations regarding the scope of the study.	Specify the characteristics of the sources of evidence used as eligibility criteria and provide a rationale. Describe all information sources (e.g., databases, dates, additional sources) and the date of the most recent search. Present the full search strategy.
Step 3: Select the studies to be included in the review	Determine inclusion and exclusion criteria. These may be iteratively derived or developed post hoc as familiarity with the literature increases.	This should be an iterative process. The team should include at least two reviewers to independently screen and select articles with disagreements resolved by a third reviewer if needed. Hold regular team meetings at the beginning, midpoint, and final stages.	State the process for selecting sources of evidence (i.e., screening and eligibility).
Step 4: Chart the data	Create and utilize a data extraction tool. Use a “narrative review” or “descriptive‐analytic” method.	Collectively develop the data extraction form. Charting should be an iterative process and the form should be continuously updated. Two authors should independently extract data from the first five to 10 studies and ensure it is consistent with the research question and purpose. Process‐oriented data may require extra planning for analysis. A qualitative content analysis approach is suggested.	Describe the methods for charting the data and processes for confirming data from investigators. List and define all data variables. Describe the methods and rationale for a critical appraisal of the data sources (if performed).
Step 5: Collate, summarize, and report the results	Present numerical analyses of the data. Perform thematic analyses. Utilize a clear and consistent reporting structure.	It is recommended to split this stage into three distinct steps: analyzing the data, reporting results, and applying meaning to the results. Analysis: includes descriptive, numerical summary, and qualitative thematic analysis. Reporting: present the results and produce the outcome that aligns with the overall study purpose or research question. Applying meaning: consider the findings as they relate to the research question, as well as future research, practice, and policy.	Describe the methods for handling and summarizing the data that were charted. Give the number of sources screened, assessed for eligibility, and included, as well as the reasons for exclusion at each stage. Present characteristics for the data and provide the citations. Present the critical appraisal (if performed). For each included source of data, summarize and present the relevant data that were charted and relate them to the review question and objectives. Summarize the main results, link to the review questions, and consider the relevance to key groups.
Step 6: Consult with key stakeholders	Engage multiple stakeholders, including experts and end‐users. This step can provide valuable insights and additional references. This is optional but encouraged.	Consultation should be an essential component. Establish a clear purpose for the consultation. Use preliminary findings to inform the consult. Develop a clear plan to select which stakeholders to consult and how the data will be collected, analyzed, reported, and integrated. Incorporate opportunities for knowledge dissemination.	Not mentioned.

Abbreviation: PRISMA‐ScR, Preferred Reporting Items for Systematic Review and Meta‐Analysis–Scoping Review Extension.

CONCLUSION

Scoping reviews can be a powerful tool to map the current literature for the purposes of determining gaps and problems within a new field or area. Once completed, a scoping review may provide new insights into existing gaps in the literature and lead to further research, innovation, and scholarship. Those new to medical education may find scoping reviews to be a useful methodology to apply when venturing into a new scholarly conversation within a particular field of study.

Gottlieb M, Haas MRC, Daniel M, Chan TM. The scoping review: A flexible, inclusive, and iterative approach to knowledge synthesis. AEM Education and Training. 2021;5:e10609. 10.1002/aet2.10609