Workplace‐based Assessment Data in Emergency Medicine: A Scoping Review of the Literature

Abstract

Objective

In the era of competency‐based medical education (CBME), the collection of more and more trainee data is being mandated by accrediting bodies such as the Accreditation Council for Graduate Medical Education and the Royal College of Physicians and Surgeons of Canada. However, few efforts have been made to synthesize the literature around the current issues surrounding workplace‐based assessment (WBA) data. This scoping review seeks to synthesize the landscape of literature on the topic of data collection and utilization for trainees’ WBAs in emergency medicine (EM).

Methods

The authors conducted a scoping review in the style of Arksey and O’Malley, seeking to synthesize and map literature on collecting, aggregating, and reporting WBA data. The authors extracted, mapped, and synthesized literature that describes, supports, and substantiates effective data collection and utilization in the context of the CBME movement within EM.

Results

Our literature search retrieved 189 potentially relevant references (after removing duplicates) that were screened to 29 abstracts and papers relevant to collecting, aggregating, and reporting WBAs. Our analysis shows that there is an increasing temporal trend toward contributions in these topics, with the majority of the papers (16/29) being published in the past 3 years alone.

Conclusion

There is increasing interest in the areas around data collection and utilization in the age of CBME. The field, however, is only beginning to emerge, leaving more work that can and should be done in this area.

McGaghie and colleagues first proposed competency‐based medical education (CBME) in their World Health Organization report in 1978, ¹ where they detailed the need to transition from a subject‐based curriculum or even an integrated curriculum to CBME in order to truly produce physicians that “can practice medicine at a defined level of proficiency, in accord with local conditions, to meet local needs.” ¹ The first step in transitioning to CBME (and realizing its full potential) is to define professional competence and its components. ¹ In the late 1990s, medical education took this first step in the form of the Canadian Medical Education Directives for Specialists (CanMEDS) roles ² , ³ and the Accreditation Council for Graduate Medical Education (ACGME) core competencies. ⁴ , ⁵

As the 21st century progressed, this transition to CBME became more concrete as specialty‐specific competencies were created. Emergency medicine (EM) was one of the first specialties to implement this form of CBME with the ACGME’s introduction of 23 EM competencies, each with five levels of milestones designed to be attained as training progresses. ⁵ , ⁶ , ⁷ More recently, the Royal College of Physicians and Surgeons of Canada EM specialty committee developed a suite of 28 entrustable professional activities to encompass the CanMEDS milestones as part of their competence‐by‐design stage‐based program of development and assessment. ⁸

One barrier to CBME is that, to accurately assess the learner’s progression through training, the associated programmatic assessments for each learner require voluminous amounts of information from multiple sources. The clinical workspace is a rich learning environment with significant potential to provide these data. ⁹ , ¹⁰ Missing the opportunities within the clinical workspace to capture trainee assessments would result in a narrower picture of each trainee’s abilities. ⁹ , ¹⁰ Programs of assessment potentially allow residencies or fellowships to strengthen the rigor and accuracy of trainee assessment by allowing for more perspectives and voices within the data, ¹¹ for larger population‐level analyses of achievement to unveil systemic biases, ¹² , ¹³ and for program evaluation to identify gaps in training programs. ¹⁴ Some organizations have found that specific programs of assessment substantially increase the frequency of assessment for individual trainees, resulting in even more data, ¹⁵ , ¹⁶ , ¹⁷ with some programs reporting the collection of thousands of data points from workplace environments. ¹⁸ As this volume of data is generated, there exists an increased responsibility to determine its value and use the data appropriately to guide both decision making and educational support of trainees. ¹⁸ , ¹⁹ To do so requires delineation of best practices and isolation of effective methods of high‐quality data aggregation that can function with large volumes of data.

The goal of this scoping review is to synthesize the literature from EM and provide key considerations for those interested in engaging in scholarship within the domains of collecting, aggregating, and reporting workplace‐based assessment (WBA) data and to guide its future use, supporting educational decision making for trainees, particularly in the context of CBME.

METHODS

We conducted a scoping review of the literature (systematic search of literature, followed by a thematic mapping of the findings) per the protocol of Arksey and O’Malley. ²⁰

Step 1: The Research Question

A core group of investigators (TC, MG, CS) met to determine the bounds of this review. Via discussion about key issues within the literature around CBME, this group determined that with regard to WBAs, the evidentiary chain is from workplace to reporting. After determining three key components (i.e., direct observation, data collection/aggregation/reporting for decision‐making and supporting trainees, data utilization by trainees), we focused on one key area: What are the primary considerations when collecting, aggregating, reporting WBA data for the diagnosis and support of trainees?

Step 2: Relevant Studies

With the assistance of a medical librarian (JW), we conducted a search of six databases (PubMed, Scopus, ERIC, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, and Google Scholar). The search was conducted on January 10, 2020. No date limit was imposed. After removing duplicates (n = 9), we were left with 189 unique titles and abstracts to review. The search terms are in Data Supplement S1, Appendix S1 (available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1002/aet2.10544/full).

Step 3: Literature Selection

Two reviewers (TC, SSS) reviewed the abstracts using Covidence (covidence.org), and a kappa statistic was calculated for this first screening for inclusion versus exclusion. All discrepancies were resolved via a consensus‐building process between the two reviewers. Full texts of the remaining papers (n = 63) were then acquired, and three reviewers (TC, WC, SSS) screened the full texts for determination of final inclusion or exclusion, based on the screening criteria. Our inclusion criteria were dictated by our initial research question. Works that highlighted procedures addressing the collection, aggregation, analysis, or report generation of WBAs for further downstream educational decision‐making (e.g., for competency committee utilization, summative decisions, and supporting trainees throughout training) were included. To facilitate study selection, we restricted our selection processes to journal articles and conference proceedings (including conference papers and white papers) that were indexed within our selected databases.

Step 4: Charting the Data

Based on our research questions and informed by previous scoping review studies completed by our lead author, ²¹ , ²² , ²³ , ²⁴ , ²⁵ we designed a data extraction tool incorporating the Cook, Bordage, and Schmidt framework for different types of scholarships (description, justification, and clarification). ²⁶ We also attempted to classify the outcomes measured in empirical studies according to Kirkpatrick's evaluation framework. ²⁷ , ²⁸ , ²⁹ Authorship team members (TC, WC, SSS) reviewed the Google Form (Mountainview, CA) extraction tool and provided feedback in an iterative fashion, refining this tool among the team based on a selected sample of four papers. The final extraction tool can be found in Data Supplement S1, Appendix S2.

Step 5: Collating, Reporting, and Summarizing the Data

Our final list of papers for full‐text review and extraction were divided among the entire authorship team. Authors were responsible for a final review of each paper regarding its suitability for our study (applying our inclusion and exclusion criteria once more). For the studies where the extractor was unsure about inclusion, a second author was asked to review the paper to render a final decision. Non–English‐language articles were excluded at this phase.

After meeting inclusion criteria, remaining papers underwent the extraction process with the aforementioned data extraction tool. During the extraction process, we used an inductive method to identify key themes related to the techniques for improving data collection, aggregation, analysis, and representation/visualization for informing decisions about trainee progress. Initially, the list of categories was restricted to common themes that our authorship team thought would be prevalent within the literature, but this list of themes was iteratively expanded throughout the data extraction process.

All summary statistic calculations were completed with either Microsoft Excel (Microsoft Corp., Redmond, WA) or SPSS version 26 (IBM Corp., Armonk, NY).

Step 6: Consultation

Upon completing our data extraction, the themes and findings from our study were reviewed with three education researchers with domain‐relevant expertise. We used this optional step within the Arksey and O’Malley process because our lead author (TC) had reflected that she had been highly embroiled in this area and wished to have external experts participating to check the comprehensiveness of our review processes. We used this consultation process as a method to engage in a form of reflexivity check. These researchers were required to have a graduate degree in education or psychometrics and measurement and at least three peer‐reviewed publications within the area of data collection, aggregation, analysis, or representation and preferably published in more than one domain. Consulting researchers were known to the research team, and many had served as coauthors with at least one member of the authorship team. For the expert consultations, one‐on‐one conversations via video conference were initiated with the lead author (TC).

Based on the suggestions from our consultations, we conducted a final mapping exercise. Using two assessment validity frameworks (Kane and Messick), we determined whether the papers mapped, or could be logically categorized to, elements of these frameworks. ³⁰ , ³¹ , ³² These mappings were thought to help new scholars better identify areas where they could engage in applying these validity frameworks.

RESULTS

The search retrieved 198 potentially relevant references, which were reduced to 63 papers after removing duplicates and screening based on title/abstract alone. These 63 studies were then reviewed in full by three authors each to evaluate whether they met our inclusion and exclusion criteria. We excluded a total of 27 more papers at this phase. Seven more papers were excluded by the team after full‐text review by the rest of the authorship team (please see the flow diagram [Figure 1] for details).

Figure 1.

Flow diagram of our literature screening and selection process.

About the Texts

The date of the papers ranged from 2008 to 2019. The majority of papers were published in the past 3 years (55%, 16/29). Figure 2 displays the trend over time.

Figure 2.

The trend of publications found using this scoping review on the topic of workplace‐based assessment in emergency medicine from 2008‐2019.

Four of the texts that we reviewed were conference‐related content (three abstracts, one conference paper), with one abstract serving as an early report of the subsequent full conference paper. ³³ Fifteen papers were descriptive papers, where authors described their design processes. There was one justification paper ³⁴ and two clarification papers. ¹² , ¹⁴ There were only six conceptual pieces (21% of all our papers) and three descriptions of innovations without evaluation data (10%, 3/29) within our findings, suggesting that the majority (66%, 19/29) of this area of EM education literature is empirically driven.

For empirically driven research (n = 19), the majority of papers were quantitative (53%, 10/19), while the remaining were qualitative work (31%, 6/19) or mixed methods (11%, 3/19). The quantitative approaches used in the studies included mainly descriptive techniques (n = 7); however, many also used inferential statistical techniques, including psychometrics (n = 3).

The majority of identified texts focused on EM only (72%, 21/29), but a large minority of our reviewed content featured other specialties (41%, 12/29). These other specialties included internal medicine, family medicine, surgery, pediatrics, psychiatry, neurology, obstetrics and gynecology, radiation oncology, and nephrology.

Due to the specialty‐oriented search, the vast majority of the papers in our final analysis focused on residents (89%, 25/29). Four articles (14%) featured multiple groups, including other groups such as medical students, fellows, practicing physicians, and members of other health care professions.

The majority of papers were conducted in or originated from North America (89%, 26/29). Of the remaining conceptual papers, one was from an international grouping of authors, one was from the International Federation of Emergency Medicine, and the other by the International Competency‐Based Medical Education collaborators.

The Forms of Evidence

Overall, studies fell into four major categories, as depicted in Table 1. Eighteen of the studies were descriptive in nature. ¹⁴ , ⁴⁹ These papers usually described a phenomenon focusing on the first step in the scientific process (e.g., observation). ²⁶ While these studies described the intervention and outcomes, they typically did not provide a comparison. Six papers were conceptual pieces (e.g., narrative review, opinion, editorial, or other theory‐advancing commentary). ⁵⁰ , ⁵¹ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ Three papers were classified as justification studies. ³⁴ , ⁴¹ , ⁵⁶ Justification studies focus on the last step in the scientific method by comparing one educational intervention with another to address whether the intervention was effective. The remaining two papers were clarification studies. ¹² , ¹⁴ These papers typically employ each step in the scientific method, starting with observations (often building on prior research), models, or theories and then using this to make and test predictions.

Table 1.

Types of Scholarship

DESCRIPTION STUDY (usually of a phenomenon): focuses on the first step in the scientific method, namely, observation. For example, intervention descriptions outcome data reported, usually no comparisons.	18
JUSTIFICATION STUDY: focuses on the last step in the scientific method by comparing one educational intervention with another to address the question(often implied): "Does the new intervention work?"	3
CLARIFICATION STUDY: employ each step in the scientific method, starting with observations (typically building on prior research) and models or theories, making predictions, and testing these predictions.	2
CONCEPTUAL WORK (i.e., narrative review, opinion, editorial, or other theory advancing commentary).	6

The reviewed publications consisted of six different methodologies. Nine studies were quantitative, ¹² , ¹⁴ , ³³ , ³⁶ , ³⁸ , ³⁹ , ⁴¹ , ⁴³ , ⁴⁸ , ⁵⁶ six were qualitative, ³⁵ , ³⁷ , ⁴² , ⁴⁵ , ⁴⁷ , ⁴⁹ and two included mixed methodologies. ¹⁷ , ³⁴ Many of the remaining publications were conceptual pieces, ⁵⁰ , ⁵¹ , ⁵² , ⁵³ , ⁵⁴ , ⁵⁵ innovation reports without results, ⁸ , ⁴⁰ or consensus conference proceedings. ⁴⁴

Validity Evidence

Validity frameworks are organizing concepts for guiding the construction of assessment tools. Data Supplement S1, Appendix S3, describes both Kane’s and Messick’s validity frameworks and provides a complete mapping of the 23 empirical papers within these two frameworks. Of the empirical papers, the majority of papers (n = 18/23) addressed at least one element of Kane’s validity framework, and nearly all addressed at least one element of Messick’s validity framework (n = 21/23). On the other hand, only two papers and one conference abstract addressed all the elements of Kane’s framework within a single paper. ³³ , ⁴³ , ⁴⁸ Meanwhile, no single paper within our study addressed all of the elements of Messick’s validity framework.

Themes From the Literature Reviewed

All of these papers could be filed under one or more of the following four categories: 1) collecting workplace‐based assessment data; 2) aggregating workplace‐based data; 3) assessment of learning: data utilization (e.g., interpretation, visualization, analysis) for promotion, remediation, diagnosis of trainee problems; or 4) assessment for learning (e.g., visualization, analysis) for supporting trainee growth and development. Approximately half of reviewed literature (52%, 15/29) focused on one of the four key themes, with the other half addressing multiple themes (48%, 14/29). Four papers discussed all four of our themes. ¹² , ⁴⁶ , ⁵² , ⁵³ There was one additional theme that was found in our literature review that focused on the functioning of competency committees, suggesting that this is an emerging area of scholarly interest. ³³ , ³⁷ , ⁴³ , ⁵¹

The majority of the papers in our study addressed the first theme of collecting workplace‐based assessment data (62%, 18/29). The second theme (processes that assist in aggregating workplace‐based data) was covered by 45% (n = 13), and the third theme (processes that assist in data utilization for assessment of learning), 48% (14/29). Our final theme of examining data utilization for assessment for learning was only found in 31% of the papers (9/29).

Theme 1: Collecting WBA Data

Papers that addressed collection of WBA data were characterized by quantitative, qualitative, or mixed‐methods approaches as well as editorials and commentaries. Studies highlighted the need to move beyond number counting and toward assessment of specific components of competency. ⁴⁴ , ⁵⁰ They emphasized the importance of purposefully selecting psychometrically sound assessments matched to competency domains of interest and ensuring that selected tools are “fit for purpose.” ⁸ , ¹⁷ , ⁴¹ , ⁴⁶ , ⁵¹ , ⁵² Authors reported not only on the value of collecting reliable quantitative data but also the utility of gathering rich qualitative data to add greater meaning to learner assessment and to facilitate data interpretation. ¹⁷ , ⁵²

Three studies warned against rater‐ or learner‐driven sampling bias in data collection. ¹² , ⁴³ , ⁴⁴ They encouraged the purposeful collection of WBA data using multiple methods, by multiple assessors, and across multiple contexts to mitigate the risks of biased sampling and to address limitations of individual assessment tools and individual rater shortcomings, such as halo or leniency effects. ¹⁸ , ⁴³ , ⁴⁶ Best practice models of programmatic assessment were described as a systematic way of organizing assessments; with regard to WBAs, this included the intentional collection of data from multiple sources over time. ⁴⁶ Two papers provided detailed descriptions of the design and implementation of their program of assessment, ¹⁷ , ⁴⁶ highlighting the benefits related to normalizing the collection of daily WBA and facilitating downstream data aggregation and interpretation.

A major challenge to collecting WBA data cited among several studies related to the feasibility of assessment methods and processes. ⁴³ , ⁵¹ , ⁵² , ⁵³ , ⁵⁵ , ⁵⁶ Caretta‐Weyer and Gisondi ⁵⁵ emphasized the need to simplify frontline assessment to be more intuitive for assessors to complete. They suggest that this could be done by designing collection forms that use task‐based components and that incorporate language that aligns with how supervisors make everyday judgments of learners. Other studies focused on the use of digital technology as a powerful tool to enhance data collection by facilitating real‐time feedback and documentation of learner performance. ¹⁷ , ³⁴ , ³⁷ , ⁵² Assessor training was also recognized as important and necessary for supporting faculty‐driven collection of learner assessment. ³⁷ , ⁵² The paper by Lockyer et al. ⁵² provides a review of the literature addressing collection of WBA data in the context of CBME, highlighting best‐practices, and challenges, while offering suggestions to optimize the process.

Theme 2: Aggregating WBA Data

Aggregating WBA data describes how data collected by trainees could be consolidated and viewed holistically at the individual, program, or institutional level. Studies in this area were characterized by both qualitative and quantitative approaches to better understand how various assessments of residents in EM fit within a broader system of assessment or align with EM‐specific milestones or are consistent across different EM settings. The most commonly used qualitative approach was constructivist grounded theory to highlight the experiences and social interactions between trainees and their assessors. ³⁵ , ⁴² , ⁴⁵ Regression approaches with more specifically multilevel modeling was one quantitative approach used to model trainee progression over time. ⁴³

A few studies offered frameworks and program descriptions that help guide effective aggregation of WBA from multiple sources. For example, Chan and Sherbino ¹⁷ describe the McMaster Modular Assessment Program (McMAP), a WBA system designed specifically for EM trainees. The assessments were EM aligned and blueprinted back to the national CanMEDS framework (i.e., a framework that describes the abilities physicians need to possess and demonstrate to be deemed effective in meeting patients’ needs).

Theme 3: Data (e.g., Visualization, Analysis) for Assessment of Learning

In addition to previous studies that focused on aggregating WBA data for a variety of purposes, we observed that studies in this area were often conducted to determine how aggregated data could be leveraged to track progression. The most frequent utilization was to identify issues (e.g., how trainees perceive their role) or determine which trainees were struggling. A theme around how clinical competence committees (CCCs) take these data to make decisions was also identified. Since these are a relatively new phenomenon, we did find that there was an intersection between the CCCs and the emerging literature around assessment of learning.

Acai and colleagues ³⁷ interviewed 16 EM attendings about their experiences with the McMAP WBA system and noted logistic concerns as to how to accumulate sufficient digitalized data in a transparent, nongameable fashion that overcomes a bias against reporting negative feedback. While this qualitative study may need to be triangulated with actual analyses to determine if this perception aligns with the reality of data acquisition, it is still important to bear in mind how the absence of data may have ramifications for deciding trainee progression. This paper nicely maps to another paper within this program of research around the sources of missing data. ¹⁴

Beeson et al. ⁴¹ report on the even larger, cross‐program milestones data collected by the ACGME where, over a 68‐day window, they collected milestone data on 5,805 residents and found that it demonstrated a consistent factor structure. However, the factor loading (i.e., the clustering of milestones together in WBA data) described within this paper’s findings suggest that there are marked limits to our ability to use the milestones for decision‐making purposes.

Approaches to data visualization and analysis included automating case logs, developing structured educational programs, creating trainee report cards using electronic health record (EHR) data, and applying machine learning algorithms to flag struggling trainees. ³³ , ⁴⁸ , ⁴⁹ Using the data aggregated in Chan and Sherbino’s McMAP system, Ariaeinejad and colleagues ³³ , ⁴⁸ conducted machine learning and regression analyses in an attempt to predict the future performance of trainees. They successfully predicted whether an individual trainee was “at risk” or “not at risk” based on the large data set. Machine learning algorithms could be used to enhance human decision making for deciding whether a resident might not be progressing in their learning or achievement.

In our validity framework analysis, we found that there was a trend toward increased use of both Kane’s and Messick’s validity frameworks in the literature. To depict this, we have chronologically mapped the literature for our review and used a heat mapping function to display the validity elements that we felt each paper displayed or addressed in Figure 3.

Figure 3.

A heat map that depicts the validity framework incorporation within the EM assessment literature over time.

Theme 4: Data Utilization (e.g., Visualization, Analysis) for Assessment for Learning

A number of studies have laid the groundwork for data‐driven interventions in real time. We perceived a shift from retrospective data analyses to prospective ones: from annual or semiannual reports that summarize a completed time period to the development of dashboards that update as the data is accumulated. Such systems are enabled by the increasingly digitalized environments in which clinical care is carried out. Shepherd et al. ⁴⁹ interviewed EM trainees and faculty to determine which performance indicators could be pulled from the EHR and meaningfully used to represent both independent and interdependent clinical actions by the trainees.

When thinking of trainee support, we must also consider the role of program evaluation and systems‐level supports that are examined in the literature. What should not be overlooked is the benefit to the program of the data collected on behalf of individuals. Dorfsman et al. ⁵⁷ observed 32 EM residents for 4 to 5 hours each and reaped a rich trove of information that pointed the way to program curriculum enhancements, in addition to the individualized feedback. Dayal et al. ¹² used EM program evaluation data to show that sex bias may exist in resident evaluations. Using real‐time algorithms, their sex‐informed program evaluation process could also be the basis for real‐time reporting and intervention, especially as the differences became more manifest as the residents advanced in the program.

DISCUSSION

As CBME becomes more prevalent within medical education, ⁵ , ⁸ , ⁵⁸ it is important to develop strategies for effective collection, aggregation, and dissemination of WBA data. This scoping review identified 29 EM publications specifically addressing this topic and identified several key patterns to inform practice and future research. This review reports these early studies within this area in hopes of providing new education scholars with insights on the scholarly conversations within our specialty.

When collecting data, multiple studies emphasized the need for using both quantitative and qualitative components to avoid the limitations of purely numeric‐based assessments. ¹⁷ , ⁴⁴ , ⁵⁰ , ⁵² Obtaining substantive comments on assessment forms can be challenging, ⁵⁹ , ⁶⁰ and future research should assess strategies for increasing qualitative comments. ⁶¹ , ⁶² , ⁶³ Recent studies have also described tools for evaluating the quality of workplace‐based assessment comments (e.g., Completed Clinical Evaluation Report Rating tool, Quality of Assessment for Learning score). ⁵⁹ , ⁶⁴ One study found that targeted feedback improved qualitative feedback quality. ⁶⁴ Future research should expand on this and determine strategies for increasing both the quantity and the quality of WBA data provided. Moreover, given the complexity and time‐pressured nature of the emergency department environment, efforts should also be made to facilitate WBAs by streamlining forms and utilizing technology to facilitate data entry and capture. ³³ , ⁵¹ , ⁵² , ⁵³ , ⁵⁵ , ⁵⁶

Data should be aggregated and presented in a manner to augment decision making or trainee support in a meaningful format. In this capacity, it would be valuable to utilize keyword technology and machine learning algorithms to identify trends to facilitate earlier interventions among “at‐risk” learners. ³³ , ⁶⁵ More recently (and outside of our present study period), a conversation has begun in the literature that describes the nature of dashboards that marshal these types of real‐time data flows. ¹⁹ , ⁶⁶ Exciting new areas of research that have more recently shown up in the literature (both within and outside of EM) include the incorporation of EHR data for trending or tracking workplace outcomes. ⁴⁹ , ⁶⁷ , ⁶⁸ , ⁶⁹ , ⁷⁰ However, enthusiasm for the possibilities of digitized data should be tempered by the ongoing need for assessor judgment of important but nondigitizable aspects of resident performance, such as discernment, conscientiousness, and truthfulness. ⁴⁵ Moreover, we must know what these data are actually capturing, since they are not always easy to interpret due to issues around interdependence of teammates or trainee–teacher dyads. ⁴⁹ , ⁷⁰ , ⁷¹ While digital systems can collect text data, meaning needs to be extracted in a way that ensures proper weighting of these important attributes. The theme around decision‐making processes and group work (e.g., clinical competence committees) also intersects with this body of literature related to data aggregation/visualization—and there is certainly much work to be done examining how the decision makers are affected by the data’s representation. Only just recently has a study describing the use of a design‐based model to develop a dashboard for clinical competency committees been published. ⁶⁶ Clearly, our review shows that there is still ample room for continued innovation and scholarship in this area.

In this new era of increased WBA data, we must begin to think about how best to incorporate them into our existing workflows, but also be deliberate about how to do this with care and thoughtfulness. Some innovations will flow easily (e.g., real‐time feedback for more robust coaching), while other changes may require more planning and care to initiate (e.g., whether to attribute successes or failures of certain interventions to trainees, their attendings, or the team at large). Ellaway et al., ⁷² , ⁷³ for example, have frequently written about these issues, and those looking to enter into this area of research should consider their thoughts and heed their warnings.

Finally, we found the validity framework analysis suggested by our consultations quite fascinating. This analysis allowed us to quickly see how much rigor has been added to the literature in recent years. We recommend that new scholars entering into the scholarly conversation around WBA data within EM may find our validity framework analyses useful to gain guidance as to the types of validity evidence they may want to highlight in their own work. Future research should also evaluate strategies for maximizing purposeful utilization of data by harnessing technology to better identify trends as well as presenting data in a more user‐friendly format. Other gaps that exist in the current literature within our field identified by our consultants in step 6 of our scoping review process included:

1. Data access and ethics around trainee data stewardship (e.g., Do our trainees know how the data is being stored? Are they aware of who owns the data about them? Who has access? Do trainees have a say in how this data can be used?)

2. Data utilization for purposes beyond single‐trainee decisions (e.g., How can we use this same data for program evaluation, faculty development, and quality improvement?).

LIMITATIONS

The authors would like to first remind readers that this review is the beginning of a scholarly conversation and not the end. While we have identified some works in the area of WBA aggregation and analysis within our field, there is certainly room for more work and clarification in this area. One of the other key limitations of our study is that much of the literature that currently exists concerning WBA data collection, aggregation, and visualization exists outside of EM. Many articles about these themes were found within our search but were excluded because they did not involve specific mention or provide guidance in EM. It is possible that some of the general literature may contain evidence that could guide EM faculty on these issues, but we felt that a concerted effort to focus on EM literature would help to shed light on opportunities for scholars in our specialty to explore and pursue. There are also some semantic nuances (e.g., feedback vs. assessment) that exist within the literature, and we anticipate that some key papers may have been relevant but used different terminology in their papers (and hence would have been excluded by our scoping review search and subsequent processes). ¹³ , ⁷⁴

Moreover, the lead author of this paper (TC) would note that she has been involved in many of the papers cited within this paper, but the team has helped her engage in checking her reflexivity throughout the scoping review process at every step—including dual coding within paper selection and inviting in external assessment experts to ensure that no literature was missing. We value her expertise but have tried our best to ensure that her perspective and vantage point did not dominate this review and that we conducted a comprehensive initial search.

We are also aware that this field is evolving, with increasing numbers of papers in more recent years. Based on this trend, we suspect that a future review on these topics will yield more high‐quality evidence over time. That said, we are certain that more work would be welcome in this area, especially as we complete our transition to a CBME era.

CONCLUSIONS

This scoping review identified a trend toward more literature about workplace‐based assessment data collection, aggregation, and visualization within EM education, with the majority of the papers being published in the past 3 years. There is a burgeoning body of EM literature within this area of assessment, and those seeking to carve out a scholarship niche in this area can look to some of the authors in our review for guidance and mentorship. Ultimately, our scoping review suggests that more work must be done to provide greater evidence for how best to collect, process, and visualize the wealth of data from both quantitative and qualitative sources in the workplace.

Supporting information

Data Supplement S1. Supplemental material.

AEM Education and Training. 2021;5:1–13