Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

Abstract

Objectives

Direct observation is important for assessing the competency of medical learners. Multiple tools have been described in other fields, although the degree of emergency medicine–specific literature is unclear. This review sought to summarize the current literature on direct observation tools in the emergency department (ED) setting.

Methods

We searched PubMed, Scopus, CINAHL, the Cochrane Central Register of Clinical Trials, the Cochrane Database of Systematic Reviews, ERIC, PsycINFO, and Google Scholar from 2012 to 2020 for publications on direct observation tools in the ED setting. Data were dual extracted into a predefined worksheet, and quality analysis was performed using the Medical Education Research Study Quality Instrument.

Results

We identified 38 publications, comprising 2,977 learners. Fifteen different tools were described. The most commonly assessed tools included the Milestones (nine studies), Observed Structured Clinical Exercises (seven studies), the McMaster Modular Assessment Program (six studies), Queen’s Simulation Assessment Test (five studies), and the mini‐Clinical Evaluation Exercise (four studies). Most of the studies were performed in a single institution, and there were limited validity or reliability assessments reported.

Conclusions

The number of publications on direct observation tools for the ED setting has markedly increased. However, there remains a need for stronger internal and external validity data.

Direct observation involves observation of the learner in the clinical or simulated setting, generating information which can then be utilized both to provide real‐time formative feedback and to generate data for global assessments of the learner. ¹ Direct observation is a commonly used method for assessment of medical trainees and is especially important in today’s age of competency‐based medical education (CBME). ² , ³ , ⁴ It can provide essential information regarding a trainee’s knowledge, behavior, and skills related to a particular context or environment. In addition to providing information for both formative and summative feedback, direct observation can also aid in deliberate practice, which is essential for developing expertise. ⁵ Direct observation of a trainee’s skills is an essential component of a workplace‐based assessment program and thereby plays a key role in both education and advancement decisions. ⁶ , ⁷ This became particularly important since the Accreditation Council for Graduate Medical Education (ACGME) created the Next Accreditation System (also known as the Milestones) in 2012. ⁸

At that time, while there were a variety of tools available for direct observation of clinical skills, very few had been evaluated in the emergency department (ED) environment. ⁷ The unique practice environment of the ED poses additional challenges for emergency medicine (EM) educators and program leadership. ⁹ , ¹⁰ Several of these challenges, including the feasibility of conducting direct observations amid patient care and supervision of acutely ill individuals, were discussed as part of a breakout session on assessment of observable learner performance in EM during the 2012 Academic Emergency Medicine (AEM) Consensus Conference on Education Research. ¹ The resulting article from this breakout session identified several strategies for assessing learner performance, including both direct observation strategies and indirect approaches (e.g., resident portfolios, procedure logs, self‐reflection). ¹ The authors then highlighted the strengths, weakness, relative costs, and available outcome data of direct observations and suggested a research agenda to address gaps in knowledge (Figure 1). ¹ The field of education research within EM has advanced substantially since that time. ¹¹ However, it is unclear to what degree this call to action has been answered by EM. Therefore, it is important to understand the current evidence to inform future research efforts and best practices. The objective of this article is to perform a systematic review of the literature on direct observation tools in EM published since 2012.

Figure 1.

2012 AEM Consensus Conference on Education Research Agenda on Clinical Skills Assessment Tools ¹ .

METHODS

Our study follows the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) guidelines for systematic reviews and was performed in accordance with best practice guidelines (Data Supplement S1, Appendix S1, available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1002/aet2.10519/full). ¹² In conjunction with a medical librarian, we conducted a search of PubMed, Scopus, the Cumulative Index to Nursing and Allied Health Literature (CINAHL), the Cochrane Central Register of Clinical Trials, the Cochrane Database of Systematic Reviews, Education Resources Information Center (ERIC), PsycINFO, and Google Scholar to include citations from January 1, 2012, to January 27, 2020. Details of the search strategy are included in Data Supplement S1, Appendix S2. After completing our initial search, we then performed a targeted search of each identified direct observation tool combined with “emergency medicine” in PubMed to identify any potentially missed articles. We specifically focused on articles published since 2012 because this was the date of the AEM Consensus Conference, which included a focus on direct observation tools. ¹ As such, we sought to identify new literature on direct observational tools within EM since that time period. We also reviewed the bibliographies of all included studies and review articles for potentially missed studies. Finally, we consulted with topic experts to help identify any further relevant studies. No funding was provided for this review.

Inclusion and Exclusion Criteria

This review sought to summarize the existing direct observational tools used for the evaluation of medical students and residents in the ED setting. Inclusion criteria included all articles directly describing direct observational tools, which could include descriptive studies, retrospective studies, prospective studies, and randomized controlled trials. Studies could be performed in the clinical ED setting or a simulated environment. We excluded narrative reviews, studies focused on other specialties, studies focusing exclusively on procedural skills, or studies where the authors did not address direct observation tools. We intentionally excluded studies focusing exclusively on procedural skills because these tools have a different focus than clinical skills assessment tools. We also prospectively planned to exclude studies not published in English or Spanish if there was no translated version available, although none were identified in our literature search.

Two investigators (MG, JJ) independently assessed studies for eligibility based upon the above criteria. All abstracts meeting the initial criteria were reviewed as full‐text articles. Studies deemed to meet the eligibility criteria on full‐text review were included in the final data analysis. Any discrepancies were resolved by in depth discussion and negotiated consensus.

Data Collection and Processing

Two investigators (JNS, RC) underwent initial training and extracted data into a predesigned data collection form. The following information was abstracted: first author name, year of publication, study title, number of participants, study country, study location (e.g., clinical, simulation laboratory), study design (e.g., qualitative, retrospective, prospective, randomized controlled trial), learner population (e.g., medical student, resident, year of training), medical specialty of the learners, tool utilized, assessor training, assessor calibration, outcome(s) measured, and the main study findings. Given the significant clinical heterogeneity of the studies, a meta‐analysis was not planned.

Quality Analysis

Two investigators (MG, JJ) underwent initial training and independently performed quality analysis using the Medical Education Research Study Quality Instrument (MERSQI). ¹³ The MERSQI is a 10‐item tool (18 maximum points), which was specifically designed for evaluating medical education research and has been demonstrated to have good inter‐rater reliability. ¹³ The investigators compared responses and resolved any discrepancies by in depth discussion and negotiated consensus. Traditionally, the MERSQI tool has a maximum score of 18 points. However, because some of the components of this tool will only apply to quantitative studies, it may artificially underscore qualitative studies. To address this, we adjusted the maximum possible points for qualitative studies to 10 (Data Supplement S1, Appendix S3).

RESULTS

Summary of Findings

The search identified 728 articles. After duplicates were removed, 634 studies were screened using titles and abstracts with 71 selected for full‐text review (Figure 2). Of these, 38 studies (n = 2,977 learners) met inclusion criteria and are discussed further below (Table 1).

Figure 2.

Flow diagram for article selection.

Table 1.

Study Characteristics

Study	Number of Participants	Country	Study Location	Study Design	Learner Population	Tool Used
Acai 2019 34	16	Canada	Clinical	Qualitative	PGY 1–5	McMAP
Ander 2012 14	289	USA	Clinical	Prospective	MS 4	RIME and Global Rating Scale
Bedy 2019 15	120	USA	Clinical	Descriptive	PGY 1–3	Milestones
Bord 2015 16	80	USA	Simulation	Prospective	MS 2–4	OSCE, Milestones
Brazil 2012 47	20	Australia	Clinical	Prospective	PGY 1	Mini‐CEX
Bullard 2018 17	30	USA	Simulation	Prospective	PGY 1	OSCE
Chan 2015 36	15	Canada	Clinical	Descriptive	PGY 1, 2	McMAP
Chan 2017 35	23	Canada	Clinical	Retrospective	PGY 2	McMAP
Chang 2017 50	273	Taiwan	Clinical	Retrospective	PGY 1	Mini‐CEX
Cheung 2019 37	45	Canada	Clinical	Prospective	PGY 1–5	O‐EDShOT
Dagnone 2016 38	98	Canada	Simulation	Prospective	PGY 1–5	QSAT
Dayal 2017 18	359	USA	Clinical	Retrospective	PGY 1–3	Milestones
Dehon 2015 19	33	USA	Clinical	Retrospective	PGY 1–4	Milestones
Donato 2015 20	73	USA	Clinical	Retrospective	PGY 1–3	Minicard
Edgerley 2018 39	57	Canada	Simulation	Retrospective	PGY 1–5	QSAT
FitzGerald 2012 21	34	USA	Clinical	Qualitative	PGY 1	Checklists
Hall 2015 40	92	Canada	Simulation	Prospective	PGY 1–5	QSAT
Hall 2017 41	79	Canada	Simulation	Prospective	PGY 1–5	QSAT
Hart 2018 22	118	USA	Simulation	Prospective	PGY 1–3	Checklist, Global Rating Scale, Milestones
Hauff 2014 23	28	USA	Clinical and Simulation	Prospective	PGY 1	OSCE, Checklist, Global Rating Scale, Milestones
Hoonpongsimanont 2018 24	45	USA	Clinical	Prospective	MS 4	Local EOS evaluation
Hurley 2015 42	57	Canada	Simulation	Prospective	PGY 3–5 and attending physicians	OSCE
Jones 2016 48	24	Australia	Clinical	Prospective	PGY 1–4	Local EOS evaluation
Jong 2018 25	34	USA	Simulation	Prospective	PGY 2–4	QSAT
Kane 2017 26	26	USA	Clinical	Prospective	PGY 3–4	SDOT
Lee 2019 49	73	Australia, NZ	Clinical	Qualitative	ND	Mini‐CEX
Lefebvre 2018 27	41	USA	Clinical	Prospective	PGY 1–3	Milestones
Li 2017 43	26	Canada	Clinical	Qualitative	PGY 1–5	McMAP
Lin 2012 51	230	Taiwan	Clinical	Prospective	PGY 1	Mini‐CEX
McConnell 2016 44	9	Canada	Clinical	Retrospective	PGY 2	McMAP
Min 2016 28	10	USA	Clinical	Prospective	PGY 1–5	Global Breaking Bad News Assessment Scale
Mueller 2017 29	71	USA	Clinical	Qualitative	PGY 3–4	Milestones
Paul 2018 30	39	USA	Clinical	Prospective	PGY 1	OSCE, Checklist
Schott 2015 31	29	USA	Clinical	Prospective	PGY 1–4	CDOT, Milestones
Sebok‐Syer 2017 45	23	Canada	Clinical	Retrospective	PGY 1–2	McMAP
Siegelman 2018 32	102	USA	Simulation	Retrospective	PGY 1–3	OSCE
Wallenstein 2015 33	239	USA	Simulation	Prospective	MS 4	OSCE
Weersink 2019 46	17	Canada	Simulation	Prospective	PGY 1–5	RAT

CDOT = Critical Care Direct Observation Tool; EOS = end‐of‐shift; ITER = McMAP = McMaster Modular Assessment Program; Mini‐CEX = Mini‐Clinical Evaluation Exercise for Trainees; MS = medical student; ND = not described; NZ = New Zealand; O‐EDShOT = Ottawa ED Shift Observation Tool; PGY = Post‐graduate year; RAT = Resuscitation Assessment Tool; RIME = Reporter, Interpreter, Manager, Educator; SDOT = Standardized Direct Observation Tool.

Twenty studies were conducted in the United States, ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² , ³³ 13 were performed in Canada, ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ three took place in Australia, ⁴⁷ , ⁴⁸ , ⁴⁹ and two were conducted in Taiwan. ⁵⁰ , ⁵¹ Thirty‐three studies involved resident physicians, ¹⁵ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³² , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁶ , ⁴⁷ , ⁴⁹ , ⁵⁰ four studies involved medical students, ¹⁴ , ¹⁶ , ²⁴ , ³³ and one did not report the learner level. ⁴⁸ Twenty‐two were prospective studies, ¹⁴ , ¹⁶ , ¹⁷ , ²² , ²³ , ²⁴ , ²⁵ , ²⁶ , ²⁷ , ²⁸ , ³⁰ , ³¹ , ³³ , ³⁷ , ³⁸ , ⁴⁰ , ⁴¹ , ⁴² , ⁴⁶ , ⁴⁷ , ⁴⁸ , ⁵¹ nine were retrospective, ¹⁸ , ¹⁹ , ²⁰ , ³² , ³⁵ , ³⁹ , ⁴⁴ , ⁴⁵ , ⁵⁰ five were qualitative studies, ²¹ , ²⁹ , ³⁴ , ⁴³ , ⁴⁹ and two were descriptive studies. ¹⁵ , ³⁶ Twenty‐five studies were performed in the clinical environment, ¹⁴ , ¹⁵ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²⁴ , ²⁶ , ²⁷ , ²⁸ , ²⁹ , ³⁰ , ³¹ , ³⁴ , ³⁵ , ³⁶ , ³⁷ , ⁴³ , ⁴⁴ , ⁴⁵ , ⁴⁷ , ⁴⁸ , ⁴⁹ , ⁵⁰ , ⁵¹ 12 in the simulation lab, ¹⁶ , ¹⁷ , ²² , ²⁵ , ³² , ³³ , ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ , ⁴² , ⁴⁶ and one used both simulation and the clinical environment. ²³

Assessor training was described in 15 studies and ranged from a 10‐minute training video to a dedicated 3‐hour training session (Data Supplement S1, Appendix S4). Assessor calibration was described in nine studies and primarily consisted of either an initial calibration session or feedback based on subsequent scoring (Data Supplement S1, Appendix S4). Among studies assessing diagnostic accuracy, the most common criterion standard was training level (n = 9), ¹⁸ , ¹⁹ , ²⁰ , ²² , ³⁰ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ followed by independent scoring from experts (n = 3), ²⁵ , ²⁶ , ⁴² clinical competency committee scores (n = 2), ¹⁵ , ¹⁹ overall ED rotation score (n = 2), ¹⁴ , ³³ in‐training assessment report (n = 2), ⁴¹ , ⁴⁷ and mean entrustment score from an alternate workplace‐based assessment tool (n = 1). ⁴⁶

Study Quality

The mean MERSQI score among the included studies was 13.1 of 18 (range = 8–15.5) for quantitative studies and 8.2 of 10 (range = 7–10) for qualitative studies (Table 2). The most common areas that studies lost points were study design, institutional sampling, and validity evidence for internal structure of the tools.

Table 2.

Quality Assessment

Study	Study Design	Sampling: Institutions	Sampling: Response Rate	Type of Data	Validity Evidence: Content	Validity Evidence: Internal Structure	Validity Evidence: Relationship to Other Variables	Data Analysis: Sophistication	Data Analysis: Appropriate	Outcome	Total
Acai 2019 34 , *	1	1.5	1	1	0	0	0	1	1	1	7.5
Ander 2012 14	1	0.5	0.5	3	1	0	1	2	1	2	12
Bedy 2019 15	1	0.5	1.5	3	1	0	1	1	1	2	12
Bord 2015 16	1	0.5	1.5	3	1	1	1	2	1	1.5	13.5
Brazil 2012 47	1	0.5	1.5	3	1	1	1	2	1	2	14
Bullard 2018 17	1.5	0.5	1.5	3	1	1	1	2	1	1.5	14
Chan 2015 36	1.5	0.5	0.5	3	1	0	1	2	1	2	12.5
Chan 2017 35	1.5	0.5	1.5	3	1	0	1	2	1	2	13.5
Chang 2017 50	1	0.5	0.5	3	1	1	1	2	1	2	13
Cheung 2019 37	1	0.5	1.5	3	1	1	1	2	1	2	14
Dagnone 2016 38	1	1.5	1.5	3	1	1	1	2	1	1.5	14.5
Dayal 2017 18	1	1.5	1.5	3	1	0	1	2	1	2	14
Dehon 2015 19	1	0.5	1.5	3	1	0	1	2	1	2	13
Donato 2015 20	1	0.5	1.5	3	1	1	1	2	1	2	14
Edgerley 2018 39	2	0.5	1.5	3	1	1	1	2	1	1.5	14.5
FitzGerald 2012 21 , *	1	0.5	1.5	3	1	0	0	1	1	1	10
Hall 2015	1	0.5	1.5	3	1	1	1	2	1	1.5	13.5
Hall 2017	1	1.5	1.5	3	1	1	1	2	1	1.5	14.5
Hart 2018 22	1	1.5	1.5	3	1	1	1	2	1	1.5	14.5
Hauff 2014 23	1	0.5	1.5	3	1	1	1	1	1	2	13
Hoonpongsimanont 2018 24	1.5	0.5	1.5	1	1	0	1	2	1	2	11.5
Hurley 2015 42	1	0.5	1.5	3	1	1	1	2	1	1.5	13.5
Jones 2016 48	1.5	0.5	1	1	1	0	0	1	1	1	8
Jong 2018 25	1	0.5	1.5	3	1	1	1	2	1	1.5	13.5
Kane 2017 26	1.5	0.5	1	3	1	0	0	2	1	1.5	11.5
Lee 2019 49 , *	1	0.5	0.5	1	1	0	0	1	1	1	7
Lefebvre 2018 27	1	0.5	1.5	3	1	0	0	2	1	2	12
Li 2017 43 , *	1	0.5	1	1	1	0	0	1	1	1	7.5
Lin 2012 51	1	1.5	1.5	3	1	0	0	1	1	2	12
McConnell 2016 44	1	0.5	1.5	3	1	0	1	2	1	2	13
Min 2016 28	1	0.5	1.5	3	1	0	1	2	1	2	13
Mueller 2017 29 , *	1	0.5	1.5	1	1	0	0	1	1	2	9
Paul 2018 30	2	1	1.5	3	1	1	1	2	1	2	15.5
Schott 2015 31	1	1.5	1.5	3	1	1	1	2	1	1.5	14.5
Sebok‐Syer 2017 45	1	0.5	1.5	3	1	0	1	2	1	2	13
Siegelman 2018 32	1	0.5	1.5	3	1	0	0	2	1	1.5	11.5
Wallenstein 2015 33	1	0.5	1.5	3	1	0	1	2	1	1.5	12.5
Weersink 2019 46	1	0.5	1	3	1	1	1	2	1	2	13.5

^*Qualitative studies where the modified MERSQI tool was used.

Direct Observation Tools

Studies evaluated 15 different direct observation tools, with the majority of the literature focusing on the following five tools. Ten studies utilized the ACGME EM Milestones, ¹⁵ , ¹⁶ , ¹⁸ , ¹⁹ , ²² , ²³ , ²⁷ , ²⁸ , ²⁹ , ³¹ seven used Observed Structured Clinical Exercises (OSCE), ¹⁶ , ¹⁷ , ²³ , ³⁰ , ³² , ³³ , ⁴² six studies utilized the McMaster Modular Assessment Program (McMAP), ³⁴ , ³⁵ , ³⁶ , ⁴³ , ⁴⁴ , ⁴⁵ five used the Queen’s Simulation Assessment Test (QSAT), ²⁵ , ³⁸ , ³⁹ , ⁴⁰ , ⁴¹ and four utilized the mini‐Clinical Evaluation Exercise (mini‐CEX). ⁴⁷ , ⁴⁹ , ⁵⁰ , ⁵¹ Additional tools included checklists, ²² , ²³ , ³⁰ a global rating scale, ¹⁴ , ²² , ²³ the Minicard, ²⁰ a non–milestone‐based end‐of‐shift evaluation, ²⁴ , ⁴⁷ the Ottawa Emergency Department Shift Observation Tool, ³⁷ the Reporter/Interpreter/Manager/Educator (RIME) framework, ¹⁴ the Standardized Direct Observation Tool (SDOT), ²⁶ the Critical Care Direct Observation Tool (CDOT), ³¹ and the Resuscitation Assessment Tool (RAT). ⁴⁶ A summary of the data for each tool is provided in Table 3.

Table 3.

Summary of the Data for Each Tool

Direct Observation Tool	Total Studies(Total Participants)	AssessmentSetting	Accuracy and Reliability	Benefits	Limitations	Resource for Example Tool
CDOT	1 (29)	Clinical	Poor inter‐rater reliability (ICC = −0.04 to 0.25) 31	Focused on critical care interventions. Mapped to milestones. Includes a qualitative comments box.	Limited to yes, no, or N/A responses. Poor inter‐rater reliability.	Schott 2015 31
Checklists	4 (219)	Clinical and Simulation	Statistically significant increase for each training level (0.52 levels per year; p < 0.001). 22 Good inter‐rater reliability (ICC = 0.81 to 0.86). 22	Checklists are targeted to each clinical presentation. May include an area for qualitative feedback. If mapped to milestones, can also be used to evaluate milestones for ACGME.	Each checklist needs to be individually designed for each chief complaint. Primarily focused on specific presentations or aspects of care. Responseoptions often limited to yes, no, or unclear. Qualitative comments vary by checklist.	FitzGerald 2012 21 Hart 2018 22 Paul 2018 30
Global Breaking Bad News Assessment Scale	1 (10)	Clinical	Resident skill increased by 90% on subsequent encounter. 28	Short and easy to complete. Study tool can be modified to include a qualitative comments box. 28	Only assesses delivery of bad news. Responses limited to yes or no.	Schildmann 2012 78
Global Rating Scale	3 (435)	Clinical and Simulation	Statistically significant increase for each training level (p < 0.05). 22 Good inter‐rater reliability for clinical management (ICC = 0.74 to 0.87) and communication (ICC = 0.80). 22	Fewer questions. Faster to perform. Can be combined with other direct observation tools.	Relies heavily on gestalt. Less granular assessment of components. No qualitative comments.	Ander 2012 14 Hart 2018 22
Local EOS Evaluation	2 (69)	Clinical	N/A	Can includes assessment of technical skills and some non‐technical skills (e.g., professionalism, interpersonal skills)	Categorizations are general with limited specific examples. Not all tools have qualitative comments.	Hoonpongsimanont 2018 24 Jones 2016 48
McMAP	6 (112)	Clinical	Data on accuracy not available. 12.7% variance between raters. 35	Learner‐centered. Individual clinical assessments were mapped to the ACGME and CanMEDS Frameworks. Tool uses behaviorally anchored scales and includes mandatory written comments.	May have a higher learning curve associated with the 76 unique assessments within the tool. Some components may not be possible to observe depending upon the patients encountered. Learners may avoid cumbersome tasks or those that they are weaker in. Faculty may avoid certain components that are harder to evaluate.	Acai 2019 34 Chan 2015 36 Chan 2017 35
Milestones	9 (911)	Clinical and Simulation	Statistically significant increase for each training level (0.52 levels per year; p < 0.001). 18 However, faculty may overestimate skills with milestones (92% milestone achievement regardless of training level). 19 Mean CCC score differed significantly from milestone scores (p < 0.001). 19 Poor inter‐reliability in one study (ICC = −0.04 to 0.019). 31	Addresses a diverse array of technical and nontechnical skills. Already utilized for summative residency assessments that are collected by the ACGME	Many of the milestones may not be applicable for a given patient or shift. Has a risk of grade inflation. 19 No qualitative comments.	ACGME Milestones 52
Mini‐CEX	4 (596)	Clinical	Did not identify any underperformers that were not already identified by the Australian Resident Medical Officer Assessment Form. 47	Includes assessment of technical skills and some nontechnical skills (e.g., professionalism, efficiency). Overall high satisfaction among both learners and assessors. 47 Includes a dedicated area for qualitative feedback (strengths and weaknesses).	Does not assess teaching, teamwork, or documentation. Focused on single patient encounters so unable to account for managing multiple patients. Some components may be skipped unless they are required for completion. 50	Brazil 2012 47
Minicard	1 (73)	Clinical	Minicard scores increased by 0.021 points per month of training (p < 0.001). 20	Includes comments for each individual assessment item. Includes an action plan at the end.	Inclusion of trainee level in descriptors for scoring may bias results.	Donato 2015 20
O‐EDShOT	1 (45)	Clinical	Statistically significant increase for each training level (p < 0.001). 37 38% variance noted in ratings between raters. 37 13 forms needed for 0.70 reliability. 37 33 forms needed for 0.80 reliability. 37	Designed specifically for the ED setting with feedback from faculty and residents. Includes an area for qualitative feedback (strengths and weaknesses). Can be used regardless of treatment area (i.e., high, medium, low acuity).	Only evaluated in a single study.	Cheung 2019 37
OSCE	7 (575)	Clinical and simulation	OSCE was positively correlated with ED performance score (p < 0.001). 33 Comparing 20‐item OSCE with 40‐item OSCE revealed no difference in accuracy (85.6% vs. 84.5%). 42 Variation in ICC between studies (ICC = 0.43 to 0.92). 17 , 42	Can assess a wide range of factors, including technical and nontechnical skills. Bullard modeled their tool after the ABEM oral board categories. 17	OSCEs may vary between sites. OSCEs typically need to be individually designed for each presentation.	Bullard 2018 17 Paul 2018 30 Wallenstein 2015 33
QSAT	5 (360)	Simulation	Statistically significant increase between PGY 1/2 and PGY 3–5 (p < 0.001). 38 , 40 Mean score increased by 10% for each training year (p < 0.01). 39 QSAT total score was moderately correlated with in‐training evaluation report score (r = 0.341; p < 0.01). 41 Moderate inter‐rater reliability (ICC = 0.56 to 0.89). 25 , 38 , 40	Provides a framework that can be customized to each specific case.	Each QSAT would need to be individually designed for each presentation. Studies limited to the simulation environment.	Hall 2015 40 Hall 2017 41 Jong 2018 25
RAT	1 (17)	Simulation	RAT was positively correlated with entrustment scores (r = 0.630; p> 0.01). 46 Moderate inter‐rater reliability (ICC = 0.585 to 0.653). 46	Builds upon QSAT with entrustable professional activities targeted towards resuscitation management. Designed using a modified Delphi study with experts.	Only assesses resuscitation management. Limited data from a single study.	Weersink 2019 46
RIME	1 (289)	Clinical	Positive correlation between RIME category and clinical evaluation score (r2 = 0.40, p < 0.01). 14 Very weak correlation between RIME category and clinical examination score. 14	Easy to use. Can be combined with other tools.	Only one study evaluated RIME in the ED. Limited assessment of professional competencies (e.g., work ethic, teamwork, humanistic qualities).	Ander 2012 14
SDOT	1 (26)	Clinical	Attending physicians were 54.4% accurate and resident physicians were 49.6% accurate when compared with the criterion standard scoring. 26	Includes assessment of technical skills and some nontechnical skills (e.g., professionalism, interpersonal skills).	Several components may not be applicable to some patient encounters. Does not include an option for qualitative comments. Lower accuracy compared with other tools. May be more time consuming than other direct observation tools.	Kane 2017 26

ACGME = Accreditation Council for Graduate Medical Education; CCC = clinical competency committee; CDOT = Critical Care Direct Observation Tool; EOS = end of shift; ICC = intraclass correlation; McMAP = McMaster Modular Assessment Program; Mini‐CEX = Mini‐Clinical Evaluation Exercise for Trainees; N/A = not available; O‐EDShOT = Ottawa ED Shift Observation Tool; RAT = Resuscitation Assessment Tool; RIME = Reporter, Interpreter, Manager, Educator; SDOT = Standardized Direct Observation Tool.

Milestone‐based Evaluations

The tool most heavily represented in our sample was the ACGME EM Milestones. The Milestones are a framework for assessing resident progress, which were developed by each specialty to address the six core competencies created by the ACGME. ⁵² Multiple authors urged caution in using ACGME milestones to create end‐of‐shift or simulation assessment tools. ¹⁹ , ³¹ The CDOT, which allows for a milestone‐based assessment of a resident during the early part of a critical resuscitation, did not demonstrate good reliability, with significant variability between raters (intraclass correlation from −0.04 to 0.019). ³¹ Dehon et al. ¹⁹ showed poor agreement at one site between end‐of‐shift milestone scores and clinical competency committee ratings, as well as similar rates of attainment of level 3 milestones for all resident levels. Alternatively, Dayal et al. ¹⁸ found that milestone scores increased 0.52 levels per year. Lefebvre et al. ²⁷ found that including narrative comments along with milestone scores on end‐of‐shift tools increased the learner assessment scores assigned by the clinical competency committee.

OSCE

Several authors examined various tools used for OSCEs. OSCEs are highly structured tools used to assess competency, with an emphasis on objective assessment measures. ⁵³ Bord et al. ¹⁶ developed an OSCE to evaluate attainment of Level 1 milestones in clerkship students, which was able to discriminate between high‐ and low‐performing students. Wallenstein and Ander ³³ compared OSCE scores with overall EM clerkship scores and found that they were positively correlated (p < 0.001). Hauff et al. ²³ described an OSCE developed as part of postgraduate orientation which showed that many of their incoming interns had not attained Level 1 milestones. Bullard et al. ¹⁷ created an OSCE modeled after the American Board of Emergency Medicine oral board examination and found high inter‐rater reliability (intraclass correlation = 0.92). The authors also noted a retained educational benefit for both the participants and the observers of the OSCE at 3 months. ¹⁷ Hurley et al. ⁴² evaluated the effect of the OSCE length on interobserver reliability and accuracy and found that no significant difference was present between the 20‐item and 40‐item checklists.

McMAP

The McMAP is a competency‐based program of assessment that includes 76 micro clinical assessments systematically mapped to key clinical tasks within EM. ³⁴ , ³⁵ Each specific task includes a checklist to orient the rater, a global assessment using behavioral anchors, and mandatory written comments. ³⁴ Each clinical task is linked to a global end‐of‐shift rating. ³⁴ Both resident and faculty reflections on the implementation of this tool have been published. ³⁴ , ⁴³ Key benefits described by faculty were the inclusion of a wide range of clinical tasks, learner‐driven emphasis, and the facilitation of more targeted specific and global feedback. ³⁴ However, the faculty also noted that there was a learning curve associated with the 76 unique assessment instruments. ³⁴ There was also concern among faculty that learners could “game the system” by selecting tasks that they were more facile with. ³⁴ Authors used the data from the McMAP implementation to comment on systematic gaps in data collection (such as the Health Advocate and Professional CanMEDS roles) ⁴⁴ and the effect of McMAP on end‐of‐year report quality for residents, ³⁶ analyze narrative comments compared with checklist scores, ⁴⁵ and describe longitudinal patterns arising from aggregate assessments. ³⁵ The McMAP tool increased resident perception of formative feedback delivery and provided a conduit for residents to seek real‐time feedback. ³⁶ , ⁴³

QSAT

The QSAT is a modification of the OSCE that incorporates a global learner assessment score and was shown to discriminate well between learner levels. ³⁸ , ⁴⁰ It has also demonstrated good inter‐rater reliability with one study reporting an intraclass correlation coefficient (ICC) of 0.89, ³⁸ while a different study reported individual ICCs ranging from 0.56 to 0.87. ⁴⁰ It has been used to provide complementary data to an in‐training evaluation report used in Canada, assessing different aspects of competence. ⁴¹ Scores have been demonstrated to increase 10% with each additional year of training. ³⁹ Interestingly, Edgerley et al. ³⁹ found that working a night shift within one day of a QSAT assessment did not significantly impact a learner’s score.

Mini‐CEX

The Mini‐CEX was originally utilized in internal medicine and has more recently been adapted to the ED setting. The Mini‐CEX is a direct observation tool which emphasizes the following domains: medical interviewing, physical examination, humanistic qualities/professionalism, clinical judgment, counseling, and organization/efficiency. ⁵⁴ Lin et al. ⁵¹ showed that most raters using this tool focused on clinical judgment, with decreased emphasis on humanistic components, while Lee et al. ⁴⁹ explored factors influencing rater judgments using the tool. Brazil et al. ⁴⁷ found that the Mini‐CEX increased formative feedback overall despite not addressing all of the performance domains. Chang et al. ⁵⁰ demonstrated that Mini‐CEX compliance was improved on a computer format, particularly among raters with less than 10 years of seniority.

Other Tools

A variety of other tools were evaluated in a more limited number of studies. Ander et al. ¹⁴ found that the RIME framework correlated well with the overall clinical evaluation score in an EM clerkship (r ² = 0.40, p < 0.01). Kane et al. ²⁶ evaluated a training session for the SDOT among attending and resident physicians, noting that even after training, the attending physicians selected the correct rating in only 54.4% of cases, while senior residents selected the correct rating in 49.6% of cases. One group described a set of four different checklists that were each targeted to evaluating specific history and physical examination skills based on chief complaints (e.g., asthma, fever in the neonate, pediatric fever, and gastroenteritis/dehydration). ²¹ Hart et al. ²² studied global rating scales for clinical management and communications, as well as a checklist mapped to the milestones, and reported a statistically significant increase with level of training (p < 0.001). They also noted an ICC of 0.74 to 0.87 for the global rating scale and an ICC of 0.81 to 0.86 for the checklist. ²² Paul and colleagues ³⁰ described an otoscopy skills‐focused tool. More recently, the Ottawa Emergency Department Shift Observation Tool was introduced as an entrustment‐based tool to evaluate a resident’s ability to manage the ED, with some validity evidence supporting its use, but further studies are needed. ³⁷ Jones and Nanda ⁴⁸ reported on a locally developed workplace‐based assessment tool which raters found useful. Weersink et al. ⁴⁶ modified the QSAT to develop the RAT, which demonstrated a positive correlation with resident entrustment scores (r = 0.630, p < 0.01) and good inter‐rater reliability (ICC = 0.59 to 0.65). Hoonpongsimanont et al. ²⁴ described a locally developed workplace‐based assessment tool used on learner performances recorded with GoogleGlass and compared this with learner self‐assessments of that same recording. Donato et al. ²⁰ evaluated a Minicard that was able to identify struggling learners and also encouraged formative feedback with action plans.

Bias in Assessment

Three studies evaluated the effect of sex on evaluation results. Dayal et al. ¹⁸ found that milestone‐based assessments used at the point of care or end of shift led to a 12.7% higher score for males compared to females regardless of assessor sex or assessor–assessee pairing. This corresponded to approximately 3 to 4 months of additional training in their study. ¹⁸ Mueller et al. ²⁹ found that female residents received less consistent feedback than their male counterparts. Feedback was particularly inconsistent regarding issues of autonomy and assertiveness. ²⁹ Siegelman et al. ³² evaluated bias in simulation‐based assessments but did not find a similar association between rater or trainee sex and score.

Multisource Feedback

Three studies specifically evaluated the use of nonphysician observers. ¹⁵ , ²⁵ , ²⁸ Bedy et al. ¹⁵ utilized ED‐based pharmacists to specifically evaluate EM resident performance of the Pharmacotherapy Milestone (PC5). They found that pharmacist input was valuable for the determination of milestone levels during the clinical competency committee meetings. ¹⁵ Jong et al. ²⁵ studied the QSAT among residents comparing physicians with nurses and emergency medical technicians (EMTs) as raters. In their study, nurses had moderate agreement with physicians (ICC = 0.65), while EMTs had excellent agreement with physicians (ICC = 0.812) for the QSAT scoring. ²⁵ Social worker evaluation of resident performance during the delivery of bad news was studied by Min et al. ²⁸ They found that this was acceptable to both residents and social workers, but that social workers tended to rate resident performance higher when compared with the resident’s self‐assessment. ²⁸

DISCUSSION

Since the 2012 AEM Consensus Conference, there has been a substantial increase in the number of publications related to direct observation tools in the ED setting. This is encouraging, as Kogan et al. ⁷ identified only six total EM‐based studies in their prior systematic review. We were able to identify 38 new studies since 2012 alone. This adds significantly to the available literature on this topic.

The most common tool utilized was the ACGME EM Milestones despite the intent that milestones would guide assessment practices instead of acting as the assessment. ⁵⁵ This is not surprising, because this is utilized by all EM residency programs as part of their assessment of residents and is required to be reported to the ACGME for reaccreditation. Therefore, it would seem reasonable to extend these summative assessments to direct observation tools. However, studies found relatively limited reliability of the measurements when used as direct observation tools. ¹⁹ , ³¹ These findings may reflect a problem with the tools used or with how clinicians are trained to use them. ⁵⁶ None of the studies included in our review discussed how the assessors were trained with regard to this assessment tool. Multiple studies have suggested that potential assessors need to be sufficiently trained (including targeted training sessions, initial calibration, and feedback on assessment scoring with regard to one’s peers) ⁵⁷ , ⁵⁸ , ⁵⁹ and that they need to see the value in improving their ability to assess their learners. ⁶⁰ , ⁶¹ , ⁶² , ⁶³

Some of the more successful tools for assessing EM learner skills include the McMAP, QSAT, OSCEs, O‐EDShOT, RAT, global rating scale, and checklists. While these tools demonstrated greater discriminatory ability, they were often limited to a small group of learners in a single program. While the Working Group on Assessment of Observable Learner Performance emphasized the need to refine previously well‐established tools (e.g., mini‐CEX, SDOT), ⁶⁴ nearly one‐third of studies described a novel tool. Unfortunately, there is limited validity evidence for most of these tools. When creating a new tool, it is important to demonstrate the validity of the measure, and future studies should seek to better establish the internal and external validity of these newer tools. This should include assessment of content, response process, internal structure, relationship to other variables, and consequences. ⁶⁴ , ⁶⁵ Moreover, studies should ensure that they follow and explicitly report adherence with recommended reporting guidelines. ⁶⁶

The overall quality of the data was lower with mean MERSQI scores of 13.1 of 18 for quantitative studies and 8.2 of 10 for qualitative studies. Many studies lost several points for insufficient validity evidence as described above. Additionally, the vast majority of studies were single‐group, cross‐sectional analyses. As the evidence advances, there will be a need for more cohort and randomized controlled trials comparing different direct observation methods. Finally, while most studies were performed at a single site, there is a need to assess these interventions across multiple institutions to better evaluate external validity.

Within our data, we found conflicting information regarding the effect of sex on direct observation tools. Sex biases and inequality have been demonstrated in EM among attending physicians, but the data among EM residents are more limited. ⁶⁷ , ⁶⁸ , ⁶⁹ , ⁷⁰ , ⁷¹ Dayal et al. ¹⁸ found that female residents had a lower overall rating on end‐of‐shift evaluations compared with male residents, while Mueller et al. ²⁹ found that female residents received less consistent feedback than male residents. Interestingly, Siegelman et al. ³² did not find a difference in scoring between male and female residents in their simulation‐based OSCE study. This may be due to the use a simulation environment, where there was a greater degree of control and external supervision, which may have led to a Hawthorne effect. Additionally, the OSCE is a binary tool (yes/no), which may be less prone to bias than the more subjective tools. Further studies should assess the role of bias in assessment and strategies to prevent this. Additionally, studies should also evaluate the impact of other nonmedical biases related to race, ethnicity, age, and primary language.

When compared with the 2012 AEM Consensus Conference on Education Research Direct Observation Tools Research Agenda (Figure 1), ¹ there remains a need for better data on the reliability and discriminatory ability of the assessment tools in the ED environment and among different assessors (Items 2 and 4). Three studies described the role of nonphysicians (e.g., pharmacists, social workers) using direct observation tools, ¹⁵ , ²⁵ , ²⁸ but none compared these assessments with a criterion standard. Additionally, there remains a need for more data on the number of direct observations, types of patient encounters, and global assessments necessary to determine competency (Items 1 and 3). The ED is a unique environment, wherein attending and resident physicians are in close proximity to each other for an extended period of time. However, time constraints may limit direct observation and variations in intervals between consecutive shifts may limit the ability to develop longitudinal assessments of progress. ⁷² , ⁷³ , ⁷⁴ , ⁷⁵ Future studies need to determine the ideal number and distribution of encounters necessary to reliably assess competence. We did not identify any studies comparing the validity of clinical metrics relative to other forms observation (Item 5), so there remains a need for further research into this area. We did not assess procedural skill acquisition (Item 6) in our review.

As the field moves increasingly toward CBME, there will be an increased need for validity and reliability evidence for direct observation tools in the ED setting. ⁷⁶ , ⁷⁷ Future research will need to build on these studies to better assess validity across institutions and among different providers as well as how best to integrate this into the clinical ED environment.

LIMITATIONS

There are several limitations with regard to this study. First, the studies had substantial heterogeneity with regard to tools and outcome measures, limiting the ability to perform meta‐analysis for any of the outcomes. Among the included studies, many utilized the tool as part of the overall assessment, which may have led to incorporation bias. Also, the criterion standard varied between studies with many using training level, which may not reflect the actual degree of clinical expertise. Only a limited number of studies adequately described assessor training and calibration. Future studies should ensure that these are fully described in the methods. Additionally, the search was limited to EM and did not include direct observation tools developed and/or evaluated in other specialties that may be applicable to the ED environment. Moreover, only three studies evaluated medical students, so it remains unclear how reliable most of these tools are in this learner group. The majority of studies were performed in North America. As such, it is unclear how this would apply in other locations. Our search excluded procedural assessment tools, so we were not able to comment on this in the article. Finally, while we searched eight databases with the assistance of a medical librarian, it is possible that we may have missed some relevant studies. However, we also performed bibliographic review of all included articles and reached out to topic experts, so we believe that the risk of this is low.

CONCLUSION

There is a burgeoning body of work within emergency medicine focusing on how we might optimize direct observation of our trainees. The majority of these articles assess the Milestones, McMAP, OSCE, QSAT, and Mini‐CEX, although validity evidence is limited. Future studies are needed to better assess the validity, reliability, and number of evaluations necessary to assess competence.

The authors would like to thank Jennifer C. Westrick, MSLIS for her assistance with the literature search.

Supporting information

Data Supplement S1. Supplementary materials.

AEM Education and Training 2021;5:1–17