An Exploration of Competency Assessment in Graduate Medical Education

Introduction

Assessment remains a major challenge to high-quality implementation of a more competency-based graduate medical education (GME) system.¹ Fortunately, the past decade has seen significant progress. In the United States, the Accreditation Council for Graduate Medical Education (ACGME) has developed the Milestones to help define the specific outcomes, or competencies, that medical trainees are expected to develop during their training.^2,3 However, training programs still lack standardized, evidence-based approaches to assess performance, use assessment data to make competency decisions, and implement assessment systems successfully. As a result, assessment processes vary substantially among residency programs,⁴ including within Clinical Competency Committees.⁵ This variation can waste educational resources, limit educational impact, compromise educational quality, and ultimately threaten ongoing efforts to make GME more competency-based.

To improve the quality of competency-based assessment, an important next step is to examine what assessment tools are used in GME and how data from those tools are used to determine trainee competency. The ACGME Common Program Requirements specify that resident evaluation occurs at specific intervals, including determination of Milestone ratings.⁶ The ACGME has guidelines for what assessment approaches can be used to inform competency decisions related to the Milestones,⁷ and other organizations provide lists of assessment tools for specific activities.^8,9 However, outside of end-of-rotation evaluations and in-training examinations, few assessments are mandated for medical trainees, and no specific assessment tool is required or recommended by the ACGME. Furthermore, many assessment tools are developed in single institutions and may not be feasible to scale outside the contexts in which they were developed. Even when an assessment tool is implemented more widely, it remains difficult to connect the results of an assessment tool to proficiency standards like the Milestones. Understanding how assessment data have been collected, aggregated, and used to inform decisions about trainee competence is key to understanding progress toward a more evidence-based medical education system.

This report characterizes what assessment tools have been used to determine clinical competency in GME and identifies assessment systems that have demonstrated feasibility in generating valid and fair scores. Using the Evidence Centered Design (ECD) framework,^10,11 we examined what assessment tools are being used in GME, what competencies they are being used to assess, and what evidence exists to support their use and implementation.

Our Approach

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) framework¹² to develop a search strategy for identifying all assessment tools used to inform competency decisions in GME. The protocol is available on Open Science Framework. PubMed, Scopus, ERIC, and Embase databases were searched in May 2024 to retrieve articles. Studies were included if they used an assessment tool to measure technical, procedural, operational, or clinical competency within any specialty in GME. Studies were excluded if they were conducted in health professions education outside of medicine, non-English texts, editorial/perspective pieces, or review articles.

A total of 2299 articles were examined during title and abstract screening. Two reviewers K.M.M. and A.P. reviewed each title and abstract, and 642 studies received a full-text review by 2 reviewers, with 454 studies selected. Conflicts regarding whether a study met criteria were resolved through discussion between the 2 reviewers to come to a consensus.

Data elements collected included study characteristics, competencies assessed, details of the assessment tool, scoring methods, implications, and validity evidence of each tool. These articles and the manually extracted data were then used to train and iteratively refine a large language model (LLM) in ChatGPT 4o. The LLM was used to extract specific data elements from the articles. For any data element that required a decision or interpretation, such as the ACGME competency domain that was assessed by the tool, the LLM was trained to provide justification for the extraction decision. This strategy was implemented to facilitate data quality assurance. All elements were manually checked by reviewers. All studies were assessed for quality using the Medical Education Research Study Quality Instrument (MERSQI).^13,14

Data were organized according to the ECD, a framework for building and implementing quality assessment systems.^10,11,15,16 Within ECD, the Conceptual Assessment Framework¹⁶ outlines specific components of an assessment argument. These components include the Competency Model, Task Model, and Evidence Model. The Competency Model specifies what knowledge, skills, or attributes should be assessed. For this report, competencies were categorized based on the 6 core competencies outlined in the ACGME Milestones. The Task Model describes the tasks being performed and elements of the task that elicit a desired construct, such as the setting and rater performing the assessment. The Evidence Model describes how a learner’s performance on an assessment is scored. Assessment tools were categorized into the following: simulation, workplace-based assessment (WBA),¹⁷ checklist, written examination, Milestones, biometric data, patient outcomes, and clinical competency committees. Additional extracted data included what type of decisions were made from the assessment data and study using the MERSQI framework (study response rate, data type, outcome type, complexity of analysis, and validity evidence). After extracting data, descriptive analyses were conducted using R (version 4.2.2024).

Evidence for Competency Assessment Tools in GME

A total of 454 studies were examined. Of these, 282 studies were conducted in the United States, 156 studies outside of the United States, and 16 across multiple countries. Study publication dates ranged from 1989 to 2024. Study design was categorized according to the MERSQI categorizations for educational studies (see Table 1). Most studies were cross-sectional, posttest, or pretest/posttest. The 5 most common specialties represented in these studies were general surgery, internal medicine, pediatrics, emergency medicine, and anesthesiology. Fifty-three studies were conducted in more than one specialty, and 147 studies occurred within one specialty at multiple institutions. The median number of participants assessed in the studies was 34, with a range of 1 to 21 499.

Table 1.

Study Characteristics of Included Studies

Variable	n
Country
United States	282
International	156
Multiple countries	16
No. of institutions within study
1	278
2	30
More than 2	146
Study design
Single-group cross-sectional or single-group posttest	240
Single-group pretest and posttest	179
Nonrandomized, 2 groups	8
Randomized controlled trial	27

The ACGME competency domain most assessed was Patient Care (Table 2), and 221 studies evaluated more than one competency in the study. Overall, the most common types of assessment tools were simulation assessment tools, workplace-based assessments, checklists, and written assessments (Table 3). In studies that evaluated a single ACGME competency, simulation was the most commonly used assessment tool. For studies that assessed multiple competencies, the Milestones were the most common tool used for aggregating assessment data. The primary rater, or person conducting the assessment, was faculty/attending in 424 studies, other residents and fellows in 17 studies, program directors in 29 studies, nurses and other health care providers in 9 studies, and other raters, such as parents or standardized patients, in 24 studies.

Table 2.

ACGME Competencies Examined in Articles With Most Commonly Associated Assessment Tool and Validity Evidence

ACGME Competency	N	Assessment Tools	N	Validity Evidence	n (%)
Patient Care	311	Simulation	153	Internal structure	121 (79)
				Content validity	128 (84)
				Relationship to other variables	109 (71)
		Workplace-based assessment	120	Internal structure	87 (73)
				Content validity	89 (74)
				Relationship to other variables	80 (67)
		Checklist	62	Internal structure	54 (87)
				Content validity	55 (89)
				Relationship to other variables	53 (85)
Medical Knowledge	130	Simulation	58	Internal structure	50 (86)
				Content validity	51 (88)
				Relationship to other variables	43 (74)
		Written assessment	44	Internal structure	31 (70)
				Content validity	33 (75)
				Relationship to other variables	28 (64)
		Checklist	32	Internal structure	28 (88)
				Content validity	29 (91)
				Relationship to other variables	28 (88)
Professionalism	55	Simulation	24	Internal structure	18 (75)
				Content validity	19 (79)
				Relationship to other variables	17 (71)
		Workplace-based assessment	19	Internal structure	17 (89)
				Content validity	18 (95)
				Relationship to other variables	16 (84)
		Checklist	12	Internal structure	9 (75)
				Content validity	10 (83)
				Relationship to other variables	8 (67)
Interpersonal and Communication Skills	73	Simulation	39	Internal structure	28 (72)
				Content validity	29 (74)
				Relationship to other variables	28 (72)
		Workplace-based assessment	23	Internal structure	17 (74)
				Content validity	19 (83)
				Relationship to other variables	16 (70)
		Checklist	18	Internal structure	12 (67)
				Content validity	14 (78)
				Relationship to other variables	11 (61)
Practice-Based Learning and Improvement	52	Simulation	21	Internal structure	19 (90)
				Content validity	19 (90)
				Relationship to other variables	15 (71)
		Workplace-based assessment	17	Internal structure	15 (88)
				Content validity	15 (88)
				Relationship to other variables	15 (88)
		Written assessment	11	Internal structure	8 (73)
				Content validity	9 (82)
				Relationship to other variables	5 (45)
Systems-Based Practice	36	Simulation	16	Internal structure	12 (75)
				Content validity	13 (81)
				Relationship to other variables	11 (69)
		Checklist	11	Internal structure	10 (91)
				Content validity	11 (100)
				Relationship to other variables	10 (91)
		Workplace-based assessment	10	Internal structure	9 (90)
				Content validity	9 (90)
				Relationship to other variables	7 (70)
All competencies assessed together	79	Milestones	23	Internal structure	13 (57)
				Content validity	17 (74)
				Relationship to other variables	13 (57)
		Workplace-based assessment	20	Internal structure	14 (70)
				Content validity	12 (60)
				Relationship to other variables	13 (65)
		Clinical Competency Committee	19	Internal structure	13 (68)
				Content validity	15 (79)
				Relationship to other variables	14 (74)

Abbreviation: ACGME, Accreditation Council for Graduate Medical Education.

Table 3.

The 5 Most Commonly Used Assessment Tool Types

Assessment Tool Type	Definition	Examples	N	MERSQI Score, Median (IQR)
Simulation	Assessment tools that aim to replicate or mirror real-life clinical situations, such as patients, anatomical regions, or clinical tasks.35,36	Objective structured clinical examination (OSCE)	171	12.50 (11.50-13.00)
Workplace-based assessment	Assessment tools that collect information about performance in an authentic working environment.37	Society for Improving Medical Professional Learning (SIMPL) assessment system Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) Entrustable professional activities (EPAs)	145	12.00 (11.00-13.00)
Checklist	Assessment tools that list the specific criteria for the skills, behaviors, or attitudes that participants should demonstrate to show successful learning from training.38	Family Meeting Behavioral Skills Checklist Instrument assessing internal medicine residents’ discharge summaries	70	12.00 (11.12-13.00)
Written examination	Assessment tools that test performance through written, computerized, or oral examinations.39,40	American Board of Surgery In-Training Examination (ABSITE) Internal Medicine In-Training Examination (IM-ITE) Certification Examinations	59	12.00 (11.00-13.00)
Milestones	Assessments of competency-based developmental outcomes (eg, knowledge, skills, attitudes, and performance) that can be demonstrated progressively by residents/fellows from the beginning of their education through graduation to the unsupervised practice of their specialties.2	ACGME Milestones	28	11.00 (9.75-11.75)
Clinical Competency Committees	Assessment tools that consist of the Clinical Competency Committee using multiple data sources to determine the ACGME Milestone ratings for trainees.	Clinical Competency Committee	22	11.75 (11.12-12.50)
Patient outcomes	Assessment tools that use patient data or outcomes as sources of assessment for residents.	Complication rates following a procedure Hospital readmissions Medical record reviews	16	13.00 (12.25-14.12)
Biometric data	Assessment tools that use biometric data, such as motion tracking, as measures of proficiency.	Eye tracking during procedures Motion tracking during procedures	6	12.50 (12.12-12.88)

Abbreviations: MERSQI, Medical Education Research Study Quality Instrument; ACGME, Accreditation Council for Graduate Medical Education.

Most studies used basic summative or average scoring methods to produce a score for a single assessment. Few studies discussed scoring methods that combined multiple assessments over time. One such method described in the articles was using cumulative sum control chart analysis (N=29). A total of 211 studies discussed feasibility of using their assessment tool; however, most of these studies only discussed local implementation. A total of 33 studies included a generalizability study related to their assessment tool. Many studies discussed some type of validity for their assessment tool (Table 2). As evidence for internal validity, 109 studies reported Cronbach’s alpha, 50 studies reported intraclass correlation coefficient, and 9 studies reported Kappa statistics. For content validity, the most cited evidence was expert consensus (152) and alignment with established competencies or guidelines (118).

No studies reported specific methodologies or techniques for translating assessment data from individual assessment tools into determinations of ACGME competencies. A total of 12 studies examined bias in assessment tools or assessment outcomes. The median MERSQI score for included studies was 12 (IQR 11-13), with scores ranging from 6 to 15.5.

Our Interpretations and Recommendations for GME

This examination found that WBAs, simulation, and checklists were the most common assessment tools used in GME. The majority of studies examined an assessment tool in one specialty and at a single institution. Across all studies evaluating a single competency, simulation was the most used assessment tool. When assessing multiple competencies in a single study, Milestones were the most common tool for aggregating assessment data. Many studies reported some validity evidence for an assessment tool, but the reported evidence was highly variable.

WBAs assess trainees in their authentic work environment, providing key insights into a trainee’s ability to perform at a specific point in time, facilitating both assessment for learning and assessment of learning.¹⁸ Simulations, by contrast, are performed in a controlled environment that can be directly modified to elicit specific knowledge and abilities and may have consistent raters and rubrics. Simulation-based assessments can be useful for ensuring a trainee masters a specific skill prior to interacting with patients. In addition, consistency in raters and rubrics may facilitate higher levels of assessment reliability. Finally, checklists were commonly used to document that a trainee can appropriately follow steps for a given task. Checklists not only emphasize the outcome of an activity but also the way the activity was carried out—both of which are critical for characterizing a trainee’s competence.¹⁹

Patient Care and Medical Knowledge were the most common ACGME competencies assessed because they are likely easier to assess with commonly used assessment tools, such as WBAs, simulations, checklists, and traditional testing formats. Some of the tools used to assess patient care, such as the Society for Improving Medical Professional Learning WBA and objective structured clinical examinations, have been broadly implemented across different institutions, specialties, and assessment settings.^20-25 Written or multiple-choice examinations continue to be a popular method for assessing Medical Knowledge. Other competencies, such as Interpersonal and Communication Skills, Professionalism, Systems-Based Practice, and Practice-Based Learning and Improvement, appear to be more difficult to observe in a way that can be reliably assessed by a rater.^26-28 Innovative tools, such as portfolios, have shown promising results for authentically assessing trainees in these competencies; however, they can be time-intensive to implement.^29,30

Most studies examined assessment tools within a single medical specialty. This finding is notable given that the assessment of competencies such as Interpersonal and Communication Skills, Professionalism, Systems-Based Practice, and Practice-Based Learning and Improvement are likely generalizable. Although there is significant variation in training environments and health care systems, one of the intended goals of Milestones 2.0 is to create harmonized language that is generalizable across different specialties and systems.²⁸ The potential for these competencies to manifest in similar ways across specialties could serve as a motivator to develop robust assessment systems through shared resources and lessons learned.

Our examination demonstrated that there is a substantial number of different assessment tools used in GME. The ingenuity evidenced by different assessment approaches has helped expand the ways in which assessment evidence is collected. While the continued development of assessment systems is to be commended, the diversity of data collection approaches could be better aligned to specific decisions that a user could make based on a learner’s performance, quantified as one or more “scores.” Assessment is an inherently practical activity³¹; therefore, unless being done for purely research purposes, assessment data should be collected to support formative feedback or specific acceleration, remediation, and certification decisions.

While several studies highlighted the possibility of using one or more observations to support a Milestone subcompetency rating, few if any provided rigorous quantitative rules for doing so. For example, WBAs were argued to be helpful in assessing trainees in their authentic work environment; however, left unaddressed is how different WBA ratings could be aligned to a subcompetency level (eg, PC2 Level 3). Aligning assessment evidence to a subcompetency requires specific mappings between proficiencies thought to drive observable performance, flexible analytical models, and explicit rules for observing multiple types of activities to cover elements of a desired subcompetency.³² Aligning assessment scores to subcompetency levels could not only improve existing assessments but could also highlight competencies with underdeveloped sources of evidence (eg, Professionalism). Similarly, understanding how to align specialty-specific entrustable professional activities (EPAs) to Milestones could help improve assessment quality and interpretation of the growing number of EPA evaluations. Additionally, more focused research on the measurement models used to aggregate observable and varied assessment evidence will be critical in achieving high-quality programmatic assessment (as proposed in the modern CBME model), which relies on multiple data points contributed by a variety of assessors and tools.³³

Recommendations

1. Simulation and WBA tools should be strongly considered when seeking to collect performance data related to Patient Care competencies. While already feasible in the case of WBAs, additional investment is needed to develop approaches across the entire range of Patient Care activities, many of which are specialty specific. Approaches to interpreting the data collected with simulation and WBAs (including specialty-specific EPA evaluations) remain underdeveloped, especially when interpreting data aggregated from multiple assessed performances and in statistically linking the observable performances to estimates of competencies.

2. Additional work is needed to develop, study, and support assessment of the Interpersonal and Communication Skills, Professionalism, Systems-Based Practice, and Practice-Based Learning and Improvement competencies. Assessment of these competencies may be generalizable across specialties. Where possible, new assessment tools should be developed and implemented while attending to issues of how to aggregate multiple observations and align overall scores to Milestone levels.

3. More studies should report sources of bias and possible mitigation strategies for reducing bias. Many studies have examined bias in assessment, but few original research studies investigate or report sources of bias for specific assessment tools. Further investigation into bias that exists when developing or implementing assessment tools would help to promote fair assessment practices and, ultimately, better assessment of trainees.³⁴

4. New tools should be scaled only after they have collected strong validity evidence, clearly articulated how the assessment data should be used to determine competency, and made rigorous efforts to optimize implementation-related factors, including characterizing which contexts are best suited for a specific assessment tool. This early-stage work should be performed in small pilots that include a representative sample of training programs.

5. The work needed to develop, test, and implement a new assessment should be performed in partnership with consortia of educators, researchers, training programs, and policymakers. This approach improves quality, ensures feasibility, and cost-effectively provides opportunity for more rapid innovation.

Caveats and Considerations

There is significant heterogeneity in current study designs, populations, interventions, outcomes, and measurement methods. This variation complicates data synthesis and makes comparisons difficult. We did not look at data pertaining to the type of training environment (ie, academic medical center, rural, etc) in which the assessments took place. Publication bias may have led to studies being skewed toward positive findings. Additionally, because we trained and used an LLM to perform the initial data extraction, it is possible that some of this data was missed or that the LLM introduced bias.

Conclusions

Assessment in GME is heterogeneous. While many assessment tools have been developed, most studies do not describe how to use the tool to inform competency decisions. To support high-quality and fair assessment in GME, future research should investigate how assessment outcomes can be used to inform competency decisions.

Author Note

*Drs Marcotte and Pradarelli are co-first authors and contributed equally to the work.

Editor’s Note

The ACGME News and Views section of JGME includes reports, initiatives, and perspectives from the ACGME and its review committees. This article was not reviewed through the formal JGME peer review process. The decision to publish this article was made by the ACGME. This is an article commissioned by the ACGME to inform the major revision of the Common Program Requirements, a 3-year process currently underway.