Entrustment and Mapping of Observable Practice Activities for Resident Assessment

ABSTRACT

Entrustable Professional Activities (EPAs) and the Next Accreditation System reporting milestones reduce general competencies into smaller evaluable parts. However, some EPAs and reporting milestones may be too broad to use as direct assessment tools. We describe our internal medicine residency curriculum and assessment system, which uses entrustment and mapping of observable practice activities (OPAs) for resident assessment. We created discrete OPAs for each resident rotation and learning experience. In combination, these serve as curricular foundation and tools for assessment. OPA performance is measured via a 5-point entrustment scale, and mapped to milestones and EPAs. Entrustment ratings of OPAs provide an opportunity for immediate structured feedback of specific clinical skills, and mapping OPAs to milestones and EPAs can be used for longitudinal assessment, promotion decisions, and reporting. Direct assessment and demonstration of progressive entrustment of trainee skill over time are important goals for all training programs. Systems that use OPAs mapped to milestones and EPAs provide the opportunity for achieving both, but require validation.

INTRODUCTION

In his 1983 article on feedback in medical education, Jack Ende wrote, “Not only are clinical skills infrequently observed, but when they are, the information so obtained does not get to where it can be most helpful—to the trainees themselves.”¹ In 1999, the Accreditation Council for Graduate Medical Education (ACGME) introduced the core competencies to improve trainee assessment.² However, many training programs still did not explicitly link resident competency attainment to patient care outcomes.² With the Entrustable Professional Activity (EPA), ten Cate suggested we should focus not on competencies directly, but rather on day-to-day activities and accomplishments of our trainees, and infer the presence of competencies.³ EPAs are broad activities of practice that the public entrusts all physicians with being capable of performing.^4,5 Many authors have suggested mapping EPAs to competencies and milestones to measure developmental progression,^6–8 and this is a focus of the reporting milestone component of the Next Accreditation System (NAS).⁹

The number and size of EPAs in a curriculum has been the subject of debate. In 2010, ten Cate and colleagues stated:¹⁰

To answer the question, how many EPAs should constitute the curriculum, one should primarily think of requirements at graduation…This will typically list anywhere from five to 15 EPAs, being not too detailed.

However, further on in the paper they stated:

EPAs … are broad responsibilities that may include smaller ones. As an example, the care for [carpal tunnel syndrome] (CTS) patients includes providing steroid injections and applying electromyography. These two smaller EPAs are components of the ‘CTS’-EPA, and must be mastered previously during the training period. This illustrates that one comprehensive EPA can be viewed as consisting of smaller, more elementary EPAs.

With this description, EPAs could be seen as a set of nesting dolls, with smaller ones fitting into bigger ones, and so on. For a broad specialty such as internal medicine, this could mean hundreds of EPAs over the course of training. Published curricula have not gone in this direction, however, ranging from 16 to 33 EPAs for internal medicine,^6,11 to 29 EPAs for psychiatry,¹² to 16 for pediatrics.¹³

EPAs and NAS reporting milestones reduce general competencies into smaller parts. However, these elements may be too broad to use as direct assessment tools. Take for example the EPA, serve as the primary admitting pediatrician for previously well children suffering from common acute problems.⁷ Which acute problems does this refer to? How should an assessor rate this EPA if the trainee manages 80 % of acute problems well? How often do training programs ensure that enough observers assess an EPA so large? In this paper, we describe an assessment system geared towards measuring entrustment of multiple specific observable practice activities (OPAs) over time, and mapping these entrustment decisions to milestones and EPAs to measure developmental progression.

SETTING AND PARTICIPANTS

The University of Cincinnati Internal Medicine Residency, based in an urban, tertiary referral medical center, consists of approximately 120 residents and 250 faculty members. Residents rotate at University of Cincinnati Medical Center, Veterans Affairs Medical Center, and multiple ambulatory clinics. Faculty members, peers, and allied health professionals provide end-of-rotation feedback.

PROGRAM DESCRIPTION

In the first phase of our project, we re-wrote the entire curriculum using OPAs as the basic unit. We created the term OPA to codify a collection of learning objectives/activities that must be observed in daily practice in order to form entrustment decisions. This labor intensive project included program directors, chief residents, and faculty members.

Evaluators assign entrustment decisions for OPAs during clinical experiences. These data are mapped directly to milestones and EPAs, which are then used for longitudinal assessment and external reporting (Fig. 1). Content OPAs (COPAs) describe specific activities that differ for each rotation depending on the discipline of medicine learned. Process OPAs (POPAs), are activities conserved across rotations, and approximately ten of these are assessed per rotation. POPAs, such as minimize unfamiliar terms during patient encounters, on first blush might not seem to stand on their own. However, we felt specific behaviors such as these that could be observed during a given clinical encounter would be higher value than using competency terms (such as communications skills) or broad EPAs. For a given rotation, we created graduated levels of OPAs—one set for PGY-1 residents and another for PGY2-4 residents. We also created peer and allied health multisource assessments consisting of POPAs only. Faculty members presented OPAs at divisional meetings for input and refinement. We combined appropriate content and process OPAs for each rotation and PGY-level to create our curriculum and end-of-rotation evaluations. After the first year, we refined OPAs based on the number and quality of assessments made. Our curriculum and assessment system now contain approximately 350 OPAs across more than 75 rotations.^14,15

Figure 1.

Observable practice activities (OPAs).

We mapped OPAs to NAS reporting milestones,¹⁶ curricular milestones,¹⁷ and end-of-training EPAs.⁶ When mapping, we asked ourselves the following question: when assessing a trainee on a given OPA, would the evaluator feel as if they were also assessing the mapped milestones and EPAs?

We assessed OPA performance using a 5-point entrustment scale:

1. Resident not trusted to perform activity even with supervision

2. Resident trusted to perform activity with direct supervision

3. Resident trusted to perform activity with indirect supervision

4. Resident trusted to perform activity independently

5. Resident trusted to perform activity at aspirational level

6. Activity was not observed on this rotation (produces no score)

Faculty members received education on the use of entrustment and were asked to justify in writing entrustment that could potentially be higher than expected.

End-of-rotation evaluation forms consist of rotation-specific OPA entrustments as well as narrative comments. Once an assessor assigns a level of entrustment for an OPA, this rating is automatically assigned to any mapped milestone or EPA. To create the semi-annual NAS milestone reports, aggregate 6-month data using the 5-point entrustment scale are normalized to the 9-point NAS reporting milestone rating form.

PROGRAM EVALUATION

Aggregate entrustment scores for mapped milestones and EPAs can be compiled for any time period, including the NAS reporting milestone 6-month reports (Fig. 2a). Because of the multiplier effect of mapping several milestones and end-of-training EPAs to each OPA, residents receive a significant amount of data. For example, during the academic year 2012–13, PGY-1 residents received an average of 1,711 NAS reporting milestone entrustment scores (range 1076–2136), PGY-2 residents an average of 764 (range 427–1995), and PGY-3 residents an average of 1263 (range 796–1814). Similar data can be derived for curricular milestones and end-of-training EPAs. PGY-2 s in our program receive fewer assessments due to a unique residency structure and parallel assessment system.^18,19

Figure 2.

a: NAS reporting milestone entrustment scores for a single PGY-3 resident for the 6-month periods of 7/1/12 to 12/31/12 (total of 495 assessments, average of 22.5 assessments per NAS milestone) and 1/1/13 to 6/30/13 (total of 927 assessments, average of 42.1 assessments per milestone). Entrustment scores are derived from entrustment decisions made by assessors for Observable Practice Activities over the course of the year. b, c, and d: comparison of entrustment curves for three PGY-1/2 residents (dark lines with circles) plotted against the entire class (grey line with diamonds) for the first five quarters of residency for a single NAS milestone (PC-2: develops and achieves comprehensive management plan for each patient). Curves such as these can be generated for every internal medicine NAS reporting milestone, curricular milestone, and end-of-training EPA. MedHub, Inc. P.O. Box 4148 Ann Arbor, Michigan 48106-4148.

Entrustment data are used to create curves for longitudinal assessment. Figure 2b–d shows a comparison of a single NAS milestone, PC-2: develops and achieves comprehensive management plan for each patient, for three PGY-1/2 residents over the first five quarters of residency. This example demonstrates progressive entrustment of the entire PGY-1/2 class over the year, moving from two (direct supervision) to three (indirect supervision). In the example, each individual resident also showed progressive entrustment, with the resident in 2b demonstrating faster entrustment than peer level, 2c demonstrating slower entrustment than peer level, and 2d demonstrating variable entrustment over time. All three residents achieved level 3 by the end of the first year. These curves are generated for each NAS reporting milestone, curricular milestone and end-of-training EPA. The clinical competency committee uses data showing rates of progression to assist with promotion decisions to higher levels of responsibility (e.g., promotion from the PGY-1 to PGY-2 years).

DISCUSSION

The process of OPA entrustment provides opportunity for immediate, structured, end-of-rotation feedback of specific clinical activities. OPAs mapped to NAS reporting milestones and EPAs allow for entrustment of these larger elements, as well as longitudinal assessment over time.

Entrustment and mapping of OPAs leads to several important questions. Did we choose the correct OPAs to assess? We engaged faculty members across all divisions to identify important OPAs, using local clinical opportunities as well as American Board of Internal Medicine content areas as a guide. The entire department had an opportunity to vet the OPAs before activation. Much like an artist drawing a few bricks to indicate a wall, we felt a few well-selected OPAs could suggest a good picture of resident performance during a rotation. This will continue to evolve and require updates with advances in knowledge.

Additionally, were the correct mapping decisions made? We used our best judgment and adhered to our guideline for mapping (when assessing a trainee on a given OPA, would the evaluator feel as if they were also assessing the mapped milestones and EPAs?). However, this process is subjective, and different programs might make different mapping decisions. This could be a strength, as each program could choose what to emphasize through mapping. This could also be a weakness, as variation could produce results difficult to compare across programs. Correct mapping decisions will require longitudinal assessment and comparison with future performance.

Another important question is, what is the appropriate rate of entrustment progression? There is no gold standard. We compared each resident to his/her peer group, as well as the absolute entrustment level for each milestone and EPA. Most residents showed progressive entrustment over time on most milestones and EPAs. However, some did not. How will program directors and competency committees handle this? In our case, we reviewed the curves in real time, and intervened early when flat or declining curves were identified. As of now, we have not promoted those residents who attained entrustment faster than peers any earlier, nor did we hold residents back who were slower. However, there is debate in the education community regarding the concept of flexible and tailored education, and systems such as ours could inform the discussion.

In general, communications skills and professionalism scores progressed fastest to each entrustment level, with the other competencies progressing more slowly. The PGY-1 and PGY-3 classes showed fairly uniform and steady entrustment over the year. However, the PGY-2 curves were variable. This is likely due to PGY-2 s and PGY-3 s serving in similar roles (team leaders, etc.) with similar responsibilities. We created separate OPAs and assessment forms for PGY-1 s, but the PGY-2 s and PGY-3 s used the same forms. We will continue to study and compare progression in the middle year, and may consider creating separate OPAs and assessment forms for PGY-2s and PGY-3 s in the future.

We also need to determine if normalizing our 5-point scale to the 9-point NAS reporting milestone scale has validity. We believe our five levels of entrustment exactly correlate with the five columns on the NAS reporting document. We have confidence that automatically aggregating hundreds of data points to complete the NAS reporting document could provide a more complete portrayal of performance than competency committee personnel manually attempting to synthesize data in order to check the appropriate boxes. Our competency committee reviews NAS reporting document data derived from entrustment decisions of the previous 6 months (with the 5-point scale normalized to 9 points), and then decides if the data supports entrusting a resident with greater responsibility and independence.

An additional question is whether mapping NAS milestones, curricular milestones, and end-of-training-EPAs provides significantly different data. We have used all three tools to identify reasons and remedies for resident struggles, but need further study to determine if this is truly valuable.

Finally, would this system work in another setting? It is possible this system would not be generalizable to other programs and settings, or that other programs might get significantly different results. We are currently sharing our work with a community-based internal medicine residency, subspecialty internal medicine programs, and other specialties to study the differences in effect.

In summary, direct assessment and demonstration of progressive entrustment of trainee skill over time are important goals for all training programs. Systems that incorporate OPAs mapped to milestones and EPAs over time provide the opportunity for achieving both, but require validation and further study.