Developing Entrustable Professional Activities as the Basis for Assessment of Competence in an Internal Medicine Residency: A Feasibility Study

ABSTRACT

BACKGROUND

Graduate medical education programs assess trainees’ performance to determine readiness for unsupervised practice. Entrustable professional activities (EPAs) are a novel approach for assessing performance of core professional tasks.

AIM

To describe a pilot and feasibility evaluation of two EPAs for competency-based assessment in internal medicine (IM) residency.

SETTING/PARTICIPANTS

Post-graduate year-1 interns (PGY-1s) and attendings at a large internal medicine (IM) residency program.

PROGRAM DESCRIPTION

Two Entrustable professional activities (EPA) assessments (Discharge, Family Meeting) were piloted.

PROGRAM FEASIBILITY EVALUATION

Twenty-eight out of 43 (65.1 %) PGY-1 s and 32/43 (74.4 %) attendings completed surveys about the Discharge EPA experience. Most who completed the EPA assessment (10/12, 83.8 %, PGY-1s; 9/11, 83.3 %, attendings) agreed it facilitated useful feedback discussions. For the Family Meeting EPA, 16/26 (61.5 %) PGY-1s completed surveys, and most who participated (9/12 PGY1s, 75 %) reported it improved attention to family meeting education, although only half recommended continuing the EPA assessment.

DISCUSSION

From piloting two EPA assessments in a large IM residency, we recognized our reminder systems and time dedicated for completing EPA requirements as inadequate. Collaboration around patient safety and palliative care with relevant clinical services has enhanced implementation and buy-in. We will evaluate how well EPA-based assessment serves the intended purpose of capturing trainees’ trustworthiness to conduct activities unsupervised.

Electronic supplementary material

The online version of this article (doi:10.1007/s11606-013-2372-x) contains supplementary material, which is available to authorized users.

INTRODUCTION

To fulfill their mission, graduate medical education programs must ensure that trainees are competent to practice medicine. The Accreditation Council for Graduate Medical Education (ACGME) six core competencies aim to address outcomes rather than the process of medical education.¹ Milestones (discipline-specific developmental achievements toward competence) ideally facilitate meaningful workplace-based assessment over time.^2–4 However, efforts to implement competency-based medical education have been stymied by practical challenges and assessment questions. Competency-based assessment tools, while potentially psychometrically strong, can prompt narrow focus on aspects of individual competencies,⁵ and improper implementation of assessment tools with inadequate faculty training limits the information gained.^4,6

Entrustable professional activities (EPAs) are a novel method of operationalizing competencies and milestones in the context of actual clinical work.⁷ By definition, an EPA reflects relevant competencies and milestones; requires skills, knowledge, and attitudes; addresses a professional task with a recognizable output; and can be observed and judged by an expert. EPAs naturally focus on holistic performance of actual physician tasks. Assessment based on EPAs addresses the need to determine whether trainees are ready for unsupervised practice, and can enable granting of entrustment of a trainee to conduct certain activities unsupervised, based on assessment of performance of those activities.

This paper describes the development and feasibility of two EPAs for competency-based assessment in a large internal medicine (IM) residency program.

SETTING AND PARTICIPANTS

Setting

The University of California, San Francisco (UCSF) IM residency includes 62 post-graduate year-1 interns (PGY-1s) who rotate to university, Veterans Affairs, and public county hospitals. Prior to and concurrent with EPA-based assessment implementation, trainees were assessed with: end-of-month global evaluations by faculty, residents, and students; 1–2 annual mini-Clinical Evaluation Exercise (mini-CEX) encounters rated by faculty in continuity clinics; and an in-training exam. Each resident has an advisor serving on a Committee on Housestaff Evaluation and Feedback (CHEF) for discussion of academic progress, career planning, and wellbeing.

PROGRAM DESCRIPTION

A Department of Medicine assessment leadership group developed two EPAs in spring 2011 and piloted them for assessing PGY1s during the 2011–12 academic year.

EPA Development

We formed an assessment leadership group consisting of the residency program director, associate chair for education, associate program director for evaluation, two department of medicine faculty with medical education expertise, and chief residents. This group organized two retreats for IM residency leaders, including other associate program directors, residency advisors, fellowship directors, chief residents and residents. At the first 2-hour retreat, participants discussed limitations of the current evaluation system, reviewed the ACGME competencies and IM curricular milestones, and learned about EPAs.^1,4,8 Participants rated the relevance of potential IM EPAs.⁹ At a second 3-hour retreat, participants reviewed the potential EPAs including comments from the first retreat. They heard two presentations on connecting curricular milestones to EPAs. The leadership group subsequently held monthly meetings with additional faculty involved in the clinical services.

EPAs

Based on rankings of potential EPAs from the residency retreats and discussions of the assessment leadership group with key stakeholders in the residency curriculum and clinical services, two pilot EPAs were selected for assessment:

1. Inpatient “Discharge” (largely achievable by PGY-1s)—aligned with interests of quality improvement and patient safety leaders.

2. “Family Meeting” to discuss difficult information with patients and families (achievable primarily by PGY-2s)—aligned with existing categorical PGY-1 curriculum within a palliative care rotation.

The leadership group developed the EPAs following the format used in a competency-based workplace curriculum for physician assistants.^10,11 As shown in online Appendices 1 and 2, each EPA includes a title, setting, learning goals, description, relevant IM competencies and curricular milestones,⁴ other information that informs performance of this EPA, and a strategy for determining whether the trainee can be trusted to perform this activity independently.

Table 1 summarizes the requirements for assessment of each EPA. Prior to the study, PGY-1s managed inpatient discharge with direct supervision from the team PGY-2/3 and team supervision by the attending. The Discharge EPA comprises a meeting between the PGY-1 and medicine ward team attending to review the discharge plan and discharge summary for a recently hospitalized patient. Using a standardized rubric based on literature summarizing elements of effective care transitions, the attending provides feedback to the PGY-1 and makes an assessment of entrustability to be used by the CHEF advisors^12–14 (online Appendix 3).

Table 1.

Summary of Two Entrustable Professional Activity (EPA) Assessments Piloted in an Internal Medicine Residency

EPA title	Discharge EPA	Family meeting EPA
EPA description	Develop and implement a safe discharge plan for an inpatient from the acute care setting	Observe a family meeting to discuss serious or sensitive news with patient and/or family
PGY level	PGY-1 to PGY-2	PGY-1 to PGY-2
Curriculum	• 1-hour didactic on writing discharge summaries	• 1 week Palliative Care Rotation includes participating in family meetings and goals of care discussions
	• Two 1-hour noon conferences on safe patient discharge–didactic and small group exercise	• 1-hour noon conference on running a family meeting
	• 1-hour interactive session for interns to review peers’ discharge summaries and give feedback	• Online modules and selected papers for self-directed learning as part of the 2-hour long critical reflection exercise
	• Monthly PGY-1 ward orientations, includes site specific key elements of patient discharge
Setting	4-week inpatient general medicine ward rotation at University Hospital	1-week inpatient palliative care rotation at Veterans Affairs hospital
Activity	For initial activity, PGY-1 discharges ward patient, including completing discharge summary	PGY-1 observes a family meeting
		PGY-2/3s lead family meeting
Assessment	• EPA rubric completed by ward attending with PGY-1	• PGY-1 writes a critical reflection based on observations of a family meeting
	• PGY-1 reflection in portfolio	• PGY-1 receives written feedback on reflection from pre-specified faculty/fellows
		• PGY-2 and PGY-3s observed leading at least three serious illness conversations and receive structured, timely feedback

PGY post-graduate year intern

The multi-year Family Meeting EPA-based assessment begins with categorical PGY-1s. In a 1-week mandatory palliative care rotation, PGY-1s observe a senior resident, fellow, or attending conduct a conversation about serious illness and/or goals of care. Each PGY-1 then completes a written reflection with a brief description of the family meeting, reconsideration of the meeting from multiple perspectives, identification of personal learning goals, and an action plan. (online Appendix 4). After entering the reflection into an electronic portfolio, the PGY-1 receives written feedback from palliative medicine faculty and fellows. This portion of the EPA assessment was piloted in 2011–12.

Faculty and Intern Development

A chief resident and associate program director, both charged with quality improvement and patient safety education, educated interns and faculty about the Discharge EPA. Orientation included a session on EPA-based assessment at a Hospital Medicine faculty meeting, email orientation information to attendings before their time on the medical service, and individual orientation for those not at the faculty meeting. Family Meeting EPA faculty development happened at a faculty meeting; the EPA assessment occurred in the PGY-1 palliative care rotation to allow time for intern education about the content, as well as an introduction to the EPA.

PROGRAM FEASIBILITY EVALUATION

We evaluated feasibility of the two pilot EPA-based assessments using several metrics for evaluating feasibility studies¹⁵: survey response rate, EPA assessment completion rates, participant satisfaction and willingness to do the EPA assessment. We also solicited barriers to doing EPA assessments. Three survey questions addressed perceived skill improvement, feedback usefulness with the EPA and a recommendation about continuing the EPA assessment, along with space for written comments about barriers to completing the activities. The Institutional Review Board approved the study.

Results

The Discharge EPA assessment was piloted at the university hospital beginning July, 2011. All 43 PGY-1 and attending pairs on service were eligible to do the EPA assessment and received surveys. Of those, 28 (65.1 %) PGY-1s and 32 (74.4 %) attendings completed surveys. (Table 2)

Table 2.

Post-Graduate Year-1 (PGY-1) and Attending Surveys: Experiences with Two Entrustable Professional Activity (EPA) Assessments

	PGY-1  N (%)	Attendings N (%)	Mean rating* (SD)
Discharge EPA
Surveys sent	43 (100 %)	43 (100 %)
Responses	28 (65.1 %)	32 (74.4 %)
Participated in discharge EPA	12 (42.9 %)	11 (34.4 %)
The inpatient discharge EPA improved my discharge planning skills.	8 (66.7 %)	5 (45.5 %)	3.55 (0.93)
The inpatient discharge EPA facilitated a useful feedback discussion.	10 (83.3 %)	9 (81.8 %)	3.36 (1.12)
I recommend continuing to use the inpatient discharge EPA.	10 (83.3 %)	7 (63.6 %)	3.36 (1.12)
Did not participate	16 (57.1 %)	21 (65.6 %)
Reason
I did not know about it	9 (56.3 %)	11 (52.4 %)
I didn’t have time	5 (31.3 %)	7 (33.3 %)
I forgot about it	1 (6.3 %)	3 (14.3 %)
No answer	1 (6.3 %)	0
Interested in participating
Yes	15 (93.8 %)	19 (90.5 %)
No	1 (6.3 %)	2 (9.5 %)
Family meeting EPA
Surveys sent	26 (100 %)
Responses	16 (61.5 %)
Participated in family meeting EPA	12 (42.9 %)	11 (34.4 %)
This EPA improved my attention to family meeting education	9 (75.0 %)		3.57 (0.90)
The EPA critical reflection was useful for my learning	6 (50.0 %)		4.04 (0.88)
I recommend continuing to use the EPA	6 (50.0 %)		4.09 (0.87)
Did not participate	4 (25.0 %)
Reason
I did not know about it	4 (100 %)
Interested in participating
Yes	2 (50.0 %)
No	1 (25.0 %)

*1–5 Likert scale, 1 = strongly disagree, 5 = strongly agree

Of the 43 pairs, 12 PGY-1s and 11 attendings who completed surveys had done the Discharge EPA assessment. Both PGY-1s and attendings responded favorably about the EPA assessment: eight PGY-1s and five attendings felt it improved their discharge planning skills, and ten PGY-1s and nine attendings felt it facilitated useful feedback discussions. Over 90 % of those who did not do the EPA assessment wished to participate. Not knowing about the EPA or not having time were the most commonly identified barriers to participating. (Table 2)

The Family Meeting EPA assessment was implemented in July, 2011 at the Veterans Affairs Hospital. Of the 26 PGY-1s who completed the palliative care service and thus received surveys, 16 (61.5 %) completed the survey and, of those, 12 completed the EPA assessment. (Table 2) The majority (12/16, 75 %) of respondents endorsed that the EPA improved their attention to family meeting education; half found the critical reflection useful for learning and recommended continuing the EPA assessment. The four non-participants cited not knowing about the EPA as the barrier to participation.

DISCUSSION

In our pilot of two EPA-based assessments in IM residency, participating PGY-1s and attendings found EPA activities useful for learning and feedback. Among non-participants, interest in participating was high. We identified barriers to participation that reflected the multiple time demands in the inpatient setting and challenges in disseminating information about a new assessment system in a large program.

We designed this initial implementation to evaluate the feasibility of using EPAs that can be the basis for milestones-based assessment in the next accreditation system (NAS); we did not study performance levels or entrustment at this pilot phase.^15,16 Our feasibility study shows that we could orient interns and faculty; both groups are willing to participate in this assessment method, and all could identify advantages to the system. However, participation rates were low. Strategies to increase adherence to completing the EPA assessments are needed. As with any assessment system, EPAs require ongoing administrative support for monitoring compliance and reminding trainees and faculty to complete expected activities. We found that frequent email reminders improved compliance, and we have increased endorsements and reminders about EPA assessments from program leadership.

Our pilot reveals barriers to and lessons about integrating EPA assessments with clinical work. We found it challenging to engage PGY-1s and faculty on an inpatient rotation with high service expectations, despite their stated interest in EPAs. Interns feel tension balancing work and learning, and the culture may prioritize immediate work duties.¹⁷ PGY-1s and attendings struggled to find time together to review a patient discharge, discuss feedback, and complete a short rubric. Limited trainee-supervisor contact¹⁸ will impede informed entrustment decisions. Better understanding of the time required for each component of EPA-based assessment and practical strategies to build EPA-based assessment into the workday are needed. Based on initial feedback, PGY-1s are now advised to login to their portfolio at the meeting to review expected elements of the discharge process and upload the completed rubric, both for time-efficiency and for exchanging technical knowledge of the portfolio. Several factors may explain the higher participation rate with the Family Meeting EPA assessment than with the Discharge EPA. Housing the Family Meeting EPA assessment within a required palliative care rotation allowed more time for EPA orientation and discussion with fewer competing clinical responsibilities.

Our implementation strategy was strengthened by collaboration with key clinical service and curriculum leaders. The Discharge EPA advances the Division of Hospital Medicine patient safety and quality agenda that prioritizes a care transitions curriculum for residents. Thus, hospitalist faculty had both educational and patient safety incentives to participate. The Family Meeting EPA occurs within an existing intern palliative care rotation with dedicated faculty interested in both education and advancing palliative care quality. Synergies between education and clinical goals are important in the current environment of increased demands for both clinical supervision and clinical productivity.¹⁹

Future studies are needed to evaluate how effectively EPA assessments capture meaningful aspects of residents’ performance, particularly trainees’ trustworthiness to practice independently in the future. Judgments of entrustment will entail collating assessment information from multiple observations from multi-disciplinary team members about performance of an activity. In complex care environments, methods of capturing an individual’s contributions to team performance will be needed. We also plan to share the pertinent curricular milestones for each EPA with interns and attendings on rubrics and portfolio pages, to build understanding of the components of expected performance. We believe that PGY-1 enthusiasm may have been lower for the Family Meeting EPA than the Discharge EPA because of the intern’s relatively passive role; the reflection exercise is a preparatory step for a senior resident activity leading family meetings.

Next steps in our pilot will be to elevate the use of EPA assessments to inform decisions of entrustment¹⁰ through review of performance by each advisor who sits on the competence committee. Because each EPA is mapped to multiple competencies and curricular milestones, this review of resident performance and entrustment can capture the new milestones-based assessment required by the ACGME. However, we are developing our strategy for evaluating curricular milestones within an EPA, and for determining whether entrustment for an EPA guarantees successful achievement of all embedded curricular milestones. Balancing innovations in assessment with regulatory requirements is a challenge in any educational program; our experience suggests that EPAs are an assessment strategy that allows for program innovations while also assessing multiple key curricular milestones.³ We will need remediation tasks for residents, identified through EPA-based assessment, who require additional experience to reach the entrustability threshold.

This study has limitations. This project represents a pilot innovation at a single institution with two EPAs to date. Not all eligible participants completed the activities or surveys. We do not know if faculty respondents completed faculty development or whether participants were more likely to do the EPA assessments. More nuanced understanding of barriers to PGY-1 and attending participation and solutions for each are needed before EPAs can be used for granting entrustment for unsupervised practice of certain activities. We cannot yet determine the degree to which our EPAs enhance education and assessment, although our learners and attendings attested to some value.

Our results show the potential for EPAs to enhance IM trainee assessment focused on competence in practice. We plan to develop additional EPA-based assessments, including EPAs currently being initiated in outpatient and emergency/urgent care settings. Future work should focus on outcome measures to show whether performance on EPAs correlates with performance of other duties and practice outcomes.