Comprehensive Assessment of Struggling Learners Referred to a Graduate Medical Education Remediation Program

Abstract

Background 

Implementation of the Next Accreditation System has provided a standardized framework for identifying learners not meeting milestones, but there is as yet no corresponding framework for remediation.

Objective 

We developed a comprehensive assessment process that allows correct diagnosis of a struggling learner's deficit(s) to promote successful remediation.

Methods 

At the University of Pennsylvania, resident learners within the Department of Medicine who are not meeting milestones are referred to the Early Intervention Remediation Committee (EIRC). The EIRC, composed of 14 faculty members with expertise in remediation, uses a standardized process to assess learners' deficits. These faculty members categorize primary deficits as follows: medical knowledge, clinical reasoning, organization and efficiency, professionalism, and communication skills. The standardized process of assessment includes an analysis of the learner's file, direct communication with evaluators, an interview focused on learner perception of the problem, screening for underlying medical or psychosocial issues, and a review of systems for deficits in the 6 core competencies. Participants were surveyed after participating in this process.

Results 

Over a 2-year period, the EIRC assessed and developed remediation plans for 4% of learners (14 of a total 342). Following remediation and reassessment, the identified problems were satisfactorily resolved in all cases with no disciplinary action. While the process was time intensive, an average of 45 hours per learner, the majority of faculty and residents rated it as positive and beneficial.

Conclusions 

This structured assessment process identifies targeted areas for remediation and adds to the tools available to Clinical Competency Committees.

Introduction

Two national surveys of internal medicine (IM) program directors (PDs) highlighted that it is common for residents to struggle during training.^1,2 An accurate assessment of a resident's primary deficit(s), while challenging, is critical to successful remediation. Evaluation data are often insufficient to correctly characterize the learner's primary deficit in a manner that will facilitate the appropriate type of remediation.

Building on the experience of a previously published remediation program,³ faculty at the University of Pennsylvania developed the Early Intervention Remediation Committee (EIRC) for struggling learners in 2014. We describe the feasibility, acceptability, and results of this process.

Methods

The University of Pennsylvania IM residency program annually has approximately 65 interns and 107 residents. Between 2014 and 2016, learners struggling with clinical performance were referred to the EIRC by the Clinical Competency Committee (CCC) at the time of the milestone ranking or at any time by the PD. The EIRC was led by 2 physician faculty members—a graduate medical education chair (0.2 full-time equivalent [FTE]) and an undergraduate medical education chair (0.2 FTE)—who oversaw the work of 12 other physician committee members. One committee member, a professionalism and communication expert (0.1 FTE), completed specialized training offered by the American Academy on Communication in Healthcare. The overall coaching program was funded by an intraentity transfer from the medical school and the department of medicine central budget. The remaining 11 faculty did not receive support for their efforts. Nine residents and fellows were recruited to serve as coaches for peers struggling with organization and efficiency. In many cases, direct observation was performed (under the direction of the EIRC chairs) by the attending physician working with the learner in continuity clinic or the inpatient setting. Administrative duties, estimated at 60 hours, were performed by 1 member of the residency program's administrative staff.

The 4 Components of the Assessment Process

Step 1:

Following referral to the EIRC, 1 of the cochairs performs an in-depth review of the learner's file using a standardized checklist (Figure 1). The review focuses on clinical performance in residency or fellowship and standardized test scores, along with an examination of the learner's performance in prior programs. Particularly in the case of a struggling intern, the undergraduate medical record often provides valuable information about the learner's trajectory. For example, an abrupt change may indicate an underlying problem with stress, burnout, or depression. A disconnect between the learner's standardized test scores and clinical performance may signal a mental health concern or a problem with clinical reasoning or communication skills.

Figure 1.

Assessment of the Learner in Need

Abbreviations: GME, graduate medical education; MSPE, Medical Student Performance Evaluation; PD, program director; UME, undergraduate medical education; USMLE, United States Medical Licensing Examination.

Step 2:

After developing a preliminary understanding of the situation, the EIRC cochair then speaks directly with evaluators to gain a more complete sense of the learner's performance. Additional direct observation, often by the assigned cochair, is usually necessary to refine the diagnosis and develop a targeted remediation plan. A sample form used for additional direct observation of the trainee struggling with clinical reasoning is provided as online supplemental material.

Step 3:

The cochair then conducts a 60-minute interview with the learner, using a tool adapted from the University of Colorado's remediation program³ that focuses on 3 areas: (1) understanding learner perception of the problem; (2) screening for concomitant life stressors, burnout, mental illness, substance abuse, and learning disability; and (3) conducting a review of deficits in each of the 6 core competencies.

Step 4:

Once the interview is complete, the cochair has sufficient information to diagnose a primary deficit in 1 of the following areas: medical knowledge, clinical reasoning, organization and efficiency, professionalism, or communication skills (Figure 2). The cochair then shares this diagnosis and a draft of a remediation plan with the full EIRC, typically via e-mail. Once the information has been shared, the EIRC edits and helps finalize a written remediation plan that is reviewed with the PD and the learner. The plan includes up to 5 specific and measureable goals for improvement, a description of where the remediation will take place, an assignment of specific faculty or resident, and a timeline and process for reassessment.

Figure 2.

Process of Referral, Assessment, and Diagnosis of Clinical Deficiency by the Early Intervention Remediation Committee

The Remediation

The remediation is typically carried out within the context of the learner's existing schedule. In a model developed by Hauer et al,⁴ remediation involves deliberate practice of the skill in question, real-time feedback by a mentor directly observing the trainee, and time for reflection. A sample remediation plan for a learner struggling with organization and efficiency is provided as online supplemental material.

The learner is then reassessed by the CCC. Throughout the remediation period, the EIRC cochair provides frequent informal progress reports that describe the learner's participation in the remediation effort and indicates whether the goals have been met. The CCC also uses new evaluation data to determine whether the trainee requires ongoing remediation or if any additional measures need to be implemented (eg, disciplinary action). Success of remediation is contingent on milestone scores and is defined by whether or not the learner has completed remediation or faced disciplinary action or termination.

Two years after program initiation, we conducted a written survey of learners after remediation and of EIRC faculty to assess hours invested and satisfaction with the program.

The study protocol was determined to be exempt from review by the Institutional Review Board at the University of Pennsylvania.

Results

Between 2014 and 2016, the EIRC performed a comprehensive assessment and developed remediation plans for 14 of 342 learners (4%; Figure 3). The majority of these learners were referred as interns, and 1 IM subspecialty fellow was referred by his PD. The most common primary deficit was organization and efficiency (n = 7), followed by medical knowledge (n = 3) and clinical reasoning (n = 3). In 6 learners (43%), an underlying medical (n = 2) or psychosocial (n = 4) issue requiring intervention was identified. Referral to a mental health provider was mandated in 2 cases and suggested in 2 others. We were able to proceed with remediation following initiation, but not necessarily completion, of the parallel intervention.

Figure 3.

Distribution of Deficits and Academic Outcomes of Learners Referred to the University of Pennsylvania Early Intervention Remediation Program (2014–2016)

All learners successfully completed remediation. Three received letters of poor performance prior to the successful completion of remediation, given the severity of deficits identified. However, with remediation, no disciplinary action was undertaken (eg, academic probation or dismissal). One learner voluntarily transferred to a program in another specialty.

The time commitment involved in remediation was significant: an average of 45 hours per learner (range, 10–100), which includes EIRC faculty and administrative staff time. Committee chairs spent approximately 15 hours per learner performing the assessment and coordinating the remediation. Usually, 1 faculty member was assigned as the primary coach and spent 5 to 20 hours per learner. Additional direct observation and feedback were then carried out by a senior resident, fellow, or faculty member. The majority of faculty and learners rated the overall process as positive.

Discussion

Our assessment process for struggling learners in an IM program has allowed us to correctly identify specific and targeted areas for remediation. We found that observations through the arc of a learner's entire training give greater context to what is directly observed and allow for the recognition of preexisting themes that contribute to current deficits. Our experience builds on the previous literature³ and provides support that a third-party remediation system is effective. To our knowledge, we are the first to describe a systematic assessment process.

We have found a relatively high incidence of primary deficits in the areas of organization and efficiency. Such an apparent deficit can represent many things, such as a more global problem with organization and time management, a primary deficit in clinical reasoning, or an underlying issue with mental well-being. Our assessment process, particularly through additional direct observation and an in-depth interview with the learner, allowed us to differentiate between these situations to provide targeted remediation.

Our study is limited by its small sample size at a single institution that had the ability to dedicate resources to the program. We feel that these fundamental concepts are reproducible if broken into specific steps. It is unclear if successful remediation would have occurred with fewer resources. Future work should compare this approach to interventions that are less resource intensive.

Conclusion

A standardized assessment process identified residents and fellows for targeted remediation early in their training. Although time intensive, the process successfully remediated most learners.