Abstract
Background
This multi-center, retrospective study was conducted to determine how resident performance deficiencies affect graduation and board certification.
Methods
Primary documents pertaining to resident performance were examined over a 10-year period at four academic anesthesiology residencies. Residents entering training between 2000 and 2009 were included, with follow-up through February 2016. Residents receiving actions by the programs’ Clinical Competency Committee were categorized by the area of deficiency and compared to peers without deficiencies.
Results
A total of 865 residents were studied (range: 127–275 per program). Of these, 215 residents received a total of 405 actions from their respective Clinical Competency Committee. Among those who received an action compared to those who did not, the proportion graduating differed (93% versus 99%, respectively, P<0.001), as did the proportion achieving board certification (89% versus 99%, respectively, P<0.001). When a single deficiency in an Essential Attribute (e.g., ethical, honest, respectful behavior; absence of impairment) was identified, the proportion graduating dropped to 55%. When more than three Accreditation Council for Graduate Medical Education core competencies were deficient, the proportion graduating also dropped significantly.
Conclusions
Overall graduation and board certification rates were consistently high in residents with no, or isolated, deficiencies. Residents deficient in an Essential Attribute, or multiple competencies, are at high risk of not graduating or achieving board certification. More research is needed on the effectiveness and selective deployment of remediation efforts, particularly for high-risk groups.
INTRODUCTION
Residency programs have a responsibility to the public and to their residents to train competent medical specialists. To fulfill this mission, residency programs monitor the progress of their residents through supervision, evaluations, and examinations. When trainees are observed to experience personal, professional, or academic difficulties, the residency program intervenes to assist these residents.
Residents with performance concerns have been extensively studied.1–22 The prevalence of residents with performance deficiencies has been reported between 6% and 26% across specialties including psychiatry, family medicine, internal medicine, and general surgery.1,9,10,17 These residents have been referred to variously as “problem residents,” “residents in difficulty,” “troublesome residents,” “problem learners,” and “residents in trouble.”1 We have chosen the term “residents in trouble,” since it conveys the attitude of concern for the performance and improvement of trainees, as we might say when our patients are “in trouble.”
The methods used to study residents in trouble vary. The majority of studies used surveys of program directors to establish the prevalence of residents with deficiencies, the most commonly deficient categories, and the remediation strategies. Results are subject to both response and recall bias. In addition, surveys typically report cross-sectional data, which are less useful in determining the proportion of residents who graduate and become board certified.
The 1999 landmark survey of the Association of Program Directors in Internal Medicine identified a 7% point prevalence of “problem residents” (defined by the American Board of Internal Medicine as “a trainee who demonstrates a significant enough problem that requires intervention by someone of authority”); 94% of programs had problem residents.18 The most frequently identified problems were medical knowledge, poor clinical judgment and inefficient use of time. A 2012 update of this survey describing deficiencies in terms of Accreditation Council for Graduate Medical Education (ACGME) core competencies showed that the most commonly deficient ACGME core competencies were Medical Knowledge, Patient Care, and Interpersonal Communication.20
The prevalence of residents in trouble has also been assessed through single center longitudinal studies within one specialty, allowing determination of graduation and board certification rates. A single center 10-year longitudinal study of surgical residents reported that 21% of 115 residents performed poorly; 75% (18/24) of these residents graduated. Deficits clustered around medical knowledge and its application, personal problems, including health, and interpersonal skills/professionalism.21 A 25-year retrospective analysis of 230 family medicine residents at a single site revealed a 9% prevalence of residents in trouble. The most common deficits identified were in medical knowledge, attitudinal problems, and interpersonal conflict.1
Despite the considerable literature on “residents in trouble,” the effect of performance gaps on graduation and board certification has not been well characterized, including in anesthesiology. There are currently over 130 U.S. anesthesiology programs accredited by the ACGME. Each program reports resident physician performance to the American Board of Anesthesiology (ABA). The aim of this 10-year multicenter study is to report the prevalence of unsatisfactory performance, characterize the nature of performance gaps and examine their effect on graduation and board certification.
MATERIALS & METHODS
With local Institutional Review Board approval (University of California at Los Angeles, Los Angeles, California), we performed a multicenter retrospective review of anesthesiology residents’ files. Residents entering anesthesia year 1 (PGY-2) training between 2000 and 2009 were included in the analysis. Data sources reviewed included: Clinical Competency Committee (CCC) reports; Residency Program Director memoranda; and the semi-annual American Board of Anesthesiology Training Reports (ABA RTR/CCC Report) in force during the study period (Table 1).
Table 1.
Graded Components of the ABA Record of Training/Clinical Competence Committee Report* Used During the Study Period
| Essential Attributes |
|
| Core Competencies |
| Patient Care |
|
| Medical Knowledge |
|
| Practice-Based Learning and Improvement |
|
| Interpersonal and Communication Skills |
|
| Professionalism |
|
| Systems-Based Practice |
|
| Clinical Skills |
|
| Overall Clinical Competence |
Essential Attributes and Overall Clinical Competence were graded as Satisfactory (meets reasonable expectations) or Unsatisfactory (falls short of reasonable expectations). Overall Clinical Competence was graded as Satisfactory only if the grade for every Essential Attribute was Satisfactory. If Overall Clinical Competence was Unsatisfactory, a description of the anesthesiologist’s most serious deficiencies was submitted with the report. Core Competency skills were graded as Satisfactory (meets reasonable expectations), Unsatisfactory (falls short of reasonable expectations), or Not Applicable (used only for those categories not required of the resident during the reporting period).
ABA = American Board of Anesthesiology
Adapted with permission from the American Board of Anesthesiology
The CCC is an ABA and ACGME mandated body composed of faculty members who meet regularly (2–4 times per year) to assess resident performance. The role of the CCC is to track and evaluate resident performance in order to follow the progress of residents in the program and advise the Program Director regarding suitability for graduation and independent practice.
The ABA RTR/CCC Report assesses resident performance based on seven “Essential Attributes” (qualities of character and professionalism deemed “essential” by the ABA; these attributes map to elements of the ACGME Core Competencies of Professionalism, Patient Care, and Interpersonal and Communication Skills), six ACGME Core Competencies, and four Clinical Skills outlined in Table 1. The Chair of the CCC and Program Director are responsible for reporting resident performance to the ABA.
Each Clinical Competency Committee at the respective residency programs established and implemented its own specific policies and procedures to evaluate the performance and progress of trainees. A pattern of poor evaluations resulted in CCC action. CCC actions included counseling, letter of concern, Unsatisfactory ABA Training report, probation, and, if deficiencies were not resolved after successive attempts at remediation, forced resignation or dismissal. Of the range of CCC actions, only the ABA training reports could be compared across programs in this study. The criteria driving other actions, for example, placing a resident on probation, are unique to each institution, and were therefore not amenable to comparison. Each program devised its own approach to remediation in general, which was tailored to the unique needs of each of the referred residents.
Study Sites
The four study sites were Massachusetts General Hospital, University of California, Los Angeles (UCLA, serving as the coordinating site), University of California, San Francisco, and University of Colorado School of Medicine. During the period under review, there were three Program Directors and a single CCC Chair at UCLA, two Program Directors and a single CCC Chair at University of California, San Francisco, and a single Program Director and single CCC Chair at Massachusetts General Hospital and University of Colorado School of Medicine. All sites are ACGME-accredited anesthesiology residency programs.
Data Collection and Outcome Measures
Residents from the four training programs who commenced clinical anesthesia year 1 between 2000 and 2009 were included in the analysis, including residents who transferred into the program at the PGY-2 level or beyond (the years devoted exclusively to training in clinical anesthesia). Since reporting criteria differ between residents and interns, only PGY-2 through PGY-4 residents were included in the study. Performance reports through 2012 were reviewed in order to include data for the class entering in 2009. ABA reports through February 2016 were used to assess board certification.
The Program Director, CCC Chair or their designees compiled data at each site. They reviewed source documents, which included ABA training reports, CCC minutes, emails and individual resident files. Data were de-identified and abstracted into a single database that included: year of entry into the residency program, program site, and, if applicable, year of 1st CCC action, level of training at the time of CCC action, the type of action, and the ACGME Core Competencies of concern. In addition, if the resident received one or more Unsatisfactory designations on the ABA RTR/CCC report, the database included a notation of the primary Unsatisfactory attribute or competency, in the judgment of the training program. Outcomes measured included whether the training program was completed (graduation) and whether ABA Board Certification was attained.
Statistical Analysis
Descriptive statistics were summarized using frequencies and percentages in contingency table format. In order to test for site to site differences, or test other associations including types of deficiencies, actions, and outcomes, we used the Pearson’s chi-square test or Fisher’s exact test as appropriate. These tests were carried out using JMP Pro 12 (SAS Institute, Cary, North Carolina).
For assessing binary outcomes (graduation or ABA certification), generalized logistic mixed effects models were constructed. Due to site to site variation discovered in our descriptive analysis, a random site effect was added to these models (Tables 2, 3, 4, 5 and 6). Mixed effects models were constructed using SAS 9.4 (SAS Institute, Cary, NC).
Table 2.
Graduation and Board Certification Rates – All Residents
| Site | Program A | Program B | Program C | Program D | Total (%) |
|---|---|---|---|---|---|
| All residents 2000–2009 |
127 | 249 | 275 | 214 | 865 |
| Graduates (%)a | 124 (98) | 244 (98) | 269 (98) | 208 (97) | 845 (98) |
| ABA certified (%)b | 122 (96) | 243 (98) | 266 (97) | 206 (96) | 837 (97) |
P =0.96 (Fisher’s Exact test) for comparisons of graduation across the four sites (Programs A–D)
P =0.78 (Fisher’s Exact test) for comparisons of ABA certification across the four sites (Programs A–D)
ABA = American Board of Anesthesiology
Table 3.
Graduation and Board Certification Rates – Residents Receiving vs. Not Receiving CCC Actions
| Site | Program A | Program B | Program C | Program D | Total (%) |
|---|---|---|---|---|---|
| Residents not receiving CCC actions |
107 | 217 | 221 | 105 | 650 |
| Graduates (%)a | 107 (100) | 217 (100) | 217 (98) | 105 (100) | 646 (99) |
| ABA certified (%)b | 107 (100) | 217 (100) | 217 (98) | 105 (100) | 646 (99) |
| Residents receiving CCC action |
20 | 32 | 54 | 109 | 215 |
| Graduates (%)a | 17 (85) | 27 (84) | 52 (96) | 103 (94) | 199 (93) |
| ABA certified (%)b | 15 (75) | 26 (81) | 49 (91) | 101 (93) | 191 (89) |
P <0.001, after controlling for site, for comparisons of graduation between all residents receiving and not receiving CCC actions
P <0.001, after controlling for site, for comparisons of ABA certification between all residents receiving and not receiving CCC actions
ABA = American Board of Anesthesiology
Table 4.
Graduation and Board Certification Rates – Residents Receiving vs. Not Receiving Unsatisfactory Designations
| Site | Program A | Program B | Program C | Program D | Total (%) |
|---|---|---|---|---|---|
| Residents not receiving Unsatisfactory designation |
121 | 224 | 268 | 134 | 747 |
| Graduates (%)a | 119 (98) | 224 (100) | 264 (99) | 134 (100) | 741 (99) |
| ABA certified (%)b | 118 (98) | 224 (100) | 261 (97) | 134 (100) | 737 (99) |
| Residents receiving Unsatisfactory designation |
6 | 25 | 7 | 80 | 118 |
| Graduates (%)a | 5 (83) | 20 (80) | 5 (71) | 74 (92) | 104 (88) |
| ABA certified (%)b | 4 (67) | 19 (76) | 5 (71) | 72 (90) | 100 (85) |
P <0.001, after controlling for site, for comparison of graduation between all residents receiving and not receiving Unsatisfactory designations
P <0.001, after controlling for site, for comparison of ABA certification between all residents receiving and not receiving Unsatisfactory designations
ABA = American Board of Anesthesiology
Table 5.
Unsatisfactory Ratings to ABA in Essential Attributes versus Other Categories
| ABA Unsatisfactory Ratings | No. of Residents (% of column) |
Graduates (% of row) |
ABA Certified (% of row) |
|---|---|---|---|
| Total Unsatisfactory Ratings (%) | 118 (100) | 104 (88) | 100 (85) |
| Essential Attributes (%) | 11 (9) | 6 (55)a* | 5 (45)b* |
| Other Categories (excluding Essential Attributes) (%) |
107 (91) | 98 (92)a* | 95 (89)b* |
P =0.0013 for comparisons of graduation between residents receiving Unsatisfactory ratings in Essential Attributes versus other categories, after controlling for site
P =0.0007 for comparisons of ABA certification between residents receiving Unsatisfactory ratings in Essential Attributes versus other categories, after controlling for site
Statistically significant after Bonferroni correction
ABA = American Board of Anesthesiology
Table 6.
Graduation and Board Certification by Number of ACGME Competencies Deficient
| Number of Deficient ACGME Competencies |
0 | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|---|
| Total number of residents |
650 | 120 | 40 | 31 | 14 | 4 | 6 |
| Graduates (% of column total)a,* |
646 (99) |
116 (97) |
39 (97) |
30 (97) |
9 (64) |
3 (75) |
2 (33) |
| ABA Certified (% of column total)b,* |
646 (99) |
114 (95) |
37 (92) |
27 (87) |
8 (57) |
3 (75) |
2 (33) |
P <0.0001 for comparison of the ordinal trend of graduation between residents receiving deficiencies in varying numbers of competencies, after controlling for site
P <0.0001 for comparison of the ordinal trend of ABA certification between residents receiving deficiencies in varying numbers of competencies, after controlling for site
Statistically significant after Bonferroni correction
ABA = American Board of Anesthesiology
ACGME = Accreditation Council for Graduate Medical Education
In order to adjust for multiple comparisons, Bonferroni correction was used. P values <0.0036 were considered statistically significant after the adjustment (alpha 0.05/14 tests = adjusted alpha of 0.0036).
Given that the follow-up period was shorter for subjects who entered late in the study period, we analyzed time from graduation to board certification, then performed a sensitivity analysis of the impact of the follow-up period on board certification rates.
RESULTS
Prevalence of Residents in Trouble
The total number of PGY-2 through PGY-4 residents at each site ranged from 127 to 275 over the study period with a total of 865 residents at all sites who participated in this study. Of the 865 residents in training at the four sites, 215 residents received a total of 405 CCC actions. Of these 215 residents, 118 residents received at least one Unsatisfactory designation on the ABA RTR/CCC Report. Across sites, the proportion of residents receiving CCC actions and Unsatisfactory designations varied considerably, from 13–51% for CCC actions and from 3–37% for Unsatisfactory designations.
Graduation and ABA Board Certification Rates
There was no significant difference in overall graduation rates among the four sites (97–98%, P =0.96, Fisher’s Exact Test, Table 2). The proportion of entering residents who achieved board certification varied from 96% to 98% (P =0.78, Fisher’s Exact Test) among the four residency programs.
When we compared residents who were the subject of any CCC action (n = 215) to residents who were not the subject of CCC actions (n = 650) we found a difference in their graduation rate (93% vs. 99%, respectively; P <0.001; OR 0.08, 95% CI [0.03–0.23], after controlling for site, Table 3). The board certification rate of residents who were subject to CCC action was also different from that of residents who did not receive CCC actions (89% vs. 99%, respectively; P <0.001; OR 0.04, 95% CI [0.02–0.13], after controlling for site). Among residents receiving CCC actions who ultimately graduated (n = 199), 191 achieved board certification.
When residents who received one or more Unsatisfactory rating(s) on an ABA RTR/CCC Report (n = 118) were compared to residents who received all Satisfactory ratings (n = 747), there was a difference in the graduation rate (88% vs. 99%, respectively; P <0.001, OR 0.03, 95% CI [0.01–0.10], after controlling for site) and board certification rate (85% vs. 99%, respectively, P <0.001, OR 0.04, 95% CI [0.01–0.09], after controlling for site, Table 4). Among residents receiving one or more Unsatisfactory ratings who graduated (n = 104), 100 achieved board certification.
Residents receiving CCC actions
The CCC might take one of several actions when confronted with a resident who was not meeting performance expectations. Of the 215 residents who received actions by the CCC, 204 received verbal counseling, 102 received written counseling, 118 received at least one grade of Unsatisfactory on the ABA RTR/CCC report and 24 were placed on probation according to the criteria of their institution.
The number of deficient ACGME Core Competencies varied among residents who were placed on probation. In descending order, the performance gaps were noted in Patient Care (n = 21), Professionalism (n = 19), Interpersonal and Communication Skills (n = 15), Medical Knowledge (n = 10), Practice-Based Learning (n = 8), Systems-Based Practice (n = 5). Of the 24 residents placed on probation, 16 residents graduated (66%); of these,15 residents went on to achieve board certification.
Residents with Unsatisfactory Designations on ABA RTR/CCC Reports
The overall graduation rate of residents receiving any Unsatisfactory designation was 88% (Table 4). Of the total cohort of 865 residents, 11 residents received a primary Unsatisfactory rating in an Essential Attribute (has no documented abuse of alcohol or illegal use of drugs (n = 4), learns from experience, knows limits (n = 2), demonstrates honesty, integrity, reliability, and responsibility (n = 2), reacts to stressful situations in an appropriate manner (n = 2), demonstrates high standards of ethical and moral behavior (n = 1), Table 5). Seven of these residents were placed on probation. Of the residents with Unsatisfactory ratings reported to the ABA, those who were lacking Essential Attributes were farless likely to graduate compared to those lacking in other categories (55% vs. 92% for those without a rating of Unsatisfactory in an Essential Attribute, P =0.0013, OR 0.09, 95%CI [0.02–0.37], after controlling for site).
Of the residents with Unsatisfactory ratings reported to the ABA, those who were lacking Essential Attributes were far less likely to obtain ABA certification than those lacking in other categories (45% vs. 89%, respectively; P = 0.0007; OR 0.08, 95% CI [0.02–0.34], after controlling for site). The overall board certification rate among residents receiving any Unsatisfactory designation on the ABA RTR/CCC Report was 85%.
Graduation by Number of Deficient ACGME Core Competencies
The CCC assessed the presence or absence of a deficiency in any ACGME Core Competency. Table 6 shows the number of residents by number of ACGME competencies deficient as well as the graduation and board certification rates for each group. For both outcomes (graduation and board certification), the rates decreased for residents with more deficiencies (p<0.0001, after controlling for site).
Frequency of Deficiencies in ACGME Core Competencies
Table 7 shows the number of residents across all four programs with deficiencies in an ACGME Core Competency. Medical Knowledge was the most commonly deficient competency (128 residents), followed by Patient Care (n = 78), Professionalism (n = 75), and Communication & Interpersonal Skills (n = 74). Practice-Based Learning & Improvement and Systems-Based Practice were the least commonly deficient competencies (n = 37 and n = 13, respectively).
Table 7.
Frequency of the Most Common ACGME Competency Deficiencies
| ACGME Competency | No. Deficient N = 865 (%) |
Graduation (% of row) |
ABA Certification (% of row) |
|---|---|---|---|
| Medical knowledge | 128 (15) | 117 (91) | 113 (88) |
| Patient care | 78 (9) | 65 (83) | 59 (76) |
| Professionalism | 75 (9) | 65 (87) | 63 (84) |
| Communication & interpersonal skills | 74 (9) | 65 (88) | 61 (82) |
| Practice-based learning/improvement | 37 (4) | 28 (76) | 25 (68) |
| Systems-based practice | 13 (2) | 7 (54) | 7 (54) |
ABA = American Board of Anesthesiology
ACGME = Accreditation Council for Graduate Medical Education
Time from Graduation to Board Certification
The median time from graduation to certification for residents in trouble was 15 months, interquartile range 10–20 months. All residents who ultimately became certified did so within five years. Two graduates who were not board certified were followed for less than five years. All statistically significant findings remained significant after a sensitivity analysis was done to determine whether a change in board certification status of these two residents would affect our findings.
Remediation
A variety of remediation methods were utilized at all four sites in order to support all residents in trouble. Each program provided a list of the remediation techniques used to address the various deficiencies. An aggregate list is presented in Table 8.
Table 8.
Remediation
| Remediation Tools | ACGME Competencies Targeted |
|---|---|
| Mentor meeting / Action plan development | All competencies |
| Learning assessment | Patient Care, Medical Knowledge |
| Board review sessions / Standardized tests (e.g., Anesthesia Knowledge Test) |
Patient Care, Medical Knowledge |
| Mandatory reading / lectures / conferences | Patient Care, Medical Knowledge |
| Limit away rotations and/or change rotations | All competencies |
| Additional 1:1 OR training | All competencies |
| Written case management worksheets with mentor follow-up |
Patient Care, Medical Knowledge, Practice-Based Learning & Improvement, Systems-Based Practice |
| Assignments with consistent faculty | All competencies |
| Assign different faculty advisor | All competencies |
| Extend training | All competencies |
| Repeated rotations | Patient Care, Medical Knowledge, Practice-Based Learning & Improvement |
| Simulation | All competencies |
| Leave of absence | Personal matters related to any competencies |
| Stress reduction / Emotional intelligence / Anger management |
Interpersonal & Communication Skills, Professionalism |
| Referrals to Psychiatry / Cultural sensitivity | Interpersonal & Communication Skills, Professionalism |
| Cognitive Behavioral Therapy | Interpersonal & Communication Skills, Professionalism |
| English as a Second Language | Interpersonal & Communication Skills |
| Physician Health / Rehabilitation / Drug diversion |
Patient Care, Interpersonal & Communication Skills, Professionalism |
| Withhold academic allowance | All competencies |
| Self-evaluation / 360° evaluation | All competencies |
This table represents a compilation of the various remediation tools used at the four residency training programs and the ACGME competencies targeted.
OR = operating room
DISCUSSION
Primary Findings and Significance
Deficiencies in multiple ACGME Core Competencies threaten graduation and board certification
Multiple deficiencies significantly decrease the likelihood of graduation and board certification, particularly when more than three ACGME Core Competency categories are involved. Roughly 3% of our study population was deficient in four or more categories. Residents with deficiencies in fewer categories are likely to graduate.
Essential Attributes are essential
Although deficiencies in an Essential Attributes were limited to a small group of residents (1.3% of the study population), the graduation and board certification rates of these residents were significantly decreased. An Unsatisfactory rating in an Essential Attribute represents a similar threat to graduation as deficiencies in multiple competency categories. Board certification rates drop below 50% for residents with deficiencies in an Essential Attribute. Deficits in essential attributes are known to be challenging to remediate.1 Substance abuse, which falls into the category of Essential Attribute deficiency, was found in 0.5% of the study population. In a large study that focused on substance abuse the graduation and certification rates were significantly decreased, and similar to the respective rates of residents with Essential Attribute deficiencies in this study.22
Graduation rates are high overall
Even residents in trouble are likely to graduate when deficiencies are isolated and do not involve Essential Attributes. Despite documented difficulties in performance during training, residents receiving CCC actions, with or without Unsatisfactory ratings, experienced a five percentage point drop in graduation rate compared to those without deficiencies.
Board Certification rates are high overall
Even residents in trouble are likely to become board certified when deficiencies are isolated and do not involve Essential Attributes. However, residents receiving CCC actions had lower board certification rates than graduation rates (89% vs. 93%, respectively).
The most common deficiency was in the ACGME category Medical Knowledge
A comparison of the frequency of deficiencies in ACGME Core Competencies shows that Medical Knowledge is by far the most common deficient competency, followed by, in approximately equal numbers, Patient Care, Professionalism, and Communication & Interpersonal Skills. This pattern may account for the overall high graduation and certification rates among our residents in trouble, as there are many readily available tools to supplement medical content knowledge.
The large variation in CCC actions and Unsatisfactory ratings between sites did not affect overall graduation or ABA certification rates
The proportion of residents who were referred to their respective CCCs varied considerably among the four residencies. We found a similar variation among the four sites in assigning Unsatisfactory ratings on ABA Training Reports. However, the programs did not differ from one another in overall graduation and board certification rates.
The variation in CCC referrals and Unsatisfactory ratings are related to the academic policy at Program D, which automatically assigns an Unsatisfactory in Medical Knowledge to residents who score below predetermined percentile levels on the In Training Examination. Using this algorithm results in a larger number of residents at Program D receiving Unsatisfactory ratings. Despite the differences between CCC practices at the various institutions, graduation and board certification rates are similar across all four programs.
Remediation Efforts
Remediation was multi-pronged and individualized to each resident, various combinations of rewards (educational stipends, eligibility for Chief Resident) and punitive measures (extended training time, Unsatisfactory training reports) were used to motivate residents to address deficiencies. A number of similar remediation efforts were employed among the four sites and were tailored to the competency in question. For instance, medical knowledge deficiencies were addressed with required reading, lecture attendance and use of additional standardized assessments (e.g., the Anesthesia Knowledge Test) while patient care deficiencies were addressed by directed clinical assignments. Faculty mentors initially addressed professionalism and communication lapses, while recurring problems frequently led to referrals to behavioral therapists. Generally, program directors and CCCs tried to identify deficiencies early, in order to initiate and escalate remediation as needed.
Comparison with Previous Research
The prevalence of residents in trouble in the present study (25%) falls within the range previously reported (6–26%), although definitions of “residents in trouble” varied.1,9,10,17 The most commonly deficient ACGME core competencies among our residents, Medical Knowledge, Patient Care, and Professionalism, were also consistent with previous findings.1,20 The graduation and board certification rates among the anesthesiology residents with performance deficits in our study (93% and 89%, respectively) were also comparable to those found in the 2006 study of family medicine residents (90% and 86%, respectively).1
Study Limitations
Our study was limited by the accuracy and completeness of the records maintained by the residency offices at the four sites. Nevertheless, we were fortunate to have low turnover of residency program personnel during the study period, facilitating data retrieval. Although all the residents included in this study were assessed by the Program Director and CCC Chair (or their designees) using a consensus approach to evaluate the comprehensive records at each site (CCC records, ABA training reports, individual resident files), the reduction of the rich narrative data on each resident to dichotomous variables in a database created the potential for oversimplification of the deficiencies.
Nearly half (44%) of residents with deficiencies in ACGME Core Competency categories had deficiencies in two or more categories, making it difficult to isolate the impact of a deficiency in a single competency on graduation or board certification. Despite the size of the cohort in this multicenter study, an assessment of interaction effects between deficiencies in two or more ACGME Core Competency categories was not feasible.
Our study may also suffer from selection bias. Although the residency programs in this study are large and geographically diverse they are all university-based and may not be representative of anesthesiology training programs as a whole.
Several residents who entered the study during the last several years of enrollment had a shorter follow-up period during which to become board certified. After a sensitivity analysis designed to see if our results would change if they were to become certified, P values and confidence intervals only changed slightly and overall conclusions were not affected.
This study was not designed to provide data on the effectiveness of remediation methods. Programs typically employ several remediation strategies simultaneously which limits assessments of individual remediation strategies.
The ACGME provides guidance to residency training programs regarding the structure and function of program CCCs.23,24 Nevertheless, every CCC operates according to internally developed policies and procedures. Milestones have now been created in an effort to create a standardized assessment of all residents across programs; however, the ACGME Milestones did not exist during the period of our study.
Conclusions
There was a wide variability in the proportion of residents receiving Clinical Competency Committee actions at the four programs. This difference between programs did not affect overall graduation rates; however, it does indicate a lack of standardization in CCC practices across programs. The present study indicates that specialty-wide policies and procedures for CCCs might assist training programs in making comparisons across programs. Standard operating procedures for all CCCs would promote internal and external consistency among programs, maintain efficiency and quality control, and provide transparency. Organizational or regulatory bodies may be helpful in establishing best practices for CCCs.
The graduation and board certification rates we have reported may serve as a benchmark for other anesthesiology programs. Graduation and board certification rates were consistently high across the four programs. There was a small reduction in graduation and board certification rates for residents receiving Clinical Competency Committee actions for ACGME core competencies; nevertheless, these residents were still more likely than not to graduate in good standing and achieve board certification, perhaps due to successful remediation strategies by the training programs. Notably, performance problems in Essential Attributes carry a poor prognosis and suggest that remediation efforts may not be satisfactory or that it may not be possible to remediate these problems in anesthesiology training. Another risk factor for failure to graduate or to achieve board certification is the presence of deficiencies in multiple ACGME Core Competency categories. Although it is beyond the scope of our data to provide an analysis of the usefulness of specific remediation techniques, it would be helpful if future work in this area examined which remediation methods are most efficient and effective in correcting particular deficiencies.
Acknowledgments
Funding Source: This study was supported by the authors’ institutions: Department of Anesthesiology and Perioperative Medicine, David Geffen School of Medicine at UCLA, Los Angeles, California. Harvard Medical School, Department of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital, Boston, Massachusetts. Department of Anesthesia and Perioperative Care, University of California San Francisco, San Francisco, California; Department of Anesthesiology, University of Colorado School of Medicine, Denver, Colorado. Statistical analyses described were partially supported by NIH /National Center for Advancing Translational Science (NCATS) UCLA CTSI Grant Number UL1TR000124.
Special thanks to Erin Lacey, MS, and Maria DellaRocco at Massachusetts General Hospital, Boston, Massachusetts; Allison Glover, BA, at University of Colorado School of Medicine, Denver, Colorado for data collection and coordination.
Footnotes
Presentations: None.
Disclosures: The authors declare no competing interests.
Contributor Information
Judi A. Turner, Email: jaturner@mednet.ucla.edu, Department of Anesthesiology and Perioperative Medicine, David Geffen School of Medicine at University of California Los Angeles (UCLA), Los Angeles, California.
Michael G. Fitzsimons, Email: mfitzsimons@partners.org, Harvard Medical School, Department of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital, Boston, Massachusetts.
Manuel C. Pardo, Jr, Email: pardom@anesthesia.ucsf.edu, Department of Anesthesia and Perioperative Care, University of California San Francisco, San Francisco, California.
Joy L. Hawkins, Email: joy.hawkins@ucdenver.edu, Department of Anesthesiology, University of Colorado School of Medicine, Denver, Colorado.
Yue Ming Huang, Email: yhuang@mednet.ucla.edu, Department of Anesthesiology and Perioperative Medicine, David Geffen School of Medicine at UCLA, Los Angeles, California.
Maria D. D. Rudolph, Email: mddr81@ca.rr.com, UCLA Simulation Center, Los Angeles, California.
Mary A. Keyes, Email: mkeyes@mednet.ucla.edu, Department of Anesthesiology and Perioperative Medicine, David Geffen School of Medicine at UCLA, Los Angeles, California.
Kimberly J. Howard-Quijano, Email: khquijano@mednet.ucla.edu, Department of Anesthesiology and Perioperative Medicine, David Geffen School of Medicine at UCLA, Los Angeles, California.
Natale Z. Naim, Email: nnaim@mednet.ucla.edu, Department of Anesthesiology and Perioperative Medicine, David Geffen School of Medicine at UCLA, Los Angeles, California.
Jack C. Buckley, Email: jcbuckley@mednet.ucla.edu, Department of Anesthesiology and Perioperative Medicine, David Geffen School of Medicine at UCLA, Los Angeles, California.
Tristan R. Grogan, Email: tgrogan@mednet.ucla.edu, Department of Medicine Statistics Core, David Geffen School of Medicine at UCLA, Los Angeles, California.
Randolph H. Steadman, Email: rsteadman@mednet.ucla.edu, Department of Anesthesiology and Perioperative Medicine, David Geffen School of Medicine at UCLA, Los Angeles, California.
REFERENCES
- 1.Reamy BV, Harman JH. Residents in trouble: an in-depth assessment of the 25-year experience of a single family medicine residency. Fam Med. 2006;38:252–257. [PubMed] [Google Scholar]
- 2.Adams KE, Emmons S, Romm J. How resident unprofessional behavior is identified and managed: a program director survey. Am J Obstet Gynecol. 2008;198:692, e1–e4. doi: 10.1016/j.ajog.2008.03.023. discussion 692 e4–5. [DOI] [PubMed] [Google Scholar]
- 3.Bernstein S, Atkinson AR, Martimianakis MA. Diagnosing the learner in difficulty. Pediatrics. 2013;132:210–212. doi: 10.1542/peds.2013-1526. [DOI] [PubMed] [Google Scholar]
- 4.Domen RE. Resident remediation, probation, and dismissal basic considerations for program directors. Am J Clin Pathol. 2014;141:784–790. doi: 10.1309/AJCPSNPAP5R5NHUS. [DOI] [PubMed] [Google Scholar]
- 5.Guerrasio J, Aagaard EM. Methods and outcomes for the remediation of clinical reasoning. J Gen Intern Med. 2014;29:1607–1614. doi: 10.1007/s11606-014-2955-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Guerrasio J, Garrity MJ, Aagaard EM. Learner deficits and academic outcomes of medical students, residents, fellows, and attending physicians referred to a remediation program, 2006–2012. Acad Med. 2014;89:352–358. doi: 10.1097/ACM.0000000000000122. [DOI] [PubMed] [Google Scholar]
- 7.Katz ED, Dahms R, Sadosty AT, Stahmer SA, Goyal D. Guiding principles for resident remediation: recommendations of the CORD remediation task force. Acad Emerg Med. 2010;17(Suppl 2):S95–S103. doi: 10.1111/j.1553-2712.2010.00881.x. [DOI] [PubMed] [Google Scholar]
- 8.Ratan RB, Pica AG, Berkowitz RL. A model for instituting a comprehensive program of remediation for at-risk residents. Obstet Gynecol. 2008;112:1155–1159. doi: 10.1097/AOG.0b013e31818a6d61. [DOI] [PubMed] [Google Scholar]
- 9.Resnick AS, Mullen JL, Kaiser LR, Morris JB. Patterns and predictions of resident misbehavior--a 10-year retrospective look. Curr Surg. 2006;63:418–425. doi: 10.1016/j.cursur.2006.05.004. [DOI] [PubMed] [Google Scholar]
- 10.Roback HB, Crowder MK. Psychiatric resident dismissal: a national survey of training programs. Am J Psychiatry. 1989;146:96–98. doi: 10.1176/ajp.146.1.96. [DOI] [PubMed] [Google Scholar]
- 11.Rosenblatt MA, Schartel SA. Evaluation, feedback, and remediation in anesthesiology residency training: a survey of 124 United States programs. J Clin Anesth. 1999;11:519–527. doi: 10.1016/s0952-8180(99)00084-7. [DOI] [PubMed] [Google Scholar]
- 12.Steinert Y. The "problem" learner: whose problem is it? AMEE Guide No. 76. Med Teach. 2013;35:e1035–e1045. doi: 10.3109/0142159X.2013.774082. [DOI] [PubMed] [Google Scholar]
- 13.Stirling K, Hogg G, Ker J, Anderson F, Hanslip J, Byrne D. Using simulation to support doctors in difficulty. Clin Teach. 2012;9:285–289. doi: 10.1111/j.1743-498X.2012.00541.x. [DOI] [PubMed] [Google Scholar]
- 14.Torbeck L, Canal DF. Remediation practices for surgery residents. Am J Surg. 2009;197:397–402. doi: 10.1016/j.amjsurg.2008.10.008. [DOI] [PubMed] [Google Scholar]
- 15.Weizberg M, Smith JL, Murano T, Silverberg M, Santen SA. What does remediation and probation status mean? A survey of emergency medicine residency program directors. Acad Emerg Med. 2015;22:113–116. doi: 10.1111/acem.12559. [DOI] [PubMed] [Google Scholar]
- 16.Williams RG, Roberts NK, Schwind CJ, Dunnington GL. The nature of general surgery resident performance problems. Surgery. 2009;145:651–658. doi: 10.1016/j.surg.2009.01.019. [DOI] [PubMed] [Google Scholar]
- 17.Yao DC, Wright SM. National survey of internal medicine residency program directors regarding problem residents. JAMA. 2000;284:1099–1104. doi: 10.1001/jama.284.9.1099. [DOI] [PubMed] [Google Scholar]
- 18.Yao DC, Wright SM. The challenge of problem residents. J Gen Intern Med. 2001;16:486–492. doi: 10.1046/j.1525-1497.2001.016007486.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Zbieranowski I, Takahashi SG, Verma S, Spadafora SM. Remediation of residents in difficulty: a retrospective 10-year review of the experience of a postgraduate board of examiners. Acad Med. 2013;88:111–116. doi: 10.1097/ACM.0b013e3182764cb6. [DOI] [PubMed] [Google Scholar]
- 20.Dupras DM, Edson RS, Halvorsen AJ, Hopkins RH, Jr, McDonald FS. "Problem residents": prevalence, problems and remediation in the era of core competencies. Am J Med. 2012;125:421–425. doi: 10.1016/j.amjmed.2011.12.008. [DOI] [PubMed] [Google Scholar]
- 21.Bergen PC, Littlefield JH, O'Keefe GE, Rege RV, Anthony TA, Kim LT, Turnage RH. Identification of high-risk residents. J Surg Res. 2000;92:239–244. doi: 10.1006/jsre.2000.5924. [DOI] [PubMed] [Google Scholar]
- 22.Warner DO, Berge K, Sun H, Harman A, Hanson A, Schroeder DR. Substance use disorder among anesthesiology residents, 1975–2009. JAMA. 2013;310(21):2289–2296. doi: 10.1001/jama.2013.281954. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.ACGME: ACGME Program Requirements for Graduate Medical Education in Anesthesiology, Accreditation Council for Graduate Medical Education. [Revised September 29, 2013. Accessed November 2, 2015]; at http://www.acgme.org/acgmeweb/portals/0/pfassets/programrequirements/040_anesthesiology_07012014.pdf.
- 24.Andolsek K, Padmore J, Hauer KE, Holmboe E. Clinical Competency Committees: A Guidebook for Programs, Accreditation Council for Graduate Medical Education, 2015. [Accessed November 2, 2015]; at https://www.acgme.org/acgmeweb/Portals/0/ACGMEClinicalCompetencyCommitteeGuidebook.pdf.