The Utility of Letters of Recommendation in Predicting Resident Success: Can the ACGME Competencies Help?

Abstract

Background

The Accreditation Council for Graduate Medical Education (ACGME) core competencies are used to assess resident performance, and recently similar competencies have become an accepted framework for evaluating medical student achievements as well. However, the utility of incorporating the competencies into the resident application has not yet been assessed.

Purpose

The objective of this study was to examine letters of recommendation (LORs) to identify ACGME competency–based themes that might help distinguish the least successful from the most successful residents.

Methods

Residents entering a university-based residency program from 1994 to 2004 were retrospectively evaluated by faculty and ranked in 4 groups according to perceived level of success. Applications from residents in the highest and lowest groups were abstracted. LORs were qualitatively reviewed and analyzed for 9 themes (6 ACGME core competencies and 3 additional performance measures). The mean number of times each theme was mentioned was calculated for each student. Groups were compared using the χ² test and the Student t test.

Results

Seventy-five residents were eligible for analysis, and 29 residents were ranked in the highest and lowest groups. Baseline demographics and number of LORs did not differ between the two groups. Successful residents had statistically significantly more comments about excellence in the competency areas of patient care, medical knowledge, and interpersonal and communication skills.

Conclusion

LORs can provide useful clues to differentiate between students who are likely to become the least versus the most successful residency program graduates. Greater usage of the ACGME core competencies within LORs may be beneficial.

Background

The screening and selection of applicants into residency programs is an inexact science. Applications include information about candidates' performance in medical school and on standardized examinations, a personal statement, and letters of recommendation (LORs). Numerous studies have evaluated the utility of objective metrics (US Medical Licensing Examination scores, medical school transcript, etc) in predicting an applicant's success and have produced mixed results.^1–6 Although LORs are consistently considered important in the resident selection process,^7–9 their predictive value is not conclusive.

The Accreditation Council for Graduate Medical Education (ACGME) core competencies have been used in assessing resident performance¹⁰ but are not a standardized component of the residency application. Recently, the Liaison Committee on Medical Education endorsed the use of a similar competencies format for the evaluation of medical students.¹¹ The goal of this study is to investigate whether references to performance measures associated with the ACGME competencies within LORs are useful in identifying candidates most likely to succeed in residency education in obstetrics and gynecology.

Methods

We performed a retrospective cohort study of residents entering the Johns Hopkins University residency program in obstetrics and gynecology through the National Residency Match Program between 1994 and 2004. This study was approved by the Institutional Review Board of Johns Hopkins Medical Institutions.

Residents meeting inclusion criteria were independently ranked by the residency program director and the department chair. The evaluators assigned each resident a numeric value between 1 (lowest) and 4 (highest) based on each evaluator's independent assessment of the resident's overall performance. Specifically, the evaluator was asked to respond to the question: “Knowing what you now know about this resident after his/her four years of training, would you select him or her again for admission into the residency program?” Each evaluator was blinded to the other evaluator's ratings as well as to the resident's original residency application. Evaluators were permitted to review the residents' performance folders, which include semiannual faculty and peer evaluations.

A response of 1 indicated minimal or no desire to “reselect” a resident, a response of 4 indicated a strong interest in readmitting a resident, and the intermediate levels encompassed scores of 2 (some interest) and 3 (much interest). Residents who did not complete the 4-year training program because of premature elective or mandated termination were automatically assigned a numeric value of 1. A mean score was calculated for each resident. A resident with a score ≤1.5 was assigned to the lowest group; a score >1.5 and ≤2.5 to the second-lowest group; a score between 2.5 and 3.5 to the second-highest group; and a score ≥3.5 to the highest group. Once ranked, all resident identities were masked to preserve confidentiality.

Deidentified applications of residents ranked in the highest and lowest groups were reviewed. LORs were independently abstracted by 2 experienced reviewers (N.A.H. and J.L.B.) and analyzed for themes based on the ACGME core competencies (patient care [PC], medical knowledge [MK], practice-based learning [PBL], interpersonal and communication skills [ICS], professionalism, and systems-based practice [SBP]) and 3 additional characteristics (research skills, leadership skills, and motivation) deemed important in defining a successful resident. table 1 illustrates examples of sentences in the LORs that pertain to each theme.

Table 1.

Sample Comments From Letters of Recommendation in Residency Applications for Each of the Accreditation Council for Graduate Medical Education Core Competencies

The mean number of times each theme was mentioned was calculated for each applicant. Results were compared between residents in the highest group (very successful residents) and residents in the lowest group (unsuccessful residents). Data were analyzed using the Fisher exact test and the Student t test. A P value of <.05 was considered statistically significant.

Results

During the 10-year study period, the program “matched” 7 to 8 residents per academic year. Seven residents left the program electively (to pursue residencies in other specialties) or upon request (because of poor clinical performance). These residents were automatically ranked in the lowest group. The residents who filled these vacated positions were excluded from analysis because they did not enter the residency program through the match. A total of 75 residents met inclusion criteria. Application data were available for all study participants.

There was substantial agreement between the 2 evaluators regarding resident performance. Approximately 50% of the residents in the groups analyzed received the same score from both evaluators. The remainder of the residents were assigned scores that differed by a value of 1 (eg, one score of 3 and one score of 4, providing a mean of 3.5).

Fourteen residents were ranked into the highest group and 15 into the lowest group. Sixty-five percent of the residents were women. Three international medical graduates were included in our analysis, with no difference between the highest and lowest groups.

A total of 110 LORs (median of 4 per resident) were reviewed. All letters referenced at least one of the core competencies. There was no temporal difference in the number of references in LORs between the beginning and the end of the study time period. The competencies of ICS and MK were most commonly noted, followed by PBL and professionalism, PC, and SBP. Of the 3 noncompetencies analyzed, motivation was most commonly cited. None of the letters recommended against accepting the applicant or contained overtly negative information.

Of 9 themes evaluated, differences in 3 themes (PC, MK, and ICS) were statistically significant between residents in the highest versus the lowest groups (table 2). For each of these themes, the residents in the highest group had almost double the number of references to the respective ACGME competency than did their counterparts in the lowest group.

Table 2.

Mean Number of References in the Letters of Recommendation for Each Theme Evaluated

Discussion

The goal of a residency training program is to produce successful residents and outstanding clinicians. Unfortunately, the process of resident selection is inexact because of the limitations of quantifiable metrics in predicting resident success.2,3,12,13 Our study demonstrates that LORs written for medical students applying to residency programs in obstetrics and gynecology may be useful in predicting resident performance at the extremes.

To our knowledge, only one published study has demonstrated that the core competencies are referenced in LORs written for applicants to residency programs in obstetrics and gynecology, finding that 93% of 70 LORs contained at least one reference to one of the competencies, and 50% referenced 3 or more.¹⁴ This study did not analyze the predictive ability of LORs in the resident selection process.

References to 3 competencies (PC, MK, and ICS) were found to be significantly different between the groups in our study. There were almost twice as many references to these 3 competencies in LORs for residents in the highest group than in LORs for residents in the lowest group. These results are consistent with the ideal goal of a residency training program, namely, the production of a competent (aka “successful”) clinician. Similar results were noted by Greenburg et al¹⁵ in an analysis of LORs for a surgical residency program. Although specific references to the core competencies were not investigated, Greenburg and colleagues note that references to “fund of knowledge” and “work habits” correlated with a high score for the LOR.

The 3 competencies that did not achieve statistical significance were SBP, PBL, and professionalism. Until recently, the former 2 have not been a common part of the lexicon that defines medical student education. This may explain why writers of LORs for medical students infrequently address these skills. For example, the competency of SBP was referenced in letters for only 15 residents compared with MK, which was mentioned in letters for every resident.

There is no uniformly accepted means of measuring a resident's overall performance or “success.” In our study, we defined “success” based on the independent rankings of 2 experienced and respected resident educators. There are myriad ways of defining success, such as fellowship matching, continuation in academic medicine, and passage of licensing boards. We preferred our definition because it allowed us to best investigate whether we can reliably identify which applicants will become residency graduates from whom we receive the greatest return on our educational investment.

The primary limitations of our study include the relatively small cohort and the fact that our study was conducted at a single university-based program. It is possible that our outcomes are not generalizable to residents in other programs. We do not believe that including residents who were prematurely terminated from our program is a study limitation. Although these residents may be considered a separate cohort, we believe that they make up an important element of a “success failure.” Residents who do not complete their training program lose personal time and introduce uncertainty and anxiety within the educational program.

With increasing emphasis on competency-based learning in medical education, it seems prudent to move toward competency-based assessments and recommendations. Not all program directors are interested in going the way of emergency medicine and adopting a formal, standardized LOR.¹⁶ We suggest that incorporating the competencies more consistently into LORs would allow for a common language and structure without mandating a complete reformation of current conventions.