Standardized Letters of Recommendation and Successful Match into Otolaryngology

Abstract

Objective

Historically, narrative letters of recommendation have been utilized in selection of applicants for Otolaryngology residency programs. In the last two application cycles, our specialty adopted a standardized letter of recommendation (SLOR). The intent was to decrease time burden for letter writers and to provide readers with an “objective” evaluation of applicants. The objective of this study is to determine attributes in the SLOR that correlate with matching into a residency program.

Study Design

We performed a retrospective study using SLOR, USMLE Step 1 scores and match outcomes of applicants who applied to our institution for the 2013 and 2014 match cycle.

Methods

We included the following variables from the SLOR in the statistical analysis to determine which ones were associated with matching: patient care, medical knowledge, communication skills, procedural skills, research, initiative and drive, commitment to Otolaryngology, commitment to academic medicine, match potential and USMLE1 scores.

Results

We identified 532 applicants and 963 SLOR. In successful applicants, scores for patient care, medical knowledge, communication skills, initiative and drive, and match potential were statistically higher (p < 0.05). Scores for professionalism, procedural skills, research, commitment to Otolaryngology, commitment to academic medicine and USMLE Step 1 scores were not higher among successfully matched applicants.

Conclusions

While SLOR can save time for letter writers and provide an objective description of applicants, the utility of individual domains within the SLOR is questionable. Additionally, it is concerning that applicants’ professionalism and procedural skills are not correlated with matching in our specialty.

Level of Evidence

NA

INTRODUCTION

Entry and matriculation through medical school serve as a gateway to become physicians; however, each specialty further selects their future colleagues through a process of applications, interviews and the match. While some training programs may rank applicants based on algorithms, the majority of programs select potential residents using poorly defined subjective measures and group discussions.¹

Applicants for Otolaryngology Head and Neck Surgery represent a highly competitive pool of applicants. In the 2014 there were 1.4 applicants for each of the 295 intern positions. The majority of our applicants have 5 or more abstracts, presentation and publications. Nearly 20% of applicants who do not match into Otolaryngology are Alpha Omega Alpha honor society members. USMLE Step 1 examination scores for our applicants are extraordinary, with an average of 248 among matching US seniors.² Given the highly competitive applicant pool, limited time for each interview and the impossibility of interviewing all applicants selecting residents is a difficult task.³

Letters of recommendations serve as an endorsement from an individual in our field. This is an endorsement from a colleague who has known and worked with the applicant. Given that interpersonal skills and clinical judgment are not directly reflected by AOA status, honors in clerkships or USMLE scores, letters of recommendation serve as a unique tool to assess these characteristics. Despite this, when objectively reviewed many letters of recommendation lack crucial information.⁴

Historically, letters of recommendation for residency have been narrative letters of recommendation.⁵ Typically applicants submit three letters of recommendations from various mentors, chairs or faculty members with whom they have had a positive experience. Each narrative letter establishes three main elements. First, they establish the relationship between the writer and the applicant. Second, the letter reiterates the the applicants performance and third, the subject is compared to his/her peers.⁵ Assessments of narrative letters of recommendations demonstrate that many are not “clear and easy to read” and narrative letters have high inter-reader variability.⁶ Additionally, intentional selection bias of selecting letter writers and the desire of letter writers to preserve institutional reputation, create conflicting goals that make it difficult to discern a strong applicant from average or weak applicant^6,7 Furthermore, the same letter does not convey the same information to different readers. In quantitative studies, different readers have been shown to have “slight to poor” agreement when reading the same letter.⁸ Because of the shortcoming’s in narrative letters of recommendation, as well as the amount of time and energy that must be spent reading and writing the letters, Emergency Medicine residency programs implemented a “standardized letter of recommendation (SLOR) in 1995.⁹ Implementation of a SLOR in Emergency Medicine demonstrated dramatic improvement in inter-rater reliability as well as a decreased time to interpret the letter from 90 seconds to 16 seconds per letter.¹⁰

In 2011 a SLOR was developed for Pediatric Otolaryngology Fellowships¹¹ and in 2012 a SLOR was implemented by the Otolaryngology Program Directors’ Organization.¹² The SLOR was broken down into 5 components, background and experience of letter writer, comparison of applicant to peers, qualification for Otolaryngology, global assessment and a summary statement.¹²

In a follow up report after implementation of the SLOR, Program Directors and Chairs reported that the SLOR saved time, allowed easier comparison and was superior in conveying relevant information.¹³ The objective of this study is to determine attributes in the SLOR that correlate with matching into a residency program.

MATERIALS AND METHODS

Institution approval from the Registrar of the University of North Carolina Chapel Hill and exemption from the Institutional Review Board was obtained. All applicants, whose match status was known and whom submitted at least one SLOR to the University of North Carolina Chapel Hill for the 2013 or 2014 match were included. Data was extracted to a de-identified database for statistical analysis using GraphPad Prism 5.0 and STATA 10.

A percentile was generated from the letter writer’s markings. A marking over “top” was considered 100% and Bottom was 0% (Figure 1). Intermediate marks were determined using the ruler function in Adobe Acrobat and dividing by the length of the line. If a letter writer marked “cannot assess” the field was left blank. A “letter of recommendation average” was computed for each letter. This was the computed by summing each individual component that was completed and dividing by the attributes that were evaluated by the letter writer. Additionally, the USMLE Step 1 score for was obtained from the application packet.

Figure 1. Standard grading scale for standardized letter or recommendation.

Each attribute within the standard letter or recommendation has a line marked “top” at the left and “bottom” at the right and “middle” in the center. The letter writer marks an X where the individual being evaluated fall for each attribute in relationship to all other medical students. To calculate a percentile the distance from the right is measured (l_applicant) as is the length of the overall line (l_possible). These values were used to determine the overall percentile using the equation 100 * l_applicant/l_possible.

Logistic regression was performed to identify the most important variable in successfully matching into an Otolaryngology program. Marginal Wald-type chi-square statistics were reported instead of joint analysis in a single model due to strong collinearity between attributes.

RESULTS

Overall 963 SLOR’s were identified and used in the study from 532 applicants. There were 920 SLOR from applicants who successfully matched and 43 SLOR from unsuccessful applicants. No SLOR’s were excluded. The variables included in the letters of recommendation and analysis include: patient care, medical knowledge, professionalism, interpersonal & communication skills, procedural skills, research, initiative & drive, commitment to academic medicine and match potential (Table 1). Additionally, the average of the quantitative portion of the SLOR was calculated and the USMLE Step 1 score was included in the analysis. Of these attributes, higher scores in patient care, medical knowledge, interpersonal & communication skills, initiative & drive, match potential and the overall average of the SLOR were statistically significant (p < 0.05) for matching (Table 1, black; Figure 2). Conversely, higher score in professionalism, procedural skills, research, commitment to otolaryngology and commitment to academic medicine were not statistically significant (p > 0.05) for successfully matching (Table 1, red; Figure 3).

Table 1.

Attribute and Definition

Qualifications for OHNS	P
Patient Care – Ability to develop and justify an appropriate differential diagnosis and a cohesive treatment plan.	0.01
Medical Knowledge – Level of general and otolaryngologic medical knowledge.	0.03
Professionalism – Quality of work ethic, altruism, professional appearance and willingness to assume responsibility	0.16
Interpersonal and Communication Skills – Ability to interact with others on the health care team and communicate in an effective and caring manner with patients	0.03
Procedural Skills – Ability to perform surgical tasks.	0.41
Research – Ability to identify a question and to formulate and execute a cogent research plan.	0.14
Initiative and Drive – Ability to stay oriented to a goal and see tasks to completion.	0.03
Global Assessment
Commitment to Otolaryngology—Head & Neck Surgery – Thoughtfulness in choosing his or her career path compared to other medical students you know.	0.07
Commitment to Academic Medicine – Likelihood of pursuing a research/academic career after residency	0.54
Match Potential – Likelihood of the candidate matching in otolaryngology.	0.02
Items Not on Survey
USMLE 1	0.14
Average Letter of Recommendation Score	0.03

Red text indicates non-significant attributes

Figure 2. Higher scores in patient care, medical knowledge, interpersonal & communication skills, initiative & drive, match potential and the overall average were associated with matching.

Applicants who did not match (green) and who did match (orange) are compared for each of the above attributes. P-values are indicated.

Figure 3. Higher score in professionalism, procedural skills, research, commitment to otolaryngology, and commitment to academic medicine were not statistically significant for successfully matching.

Applicants who did not match (green) and who did match (orange) are compared for each of the above attributes. P-values are indicated.

Non-linear regression was performed to identify the most important variable in successfully matching into an Otolaryngology program; however the correlation between variables was such that the analysis could not detect significant variables.

USMLE Step 1 scores were not statistically higher in successful applicants. The average USMLE Step 1 score was 243 for successful applicants versus 238 for non-successful applicants (p = 0.14, Figure 4). To assess if attributes in the SLOR were reiterating the USMLE Step 1 score in a different context, the USMLE Step 1 score was plotted against the individual score for each component of the SLOR. Linear regression was performed and the slope and 95% confidence interval were calculated (Table 2). Medical knowledge, procedural skills, and commitment to academic medicine were all positively associated with USMLE Step 1 score, with 95% confidence intervals not covering zero. The remainder of the attributes within the SLOR had no relationship with USMLE Step 1 scores.

Figure 4. USMLE Step 1 scores are not significantly higher for matching applicants.

Applicants who did not match (green) had an average score of 243 compared to 238 for non-matching applicants (orange; p = 0.14)

Table 2.

USMLE 1 Score correlation with components of the standardized letter of recommendation.

Attribute	Slope	95% C.I.
Patient Care	0.075	−0.009 to 0.160
Medical Knowledge	0.188	0.106 to 0.270
Professionalism	0.005	−0.060 to 0.071
Communication Skills	−0.008	−0.104 to 0.086
Procedural Skills	0.096	0.002 to 0.190
Research	−0.023	−0.141 to 0.095
Initiative & Drive	0.032	−0.036 to 0.100
Commitment to Otolaryngology	0.045	−0.019 to 0.109
Commitment to Academic Medicine	0.124	0.001 to 0.247
Match Potential	0.201	0.086 to 0.316
Average	0.072	0.007 to 0.138
USMLE 1	1.000	n/a

Red text indicates the 95% confidence interval of the slope crosses zero. Blue text indicates the 95% confidence interval of the slope is greater than zero.

Based on a Marginal Wald-type chi-square, patient care was considered as the most important factor with the largest Wald-type chi-square statistic (Wald=5.89, p-value=0.015; Table 3). Initiative & drive has a similar impact on successful match with Wald-type chi-square statistic (Wald=4.92, p-value=0.027; Table 3).

Table 3.

Marginal Wald-type chi-square analysis of variables are that correlated with a successful match.

Attribute	Wald Chi Square	Sig
Patient Care	5.873	0.015
Medical Knowledge	4.023	0.045
Communication Skills	4.104	0.043
Initiative & Drive	4.921	0.027
Match Potential	3.808	0.051

DISCUSSION & CONCLUSIONS

Standardized letters of recommendation have eased the efficiency of both reading and writing letters of recommendation and have provided a framework that is more analytical for evaluating and comparing applicants. The instructions for letter writers on the SLOR indicate they should “compare the applicant to other medical students (including those entering other specialties) by indicating with an “X” on the scales below where you rate him or her” (Figure 1). Since the instructions indicate that the applicants should be compared to all medical students (not just Otolaryngology applicants), it is expected that applicants for Otolaryngology would be above average and indeed this is true. The average score SLOR was over 80%. Specifically, the overall average was 85.6% and 83.3% for matching and non-matching applicants respectively (p = 0.03; Figure 2F). Of all responses on the 964 SLOR that were reviewed, 0.3% had scores that were below 50%. Like the fictional town of Lake Wobegon, where all the children are all above average,¹⁴ nearly all of our applicants are above average according to the letter writers. The dynamic range of these metrics could be improved if the letter writer were instructed and accurately compared the applicant to their fellow Otolaryngology applicants, which should move the average towards 50%.

Our analysis demonstrates that scores in the domain of patient care, medical knowledge, interpersonal & communication skills, initiative & drive and match potential are significantly higher (p < 0.05) for applicants who match (Table 1, Figure 2). Despite these statistically significant differences, the outliers on both sides make the utility of an individual score in predicting match success or to identify a competitive candidate difficult at best. Of the 963 SLOR, the top five overall scores ranged from 98.1 to 100. One of the applicants with the top scoring SLOR did not match into Otolaryngology. Conversely, the lowest five scores ranged from 46.4 to 50.6 and four of these individuals successfully matched into Otolaryngology. This indicates that even while the overall SLOR is correlated with matching into Otolaryngology residency it alone is not predictive of match status.

The attributes on the SLOR that are not correlated with successfully matching are perhaps the most interesting aspect of this study (Table 1, Red; Figure 3). These attributes were professionalism, procedural skills, research, commitment to Otolaryngology and commitment to academic medicine (Figure 3).

“Commitment to Otolaryngology” on the SLOR is defined as “thoughtfulness in choosing his or her career path compared to other medical students you know” and was not correlated (p = 0.07) with matching into our field (Figure 3D). It would be interesting to see if individuals who scored low in the commitment to Otolaryngology had high attrition rates from clinical practice in Otolaryngology. More concerning perhaps, is that neither professionalism nor procedural skill are correlated with matching into our surgical specialty.

Otolaryngology is a surgical specialty that requires a high degree of precision and manual dexterity. It is alarming the applicant’s “procedural skills - ability to perform surgical tasks” as judged by our colleagues who have operated with these applicants does not significantly influence the ability of an applicant to match (p = 0.41, Figure 3B). Testing of surgical skill aptitude at interviews has an unclear benefit in selecting residents who can be trained to become skilled surgeons^15,16 and our data suggests we do not select future residents based on our colleagues endorsement of an applicants technical abilities either. Surgical specialties and even applicants would benefit from a predictive instrument to identify individuals who may not be fit to become a surgeon.

Another concerning finding in our study was the lack of correlation between matching and professionalism. The SLOR defines professionalism as “quality of work ethic, altruism, professional appearance and willingness to assume responsibility.” These attributes, while admirable, were not correlated with matching (p = 0.16, Figure 3A). If the assessment of the letter writers is correct, it is concerning that we, as a specialty, are selecting the next generation of Otolaryngologist with questionable work ethic, willingness to assume responsibility and procedural skills.

Neither research nor commitment to academic medicine as evaluated on the SLOR are correlated with successfully matching (Figure 3C, E). It is well known that some programs strive to train the next generation of academic Otolaryngologist while other programs have no qualms about training Otolaryngologists for the community, so this finding is not unexpected. It can be argued the desire to stay in academic medicine is more likely influenced by the environment of the residency training program than any experience prior to residency.

Despite successful applicants having more abstracts, publications and research experiences based on the NRMP data,² there was no correlation between matching and the SLOR research domain. Research on the SLOR is defined as the “ability to identify a question and to formulate and execute a cogent research plan.” While this is a lofty definition, unless a letter writer has worked on research projects with the applicant it would be very difficult to assess. Overall 28% of SLOR’s reviewed did not include a score for research while most other categories had completion rates greater than 95%. Perhaps quantifying an applicant’s research experience is better suited to their abstracts, publications and research experience.

Letters of recommendation in general, are meant to provide residency programs information about applicants that cannot be directly inferred from other components of the application. To determine if the SLOR is providing information that is independent of the USMLE Step 1 score, scores were plotted against the individual components of the SLOR and linear regression was performed to determine slope and 95% confidence interval. A slope of zero or 95% confidence interval that cross zero, indicates components of the SLOR are independent of USMLE Step 1 score. Medical knowledge and match potential were most directly associated with USMLE Step 1 score (Table 2). It is interesting to note that the other two variables that were found to be associated with USMLE 1 performance were commitment to academic medicine and procedural skills (Table 2). The remaining aspects of the SLOR were independent of the USMLE Step 1 score (Table 2), indicating that the letter does provide unique information, and is not just a reformulation of an applicants USMLE Step 1 score.

The standardized letter of recommendation has increased efficiency for letter writers and readers. While the attributes within the SLOR have high collinearity, this is logical in that an excellent applicant would be expected to excel in the majority of traits that are assessed in a letter or recommendation. Matching is a complex process that is dependent on multiple variables, some of which cannot be objectively measured. Interactions and impression during an interview are critical components of an application that are not captured in this type of study. An additional weakness of this study is the low number of SLOR for applicants who did not match. Of the 963 SLOR only 43 of them were from applicants who did not match.

Based on these results we need to continue to refine our methods, especially in identify applicants who demonstrate a high level of professionalism and technical skill.