CORR Insights®: Are Narrative Letters of Recommendation for Medical Students Interpreted as Intended by Orthopaedic Surgery Residency Programs?

Where Are We Now?

Historically, the acquisition of surgical skills in North America has been a highly variable and unstructured affair. Formal and comprehensive postgraduate education was uncommon in North America at the turn of the 20^th century. Instead, surgical education was largely limited to a combination of individual apprenticeships, short crash courses, and self-directed learning, whereby general practitioners who self-identified as surgeons operated on patients with minimal formal training [2].

By the middle of the 20^th century, surgical training programs in the United States and Canada had become relatively standardized through the proliferation of hospital-based residency programs [2]. This was further supported by the formation and growth of surgical societies and boards, which developed and promoted education and certification standards [11].

While postgraduate programs improved the quality of surgical education, selection into these programs remained a relatively unstructured process. Without agreed upon timelines and processes, hospitals competed with one another to be the first to fill their intern positions. Hospitals weren’t above pressuring candidates to commit to their program before they’d had a chance to interview with other institutions [5, 12].

With the introduction of formal residency and fellowship match structures starting in the 1950s, the process became fairer for both applicants and programs. More standardized selection criteria (typically based around measures of academic performance in medical school) could be applied. Programs could now interview all potentially desirable candidates and vice versa, and final selections were done using a simultaneous matching algorithm coupled with a contractual obligation between the matched applicant and program [5].

Most recently, with increasing adoption of competency-based surgical residency curricula, there is growing emphasis on the use of reliable measures of clinical skill during training to ensure that learners progress toward and ultimately achieve competency to enter independent practice by the time of graduation [10].

With these advancements, the selection process has become relatively standardized, and learners are more commonly being objectively evaluated on their clinical skills. However, one remaining challenge is that many of the selection criteria traditionally used by programs have, at best, poor correlation with both technical and nontechnical clinical performance in training and in practice. For example, 91% of orthopaedic programs in the United States recently reported using United States Medical Licensing Examination (USMLE) Step 1 scores as a selection filter for admission [8] despite evidence that these are not correlated with reliable measures of clinical skill (as distinct from performance on board examinations or subjective global evaluations by faculty) [7].

The recent study from Egan et al. [3] adds yet more evidence questioning our current approach for selecting residents into orthopaedic residency [3]. The authors queried the writers of 247 narrative letters of recommendation on their intended description of the strength of the applicant, and compared these to the strength of recommendation perceived by readers of these same letters, who were members of a residency selection committee. The authors found generally poor correlation between intended and perceived meaning, ultimately questioning the value of these letters in the selection process. Based on these discoveries, orthopaedic residency programs should review the specifics of the reference letters that they request of applicants, with the aim of maximizing their potential value in the application process. This may include requesting that specific aspects of the applicant’s candidacy be addressed, that a recommended structure or format for assessment be used, and that one or more letters come from a more experienced educator and letter writer (for example, the chairman or program director of the applicant’s home program). Additionally, programs should consider their application review and rank-ordering processes to ensure that reference letters are not given excessive weight during the decision-making process. One strategy might include scoring separate components of the application, with relatively low weighting of reference letter assessments.

Where Do We Need To Go?

With the upcoming transition of the USMLE Step 1 to a pass/fail score, and as we continue to gather more evidence calling into question the current selection criteria used for surgical training, programs are increasingly left to wonder how they can select learners that are most likely to succeed. Similarly, learners are increasingly left wondering how they can appropriately determine whether they are likely to succeed in a chosen specialty. There is nothing more disappointing for everyone involved than a learner who spends years struggling in a residency, and despite tremendous personal effort and support from his or her colleagues and mentors, the individual is unable to succeed and must switch to a different specialty.

Recent evidence has shown that there is great variability in the acquisition of surgical technical skill for novice learners (those with no prior surgical exposure), even when presented with similar dedicated training curricula [6]. While competency-based curricula with early and frequent skills-based evaluation may allow programs to identify poorer performing residents earlier, it’s unclear whether early, aggressive, and individualized support will allow the poorest performing trainees to reach competence.

Consequently, there remains a pressing need to identify selection criteria that can reliably predict future performance with measures that directly correlate with providing high-quality, patient-centered care as an independent surgeon. These include the acquisition and refinement of both technical and nontechnical skills and achievement of a consistent level of performance at or beyond the minimum level of competence expected of a practicing surgeon.

Going forward, we need more research within the domain of selection criteria for training. This includes identifying more reliable predictors of technical skill, nontechnical skill, as well as performance in independent practice. The failure of most current selection criteria to adequately predict performance suggests that novel, specialty-specific instruments may need to be developed, evaluated, and ultimately incorporated at the medical student level to better predict future performance.

How Do We Get There?

In the near-term, as a way to reduce the poor correlation between writer and reader interpretations of narrative letters of reference, referees and training programs should strongly consider the use of standardized letters of reference that specifically address candidate characteristics considered important in the selection process. This would help remove the ambiguity around the strength of individual referees’ recommendations highlighted in the current study. The American Orthopaedic Association has a standardized reference letter template that can be used for this purpose [1]. However, it must be recognized that while standardized letters have been shown to be more reliable and easier to interpret [4], it still remains unclear whether these will be any better at predicting future performance [9].

Achieving competence as an independent surgeon is characterized not only by an individual’s performance at a single point in time, but also how performance progresses over time. Consequently, with respect to predictors of technical skill, it may be important to evaluate longitudinal progression over time to discriminate differences in learning curves and better predict future performance [6]. For example, at present, the technical skills potential of a candidate for surgical residency training is typically assessed through a brief period of observation in the operating room, and communicated by a narrative statement or percentile score ranking a student’s “hands” in the operating room. In contrast, a structured assessment of technical skill acquisition over time while in medical school might better inform decision-making among both trainees and training programs. This might include repeated performance of surgical tasks such as knot tying, screw placement, or arthroscopic tasks in the simulated environment, assessed using either expert rating or automated video assessment algorithms. However, considerable work remains to be done in developing and validating these types of assessments. Ultimately, the continued success and growth of the specialty of orthopaedic surgery will depend on selecting candidates for training who are diverse in their backgrounds and experiences but similar in their passion for the field and likelihood to succeed in becoming competent independent surgeons. The results from the current study [3] should motivate the profession to continue seeking better ways to select for these qualities in residency applicants.