Should Canadian urology adopt structured reference letters for residency matching?

Introduction

Senior medical students participate in a yearly match for entry-level postgraduate positions in Canada through the Canadian Resident Matching Service (CaRMS). As part of the application, aspiring residents must ask medical school preceptors to submit letters of reference. In Canada, letters of reference from clinical preceptors practicing in the field of interest of the applicant are consistently the most important application documents evaluated by selection committees when ranking potential residents.^1–3 A letter of reference is a detailed letter that describes a referee’s experiences with the applicant and recommends them for further training in their specialty of choice. While general guidelines for letters of reference can be found on the CaRMS website, traditional residency letters of reference are narrative, non-standardized, and do not follow a prescribed template.

Context

In August 2021, the Association of Faculties of Medicine of Canada (AFMC) Resident Matching Committee announced that it had approved the structured reference letters (SRL) developed for the family medicine (FM) and emergency medicine (EM) disciplines.⁴ This is the first time in Canada that SRLs are required by all postgraduate medical programs of a medical specialty for the R-1 match. Initially announced for the 2021 match cycle, the FM SRL was delayed due to concerns regarding the timing of the announcement, the content of the form, and the lack of stakeholder engagement. The current FM SRL and new EM SRL can be found in the online Appendix (available at cuaj.ca).

In addition to demographics and context of the working relationship between the referee and the applicant, both SRLs require that referees evaluate various areas/attributes (such as work ethic, teachability, and communication) using either a rank (top 50% to top 5%) or a qualitative scale (poor to excellent). Both SRLs also reserve space for brief supporting narrative statements. While SRLs have not been implemented for Canadian surgical programs, a study published in 2017 found that about half of Canadian surgical program directors would recommend the use of SRLs and think that this would improve the selection process.⁵

Evidence supporting SRLs

There is evidence supporting the use of SRLs (also known as standardized letter of recommendation or standardized letter of evaluation) for residency applications, primarily from the U.S. EM match, where over half of programs require at least one EM-specific SRL. The U.S. EM SRL was first introduced in 1999 by the Council of EM Residency Directors to provide a more objective evaluation of actual performance that can be used to compare applicants for optimal program fit.⁶ It has been shown to decrease writing time for referees and reviewing time for application reviewers, facilitate interpretation with high interrater reliability, and most importantly, predict resident performance in core residency competencies.^7–9 SRLs can also reduce gender bias in trainee selection, as shown in a study of letters submitted to an otolaryngology head and neck surgery residency in the U.S.¹⁰

SRLs in the Canadian urology setting

Many of the advantages listed above will likely apply to the Canadian urology setting. For example, there is evidence of significant linguistic differences and gender bias existing among letters of reference written for women applying to urology in the U.S.¹¹ This is likely to be true for Canadian urology applicants as well; SRLs could mitigate these gender disparities by standardizing large parts of the evaluation process. SRLs may also benefit late-comers to urology who may not have developed strong bonds with urology mentors outside of the clinical setting. An SRL would allow a leveled and focused evaluation of the applicant, regardless of the relationship of the assessor and the trainee outside of the rotation setting. In the same vein, SRLs may alleviate the pressure on all applicants to obtain three strong personalized letters with at least one being an SRL. As an SRL can’t vary in content, there may be concerns that it will not capture the varying priorities of urology programs. However, different programs may apply different weights to each section of an SRL a priori. Lastly, with limited in-person “standardized” and direct clinical assessment of applicants, standardizing reference letters may mitigate the lack of interaction between program and applicant when selecting future residents, as they are theoretically written based on extended direct clinical evaluation.

There are, however, many potential limitations and pitfalls of an SRL in the Canadian urology setting. While the Canadian urology match is competitive, the absolute number of applicants remains small compared to other specialties (such as FM and EM). The number of applicants to urology has ranged from 45–75 per year since 2012.¹² This pales in comparison to the 1663–2162 and 166–227 annual applicant range for FM and EM, respectively.¹² As such, whereas a thorough review of applications without SRLs may not be possible in FM and EM, it is more feasible in urology. Similarly, considering the smaller number of applicants, urology has the capacity to interview a larger proportion of applicants, making an objective interview invitation cutoff with SRLs less essential.

Canadian academic urology is a small community. This proximity facilitates the evaluation of traditional letters of reference without the need for standardization. Standardization itself may be problematic when attempting to capture subjective measures. Non-cognitive attributes (interpersonal skills, empathy, etc.) are increasingly important when selecting future residents; it is questionable whether candidates can accurately and unbiasedly be compared by clinical preceptors on these measures in a standardized way. Likewise, some SRLs, like the initial iteration of the FM SRL, may inappropriately criticize certain behaviors (such as expressing sadness) and expect a certain level of performance rather than encourage trainees to ask for help and improve on their skills. Standardization may also be ill-fitted to Canadian urology, where each program varies slightly in institutional objectives and faculty phenotype, and as such, emphasize different aspects of the application, requiring tailored applications. Lastly, applicants may have longstanding mentor-student relationship with a urologist in the context of research. If they are not afforded the opportunity to work with their mentor in a clinical setting, SLR may preclude a letter from said mentor that may provide a more holistic assessment of the applicant.

Additional considerations

Future research is needed to evaluate the benefits of SRLs in the Canadian urology context. An initial study could emulate that of Perkins et al, who sent an SRL to writers of traditional letters of reference for each applicant to assess interrater reliability and compare the effect of SRL vs. traditional letters on student rankings.¹³ In addition to the SRL itself, there should be considerations on the implementation of SRLs. For example, while Canadian FM and EM programs required that all submitted letters be SRLs, Canadian urology can consider requiring a single SRL, with all other letters being standard narrative letters of reference. Other considerations include the use of group SRLs, which are written by whole departments, comparing all trainees that have rotated on their service.

If a Canadian urology-specific SRL is to be developed, it is important to ensure adequate stakeholder engagement — at a minimum, all urology training programs and trainees at each level should be represented. The development process itself must be defined, as there is no evidence-based validated approach. Considerations such as included rubrics and adequate psychometrics measures will be important to account for varying reference scales of referees.

Conclusions

It is increasingly relevant to innovate the CaRMS match considering the ever-increasing pressure on residency programs to rank potential residents with fixed resources, reduced visiting electives, and switch to virtual interviews in the context of the COVID-19 pandemic and possibly moving forward as well. While there are many advantages to a urology-specific SRL for the Canadian match, there are potential pitfalls and limitations. If the SRL is adopted, its development and adoption should engage all relevant stakeholders, including trainees, and proceed in a stepwise fashion.