Reimagining Residency Selection: Part 3—A Practical Guide to Ranking Applicants in the Post-COVID-19 Era

The COVID-19 pandemic has inextricably changed academic medicine and medical education, including a complete overhaul of the residency and fellowship selection process. Applicant interviews are now virtual, and visiting electives, often serving as “audition rotations,” are cancelled or highly restricted by national bodies.^1,2 Previously, we reviewed the topics of recruitment³ and interviews.⁴

In this guide, we explore findings from fields outside of medicine, such as the social sciences and business, to inform residency selection with a focus on file review and rank list determination.

We also summarize recent evidence around trainee assessment during candidate ranking, since it may be worthwhile for program leaders to harness the most recent evidence while reimagining their systems. Studies show that these systems are flawed with rater and systemic biases. In the era of the Me Too and Black Lives Matter movements, there is a compelling case to go beyond simple digital conversion of processes and approach this task as an opportunity to redesign a better system for the next generation of learners and patients.

Revisiting Selection and Ranking

Program directors use the Electronic Residency Application Service (ERAS) or Canadian Residency Match System (CaRMS) application data to attempt to predict which applicants will be most successful in their program. The COVID-19 pandemic has resulted in changes to the standard data typically used for rank order decisions. Traditionally, specialty-specific letters of recommendation and in-person interviews have played an important role in determining applicant rank order position.^5–7 Applicants are likely to have fewer specialty-specific letters of recommendation in the 2020–2021 application cycle. Emergency medicine (EM) program directors have created a template for a non-specialty letter of reference to guide faculty in non-EM specialties regarding the specific characteristics and assessments prioritized in EM.⁸ Specialties that emphasize letters of recommendation written by faculty within the specialty may consider creating similar templates.

Programs may also want to more explicitly obtain information during interviews, given that interviews will now be conducted virtually, and the actual time spent directly engaging with applicants may be truncated. Providing implicit bias training to interviewers and using a standardized, structured interview process (eg, multiple mini interviews) may help programs maximize objectivity and garner information that is more likely to be predictive of an applicant's future performance.^9–13

Some programs have employed scoring rubrics or mathematical models to assist in the selection process.^14–16 The predictive ability of various application data or summative scoring rubrics on future resident performance has been mixed.^14,15,17,18 Previously used scoring rubrics may need to be restructured or reweighted, by potentially incorporating diversity as a metric to mitigate existing structural bias within medicine. Expanding screening tools to capture a broader spectrum of candidate aptitudes may help to create more diverse classes of residents. Additionally, programs may consider the blinding of applicant gender, first names, or excluding extracurricular activities to promote equity during the ranking process.¹⁹ Extracurricular activities may provide insights regarding applicant interests, but may also lead to bias related to socioeconomic status.¹⁹

Reimagining Selection and Ranking in the Post-COVID-19 World

Although not a universal practice, some programs examine candidates' social media profiles as a screening instrument or tool for selection.^20–23 Those programs that have used social media review as part of the selection process have found mixed results.^20–23 Given the limitations that a virtual recruitment season has placed on programs, review of applicant social media pages may play an increased role in the selection process.

Alternative modalities for assessment may provide additional information to program directors. Some programs have incorporated assessment of technical skills or critical thinking into their application process.^24,25 Having applicants participate and be assessed in virtual educational sessions (eg, simulation, skills lab, case-based discussions) in real time or via video may provide additional insight into their capabilities. These assessments could take place prior to selection for interviews or during the interview processes. Limited data suggested that incorporating pre-interview assessment does not negatively influence applicant perceptions of the program.²⁶ Additionally, consideration of assessment beyond the cognitive- and skill-based realms, such as personal and professional characteristics, may help program directors capture a more holistic representation of applicants. Several different assessments have been used in medical education to measure non-cognitive dimensions such as situational judgement, personality traits, and professional characteristics.^{11,13,27–30} Limited data supported that these types of assessments predict rank list position and future performance, and are more predictive than traditional cognitive assessments.^{13,27,28,30–32} The Table summarizes evidence-informed tactics for scoring and ranking application elements.

Table.

Evidence-Informed Tactics for Scoring and Ranking Application Components

Best Practices	Specific Tactics	Considerations During COVID
Standardize incoming assessment data and align with specialty values.	Create a standardized template for letters of recommendation that address the characteristics prioritized by the specialty.	Consider creating separate templates for faculty letter writers within and outside the specialty. The effect of a meaningful letter of recommendation is heightened due to the inability to participate in visiting rotations and the lack of subsequent external sources of input.
Use objective methods to screen applications that are fair and aligned with program values.	Create a comprehensive scoring rubric with various weights according to program values specifically incorporating diversity as a metric. Provide implicit bias training for all application screeners. Consider deliberately blinding reviewers to aspects of files to minimize bias during the screening process. Consider screening social media sites utilizing a structured approach.	A holistic review and comprehensive scoring rubric may require more faculty resources to screen applications. The increased usage of social media creates new material for applicant screening.
Standardize interviews according to program priorities.	Use a standardized, structured interview process, which involves the same questions and order for each applicant. Questioning should be behavioral and situational in nature.	Interview questions should focus on topics that are high priorities for the program. Virtual interviews can limit the applicant's ability to effectively present their strengths and may be open to more inherent biases. A standardized approach is one way to try to limit bias.
Consider innovative assessment strategies.	For specialties with a significant procedural component, incorporate an assessment of technical skills (eg, suturing, intubation). Assess critical thinking skills with virtual simulation of small group case-based discussions. Incorporate assessment of noncognitive skills (eg, professionalism, personality traits).	Alternative assessments can be done prior to the interview or conducted during the virtual interview process. Use of commercial assessments often requires funds.

Considerations for Rank Order Determination

Because the transition to virtual experiences in the graduate medical education (GME) application process has been abrupt, processes are rife with uncertainty for applicants and programs. The methods for determining rank list order should minimize implicit bias, maximize diversity and inclusion, and prioritize factors most likely associated with resident success, as defined by each program. The first priority of a program's rank list is to ensure that all positions are filled. Given the efficiency and cost-savings of virtual interviews, students may apply to and interview at an even greater number of programs this year. Program directors may wish to implement processes to assess applicant interest in the program prior to interviewing, as well as increase the number and diversity of the candidates invited to interview.

With reductions and changes in the information available to rank list discussions, programs may benefit from early consensus decisions, prior to the interview season, regarding what applicant factors matter most to the program. This determination will consider the current program strengths, weaknesses, and vision. Explicitly naming these factors can help ensure they are elicited during the selection process and are thus available for and considered during rank list development. Program directors can apply published strategies for minimizing implicit bias into rank meetings, and consider adding diversity as a metric in ranking spreadsheets and rubrics.³³

Finally, because selection committee activities may be conducted virtually as well, it is important to consider how virtual interactions may change the usual consensus processes. Literature regarding group decision-making in clinical competency decisions may be relevant to rank meeting discussions.³⁴ Hauer and colleagues suggested that attention to group size, group understanding of its work, the role of the leader, information-sharing procedures, and time pressures is necessary to ensure optimal outcomes.³⁴

Conclusions

Changes as a result of the COVID-19 pandemic necessitate changes to the GME selection process. The limited availability of specialty-specific letters of recommendation and other traditional application elements, in addition to the elimination of in-person interactions, requires revisions to screening rubrics and consideration of alternative assessments. Program leadership may want to create processes to assess applicant interest, reach early consensus agreement on which applicant factors are prioritized and thereby assessed, and deliberately incorporate procedures that ensure equity.