A Renewed Call for a More Equitable and Holistic Review of Residency Applications in the Era of COVID‐19

The COVID‐19 pandemic has highlighted and exacerbated racial disparities within health care; ¹ , ² , ³ , ⁴ unfortunately, the pandemic may also exacerbate the impact of disparities that exist within medical student evaluation. Without proper acknowledgement or intervention, the outsized impact of COVID‐19 on the underrepresented‐in‐medicine (URiM) student will have significant consequences. As most clinical rotations were ceased, medical schools devised strategies to ensure that students complete their medical education curriculum in a timely fashion. ⁵ To that end, and for safety reasons, the decision to eliminate away rotations for this upcoming residency application cycle became necessary. We believe that in the absence of away rotations and evaluations, there is a significant risk of defaulting to an overreliance on inherently biased evaluation measures during residency application review. There is well‐documented evidence demonstrating how racial biases influence grading, standardized test scores, honor society membership selection, and the Medical Student Performance Evaluation (MSPE). Therefore, giving unbalanced consideration to these measures could disproportionately impact URiM students in the upcoming application process. The field of emergency medicine (EM) has the opportunity to acknowledge this and past disparities to change how we evaluate students' potential for success in residency.

The value of away rotations and evaluations for students are multiple, including opportunities to work within a different environment, to demonstrate growth in both clinical and interpersonal skills, and to serve as “audition” rotations. For residency programs, purposefully or not, evaluations from away rotations have served as a counterbalance to traditional evaluation measures rife with bias affecting URiM students. Additionally, programs have prioritized the Standardized Letter of Evaluation (SLOE) from EM rotations in the application review process. ⁶ Commonly, programs required at least one SLOE outside of an applicant’s home institution. Because of the pandemic’s constraints on the clinical learning environment, the Council of Resident Directors in Emergency Medicine (CORD) has recommended students obtain one EM SLOE and one off‐service SLOE (O‐SLOE) for evaluation of clinical performance in other specialties. However, the absence of a second EM SLOE influences a program’s ability to assess an applicant’s clinical skills and potential for success in EM. The degree to which biases influence EM SLOE rankings and language is insufficiently studied. However, away‐rotation SLOEs, completed by faculty unaffiliated with a student’s home institution, may be perceived as the most objective assessment of clinical performance and potential trajectory as a resident. Elimination of away rotations serves to disproportionately negatively impact URiM students, because residency programs may now unintentionally give greater consideration to biased evaluation measures.

Although clerkship evaluations are intended to provide an accurate assessment of a student’s clinical abilities, racial disparities exist within clerkship grades with URiM and non‐URiM minority students often receiving lower grades and poorer clinical evaluations. ⁷ Additionally, despite receiving the same grade, URiM students’ narrative evaluations often include wording that is associated with personal attributes (e.g., “pleasant”) versus observed competencies (e.g., “knowledgeable”). ⁸ Reducing a student to personal attributes is potentially harmful because it fails to acknowledge competencies that are more predictive of future performance.

While Alpha Omega Alpha (AOA) Honor Society membership is a distinction that can influence interview selection, URiM students are less likely to be chosen as members. One study demonstrated that White students were six times likelier than Black students and twice as likely as Asian students to be members of AOA. ⁹ Although some medical schools have suspended their affiliation with AOA, the use of this distinction in the application review process continues and creates the potential to reduce the pool of interviewed URiM applicants. ¹⁰ , ¹¹

The MSPE attempts to provide an objective evaluation of a student's cumulative performance. However, across medical schools, the data provided in the MSPE lacks consistency. ¹² More alarmingly, the MSPE contains biases influenced by race and gender. ¹³ , ¹⁴ One study found that the MSPEs of White students were likely to contain the words “bright, exceptional, outstanding, best,” whereas Black students were often described as “competent.” This term “competent” was associated with a positive connotation in only 37 and 33% for Black and Latinx students, respectively. ¹³ Instead of giving such descriptors undue influence in interview selection and match list ranking, evaluators could utilize the MSPE to provide background information about the applicant and evidence of academic trajectory.

Underrepresented‐in‐medicine students are also subject to structural bias in standardized testing. Educational inequities, systemic racism, household income disparities, and housing insecurities have all been shown to negatively influence the standardized test scores of URiM students. ¹⁵ Using the USMLE to screen residency applicants becomes increasingly problematic as Step 1 scores are not a reliable predictor of residency success. ¹⁵ Obstetrics/gynecology literature demonstrated that while testing scores predicted rank on the residency program match list and matching success, they failed to correlate with residency performance. ¹⁶ Additionally, a systematic review showed that standardized testing correlated with in‐training examination scores but poorly with faculty evaluations. ¹⁷ The utilization of standardized test scores to predict success as a resident does not hold validity. Because USMLE Step 1 was never intended to be a stratification tool and will soon be reported as pass/fail, residency programs should deemphasize this test by eliminating score filters and treating the exam as pass/fail now. This approach may result in a more diverse pool of applicants from which residency programs can select. ¹⁵ , ¹⁸

The pandemic has created an opportunity to improve on the holistic evaluation of students and create a paradigm shift in what constitutes a qualified candidate. Holistic evaluation or review involves giving flexible, individualized, and balanced consideration to an applicant’s experiences, attributes, and metrics to assess their potential for success. ¹⁹ Reformatting the selection process will ensure that the field better reflects the diversity of our patients; the benefits of representation are many, the foremost of which is improved patient care outcomes. ²⁰ , ²¹ , ²² , ²³ Further, recognizing and valuing non‐cognitive characteristics is important as these serve as more reliable predictors of success in residency. Noncognitive factors such as athletic/musical talent and interviews have shown a moderate correlation with positive faculty evaluations. ¹⁷ Reviewers can look beyond even athletic/musical talents of applicants and carefully evaluate elements such as work history, service record, obstacles/challenges faced, and languages spoken of each potential candidate. ²⁴ Another study demonstrated that in addition to high achievement in sports and music, medical school attended was a predictor of success in residency. ²⁵ However, recognition that a singular piece of data can be nuanced and is not infallible is important. For instance, evaluating applicants based on the medical school attended can potentially harm URiM students as representation may not be uniform across institutions. Residency application evaluators should be cautioned against overreliance on traditional, easily identifiable metrics and give appropriate weight to other predictors of residency success. A holistic approach to reviewing residency applicants will decrease the influence of bias and increase the accuracy of assessment.

Holistic application review has proven to increase diversity in Undergraduate Medical Education (UME) and Graduate Medical Education (GME). ²⁶ , ²⁷ , ²⁸ One Internal Medicine residency program’s approach to improving diversity can serve as a blueprint for application review. ²⁷ Prior to applicant selection, faculty were required to undergo Implicit Bias Training. And, as a framework for application review, attributes that aligned with the mission of the medical school and residency program were identified. These attributes included commitment to serve the underserved, demonstrations of leadership, Spanish fluency and self‐identification of race/ethnicity that represented the diverse population of their city. This program saw a substantial increase in the number of URiM students matriculating to the program (12.5% to 31.4%). As part of an initiative to increase diversity, the Highland EM program described changes to their residency application review in 2006. ²⁸ This process eliminated a USMLE score filter, the applicant score sheet was revised to decrease the weight of certain aspects of the application (MSPE, USMLE scores, and SLOE), and applicants’ alignment with the program's mission was emphasized. These measures led to an increase in the percentage of graduating URiM residents from 12.1% to 27.1%. These programs demonstrate that a holistic approach to application review that includes assessment of applicants’ non‐cognitive characteristics and alignment with a program’s mission is feasible and impactful.

The pandemic and the resulting inability to complete an away rotation present a challenge for all applicants. Notably, these challenges are magnified for students who attend institutions without a home EM program, many of whom are URiM students studying at historically Black institutions. The absence of away rotation evaluations creates the risk of overemphasizing traditional biased evaluation tools which would have a disproportionate negative impact on URiM students. The pandemic has created a tremendous opportunity for residency leaders to fundamentally change application review by committing to the holistic evaluation of each applicant and deemphasizing traditional evaluation measures. The EM community can lessen the outsized burden that the COVID‐19 pandemic has placed on URiM students and continue to lay the foundation for a diverse workforce.

AEM Education and Training 2021;5:135–138