Engaging Choice Architecture: Interventions in the Emergency Medicine Standardized Letter (eSLOE 2.0) to Adjust Rater Behaviors

ABSTRACT

Background

Residency programs are utilizing standardized letters of evaluation with increasing frequency to obtain concise and discerning information about applicants. However, letters show a narrow distribution of ratings, leading to difficulties differentiating between applicants.

Objective

To determine if choice architecture, through 2 interventions on the electronic emergency medicine Standardized Letter of Evaluation 2.0 (eSLOE 2.0), can increase the spread of ratings on letters written for emergency medicine applicants.

Methods

On the eSLOE 2.0 Part B for the 2024-2025 application cycle, radio buttons were prepopulated to a “3” on a Likert scale of 1 to 5 (rather than blank), and a pop-up message prompted letter writers to provide narrative justification for ratings of “1” (lowest) or “5” (highest). Ratings of applicants after this intervention were compared with ratings from 2 prior application cycles.

Results

A total of 17 727 letters (5938 from 2024-2025; 11 789 from 2022-2024) were analyzed. Post-intervention, there was a decrease in ratings of “5” across all domains in Part B, with the absolute difference in the percentage of students rated “5” ranging from -12.3% to -19.7%. There was a post-intervention increase in percentage of “3” and “4” utilization across all domains ranging from +3.9 to +11.0 and +7.2 to +11.0 respectively. Ratings of “1” and “2” remained stable post-intervention.

Conclusions

Prepopulating SLOE Part B ratings to 3 out of 5 and pop-up messages prompting letter writers to justify top or bottom ratings led to a shift in the distribution of scores and a reduction in the percentage of top ratings in the 2024-2025 cycle, as compared to the prior 2 cycles.

Introduction

The Standardized Letter of Evaluation (SLOE) was developed in emergency medicine (EM) to evaluate residency applicants in a concise and standardized manner.¹ Program directors rate the SLOE as the most valuable aspect of an EM residency applicant’s file.² The importance of the SLOE has increased in recent years given the decrease in other differentiating information, with United States Medical Licensing Examination Step 1 and many clerkships moving to pass/fail grading.^3,4 However, there has been a trend toward positively skewed scores within the SLOE, raising concerns that the intent is not being met.⁵ This trend has been noted in other specialties that utilize standardized letters as well.^6-8 The causes are likely multifactorial, related to medical schools’ desire for their students to successfully match and in some specialties attributed to a strong applicant pool.^6-10

Prior attempts to minimize inflation and improve discrimination began in the 2016-2017 application cycle with the adaptation of an electronic system that allowed the EM community to track rating and ranking distributions. Auto-population of each institution’s previous year’s scores allowed letter readers to assess the historic grading distribution for a particular institution or letter writer. Institutional dashboards helped faculty visualize their overall and Global Assessment ranking distribution. This electronic format improved the spread of applicant Global Assessment scores compared to the 2011-2012 EM Standardized Letter of Recommendation but did little to affect the positive skew in ratings of other qualifications for EM.¹¹ With the Coalition for Physician Accountability recommending increased use of standardized letters for all specialties, the push to have these letters be as forthcoming and meaningful as possible is of utmost importance for all specialties.¹²

For the 2024-2025 application cycle, choice architecture was employed on the revised electronic SLOE (eSLOE 2.0) Part B section, which assesses student performance in 7 domains related to professionalism and communication. Applicants are rated from 1 to 5 with a rating of “5” being described as an “exceptional EM applicant” and a rating of “1” describing a “minimally acceptable EM applicant.”

Choice architecture is a behavioral tool that designs how choices are presented in an effort to influence individuals’ decision-making. Choice architecture has been widely studied in behavioral sciences and heavily utilized in policy making and marketing.^13-15 While a 2007 study of EM rotation grading at a single institution showed a change in distribution of grades with renaming of honors to “top 5%,”¹⁰ we believe ours is the first study investigating choice architecture’s ability to influence evaluation behavior patterns in a widely used national standardized letter across multiple institutions and multiple application cycles.

The purpose of this study is to determine whether 2 electronic choice architecture interventions, implemented simultaneously to the eSLOE 2.0 Part B, diminished the positive ratings skew.

Methods

This is a retrospective cohort study comparing anonymized eSLOE 2.0 Part B evaluations of medical student performance from the aggregate 2022-2023 and 2023-2024 residency application cycles (pre-intervention) to the 2024-2025 residency application cycle (post-intervention). A complete eSLOE 2.0, including Part B, can be found in the online supplementary data. Permission was obtained from the Council of Residency Directors in Emergency Medicine (CORD) Board of Directors to access this data. No individually identifiable data was used or accessed.

For the 2024-2025 residency application cycle, all 7 Part B ratings were prepopulated to “3” (out of 5), requiring the letter writer to adjust this default rating if they believed a candidate deserved a higher or lower ranking. In the pre-intervention eSLOE 2.0, ratings on Part B were left blank. Additionally, a pop-up window appeared if a “1” (lowest rating) or “5” (highest rating) were selected, prompting the letter writer to provide further explanation in the narrative section for this response to these questions in Part B. Both interventions took place in the same application cycle for all eSLOEs.

Ratings from the eSLOE 2.0 Part B during the pre- and post-intervention application cycles were collected and analyzed via Excel (Microsoft). Descriptive statistics were used to summarize the distribution of ratings pre- and post-intervention. Absolute differences in the percentage of students rated in each category were calculated.

This study was deemed exempt by the institutional review board at Cooper University Hospital.

Results

Data were extracted from 17 727 eSLOE 2.0 letters: 5938 post-intervention and 11 789 pre-intervention. We found a decrease in the ratings of “5” across all domains in Part B with the greatest absolute difference observed for “Punctuality, attendance, and preparation for duty” (-19.7%; 95% CI, -21.2 to -18.3) and “Timeliness and responsiveness in completing administrative tasks” (-19.5%; 95% CI, -21 to -18.1) and the least difference in “Dependability, responsibility, initiative, and work ethic” (-12.3%; 95% CI, -13.8 to -10.7). The changes resulted in a more normal distribution of Part B ratings (Figure). The absolute difference in the percentage of students rated “5” for each characteristic ranged from -12.3% to -19.7% (Table). Increased utilization of “3” and “4” ratings were noted post-intervention with an absolute difference in percentage ranging from +3.9% to +11.0% and +7.2% to +11.0% respectively (Table). Ratings of “1” and “2” were overall low with differences ranging from -0.1% to -0.03% and -0.4% to +0.2% respectively (Table).

Figure.

Emergency Medicine Standardized Letter of Evaluation Part B Rating Distributions, 2024-2025 (Post-Intervention) vs 2022-2024 (Pre-Intervention)

Table.

Absolute Difference in Percentage of Students for Each Rating of 1 to 5 in Part B of the Emergency Medicine Standardized Letter of Evaluation (eSLOE 2.0)

eSLOE 2.0 Part B Category	Absolute Difference in Percentage of Students Rated “5” (95% CI)	Absolute Difference in Percentage of Students Rated “4” (95% CI)	Absolute Difference in Percentage of Students Rated “3” (95% CI)	Absolute Difference in Percentage of Students Rated “2” (95% CI)	Absolute Difference in Percentage of Students Rated “1” (95% CI)
Compassion, sensitivity, and respect toward patients and team members	-13.2 (-14.7 to -11.7)	+8.5 (7.0 to 10.1)	+4.9 (3.7 to 6.0)	-0.1 (-0.4 to 0.1)	-0.03 (-0.1 to 0.03)
Receptivity to feedback and ability to incorporate feedback	-12.6 (-14.1 to -11.1)	+9.2 (7.6 to 10.7)	+3.9 (2.7 to 5.1)	-0.4 (-0.8 to 0.05)	-0.09 (-0.2 to 0.03)
Dependability, responsibility, initiative, and work ethic	-12.3 (-13.8 to -10.7)	+7.8 (6.3 to 9.4)	+4.7 (3.5 to 5.8)	-0.09 (-0.5 to 0.3)	-0.1 (-0.2 to 0.01)
Punctuality, attendance, and preparation for duty	-19.7 (-21.2 to -18.3)	+11.0 (9.5 to 12.5)	+9.1 (7.8 to 10.3)	-0.2 (-0.6 to 0.1)	-0.09 (-0.2 to 0.02)
Timeliness and responsiveness in completing administrative tasks	-19.5 (-21.0 to -18.1)	+8.6 (7.1 to 10.2)	11.0 (9.6 to 12.3)	-0.05 (-0.4 to 0.3)	-0.04 (-0.2 to 0.1)
Interpersonal and communication skills with patients and family members	-13.3 (-14.8 to -11.8)	+7.7 (6.2 to 9.3)	+5.8 (4.6 to 7.0)	-0.1 (-0.4 to 0.2)	-0.09 (-0.2 to 0.01)
Interpersonal and communication skills with faculty, residents, and health care professionals	-12.6 (-14.1 to -11.1)	+7.2 (5.7 to 8.7)	+5.3 (4.1 to 6.6)	+0.2 (-0.3 to 0.06)	-0.1 (-0.2 to 0.01)

Discussion

Prior attempts to improve score spread and decrease presumed rating inflation for standardized letters have been largely unsuccessful.¹⁶ In an effort to provide a potentially more useful letter with a broader spread of ratings, choice architecture was trialed to determine the effect assessment of applicants. Our results support the hypothesis that simple changes utilizing choice architecture can lead to a more normal distribution of student characteristics.

A prior meta-analysis shows that decision structure interventions are more effective in changing behavior than decision information or decision assistance interventions.¹⁴ Choice default in particular has been found to be one of the most powerful choice architecture techniques to nudge people toward a desirable behavior.¹⁴ Thus, we speculate that prepopulating ratings to a midpoint on Part B of the letter could have had a greater influence of behavior than the pop-up boxes prompting letters to describe the extremes of grading in the narrative.

Other key considerations when introducing choice architecture into an assessment instrument include its potential impact on content and response process validity, as changes to the tool’s structure may alter evaluator behavior.¹⁷ While one interpretation of these findings is that the interventions reduce inflation and better align with true applicant performance, an alternative explanation is that they suppress extreme ratings independent of actual performance, potentially underrepresenting high (or low) performers. Choice architecture also intentionally modifies raters’ cognitive and behavioral processes. Although it may promote more deliberate consideration and discourage automatic assignment of top scores, these same mechanisms increase cognitive effort and may reflect “user energy” rather than applicant performance, as default options are known to shape decision behavior through cognitive effort minimization and implied endorsement.¹⁸ Although our study design does not allow us to distinguish between these explanations, we observed greater changes in ratings of “4” and “5” compared with the auto-populated “3,” suggesting that raters regularly changed the prechecked rating rather than simply choosing the path of least resistance, which argues against a pure default effect explanation.

Standardized letters are highly utilized for residency applications across specialties. Specialties considering similar choice architecture interventions to combat assessment rating inflation should communicate clearly with their community as changes may lead to altered patterns of ratings, which, if unrecognized, could negatively impact applicants’ residency application success.

Our study has several limitations. As a single-specialty investigation, generalizability to other specialties may be limited. We did not perform subgroup analyses of raters or sites to assess degree of variation between programs. We also cannot fully separate the effects of the interventions from potential changes in the applicant pool or broader context. Effects on selection fairness, predictive validity for downstream outcomes (interview offers, rank list placement, resident performance), and greater reliability remain to be evaluated. Future work should further characterize whether raters engaged more deeply with the assessment when a prepopulated rating was presented.

Conclusions

Employing choice architecture techniques including choice default (Part B ratings prepopulated to a “3” out of 5) and a decision assistance intervention (pop-up windows for extremes of scale selection) led to a shift in the distribution of scores and a reduction in the percentage of “5” ratings in the 2024-2025 cycle, as compared to the prior 2 cycles.

Author Notes

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.

Disclosure: AI was not used in writing this article.

This work was previously presented as a poster at the Council of Residency Directors in Emergency Medicine Academic Assembly, March 2-5, 2025, Seattle, Washington, USA.

Editor’s Note

The online supplementary data contains a complete eSLOE 2.0, including Part B.