Underrepresented in Medicine and International Graduate Bias in the CASPer Situational Judgment Test for Ophthalmology Residency Applicants in 2022–2023

Abstract

Purpose:

Situational Judgment Tests (SJTs) are increasingly utilized in medical trainee recruitment decisions yet little is known of their potential biases. This study was designed to evaluate the Computer-Based Assessment for Sampling Personal Characteristics (CASPer) SJT scores of ophthalmology residency applicants and compare these to Step 1 and Step 2 scores in different demographic groups.

Methods:

All applicants to the ophthalmology residency match completed CASPer during the 2022–2023 recruitment cycle. Applicant scores were cross-referenced within the SF Match Residency Application System to collect data on applicant gender, self-identified Under-Represented in Medicine (URiM) status, and International Medical Graduate (IMG) status, as well as Step 1 and Step 2 scores.

Results:

A total of 760 of 765 (99.3%) of applicants with CASPer scores were identified in SF Match, 41% of which were female, 15% URiM, and 11% IMG. Males scored slightly higher on all three tests, with the mean difference in Step 1 (4.3) reaching a statistical significance (Cohen’s d = 0.29, p < 0.001). When comparing URiM and IMG applicants, these groups scored significantly lower than their counterparts on all three tests (p all <0.001), with the magnitude of mean difference greater for CASPer than Step 1 or 2. There were no differences between genders in CASPer percentile scores; however, URiM applicants performed on average 17% and IMG applicants 32% lower than their counterparts.

Conclusions:

The CASPer SJT demonstrated significant bias against URiM and IMG ophthalmology residency applicants. This bias was even greater than Step 1 and Step 2 performance, suggesting the continued need for caution when utilizing any standardized testing in recruitment decisions.

1. Introduction

Traditional recruitment metrics in medical education have the potential to exacerbate bias in admissions decisions for underrepresented applicants,^1–3 leading governing organizations to advocate for more review processes when evaluating applicants.^4,5 This, in part, has led to the recent proliferation of assessments intended to measure non-academic attributes.⁶ Situational Judgment Tests (SJTs) assess candidate responses to predetermined interpersonal scenarios and are increasingly utilized by medical schools and residency programs for holistic admissions purposes.⁶

The Computer-based Assessment for Sampling Personal Characteristics (CASPer) SJT is required for admission to over a dozen medical schools,⁷ and recently a distinct version for Graduate Medical Education (GME) recruitment has been developed.⁸ CASPer has been suggested to increase representation for underrepresented groups in medical school admissions compared to traditional academic metrics,⁹ although this is still controversial.⁷

During the 2022–2023 recruitment cycle, all applicants to the ophthalmology match were required to complete CASPer as part of a pilot project. To our knowledge, this is the largest and most inclusive GME level dataset utilizing CASPer or any other SJT. This study was designed to evaluate the CASPer scores of applicants to the ophthalmology residency match in 2022 and compare SJT and traditional standardized test performance to demographic information including gender as identified by usage of pronouns in the application, self-identified Under-Represented in Medicine (URiM) status, and International Medical Graduate (IMG) status.

2. Methods

As part of a pilot project sponsored by the American University Professors of Ophthalmology (AUPO), the governing body of academic ophthalmology in the United States, all applicants during the 2022–2023 recruitment cycle were required to take CASPer. Applicant CASPer scores were provided to program directors (regardless of whether they applied to their individual programs) to be utilized per their arm of the project. Investigators at Montefiore Medical Center/Albert Einstein College of Medicine, Oregon Health and Science University/Casey Eye Institute, and the University of Kentucky cross-referenced CASPer scores with applicants to their programs through the SF Match Residency Application System¹⁰ to collect additional data on applicant gender, self-identified URiM status, domestic versus international medical school location, United States Medical Licensing Examination (USMLE) Step 1 score, and USMLE Step 2 score. Additional data on remaining applicants was provided by SF Match through a data sharing agreement with the AUPO. A voluntary checkbox within the SF Match application identified candidates as URiM, which was defined by the AUPO and SF Match as Black or African American, Hispanic or Latino, and/or Native American (American Indian/Alaska Native/Native Hawaiian).¹⁰ Gender was determined by pronoun usage within the letters of recommendation and Medical Student Performance Evaluations; all applications exclusively utilized he/him/his and she/her/hers pronouns. Graduates of medical schools outside the United States, Puerto Rico, and Canada were considered IMGs.¹¹ The Step 1 and Step 2 percentile differences between groups were estimated from the USMLE normative table.¹⁴

The 2022 CASPer assessment was a 12-section test with 4 written and 8 video sections. In each section, applicants were presented with a scenario and answered three questions. A different trained rater assigned a 1–9 Likert scaled score to each section, and the 12 ratings were averaged into a single score. The scores were converted to a z-score and percentile, which were provided to programs for review.¹²

Descriptive statistics of the study were summarized by count (%) for categorical variables and mean (standard deviation) or median (interquartile range, IQR) for continuous or ordinal variables. Three scores (Step 1, Step 2, CASPer Z-score) were compared by gender, self-identified URiM status, IMG, and domestic URiM (URiM among non-IMG) using a two-sample t-test. The mean difference and its 95% confidence interval (CI) of the score by two groups were estimated and Cohen’s D was calculated for the standardized effect size to compare across three scores.¹³ Similarly, the percentile of each score (a higher percentile corresponding to a better result) was compared between groups using a Mann–Whitney U-test and the median difference of two groups and its 95% CI were estimated. To reduce a potential bias by the availability of score, a sensitivity analysis was performed among the applicants with all three scores. Lastly, relationship in the scores was examined using a scatter plot whether the CASPer Z-score relative to Step score was consistently higher or lower for a group of interest, and whether the relationship (slope) between CASPer Z-score and Step score differed by groups, which was tested by a linear regression on CASPer Z-score with a Step score, group variable, and an interaction term of Step score and the group.

This study was approved by the Institutional Review Board (IRB) at the University of Kentucky. All participating sites also received local IRB approval. The manuscript was also reviewed and approved by the SF Match Oversight Committee of the AUPO.

3. Results

A total of 760 of the 765 (99.3%) applicants with CASPer scores were identified in SF Match. Three-hundred and twelve applicants were female (41.1%), 116 were URiM (15.3%), 83 were IMG (10.9%), and 94 URiM (81.0%) were from a domestic medical school¹¹ (12.3%) (Table 1). A total of 669 (88.0%) applicants reported a numeric score for Step 1, with a mean of 240.9 and range of 194–269. A total of 578 (76.1%) applicants reported a numeric score for Step 2, with a mean of 251.2 and a range of 192–283. All 760 applicants completed the CASPer assessment, with a mean Z-score of 0.02 and range of −3.09 to 2.47. When comparing applicants with at least one of the USMLE scores missing (N = 242) to applicants with both scores (N = 518), IMG applicants were significantly more likely to report both USMLE scores than not (p < 0.001), and those reporting only Step 1 scores had a significantly higher mean score (mean = 243.9 ± 15.6) than those reporting both (mean = 240.9 ± 15.2) (p = 0.006). When comparing gender, males scored higher on all three tests, with only the mean difference in Step 1 (4.3) reaching significance (Cohen’s d = 0.29, p < 0.001). There were significant differences when comparing non-URiM to URiM for all three tests, with mean differences of 7.7 (d = 0.51, p < 0.001), 7.7 (d = 0.59, p < 0.001), and 0.6 (d = 0.59, p < 0.001) on Step 1, Step 2, and CASPer Z-score, respectively. Similarly, there were significant differences when comparing non-IMG to IMG for the three tests, with mean differences of 9.3 (d = 0.62, p < 0.001), 10.1 (d = 0.78, p < 0.001), and 1.2 (d = 1.22, p < 0.001) on Step 1, Step 2, and CASPer Z-score, respectively. CASPer showed a significantly higher difference for IMG applicants (d = 1.22, 95% CI = 0.98–1.45) than Step 1 (d = 0.62, 95% CI = 0.38–0.86) or Step 2 (d = 0.78, 95% CI = 0.38–0.86). The differences between non-URiM and URiM applicants remained significant when analyzing exclusively URiM applicants from domestic medical schools, with mean differences of 6.0 (d = 0.42, p = 0.002), 7.1 (d = 0.57, p 0.001), and 0.5 (d = 0.58, p < 0.001) on Step 1, Step 2, and CASPer Z-scores, respectively (Table 1). When comparing non-URiM and URiM applicants from international medical schools, only Step 1 scores remained significant, with mean differences of 9.3 (d = 0.51, p = 0.063), 4.2 (d = 0.27, p = 0.334), and 0.3 (d = 0.25,p = 0.289) on Step 1, Step 2, and CASPer Z-scores, respectively (see Fig. 1).

Table 1.

USMLE and CASPer score performance by gender, URiM, and IMG.

		Number	Step 1	Step 2	CASPer Z-score
Total		760	241 (15.2)	251 (13.5)	0.02 (1.06)
Gender	Female	312	238 (15.5)	251 (13.4)	0.0 (1.1)
	Male	448	243 (14.7)	252 (13.6)	0.0 (1.00)
	Difference		−4.3 (−6.7, −2.0)	−1.4 (−3.7, 0.8)	0.0 (−0.1, 0.2)
	Cohen’s d		−0.29 (−0.44, −0.13)	−0.11 (−0.27, 0.06)	0.01 (−0.13, 0.16)
	P		<0.001	0.202	0.869
URiM	URiM	116	234 (17.0)	245 (15.0)	−0.5 (1.1)
	Non-URiM	644	242 (14.6)	252 (12.9)	0.1 (1.0)
	Difference		−7.7 (−11.3, −4.1)	−7.7 (−11.1, −4.3)	−0.6 (−0.8, −0.4)
	Cohen’s d		−0.51 (−0.73, −0.3)	−0.59 (−0.82, −0.35)	−0.59 (−0.79, −0.39)
	P		<0.001	<0.001	<0.001
IMG	IMG	83	233 (18.5)	242 (15.6)	−1.1 (1.2)
	Non-IMG	677	242 (14.4)	253 (12.7)	0.2 (1.0)
	Difference		−9.3 (−13.7, −4.9)	−10.1 (−13.9, −6.4)	−1.2 (−1.5, −0.9)
	Cohen’s d		−0.62 (−0.86, −0.38)	−0.78 (−1.03, −0.53)	−1.22 (−1.45, −0.98)
	P		<0.001	<0.001	<0.001
Domestic URiM	URiM	94	236.7 (15.4)	246 (14)	−0.3 (1.0)
	Non-URiM	583	243 (14.1)	253 (12.2)	0.2 (0.9)
	Difference		−6.0 (−9.7, −2.4)	−7.1 (−10.7, −3.4)	−0.42 (−0.66, −0.18)
	Cohen’s d		−0.42 (−0.66, −0.18)	−0.57 (−0.83, −0.31)	−0.58 (−0.8, −0.36)
	P		0.002	<0.001	<0.001

USMLE = United States Medical Licensing Exam; URiM = Under-Represented in Medicine; IMG = International Medical Graduate. Values are mean (Standard Deviation). P-value is by two-sample t-test.

Fig. 1.

Score difference in USMLE and CASPer score performance by gender, URiM, IMG and domestic URiM. A dot represents the standardized mean difference (Cohen’s d) and its error bar represents its 95% confidence interval. IMG: International. URiM: Under-Represented in Medicine.

When comparing mean CASPer percentiles, there was no difference between the two genders; however, URiM performed 17% lower and IMG 32% lower compared to non-URiM and IMG applicants, respectively. This was a greater difference than either Step 1 or Step 2 for both groups (Table 2) (Fig. 2).

Table 2.

USMLE and CASPer median percentile score differences by gender, URiM, and IMG.

		Number	Step 1	Step 2	CASPer
Total		760
Gender	Female	312	60 (40–78.5)	54 (31–80)	55 (26–75.25)
	Male	448	60 (45–81)	50 (23–79)	55 (26–79)
	Difference		−9 (−10, 0)	0 (−10, 0)	0 (−4, 5)
	P		<0.001	0.226	0.787
URiM	Non-URiM	644	60 (40–81)	54 (42–80)	55 (29–80)
	URiM	116	50 (23–71)	42 (14–68)	29 (10.75–55.75)
	Difference		−11 (−20, −8)	−14 (−23, −11)	−17 (−24, −11)
	P		<0.001	<0.001	<0.001
IMG	Non-IMG	677	60 (50–81)	68 (42–80)	60 (33–82)
	IMG	83	40 (23–71)	31 (14–54)	12 (15–32)
	Difference		−14 (−21, −9)	−23 (−28, −14)	−32 (−39, −24)
	P		<0.001	<0.001	<0.001

USMLE = United States Medical Licensing Exam; URiM = Under-Represented in Medicine; IMG = International Medical Graduate. Values are median (Inter Quartile Range), P-value by Mann–Whitney U-test. Note that the difference in the median is the median of the difference between the two groups.¹⁴

Fig. 2.

Percentile score difference by gender, URiM, and IMG. A dot is a median of the percentile difference and its error bar is its 95% confidence interval. IMG: International. URiM: Under-Represented in Medicine.

An additional analysis was performed to determine if two applicants of the same Step score from each group (gender, URiM, IMG, domestic URiM) were expected to have the same CASPer score. The expected CASPer score, indicated by a linear regression line in Supplemental Appendix 1, was lower for URiM applicants than non-URiM applicants, either predicted by Step 1 or Step 2. IMG and domestic URiM applicants also had lower expected CASPer score than non-IMG and domestic non-URiM applicants. In addition, the gap in the expected CASPer score between domestic URiM and non-URiM applicants was accentuated with higher Step 1 scores (p = 0.084, Supplemental Appendix 2); between IMG and non-IMG applicants (p = 0.022); and between domestic URiM and non-URiM (p = 0.007). On the other hand, there was no significant interaction by groups between CASPer and Step 2 scores.

4. Discussion

This evaluation of ophthalmology residency applicants in 2022–2023 found significant levels of discrepancy in both traditional standardized testing and CASPer for URiM and IMG applicants. Utilizing the definition of bias as “a type of systematic error that can distort measurements and/or affect investigations and their results,”¹⁵ these discrepancies were biased. The bias was greater for CASPer than either USMLE exam, with URiM applicants on average scoring 17 percentile and IMG 32 percentile lower than their counterparts on CASPer. This study provides a comprehensive analysis of a captive population and suggests the Undergraduate and Graduate Medical communities should exercise caution when utilizing these tests individually or in combination for recruitment decisions.

In an era of increasing numbers of residency applications, holistic review is time- and resource-intensive for selection committees.¹⁶ Understandably, programs desire discrete and reproducible metrics to aid in decision-making while simultaneously conducting more equitable evaluation of applicants. Situational Judgment Tests hold promise to reduce bias, as they can be standardized, computer-based, and cost-effectively scalable to a large target population.¹⁷ A systematic review and meta-analysis of SJTs in medical selection through 2019 found considerable heterogeneity, but in general, modest incremental and predictive validity over academic or cognitive testing. At the same time, the review also found a wide variety of outcome measures utilized to provide evidence of the validity of these SJTs, highlighting the need for robust construct-relevant measures to be used in medical SJT validation;⁶ not only are different SJTs potentially measuring disparate attributes, it may be unclear what the tests are predicting.¹⁸ This raises concerns about the perpetuation of existing or introduction of new biases with these tests.¹⁹

The CASPer SJT was initially developed in Canada to correlate with the Multiple Mini-Interview.²⁰ Since then, it has been adopted by a growing number of medical schools in the United States as well as several GME programs.⁸ However, whether the SJT decreases bias for URiM applicants remains controversial. A simulation of applicants to the New York Medical College from the 2015–2016 cycle utilized different weighted combinations of Grade Point Average (GPA), MCAT, and CASPer z-scores. The authors found when CASPer received twice the weight of both GPA and Medical College Admissions Test (MCAT), the school would have admitted 3 additional African American and 8 more Hispanic students than utilizing GPA and MCAT alone. If each of the three measures received equal weighting, the numbers gained would have been only 2 and 1, respectively.⁹ Conversely, an evaluation of applicants interviewing at the University of North Carolina School of Medicine in 2018 and 2019 found URiM applicants may be further disadvantaged if CASPer was weighted in the screening process.⁷ To our knowledge, there exists no literature evaluating CASPer in United States GME recruitment. Our data demonstrate significant bias in the CASPer assessment for self-identified URiM and IMG applicants. Unlike the New York Medical College study,⁹ the differences were greater with CASPer than either of the traditional academic measures, Step 1 and Step 2, which suggests that any weighted combination would have induced further bias against these groups.

Recently, the Supreme Court of the United States reached a landmark ruling that prohibits the use of race-based affirmative action in college admissions.²¹ This decision will necessarily change admission processes at all levels of higher education, including within GME. As diversifying the physician work force is a significant factor in addressing health care disparities and access, we must do our best to recruit and retain URiM applicants in medicine.^22–24 With the Supreme Court decision, medical schools and residency programs may rely on conducting holistic reviews of candidate applications and implementing bias reduction practices during the recruitment and interview process. Our data indicates that CASPer may be introducing unwanted biases against diverse applicants instead, so we must exercise caution in adopting CASPer and other SJTs in our selection process — especially in a post-race-based affirmative action era in which we search for alternative methods to recruit diverse, talented applicants. Additionally, we may not even have a means to evaluate our success in this area without the demographic data needed to conduct this sort of analysis in the future, making the current analysis especially important to report.

There are limitations to this study. It is inclusive of only one GME specialty during a single recruitment cycle. The CASPer exam was novel for most applicants, and there could be a learning effect that decreases bias if the test is regularly utilized and more familiar to applicants in the future. Indeed, a coaching program is already available for CASPer in Canada and has demonstrated an increase in URiM examinee knowledge and self-perceived competency with the test.²⁵ These findings and limitations both indicate the need for additional investigation and scrutiny with continued use of SJTs in medical recruitment decisions.

5. Conclusions

These data demonstrate significant bias in the CASPer situational judgment test against URiM and IMG ophthalmology residency applicants in 2022–2023. While these findings are specific to one specialty, they suggest everyone invested in recruitment should approach SJTs with caution given the potential for introducing additional bias against underrepresented student groups.

Appendix

Supplemental Appendix 1.

Association of CASPer Z-score with Step 1 and 2 scores. A linear regression was fitted on CASPer Z-score with Step 1 or Step 2 score, stratified by a group. Values are beta coefficient, a change in CASPer Z-score per 10 score change in Step 1 or Step 2. P for interaction indicates whether these linear trends (β) are different by groups.

β for CASPer Z-score per 10 unit of step score and 95% CI	Male	Female	P for interaction
Step 1	0.16 (0.1, 0.23)	0.2 (0.12, 0.28)	0.467
Step 2	0.2 (0.13, 0.28)	0.31 (0.21, 0.41)	0.093
	Non URiM	URiM
Step 1	0.18 (0.12, 0.23)	0.06 (−0.06, 0.18)	0.084
Step 2	0.22 (0.16, 0.29)	0.18 (0.04, 0.32)	0.600
	Non-IMG	IMG
Step 1	0.11 (0.05, 0.16)	0.26 (0.12, 0.4)	0.022
Step 2	0.17 (0.11, 0.23)	0.24 (0.06, 0.41)	0.411
	Domestic non-URiM	Domestic URiM
Step 1	0.12 (0.06, 0.18)	−0.08 (−0.22, 0.06)	0.007
Step 2	0.16 (0.09, 0.22)	0.1 (−0.07, 0.26)	0.495

Supplemental Appendix 2.

A scatter plot between CASPer Z-score and Step 1 or 2 scores. A line in color represents a fitted linear regression and a shaded region represents 95% CI of the prediction.