This study describes the adoption of bias reduction practices in underrepresented groups in ophthalmology residency recruitment and attempts to determines which practices are effective for increasing these residents.
Key Points
Question
What is the current state of bias reduction practices in US ophthalmology residency recruitment and which practices are associated with resident diversity?
Findings
In this survey study that included over 60% of US ophthalmology residency programs, use of multiple bias reduction tools provided to selection committees was positively associated with increased resident diversity, but use of interview rubrics was negatively associated.
Meaning
Use of multiple bias reduction tools and critical reevaluation of interview rubrics may help increase ophthalmology resident diversity.
Abstract
Importance
Best recruitment practices for increasing diversity are well established, but the adoption and impact of these practices in ophthalmology residency recruitment are unknown.
Objective
To describe the adoption of bias reduction practices in groups underrepresented in ophthalmology (URiO) residency recruitment and determine which practices are effective for increasing URiO residents.
Design, Setting, and Participants
This cross-sectional survey study used an 18-item questionnaire included in the online survey of the Association of University Professors in Ophthalmology (AUPO) Residency Program Directors. Data collection occurred from July 2022 to December 2022. The data were initially analyzed on January 16, 2023. Participants included residency program directors (PDs) in the AUPO PD listserv database.
Main Outcomes and Measures
Descriptive analysis of resident selection committee approaches, evaluation of applicant traits, and use of bias reduction tools. Primary outcome was diversity assessed by presence of at least 1 resident in the last 5 classes who identified as URiO, including those underrepresented in medicine (URiM), lesbian, gay, bisexual, transgender, queer, intersex, and asexual plus, or another disadvantaged background (eg, low socioeconomic status). Multivariate analyses of recruitment practices were conducted to determine which practices were associated with increased URiO and URiM.
Results
Among 106 PDs, 65 completed the survey (61.3%). Thirty-nine PDs used an interview rubric (60.0%), 28 used interview standardization (43.0%), 56 provided at least 1 bias reduction tool to their selection committee (86.2%), and 44 used postinterview metrics to assess diversity, equity, and inclusion efforts (67.7%). Application filters, interview standardization, and postinterview metrics were not associated with increased URiO. Multivariate logistic regression analysis showed larger residency class (odds ratio [OR], 1.34; 95% CI, 1.09-1.65; P = .01) and use of multiple selection committee bias reduction tools (OR, 1.47; 95% CI, 1.13-1.92; P = .01) were positively associated with increased URiO, whereas use of interview rubrics (OR, 0.72; 95% CI, 0.59-0.87; P = .001) and placing higher importance of applicant interest in a program (OR, 0.83; 95% CI, 0.75-0.92; P = .02) were negatively associated. URiM analyses showed similar associations.
Conclusions and Relevance
Ophthalmology residency interviews are variably standardized. In this study, providing multiple bias reduction tools to selection committees was associated with increased URiO and URiM residents. Prioritizing applicant interest in a program may reduce resident diversity. Interview rubrics, while intended to reduce bias, may inadvertently increase inequity.
Introduction
Ophthalmology remains one of the least diverse fields in medicine, with studies showing substantial underrepresentation with respect to race, ethnicity, and gender in the residency applicant, resident, faculty, and leadership levels.1,2 Among US medical school matriculants in 2019, 21.5% were underrepresented in medicine (URiM), defined by the American Association of Medical Colleges (AAMC) as those who identify as Black or African American, Hispanic or Latinx/o/a, Native American, and/or Pacific Islander.3 This mirrored the US population in 2019, when 18.5% identified as Hispanic and 12.2% as Black.4 However, only 5.8% of ophthalmologist residents identified as URiM the same year.5 According to the AAMC medical student matriculant questionnaire, 15.9% identified as lesbian, gay, bisexual, transgender, queer, intersex, and asexual plus (LGBTQIA+) in 2023.6 Though understudied, there is low LGBTQIA+ representation in ophthalmology, with LGBTQIA+ trainees and ophthalmologists reporting challenges secondary to burnout and discrimination.7,8,9 Other factors, such as parental income and education, are also predictors of entry into medicine. Only 11.4% of medical school matriculants were first-generation college graduates in 2019, with students from lower socioeconomic backgrounds facing numerous challenges in the residency application process.3,10
Growing representation in ophthalmology is crucial for several reasons. It has been shown that patients report greater care satisfaction when care teams have diverse life experiences and backgrounds, particularly when there is racial concordance between patients and health care professionals.11 Diversity within health care teams enhances our understanding of social determinants of health, which may further improve patient adherence to treatment.12,13 Diverse physicians are also more likely to work with underserved communities, improving health equity and access to care.14
Increasing workforce diversity in ophthalmology requires examining and replacing the conventional system of recruitment and selection that have amplified barriers toward underrepresented groups. Strategies at the residency application level include holistic application review and bias reduction during the interview process. The AAMC and Accreditation Council for Graduate Medical Education (ACGME) have curated best practices for holistic recruitment and the promotion of diversity, equity, and inclusion (DEI).15,16 Standardized or structured interviews, in which the same question is asked to all applicants, is one of the most effective methods.17,18,19,20 Selection committee implicit bias training21,22,23 and blinded interviews (applicant file is not reviewed by interviewers)24 have also shown to reduce bias in other specialties. Scoring rubrics, either to prescreen applicants or to assess them during the interview, provide further standardization and reduce affinity and conformity biases.19,20,23
The prevalence of bias reduction practices in ophthalmology is unknown. Furthermore, it is unclear if any specific bias reduction practice is effective in increasing representation in ophthalmology. This study aims to better understand ophthalmology residency interview structures and adoption of bias reduction practices in residency recruitment, ultimately to identify best practices to increase diversity.
Methods
Study Design
An anonymous, voluntary 18-item online questionnaire was developed on Qualtrics by 2 authors (O.E.U. and A.F.). It was sent to a pilot group consisting of the current residency program director (PD), 2 current associate PDs, and 2 past PDs of Oregon Health & Science University Casey Eye Institute, and was edited with their input. The questionnaire was approved for distribution to the Association of University Professors of Ophthalmology (AUPO) Program Director Council listserv and consisted of 106 residency PDs in the US. The survey had 5 sections: demographic information, approach to selection committee composition, use of application filters, desired applicant qualities, interview structure and applicant ranking, and bias reduction tools (eFigure in Supplement 1). The list of bias reduction tools was curated by the authors from AAMC best practices and the anti-bias literature, with an optional free text to report other practices (eTable 1 in Supplement 1). Only interview scoring rubrics, not preinterview rubrics, were assessed. Each PD reported if they had at least 1 resident in the last 5 classes who identified as underrepresented in ophthalmology (URiO), a term we used to include URiM, LGBTQIA+, or another disadvantaged background determined by each PD with required explanation in a free text box. Data were collected during July through December 2022, with 2 individual reminder emails sent to each PD on the listserv 2 weeks and 2 months after initial survey distribution.
The study followed the American Association for Public Opinion Research guidance for institutional review boards and survey researchers and approved by Oregon Health & Science University’s institutional review board. Individual electronic consent to participate was obtained from each participant and no stipend or other incentive to participate was provided.
Statistical Analysis
Descriptive analysis was used to summarize demographic information. Primary dependent variable was the total number of URiO identities (question 18) reported as a score on a scale from 0 to 5, 1 point each for Black or African American, Hispanic or Latinx/a/o, LGBTQIA+, Native American or Pacific Islander, and other, as well as URiM identities (question 18) reported as a score from 0 to 3, 1 point each for Black or African American, Hispanic or Latinx/a/o, and Native American or Pacific Islander. We compared parameters of interest between programs with high (3 or higher for URiO and 2 or higher for URiM) and low scores, in addition to identifying predictors of select URiO identities.
The Mann-Whitney U test was used for continuous independent variables and χ2 test for categorical independent variables. One-way analysis of variance analyses of predictors of high URiO and URiM were conducted for ranked variables (question 3, question 7, and question 10) and logistic regression for all other variables. Multiple logistic regression analyses were conducted to determine factors associated with high URiO and URiM, with variables meeting significance of P < .10 in univariate analyses included in multivariate analyses. Analysis was performed using XLSTAT for Microsoft (Microsoft Corp). All P values were 2-sided and not adjusted for multiple comparisons.
Results
Demographics and Application Filters
A total of 65 of 106 respondents (61.3%) were included in the study (Table 1). Mean number of residents per class was (SD, 1.3) 4.5 (range, 2-8). Within the past 5 classes, 47 had Black or African American resident (72.3%), 53 reported having a Hispanic or Latinx/a/o resident (81.5%), 9 had Native American or Pacific Islander (13.8%), and 39 had a resident who identified as LGBTQIA+ (60.0%). Thirty-nine residents were identified as a member of a disadvantaged background (60.0%), which included first-generation college students (14 [21.5%]), upbringing in a rural background (8 [12.3%]), international medical graduates or non–US citizens (6 [9.2%]), and low socioeconomic status (3 [4.6%]). Only 1 program reported no URiO residents (1.5%).
Table 1. Demographic Characteristics and Application Filters With Underrepresented in Ophthalmology (URiO) and Underrepresented in Medicine (URiM) Scores.
| Characteristic | Study cohort (n = 65) (%) | URiO score, mean (SD) | P value | URiM score, mean (SD) | P value |
|---|---|---|---|---|---|
| Locationa | |||||
| East South Central | 7 (10.8) | 3.0 (1.4) | .99b | 1.6 (1.0) | .35b |
| West South Central | 7 (10.8) | 2.9 (0.9) | 1.6 (0.5) | ||
| East North Central | 15 (23.1) | 2.8 (1.1) | 1.6 (0.7) | ||
| West North Central | 8 (12.3) | 2.9 (1.5) | 1.7 (0.5) | ||
| New England | 3 (4.6) | 2.7 (0.6) | 1.0 (1.0) | ||
| Mid-Atlantic | 6 (9.2) | 3.0 (0.6) | 2.0 (0.6) | ||
| South Atlantic | 11 (16.9) | 2.7 (0.9) | 1.6 (0.5) | ||
| Mountain | 3 (4.6) | 2.3 (0.6) | 2.0 (0.0) | ||
| Pacific/Puerto Rico | 5 (7.7) | 3.4 (1.3) | 1.8 (1.1) | ||
| Application filters | |||||
| AOA Honor Society membership | |||||
| Yes | 21 (32.3) | 2.8 (1.6) | .66 | 1.6 (0.7) | .79 |
| No | 44 (67.7) | 2.9 (1.7) | 1.7 (0.7) | ||
| Doctor of Osteopathic Medicine status | |||||
| Yes | 16 (24.6) | 2.9 (1.6) | .74 | 1.6 (0.7) | .90 |
| No | 49 (75.4) | 2.8 (1.6) | 1.6 (0.7) | ||
| Gold Humanism Honor Society membership | |||||
| Yes | 23 (35.4) | 2.8 (1.6) | .92 | 1.6 (0.8) | .66 |
| No | 42 (64.6) | 2.9 (1.7) | 1.7 (0.7) | ||
| International medical graduate status | |||||
| Yes | 26 (40.0) | 3 (1.1) | .43 | 1.6 (0.7) | .93 |
| No | 39 (60.0) | 2.7 (1.1) | 1.6 (0.8) | ||
| Medical school grades | |||||
| Yes | 26 (40.0) | 3.0 (1.0) | .48 | 1.7 (0.8) | .44 |
| No | 39 (60.0) | 2.8 (1.2) | 1.6 (0.7) | ||
| Medical school reputation/rank | |||||
| Yes | 17 (26.2) | 3.0 (0.9) | .66 | 1.6 (0.8) | .90 |
| No | 48 (73.8) | 2.8 (1.2) | 1.6 (0.7) | ||
| Reapplicant status | |||||
| Yes | 11 (16.9) | 2.9 (1.1) | .39 | 1.8 (0.6) | .49 |
| No | 54 (83.1) | 2.8 (1.1) | 1.6 (0.8) | ||
| USMLE scores | |||||
| Yes | 35 (53.8) | 2.7 (1.1) | .55 | 1.6 (0.8) | .98 |
| No | 30 (46.2) | 2.9 (1.1) | 1.6 (0.7) | ||
| None of the above | 19 (29.2) | NA | NA | NA | NA |
| No. of application filters, mean (range) | 2.7 (0-8) | NA | NA | NA | NA |
| No. of interview invitations, mean (range) | 66 (48-90) | NA | NA | NA | NA |
| No. of residents per class, mean (SD) | 4.5 (1.3) | NA | NA | NA | NA |
Abbreviations: AOA, Alpha Omega Alpha; NA, not applicable; USMLE, United States Medical Licensing Examination.
See eFigure in Supplement 1 for states located in each geographic region.
Comparison of Central US (East/West/North/South Central) with remainder of locations.
Forty-eight PDs used preapplication filters to screen applicants (73.8%). The most common application filters were US Medical Licensing Examination scores (35 [53.8%]). A total of 17 PDs reported not using any application filters (26.2%). Use of application filters or the number of application filters was not associated with increased URiO or URiM (Table 1).
Selection Committee Composition and Applicant Characteristics
Several factors were rated as most or very important for PDs when composing their selection committee. Representation of residency program leadership was ranked by 35 PDs in this high-importance category (53.8%). This was followed by ethnic diversity and representation of people involved in resident education (both 34 [52.3%]), gender diversity (33 [50.8%]), subspecialty representation (19 [29.2%]), diversity in age (14 [21.5%]), and residency alumni representation (2 [3.1%]) (eTable 2 in Supplement 1). Perceived importance of gender of selection committee composition were similar between programs with high or low URiO. URiM analysis showed similar associations.
When asked to rank desired applicant qualities, most PDs ranked excellence in clinical rotations as the most important factor (26 [40.0%]), followed by having a growth mindset (13 [20.0%]). Demonstrated interest in the program was ranked most important by 6 PDs (9.2%) (eTable 2 in Supplement 1). Compared with programs that did not have an LGBTQIA+ resident, those that did placed more importance on selecting applicants underrepresented in ophthalmology (odds ratio [OR], 1.53; 95% CI, 1.05-2.23; P = .03).
Interview Goals
Thirty-one PDs ranked the interview as the most important factor in forming their rank list (47.7%), with a mean score of 1.7 (SD, 0.8; range 1.0-4.0). Mean number of applicants interviewed was (SD, 15.3) 54.4 (range, 18.0-90.0). Most PDs reported the most important goal of the interview was selecting the most qualified candidates for their program (30 [46.2%]) (eTable 2 in Supplement 1). Compared with programs without, those with Black or African American (U = 336.0; P = .11), Hispanic or Latinx/a/o (U = 238.5; P = .08), or LGBTQIA+ (U = 551.5; P = .75) residents placed less emphasis on selecting candidates with the best fit.
Bias Reduction Strategies
A total of 28 PDs reported interview standardization (43.1%), with 24 PDs reporting standardization over 50% of the interview. URiO scores between PDs who used interview standardization and those who did not were similar (Table 2). However, stratified analysis revealed PDs who used more than 50% standardization of the interview were more likely to have high URiO scores (OR, 1.2; 95% CI, 0.95-1.78; P = .08).
Table 2. Comparison of Bias Reduction Tools Used in Programs With High and Low Underrepresented in Ophthalmology (URiO) and Underrepresented in Medicine (URiM) Scores.
| Characteristic | Study cohort (n = 65) | URiO | URiM | ||||
|---|---|---|---|---|---|---|---|
| <3 (n = 26) | ≥3 (n = 39) | P value | <2 (n = 26) | ≥2 (n = 39) | P value | ||
| Use of interview standardization | 38 (58.5) | 14 (53.8) | 24 (61.5) | NA | 15 (57.7) | 23 (59.0) | NA |
| 0%-25% | 3 (4.6) | 3 (11.5) | 0 | .54 | 2 (7.7) | 1 (4.3) | .62 |
| 25%-50% | 11 (16.9) | 4 (15.4) | 7 (18.0) | 5 (19.2) | 6 (15.4) | ||
| 50%-75% | 13 (20.0) | 3 (11.5) | 10 (25.6) | 4 (15.4) | 9 (23.1) | ||
| >75% | 11 (16.9) | 4 (15.4) | 7 (18.0) | 4 (15.4) | 7 (17.9) | ||
| Use of rubric/scoring tool | 39 (60.0) | 22 (84.6) | 17 (43.6) | <.001 | 19 (73.1) | 20 (51.3) | .08 |
| Use of selection committee bias reduction tool | 56 (86.2) | 20 (77.0) | 36 (92.3) | 19 (73.1) | 37 (94.9) | ||
| AAO/AUPO resources | 35 (53.8) | 12 (46.2) | 23 (59.0) | .08 | 11 (42.3) | 24 (61.5) | .01 |
| Departmental/institutional workshop interactive/in person | 32 (49.2) | 9 (34.6) | 23 (59.0) | 10 (38.5) | 22 (56.4) | ||
| Departmental/institutional workshop online | 37 (56.9) | 13 (50.0) | 24 (61.5) | 8 (30.8) | 29 (74.4) | ||
| Journal club | 22 (33.8) | 6 (23.1) | 16 (41.0) | 9 (34.6) | 13 (33.3) | ||
| Web-based (eg, IAT or other online module) | 33 (50.8) | 11 (42.3) | 22 (56.4) | 10 (38.5) | 23 (59.0) | ||
| Other | 6 (9.2) | 1 (3.8) | 5 (12.8) | 1 (3.8) | 5 (12.8) | ||
| Mean No. of selection committee bias reduction tools (SD) | 2.5 (1.5) | 2.0 (1.5) | 2.8 (1.5) | .03 | 1.8 (1.6) | 2.9 (1.3) | .07 |
| Use of postinterview metrics | 44 (67.7) | 18 (69.2) | 26 (66.7) | 17 (65.4) | 27 (69.2) | ||
| Review of diversity in residency classes | 42 (64.6) | 16 (61.5) | 26 (66.7) | .83 | 15 (57.7) | 27 (69.2) | .75 |
| DEI responses in ACGME survey | 23 (35.4) | 9 (34.6) | 14 (35.9) | 7 (26.9) | 16 (41.0) | ||
| Other | 6 (9.2) | 4 (15.4) | 2 (5.1) | 2 (7.7) | 2 (5.1) | ||
Abbreviations: AAO, American Academy of Ophthalmology; ACGME, Accreditation Council for Graduate Medical Education; AUPO, Association for University Professors of Ophthalmology; DEI, diversity, equity, and inclusion; IAT, Implicit Association Test; NA, not applicable.
Interview scoring rubrics were used by 39 PDs (60.0%). Programs with high URiO scores used interview rubrics less than programs with lower scores (OR, 0.13 95% CI, 0.04-0.46; P = .001). Similarly, programs with high URiM scores used rubrics less (OR, 0.37; 95% CI, 0.13-1.07; P = .07) (eTable 3 in Supplement 1).
Fifty-six PDs (86.2%) reported providing at least 1 bias reduction tool to their selection committee with departmental and/or institutional online modules (37 [56.9%]) and AUPO/American Academy of Ophthalmology resources, such as the DEI Education Page (35 [53.8%]) being most common.25 The mean number of selection committee bias reduction tools used was 2.5 (range, 1-5). More programs that used bias reduction tools reported high URiO (OR, 1.28; 95% CI, 1.00-1.86; P = .05) and URiM scores (OR, 1.54; 95% CI, 1.05-2.59; P = .03) (eTable 3 in Supplement 1). In addition, a large proportion of PDs with Black or African American residents used at least 1 tool compared with those who did not (n = 65; χ21 = 9.76; P = .002). Programs with high URiO and URiM scores reported use of more selection committee bias reduction tools compared with those with low scores (URiO, U = 362.5; P = .03 and URiM, U = 317.5; P = .01) (Table 2).
A total of 44 PDs used postinterview metrics to assess their DEI efforts (67.7%). Specific metrics included analysis of URiO trend in residency applicants and classes (42 [64.6%]) and evaluation of DEI question responses in the ACGME annual survey completed by residents and faculty (23 [35.4%]). Within other responses, review of US census data for their region and comparison with recruitment statistics was used by 2 PDs (3.1%). There was no difference in use of postinterview assessments between programs with high and low URiO (n = 65; χ21 = 0.05; P = .83) or in those with high and low URiM scores (n = 65; χ21 = 0.11; P = .75) (Table 2).
Predictors of Resident Diversity
Univariate analysis demonstrated several parameters associated with increased URiO and URiM (eTables 3 and 4 in Supplement 1), including number of residents per class and use of selection committee bias reduction tools. Notably, programs that prioritized demonstrated interest by applicant were more likely to have lower URiO (F = 2.87; P = .02) and URiM (F = 3.79; P < .001) scores (eTable 3 in Supplement 1).
Multivariate logistic regression analyses showed that prioritizing demonstrated interest in a program (OR, 0.83; 95% CI, 0.75-0.92; P = .02) and using an interview rubric (OR, 0.72; 95% CI, 0.59-0.87; P = .001) were negatively associated with URiO scores. Increased number of selection committee bias reduction tools (OR, 1.47; 95% CI, 1.13-1.92; P = .01) and larger residency class size (OR, 1.34; 95% CI, 1.09-1.65; P = .01) were predictors of increased URiO (Figure). URiM analyses showed similar associations.
Figure. Multivariate Logistic Regression Analyses of Predictors of High Underrepresented in Ophthalmology and Underrepresented in Medicine Residents.
OR indicates odds ratio.
Discussion
The use of bias reduction practices in ophthalmology residency recruitment is varied. To our knowledge, this study is the first to explore interview structures in ophthalmology, as well as investigate the association of bias reduction strategies on program diversity. We used the term URiO, which expanded on the definition of diversity and found each unit increase in residency class size and selection committee bias reduction tool was associated with 34% and 47% more likelihood of a high URiO score, respectively. Those who used interview rubrics and those who prioritized applicant interest in the program were 28% and 17% less likely to have a high URiO score, respectively. Our findings highlight the complexity of bias reduction tools and the impact certain applicant characteristics may have in residency selection.
There were notable differences in the perceived importance of applicant qualities and interview goals among residency programs. Programs with LGBTQIA+ residents placed more importance on selecting applicants URiO, which may explain the strong concordance of URiO and URiM scores. PDs in programs with more diverse representation also placed less emphasis on selecting the candidate with the right fit. This term can promote homogeneity due to affinity or pedigree biases instead of diversity of attributes and life experience. Creating a culture of belonging, rather than encouraging fit with what we decide to be a norm, is the basis of DEI. A resident selection process that looks for an applicant who would add to their program rather than one who would fit into the program may reduce harmful biases that limit opportunities for both applicants and the program. Diverse selection committees may further help ensure against replicating narrow sets of traits and assets in the selection process.
Our results show that providing selection committees with multiple bias reduction tools rather than a single tool and having a larger residency class are positively associated with high URiO. Although, to our knowledge, no study previously has analyzed the association of specific bias reduction tools in residency recruitment, our findings support the effectiveness of holistic review, designated as an AAMC and ACGME best recruitment practice and used by other specialties.15,16,26 In internal medicine, a pilot standardized interview program raised the proportion of URiM applicants from 16.0% to 24.5% and proportion of matriculating URiM residents from 12.5% to 31.7%.27 A holistic review program assigning value to lived experience and de-emphasizing USMLE scores increased URiM representation in psychiatry.28 Though applicant filters and interview standardization were not associated with high URiO in multivariate analysis, we found PDs who used more than 50% interview standardization were more likely to have high URiO scores. This study did not explore preinterview rubrics or other screening strategies, so further research in the effect of these practices on representation is warranted.
Our study found emphasis of applicant interest in a program to be negatively associated with resident diversity. Though information on applicant interest may be valuable, prioritizing this can have unintended negative consequences. URiO applicants may have less access to program opportunities or mentors who can advocate on their behalf. In our personal experience not examined in this survey, URiO applicants were much less likely to have messages sent on their behalf informing the program of their interest from a mentor. We believe selection committees should be aware of this and ensure applicants are considered and evaluated equitably.
The use of the interview rubric was also associated with lower representation of URiO. This is a surprising finding, as creating a predetermined rubric that considers qualifications is a standard recommendation as a bias reduction tool. While we did not examine the specifics of such a rubric, it is possible that certain rubrics create rigid systems that are biased against URiO applicants, especially if the rubrics are constructed to favor traditional measures of achievement, such as publications and awards, but do not take into consideration life experiences that demonstrate resilience and resourcefulness. Conventional bias-reduction wisdom of creating rubrics may not be flawed, but the rubric must reflect a commitment to holistic review to be effective. Fortunately, many residency programs are placing more importance on distance traveled in application review.29 While national standardization of rubrics would be impractical, these rubrics can be screened by an ACGME and/or AUPO–endorsed third-party organization to aid in question creation, scoring methodology, and bias reduction. Webinars on rubric formation can be provided to PDs through AUPO or a third-party organization. By ensuring the proper design of rubrics and interview questions, we can strive to make the interview process more equitable.
Limitations
The study had several limitations. Though our study had a high survey response rate, the sample population may have been biased and there may be PDs who are not on the AUPO PD listserv, which may contribute to sampling bias. Some residency programs may use a prescreening rubric, which was not investigated in the study. Similarly, we did not ask PDs if their rubrics were traditional or holistic, as these terms are subjective without clear definitions. This may have permitted further analyses and changed the interpretation of rubric use in our study. The selection of residents with a disadvantaged background was at the discretion of each PD. Though we did not perform subanalyses on this group due to response variety, we showed that over one-third of PDs reported having a resident whom they identified as disadvantaged—a statistic that underscores need for further research. The survey was only completed by PDs who may not share the same views as other selection committee members. Only the presence of underrepresented groups, rather than the total number of underrepresented residents, was studied to provide ease of recall to the PD, so this analysis could not identify strategies to increase the number of underrepresented residents in each URiO category. The survey assumed the PD was aware of resident ethnicity, identity, and sexual orientation in the past 5 classes, which is prone to recall bias and may result in inaccuracies in URiO reported. Lastly, our study evaluated associations and cannot determine, directly, a cause-and-effect relationship.
Conclusions
Overall, our study suggests that increasing the number of selection committee bias reduction tools, reducing importance placed on an applicant’s interest in the program, and critically evaluating interview rubrics are potential strategies that can be implemented by PDs to increase resident diversity. Though interview standardization was not associated with high URiO in multivariate analysis, we found PDs who standardized more than half of their interviews were more likely to have high URiO scores. Interview standardization also has demonstrated effectiveness in the literature, so we recommend programs consider at least partial standardization of their residency interviews. More research on interview rubrics is needed to address their potential contribution to bias in the residency selection process.
eFigure. Survey Questions
eTable 1. List of bias reduction tools and rationale behind their use (page 5)
eTable 2. Descriptive statistics of selection committee composition, interview structure, and applicant ranking (Survey sections 2 and 4)
eTable 3. Univariate analysis of variance (ANOVA) of rank-based survey questions and URiO and URiM
eTable 4. Univariate logistic regression analyses of program size and bias reduction tools on resident diversity
Data sharing statement
References
- 1.Fairless EA, Nwanyanwu KH, Forster SH, Teng CC. Ophthalmology departments remain among the least diverse clinical departments at United States medical schools. Ophthalmology. 2021;128(8):1129-1134. doi: 10.1016/j.ophtha.2021.01.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Gill HK, Niederer RL, Shriver EM, Gordon LK, Coleman AL, Danesh-Meyer HV. An eye on gender equality: a review of the evolving role and representation of women in ophthalmology. Am J Ophthalmol. 2022;236:232-240. doi: 10.1016/j.ajo.2021.07.006 [DOI] [PubMed] [Google Scholar]
- 3.Association of American Medical Colleges . Applicants, matriculants, enrollment, graduates, MD-PhD, and residency applicants data. Accessed February 22, 2024. https://www.aamc.org/data-reports/students-residents/report/facts
- 4.United States Census Bureau Population Statistics . Population Estimates by Age. Sex, Race and Hispanic Origin; 2019. [Google Scholar]
- 5.Aguwa UT, Srikumaran D, Canner J, et al. Trends in racial and ethnic diversity of ophthalmology residents and residency applicants. Am J Ophthalmol. 2022;240:260-264. doi: 10.1016/j.ajo.2022.03.006 [DOI] [PubMed] [Google Scholar]
- 6.Association of American Medical Colleges . Matriculating student questionnaire (MSQ). Accessed February 22, 2024. https://www.aamc.org/data-reports/students-residents/report/matriculating-student-questionnaire-msq
- 7.Duong AT, Law JC, Ramirez DA, Woreta F, Sun G. Advancing lesbian, gay, bisexual, transgender, and queer (LGBTQ+) diversity, equity, and inclusion in ophthalmology. Ophthalmology. 2022;129(11):1232-1234. doi: 10.1016/j.ophtha.2022.07.021 [DOI] [PubMed] [Google Scholar]
- 8.Chang TC, A R, Candelario C, et al. LGBTQ+ identity and ophthalmologist burnout. Am J Ophthalmol. 2023;246:66-85. doi: 10.1016/j.ajo.2022.10.002 [DOI] [PubMed] [Google Scholar]
- 9.Kelly T, Rodriguez SB. Expanding underrepresented in medicine to include lesbian, gay, bisexual, transgender, and queer individuals. Acad Med. 2022;97(11):1605-1609. doi: 10.1097/ACM.0000000000004720 [DOI] [PubMed] [Google Scholar]
- 10.Association of American Medical Colleges . An Updated Look at the Economic Diversity of US Medical Students. Accessed February 22, 2024. https://www.aamc.org/data-reports/analysis-brief/report/updated-look-economic-diversity-us-medical-students
- 11.Jetty A, Jabbarpour Y, Pollack J, Huerto R, Woo S, Petterson S. Patient-physician racial concordance associated with improved healthcare use and lower healthcare expenditures in minority populations. J Racial Ethn Health Disparities. 2022;9(1):68-81. doi: 10.1007/s40615-020-00930-4 [DOI] [PubMed] [Google Scholar]
- 12.Woreta FA, Gordon LK, Knight OJ, Randolph JD, Zebardast N, Pérez-González CE. Enhancing diversity in the ophthalmology workforce. Ophthalmology. 2022;129(10):e127-e136. doi: 10.1016/j.ophtha.2022.06.033 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.McQuaid EL, Landier W. Cultural issues in medication adherence: disparities and directions. J Gen Intern Med. 2018;33(2):200-206. doi: 10.1007/s11606-017-4199-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Moy E, Bartman BA. Physician race and care of minority and medically indigent patients. JAMA. 1995;273(19):1515-1520. doi: 10.1001/jama.1995.03520430051038 [DOI] [PubMed] [Google Scholar]
- 15.Association of American Medical Colleges . Best practices for conducting residency program interviews. Accessed February 22, 2024. https://www.aamc.org/services/admission-interview-foundations/residency
- 16.Accreditation Council for Graduate Medical Education . ACGME equity matters holistic recruitment toolkit. Accessed February 22, 2024. https://dl.acgme.org/learning-paths/acgme-equity-matters-holistic-review-toolkit
- 17.Marcus-Blank B, Dahlke JA, Braman JP, et al. Predicting performance of first-year residents: correlations between structured interview, licensure exam, and competency scores in a multi-institutional study. Acad Med. 2019;94(3):378-387. doi: 10.1097/ACM.0000000000002429 [DOI] [PubMed] [Google Scholar]
- 18.Black C, Budner H, Motta AL. Enhancing the residency interview process with the inclusion of standardised questions. Postgrad Med J. 2018;94(1110):244-246. doi: 10.1136/postgradmedj-2017-135385 [DOI] [PubMed] [Google Scholar]
- 19.Bergelson I, Tracy C, Takacs E. Best practices for reducing bias in the interview process. Curr Urol Rep. 2022;23(11):319-325. doi: 10.1007/s11934-022-01116-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Costa PCGA. Strategies to increase diversity in surgical residency. Curr Surg Rep. 2021;9(5):11. doi: 10.1007/s40137-021-00288-4 [DOI] [Google Scholar]
- 21.Kallianos KG, Webb EM, Hess CP, Talbott J, Bucknor MD. Use of the implicit association test to improve diversity in radiology. J Am Coll Radiol. 2019;16(7):976-979. doi: 10.1016/j.jacr.2019.01.010 [DOI] [PubMed] [Google Scholar]
- 22.Phillips MR, Charles A. Addressing implicit bias in the surgical residency application and interview process for underrepresented minorities. Surgery. 2021;169(6):1283-1284. doi: 10.1016/j.surg.2021.01.018 [DOI] [PubMed] [Google Scholar]
- 23.Chi B, Krull C, Akamagwuna U, Huang D. Incorporating holistic review in recruitment in a physical medicine and rehabilitation residency program. Am J Phys Med Rehabil. 2022;101(9):859-863. doi: 10.1097/PHM.0000000000002000 [DOI] [PubMed] [Google Scholar]
- 24.Haag J, Sanders BE, Walker Keach J, Lefkowits C, Sheeder J, Behbakht K. Impact of blinding interviewers to written applications on ranking of gynecologic oncology fellowship applicants from groups underrepresented in medicine. Gynecol Oncol Rep. 2022;39:100935. doi: 10.1016/j.gore.2022.100935 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.American Academy of Ophthalmology . Diversity and inclusion education. Accessed February 22, 2024. https://www.aao.org/education/diversity-and-inclusion-education
- 26.Marbin J, Rosenbluth G, Brim R, Cruz E, Martinez A, McNamara M. Improving diversity in pediatric residency selection: using an equity framework to implement holistic review. J Grad Med Educ. 2021;13(2):195-200. doi: 10.4300/JGME-D-20-01024.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Aibana O, Swails JL, Flores RJ, Love L. Bridging the gap: holistic review to increase diversity in graduate medical education. Acad Med. 2019;94(8):1137-1141. doi: 10.1097/ACM.0000000000002779 [DOI] [PubMed] [Google Scholar]
- 28.Barceló NE, Shadravan S, Wells CR, et al. Reimagining merit and representation: promoting equity and reducing bias in GME through holistic review. Acad Psychiatry. 2021;45(1):34-42. doi: 10.1007/s40596-020-01327-5 [DOI] [PubMed] [Google Scholar]
- 29.McFarling UL. How one medical school became remarkably diverse—without considering race in admissions. Accessed February 22, 2024. https://www.statnews.com/2023/03/07/how-one-medical-school-became-remarkably-diverse-without-considering-race
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eFigure. Survey Questions
eTable 1. List of bias reduction tools and rationale behind their use (page 5)
eTable 2. Descriptive statistics of selection committee composition, interview structure, and applicant ranking (Survey sections 2 and 4)
eTable 3. Univariate analysis of variance (ANOVA) of rank-based survey questions and URiO and URiM
eTable 4. Univariate logistic regression analyses of program size and bias reduction tools on resident diversity
Data sharing statement
