Factors influencing emergency medicine residency choice: Diversity, community, and recruitment red flags

Abstract

Background

Emergency medicine (EM) applicants consider many factors when selecting residency programs. Prior studies have demonstrated that applicants consider geography as well as modifiable/nonmodifiable program factors. Less attention, however, has been paid to underrepresented groups. Additionally, the prevalence and characteristics of “red flags,” or factors that may lead an applicant to lower a program's rank or not rank it at all, remain unknown in EM. Our objective was to describe the factors that influence current EM‐bound medical students’ residency selection focusing on underrepresented applicants and red flags encountered during the recruitment process.

Methods

We conducted a mixed‐methods survey study of EM‐bound graduates from U.S. medical schools in the 2020 application cycle. Quantitative analysis included descriptive statistics, measures of central tendency, 95% confidence intervals (CIs), nonparametric tests for ordinal data, and logistic regression. For the qualitative portion of the study, two independent reviewers performed a thematic analysis of the red flag free‐text responses. Discrepancies were addressed via consensus with third‐party oversight.

Results

Our survey response rate was 49%, and most applicants considered both geographic and program factors. Underrepresented applicants prioritized program diversity, program commitment to the underserved, neighborhood/community, and patient population. Of all respondents, 71% reported red flags. Women had a significantly higher odds of encountering red flags (odds ratio = 1.64, 95% CI = 1.25 to 2.18). Red flags included seven key themes: violations of regulatory standards, program characteristics, interview day experience, program culture, interpersonal interactions, lack of fit, and quality of life; subthemes included lack of diversity and racism.

Conclusions

Modifiable/nonmodifiable program factors and geography continue to influence EM‐bound applicants’ residency choices. Underrepresented applicants place a higher value on diversity, community, and patients served. Residency programs should consider modifiable factors and self‐assess for red flags to successfully recruit the next generation of EM physicians.

INTRODUCTION

Medical students applying in emergency medicine (EM) prioritize programs on their rank lists based on limited data and their interactions with residency programs. In 2012, Love et al.¹ provided a baseline description of the factors that applicants consider; however, little attention was paid to underrepresented groups such as female applicants, applicants of underrepresented races and ethnicities, and applicants who identify as gender and sexual minorities. Our study is designed to provide a current update focusing on factors prioritized by underrepresented groups.

EM residency program directors seek to recruit competitive candidates from an ever‐growing applicant pool.², ³, ⁴ EM‐bound medical students consider many factors when evaluating potential training programs, including geography and modifiable/nonmodifiable program factors.¹ Underrepresented applicants strongly consider diversity when choosing a specialty and residency program and are more likely to prioritize programs they perceive as diverse.⁵, ⁶, ⁷ Additionally, the ACGME requires programs to “engage in practices that focus on mission‐driven, ongoing, systematic recruitment and retention of a diverse and inclusive workforce of residents …”⁸ Yet, fewer than half of EM programs have implemented diversification strategies recommended by our governing bodies.⁹ Troublingly, despite the influence of program diversity on applicants and the recommendations of our accrediting bodies, female and underrepresented medical students are less likely to apply in EM,¹⁰ and EM residencies are less representative than almost every other specialty.¹¹ Also, concerningly, EM does not rank among the most inclusive specialties for sexual and gender minorities.⁶

Understanding the factors that influence residency choice among underrepresented applicants, specifically female applicants; those who are members of racial and ethnic groups who are underrepresented in medicine (URiM); and those who identify as lesbian, gay, bisexual transgender, queer/questioning, intersex, asexual, and other gender/sexual minorities (LGBTQIA+) is crucial for inclusive recruitment. We also sought to understand the factors influencing residency choice among nontraditional applicants or those applicants who came to medicine as a second career, later in life, after seeking other degrees, etc. Along with identifying factors that positively contribute to applicant decision making, it is necessary to determine factors that contribute negatively, especially “red flags” or those with a profoundly negative impact. There is little published literature regarding red flags in residency recruitment, particularly in EM.¹²

Our primary objective was to determine the factors influencing the EM applicant rank list creation during the 2019 to 2020 application season. Our secondary objective was to determine if those factors are different for underrepresented applicants. Our final objective was to quantify the prevalence of and characterize red flags encountered during the recruitment process.

METHODS

Study setting and population

We conducted a cross‐sectional mixed methods survey study of medical students applying to EM programs in the United States who applied through the Electronic Residency Application Service (ERAS).¹³ This study was acknowledged as exempt by the primary author's institutional review board. We identified potential participants from the cohort of medical students who applied to one or more of four EM residency programs (Johns Hopkins University School of Medicine, Wayne State University School of Medicine, Baylor College of Medicine, and Maimonides Medical Center). Each program provided a list of email addresses for their applicants and duplicates were removed.

Survey content and administration

We reviewed the literature and began survey development based upon the original items published previously by Love et al. to maximize content validity.¹, ¹⁴ We added nonoverlapping program and location factors based on other studies in the literature.¹⁵, ¹⁶, ¹⁷, ¹⁸ We added one question regarding the distinguishing factor between an applicant's first‐ and second‐choice programs¹⁶ and one question addressing recruitment red flags.¹² Red flags are specifically defined as, “factors that caused applicants to rank a program lower than other programs or not at all.”¹² We piloted our survey with current EM postgraduate year 1 (PGY‐1) residents at four residency programs. We revised the survey based on their feedback and eliminated low‐yield/infrequent factors. Our final survey instrument consisted of 21 multiple‐choice, completion, and free‐text items (Data Supplement S1, Appendix S1, available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1002/aet2.10638/full). Rank list certification occurred on February 26, 2020, and our survey was administered via a Qualtrics email distribution on February 27, 2020.¹⁹ Three reminder emails were sent to nonresponders on an approximately weekly basis and the survey closed on March 19, 2020. Matchday was March 20, 2020. Respondents were not compensated for their time but were entered into a random drawing for one of four gift cards of nominal value for their participation.

All completed survey responses from participants enrolled in U.S. medical schools who ranked at least one EM program were included. Surveys that were incomplete, were completed by international medical graduates, or were completed by applicants who indicated that they did not rank any EM programs were excluded.

Data analysis

We calculated and reported simple descriptive statistics, measures of central tendency, and 95% confidence intervals (CIs) for items with discrete answer choices. Means were compared using t‐tests and proportions were compared using chi‐square statistics. Ordinal responses were compared using the Kruskal‐Wallis test by rank. We employed the Bonferroni correction to address multiple testing.²⁰, ²¹ To assess the relationship between identification of red flags and applicant's characteristics, we employed adjusted logistic regression. Stata 13 was used for data analysis.²²

We performed qualitative analysis on free‐text data using a thematic approach with a constructivist/interpretivist paradigm.²³, ²⁴ Data were independently reviewed by two analysts (JJ and LS) experienced in qualitative methods. They examined data line by line to identify recurring concepts and assign codes, which were further refined into themes using the constant comparative method.²⁵ The two analysts then met to establish a final coding scheme. Then, one of the initial analysts (LS) and a third analyst (EO) independently applied the coding scheme to all data. Discrepancies were resolved by in‐depth discussion and negotiated consensus.

RESULTS

Our overall survey response rate was 49% (1,378/2,810). After 164 international medical graduates and 31 applicants who did not apply to at least one EM residency program were excluded, our final sample included 1,183 completed surveys comprising 44% of all U.S. applicants (1,183/2,661).²⁶, ²⁷

Participant characteristics are displayed in Table 1. These applicants came from medical schools across the United States with 27% from the Midwest, 25.6% from the Northeast, 30.6% from the South, and 14.7% from the West, and 2.1% indicated that region was not applicable. The vast majority (94%) of EM applicants considered a combination of geography and program characteristics in EM applicant match decision making (Figure 1) and the relative importance of these factors did not differ between our sample and the sample described by Love et al. (Supplement S1 and Table S1).¹

TABLE 1.

Demographics and characteristics of U.S. applicants to EM residency 2019–2020 completing rank list survey

Age (years), mean (±SD), range		28.0 (±3.0), 18–45
Gender
	Male	655 (55.4)
	Female	523 (44.2)
	Other or nonbinary	4 (0.3)
	Prefer not to answer	1 (0.1)
LGBTQIA+
	Yes	128 (10.8)
	No	1,039 (87.8)
	Prefer not to answer	16 (1.4)
URiM
	Yes	274 (23.2)
	No	882 (74.6)
	Prefer not to answer	27 (2.3)
Race
	White/Caucasian	771 (65.6)
	Black or African American	92 (7.8)
	American Indian or Alaskan Native	2 (0.2)
	Asian	155 (13.2)
	Native Hawaiian or other Pacific Island	2 (0.2)
	Multiracial (please specify)	72 (6.1)
	Other (please specify)	50 (4.3)
	Prefer not to answer	31 (2.6)
Ethnicity (Hispanic/Latino)
	Yes	122 (10.3)
	No	1,046 (88.6)
	Prefer not to answer	13 (1.1)
Nontraditional
	Yes	482 (40.7)
	Prefer not to answer	12 (1.0)
	No	689 (58.2)
Marital status
	Committed partner/married	408 (34.5)
	Divorced	12 (1.0)
	Widowed	1 (0.1)
	Separated	1 (0.1)
	Single (never married)	753 (63.7)
	Prefer not to answer	8 (0.7)
Region
	Midwest	316 (27.0)
	Northeast	299 (25.6)
	South	358 (30.6)
	West	172 (14.7)
	Not applicable	25 (2.1)
Fellowship
	Yes	345 (29.2)
	No	166 (14.0)
	Unsure	672 (56.8)
Academic career
	Yes	378 (32.0)
	No	206 (17.4)
	Unsure	599 (50.6)

Data are reported as n (%) unless otherwise specified.

Abbreviations: LGBTQIA+, lesbian, gay, bisexual, transgender, queer/questioning, intersex, asexual, and others; URiM, underrepresented in medicine.

FIGURE 1.

Applicant responses to the prompt, “Which choice describes the relative importance of location versus program characteristics in your decision of where to rank programs?”

In our cohort, we found that applicants attended a mean (±SD) of 12.6 (±4.6) interviews and residents ranked a mean (±SD) of 12.5 (±4.4) programs. Female applicants interviewed at and ranked more programs than men (interviewed female 13.3 [±4.2], interviewed male 12.0 [±4.7], ranked female 13.1 [±3.9], ranked male 12.0 [±4.2]; p < 0.0001 for both comparisons), while nontraditional applicants interviewed at and ranked fewer programs than traditional applicants (interviewed nontraditional 12.0 [±4.6], interviewed traditional 13.0 [±4.5], ranked nontraditional 11.9 [±4.4], ranked traditional 12.9 [±4.3]; p < 0.001 for both comparisons).

First‐ and second‐choice programs

Program location was the most common factor applicants used to prioritize their first and second choice in residency programs (Figure 2). Given that 13% (157/1,175) of respondents provided free‐text responses, we opted to perform a qualitative thematic analysis of the free‐text responses revealing themes that ranged from the clinical environment and practical program characteristics—including program length—to the sense of personal connection (Table 2).

FIGURE 2.

Applicant responses to the prompt, “Why was your 2nd ranked program not your 1st? (pick the most important reason)”

TABLE 2.

Qualitative analysis of free‐text responses to the following question: “Why was your 2nd ranked program not your 1st? (pick the most important reason)”

Theme	Subtheme	Example
Program characteristics	Program length	“I love my 2nd choice, it's 4 year academic and I decided to place 1st a similar program that is 3‐year”
	Program location	“Location + weather conditions (Chicago is too cold!!)”
	Program stability	“A lot of new (good) changes but not sure if they have found homeostasis yet”
	Program reputation	“Reputation, brand name”
	Program logistics	“Commute time between sites”
	Institution type	“Primarily academic vs primarily community”
	Alignment with professional goals and aspirations	“Absolutely loved my 2nd choice. However, it just didn't match my professional and personal goals as much as my top ranked program did.”
Clinical environment	Breadth and depth of clinical experiences	“Patient volume, acuity, and variety”
	Patient population	“Patient population‐ wanted more Spanish‐speaking patients”
Personal connection	Perception of “fit”	“It felt right.”
	Familiarity and prior experience with program	“I had a great audition at my first choice”
	Interpersonal connection with program leadership	“… faculty were just slightly less charming than the number one”

Location factors

In our subgroup analyses we found differences in the relative importance of location factors between genders and between those who considered themselves URiM, LGBTQIA+, and nontraditional applicants and those who did not (Tables 3A and 3B). Specifically, females ranked the average importance of neighborhood/community and patient population higher than males. URiM applicants ranked neighborhood/community and patient population as well as ability to live in a particular setting (urban, suburban, and rural) and cost of living as more important when compared with non‐URiM applicants. Members of the LGBTQIA+ community ranked ability to live in a particular setting, neighborhood and community, and patient population as having greater average importance when compared with non‐LGBTQIA+ applicants. Nontraditional applicants ranked program location as less important when compared with those who consider themselves traditional applicants.

TABLE 3A.

Relative importance of location factors stratified by gender and URiM

Factors	Female	Male	p‐value	URiM	Not URiM	p‐value
Geographic location	523, 4.3 (±0.9)	654, 4.3 (±0.9)	0.95	274, 4.4 (±0.9)	881, 4.3 (±0.9)	0.60
Opportunities for partner	514, 2.8 (±1.7)	647, 3.1 (±1.6)	0.02	268, 2.8 (±1.7)	871, 3.0 (±1.7)	0.16
Proximity of partner or family	521, 3.6 (±1.4)	653, 3.6 (±1.4)	0.97	274, 3.5 (±1.4)	878, 3.6 (±1.4)	0.89
Cost of living	522, 2.9 (±1)	652, 2.9 (±1.1)	0.94	273, 3.0 (±1.1)	879, 2.8 (±1.1)	0.00
Ability to live in a particular setting (urban, suburban, rural)	521, 3.5 (±1.1)	652, 3.3 (±1.1)	0.01	274, 3.6 (±1.1)	877, 3.3 (±1.1)	0.00
Neighborhood/community	523, 3.4 (±1.1)	653, 3.1 (±1.2)	0.00	274, 3.6 (±1.2)	880, 3.1 (±1.1)	0.00
Extracurricular and/or recreational activities	523, 3.2 (±1)	651, 3.3 (±1.1)	0.02	274, 3.3 (±1.1)	878, 3.3 (±1.1)	0.79
Patient population	522, 3.9 (±1)	652, 3.5 (±1.1)	0.00	273, 4.1 (±1.0)	879, 3.6 (±1.1)	0.00

Data are reported as number of respondents, sample mean (±sample SD). Factors are ranked from 1 to 5 where 1 = not at all important, 2 = slightly important, 3 = moderately important, 4 = very important, and 5 = extremely important. Neighborhood/community includes (culture, diversity, political climate, school systems, etc.). Based on the Bonferroni correction, p‐values less than 0.006 are considered statistically significant.

Abbreviation: URiM = applicants who identify as underrepresented in medicine.

TABLE 3B.

Relative importance of location factors stratified by LGBGQIA+ and nontraditional applicants

Factors	LGBTQIA+	Not LBGTQIA+	p‐value	Nontraditional	Not nontraditional	p‐value
Geographic location	128, 4.4 (±0.8)	1038, 4.3 (±0.9)	0.99	482, 4.2 (±1.0)	688, 4.4 (±0.8)	0.00
Opportunities for partner	127, 2.9 (±1.7)	1023, 3.0 (±1.7)	0.43	476, 3.1 (±1.6)	678, 2.9 (±1.7)	0.17
Proximity of partner or family	128, 3.2 (±1.5)	1035, 3.6 (±1.4)	0.01	481, 3.5 (±1.4)	686, 3.6 (±1.4)	0.12
Cost of living	128, 2.7 (±1.1)	1035, 2.9 (±1.1)	0.04	480, 2.9 (±1.1)	687, 2.8 (±1.1)	0.07
Ability to live in a particular setting (urban, suburban, rural)	128, 3.8 (±1.1)	1034, 3.4 (±1.1)	0.00	481, 3.4 (±1.1)	685, 3.4 (±1.1)	0.35
Neighborhood/community	128, 3.8 (±1.0)	1037, 3.2 (±1.2)	0.00	482, 3.4 (±1.2)	687, 3.2 (±1.1)	0.01
Extracurricular and/orrecreational activities	127, 3.4 (±1.0)	1036, 3.2 (±1.1)	0.17	481, 3.2 (±1.1)	686, 3.3 (±1.0)	0.08
Patient population	126, 4.0 (±1.1)	1037, 3.7 (±1.1)	0.00	480, 3.7 (±1.1)	687, 3.7 (±1.1)	0.24

Data are reported as number of respondents, sample mean (±sample SD). Factors are ranked from 1 to 5 where 1 = not at all important, 2 = slightly important, 3 = moderately important, 4 = very important, and 5 = extremely important. Neighborhood/community includes (culture, diversity, political climate, school systems, etc.). Based on the Bonferroni correction, p‐values less than 0.006 are considered statistically significant.

Abbreviation: LGBTQIA+ = applicants who identify as lesbian, gay, bisexual, transgender, queer/questioning, intersex, asexual, and others.

Program factors

We also found differences in the relative importance of program factors between genders and applicants who consider themselves URiM or LGBTQIA+ compared to those who do not (Tables 4A, 4BA and 4B). However, we did not find any statistically significant differences between nontraditional applicants from those who do not self‐identify in this category. Males ranked program length of higher average importance while females ranked the average importance of diversity within the program and commitment to the underserved community of higher average importance. Applicants who identified as URiM ranked diversity within the program, program commitment to the underserved, and second look/shadowing experiences of higher average importance than did their non‐URiM counterparts. Applicants who did not identify as LGBTQIA+ ranked program length of greater importance, while members of the LGBTQIA+ community ranked diversity within the program and program commitment to the underserved as having greater importance.

TABLE 4A.

Relative importance of program factors stratified by gender and URiM

Factors	Female	Male	p‐value	URiM	Not URiM	p‐value
Program length(3 years vs. 4 years)	523, 3.1 (±1.5)	654, 3.6 (±1.4)	0.00	274, 3.2 (±1.5)	881, 3.5 (±1.5)	0.01
Program type (academicvs. community)	523, 3.4 (±1.1)	654, 3.4 (±1.1)	0.57	274, 3.3 (±1.1)	881, 3.5 (±1.1)	0.02
Variety of training sites	522, 3.2 (±1.1)	653, 3.2 (±1.1)	0.65	273, 3.2 (±1.1)	880, 3.2 (±1.1)	0.73
Compensation/benefits/vacation time/family leave	522, 2.7 (±1.1)	654, 2.9 (±1.1)	0.01	273, 3.0 (±1.2)	881, 2.8 (±1.1)	0.01
Program reputation	523, 3.4 (±1.0)	653, 3.5 (±1.1)	0.89	274, 3.5 (±1.1)	880, 3.4 (±1.0)	0.36
Diversity within the program(residents and faculty)	522, 3.6 (±1.1)	654, 3.0 (±1.3)	0.00	273, 4.0 (±1.1)	881, 3.0 (±1.2)	0.00
Program commitment to the underserved community	522, 3.9 (±1.1)	653, 3.2 (±1.2)	0.00	273, 4.2 (±1.0)	880, 3.3 (±1.2)	0.00
Interview day experience	654, 3.9 (±0.9)	523, 4.1 (±0.8)	0.01	274, 4.1 (±1.0)	881, 4.0 (±0.9)	0.02
Experience with residents	654, 4.2 (±0.8)	523, 4.3 (±0.8)	0.12	274, 4.3 (±0.9)	881, 4.3 (±0.8)	0.84
Experience with faculty	654, 4.1 (±0.8)	523, 4.2 (±0.8)	0.09	274, 4.2 (±0.8)	881, 4.2 (±0.8)	0.93
Rotation in that ED	653, 3.5 (±1.4)	522, 3.5 (±1.4)	0.90	274, 3.3 (±1.4)	879, 3.5 (±1.4)	0.04
Second look/shadowing	649, 1.7 (±1.1)	521, 1.7 (±1.2)	0.78	272, 1.9 (±1.3)	876, 1.7 (±1.1)	0.00
Didactic program/conference	652, 2.6 (±1.1)	522, 2.6 (±1.1)	0.91	274, 2.7 (±1.2)	878, 2.5 (±1.1)	0.09
Core rotations (ICU, peds, elective, etc.)	652, 2.9 (±1.1)	523, 2.9 (±1.1)	0.72	274, 3.0 (±1.1)	880, 2.9 (±1.1)	0.42
Other educational opportunities	654, 3.2 (±1.1)	522, 3.2 (±1.0)	0.32	274, 3.3 (±1.1)	880, 3.2 (±1.1)	0.14

Data are reported as number of respondents, sample mean (±sample SD). Factors are ranked from 1 to 5 where 1 = not at all important, 2 = slightly important, 3 = moderately important, 4 = very important, and 5 = extremely important. Other educational opportunities include austere medicine, research, EMS, hyperbarics, ultrasound, etc. Based on the Bonferroni correction, p‐values less than 0.003 are considered statistically significant.

Abbreviation: URiM = applicants who identify as underrepresented in medicine.

TABLE 4B.

Relative importance of program factors stratified by LGBTQIA+ and nontraditional applicants

Factors	LGBTQIA+	Not LGBTQIA+	p‐value	Nontraditional	Not nontraditional	p‐value
Program length (3 years vs. 4 years)	128, 2.9 (±1.5)	1038, 3.5 (±1.5)	0.00	482, 3.3 (±1.5)	688, 3.5 (±1.4)	0.12
Program type (academicvs. community)	128, 3.6 (±1.0)	1038, 3.4 (±1.1)	0.20	482, 3.4 (±1.1)	688, 3.5 (±1.1)	0.08
Variety of training sites	127, 3.1 (±1.1)	1037, 3.2 (±1.1)	0.08	482, 3.2 (±1.1)	686, 3.2 (±1.1)	0.50
Compensation/benefits/vacation time/family leave	128, 2.8 (±1.1)	1037, 2.8 (±1.1)	0.55	481, 2.9 (±1.1)	688, 2.8 (±1.1)	0.02
Program reputation	128, 3.5 (±1.1)	1037, 3.4 (±1.0)	0.23	481, 3.5 (±1.1)	688, 3.4 (±1.0)	0.72
Diversity within the program (residents and faculty)	128, 3.8 (±1.1)	1037, 3.2 (±1.2)	0.00	481, 3.4 (±1.2)	688, 3.2 (±1.2)	0.02
Program commitment to theunderserved community	128, 4.0 (±1.2)	1036, 3.5 (±1.2)	0.00	481, 3.6 (±1.2)	687, 3.5 (±1.2)	0.09
Interview day experience	128, 4.1 (±0.8)	1038, 4.0 (±0.9)	0.21	482, 4.1 (±0.9)	688, 3.9 (±0.9)	0.01
Experience with residents	128, 4.2 (±0.8)	1038, 4.3 (±0.8)	0.06	482, 4.3 (±0.8)	688, 4.3 (±0.8)	0.94
Experience with faculty	128, 4.0 (±0.8)	1038, 4.2 (±0.8)	0.03	482, 4.2 (±0.8)	688, 4.1 (±0.8)	0.07
Rotation in that ED	127, 3.4 (±1.4)	1037, 3.5 (±1.4)	0.37	481, 3.4 (±1.4)	687, 3.5 (±1.4)	0.16
Second look/shadowing	128, 1.7 (±1.1)	1031, 1.8 (±1.2)	0.64	479, 1.8 (±1.2)	684, 1.7 (±1.1)	0.43
Didactic program/conference	127, 2.5 (±1.0)	1036, 2.6 (±1.1)	0.45	482, 2.7 (±1.2)	685, 2.5 (±1.1)	0.05
Core rotations (ICU, peds, elective, etc.)	127, 2.8 (±1.1)	1038, 2.9 (±1.1)	0.46	482, 3.0 (±1.1)	687, 2.9 (±1.1)	0.12
Other educational opportunities	128, 3.3 (±1.1)	1037, 3.2 (±1.1)	0.36	481, 3.3 (±1.1)	688, 3.1 (±1.1)	0.02

Data are reported as number of respondents, sample mean (±sample SD). Factors are ranked from 1 to 5 where 1 = not at all important, 2 = slightly important, 3 = moderately important, 4 = very important, and 5 = extremely important. Other educational opportunities include austere medicine, research, EMS, hyperbarics, ultrasound, etc. Based on the Bonferroni correction, p‐values less than 0.003 are considered statistically significant.

Abbreviation: LGBTQIA+ = applicants who identify as lesbian, gay, bisexual, transgender, queer/questioning, intersex, asexual, and others.

Red flags

Overall, 71% (842/1,183) of applicants reported a red flag experience. In our unadjusted analysis we found that odds of identifying a red flag were higher for female applicants (odds ratio [OR] = 1.62, 95% CI = 1.26 to 2.11), and in our adjusted analysis, we found that female applicants had 64% higher odds of identifying red flags than did men after controlling for age, URiM and nontraditional student status, and sexual orientation (OR = 1.64, 95% CI = 1.25 to 2.18). We found no significant differences in odds of identifying red flags among applicants who identify as URiM, nontraditional, or LGBTQIA+. Major themes that emerged from qualitative analysis regarding red flags include violations of regulatory standards, program characteristics, the interview day experience, program culture, specific interpersonal interactions, lack of fit, and quality of life (Table 5).

TABLE 5.

Qualitative analysis of responses to the question: “Please describe any “red flags” that caused you to significantly lower the position of a program on your rank list or not rank it at all”

Theme	Subtheme	Example
Violations of regulatory standards		“Match violations during interview (asked if I was married, where I applied, where I interviewed, plan for kids, and offered to get in contact with other PD’s for me)”
Program characteristics	Program location	“A city with little opportunity for me outside of work”
	Program reputation	“Reputation for being malignant”
	Program stability	“Having recently lost ultrasound director and struggling to replace them”
	Lack of high‐quality educational opportunities	“Not enough hands‐on opportunities, lack of ICU exposure.”
	Lack of procedural experience	“Senior residents taking procedures from younger residents”
	Institutional resources	“Did not have a cath lab at the hospital”
Interview day experience		“Obvious lack of respect for interviewees time and effort to be there, telling us that we would need to do a "second look" in order to be ranked, not having important logistic items such as shift lengths determined at time of interview”
Program culture	Lack of diversity	“One program had almost no female faculty or faculty of color”
	Lack of transparency	“Asking a question on interview day and not getting a straight answer from multiple people.”
	Social culture	“Residents who only had fun together when drinking”
	Lack of resident engagement	“No residents showing up to eat lunch with us, incredibly unenthusiastic resident giving tour”
	Poor intraresident rapport	“Behavior of some residents. Mostly when residents or faculty would openly bad mouth one another in front of me.”
Interpersonal interactions	Discrimination	“Micro aggressions of faculty toward me, likely being an African American.”
	Perceived lack of interest in candidate	“Asking ‘what questions do you have for me’ early in the interview. Making it obvious my application was not read.”
	Negative interactions with program leadership/personnel	“Speaking poorly of other programs, arrogant faculty members.”
	Negative interactions with residents	“Chief resident dropping multiple F bombs during interview”
Lack of fit		“Residents who lack similar interests to mine”
Quality of life	Resident burnout	“Signs of burnout, lack of enthusiasm for caring for patients/practicing.”
	Unmanageable workload	“Residents being tired, overworked, and subtly hinting at wishing it was another way.”

DISCUSSION

Our survey of a national pool of EM‐bound medical students found differences in the importance of specific program and location factors among underrepresented groups applying to EM. These findings can be used to guide strategies that may recruit more diverse applicants to EM residencies. We also describe a high incidence of red flags across candidates, which warrant further attention.

This study builds on prior literature to describe influential factors in the residency recruitment and application process. Our findings are aligned with prior work by Love et al.¹ When compared with that study, our respondents came from similar regional locations across the United States; however, a slightly higher proportion of female applicants responded to our survey. The number of applicants who considered a combination of both program factors and location in both studies was approximately 95% and were similarly distributed. We similarly found that both modifiable and nonmodifiable factors influence rank decisions; however, our study provides additional details regarding underrepresented groups. While some of the factors that emerged are nonmodifiable such as program length and location, there may be opportunity for programs to intervene to recruit their top applicants. An important theme that may be modifiable is the relationship the candidate felt with the program.

A key finding of our study was the relative importance of program diversity, neighborhood, community, and patient population for applicants identifying as female, URiM, and LGBTQIA+. This may reflect a greater importance placed on inclusivity and equity for these applicant groups who may have been directly or indirectly impacted by explicit and implicit bias during their undergraduate medical school experience. Program leaders may consider identifying and sharing qualities such as program diversity, inclusivity of communities and neighborhoods, and program commitment to the underserved to highlight factors that are of significant importance for underrepresented applicants.⁵, ⁹, ²⁸, ²⁹ Understanding these factors that influence underrepresented applicants rank decisions and employing strategies that have been shown to increase diversity in other programs may help program directors to increase diversity and inclusion within their residencies.³⁰

Similar to reports in other specialties, we found that an alarmingly high proportion (~75%) of our cohort encountered red flags as part of the recruitment process.¹² Among our cohort of EM applicants we identified critical themes and subthemes of match violations, lack of diversity, and overt racism. We also found that female applicants experienced red flags at higher rates, which is not surprising given prior literature suggesting female applicants are asked “illegal” questions more often than their male counterparts.³¹

Program leadership should be reflective about areas for real or potential red flags in their recruitment process and take steps to mitigate them. They may consider going further by surveying applicants to determine if any red flags were encountered during the residency recruitment and selection process. By identifying these potential red flags, especially those that may further marginalize already underrepresented groups, leadership may be able to increase recruitment of diverse applicants. Additionally, the community of leaders in EM education should not tolerate violations of regulatory standards to ensure that all applicants are able to experience safe and fair recruitment.

We found that EM applicants continue to encounter match violations, lack of diversity, and racism despite appeals from our governing bodies.⁹ The high prevalence of red flags that we encountered in this study should be a resounding call that we must take measure of the processes by which we select and recruit the next generation of emergency physicians to ensure that they are safe and welcoming to all.

LIMITATIONS

Our study has several limitations. First, our study was survey‐based and is subject to sampling bias. We believe this is mitigated somewhat by the fact that our survey represents 44% (1,183/2,661) of the total population of U.S. EM‐bound applicants.²⁶, ²⁷ Additionally, we found that the geographic distribution, gender representation, and relative preference for program factors versus location are similar between our study and prior literature.¹ While we attempted to gather information about many of the factors that influence EM‐bound medical student residency choice, including open‐ended responses, some important factors are likely to have been missed. Finally, while we based our measurements of diversity on traditionally used demographic characteristics that are routinely reported, these factors do not encapsulate all aspects of diversity, such as socioeconomic factors.

CONCLUSIONS

Both programmatic and location‐related factors continue to influence emergency medicine–bound applicants’ choices when constructing their final rank lists. We found that program diversity, neighborhood, community, and patient population were greater priorities for applicants who identify as female, URiM, and LGBTQIA+. We also found that most applicants experience red flags as part of the recruitment process. Program directors and recruitment committees should self‐assess for red flags and highlight program diversity, community, and commitment to the underserved to create a welcoming environment for diverse applicants and build a workforce that reflects the patients it serves.

CONFLICT OF INTEREST

The authors have no potential conflicts to disclose.

AUTHOR CONTRIBUTIONS

Paul L. Weygandt—study concept and design, acquisition of the data, analysis and interpretation of the data, drafting of manuscript, critical revision of the manuscript for important intellectual content, and statistical expertise. Laura Smylie—study concept and design, analysis and interpretation of the data, drafting of manuscript, and critical revision of the manuscript for important intellectual content. Edgardo Ordonez study concept and design, analysis and interpretation of the data, drafting of manuscript, and critical revision of the manuscript for important intellectual content. Jaime Jordan—analysis and interpretation of the data, drafting of manuscript, critical revision of the manuscript, and statistical expertise. Arlene S. Chung—study concept and design and critical revision of the manuscript for important intellectual content.

Supporting information

Data Supplement S1. Supplemental material.

Weygandt PL, Smylie L, Ordonez E, Jordan J, Chung AS. Factors influencing emergency medicine residency choice: Diversity, community, and recruitment red flags. AEM Educ Train. 2021;5:e10638. 10.1002/aet2.10638