How do emergency medicine applicants evaluate residency programs in the post–COVID‐19 era?

Abstract

Objectives

Pandemic disruptions to interviews and away rotations led applicants to rely on alternative sources of information about residency programs. We sought to compare program characteristics important to emergency medicine (EM)‐bound applicants from before and after the pandemic. We also investigated the sources of information pandemic impacted applicants used during residency recruitment.

Methods

This was a multi‐institutional, cross‐sectional survey of applicants to EM residency programs. We asked applicants about important factors in choosing a program and compared their response to results from 2019 National Residency Match Program. We also asked about alternative information sources used during this time of restricted access to programs of interest.

Results

We surveyed 414 applicants from 40 medical schools and attained a response rate of 38.4%. Compared to 2019 applicants, our respondents identified morale of residents and quality of faculty as important factors in choosing a residency. Our subjects cited websites of the program and hospital affiliate, current residents, faculty/mentor advice, Reddit, and Doximity as sources of program information.

Conclusions

Compared to 2019, our EM‐bound applicants who, because of the pandemic, were unable to visit programs of interest valued resident morale and faculty quality as factors in choosing a residency program. Without in‐person visits, our subjects also had to use both virtual resources (e.g., websites) and traditional sources (e.g., mentor advice) to investigate a program's culture, reputation, and diversity and inclusion. Residency programs should monitor their online presence now that this has become an alternative source of information for applicants during curtailment of in‐person visits.

INTRODUCTION

Traditionally, EM‐bound applicants have cited the same top five factors as most important to ranking residency programs of interest in preparation for the residency match. ¹ , ² , ³ These factors are goodness of fit with the program, the interview day experience, desired geographic location, quality of the residents, and reputation of the program. The COVID‐19 pandemic disrupted residency recruitment and interviewing due to restrictions on away rotations and travel for in‐person interviews. ⁴ , ⁵ , ⁶ Consequently, medical student applicants were unable to use direct personal experiences for investigating residency programs and developing rank lists. ⁷ , ⁸ In this new restrictive environment, would residency applicant's priorities change? And how would applicants determine which program was right for them?

The current pool of applicants is largely composed of individuals from Generation Y, individuals born between 1981 and 1996, and Generation Z, individuals born after 1996. Both generations are reputed to be more technologically savvy and more reliant on internet and virtual technologies than previous generations. ⁹ , ¹⁰ While internet resources have been available for over a decade, the role of these resources in the residency recruitment process remains ambiguous. The literature suggests that even though there may have been some progression toward applicant's use of social media and online resources for investigating residency programs before COVID‐19; applicants in some specialties such as diagnostic radiology and orthopedic surgery residencies were still primarily reliant on program information about residency programs gleaned through in‐person interactions. ¹¹ , ¹² Other studies suggest that Gen Y and Z applicants have been making increased use of internet content to evaluate residency programs for some time. ⁹ , ¹³ Schlitzkus et al., ⁹ for example, profiled Generation Y as active social networkers and sharers of information through weblogs, discussion threads, YouTube, Twitter, or Facebook. ⁹ These authors proposed that to attract and engage Gen Y applicants, residency programs needed to create an image of being technology savvy by generating their own internet content through these same platforms. Rolston et al. ¹³ studied the impact of Doximity, a website that introduced residency program rankings in 2014, and found that 17% of medical students used Doximity information for making their rank lists.

The primary purpose of our project was to determine whether medical student applicants used different criteria for evaluating residency programs of interest as a result of the COVID‐19 pandemic. A secondary objective was to identify the information resources used by medical student applicants to investigate potential residency programs during the pandemic, given strict limitations placed on away rotations and in‐person interviews.

METHODS

Population and setting

Our study was a multi‐institutional, cross‐sectional survey of fourth‐year medical students graduating from allopathic and osteopathic medical schools in the United States in 2021 who entered the National Resident Matching Program (NRMP) in pursuit of an emergency medicine (EM) residency. This cohort of students was the first to navigate the formal recruitment and match process without benefit of elective preview (away) rotations or face‐to‐face personal interviews.

Sampling strategy

We arrived at an estimate of the total population of senior applicants to EM residency programs during the 2020–2021 NRMP cycle (Total N = 2396; MD seniors = 1713, DO seniors = 683) through a step‐by‐step process of estimating medical school class sizes from the percentages of total U.S. allopathic (MD, 1713 of 18,108; 9.5%) and osteopathic (DO, 683 of 5968; 11.4%) senior medical students who successfully matched in EM in 2020. ¹⁴ , ¹⁵ We then applied a formula provided by Dillman et al. ¹⁶ to calculate a sample size that would achieve a 95% confidence interval, with ±3% sampling error. We used a nested, stratified sampling design to sample students by the geographic regions defined by the Association of American Medical Colleges (AAMC; northeast, central, south, and west) and medical school type (allopathic or osteopathic). We then randomly derived our sample from each stratum (region and type of school) to obtain a sample of 417 students from 18 osteopathic and 658 MD students from 50 allopathic medical schools (Appendix S1).

Measurements/instrumentation

Our survey instrument consisted of three parts. The first included items adapted from the NRMP Applicant Survey (used with permission), which asked medical students to identify factors important to them in choosing a residency program. ¹⁷ Instead of the two‐step process used by the NRMP in which students only rate those factors they identify as important, we asked our respondents to rate the importance of all 44 common factors using a scale of 1 (not at all important) to 5 (extremely important). We chose this alternative implementation method to ensure that all respondents provided ratings of all factors cited in past studies as having influenced EM residency choice: the (1) program's culture, (2) program's diversity and inclusion, and (3) program's and faculty's reputation. ¹⁸ We also added or revised factors to make them relevant to EM applicants. We added program duration since EM applicants can choose between 3‐ and 4‐year programs, and we changed the NRMP factor “graduates' passage rates of American Board of Medical Specialties (ABMS)” to “graduates' passage rates of American Board of Emergency Medicine (ABEM).”

The second part of our instrument asked applicants to select the top three sources of information they used to evaluate program characteristics, specifically a residency program's culture, reputation, diversity/inclusion, and faculty reputation. The list of information sources presented was generated by study authors and pilot‐tested with non–EM‐bound medical students from the University of Massachusetts. These students identified additional sources of information to add to our instrument, which included conventional social media applications such as Twitter and Instagram, professional society websites such as that of the Emergency Medicine Resident Association (EMRA), and health care–oriented social media platforms such as Doximity. Finally, we included demographic questions (Appendix S1).

Data collection

Our study relied on cooperation from targeted medical schools for access to our sample of medical students. We used institutional websites, the Council of Residency Directors in Emergency Medicine (CORD‐EM) directories, and the Society for Academic Emergency Medicine (SAEM) Clerkship Directory to obtain contact information for EM clerkship directors, their coordinators and/or advising deans. We sent an email requesting help by either: (1) forwarding a survey link to their EM applicants or (2) providing the names and emails of their EM applicants so we could survey them directly. When a contact person could not be identified for a target school, we replaced them with a similar school from the same region of the United States. We administered the survey between January and March 2021 prior to Match Day.

Data analysis

We performed descriptive statistics to profile respondent demographics and their ratings of factors of importance in choosing a residency program. To investigate potential changes in factors participants found important, we compared our results to those of EM applicants from previous Match cycles published in the NRMP Applicant Survey Reports. ¹ , ² , ³ We also conducted bias analyses using chi‐square tests of association to test our medical school samples for response bias related to our strata: region of the United States and type of medical school, allopathic versus osteopathic. Respondents who selected a racial/ethnicity category of African American, African or Black, American Indian, Alaskan Native, Native Hawaiian, Chicano or Mexican, Chicano‐White, Hispanic‐White, Hispanic, and Latinx were classified as individuals from historically minoritized populations (HMPs).

We calculated weighted rankings by reverse coding participant's ranks of information sources, assigning three points to their first choice, two to their second, and one to their third. We then summed the points across respondents to obtain a weighted rank score for each information source. Finally, we summed the weighted ranks for first, second, and third choices to create a total rank score for each information source. Using this strategy, the information source with the largest total rank score was considered the most important source of information. We then reranked the sources of information by total rank score for respondents to determine their sources of information for a program's culture, diversity–inclusion, program reputation, and faculty reputation. The institutional review board at the University of Massachusetts determined this study to be exempt from review (ID# H00022040).

RESULTS

To reach our sampling goal of 417 DO and 658 MD medical students, we submitted inquiries for support to 68 medical schools. During the data collection process, we were required to adjust our sample based on the interest and cooperation of each sampled medical school in helping with our survey. Ultimately, we were able to sample 414 medical students from 40 medical schools, from whom we received 159 responses (38.4%).

Bias analysis

Our bias analysis of the two strata used to guide our sampling suggested that our sample was representative of all four regions of the United States. We had slightly higher participation of medical schools from the northeastern and southern regions of the United States than the central and western regions, but these differences were not statistically significant. We did, however, experience a bias related to medical school type, with significantly more allopathic medical schools than would have been expected from the number sampled in each category (χ² = 6.6, df = 2, p = 0.013; Table 1). In other words, two‐thirds of the sampled allopathic medical schools participated in the study, while only one‐third of sampled osteopathic medical schools participated.

TABLE 1.

Number of our sampled medical schools who participated and did not participate by region of the United States and medical school type

	Respondents		Nonrespondents		Total
U.S. Region
Central	8	(53.3)	7	(46.7)	15	(22.1)
Northeastern	11	(68.8)	5	(31.2)	16	(23.5)
Southern	16	(64.0)	9	(36.0)	25	(36.8)
Western	5	(41.7)	7	(58.3)	12	(17.6)
Total	40	(58.8)	28	(41.2)	68	(100)
	χ2 = 2.6, df = 3, p = 0.46
Medical school type
Allopathic	34	(68.0)	16	(32.0)	50	(73.5)
Osteopathic	6	(33.3)	12	(66.7)	18	(26.5)
Total	40	(58.8)	28	(41.2)	68	(100)
	χ2 = 6.6, df = 1, p = 0.013 a

Note: Data are reported as n (%). Percentage of responding and nonresponding schools are based on row totals. Percentages in the total column are based on the 68 medical schools that were included in the study. Chi‐square tests of association served as bias tests.

^a Our sample of medical schools was slightly biased toward allopathic medical schools since more of them and fewer osteopathic schools participated than would have been expected from the number of schools in each category sampled.

Participant demographics

Among the 159 medical students who participated in our study more than half were from medical schools in the southern region 59.7% (n = 95), 89% (n = 142) were from allopathic medical schools, 42% (n = 67) identified as female, more than three‐fourths (n = 120) were in the 25‐ to 28‐year age group, and 12.6% (n = 20) identified as ethnicities or races that were from HMPs (Table 2).

TABLE 2.

Number and percentage of medical students who participated in our study by age group, gender identity, and self‐identified race/ethnicity

U.S. region (as defined by the AAMC)
Central	32	(20.1)
Northeastern	26	(16.4)
Southern	95	(59.7)
Western	6	(3.8)
Total	159	(100)
Medical school type
Allopathic	142	(89.3)
Osteopathic	17	(10.7)
Total	159	(100)
Gender
Female	67	(42.1)
Male	89	(56.0)
Nonbinary	1	(0.6)
Missing	2	(1.2)
Total	159	(100)
Age group
<25	3	(1.9)
25–28	120	(75.5)
29–31	21	(13.2)
32–35	12	(7.5)
>36	2	(1.3)
Missing	1	(0.6)
Total	159	(100)
Race/ethnicity
African American	3	(1.9)
African	2	(1.3)
African‐African American	1	(0.6)
African‐American White	1	(0.6)
American Indian	1	(0.6)
Asian Indian	11	(6.9)
Chicano or Mexican	2	(1.3)
Chicano‐White	3	(1.9)
Chinese	4	(2.5)
Chinese‐White	1	(0.6)
Cuban‐Dominican	1	(0.6)
Hispanic‐White	1	(0.6)
Japanese	1	(0.6)
Middle Eastern	4	(2.5)
Native Hawaiian	1	(0.6)
White	107	(67.3)
Other Asian	4	(2.5)
Other Hispanic or Latinx	5	(3.1)
Prefer not to say	6	(3.8)
Total	159	(100)

Abbreviation: AAMC, Association of American Medical Colleges.

Important factors to choosing a program

Program features that our respondents prioritized as important were considerably different than those of the NRMP EM seniors from 2015, 2017, or 2019 (Table 3). ¹ , ² , ³ While both groups considered “perceived goodness of fit,” “quality of residents in the program,” and “desired geographic location” as important, our respondents also identified “morale of current residents,” “quality of faculty,” and “diversity of patient problems” as important factors in their decision making. “Work–life balance” and “reputation of the program” were rated more important by the 2019 NRMP EM seniors than by our sample of EM‐bound seniors.

TABLE 3.

Factors related to selecting a residency program for application from the 2019 NRMP's Applicant Survey Figure EM‐1 (Used with permission; N = 796) and the fourth‐year medical students from the current study (N = 158)

	NRMP EM seniors‐2019			Current study
Factor	% citing as a factor a	Mean	Rank	% citing as a factor b	Mean	Rank
Desired geographic location	89	4.6	1	97	4.35	5 (t)
Perceived goodness of fit	86	4.8	2	98	4.57	1
Reputation of program	78	4.0	3	92	3.66	15
Work/life balance	72	4.4	4	95	4.14	10
Quality of residents in program	70	4.5	5	98	4.46	4
Quality of educational curriculum and training	65	4.5	6	99	4.24	8
Quality of faculty	65	4.4	7	100	4.48	3
Quality of program director	63	4.4	8	97	4.33	7
Cost of living	61	3.6	9	70	3.15	28 (t)
Social and recreational opportunities of the area	60	4.1	10	86	3.69	14
Diversity of patient problems	55	4.3	11	97	4.35	5 (t)
Balance between faculty supervision and resident autonomy	55	4.1	12	96	4.08	11
Morale of current residents	51	4.4	13	98	4.53	2
Academic medical center program	51	3.9	14	82	3.29	25
Program's flexibility to pursue electives and interests	49	4	15	92	3.85	12
Career paths of recent program grads	48	3.9	16	88	3.84	13
Quality of hospital facilities	47	3.8	17	87	3.47	17 (t)
Opportunities to perform specific procedures	45	4	18	74	3.34	21 (t)
Future fellowship training opportunities	43	3.6	19	85	3.6	16
Future job opportunities for myself	42	4.1	20	92	4.23	9
Size of program	42	3.4	21	71	2.94	32
Job opportunities for my spouse/significant other	37	4.4	22	54	2.69	38
Cultural/racial/ethnic/gender diversity of geographic location	37	4.2	23	79	3.47	17 (t)
Quality of ancillary support staff	34	3.8	24	80	3.32	24
Cultural/racial/ethnic/gender diversity of institution	33	4.2	25	72	3.21	26
Preparation for fellowship training	33	3.9	26	79	3.33	23
Support network in the area	32	4	27	70	3.46	19
Size of patient caseload	32	3.9	28	78	3.19	27
Supplemental income (moonlighting) opportunities	32	3.5	29	62	2.83	35
Vacation/parental/sick leave	28	3.7	30	65	2.97	31
Salary	28	3.4	31	52	2.49	42
Community‐based setting	27	3.4	32	65	2.88	33
Availability of EHR	24	3.7	33	60	2.99	30
Opportunities for international experience	24	3.6	34	49	2.6	41
Opportunities to conduct research	19	3.7	35	40	2.35	44
Having friends at the program	12	3.3	36	54	2.66	40
ABMS (ABEM) pass rates	11	4.2	37	68	3.15	28 (t)
Call schedule	11	3.5	38	60	2.84	34
Opportunities for training in systems‐based practice	11	3.5	39	60	2.68	39
Area schools for my children	6	4	40	13	1.43	46
Other benefits	6	3.6	41	60	2.75	36
Quality of ambulatory care facilities	5	3.5	42	60	2.71	37
Presence of a prior Match violation	4	4	43	36	2.17	45
Alternative duty hours	3.4	3.4	44	50	2.38	43
Length (in years) of program				51	3.41	20
Cultural/racial/ethnic/gender diversity of program				51	3.34	21 (t)

Abbreviations: ABEM, American Board of Emergency Medicine; ABMS, American Board of Medical Specialties; EHR, electronic health record; NRMP, National Residency Match Program.

^a NRMP seniors only rated those factors that they identified as important to them, so the first column represents the number who selected to rate that factor.

^b Our survey respondents rated all factors, so the percentage citing as factor represents those who said that the factor was moderately, very, or extremely important. Both studies used instruments that used the following key: (1) = not at all important; (2) = slightly important; (3) = moderately important; (4) = very important; (5) = extremely important. The last two factors were added to the survey for the current study.

Results related to choosing a residency program based on diversity of the institution or geographic location were ranked comparably by both groups. The 2019 NRMP EM seniors' rankings for these factors were 25 and 23, respectively, while our sample of EM bound seniors' rankings were 23 and 17 (tie). The diversity of the program was not asked on the 2019 NRMP Applicant Survey, but our sample's ranking of that item was close to the results of the other two factors (21 [tie]).

Sources of program information

Respondents identified the residency program's website and faculty/mentor advice as common sources of information for evaluating all program characteristics: culture, diversity & inclusion, and both program and faculty reputation (Table 4). The program's website was a primary (ranked top three) source of information for all but the characteristic of program reputation, while faculty/mentor advice was the primary source for all but diversity and inclusion. Sources we might think of as outside of the control of the residency program such as Reddit, EM Residents' Association (EMRA) website and peers from the applicant's medical school were the primary sources of information for investigating both program and faculty reputation.

TABLE 4.

Weighted ranks of sources of information for each information topic: program culture, diversity and inclusion, program reputation, and faculty reputation

	Weight rank
Topic of Information and information source	1	2	3	Total	Global a rank
Program culture
Residents from program of interest	232.5	30	11.5	274	1
Residency program website	49.5	68.67	22	140.17	2
Faculty/mentor advice	30	53.32	19.34	102.66	3
Reddit	21	31	24.83	76.83	4
Residents from my own institution	13.5	26.33	10	49.83	5
Peers from my medical school	13.5	19.67	12.83	46	6
Program diversity and inclusion
Residency program website	237	58	15.5	310.5	1
Residents from program of interest	107.5	82.67	17	207.17	2
Website of hospital or medical center affiliate	12	32	24.33	68.33	3
Reddit	18	16.66	20.84	55.5	4
Faculty/mentor advice	15	16	13	44	5
Instagram	4.5	21	16.33	41.83	6
Program reputation
Faculty/mentor advice	123	71	18.5	212.5	1
Doximity residency navigator	138	50	12	200	2
Reddit	48	40.67	20.5	109.17	3
Peers from my medical school	30	32	14.5	76.5	4
EMRA website	33	20	12	65	5
Residency program website	28.5	14	12.5	55	6
Faculty reputation
Residency program website	133.5	38	15	186.5	1
Faculty/mentor advice	105	65	16	186	2
Residents from program of interest	97.5	52.66	21	171.16	3
Reddit	19	24	16	59	4
EMRA website	15	14	9	38	5
Peers from my medical school	12	16.67	6	34.67	6

Note: The number of points was assigned from reverse coding of medical students top three sources of information (first choice = 3, second choice = 2, and third choice = 1). Total weighted ranks are the sum of the weighted ranks for first, second, and third choices. The final rank was assigned according to the order of the total weighted ranks. Only the first six ranked sources of information are listed for each topic.

Abbreviation: EMRA, Emergency Medicine Resident Association.

^a Global ranks are the average weighted rankings of the sixteen sources of information and an “other” option. Students were permitted to choose their top three sources of information for each topic. Responses were weighted so that the first choice was assigned a “3,” second choice a “2,” and the third choice a “1.” Weighted rankings were summed to create a total score for each resource; then items were reranked based on the total score.

DISCUSSION

The top five factors used by medical students to select residency programs for application were somewhat different between the pre‐COVID NRMP EM Match cycles and our sample of applicants from the post‐COVID, 2021 NRMP EM Match cycle. While both groups ranked “perceived goodness of fit” and “desired geographic location” in their top five factors, the post‐COVID group also prioritized “morale of the current residents,” “quality of faculty,” and “diversity of patient problems” as important factors. We surmise that these findings might be explained by the concern applicants had regarding the impact of the pandemic on residency programs, particularly the impact on residents.

While all applicants considered traditional sources of information, such as discussions with current residents, residency and affiliate hospital websites, faculty/mentor advice, and geographic location, to be of great importance, they also found nontraditional sources useful, including websites like Reddit. This finding may be alarming to some, but if we consider that websites like Reddit and Doximity are private (anonymous) exchanges of information, applicants may be using them the same way consumers use Yelp, with caution and in conjunction with other sources of information.

While many residency programs have invested time and effort into developing an online presence on platforms such as Twitter and Instagram, ¹⁹ these did not appear to be useful resources to applicants. Even widely recognized online resources like EMRA and Academic Life in EM (ALiEM) were not considered important sources by our respondents. On the other hand, Reddit was ranked third as an important resource for evaluating a program's reputation and fourth for evaluating program culture and diversity and inclusion.

LIMITATIONS

Limitations of this study include the fact that we encountered challenges with directly accessing our study population, which subsequently reduced the effectiveness of our carefully crafted sampling strategy. With some schools we had difficulty connecting with representatives who were able to identify and communicate with their EM‐bound medical students. Other schools declined participation on behalf of their students due to privacy concerns. In contrast, one medical school delivered our survey to not only their own fourth‐year students but also their visiting students from other medical schools altering their representation. Although we were inclusive of both allopathic and osteopathic medical students, we received a slightly higher response from allopathic students. While these limitations may have resulted in response biases for which we are unable to account, we think our findings are useful in helping residency programs to understand the impact of their online presence during the recruitment process, particularly during and after of the COVID‐19 pandemic. Future research is needed to determine whether our study of EM‐bound students is generalizable to applicants of programs in other specialties and whether applicants will continue to rely on nontraditional sources of information after the pandemic dissipates.

CONCLUSIONS

Our study suggests that compared to applicants from the match prior to the COVID‐19 pandemic, our subjects prioritized resident morale in creating their rank lists. We also found that emergency medicine–bound applicants who had experienced restricted access to residency programs during recruitment season made increased use of virtual resources (e.g., websites) to prepare for the residency match. We also found that applicants made use of less prominent social media platforms such as Reddit or Doximity but not more prominent platforms such as Twitter.

AUTHOR CONTRIBUTIONS

Cassandra Mackey, Jacob Feldman, Cynthia Peng, David P. Way, and Anne Messman were all involved in study concept and design, acquisition of the data, analysis and interpretation of the data, drafting of the manuscript, and critical revision of the manuscript. There was no funding involved in this study.

CONFLICT OF INTEREST

The authors declare no potential conflict of interest.

Supporting information

Appendix S1 Supporting information

Mackey C, Feldman J, Peng C, Way DP, Messman A. How do emergency medicine applicants evaluate residency programs in the post–COVID‐19 era? AEM Educ Train. 2022;6:e10805. doi: 10.1002/aet2.10805