Applicant and Program Director Perceptions of Second Look Events During the 2023 Radiation Oncology Residency Match

Abstract

Purpose

Virtual radiation oncology (RO) residency interviews may impair applicant and program evaluation. Second look events (SLEs) exist; however, the frequency, nature, and implications are unknown. We surveyed applicants and program directors (PDs) to characterize the 2023 RO Match SLEs and assess perspectives.

Method and Materials

An online, anonymous survey was distributed to 2023 RO Match applicants and American College of Graduate Medical Education-accredited RO PDs post-Match. Number and percentage are reported as response per question. Likert-type scores (1, strongly agree; 5, strongly disagree) are reported as median, IQR.

Results

Responses were received from 51 of 246 applicants (21%) and 52 of 88 PDs (59%). Forty applicants (87%) were offered in-person and virtual SLEs; 20 (51%) and 17 (44%) applicants were invited to 1 to 3 and 4 to 6 events, respectively. Most invited applicants attended none (21, 54%). Applicants reported that all (21, 54%) or some (16, 41%) programs communicated intentions to finalize rank order lists (ROLs) before SLEs. Most applicants (29, 74%) agreed that SLEs were optional without ROL consequences (median, 2, IQR 1-3). Applicants declined in-person SLEs due to city/facility indifference (10, 43%), finances (10, 43%), and logistics (9, 39%). Most (12, 86%) in-person SLE attendees agreed that SLEs influenced their ROL (median, 2, IQR 1-2). Nineteen PDs (40%) reported offering SLEs, with 18 of 19 being in-person. PDs who did not offer SLEs cited ethical concerns (13, 45%) and institutional policies (11, 38%). All PDs reported that SLEs were optional, and 18 of 19 explained that the SLE would be without ROL consequences. SLEs mostly occurred in February before (11, 58%) and after (15, 79%) ROL submission.

Conclusions

In-person SLEs occurred during Match 2023. All PDs considered SLEs optional which was trusted by most applicants. Attendance at in-person SLEs influenced applicants’ ROLs; however, finances and logistics impaired applicant attendance. Further work is needed to appreciate SLE implications and ensure equitable residency recruitment.

Introduction

In response to the COVID-19 pandemic, The Coalition for Physician Accountability's Work Group recommended the conversion of residency interviews from in-person to online or phone formats in May 2020.¹ Traditionally, in-person residency interviews served as opportunities for students to explore residency programs, including the medical facilities and city where an applicant would live during training.² Studies comparing virtual interviews to historical in-person interview offerings suggest advantages to the virtual format: reduced costs, avoidance of scheduling conflicts, and avoidance of educational interruptions.3, 4, 5, 6 These benefits, however, occur at the expense of applicant perception of the programs.⁶

Three application cycles after the start of the pandemic, many programs offer only virtual interviews. Due to the disadvantages of this format, some programs chose to offer second look events (SLEs). These events are opportunities for applicants to attend additional programming to learn further about the residency program and/or institution either virtually or in-person after their initial virtual interview. SLEs in the residency application cycle predate the conversion to exclusively virtual interviews and were previously offered by select specialties.^7,8 The controversy regarding financial burdens and pressures imposed on the applicants to attend SLE invitations remains unresolved.⁹ Nevertheless, as travel restrictions ease and programs acclimate to virtual interviews, SLEs are re-emerging across specialties.^10,11 Currently, there is no guidance to residency programs from governing bodies of graduate medical education such as the National Residency Matching Program (NRMP) or Association of American Medical Colleges (AAMC) regarding SLEs.^12,13 The AAMC, however, does provide general guidance to applicants in “Careers in Medicine”¹⁴ that reinforces the voluntary nature.

The impact of SLEs on the Match process, namely the potential alteration of an applicant's or program's rank order list (ROL) after the initial interview, is unknown within the field of Radiation Oncology (RO). Thus, we conducted a survey of both applicants and program directors (PDs) to better characterize the nature and frequency of SLEs during the 2022 to 2023 RO residency application cycle. We aim to investigate the potential impact of SLEs, including the perceived benefits and possible barriers to universal participation.

Methods and Materials

Study population

An “applicant” in this study was defined as any person who applied to a RO residency program using AAMC Electronic Residency Application Service in the NRMP Main 2022 to 2023 residency match cycle through February 2023.¹⁵ Because SLEs concluded before the NRMP Supplemental Offer and Acceptance Program process, all applicants who applied to RO through only the Supplemental Offer and Acceptance Program were excluded. A “program director” (PD) is defined as a physician who oversees any of the American College of Graduate Medical Education (ACGME) accredited RO residency programs.

Survey design and instrument

A committee of 2 RO applicants from the 2023 Match, one RO resident, 2 RO PDs, and one additional RO faculty developed surveys to be distributed to applicants (Table E1) and PDs (Table E2). The applicant survey asked about applicant perceptions of, barriers to participation in, and any resultant behavioral changes after participation in SLEs. The questions were designed to incorporate both published¹¹ and anecdotal experiences. The PD survey inquired about logistics, financial support, and components of SLEs, and assessed perceptions and barriers to participation. Demographic information for both applicants and programs were captured. Both study surveys used a Likert-type scale (1, Strongly Agree; 2, Agree; 3, Neutral; 4, Disagree; 5, Strongly Disagree) with additional Yes/No and free-response questions when applicable. Both surveys used the Research Electronic Data Capture (REDCap) platform^16,17 for data collection, which was hosted by a single institution. The surveys used branching logic to hide irrelevant questions based on responses to previous questions, thus the number of questions per individual varied. Each survey contained a maximum of 30 questions and took approximately 5 to 10 minutes to complete. The study was exempted by the (SUNY Upstate Medical University) Institutional Review Board.

Survey invitation

Applicants who applied to the (SUNY Upstate Medical University) RO residency program for the 2023 Match were invited to voluntarily participate in the applicant survey in May 2023 (after Match Day 2023). The survey invitation was sent electronically to individual email addresses that were identified using Electronic Residency Application Service. A reminder email was sent after 2 weeks. To augment this interviewed applicant pool and better reflect the diversity of the larger RO applicant population, the survey was also advertised on Twitter.

All PDs from the 2023 ACGME-accredited RO residencies were invited to voluntarily participate in the PD survey in May 2023. The survey invitation was sent electronically to email addresses that were identified and verified using ACGME, American Medical Association Fellowship and Residency Interactive Database website, PubMed, and/or official institutional websites as of May 2023. A reminder email was sent after 3 weeks. The PD survey was also advertised on the electronic monthly newsletter of Association for Directors of Radiation Oncology Programs.

Statistical analysis

All returned applicant and PD surveys were included in the analysis. Descriptive statistics were used to analyze and describe applicant and PD demographics and responses. Responses were reported as number and percentage based on the number of responses for each individual question. The Fisher's exact test was used to compare categorical variables. Ordinal responses were compared between groups with the Kruskal-Wallis test. The Wilcoxon rank-sum test was used to compare 2 groups with respect to median values, and the Kruskal-Wallis test was used to compare medians of more than 2 groups. Statistical analyses were performed using SAS 9.4 software (SAS Institute).

Results

Applicants

Of the 82 applicants invited through email using a single institutional interview list, 18 (22%) successfully completed the survey. Applicants invited through social media provided an additional 33 replies. In total there were 51 respondents (44 complete responses), which comprises 21% of the total applicant pool (n = 246). The demographics of applicant respondents are summarized in Table 1. Survey responses are summarized in Table 2. Of all respondents, 40 (87%) were invited to a SLE of any format. Applicants were invited to the following number of SLEs: 13 (20, 51%), 4 to 6 (17, 44%), and 10 to 12 (1, 3%). Most applicants declined to attend any SLE (21, 54%); with additional cohorts attending 1% to 24% (6, 15%) and 25% to 49% (6, 15%) of invitations. Thirty applicants (67%) were offered a virtual and/or hybrid SLE, and 37 (84%) were offered an in-person SLE. Attendance rates were not significantly different between in-person and virtual events, 14 of 37 (38%) versus 10 of 30 (33%), respectively (P = .8).

Table 1.

Applicant survey respondent demographics and comparison with SLE attendance

	Applicants offered SLEs
Demographic information	Attended No SLE (n = 12)	Attended virtual SLE only (n = 4)	Attended in-person SLE only (n = 8)	Attended virtual and in-person SLE (n = 6)	P value	Total N = 30
NRMP applicant type					.08
US MD senior	8 (67%)	1 (25%)	8 (100%)	5 (83%)		22 (73%)
US MD grad	1 (8%)	2 (50%)	0 (0%)	0 (0%)		3 (10%)
US citizen IMG	2 (17%)	0 (0%)	0 (0%)	1 (17%)		3 (10%)
Non-US citizen IMG	1 (8%)	1 (25%)	0 (0%)	0 (0%)		2 (7%)
Self-identified gender
Female	6 (50%)	1 (25%)	3 (38%)	1 (17%)	.57	11 (37%)
Male	6 (50%)	3 (75%)	5 (63%)	5 (83%)		19 (63%)
Self-identified Ethnicities/race					.08
White/Caucasian	8 (73%)	2 (50%)	1 (13%)	3 (50%)		14 (48%)
Latino/Latina	1 (9%)	1 (25%)	0 (0%)	0 (0%)		2 (7%)
Black or African American	0 (0%)	1 (25%)	1 (13%)	0 (0%)		2 (7%)
Asian	2 (18%)	0 (0%)	4 (50%)	2 (33%)		8 (28%)
Mixed	0 (0%)	0 (0%)	2 (25%)	1 (17%)		3 (10%)
Missing	1	0	0	0		1
Applicant regionality					.74
Northeast	4 (33%)	0 (0%)	2 (25%)	1 (17%)		7 (23%)
Midwest	1 (8%)	0 (0%)	1 (13%)	2 (33%)		4 (13%)
South	6 (50%)	3 (75%)	4 (50%)	2 (33%)		15 (50%)
West	0 (0%)	1 (25%)	1 (13%)	1 (17%)		3 (10%)
Outside of US and territories	1 (8%)	0 (0%)	0 (0%)	0 (0%)		1 (3%)

Abbreviations: IMG = international medical graduate; SLE = second look event.

Table 2.

Applicant survey responses

Survey questions	All applicants (n = 51) N (%)	Median (IQR)*
Invitations
Were you offered second look experiences of any kind (in person, hybrid, virtual)?	N = 46
Yes	40 (87%)
No	4 (9%)
Unsure	1 (2%)
Prefer not to answer	1 (2%)
How many Radiation Oncology residency programs offered you a second look of any kind (in person, hybrid, virtual):	N = 39
1-3	20 (51%)
4-6	17 (44%)
10-12	1 (3%)
Prefer not to answer	1 (3%)
Of the second look events offered, approximately how many did you attend either virtually or in-person?	N = 39
100%	1 (3%)
75%-99%	2 (5%)
50%-74%	3 (8%)
25%-49%	6 (15%)
1%-24%	6 (15%)
0%	21 (54%)
Perceptions
In the invitation for the second look event, I was informed that the program rank list would be finalized before the event:	N = 39
Yes, by all programs	21 (54%)
Yes, by some programs	16 (41%)
No	2 (5%)
I felt the second look offering was truly optional and would not affect my rank to the program:	N = 39	2 (1, 3)
Strongly agree	16 (41%)
Agree	13 (33%)
Neutral	3 (8%)
Disagree	5 (13%)
Strongly disagree	2 (5%)
Virtual/hybrid second looks
Were you offered any virtual and/or hybrid Radiation Oncology second looks?	N = 45
Yes	30 (67%)
No	14 (31%)
Prefer not to answer	1 (2%)
Did you attend any virtual Radiation Oncology second looks?	N = 30
Yes	10 (33%)
No	20 (67%)
Why did you choose to attend the virtual second look?†	N = 10
I still had questions and/or wanted more information.	4 (40%)
I wanted more interaction with the residents and faculty.	6 (60%)
I wanted more interaction with coapplicants.	5 (50%)
I felt obligated to go to remain a competitive applicant.	6 (60%)
Reasons for declining any virtual second looks†:	N = 20
I was not interested in the program/not ranking them highly.	11 (55%)
I was not available for the scheduled session.	4 (20%)
I did not feel that an additional virtual meeting would offer much more than other previous virtual interview activities.	18 (90%)
I had "Zoom" fatigue.	15 (75%)
Not applicable	2 (10%)
Not interested in second looks in general	1 (5%)
Attending the second look event virtually provided me with valuable program information that was not apparent during my virtual interview:	N = 10	3 (2, 3)
Strongly agree	1 (10%)
Agree	2 (20%)
Neutral	5 (50%)
Disagree	2 (20%)
Attending the second look event virtually provided me with a better feel of program culture than the virtual interview activities:	N = 10	2.5 (2, 4)
Strongly agree	1 (10%)
Agree	4 (40%)
Neutral	2 (20%)
Disagree	2 (20%)
Strongly disagree	1 (10%)
Attending the second look event virtually influenced my final program rank list:	N = 10	2.5 (2, 3)
Strongly agree	2 (20%)
Agree	3 (30%)
Neutral	4 (40%)
Disagree	1 (10%)
In-person second looks
Were you offered any in-person second look events?	N = 44
Yes	37 (84%)
No	6 (14%)
Prefer not to answer	1 (2%)
Did you attend any in-person Radiation Oncology second looks?	N = 37
Yes	14 (38%)
No	23 (62%)
Why did you choose not to attend the in-person second look?†	N = 23
Not interested in the program/not intending to rank them highly/at all	5 (22%)
Familiarity with the area	1 (4%)
Familiarity with the department	4 (17%)
Did not feel the need to see the facility/city to make the decision	10 (43%)
Financial considerations/burden	10 (43%)
Logistical considerations	9 (39%)
Other (short on time, not living in United States)	2 (9%)
Why did you choose to attend the in-person second look?†	N = 14
Interest in the program/intending to rank them highly	14 (100%)
Ability to physically see the facility	11 (79%)
Ability to physically experience the city (traffic, restaurants, etc)	10 (71%)
Interest in meeting other applicants	11 (79%)
Feeling of obligation to remain a competitive applicant	3 (21%)
“Meet and greet”/informal in-person time with residents and faculty	7 (50%)
Additional activities (eg, dinner with residents ± faculty, tours, etc)	8 (57%)
Financial incentive (program was offering to offset the cost)	9 (64%)
Desirable location (eg, vacation, combining with other travel plans)	3 (21%)
Other (interest in meeting faculty with specialized expertise)	1 (7%)
Attending the second look event in-person influenced my final program rank list:	N = 14	2 (1, 2)
Strongly agree	6 (43%)
Agree	6 (43%)
Neutral	1 (7%)
Disagree	1 (7%)
Attending the second look event in-person provided me with valuable program information that was not apparent during my virtual interview:	N = 14	2 (1, 2)
Strongly agree	5 (36%)
Agree	9 (64%)
Attending the second look event in-person provided me with a better “feel” of the program culture than the virtual interview activities:	N = 14	2 (1, 2)
Strongly agree	6 (43%)
Agree	5 (36%)
Neutral	2 (14%)
Disagree	1 (7%)
I felt that there were barriers preventing me from attending a second look with an in-person component:	N = 14	2.5 (2, 3)
Strongly agree	2 (14%)
Agree	5 (36%)
Neutral	4 (29%)
Disagree	3 (21%)
Barriers to attending an on-site second look†	N = 7
Your medical school required a rotation with limited absence policy	6 (86%)
Being a resident with limited scheduling flexibility	0 (0%)
Residing outside of the United States and needing a visa	0 (0%)
Financial burden	6 (86%)
Family obligations	1 (14%)

^⁎Median and IQR calculated as follows: 1 = Strongly Agree, 2 = Agree, 3 = Neutral, 4 = Disagree, 5 = Strongly Disagree.

^†Participants were allowed to select all that applied for these questions.

Applicants reported being informed by all programs (21, 54%) or some programs (16, 41%) that the program ROL would be finalized before the event. Most applicants (29, 74%) strongly agreed or agreed that SLEs were truly optional and would not affect their ranking in the program (median, 2, IQR 1-3).

Ten applicants (33%) chose to participate in virtual and/or hybrid SLEs. Applicants chose to attend to gain additional information (4, 40%), resident/faculty interaction (6, 60%), coapplicant interaction (5, 50%), or assurance of remaining a competitive applicant (6, 60%). More applicants agreed that virtual SLE attendance provided them with a better “feel” of program culture (median, 2.5, IQR 2-4) rather than valuable program information (median, 3, IQR 2-3) compared with the virtual interview day. Applicants declined virtual or hybrid SLE offerings (20, 67%) due to insufficient interest in the program (11, 55%), schedule conflict (4, 20%), disbelief in sufficient information gains with another virtual activity (18, 90%), or “Zoom” fatigue (15, 75%).

Fourteen applicants (38%) reported attending an in-person SLE due to interest in the program (14, 100%), facility exploration (11, 79%), city exploration (10, 71%), socialization with coapplicants (11, 79%), remaining a competitive applicant (3, 21%), interaction with residents and faculty (7, 50%), offered activities (eg, dinner with residents, tours; 8, 57%), financial incentives (9, 64%), and/or a desirable location (eg, vacation; 3, 21%). All applicants agreed that in-person SLE attendance provided them with valuable program information (median, 2, IQR 1-2), and most agreed that it provided them a better feel of the program culture (median, 2, IQR 1-2) compared with the virtual interview day. Applicants declined in-person SLEs for the following reasons: lack of program interest (5, 22%), established familiarity with the area (1, 4%), established familiarity with the department (4, 17%), no need to see facility/city to make decisions (10, 43%), finances (10, 43%), logistics (9, 39%), and/or other reasons (2, 9%).

Seven applicants agreed (50%) that barriers prevented in-person attendance (median, 2.5, IQR 2-3) and cited specific impairments including required medical school rotations (6, 86%), financial burden (6, 86%), and family obligations (1, 14%).

More applicants who attended in-person SLEs agreed that the event influenced their ROL compared with those who attended virtual SLEs (median, 2, IQR 1-2 vs median, 2.5, IQR 2-3, P = .11). There were no demographic factors that were significantly associated with attendance of SLEs (Table 1).

Program directors

Of the 89 ACGME accredited RO residency programs, 88 PD emails were able to be identified and received a survey invitation with 48 complete and 4 partial responses (59% total response rate). Demographics of all respondent programs are summarized in Table 3. Of the reported demographics, location of the program was significantly associated with offering SLEs (P = .03). Most programs who offered SLEs were in the Southern (10, 53%) and Western (5, 26%) United States. PD survey responses are summarized in Table 4.

Table 3.

Program director survey respondent demographics

Demographic Information	Program directors (n = 48)	Offered any SLE during 2023 match (n = 19)	Did not offered any SLE during 2023 match (n = 29)	P value
No. of residents in program				.17*
1-4	8 (17%)	2 (11%)	6 (21%)
5-8	19 (40%)	7 (37%)	12 (41%)
9-12	13 (27%)	6 (32%)	7 (24%)
13-16	5 (10%)	4 (21%)	1 (3%)
≥17	3 (6%)	0 (0%)	3 (10%)
Residency candidacy spots for July 2024 (PGY2) start date:				.58*
1	13 (27%)	3 (16%)	10 (34%)
2	19 (40%)	9 (47%)	10 (34%)
3	11 (23%)	5 (26%)	6 (21%)
4	3 (6%)	2 (11%)	1 (3%)
5	1 (2%)	0 (0%)	1 (3%)
6	1 (2%)	0 (0%)	1 (3%)
No. of applicants interviewed for July 2024 (PGY2) start date:				.15*
1-20	5 (10%)	1 (5%)	4 (14%)
21-40	32 (67%)	12 (63%)	20 (69%)
41-60	8 (17%)	3 (16%)	5 (17%)
61-80	3 (6%)	3 (16%)	0 (0%)
Program regionality
Northeast	11 (23%)	1 (5%)	10 (34%)	.03†
Midwest	11 (23%)	3 (16%)	8 (28%)
South	16 (33%)	10 (53%)	6 (21%)
West	10 (21%)	5 (26%)	5 (17%)

^⁎Kruskal-Wallis test.

^†Fisher's exact test.

Abbreviation: PGY2 = postgraduate year 2.

Table 4.

Program director survey responses

	All PD (n = 52) N (%)	Median (IQR)*
Recruitment offerings prior and during COVID-2019 pandemic
Before the COVID-19 pandemic (2019-2020 residency match cycle or earlier), did your program offer optional virtual recruiting opportunities for radiation oncology residency of any kind (eg, virtual meet and greet or program overviews?)	N = 48
Yes	3 (6%)
No	45 (94%)
Before the COVID-19 pandemic (2019-2020 residency match cycle or earlier), did your program offer any optional radiation oncology-specific second looks of any kind?	N = 48
Yes: in-person with formalized schedule of events	1 (2%)
Yes: in-person with informal schedule/standing offer to visit department	8 (17%)
Yes: virtual only	1 (2%)
No	38 (79%)
After the beginning of the COVID-19 pandemic (2020-2021 residency match cycle or later), did your program offer any optional virtual recruiting opportunities for radiation oncology residency of any kind (eg, virtual meet and greet or program overviews?)	N = 48
Yes	41 (85%)
No	7 (15%)
Current and future second look offerings
During the 2022-2023 residency match cycle, did your program offer an optional radiation oncology-specific second look of any kind?	N = 48
Yes	19 (40%)
No	29 (60%)
Why did your program not offer an optional second look?†	N = 29
Not interested/did not see the added value.	6 (21%)
Was unaware this was an option, other programs were offering them.	6 (21%)
Institutional policy (hospital, GME) did not allow.	11 (38%)
Ethical concerns	13 (45%)
Financial limitations	3 (10%)
Concerns about NRMP rules/fines	3 (10%)
Other (have an institutional SLE for underrepresented minority students)	1 (3%)
I feel that not offering a second look was detrimental in our recruitment effort:	N = 29	3 (2, 3)
Strongly agree	3 (10%)
Agree	8 (28%)
Neutral	12 (41%)
Disagree	4 (14%)
Strongly disagree	2 (7%)
I plan on using the new NRMP program ROL voluntary locking option to offer a second look for this upcoming recruitment season:	N = 48
Yes	11 (23%)
No	12 (25%)
Unsure	25 (52%)
Second-look invitations
Who was invited to your second look?	N = 20
All applicants offered an interview, regardless of interview status.	4 (20%)
All applicants offered and completed the interview.	14 (70%)
Select applicant population (underrepresented minorities).	2 (10%)
Did messaging regarding the second look offering discuss it being optional vs mandatory?	N = 19
Yes	19 (100%)
No	0 (0%)
Did messaging regarding the second look offering discuss it not affecting the ROL?	N = 19
Yes	18 (95%)
No	1 (5%)
The second look offering was truly optional and did not affect our program's ROL in any way?	N = 19	1 (1, 1)
Strongly agree	19 (100%)
Characteristics of second look
What format was used for the second look?	N = 19
In-person with formalized schedule of events	9 (47%)
In-person with informal schedule/standing offer to visit the department	8 (42%)
Hybrid with in person and virtual	1 (5%)
Virtual only	1 (5%)
When was/were the second look(s) scheduled?†	N = 19
December	1 (5%)
January	1 (5%)
Early February (1-14): before submitting ROL	6 (32%)
Early February (1-14): after submitting ROL	7 (37%)
Late February (15-28): before submitting ROL	5 (26%)
Late February (15-28): after submitting ROL	8 (42%)
Prefer not to answer	2 (11%)
Which items were gifted to applicants in person or via postage for the second look? Please do not include any gifts related to the interview.†	N = 19
Food: includes meals, snacks, and beverages	8 (42%)
Printed literature on the program/department	6 (32%)
Logoed items: mugs/cups, pens, lanyard, bags	4 (21%)
Other (one night hotel)	1 (5%)
None	10 (53%)
In-person second look offerings
If your program offered a second look with an in-person component, what aspects were financially covered by your institution/department?†	N = 18
Travel (airfare, gas reimbursement, etc)	2 (11%)
Meals	8 (44%)
Accommodation/lodging (hotel, housing with a resident)	4 (22%)
Prefer not to answer‡	9 (50%)
If your program offered an in person second look, what activities were offered?†	N = 18
Pre “second look” dinner with residents ± faculty	4 (22%)
Informative lecture about the department	6 (33%)
Physical tour of the facility	13 (72%)
Physical tour of the city	4 (22%)
“Meet and greet”/informal interaction time with residents	14 (78%)
“Meet and greet”/informal interaction time with program leadership	9 (50%)
“Meet and greet”/informal interaction time with faculty	7 (39%)
Prefer not to answer	1 (6%)

Abbreviations: GME = Graduate Medical Education; N/A = not applicable; NRMP = National Residency Matching Program; ROL = rank order list; SLE = second look events.

^⁎Median and IQR calculated as follows: 1 = Strongly Agree, 2 = Agree, 3 = Neutral, 4 = Disagree, 5 = Strongly Disagree.

^†Participants were allowed to select all that applied for these questions.

^‡Given the absence of a “nothing” option, 5 PDs clarified that no costs were covered.

Before the COVID-19 pandemic most programs did not offer any optional virtual recruiting opportunities (45, 94%) or optional SLEs (38, 79%). For the 2022 to 2023 match cycle, 19 (40%) programs offered a RO-specific SLE. PDs reported reasons that SLEs were not offered including perceived lack of value (6, 21%), unaware of possibility or other programs offerings (6, 21%), against institutional policy (11, 38%), ethical concerns (13, 45%), financial limitations (3, 10%), and/or NRMP rules/fines concerns (3, 10%). Eleven PDs (38%) agreed that not offering an SLE was detrimental to recruitment efforts (median, 3, IQR 2-3). PDs reported differing plans to use the new NRMP Program ROL voluntary locking option that is under consideration (allows for programs to certify their ROL ahead of the traditional deadline, potentially before SLEs) before SLEs this next cycle: yes (11, 23%), no (12, 25%), and unsure (25, 52%).

PDs reported offering SLE invites mainly to applicants who were offered an interview (4, 20%) or completed an interview (14, 70%); 2 programs offered SLEs only to students considered underrepresented in medicine. All PDs reported that SLE invitation messaging to the applicants included statements that the SLEs were optional (19, 100%), and almost all reported that the invitation messaging had no effect on the program's ROL (18, 95%). All PDs strongly agreed with the statement that the SLE was optional and did not affect the ROL in any way (19, 100%).

Programs offered SLEs that were in-person with formalized schedule (9, 47%) or an informal standing offer (8, 42%), virtual (1, 5%), or hybrid (1, 5%). Programs with formalized schedule SLEs offered 1 (5, 26%), 2 (4, 21%), or 3 (3, 16%) independent sessions. The SLEs were offered during the following periods: December through January (2, 10%), early February before (6, 32%) and after (7, 37%) submitting ROL, and late February before (5, 26%) and after submitting ROL (8, 42%).

Programs financially covered the following: travel costs (2, 11%), meals (8, 44%), lodging (4, 22%), prefer not to answer (9, 50%), with 5 added comments clarifying that no assistance was given to general applicants. During in-person SLEs, the following activities were offered: dinner with residents/faculty (4, 22%), lecture about the department (6, 33%), facility tour (13, 72%), city tour (4, 22%), resident interaction (14, 78%), program leadership interaction (9, 50%), and faculty interaction (7, 39%).

Discussion

The COVID-19 pandemic radically transformed the medical residency recruitment process from traditional reliance on in-person residency interviews. To our knowledge, this study was the first to attempt to characterize the SLE offerings related to the RO residency match in the postpandemic era, including the perceptions of both involved applicants and PDs. Our study found that 40% of PDs reported offering SLEs to applicants during the 2022 to 2023 recruitment cycle, most with an in-person component. Applicants reported they derived additional information from these experiences and that SLEs altered their ROL, this alteration was seen more in applicants attending in-person SLEs. PDs unanimously reported that the event offerings were truly optional and did not affect the program ROL in any way.

Prior studies have attempted to understand the implications of additional virtual programming and SLEs in other specialties. Studies on applicant perceptions suggest that resident camaraderie, program culture, and an understanding of the program's city cannot be fully appreciated in the virtual interview format.^6,10,11 One survey of Diagnostic Radiology residency applicants showed that attendance of in-person SLEs in the setting of exclusively virtual interviews imparted a better understanding of the program and city, which in turn influenced the applicant's final ROLs.¹¹

Given the recent re-emergence of these SLEs postpandemic, the value for applicants is unknown. Although most applicants reported being offered a SLE, often receiving up to 6 unique event invitations, SLEs were seemingly poorly attended as most invited applicants attended none. It appears that there is greater interest in in-person SLEs compared with virtual or hybrid events, based on slightly higher attendance rates, even with additional financial and logistical barriers. It is logical that applicants might question the value of an additional virtual experience when much of the anecdotal criticism related to virtual interviews centers on the inability to physically experience the local area or facility where the applicant would commit to training.

Although in-person SLEs certainly mitigate this deficit, they are accompanied by another unique set of challenges. There are serious concerns related to possible inequities in the process, when applicants are required to pay for flights, lodging, and other travel costs to attend these in-person events. This concern was reflected in our study, as many applicants cited finances as a barrier to attending these events. This is noteworthy as the cost of attending in-person SLEs likely varies for each experience and is not well known. Although the introduction of virtual interviews has largely exempted applicants from the financial burdens of in-person interviews,² the financial burden of SLEs may prohibit some applicants from attending. This creates a potential disadvantage for individuals who cannot afford these trips. In the most favorable scenario, those who cannot attend sacrifice potentially valuable information to inform their own ROL. After concluding our analysis, it does not seem that the feared consequence of nonattendance, such as being moved down on a program's ROL, occurs.

In our study, we found that applicants did factor in finances, and individual program contributions to these costs, when choosing to attend SLEs. Programs seemed less restricted by financial considerations as an impediment to offering SLEs, as this barrier was only cited by 10% of PDs. However, PDs differed regarding assignment of responsibility for the costs of in-person SLEs. Many PDs offered financial support to all applicants to attend their in-person-SLE characterized as travel, accommodations, and/or meals. There were a small number of PDs who did not, or could not, offer to cover costs associated with in-person SLE attendance for all-comers. A small cohort of PDs offered cost-mitigation scholarships and measures for applicants from underrepresented and economically disadvantaged backgrounds. It remains unclear if applicants who qualified for financial accommodations were aware of such opportunities or felt empowered to apply. Additional research is needed to understand these disparities and how to best address them to ensure equitable access of in-person SLEs to applicants.

Another striking finding was the applicant perception of the true voluntary nature of SLEs. All program directors reported messaging of the optional nature of the SLE in the invitation, with nearly all reporting specific notifications that attendance would not affect ROLs. Reassuringly, this resulted in most applicants reporting belief that attendance at the SLE would not affect their position on the program's ROL. However, there were applicants who did not believe that these programs were truly optional, therefore creating a feeling of obligation to attend a SLE to remain a competitive applicant at that program. This discordance between PD messaging and applicant belief exposes the concerning possibility of applicant mistrust toward programs. This may stem from the fact that programs can have the ability to change their ROLs after the SLE, as approximately half of all PDs reported offering SLEs before finalizing their program ROLs. This issue may be addressed with the proposed NRMP Program ROL voluntary locking feature,¹⁸ which encourages programs to certify their ROLs by a certain date before offering SLEs. Some programs in our study reported that they would plan to use this feature in the next residency interview cycle; however, most PDs were unsure about this decision. Utilization and advertisement of this locking feature may help to alleviate applicant concerns and allow for transparent decision-making regarding SLE attendance. Depending on rates of program adoption, future studies may better reflect applicant belief in the pure intentions of SLEs, allowing the applicant increased familiarity with program city and facilities.

The critical question of who may most benefit from SLEs remains. Also unanswered is if SLEs are necessary for the Match process in the virtual interview format. Applicants’ ROLs were influenced by SLEs, with the assumption that changes were to the benefit of the programs offering them. In theory, this could be disadvantageous to programs who were unaware that other programs were offering such events during the Match, or those who have institutional barriers. Reported program geographic distribution was similar to a previously published RO residency geographic region profile.¹⁹ Significantly more programs in the South offered SLEs, and additional research is needed to correlate program SLE offers with successful applicant matches. This survey also illuminates several barriers to SLE participation, both from programs and applicants alike. Guidance from governing bodies such as AAMC and NRMP with best practices for SLEs may help to ensure equitable SLE access for both programs and applicants. Moving into the 2023 to 2024 recruitment season, multiple programs have already expressed intentions to return to in-person interviews. The role of SLEs will continue to evolve with these changes.

Our study had several limitations that hinder the applicability of the conclusions. We observed an expected (low) response rate, particularly among applicants, likely multifactorial but certainly related to the limitations of the survey dissemination methods that did not include direct email contact of all applicants. Also, use of social media for survey distribution creates the inability to prevent multiple completions by a single person, particularly by some applicants who may have received invitations via email and social media. Together, these factors limit the generalizability of the results as the population sampled many not be truly representative of larger applicant pool because only applicants who voluntarily use social media and applied to a single institution were contacted. Finally, programs and applicants were surveyed independently, and responses cannot be matched nor compared.

Conclusion

SLEs, notably with in-person components, occurred during the 2022 to 2023 RO residency recruitment cycle. Although many applicants were offered several SLE invitations, most chose not to attend any. All PDs considered SLEs optional without ROL consequences, which was trusted by most applicants. Attendance at both virtual and in-person SLEs was motivated by a multitude of factors. SLEs, especially those offered in-person, did influence an applicants’ ROL. Financial burden was a frequently noted barrier to attending in-person SLEs. Attention to this factor is necessary with future SLE offerings, as this may be a driver of inequitable SLE attendance and, potentially, Match outcomes. Programs cited ethical concerns and institutional policies that impeded their consideration of offering a SLE. Further research is needed to fully appreciate the scope and implications of SLEs to ensure equitable residency recruitment.

Disclosures

Steve E. Braunstein discloses consulting fees from Novocure and Icotec; honoraria from Rad Onc Questions, LCC; Senior editor for IJROBP, outside of submitted work. Jillian R. Gunther discloses grant funding from the National Institutes of Health R25 for Summer Medical Student Research Program and General National Institutes of Health, National Cancer Institute Cancer Center Support Grant; honoraria from Osler Institute and University of Maryland Board Course; Radiation Oncology Education Collaborative Study Group Board Membership, outside of submitted work.