Understanding the Urology Program Directors Perspective on the Current Resident Selection Process: The Society of Academic Urologists National Survey of Urology Program Directors

Abstract

Objective

The traditional residency selection process was altered dramatically by the SARS CoV-2 (COVID-19) pandemic. For the 2020-2021 application cycle in-person interviews were transitioned to the virtual format. What was thought to be a temporary transition has now become the new standard with continued endorsement from the Association of American Medical Colleges (AAMC) and the Society of Academic Urologists (SAU) for virtual interviews (VI). We sought to assess the perceived efficacy and satisfaction of the VI format from the urology residency program director's (PDs) perspective.

Design

A designated SAU Taskforce on “Optimizing the Applicant Experience in the Virtual Interview Era” developed and refined a survey composed of 69 questions on VI and was distributed to all urology program directors (PD) of member institutions of the SAU. The survey focused on candidate selection, faculty preparation, and interview day logistics. PDs were also asked to reflect on the impact of VI on their match results, recruitment of underrepresented minorities and female gender, and what their preference would be for future applications cycles.

Participants

Urology residency PDs (84.7% response rate) between January 13, 2022 – February 10, 2022 were included in the study.

Results

Most programs interviewed a total of 36 to 50 applicants (80%), with an average of 10 to 20 applicants per interview day. The top 3 ranked criteria for interview selection reported by urology PDs surveyed included letters of recommendation, clerkship grades, and USMLE Step 1 score. The most common areas of formal training for faculty interviewers were diversity, equity and inclusion (55%), implicit bias (66%), and review of the SAU guidelines on illegal questions (83%). Over half (61.4%) of PDs believed that they were able to accurately represent their training program through the virtual platform, while 51% felt that VI did not afford similar assessments of applicant as in-person interviews. Two-thirds of PDs believed the VI platform improve access for all applicants to attend interviews. Focusing on the impact of the VI platform for recruitment of underrepresented minorities (URM) and female gender applicants, 15% and 24% reported improved visibility respectively for their program, and 24% and 11% reported increased ability to interview URM and female gender applicants respectively. Overall, in-person interviews were reported to be preferred by 42%, and 51% of PDs desired VIs to be included in future years.

Conclusions

PDs opinion and role of the VIs into the future is variable. Despite uniform agreement of cost savings and belief that VI platform improves access for all, only half of PDs expressed interest of the VI format being continued in some form. PDs note limitation of VI in the ability to comprehensively assess applicants as well as the in-person format. Many programs have begun to incorporate vital training in the areas of diversity equity and inclusion bias, and illegal questions. There is a role for continued development and research on ways to optimize virtual interviews.

Introduction

The SARS CoV-2 (COVID-19) pandemic has greatly affected medical education and residency recruitment to urology programs across the country. Starting with the 2020-2021 application cycle, the Association of American Medical Colleges (AAMC) and the Society of Academic Urologists (SAU) adopted policies to transition from in-person to virtual interviews (VI). The AAMC and SAU have since continued to endorse the virtual platform as the primary method of interviewing candidates for upcoming application cycles.^{1 , 2}

The shift in the interview process to an entirely virtual format, raises several questions with regard to the suitability and sustainability of VI for both applicants and programs moving forward. The SAU prioritizes continued re-assessment of the VI process and thus established a taskforce working in conjunction with the Board of Directors to explore this process further. Accordingly, a survey was developed and distributed to PDs who took part in the 2020-2021 and 2021-2022 application cycles. The primary objective of the study was to assess the perceived efficacy and satisfaction of the VI platform among urology residency PDs as it pertained to residency candidate assessment and selection. Secondary outcomes included investigation into 1) the logistics surrounding the VI format, and 2) the recruitment of underrepresented minorities (URM) and women.

Materials and Methods

A designated SAU Taskforce on “Optimizing the Applicant Experience in the Virtual Interview Era” developed and refined a survey composed of 69 questions including multiple choice, Likert scale, rank order and free text responses (see survey in Appendix 1). The anonymous survey was distributed by the SAU to urology PDs at member institutions of the SAU between January 13, 2022 and February 10, 2022.

Survey questions pertained to respondent demographics, candidate interview selection methods, virtual interview logistics, and faculty preparation for interviews, including formal bias training. The survey also included questions regarding perceived implications of the virtual interview platform on recruitment of underrepresented minorities and applicants of female gender. Last, program directors were queried about the impact of virtual interviews on the Urology Residency Match and prompted to share their interview preferences (i.e., virtual, hybrid, in-person) for future application cycles.

Descriptive variables were presented as percentiles relative to the total number of responses for specific questions. Chi-squared tests were used to compare categorical variables including applicants interviewed in total and per day, interview logistics (ratio of faculty to applicants, duration and number of interviews, and duration of interview day), and interview format preferences across AUA sections. All analyses were conducted using SAS 9.4 software (SAS institute, Cary, NC). A significance level of 0.05 was used for statistical testing.

Results

Survey invitations were directly distributed by the SAU to 105 ACGME-accredited urology programs; there were a total of 89 unique program director responses for a response rate of 84.7%. All AUA sections were represented. Program director demographics are provided in Table 1 .

Table 1.

Program Director Demographics

	All respondents n (%)
All respondents	89 (100)
Gender
Male	65 (73)
Female	23 (26)
Did not answer	1 (1)
Years as Program Director
< 1 year	6 (7)
1-3 years	25 (28)
4-5 years	20 (23)
6-10 years	17 (19)
>10 years	21 (23)
Program characteristics (residents per class)
1 resident per class	7 (8)
2 residents per class	33 (37)
3 residents per class	33 (37)
4 residents per class	13 (15)
>5 residents per class	3 (3)
Administrative Position
Assistant Professor	12 (14)
Associate Professor	34 (38)
Professor	25 (28)
Clinical Assistant Professor	4 (4)
Clinical Associate Professor	6 (7)
Clinical Professor	8 (9)
AUA Section
Mid Atlantic	9 (10)
New England	6 (7)
New York	9 (10)
North Central	20 (23)
Northeastern	4 (4)
Western	13 (15)
South Central	11 (12)
Southeastern	17 (19)

Virtual Interviews: Applicant Selection Criteria, Cost, and Impact on Match Outcomes

Program directors were asked to rank specific criteria for candidate interview selection (Fig. 1 ). In descending order, the top three ranked considerations for interview selection included: letters of recommendation, clerkship grades, and USMLE Step 1 scores. Note that receipt of a preference signal was not included as choice. The virtual format was reported to be more economical than in-person interviews by 85% of PDs (71/84). The average reported cost savings per program was $,4000 (SD $3,100).

FIGURE 1.

Program Director rankings of specific criteria for candidate interview selection. Program Directors were asked to rank the following: 1) United States Medical Licensing Exam (USMLE) Step 1 score, 2) USMLE Step 2 Score, 3) MSPE, 4) Class Rank, 5) Letters of Recommendation, 6) Alpha Omega Alpha, 7) Clerkship Grades, 8) Gold Humanism Honor Society, 9) Medical School Reputation, 10) Geographical factors, and 11) Research.

Overall, 62% of PDs (51/83) reported they were able to accurately represent their training programs via the virtual platform. With regard to the ability to assess applications, 51% (42/83) of PDs felt virtual interviews did not afford the same level of assessment as in-person interviews, compared to 40% (34/83) of PDs who felt the platforms were tantamount. A negative impact of VIs on 2020-2021 match outcomes was reported by 19% (16/83) of PDs, whereas on the other end, 18% (15/83) of PDs reported that VIs had a positive impact their match outcomes.

A total of 51% (42/83) of PDs expressed a desire to have VIs be included in future application cycles (Fig. 2 ) in either a virtual only (26%) or hybrid (25%) format. In-person interviews were reported to be preferred by 42% (35/83) of PDs, and 7% (6/83) of PDs felt that the choice of interview format should be institution-specific.

FIGURE 2.

Program Director interview format preference. Respondents were asked to select their preference for conducting interviews for future application cycles.

Virtual Interview Logistics

The median number (IQR) of applications received and reviewed after an initial screen was 350 (281-400) and 270 (150-366), respectively. A median of 40 (32-49) applicants were invited for interviews. Compared to in-person interview years, 57% (47/82) of PDs reported no difference in the number of applicants interviewed, whereas 34% (28/82) reported an increased number of interviews. There were no differences in the total number of applicants interviewed across AUA sections (p = 0.68).

VIs were most often held on weekdays (72%). Approximately 52% of PDs reported faculty were able to remain clinically productive on interview days, although a comparable proportion of 48% felt the contrary. The majority of programs (80%) interviewed a total of 36 to 55 applicants, with most PDs (73%) reporting interviewing 10 to 20 applicants per day.

A typical interview day for faculty consisted of 5 to 8 separate VI (66%) lasting 11 to 15 minutes each (64%) or 16 to 20 minutes (27%) in duration. During a VI day, applicants meet with 4 to 10 (71%) or >10 faculty (28%) with a 2:1 (53%) or 1:1 (36%) faculty to applicant ratio per interview.

Compared to prior in-person interview formats, 40% of PDs reported changes to the pre-existing interview structure and cited the following as reasons for change: zoom fatigue, being unable to perform aptitude assessments, and increasing the faculty to applicant ratio per interview. Organizational changes reported to the interview day included shorter interviews, less down-time between interviews, use of faculty panels and resident/faculty interviewer pairings, increased number of interviews, an overall shorter interview day, use of breakout rooms, and the use of standardized questions. Most PDs (77%) reported interviewers not being blinded to applicant information prior to the VIs. Of those who reported being blinded, several PDs reported excluding USMLE scores, pictures, and various demographic information (date of birth, citizenship and/or visa status).

Virtual Interview Preparation and Training

Formal training (Fig. 3 ) provided to interviewers most commonly consisted of the following topics: diversity equity and inclusion (DEI) (55% of programs), implicit bias (66% of programs), and review of SAU guidelines on illegal questions (83% of programs). Materials and endeavors utilized for these trainings included SAU guidelines/recommendations, materials provided by the Graduate Medical Education office, institutional DEI training programs, and Grand Rounds presentations.

FIGURE 3.

Training and education materials/endeavors used by programs for training prior to virtual interviews. Respondents reported formal training in the following prior to interview season: 1) Virtual interviewing, 2) use of the virtual platform, 3) implicit bias and other common biases, 5) diversity and equity training, and 6) training on illegal questions per SAU guidelines.

Equity in the Virtual Interview Process: URM and Gender Considerations

The approach to recruitment of URM and females was reported to be no different from the rest of the applicant pool by 48% and 51% of PDs, respectively. Over half of PDs felt neutral regarding the use of VIs to increase program visibility to URM (53%) or female (42%) applicants, with only 15% and 24% reporting improved visibility respectively (Fig. 4 ). The VI format was thought to increase the ability to interview URM and female applicants by 24% and 11% of PDs, respectively. Two-thirds of PDs surveyed believed the VI platform helped increase access for all applicants to attend interviews. There was no difference this perception of increased interview accessibility across different AUA sections (p = 0.65).

FIGURE 4.

Program Director perspective regarding equity through the use of a virtual interview platform. Program Directors were asked to indicate their level of agreement based on statements above.

Discussion

The SAU Taskforce on optimizing the VI experience was established to better assess the impact of this platform on the experiences of key stakeholders (programs and applicants) and resultant Match outcomes. This survey-based study gathered data from PDs of 89 urology residency programs (84.7% response rate) across the United States regarding their experiences with VI. Overall, 85% of programs reported a reduction in costs, and over two-thirds noted the VI platform helped promote applicant access to attend interviews. A total of 62% of PDs agreed they were able to portray their program accurately using this platform. Despite these findings, there continues to be no consensus among program leaders with regard to the efficacy of VI to both comprehensively assess applicants or to recruit URM or female applicants. There are mixed reviews about the continued use of virtual platforms in the resident interview process.

There have been questions and concerns raised regarding efficacy of VI to assess applicant academic motivations, professionalism, interpersonal skills, and degree an applicant could best be incorporated into the residency program.^{3 , 4} Understanding the effectiveness of the VI format in the ability to evaluate a prospective applicant has been a major focus for the SAU. In a previous cross-sectional study of 42 PDs of ACGME-accredited urology programs,⁵ over two-thirds felt that the VI format was less effective than in-person interviews in subjectively assessing whether an applicant would incorporate well into the current residency program, their commitment to urology, personality and communication skills and their ability to function as a resident. As a result half of PDs in their study reported a shift towards objective metrics in ranking applicants compared to prior years. Our study included responses from 83 PDs, and close to half (51%) similarly felt that VIs did not allow for the same level of subjective assessment as in-person interviews while 40% (34/83) believed the 2 interview techniques were equivalent. It is unclear how the decrease in subjective assessment with the VI format may have impacted program rank list, but in our study only 19% of PDs felt to the VIs negatively impacted their 2020-2021 Match outcomes. In fact, 57% did not believe the VI format negatively impacted the Match at all.

What defines an optimal interview experience is not clear and dependent on the stakeholder. What the candidate perceives as ideal may vary from what is needed for the program. Prior in-person interview days were traditionally conducted at the discretion of the residency program and were composed of either a series of individual, paired, or panel interviews. Within the VI cycle, overall, we found that over two-thirds of programs held 5 to 8 separate VIs per day, applicants meet with 4 to 10 faculty in 2:1 (53%) or 1:1 (36%) faculty to applicant ratios, and interviews occurred primarily during weekdays (71%). This is in agreement with a previous study of senior medical students applying to urology residency programs that favored 5 to 7 total interviews comprising all faculty in an interview day that lasted half to three-fourths of the work day.⁶ The survey explored the logistics of the VI day compared to in-person interviews from the program's perspective as well. Overall, 40% of PDs reported changes to the interview day due to concerns of zoom fatigue and no ability to perform aptitude testing. Commonly cited changes including shifting to the use of interviewer pairings or panels per applicant, shorter and more structured interviews with use of standardized questions. Use of the multiple mini-interview and structured interviews with standardized questions has been shown to increase diversity,⁷ assess non-cognitive traits,⁸ and decrease bias^{9 , 10} in the interview. The authors hope to continue to analyze subsequent application cycles to review VI processes that lead to the most optimal structured and standardized approach to evaluating applicants.

Prior to the institution of VI, an SAU Taskforce was established in March of 2018¹¹ to help address violations12, 13, 14 of the AUA Urology Match Guidelines.¹⁵ The Taskforce put forth a set of recommendations to help adopt the National Residency Match Program's (NRMP) Code of Conduct. These recommendations included: discouraging programs from asking applicants to disclose rank considerations, refraining from asking illegal or coercive questions, declining to require second look visits, discouraging postinterview communication, and providing faculty development to help standardize the interview process and minimize explicit and implicit bias. In our study, only 55% of PDs reported preinterview formal training regarding illegal questions (described SAU guidelines), DEI, or implicit bias. Even fewer PDs (39%) reported formal training on virtual interviewing. Most commonly reported educational materials/endeavors utilized by programs for interviewer preparation included SAU guidelines/recommendations, material provided directly from the office of Graduate Medical Education, institutional-specific DEI training and program-specific Grand Rounds presentations for faculty. This is an area that requires further exploration regarding ongoing faculty development initiatives to uphold the most ethical standards of conduct during the Match process.

Another area of our study that merits discussion involves patterns in URM and female applicant interview recruitment using a purely virtual platform. Approximately 12% and 11% of PDs believed VIs improved program visibility for URM and female applicants, respectively. Few PDs felt the use of a VI format increased the ability of programs to offer more interview invitations to URM (31%) and female (15%) applicants. Ultimately respondents remained divided as the efficacy of VIs to recruit these traditionally under-represented groups (53% for URM recruitment and 57% for the recruitment of women). Leveraging the virtual interviewing platform to promote candidate diversity remains an active area of interest for ongoing work.

Urology residency programs have previously reported experiencing an average of ∼$3,000 in interview related costs per cycle.⁵ In our study, the majority of PDs similarly reported average cost savings of $4,000 per program. The most commonly cited reasons for the reduction in interview-related costs included: absence of pre- or postinterview dinners, social events, and interview day meals. Undoubtedly, VIs has significantly reduced the financial strain on both applicants and programs. Financial burden to programs is not just in cost of the interview day but also in lost clinical productivity. About 52% of PDs reported accommodation of clinical activity for part of the day in the process of scheduling interviews. It is unclear if these findings are unique to VIs.

Our study has several limitations that warrant discussion. First, there was a total of 143 urology residency programs that participated in the 2020-2021 AUA Match.¹⁶ Our study captured 89 unique PD responses (58%) making generalizability of our findings limited. Nevertheless, our response rate (84.7%) was particularly high, given 105 programs were contacted by the SAU and included in the study. Perspectives from PDs in all AUA sections were represented. Second, the study was cross-sectional in nature, and therefore we are unable to assess the changes over time that may have occurred with successive cycles of virtual interviewing. Third, recall bias may affect the reported results, as the survey was distributed following interview season. Future studies comparing resident selection between in-person and the VI format will be needed to better understand the efficacy across interview platforms and the implications on Match outcomes.

Conclusion

To the best of our knowledge, this is the largest nation-wide investigation of urology PD perceptions of virtual interviewing and resident recruitment process. The platform seems to afford cost savings for programs and is perceived to expand applicant access to attend interviews. Nevertheless, PDs do note limitations in their ability to comprehensively assess an applicant during the purely virtual encounter. Half of PDs expressed interest in continuing the VI format in some form; a minority (19%) reported a negative impact of VI on Match outcomes. Future work is needed to determine the impact of this platform on residency recruitment and Match outcomes. There may be a role for the continued development of resources, including those pertaining to Match guidelines and implicit bias training, to enhance the interview experience.