In-Person Second Look During a Residency Virtual Interview Season: An Important Consideration for Radiology Residency Applicants

Abstract

Rationale and Objective

The transition to an all-virtual residency interview process, in response to the COVID-19 pandemic, has seen numerous benefits beyond applicant and program safety. These benefits include improvements in equity, access and even lowering the applicant's carbon footprint. However, a significant portion of applicants have concerns with being unable to visit a program and interact with residents and faculty in person. A second look opportunity for radiology residency applicants was developed to address applicant concerns while maintaining an equitable interview process for all.

Materials and Methods

The second look opportunity took place after our program's final rank list was completed. Second look attendees completed a 10-question online survey after completion of our program's second look to ascertain financial obstacles of attending, the importance of a second look opportunity, and reasons why applicants chose to attend.

Results

24/24 (100%) of attendees completed the survey. The majority of attendees were from >100 miles away (54%). Only 21% felt that the financial burden of the second look was substantial. However, this response was more common among attendees from >100 miles away (p = 0.013). All applicants surveyed chose to attend the second look in order to see the radiology department facilities and to meet the faculty in person. All applicants agreed or strongly agreed that they learned more about the residency program through in-person discussions with residents and faculty than they were able to remotely and that the in-person second look opportunity is a valuable tool to help make an informed rank decision when interviewing virtually.

Conclusion

The second look opportunity offers multiple benefits for applicants to help make an informed rank list decision during a virtual interview season.

Introduction

Over the past two interview application cycles (2020-2021 and 2021-2022), the residency and fellowship interview format abruptly transitioned from the traditional in-person to a virtual interview format in response to the COVID-19 pandemic. This rapid transition was the result of recommendations from the Coalition for Physician Accountability, a cross-organizational group composed of multiple stakeholders including the Association of American Medical Colleges (AAMC) and National Resident Matching Program (NRMP) (1, 2, 3). A position statement from the Association of Program Directors in Radiology (APDR) and the Association of Program Directors in Interventional Radiology (APDIR) supported and reinforced this position for all diagnostic and interventional radiology programs (4). Radiology program directors immediately pivoted to this new virtual interview format through collaboration and sharing best practices (5).

After what was largely viewed as a successful virtual interview season in 2020-2021, additional benefits of continuing the virtual interview format became clear. Both programs and applicants rated virtual interviews highly (6). Benefits from virtual interviews went far beyond the safety of applicants and interviewers during this ongoing pandemic. One additional benefit of the virtual interview format over the traditional in-person interview is the significant cost savings for applicants, improving equity and access for all applicants (7). Most medical students are already burdened with debt, and expenses incurred from the application process and interview season travel can easily exceed several thousand dollars; a potentially restrictive financial burden for some applicants (7). Decreasing travel among applicants also contributes to reducing the carbon footprint of the interview season, an aspect that nearly half of applicants feel very strongly about (7).

Despite the resounding success of the first virtual interview season, there have been concerns raised among applicants with a strictly virtual interview season (6). One primary concern includes adequately understanding the culture of a program and how it relates to program “vibe”, which normally becomes apparent through more casual conversations and interactions during the interview day. Applicant lack of familiarity with the program's city or surrounding area is also concerning for some applicants with one survey showing that almost one-third of applicants were “reluctant to rank programs in cities with which I was not familiar” (6). An additional concern to strictly virtual interviews is parity for applicants who may not have a home academic radiology department to acquire a general familiarity with a radiology residency and allow for a better understanding of the applicant's own priorities in searching for an optimal training program (6).

Radiology programs have largely adapted and shared best practices to address some of these concerns. Expanding and optimizing departmental websites, social media accounts, program videos, and even virtual reality have all been suggested as ways to improve the applicant's understanding of the program culture, facilities, and immediate geographic area (5, 8, 9, 10, 11, 12, 13, 14, 15).

Our department institutional officer (DIO) in agreement with the graduate medical education committee (GMEC) approved second look opportunities for applicants to address some concerns raised over a strictly virtual interview season. Our radiology residency program made the decision to provide applicants with this optional second look opportunity while ensuring the event had no influence on resident selection or ranking. Efforts were also made to provide this event with as little expense to the applicant as possible. The authors hypothesize that providing a second look opportunity for radiology residency applicants conveys additional important information and perspective about the radiology program not easily replicated in the virtual environment.

Materials and Methods

The IRB deemed this study exempt from IRB review and approval. Residency interviews were conducted in a virtual format per recommendations from the AAMC. Prior to the second look, each applicant participated in one of ten virtual interview days conducted during either a week in November or a week in December. The virtual interview included four 15-minute interviews with members of the resident selection committee, a midmorning round table discussion with the department chair, a short high yield resident noon conference, and finally rotating small group breakout rooms in which each applicant was paired with 2-3 residents. Invitations to a second look event to take place at a hotel adjacent to the primary hospital site were sent to all applicants who participated in a virtual interview. Applicants were informed at the virtual interview and on follow-up emails that the second look event would take place after the rank list was finalized. In efforts to minimize the cost incurred by the applicants, a one-night hotel stay would be provided at the hotel if needed. The majority of the residents and faculty who participated in the programming were not involved in the interview or selection process, in hopes of expanding the applicants’ view of the department.

Location

The second look event took place in the city of Cincinnati, Ohio. The city of Cincinnati is located in the Midwest and has an international airport across the Ohio River in northern Kentucky, approximately 20 miles from the University of Cincinnati Medical Center (UCMC). Cincinnati is also served by Amtrak passenger trains which make three weekly trips between Chicago and New York City, with the train station located approximately 5 miles from UCMC. Within the city, a streetcar connects much of the downtown region, although the closest stop only comes within two miles of UCMC.

Safety

The event took place in a large conference room with multiple spaced-out roundtables and intermittent seating to allow for social distancing. All applicants, residents and faculty were required to wear masks except when eating or drinking. Hand sanitizers were available at all entry sites and within the conference room.

Itinerary

The final day's itinerary for the second look was sent to all respondents 3 weeks prior to the second look which would take place on January 21^st, 2022. After a short welcome from the Program Director and Associate Program Directors, the applicants ate lunch and socialized with the residents. Applicants were then broken into 4 small groups and lead on walking tours of the primary hospital site, and the adjacent children's hospital and Veteran Affairs hospital. Applicants then returned to the conference room for short 5-10 minute presentations from faculty focusing on topics unique to their area of interest which included health care advocacy, international volunteer opportunities, informatics, and 3D printing. A final ice cream float social event farewell took place among applicants, faculty, residents and fellows.

Survey

At the end of the second look, a QR code linking to a short 10-question survey was distributed to all attendees (see Appendix A). The survey was created, and data was collected using SurveyMonkey (Palo Alto, CA). Applicants were sent an email reminder to fill out the survey three days after the second look and the survey closed one week after the second look. Data was exported to Microsoft Excel for analysis. Survey results were analyzed using Fisher exact test and p values <.05 were considered significant.

Results

A total of 121 virtual interview applicants received invitations to the second look event. From this group, 24 applicants attended the event. Twenty-four surveys were completed by the applicants for a response rate of 100%. The majority of attendees were from areas >100 miles away (13/24, 54%) and a large portion were from >200 miles away (10/24, 42%). Only a small minority of applicants (4/24, 17%) were from 0-50 miles away (Figure 1 ). Only 21% of applicants felt that the financial burden of the second look was substantial (5/24, 21%). However, this response was more common among attendees from >100 miles away (p-value=0.013). A minority of applicants agreed (6/24, 25%) that efforts to offset the financial burden through complimentary hotel room and provided meals influenced their decision to attend the second look, the majority of applicants (13/24, 54%) felt neutral. There was no statistical significance between applicants from further away being influenced to attend more by the budgetary offset of complimentary hotel rooms and meals (p-value = 0.462) or applicants from further away having attended the second look being more likely to have this event influence their rank list (p-value = 0.489) (Table 1 ).

Figure 1.

Distance the second look applicants traveled to attend the event. Forty-two percent of applicants traveled from >200 miles away, 29% traveled from 50-100 miles away, 12% traveled from 100-200 miles away, and 17% traveled from 0-50 miles away.

Table 1.

Survey results show a statistically significant relationship between the distance applicants travelled and perceived financial burden.

Statistical Measurements for Significance Comparing Relationship Between Second Look Applicant Distance Traveled to Survey Question Responses.
Applicant response	Total number of responses to Strongly Agree, Agree or Neutral from >100 miles (% of total)	Total number of responses to Strongly Agree, Agree or Neutral from 0-100 miles (% of total)	Statistical Relationship to Strongly Agree, Agree or Neutral (p value)
Felt Financial Burden was more substantial	4 (31)	1 (9)	0.013
Influenced the Applicant's Rank List	5 (45)	7 (53)	0.489
Were Influenced to Attend the Second Look by Budgetary Offset Measures	10 (91)	9 (69)	0.462

When asked about reasons to attend the second look, all survey applicants selected to “See the radiology department facilities” and “Meet the faculty in person.” Almost all of the applicants selected to “Meet the residents in person” (23/24, 96%) and a little over half of the applicants chose to attend to “Learn more about living in the city of Cincinnati (14/24, 58%). Three applicants chose “Other” with two of the three filling in to “Meet other applicants” (8%) (Figure 2 ). Despite a little over half the applicants choosing to attend the second look to learn more about the city (14/24 58%), a strong majority (19/24, 79%) felt that the second look helped them get a better feel for the city with four of the responses choosing “Not applicable”, presumably due to already being familiar with the city prior to the second look.

Figure 3.

The applicants gained a better feel for the city of Cincinnati after the second look with 95% of applicable responses Strongly Agreeing or Agreeing. Four applicants selected “Not applicable” presumably being already familiar with the city before the second look. (Color version of figure is available online.)

Figure 2.

Reasons for attending the second look event. One hundred percent of applicants chose to see the radiology department facilities and meet the faculty in person and 96% selected to meet the residents in person. Fifty-eight percent selected to learn more about the city while 13% chose “other”. (Color version of figure is available online.)

All applicants agreed or strongly agreed (7/24, 29% and 17/24, 71% respectively) that they learned more about the residency program through in-person discussions with residents and faculty than they were able to remotely. Additionally, all applicants agreed or strongly agreed (6/24, 25% and 18/24, 75% respectively) that the in-person second look opportunity was a valuable tool to help make an informed rank decision when interviewing virtually. Nearly all applicants agreed or strongly agreed (9/24, 38% and 12/24, 50% respectively) that the in-person second look influenced their rank list (Table 2 ). When asked in the future if a hybrid model system of both virtual and in-person residency interviews is conducted, should the radiology department also offer an optional in-person second look to all of those applicants who interviewed virtually, all responded yes (24/24, 100%).

Table 2.

There was unanimous agreement among applicants attending the second look that they learned more about the residency program through in-person discussions with residents than they were remotely and that the in-person second look is a valuable tool to help make an informed rank decision.

Second Look Applicant Responses to Benefits of Second Look Program During the Virtual Interview Season
Survey Question	Strongly Agree/Agree	Strongly Disagree/Disagree	Neutral
I learned more about the residency program through in-person discussions with residents and faculty than I was able to remotely.	24 (100%)	0 (0%)	0 (0%)
The in-person second look opportunity provided by the department of radiology is a valuable tool to help me make an informed rank decision when interviewing virtually.	24 (100%)	0 (0%)	0 (0%)
The in-person second look has influenced my rank list.	21 (87.5%)	0 (0%)	3 (12.5%)

Half of the applicants (12/24) chose to share their final thoughts on the second look experience in addition to opportunities to improve the second look experience. Five of these 12 comments specifically mentioned gaining important insight into the “atmosphere”, “fit”, or “culture” of our program. Additional specific responses include:

• 1“For me, it was easier to ask residents questions about the program in person rather than over zoom.”
• 2“I highly appreciate that radiology leadership made sure to emphasize that second look would occur after rank decisions.”
• 3“This alone (second look) will result in me ranking this program higher as I haven't had the opportunity to see any other programs in person.”
• 4“…it did allow me to gain a different perspective that made me more confident in my rank compared to places I had just interviewed with virtually.”
• 5“Was really important to meet the residents, faculty, and other applicants. Zoom does not do a good job of connecting applicants (future co-residents) with one another.”

Discussion

The purpose of this study was to determine the importance of second look opportunities for applicants during the virtual radiology residency interview season. Previous manuscripts have supported the notion that the virtual interview format overall is an effective way to safely conduct residency interviews during a global pandemic, but additional benefits beyond safety including equity, cost, and effect on climate change, support continuing virtual interviews indefinitely (7, 10, 11, 14). However, there remain drawbacks for a significant portion of the applicant pool who desire the opportunity to have some form of in-person experience available in order to thoughtfully measure and compare programs, for which they will spend the next four to six years of their training.

Other residency and fellowship programs have implemented second look opportunities to address shortcomings of an all-virtual interview season. A significant majority (85%) of urology residency applicants surveyed during the 2020/2021 interview cycle described visiting the city of the program they were applying had a moderate, large or very large impact on their rank list (16). In addition, only 5% of urology applicants agreed that a city visit could be replicated in an online platform (14). This compares with just over half of our radiology second look applicants listing a reason for attending the second look was to learn more about the city. Moran et al. describe nearly one-third of radiology applicants surveyed during the virtual interview season as “reluctant to rank programs in cities with which I was not familiar” (7). For a subset of applicants, having the opportunity to familiarize themselves with the city and geography is pivotal when formulating preferences in training programs.

Program culture or “vibe” remains a top consideration for applicants when formulating their rank list (17). Program innovations including enhanced websites with descriptive videos, virtual reality, increased engagement through social media, and virtual meet and greets have likely significantly improved the transparency of program culture for the applicant, but not to the point of replacing in-person experiences. Unstructured interactions participated in and observed by applicants throughout the second look both added to and enhanced perceptions of our program obtained during the virtual interview day. One hundred percent of respondents on our survey agreed they were able to learn more about our residency program through in-person interactions with residents and faculty than they were able to virtually. This overwhelming response is even more impressive given the extensive emphasis our program already places on illuminating our program's culture through small group resident break-out rooms with applicants during the virtual interview day, multiple active social media accounts, robust transparent residency website, and professionally made videos highlighting our program's culture.

There remain many concerns with mixing both in person and virtual experiences during the residency interview process, most of which center around equity. For this reason, the AAMC released guidance for the 2022-2023 interview cycle that encourages a single interview format (virtual or all in-person) and strongly discourages a hybrid system that could further exacerbate inequities if applicants from disadvantaged backgrounds interviewed virtually at a higher rate (18). There is also potential for inequity with second look opportunities if only applicants with financial means attend and the second look attendance influences the resident selection process. Therefore, it is essential that any second look opportunity occur after the completion of the resident selection process, and consideration is taken to minimize interactions between applicants and those faculty that participate in the resident interview/selection process (17, 19). This disassociation between the second look and resident selection process was emphasized during our virtual interview days and in all email correspondence with the applicants.

Applicant cost burden in traveling to second look opportunities is another important consideration and significant advantage of the virtual over in person interview format. Our attempts to mitigate these costs through provided meals and complimentary hotel rooms was likely reflected in the majority of second-look attendees disagreeing (79%) that the financial burden was substantial. However, when neutral and agree answers to this question were combined, a significant correlation (p=0.013) between those living >100 miles away tended to believe there was more of a financial burden. This could be explained by other costs not reimbursed by the department including time spent attending the second look as well as flights, gas, and additional travel related expenses.

In making plans for this second look opportunity, our program, based on discussions with our resident selection committee, correctly anticipated a 15-20% attendance rate from our virtual interview applicant pool. What our program did not anticipate was the excitement and overwhelming positive response from our cohort of applicants toward the second look based on our survey results and 100% response rate to the survey. Universal agreement among the second-look attendees that they learned more about the residency program through in-person discussion with residents and faculty than they were able to remotely and that the in-person second look was a valuable tool to help them make an informed rank decision when interviewing virtually was astounding. Additionally, 88% replied that the in-person second look influenced their rank list. These results support the notion that for some applicants, the option to have a low-stakes in-person experience at a program is extremely important and influential in their overall residency program assessment and rank list decision.

An additional unexpected observation in the survey was the opportunity to meet other fellow applicants in person. To the authors’ knowledge, the importance of in person interactions between residency applicants on the interview trail has not been surveyed but may be an important consideration when evaluating the need for a second look opportunity during a virtual interview season. The bonds formed among applicants during this shared experience could set up important connections and friendships as they matriculate into residency and become radiologists.

Most diagnostic radiology residency programs interview applicants from October to January with the NRMP opening rank list selection the beginning of February with a deadline for the applicant and program to certify the rank order list in the beginning of March. (20) This leaves a short window to conduct a second look opportunity between the finalization of the program's rank list and the applicant's rank order list certification deadline. If more programs adopt second look opportunities there is significant potential for overlapping dates, limiting the applicants’ ability to attend all of the second looks that they desire. In order to provide the maximum amount of second look opportunities for residents, the APDR, NRMP, and AAMC may need to revisit the match calendar with recommendations to complete the virtual interviews earlier and expand the time frame for second look opportunities.

This study does have limitations in that this was a single institution sample size and small number of applicants surveyed. This small number, in addition to the uniform agreement among those surveyed, made it challenging to find statistically significant correlations among this cohort of applicants. However, the response rate of 100% and near unanimous agreement for certain questions suggests that these second look opportunities should be strongly considered for programs offering only a virtual interview option. Future larger multi-institutional studies should be considered to further evaluate the importance of the second look opportunity in addition to exploring reasons why applicants chose not to attend.

Conclusion

The second look opportunity for applicants during an all-virtual interview season offers numerous benefits for applicants and implementation should be strongly considered by radiology programs. This in-person experience for some applicants provides essential information on a program's culture, geography, and department facilities and ultimately can impact where the applicant ranks a program.

Appendix A

• 1)
  What were the reasons you chose to attend the second look? (Choose all that apply)

  • a
    See the radiology department
  • b
    Meet the residents in person
  • c
    Meet the faculty in person
  • d
    Learn more about living in the city of Cincinnati
  • e
    Other (please specify)
• 2)
  How far did you travel to attend the second look?

  • a
    0-50 miles
  • b
    50-100 miles
  • c
    100-200 miles
  • d
    >200 miles
• 3)
  I learned more about the residency program through in-person discussions with residents and faculty than I was able to remotely.

  • a
    Strongly agree
  • b
    Agree
  • c
    Neutral
  • d
    Disagree
  • e
    Strongly disagree
• 4)
  The second look opportunity helped me get a better feel for the city of Cincinnati.

  • a
    Not applicable. I was already familiar with the city
  • b
    Strongly agree
  • c
    Agree
  • d
    Neutral
  • e
    Disagree
  • f
    Strongly disagree
• 5)
  The financial burden of attending the second look was substantial.

  • a
    Strongly agree
  • b
    Agree
  • c
    Neutral
  • d
    Disagree
  • e
    Strongly disagree
• 6)
  Efforts to offset the financial burden to applicants through complimentary hotel rooms and provided meals influenced my decision to attend the second look.

  • a
    Strongly agree
  • b
    Agree
  • c
    Neutral
  • d
    Disagree
  • e
    Strongly disagree
• 7)
  The in-person second look opportunity provided by the department of radiology is a valuable tool to help me make an informed rank decision when interviewing virtually.

  • a
    Strongly agree
  • b
    Agree
  • c
    Neutral
  • d
    Disagree
  • e
    Strongly disagree
• 8)
  The in-person second look has influenced my rank list.

  • a
    Strongly disagree
  • b
    Agree
  • c
    Neutral
  • d
    Disagree
  • e
    Strongly disagree
• 9)
  In the future, if a hybrid model system of both virtual and in-person residency interviews is conducted, should the radiology department also offer an optional in-person second look to all those who interviewed virtually?

  • a
    Yes
  • b
    No
  • c
    Unsure
• 10)Please let us know any final thoughts or opportunities to improve this experience.