Perceptions of a Virtual Interview Exercise for Ophthalmology Residency Applicants

Abstract

Importance  In-person interviews have traditionally been considered a crucial component of the residency application process. Virtual interviews (VIs) became the standard format for the 2020 to 2021 application cycle due to the novel coronavirus disease 2019 (COVID-19) pandemic. VIs offer a new perspective and challenge to this process which warrants unique considerations and further understanding of effects on applicants.

Objective  This study aimed to assess the perceived efficacy of a VI preparedness exercise for ophthalmology residency applicants in the 2021 residency application cycle.

Design, Setting, and Participants  A cross-sectional survey was distributed online. All participants in a mock VI exercise conducted via video-telecommunication technology were invited to complete the survey. Data collection occurred from October 12, 2020, to November 2, 2020. A follow-up survey after a match results released was distributed to all participants and data collection occurred from February 18, 2021, to February 25, 2021.

Main Outcome and Measures  Applicant demographics, comfort, and attitudes toward VIs and VI practice were the primary measurements of this study.

Results  Responses to the initial survey were received from all 35 participants (100%) in the VI mock interviews. There was a statistically significant difference between the pre- and postinterview responses for “How prepared do you feel for virtual interviews with residency programs?” ( p  = 0.0003) and “How likely are you to practice virtual interviews with someone you know?” ( p  = 0.0023). Participants reported feeling more prepared for VIs with residency programs after the mock interview ( p  = 0.002). A greater proportion of participants responded with “Very Likely” after the mock interview in comparison to before the interview to the questions “How likely are you to practice interviews with someone you know?” ( p  = 0.039) and “How likely are you to practice virtual interviews in the same room/area as you will during the official interview season?” ( p  = 0.021). Of the 35 original participants, 20 completed the follow-up survey. There were an equal number of participants who responded either “Helped Somewhat” ( n  = 9) or “Helped Greatly” ( n  = 9) to “How much did the VI mock exercise help you for the actual interview season?” in the follow-up survey. The majority of follow-up survey respondents (17/20) reported that they had additional practice in the virtual environment for interviews after the VI mock exercise. There was no significant difference in perceived helpfulness of the VI mock exercise during the actual interview season between matched and unmatched participants.

Conclusion and Relevance  As residency applicants prepare for future VIs, practice and adequate preparation will be essential. In this study, implementation of a VI preparedness exercise had a positive impact on applicants' perception of their preparedness and intention to practice the format in the future.

On May 8, 2020, the Program Director Council (PDC) and the Medical Student Education Councils (MSEC) of the Association for University Professors of Ophthalmology (AUPO) in conjunction with the San Francisco Match and AUPO Board of Trustees released recommendations for the 2021 Ophthalmology Residency Application Cycle. ¹ These recommendations were published in response to the changing national landscape brought about by the novel coronavirus disease 2019 (COVID-19) pandemic and its anticipated impact on the residency application and interview cycle. These recommendations addressed away rotations, research, community service/volunteering, United States Medical Licensing Examination (USMLE) step 2, the medical student performance evaluation (MSPE), letters of recommendation, and the general application timeline. Included in these recommendations was guidance urging residency programs to conduct virtual interviews (VIs) instead in-person interviews. The VI format offers new challenges and opportunities for both residency programs and applicants.

In a 2013 survey, Yousuf et al reported different factors that influenced how applicants ordered their rank lists when applying to ophthalmology residency programs. ² One of the highest ranked factors was applicants' perception of the relationship between faculty members and residents. Factors associated with lower rankings included perception of low resident morale and poor interview experience. These findings were predicated on the ability of applicant's interaction with faculty and residents face-to-face. Under the new VI format, opportunities for this type of engagement would theoretically be limited. Still, the VI is the applicant's opportunity to share their story, personality, and qualifications. With the change in format, dependence on technology and limitations of nonverbal cues, the virtual platform requires a level of comfort with new variables.

A multidisciplinary task force at the University of California San Francisco found that the following were key areas to address to allow for evidence based best practices: adopting standardized interview questions, consciously addressing implicit bias, providing practice opportunities to trainees, hosting social events, and collecting data for assessment of the virtual process for further exploration. ³ This dialogue in other specialties led a group of residency program directors in pediatrics to identify several specific factors to address when preparing for the official interview day. These include correcting issues with technology that prohibit clear and smooth communication, providing phone numbers to use as backup in case technical difficulties arise, and addressing challenges with lighting and camera angle. These factors led researchers to suggest recording practice sessions to test these variables that are unique to the virtual setting.” ⁴ To give prospective applicants, in this year's application cycle, a source of familiarity with these considerations in mind, we created a medical student–led mock VI initiative for the 2020 to 2021 cycle of ophthalmology residency applicants and conducted a pre- and post-survey of participants to assess the perceived effectiveness of these sessions in improving applicants' comfort with the VI format.

Methods

Mock Virtual Interviews

This study adhered to the tenets of the Declaration of Helsinki. The University of Miami Institutional Review Board (IRB) waived the need for IRB approval. Mock VI practice opportunities were developed by the authors (B.A., V.L., A.P., A.W., S.A., and .A.C). These mock VI sessions were free, student led, nonsponsored, and not affiliated with specific institutions. Announcements were marketed and disseminated via social media platforms including Twitter and Instagram starting on August 2, 2020. A Google document requesting name, phone number, e-mail, and medical school of each voluntary participant was made available on August 22, 2020, and announced on the previously listed forums along with other additional forums. These additional forums included the Honestly Bilal podcast, the “OphthoMatch 2020–2021” applicants' discussion Google sheets document (Alphabet Inc., Mountain View, CA), and the associated anonymous discussion channel for all applicants on the web application Discord (San Francisco, CA). Sign-ups were closed once a maximum of 60 participants had been reached.

Once the sign-up was closed, the mock interviewers internally coordinated availability via the web site, namely, SignUp Genius (SignUp Genius, Charlotte, NC). Once finalized, a separate SignUp Genius link was created and distributed via e-mail on September 28, 2020, to participants who then picked a date and time of their choice between October 12, 2020, and November 2, 2020. Using the Zoom (Zoom Video Communications, San Jose, CA) video-telecommunications platform, meetings were created, scheduled, and distributed to each interviewer and interviewee under a single account. To limit external bias and to promote a student-oriented collaborative culture of this event, no application materials or USMLE/COMLEX: Comprehensive Osteopathic Medical Licensing Examination of the United States scores were required or requested, and only the 6 originating student members of the initiative (BA, VL, AP, AW, SA, AC) assumed roles as mock interviewers.

Immediately before each 20-minute mock VI, each interviewee completed the preinterview survey which was distributed as a Google form ( Appendix ). All interviewees were given the option to have their session recorded for viewing at their own convenience once the call ended. Interviewers selected questions from a list of 19 questions from the article “The Interview Trail: How to Succeed on your Residency/Fellowship Interview,” written by J.S. ( http://www.retinapodcast.com/equal-round-and-reactive/2017/10/20/the-interview-trail-how-to-succeed-on-your-residencyfellowship-interview ). At the end of each VI, each interviewee was then sent the postinterview survey, another Google form ( Appendix ) which included identical questions and answer choices like preinterview survey. Interviewees were also sent the recording of the interview if they opted in to be recorded. A follow-up survey was distributed via e-mail to all 35 participants after the match starting on February 18, 2021. This Google form was closed on February 25, 2021.

Statistical Analyses

Descriptive statistics and data analyses were performed using IBM SPSS Statistics, version 27 (IBM Corp., Armonk, NY). The data were not normally distributed, so nonparametric tests were performed. The Kruskal–Wallis test, the nonparametric counterpart of one-way analysis of variance (ANOVA), was performed to determine if there was a difference in survey responses between groups divided by gender, ethnicity, region, and level of postgraduate education. The Wilcoxon's signed-rank test for hypothesis testing of repeated measurements on a single sample was used to evaluate whether or not the survey responses before and after the mock interview were different. All tests were two tailed, and test significance was determined at the α level of 0.05.

Results

Of the initial 60 participants interested who filled out the initial sign-up document, 41 followed up via the SignUp Genius form. Out of the 41 participants scheduled, 35 participants attended their respective sessions and completed both surveys, and 23 participants opted to record their session. In total, participants represented 27 allopathic medical schools, 2 osteopathic medical schools, and 5 international medical schools. When considering the demographic data of the participants ( Table 1 ), there were no statistically significant differences in survey responses between different genders, races/ethnicities, medical school locations, and level of postgraduate education. The average scores of the participants' pre- and postinterview survey responses on a 5-point Likert's scale are summarized in Table 2 .

Table 1. Demographic information of the virtual interview participants.

Characteristic	Number (%)
Gender
• Female	18 (51)
• Male	17 (49)
Race/Ethnicity
• White	19 (54)
• Asian/Pacific Islander	13 (37)
• Black or African American	1 (3)
• Hispanic or Latino	1 (3)
• Other	1 (3)
Medical school location by U.S. region
• Northeast	5 (14)
• Midwest	10 (29)
• South	8 (23)
• West	7 (20)
• International	5 (14)
Postgraduate degrees
• Masters of science	5 (14)
• PhD	1 (3)
Total number of participants	35 (100)

Table 2. Descriptive statistics of pre- and post-interview survey responses.

Survey question	Preinterview (mean ± SD)	Postinterview (mean ± SD)	p -Value a
How prepared do you feel for virtual interviews with residency programs?	2.9 ± 1.0	3.6 ± 0.9	0.0003 b
Mean difference in score (95% CI)	+0.66 (0.15–0.96)
How likely are you to practice virtual interviews with someone you know?	4.4 ± 0.7	4.7 ± 0.5	0.0023 b
Mean difference in score (95% CI)	+0.31 (0.13–0.50)
How likely are you to practice virtual interviews?	4.4 ± 0.5	4.8 ± 0.4	0.20
Mean difference in score (95% CI)	+0.11 (−0.07 to 0.30)
How likely are you to practice virtual interviews in the same room/area as you will during the official interview season?	4.3 ± 1.1	4.7 ± 0.6	0.07
Mean difference in score (95% CI)	+0.37 (−0.03 to 0.77)

Abbreviations: CI, confidence interval; SD, standard deviation.

Note: Likert's scale: 1 = very unprepared/unlikely, 2 = somewhat unprepared/unlikely, 3 = neutral/undecided, 4 = somewhat prepared/likely, 5 = very prepared/likely.

^aWilcoxon's signed-rank test.

^b Indicates statistically significant difference ( p  < 0.05).

Participants felt significantly more prepared for virtual residency interviews after the mock interview (mean rank = 164.5) than before the mock interview (mean rank = 6.5; Z  = − 3.57, p  = 0.0003). The median Likert-scaled scores were 3 for the preinterview and 4 for the postinterview surveys. Prior to the mock interview, only 12 participants had responded “Somewhat prepared” to the question, “How prepared do you feel for virtual interviews with residency programs?.” In the postinterview survey, most participants (22/35) responded “Somewhat prepared” ( Fig. 1A ).

Fig. 1.

( A ) Mock virtual interview participant self-reported preparedness pre and post mock virtual interviews. ( B ) Mock virtual interview participant self-reported willingness to practice interviews pre and post mock virtual interviews.

Participants were significantly more likely to practice VIs with someone they knew after the mock interview (mean rank = 84) than before the mock interview (mean rank = 7; Z  = − 3.05, p  = 0.0023). The median Likert-scaled scores of the survey question “How likely are you to practice virtual interviews with someone you know?” were 4 for the preinterview and 5 for the postinterview surveys. All of the participants who responded “Neutral/Undecided” in the preinterview survey ( n  = 4) changed their response to “Somewhat Likely” after the interview. In the postinterview survey, the majority of participants (24/35) responded “Very Likely” ( Fig. 1B ).

There were no statistically significant pre- and postinterview differences in the responses to the question “How likely are you to practice virtual interviews?” ( p  = 0.20). Most participants responded “Very Likely” to “How likely are you to practice virtual interviews?.” Most participants also responded “Very Likely” to “How likely are you to practice virtual interviews in the same room/area as you will during the official interview season?” ( Fig. 1B ). Of the five participants who responded in the preinterview survey “Very Unlikely,” “Somewhat Unlikely,” or “Neutral,” only one participant did not change their answer to “Very Likely” after the mock interview. However, there were no statistically significant differences in the pre- and postinterview responses ( p  = 0.07).

Of the 35 participants who attended the VI mock sessions, 20 completed the follow-up survey. There were an equal number of participants who responded either “Helped Somewhat” ( n  = 9) or “Helped Greatly” ( n  = 9) to “How much did the VI mock exercise help you for the actual interview season?” in the follow-up survey. There were two participants who responded to the same question with “Helped Very Little.” The majority of follow-up survey respondents (17/20) reported that they had additional practice in the virtual environment for interviews after the VI mock exercise and a majority ( n  = 13, 65%) practiced once to five times ( Fig. 2 ). The participants of the follow-up survey either did not have a preference for mock interviewers ( n  = 10) or preferred to practice with faculty or current trainees ( n  = 10). Most participants (15/20) felt that not sharing their application materials added to a collegial practice setting.

Fig. 2.

Mock virtual interview participant self-reported number of virtual interview practice sessions versus final ophthalmology residency match status.

Most follow-up survey respondents (15/20) matched into an ophthalmology residency program. There were three participants who did not match, and two participants who did not report their match status. Of the 15 participants who reported matching, 67% matched into a program that was one of their top three choices on their rank order list ( Fig. 3 ).

Fig. 3.

Self-reported final ophthalmology residency match result position on individual submitted rank list for mock virtual interview participants.

A Mann–Whitney U -test showed that there was no difference in how helpful the VI mock exercise was during the actual interview season between matched and unmatched participants. There was a strong positive relationship ( r  = 0.573, p  = 0.026) between how helpful the VI mock exercise was during the actual interview season and the number choice of the ophthalmology residency program that participants matched on their rank list. This relationship suggested participants who matched at programs lower on their rank list (i.e., fifth or sixth choice) were likely to find the VI mock exercise more helpful during the actual interview season than participants that matched at programs higher on their rank list (i.e., first or second choice).

There was no association between how helpful the VI mock exercise was perceived to be during the actual interview season and how many times the participants practiced virtual interviewing. A Mann–Whitney U -test showed that there was no difference between the matched and unmatched participants in the number of times they practiced virtual interviewing after their VI mock exercise. There was also no association between the number of times participants practiced virtual interviewing and the number of choice of the program they matched on their rank lists.

Discussion

This study illustrates the perceived impact of more formalized preparation for VIs on students applying for ophthalmology residency in the 2020 to 2021 match. The data from our study indicate that from a small sample pool of applicants, nationally and internationally, students felt more prepared after a mock VI exercise. Additionally, on completion of our mock VI exercise, participants were more likely to be inclined to practice VIs with someone they already knew.

All specialties under the recommendation of the Association of American Medical Colleges, along with ophthalmology under the AUPO, subscribed to the VI format, this residency application season is to mitigate the risk of COVID-19. Medical students and residency programs across the country adapted to this alternative format, but navigating the nuances of the virtual platform may be challenging, and little is known about how applicants perceived the benefits of mock VIs.

There are many technical variables that are unique to VIs, such as maintaining eye contact with a camera, a relative lack of visual cues and body language, adjusting lighting/background, and optimizing audiovisual and internet technology. ^{5 6} Practicing mock interviews with the intention to develop comfort with these variables may allow students to feel more prepared and confident. Previous studies have indicated that mock interviews in a nonvirtual format that boosts interviewees' confidence and ultimately improves performance on interview day. ⁷

The interview is an important element of the ophthalmology residency application. Previous studies have shown that the majority of program directors in many medical specialties place the greatest weight on the interview when reviewing applicants. ⁷ More specific to ophthalmology, a previous study of ophthalmology residency selection committees, who were surveyed, suggested that interviews were the best predictor of future resident performance. ⁸ However, practice and personal assessment may contribute to improved match outcomes by increasing the odds of superior interview performances. This may be especially true for candidates not considered to be as strong in other areas of their application such as test scores, medical school ranking, or honor society status. In the current study, we showed that applicants who underwent mock VIs felt more prepared for their upcoming actual VIs. Our follow-up survey showed that a majority of these participants who responded were successful in matching.

The virtual format may offer several other benefits in addition to mitigating viral infection risk. VIs offer cost and time advantages, along with the potential to utilize a hybrid approach of screening applicants virtually before an onsite interview. ^{9 10} VIs may also facilitate the very time-sensitive Supplemental Offer and Acceptance Program (SOAP) process with unmatched applicants and unfilled programs. ¹¹ Nonetheless, time will tell if the potential merits of this format are strong enough to replace or supplement the benefits of traditional formats in years to come.

Limitations

This study has several limitations, including the small sample size. The format of the mock VIs in this study was conducted in a manner where no applicant material was necessary to be eligible to participate to increase participation and limit fear of judgement by fellow peers. With the lack of information about a participant's board scores, research, clinical grades, leadership, and activities, the interviewers were unable to simulate potential questions surrounding those topics. This was intentional for this cohort to maintain confidentiality given that interviewees and interviewers were all medical students applying for ophthalmology residencies in the same year. However, there may be benefits to interviewing without other application materials. One study of surgical residency candidates showed that knowledge of the complete application of candidates contributed to interviewer bias. ¹² Another limitation of the current study was the lack of formal training and standardization of the interviewers. Participants overall, who responded to our follow-up survey, found the VI mock exercise helpful for the actual interview setting and continued practicing in the virtual setting. Of those who responded to our follow up survey, with regard to where they matched on their rank list, 67% matched into one of their top three ranked programs. It is noteworthy that there is a positive correlation between matching lower on one's rank list and finding the mock VI exercise helpful; however, the sample size was extremely small and there are multiple potential confounding factors. For example, applicants with less access to practice resources at their home institution may also be less “competitive” in residency applications due to factors such as access to letters of recommendation, prestige of school, and access to scholarly projects.

Conclusion

The COVID-19 pandemic has created incredible uncertainty for medical students applying to residency programs throughout the United States, including the new VI format. Our study demonstrated potential benefits of mock VI preparation for residency applicants. As we gain more information from this year's application cycle, future studies will further clarify how best VIs can be conducted, and how applicants can optimally be prepared for the residency interview.

Appendix.

Preinterview Survey and Postinterview Survey

1. Gender

   • Male

   • Female

   • Nonbinary

   • Prefer not to say

   • Other

2. If you answered “Other”, please type in your gender identity:

3. Please specify your ethnicity

   • White

   • Black or African-American

   • Asian/Pacific Islander

   • Native American or Indian American

   • Hispanic or Latino

   • Other

4. What region in the United States do you attend medical school?

   • Northeast (CT, ME, MA, NH, RI, VT, NJ, NY, PA)

   • Midwest (IL, IN, MI, OH, WY, IA, KS, MN, MO, NE, ND, SD)

   • South (DE, DC, FL, GA, MD, NC, SC, VA, WV, AL, KY, MS, TN, AR, LA, OK, TX)

   • West (AZ, CO, ID, NM, MT, UT, NV, WY, AK, CA, HI, OR, WA)

   • I attend medical school in another country.

5. Have you earned a postgraduate degree? (MSc, MBA, MA, MFA, MEng, Ph.D., etc.)

   • Yes

   • No

6. If you answered yes above, what type of postgraduate degree have you earned?

7. How prepared do you feel for virtual interviews with residency programs?

   • Very unprepared

   • Somewhat unprepared

   • Neutral

   • Somewhat prepared

   • Very prepared

8. How likely are you to practice virtual interviews with someone you know?

   • Very unlikely

   • Somewhat unlikely

   • Neutral/undecided

   • Somewhat likely

   • Very likely

9. How likely are you to practice virtual interviews?

   • Very unlikely

   • Somewhat unlikely

   • Neutral/undecided

   • Somewhat likely

   • Very likely

10. How likely are you to practice virtual interviews in the same room/area as you will during the official interview season?

    • Very unlikely

    • Somewhat unlikely

    • Neutral/undecided

    • Somewhat likely

    • Very likely

Follow-up Survey

1. How much did the VI mock exercise help you during the actual interview season?

   • It helped very little

   • It helped somewhat

   • It helped greatly

2. How many times did you practice in the virtual environment (i.e., With Zoom, Webex, and others) after the VI mock exercise?

   • 0 times

   • 1–5 times

   • 6–10 times

   • >10 times

3. Did you prefer medical students as the mock interviewers during the VI mock exercise?

   • I preferred practicing with faculty or current trainees

   • I did not have a preference

   • I preferred practicing with other applicants

4. Did you feel not sharing your application material (i.e., USMLE scores and research, extracurricular) added to a nonjudgmental and collegial environment for the VI mock exercise?

   • No, I preferred having my application viewed

   • I did not have a preference

   • Yes, I felt not sharing my application materials added to a collegial practice setting

5. Did you match into a residency in ophthalmology?

   • Yes

   • No

6. Which of the following reflects where you matched on your rank list?

   • First choice

   • Second choice

   • Third choice

   • Fourth choice

   • Fifth choice

   • Sixth choice

   • Seventh choice

   • Eighth choice

   • Ninth choice

   • Tenth choice

   • Eleventh choice

   • Twelfth choice