Implication Statement
COVID-19 pandemic restrictions abruptly changed the way interviews for medical school admissions have been conducted. This study is unique as it highlights the first successful virtual synchronous multiple mini interview (MMI) in Canada. Our low technical incident rate, troubleshooting strategies and approach may reassure other medical schools considering conducting a virtual MMI. Success was achieved with collaboration, a strong organizational and communication strategy, learning along the way and a priori contingency plans. Virtual interviewing in academic medicine is likely here to stay, and future work to highlight the impact on applicants will help to build on the diversity mission in undergraduate medicine admissions.
Énoncé des implications de la recherche
Les restrictions liées à la pandémie de la COVID-19 ont brusquement changé la façon de mener les entrevues dans les processus d'admission aux programmes de doctorat en médecine. Notre étude est unique dans la mesure où elle présente la réussite, pour la première fois au Canada, d'une mini-entrevue multiple (MMI) en mode virtuel synchrone. Notre faible taux d'incidents techniques, nos stratégies de dépannage et notre approche peuvent rassurer les facultés de médecine qui envisagent d'effectuer des MEM virtuelles. Ce succès a été obtenu grâce à la collaboration, à une solide stratégie d'organisation et de communication, à l'apprentissage en cours de route et à la préparation de plans d'urgence. L'entrevue virtuelle n'est pas près de disparaître des programmes de doctorat en médecine, et les travaux futurs visant à mettre en évidence son impact sur les candidats contribueront à renforcer la promotion de la diversité dans le processus d'admission.
Introduction
The University of Saskatchewan College of Medicine (U of S COM) uses the multiple mini interview (MMI)1 to assess its applicants. Prior to COVID-19 restrictions,2 virtual interviewing in academic medicine was rare, occurring in select post-graduate programs.3 The purpose of this paper is to highlight the methodology used by U of S COM to conduct a successful synchronous virtual MMI in their medical school admission process, the first school in Canada to do so, with the goal of having other schools learning from its approach.
Innovation
A Plan-Build-Test-Deploy method was used to ensure that the videoconferencing platform, Webex, was adequate to support the planned virtual MMI. Given the short timeline in pivoting to the virtual MMI based on COVID-19 restrictions and concerns, Webex was chosen because it was the only supported platform available for use by the U o f S COM and it was familiar to our IT personnel. Due to time constraints, a pilot interview was not conducted. Pertinent information about the virtual MMI was emailed to applicants and assessors. Interviews were accessed through meeting codes given to applicants prior to the interview and the waiting room function ensured only one applicant was in an interview at any time. A buzzer system was used for timing of the MMI stations. Interview scenarios were presented using placards through the videoconferencing camera during the three-minute reading time. Each interview lasted ten minutes, then applicants entered another meeting code to complete their next interview, and this pattern was repeated for all four interviews.
Utilizing strict COVID-19 safety protocols, assessors (n = 36) and staff (n = 20) were on site while applicants (n = 288) connected virtually. Tactics utilized included: cohorting personnel, private rooms for assessors, reducing the number of stations and building in flex time, having a strong communication strategy with training materials and testing platforms, and contingency plans such as using WhatsApp for connectivity issues. This study was granted an exemption by the University of Saskatchewan Behavioural Research Ethic Board as per Article 2.1 of the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans – TCPS 2 (2018).
Outcomes
Synchronous mini-interviews (n = 1152) were conducted over two days. The technical incident rate was 3.2% (n = 37) with a declining trend indicating ongoing improvements (Table 1). All incidences were resolved through our contingency plans and applicants experiencing a technical incident were not negatively disadvantages after review of their scores.
Table 1.
Technical incident report with resolutions during MMI
| Technical Incident | Saturday am (n = 72) |
Saturday pm (n = 72) |
Sunday am (n = 72) |
Sunday pm (n = 72) |
Resolution |
|---|---|---|---|---|---|
| Virtual Connection Issues (Audio/Visual/Both) | 6 (2.1%) | 5 (1.7%) | 6 (2.1%) | 5 (1.7%) | Transition interview to WhatsApp (15) Disconnect and reconnect (4) Interview moved to end of circuit (3) |
| Timing of Interviews (initiated early or delayed) | 7 (2.4%) | 3 (1.0%) | 0 | 1 (0.3%) | Delaying entire circuit (5) Specific timing adjustments off the main circuit (4) Communicating buzzer procedures with assessor (1) Communicating logging in procedure with assessor (1) |
| Applicant Instruction Issues (Applicants attempting to enter into wrong interview at the wrong time) | 3 (1.0%) | 1 (0.3%) | 0 | 0 | Applicants not admitted into the interview by assessor and communicating with applicants over phone to remind them of their passcodes and the order of their stations. Sent additional communication to remaining applicants to make them aware of this issue. |
Overall, 95.7% (n = 66) of surveyed assessors (95.8% response rate) strongly agreed or agreed that they were provided enough information to adequately assess the applicants and 81.5% (n = 208) of surveyed applicants (88.5% response rate) strongly agreed or agreed that the virtual method allowed them to adequately communicate with the assessor.
The G coefficient (reliability of applicant scores) of the virtual MMI was acceptable at 0.61 and comparable to previous MMI administrations (2019 = 0.68, 2018 = 0.76, 2017 = 0.68).
Compared to our traditional MMI, the virtual format was approximately one third the cost to the U of S COM and also saved applicants the cost of travel and accommodations.
Next steps
Virtual interviewing has many advantages, including increased convenience, flexibility and decreasing the need for travel, which may lead to improved access to a broader and more diverse applicant pool while decreasing the costs for all.4,5 Although we used Webex, other similar software applications would likely also be successful. Improving access and decreasing costs are key when addressing equity and enhancing diversity in admissions.6 Further study is needed to assess the impact of virtual interviewing on student selection and its potential role in decreasing the barriers of medical school admissions.
Acknowledgments
The authors wish to thank all the staff and assessors that made the University of Saskatchewan College of Medicine 2020 MMI a success.
Conflicts of Interest
All authors have no conflicts of interest to declare.
Funding
None.
References
- 1.Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: the multiple mini-interview. Med Educ. 2004;38(3):314-26. 10.1046/j.1365-2923.2004.01776.x [DOI] [PubMed] [Google Scholar]
- 2.Government of Saskatchewan. COVID-19: Saskatchewan declares state of emergency, imposes additional measures to protect Saskatchewan residents. https://www.saskatchewan.ca/government/news-and-media/2020/march/18/covid-19-state-of-emergency. Published March 18, 2020. [Accessed February 28, 2021].
- 3.Chandler NM, Litz CN, Chang HL, Danielson PD. Efficacy of videoconference interviews in the pediatric surgery match. J Surg Educ. 2019;76(2):420–426. 10.1016/j.jsurg.2018.08.010 [DOI] [PubMed] [Google Scholar]
- 4.Hariton E, Bortoletto P, Ayogu, N. Residency interviews in the 21st Century. J Grad Med Educ. 2006;8(3):322–324. 10.4300/JGME-D-15-00501.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Daram SR, Wu R, Tang S. Interview From Anywhere: Feasibility and utility of web-based videoconference interviews in the gastroenterology fellowship selection process. Am J Gastroenterol. 2014;109(2):155–9. 10.1038/ajg.2013.278 [DOI] [PubMed] [Google Scholar]
- 6.Talamantes E, Henderson MC, Fancher TL, Mullan F. Closing the gap-making medical school admissions more equitable. N Engl J Med. 2019;380(9):803-5. 10.1056/NEJMp1808582 [DOI] [PubMed] [Google Scholar]