Virtual Reality as an Interview Technique in Evaluation of Emergency Medicine Applicants

Abstract

Need for Innovation

Current interviewing strategies and the standardized letter of evaluation may not provide enough insight into preferred resident characteristics. Emergency medicine (EM) residency programs are challenged with identifying trainees who can problem solve, communicate, and work well with fellow health professionals.

Background

Structured interviews have previously been used and can help predict success but candidates have reported a negative impression with their use.

Objective of Innovation

This structured virtual reality (VR) interviewing method was designed so that interviewers can observe the communication abilities, subtle personality traits, and teamwork skills of applicants interviewed at an EM residency program.

Development Process

A consumer VR headset became available and in combination with an interactive team game was incorporated into a standardized team‐based interview session. This session was designed to allow observation of candidates’ communication, problem solving, and teamwork skills.

Implementation Phase

Surveys were collected to examine the satisfaction of EM residency applicants who participated in this novel standardized interviewing method using a VR headset. After the submission of rank lists, but prior to Match Day, those who interviewed were e‐mailed a voluntary, anonymous, and confidential survey asking about their interview experience, specifically about the VR portion. The survey was sent to 102 applicants with 63 responses for a 62% response rate at the completion of the 2015 to 2016 interview season.

Outcomes

Overall study findings suggested that participants had a highly favorable impression of the VR portion of the interview. Specifically, participants reported that this interview technique was appropriate and worthwhile. Additionally, participants attested that the Oculus portion of the interview gave insight to their work ethic, personality, and communication skills and how they work with others.

Reflective Discussion

The novel interviewing method used in this study allowed interviewers to gain insight beyond that of the paperwork and brief face‐to‐face interaction. Study findings suggest that interviewees accepted the use of this novel interview method. It has been incorporated into our interview process for three consecutive years.

Need for Innovation

Emergency medicine (EM) physicians need to possess superior interpersonal skills especially in times of stress to effectively work with patients, families, staff, and physicians of all backgrounds. Consequently, EM residency programs are challenged with identifying potential trainees who are adaptable and hardworking and can problem solve and communicate well, based solely on residency interviews.1 Unfortunately, current interviewing strategies, including the standardized letter of evaluation (SLoE) may not give insight into these characteristics.

Background

According to the National Resident Matching Program 2016 Program Director Survey, 95% of programs look at an applicant's interpersonal skills and interactions with the faculty in the ranking process.2 Ultimately, the goal of the interview day for both programs and potential residents is to demonstrate positive attributes attractive to both parties and identify a good fit between each.3

The standardized multiple mini‐interview (MMI) method allows for an objective comparison of applicants’ responses. The MMI method is designed to assess noncognitive skills in a structured format.4 This technique is poorly viewed by candidates because it feels impersonal, possibly resulting in a lower candidate ranking of the program.1, 4, 5 Such structured activities can predict success in future residents; however, MMIs do not seem to correlate with communication abilities, professionalism, or disciplinary action during residency.6, 7

New interview methods are being implemented to evaluate interpersonal aspects of the applicant and to improve applicant comparability during interviews. One such example is the standardized video interview (SVI) that was piloted in 2016. Currently, the American Association of Medical Colleges (AAMC) in combination with the Council of Emergency Medicine Residency Directors (CORD), Society of Academic Emergency Medicine (SAEM), Emergency Medicine Residency Association (EMRA), and the American Academy of Emergency Medicine (AAEM) have been working to develop the SVI program.8, 9 The SVI is designed as a unidirectional video interview where students respond to each of six sequential open‐ended questions while seated in front of a computer web camera. They have 30 seconds to think about the question and up to 3 minutes to respond to each. The recorded responses are then assigned a grade by a group of reviewers from the independent company HireVue. The AAMC designed the system to help program directors assess “interpersonal and communications skills and knowledge of professional behaviors (professionalism).”8, 9 In 2018, the AAMC is encouraging the use of SVIs for EM residency applicants and may mark applications as incomplete without it.10 The validity and acceptance of the SVI is still to be determined.8 The study sought to develop an alternative method for assessing candidates during an interview. Since programs are not convinced of the overall utility of SVI, a program‐specific evaluation that allows for objective observation of characteristics may be of more benefit on a smaller scale.

Objective of Innovation

The objective of this innovation was to develop a standardized technique to screen and evaluate EM resident candidates’ communication, leadership, and teamwork skills. This innovation needed to still be well received and still positively impact the perception of the program.

Development Process

A novel interview technique using a VR headset was designed as an interview method during the 2015 to 2016 application cycle for EM residency. An off‐the‐shelf game that required careful verbal communication skills to describe visual findings, logical reasoning, and leadership abilities was chosen by program faculty. Other VR games were evaluated but did not meet the expected teamwork and communication skills desired to evaluate the candidates or required background knowledge to be functional.

After the completion of the session, individuals were asked standardized questions to allow objective evaluation of each applicants’ understanding about the role of the EM physician in the delivery of health care, as a corollary to the preceding task. Each individual was also asked to describe his or her role in the team situation. The session was designed to fit a finite time frame with a pre‐recorded video introduction and format that was one part of a larger interview session.

To the best of our knowledge there is no prior research on the use of virtual reality (VR) in the evaluation of residency applicants. There are observational studies on the user experience of VR games with team interaction requirements. It has been found that verbal communication is an important factor for acceptability by users.11 VR is being studied as a method for improving communication skills and stimulus response in certain settings.12, 13, 14

Implementation Phase

All applicants were screened with a standard approach: scoring and evaluating each applicant by reviewing SLoEs, standardized examination scores, personal statements, medical school transcripts, and life experiences. A total of 186 invitations to interview were offered and 102 applicants came to interview.

All interviewees participated in the program's standard interview day, including an introduction to the department, tour of the campus and facilities, and four interviews (three faculty, one chief resident). In addition, all interviewees were assigned into groups of two or three (depending on the number of interviewees) and participated in the novel interview session using a Samsung VR headset powered by Oculus (Gear VR Innovator Edition) and an accompanying Samsung Galaxy S6 phone.

For the novel interview session, applicants were first shown a 5‐minute orientation video of the VR headset and the commercially available game “Keep Talking and Nobody Explodes” that relies on team communication to disarm a virtual bomb. The introduction video described the goals of the game and how to use the physical game manual to solve logic puzzles. The teams were allowed 5 minutes to review the manual and decide how to proceed. The group self‐selected who would wear the headset first and who would use the manual to guide their partner(s). No interviewer guidance was given.

The interviewee wearing the headset could see only inside the three‐dimensional environment and had to describe the components of the bomb's box. Their partner used the paper manual to decipher logic puzzles to guide their partner on how to disarm the virtual bomb. After the first bomb defusing attempt the team members traded places. After all team members had tried defusing the bomb, each was individually interviewed about his or her experience using a standard script. The whole group then returned to the room for debriefing and sharing, including how this activity corresponded to activities performed by an ED physician. The debrief questions were designed to reveal an understanding of the diverse role of the EM physician as both a team leader and a supporting team member in the overall delivery of health care.

During the VR activity, we assessed the applicants’ leadership style, teamwork, and communications skills. Two interviewers—one faculty and one chief resident—could see the screen of the VR headset on a Macbook laptop in the room by using a screen mirroring program on the phone (MirrorOp). The interviewers were blinded to the applicants’ file, to prevent bias. The designer of the activity was also the faculty member conducting the Oculus interview session and was present for all sessions during the interview season to limit concerns related to reliability of the interview data.

The selection committee saw benefit from the VR session, but wanted feedback from the interviewees. A survey was created to evaluate their perceptions. After the submission of rank lists, but prior to Match Day, those who interviewed were e‐mailed a voluntary, anonymous, and confidential survey asking about their interview experience and specifically about the VR portion. Institutional Review Board approval was obtained prior to any data collection. This time period was selected to minimize bias in responses or fear of participation in the survey affecting program scoring or ranking. The e‐mail account used to invite the candidate for interview was used for survey distribution.

Survey questions addressed applicant's satisfaction with the preinterview arrangements, the interview day itself, and the VR portion of the interview. The survey consisted of 25 questions of which eight related to the VR experience. Likert responses to each question ranged from strongly disagree to strongly agree, with a not applicable option (see Table 1 for sample survey questions). Questions regarding their interview session and appropriateness of virtual interviews were also asked.

Table 1.

Sample Survey Questions

Question Prompt	Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree	N/A
The interview team treated me with professionalism and respect	1	2	3	4	5	6
I was given a fair opportunity to present my qualifications	1	2	3	4	5	6
Overall, I was satisfied with my TTUHSC interview experience	1	2	3	4	5	6
The Oculus VR portion was appropriate for an interview	1	2	3	4	5	6
The Oculus VR portion was worthwhile	1	2	3	4	5	6
The Oculus VR portion gave insight to my work ethic	1	2	3	4	5	6
The Oculus VR portion gave insight to my personality	1	2	3	4	5	6
The Oculus VR portion gave insight to my communication skills	1	2	3	4	5	6
The Oculus VR portion gave insight to how I work with others	1	2	3	4	5	6
The Oculus VR portion of the interview positively affected my ranking of this program	1	2	3	4	5	6
Overall, I was satisfied with the Oculus VR portion of the interview	1	2	3	4	5	6

Outcomes

A total of 102 applicants conducted an in‐person interview, of whom 63 completed the voluntary survey online. However, several study participants did not complete the entire survey, resulting in varying response numbers per question (n = 26–63). Table 2 provides responses to the questions related to the VR portion of the Interview.

Table 2.

Survey Responses

	Candidate Response
Question Prompt	Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree	N/A
This portion was appropriate for an interview (n = 32)	1	0	0	0	31	0
This portion was worthwhile (n = 39)	1	0	0	0	38	0
This portion gave insight to my work ethic (n = 27)	1	0	0	0	25	1
This portion gave insight to my personality (n = 27)	1	0	0	0	25	1
This portion gave insight to my communication skills (n = 39)	2	0	0	0	37	0
This portion gave insight to how I work with others (n = 37)	2	0	0	0	35	0
This portion of the interview positively affected my ranking of this program (n = 23)	1	0	0	0	20	2
Overall, I was satisfied with this portion of the interview (n = 38)	1	0	0	0	37	0

n = the number of respondents to each question.

Overall, participants strongly agreed that they were treated with professionalism and respect (98%), were given a fair opportunity to present their qualifications (98%), and were overall satisfied with their interview experience (96%). For the VR portion of the interview, participants reported an overall positive impression, with 97% stating that it was appropriate for an interview and worthwhile. The participants strongly agreed that the VR portion gave insight to their work ethic (96%), personality (96%), and communication skills (95%) and how they work with others (95%). Additionally, 95% stated that the VR portion positively affected their ranking of our program. Moreover, 81% told other people about their interview experience with our program. Separately, 84% of the participants believed that a virtual interview would assist with the financial and time challenges of interview season, but only 46% felt that a virtual (e.g., Skype) interview alone would be appropriate for future residents.

Reflective Discussion

Although the idea of using a standardized interview is not novel,5, 15, 16 to our knowledge, this is the first use of a VR headset for this purpose, certainly in the field of EM. The combination of technology and gaming revealed underlying personality traits, communication, and teamwork skills of many individuals’ far more than standard interviews. Interviewees were noted to remove suit coats, loosen ties, and slump or slouch in their chairs. One activity required the description of strange keys and characters, often leading to vulgar language, giving unique insight into applicant personality. Verbal arguments and obvious frustration occurred when working with less‐than‐proficient team members. These types of observations helped to paint a larger personality picture of the candidate. Such unique occurrences were noted on the interview form and were taken into consideration in the global evaluation. Some of these observations, although out of place in a standard interview session, were contextually appropriate and gave a more representative sample of the personality traits of the candidate.

No specific weighting of this activity determined rank list position for the applicants, but was used often to confirm or describe what was alluded to in other portions of the application or interview sessions. The communication and interaction with a relative stranger was the primary interest of the exercise. All observers found the additional insight assisted with confirming evaluations.

From the interviewees’ perspective, the study findings suggest that the preinterview interactions and interview day were well received. Additionally, the VR portion of the interview was thought to be appropriate and worthwhile. Moreover, they agreed that it was effective in assessing their work ethic, personality, communication, and teamwork skills. With most applicants having numerous interviews, it is imperative that programs begin to introduce novel interview methods to break the often‐mundane process. This novel interview method is still in use and has been enhanced by improved technology.

The study's primary limitation was sample size, with 62% of the total interviewees participating in the survey and several not entirely completing the survey. In addition, this was a cross‐sectional study, thus strictly descriptive in nature. However, it does offer insight into how participants perceived their VR portion of the interview and how they may respond to the mandated SVI in 2018. The survey was distributed after the interview season was over to allow objective responses but could lead to recall bias due to the temporal delay in survey delivery. Additionally, there is a cost and logistic barrier to the addition of a VR system that may prevent its widespread adoption by some programs. While this interview year utilized a consistent faculty observer to minimize bias, this may not always be possible and training for how to observe and assess candidates may require local and individualized training.

In summary, standard interviews are currently used to assess each interviewee's attributes. However, it may be useful to introduce more unconventional interview methods to challenge candidates and potentially gather information on often unseen characteristics. The VR session allowed interviewers to gain insight past that of the paperwork and brief face‐to‐face interactions. Study findings suggest that interviewees approve of the use of novel interview methods such as the VR interview session. Our selection committee appreciates the information about communication abilities, teamwork skills, and personality traits provided by this activity as one part of the interview process. It has now been incorporated into our interview process for 3 consecutive years.

AEM Education and Training 2018;2:328–333