Critical Elements in the Experience of Virtual Reality Job Interview Training for Unemployed Individuals with Serious Mental Illness: Implications for IPS Supported Employment

Abstract

Objectives:

Individuals with serious mental illness (SMI) who re-enter the labor market after extended unemployment may benefit from exposure to job interview training. This study explored the processes and perceived benefits of preparing for interviews using Virtual Reality Job Interviewing Training (VR-JIT) among employment specialists and clients within the Individual Placement and Support (IPS) model of supported employment.

Methods:

This study analyzed secondary qualitative data from a randomized controlled trial (RCT) of VR-JIT including qualitative focus groups comprised of IPS employment specialists (n=11) and IPS clients (n=13), semi-structured interviews with IPS employment specialists (n=3), and semi-structured interviews with IPS clients (n=3). Additionally, semi-structured interviews with IPS employment specialists (n=8) who naturalistically implemented VR-JIT at four community mental health agencies independent of the RCT. All focus group and interview data were analyzed using grounded theory methodology.

Results:

Three main processes were viewed by employment specialists (and their clients) as beneficial for individuals with SMI receiving IPS with VR-JIT: 1) exposure to a simulated interview in a safe environment; 2) practicing and receiving job interviewing feedback; and 3) improved confidence and motivation in job seeking.

Discussion:

VR-JIT improved IPS participants’ confidence in job seeking through proposed mechanisms of exposure to a simulated job interview and repetition and practice of job interview skills. These critical elements indicate that VR-JIT has the potential to improve IPS client engagement particularly with those who have had prolonged periods of unemployment.

Individuals with serious mental illness (SMI; e.g., schizophrenia-spectrum, bipolar disorders) who receive treatment within the public mental health system consistently have unemployment rates as high as 80% to 90% (Drake, Bond, & Becker, 2012; SAMHSA, 2012). Living with a psychiatric disability that is accompanied by long periods of unemployment leads to marginalization from the competitive labor market, as well as presents additional barriers to employment such as a loss of job-related social skills (Furman, et al., 1979; Corbiere, et al., 2011).

In an effort to address the historical underemployment of individuals with SMI, the Individual Placement and Support (IPS) model of supported employment has emerged as the field’s primary evidence-based practice to help individuals with SMI obtain competitive employment (Drake, Bond, & Becker, 2012). Specifically, IPS focuses on rapid job placement through job development and providing ongoing supports, is based on client interest and not work history, and provides benefits counseling (Drake, Bond, & Becker, 2012) and is more effective at improving competitive work outcomes than other vocational rehabilitation programs (Frederick & VanderWheele, 2019). Although IPS is the field’s strongest model of supported employment, approximately 45% of persons who participate in IPS do not obtain employment outcomes (Bond, et al., 2020). Thus, IPS clients who are unable to obtain employment may benefit from adjunct services integrated within the context of IPS.

Recent studies have emerged demonstrating that clients who stay in IPS longer without achieving employment may benefit from adjunct services (McGurk, et al., 2015). One of these adjunctive services is Virtual Reality Job Interview Training (VR-JIT; Smith, et al., 2014). Specifically, VR-JIT is an internet-based job interview simulator that facilitates repeated practice of job interviews with a virtual hiring manager from which trainees receive four levels of automated feedback. Prior evaluations have demonstrated VR-JIT’s efficacy in lab-based studies with adults with SMI (e.g., Smith, Fleming, Wright, Jordan et al., 2015; Smith, Fleming, Wright, Roberts et al., 2015), and effectiveness at improving competitive employment rates overall, shorter time-to-employment, job interviewing skills, interview anxiety, and interview confidence among clients minimally engaging in IPS (Smith, Smith et al., 2022). However, research has not yet evaluated the experiences of IPS employment specialists and clients who use VR-JIT in order to more fully understand its possible benefits and how it works. This qualitative study explored the processes and perceived benefits of using VR-JIT by IPS employment specialists who implemented the tool as well as their clients when that data was available.

Methods

Research Settings & Participants

Qualitative data representing IPS employment specialists who implemented VR-JIT with their clients was collected across two settings, while additional IPS client data was available from one setting. The first setting was a recent randomized controlled trial (RCT) evaluating the effects of adding VR-JIT to an IPS program at a large community mental health agency. Meanwhile, the second type of setting included four community mental health agencies who were using VR-JIT naturalistically (See Figure 1). We used methods consistent with grounded theory to analyze both sets of data to explore the underlying experiences of using VR-JIT. (Glaser & Strauss, 1967; Starks & Brown Trinidad, 2007). All research procedures were reviewed and approved by the institutional review boards at the University of Michigan and Northwestern University Feinberg School of Medicine.

Figure 1.

Qualitative Data Collection

Randomized Controlled Trial.

We conducted focus groups and semi-structured interviews with IPS employment specialists and adult IPS recipients who participated in an RCT of VR-JIT (Smith, Smith et al., 2022). This RCT had 54 adult IPS participants in IPS+VR-JIT and 36 participants in IPS as usual (N=90). In addition, 25 of 36 eligible IPS employment specialists participated in focus groups. Participants included employment specialists who were trained to implement high fidelity IPS supported employment and their clients who had completed at least one session with VR-JIT. Inclusion criteria for employment specialists were: 1) received formal training (from our research team) to implement VR-JIT; and 2) providing IPS supported employment services to individuals with SMI.

Inclusion criteria for IPS clients in the VR-JIT RCT were: 1) active engagement in IPS at the time of enrollment (i.e., at least one contact with employment specialist in past 30 days; 2) > 18 years of age; 3) a diagnosis of schizophrenia, schizoaffective disorder, major depressive disorder (any type), or bipolar disorder (types I & II; with or without psychotic features); 4) English fluency; 5) at least a fourth grade reading level [via the Wide Range Achievement Test-V] (Wilkinson & Robertson, 2017); 6) currently unemployed/underemployed and planning to interview for a job within the next four weeks; 7) willing to provide a collateral contact (friend or family member to help prevent loss-to-follow-up); and 8) willing to be video-recorded. Exclusion criteria were: 1) no verifiable contact information; 2) documented other disability or medical illness that could significantly compromise cognition interfering with using VR-JIT (e.g., traumatic brain injury); 3) documented uncorrected vision or hearing problems that prevent VR-JIT use; and 4) meeting criteria for suicidality over the past 30 days (based on the Columbia-Suicide Severity Rating Scale) (Posner, et al., 2011). Notably, 39% of the participants randomized to the IPS+VR-JIT had worked competitively in the past two years.

Community Mental Health Agency Sites.

We conducted semi-structured interviews with IPS employment specialists at four community mental health agencies in Illinois who naturalistically piloted VR-JIT with their caseload after a brief introduction to the technology by the state IPS lead fidelity reviewer. A templated recruitment email was shared with all Illinois IPS providers to request participation in this study among those employment specialists whose agencies had begun to pilot VR-JIT implementation (no agencies were delivering VR-JIT beyond a pilot).

Intervention

VR-JIT is an internet-based job interview simulator (commercially licensed by SIMmersion LLC, www.simmersion.com) that facilitates practicing job interviews with a virtual hiring manager, Molly Porter. Trainees who practice simulated job interviews with Molly receive four levels of automated feedback during their practice sessions. When Molly asks a question, trainees review 10–15 response options and then select their choice by speaking aloud to trigger Molly’s next question. Concurrently, trainees receive automated feedback in real-time and via transcript review, a review of their performance, and an overall score from 0 to 100 (higher scores reflecting stronger performance) at the end of each interview (Smith, et al., 2014). VR-JIT includes a job application for eight positions that informs the virtual interview, and an e-Learning curriculum provides interview preparation advice. Based on review and feedback by community stakeholders and the job interview literature (Bell & Weinstein, 2011; Huffcutt, 2011), VR-JIT highlights eight interview skills designed to convey an applicant’s positive attributes such as conveying oneself as a hard worker, being easy to work with, behaving professionally, possessing negotiation skills, sharing things positively, sounding honest, sounding interested in the job, and appearing comfortable during the interview. During the aforementioned RCT, participants completed approximately one to two sessions per week over 6–8 weeks (Smith, Smith et al., 2022). Notably, employment specialists discussed the virtual interview training experiences with their clients (~30 minutes) to help contextualize their learning. This discussion was not prompted and reflected a natural integration of VR-JIT in IPS.

Data Collection

Randomized Controlled Trial Site.

Five focus groups were hosted during the RCT to ask IPS employment specialists (two focus groups; n=11) and IPS clients (three focus groups; n=13) about their VR-JIT experiences. Focus groups asked questions about using VR-JIT, including likes and dislikes, and employment specialists’ feasibility of using it with clients. Following focus group data analysis, the research team created more in-depth semi-structured interview scripts, and conducted interviews with three IPS employment specialists and three IPS clients. The IPS clients were recruited using convenience sampling (i.e., via inquiries among clients who used VR-JIT and still accessible to the IPS program).

Community Mental Health Agency Sites.

Following the recruitment email sent to agencies, interested employment specialists contacted the research team and provided informed consent before completing a semi-structured interview over web-based teleconferencing. Eight IPS employment specialists from these sites participated. The interviews consisted of questions about their exposure to VR-JIT, use, likes and dislikes about the tool, and client progress.

Participant Characteristics.

Table 1 displays the demographics of both IPS employment specialists and IPS clients. IPS clients were older (M=49.6, SD=13.1) than employment specialists (M=34.3, SD=9.21), while more employment specialists were white (54%) than IPS clients (38%). Sixty-eight percent of IPS employment specialists had a Bachelor’s degree, 21% had a Master’s degree, and 11% had completed some college. Notably, 77% of the IPS clients who participated in the focus groups were unemployed for more than two years at the time of the study.

Table 1.

Participant Demographics

		Participant Type
		Staff participants	Client participants	Total
Age	Mean (SD)	34.3 (9.21)	49.6 (13.1)
Race	White	12	6	17
	Black	6	8	14
	Latinx	2	0	2
	Asian	2	0	2
	More than one race	0	2	2
Total		22	16	38

Data Analysis

Data analysis relied on methods consistent with grounded theory where data were first analyzed inductively using a constant comparative approach, and later theoretical coding was used to gain a deeper understanding of themes (Corbin, & Strauss, 2008). A team comprised of a postdoctoral research fellow with training in grounded theory research and two research coordinators trained in qualitative coding analyzed all data which were transcribed using Zoom and/or by a research assistant and then uploaded into Dedoose software for qualitative analysis (Dedoose, 2021). First, all individual and focus group transcripts were reviewed for clarity. Then, the team used an open coding technique to identify codes within the scope of experiences with VR-JIT within IPS. After the first round of open coding was completed, the three coding team members met to discuss, and then organize codes into theme areas using selective coding to create content areas, which were evaluated for saliency of topic and frequency of their appearance. Discrepancies were discussed within the team and reviewed with the Principal Investigator when needed. This round of inductive coding and analysis led to the diagramming of an initial model of the processes involved in using VR-JIT for both IPS clients and employment specialists. Following the construction of the initial diagram, theoretical coding (Corbin, & Strauss, 2008) was utilized to apply relevant existing theoretical research on virtual and other skills-training interventions to the original themes such as the known evidence for exposure/in vivo/virtual treatments (Opris, et al., 2012). Reading the theories behind exposure treatments improved our understanding of the elements of simulation and practice within our themes. Finally, we applied theoretical integration between the existing theory of exposure treatments and our inductive themes resulted in a final proposed theoretical model shown in Figure 2 (Corbin, & Strauss, 2008).

Figure 2.

Theoretical Model

Results

Qualitative data across all sources uncovered three processes associated with completing VR-JIT sessions which ultimately improved confidence in job searching. These themes include: 1) exposure to a simulated job interview in a safe environment; 2) practicing and receiving job interviewing feedback; and 3) improved confidence and motivation in job seeking.

Exposure to a Simulated Job Interview in a Safe Environment

When interviewed about their experiences with VR-JIT, both IPS clients and employment specialists felt that the exposure to a simulated job interview held in a safe, low-risk environment provided a comfortable way to learn about job interviewing.

“It was a really, really, good opportunity and a really good experience of what job interviews are really about. I learned something from it.” – IPS Client 4

“I think with Molly, it felt like a job interview. It didn’t [only] feel like coming to see my job person. [That’s] just coming and trying to find out information how to go to an interview. But with Molly, it seems like you’re actually in an interview.” – IPS Client 1

In this case, the IPS client above likened the VR-JIT experience with interviewer Molly Porter to be more like an actual job interview when compared to talking about job interviewing with their employment specialist. IPS employment specialists also described the exposure and simulation effects of practicing VR-JIT interviews:

“What I liked the most, is something I talked about earlier, it just provides a very safe situation for someone to answer the more fundamental, high frequency questions so they can have some comfort in what they’re doing.” – IPS Employment Specialist 4

“It’s good to have a comforting space to get your first exposure to what an interview might be like and practicing some of the more difficult questions, having a shot at it with nothing on the line.” – IPS Employment Specialist 4

These quotes emphasize how working with the simulated interviewer (Molly Porter) exposed IPS clients to what a real interview was like within a safe space. This simulation of a job interview may be a critical component of VR-JIT for individuals with SMI after long periods of unemployment.

Practicing and Receiving Job Interviewing Feedback

Next, IPS Employment Specialists emphasized how the process of repeatedly practicing how to answer sample questions during the simulated job interview was beneficial for improving IPS participants’ job interview skills and confidence.

“I do like that it does provide those appropriate responses so that people like him that don’t know those responses can, you know, hopefully kind of learn them from doing Molly sessions over and over again.” – IPS Employment Specialist 5

In this case, the ES felt that the client’s lack of recent work history and associated loss of job interview knowledge was addressed through the activities of practicing job interviews and receiving feedback from VR-JIT. The following two excerpts support this with further examples of gleaning job interview knowledge through practicing with VR-JIT.

“…it did help with me being able to kind of reiterate things that I was already telling my clients, [that] are good in interviews, are appropriate in interviews, are things you want to share in interviews. It kind of helped to reiterate those good things, and, also, frown on the bad things.” – IPS Employment Specialist 6

“A lot of the questions she asked were really helpful. You know, how to work with a team, if you see somebody doing something wrong would you report it to your boss or you know, how you felt as far as your confidence in getting the job. It was interesting, you know, I never really have been on an actual interview like that because, like when I went to [work at last job] I got the job and I was doing the Molly interview at the same time and it made me more confident. I’ll be really honest I wasn’t able to keep the job due to my disability but when I would go to work I felt confident as far as working with people - it really captured how you present yourself when you are going into an interview with a person and you want to get a job and some of the questions she asked, a lot of the questions she asked, I scored pretty good on a lot of those interviews.” – IPS Client 7

These excerpts illustrate how employment specialists and IPS clients perceived the practice and repetition of sample interview questions and responses to help teach relevant skills that may have been lost as a result of long periods of unemployment. Employment specialists described how they integrated clients’ learning into their conversations:

“So the multiple choice questions were definitely something that I found interesting. It gave the clients…options to pick through what they were going to say or not going to say and so it helped me in a way where it was like I was able to kind of box those options in my head, you know. And when I would talk about these sessions later on with the clients, it was like, well why didn’t you choose this option and then talked through why it would be a negative response. And it would help me [to] understand what different responses were possible that these clients would come up with.” – IPS Employment Specialist 3

In this way, employment specialists who helped to facilitate VR-JIT for IPS clients were also able to draw on the practice questions as a teaching aid in their job skill training work.

Improved Confidence and Motivation in Job-Seeking

Finally, employment specialists thought that practicing job interviewing skills in VR-JIT increased clients’ confidence in job seeking, as illustrated by the following quotes:

“When you have a program like Molly coupled with IPS, it brings something out of them, like, a different level of confidence. And that’s what I encountered. Even their meetings with me became more productive…” – IPS Employment Specialist 2

“…a big part of what it was for me was it pushes me to understand, you know, I really have to work harder and I really can get a job, you know, it’s really possible.”

– IPS Client 3

“My experience would be just watching them, from the time that they were referred to me and then I started working with them, you know, orienting them to IPS, but then I noticed once the members who actually refer to Molly and actually participated in the Molly study, it was like I seen a difference in their communication style. Their level of not being as shy or that awkwardness. Molly really helped them to again break out of their shell; to be able to kind of keep those mental health symptoms at bay, you know, whether again if it was the social anxiety or just being nervous; to communicate with others. It just … I [saw] a difference in their level of confidence.”– IPS Employment Specialist 6

“So it’s like wake up, turn the lightbulb on, get on your Ps and Qs cause this lady [Molly] is trying to help, you know, you’re trying to get a job and this lady, she’s not playing with you. She’s asking you questions that are [essential] to going back to work. I haven’t worked in 21 years. And so this was a really big step for me. And it helped me get my confidence back again and stuff.” – IPS Client 5

“It made our meetings more productive because the members were able to, or, I want to say, feel more comfortable with communicating with me as far as their job goals were concerned, helping them to be even more motivated to want to find work, and then it’s like they became excited because they wanted to utilize the tools that they gained from Molly. Like, I’ve been practicing so I’m ready, and I’m like, okay, well, let’s go out and job search, let’s go get this job.” – IPS Employment Specialist 2

This last quote indicates that an increase in motivation for job seeking may have been associated with this improved confidence.

Discussion

IPS employment specialists from several community mental health agencies implementing VR-JIT for individuals with SMI found multiple benefits from using the tool with clients. First, exposure to a practice job interview in a virtual, yet realistic, environment without the risk of failing in a real job interview was seen as beneficial for clients. Second, using VR-JIT allowed IPS clients to repeatedly practice interviews and use effective job interview language across eight targeted job interview skills while receiving relevant automated feedback. Third, these elements of practicing job interviews with VR-JIT seemed to improve the confidence and motivation of IPS clients with job seeking and thus, improved their overall engagement with job seeking activities with program employment specialists.

Improving Connection to the Job Seeking Process and Engagement with IPS for Unemployed Individuals with SMI

The IPS model focuses on providing direct assistance to individuals with SMI in finding jobs in competitive workplaces as well as supporting them in these jobs (Drake, et al., 2012). However, studies have shown that groups who may be affected by long-periods of unemployment or disability may fail to connect with this job seeking process while receiving mental health services for a variety of reasons. VR-JIT may be advantageous for these individuals by providing basic exposure to job interviewing and a chance to practice job interviewing skills (Smith, Smith et al., 2022). Other benefits of VR-JIT reported in this study included improved confidence and motivation in job-seeking among IPS participants.

Virtual job interview simulation as exposure

Both IPS employment specialists and IPS clients described an increase in comfort with job interviewing attributed to practicing with the virtual hiring manager in VR-JIT in a low-risk setting. This may have parallel benefits to those in exposure therapy, a well- established treatment for anxiety disorders that facilitates exposure to feared stimuli during in vivo and virtual environments (Garcia-Palacios, et al., 2001). Specifically, prior research suggests virtual exposure to feared stimuli results in a high level of success to overcome the phobia (Opris, et al., 2012; Smith, Smith et al., 2022). While exposure therapy encompasses gradual and systematic exposure to a phobia trigger, virtual reality exposure therapy enhances this experience by allowing the person more control over the trigger situation while “being there.” (Opris, et al., 2012). We could see this exposure via VR-JIT where IPS clients increased their gradual exposure to interviewing (with increasing difficulty) and potentially elevating their anxiety in a low-risk setting. Moreover, participants won’t face some of the negative effects of a real job interview such as difficulty with transportation to get to the interview or not getting the position. Along these lines, multiple RCTs evaluating the effectiveness of virtual in real-world settings demonstrated that participants using VR-JIT had reduced job interview anxiety and improved confidence and motivation following VR-JIT sessions compared to services as usual (Smith, Parham et al., 2022). These positive reports of confidence and motivation, and reduced anxiety may be attributed in part to the actual experiential exposure from participating in the VR-JIT sessions.

Behavioral rehearsal of job interview responses improved job-related social skills and job interview self-confidence

Behavioral rehearsal is a core teaching component in interventions targeting improved interpersonal skills, such as social skills training (Bellack, et al., 2004) and coping skills enhancement (Tarrier, et al., 1993). These models are incorporated into evidence-based programs for SMI such as Illness Management & Recovery, which teaches individuals a variety of skills and content related to managing their overall wellness (Gingerich & Mueser, 2005). An early study found that teaching and practicing job interview skills with hospitalized psychiatric patients improved the social competence often lacking during job interviews, particularly when individuals have not been in the workforce for some time (Charisiou, et al., 1989). These improvements in job-related social skills and job interview self-efficacy were attributed to practicing job interviews. Following this early research, a previous study in a lab setting found that the number of virtual job interviews practiced within VR-JIT led to improved post-test interview skills, which contributed to a higher likelihood of receiving job offers (Smith, et al., 2017). Thus, the results of this qualitative study provide a new depth of understanding with regard to why practicing job interview questions and answers may help facilitate employment outcomes and a stronger sense of confidence in job searching.

Limitations

The study results should be considered within the context of its limitations. First, IPS clients who completed focus groups and interviews represent those who volunteered or agreed to participate and are not representative of all participants who used VR-JIT. Second, IPS clients who used VR-JIT from the four community (non-RCT) IPS sites were not recruited, so their perspectives are missing from these findings. Third, while a more balanced sample of IPS employment specialists whose clients used VR-JIT were included in the focus groups, their contributions were only observational on behalf of their clients and must be interpreted as such. Finally, this is a qualitative and experiential study of IPS clients and not a reflection of the fidelity and rigor of IPS implementation.

Conclusion

In conclusion, this qualitative study of IPS employment specialists and clients perceived exposure to and use of VR-JIT as improving one’s confidence and motivation for job-seeking. Notably, the elements of the VR-JIT experience (e.g., repeated practice and feedback) for IPS clients may have offered additional support towards improving their understanding and comprehension of the job-seeking experience. Thus, VR-JIT might be particularly helpful for IPS clients with longer periods of unemployment or who have struggled to get jobs since joining an IPS program. As such, IPS supported employment programs may consider implementing VR-JIT as an adjunctive intervention.

Impact & Implications Statement:

This study found that Virtual Reality Job Interview Training (VR-JIT) provided needed exposure and practice with job interviewing in a safe environment for unemployed individuals with serious mental illness. VR-JIT has the potential to enhance the IPS model of supported employment for this population.