Acceptability and Feasibility of Peer Specialist-Delivered Virtual Reality Job Interview Training for Individuals with Serious Mental Illness: A Qualitative Study

Abstract

This study explored peer specialists’ perspectives on delivering vocational interventions, especially Virtual Reality Job Interview Training (VR-JIT). Five focus groups of peer specialists (N = 34) explored their beliefs about vocational services, including VR-JIT. We trained eight peer specialists to be VR-JIT “instructors” and interviewed them about using VR-JIT in their mental health practice. Generally, participants discussed tailoring their vocational services to include sharing their story of recovery. Specifically, participants perceived VR-JIT as acceptable and feasibly implemented within their practice. Overall, participants viewed VR-JIT as a higher level of service and they would be uniquely qualified to support consumers using the tool.

Introduction

Although 70% of people living with serious mental illness (SMI) want to work, only 10–15% do so (Dickinson, Bellack, & Gold, 2007; Rosenheck et al., 2006). However, the right supports can lead to employment (Mueser, Bond, Drake, & Resnick, 1998; Salkever et al., 2007). Beyond providing needed income to a population disproportionately living in poverty (Vick, Jones, & Mitra, 2012), employment improves the quality of life of people living with SMI (Corrigan, Mueser, Bond, Drake, & Solomon, 2008; Dunn, Wewiorski, & Rogers, 2008; Warner, 2009). Peer specialists are individuals with a lived experience of SMI who are trained to help others with similar life experiences (Chinman et al., 2017). Forty-four states have established programs to train and certify peer specialists. (Center on Integrated Health Care & Self-Directed Recovery, 2016). Compared with non-peer staff, peer specialists enhance recovery outcomes and reduce psychiatric hospitalizations among people with SMI (Chinman et al., 2014; Klee, Chinman, & Kearney, 2019). While the most common roles of peer specialists reflect providing direct support (e.g., one-on-one peer support, facilitation of focus groups, recovery planning, and education), they also work in rehabilitation roles and perform tasks such as housing, educational, and vocational assistance (Cronise, Teixeira, Rogers, & Harrington, 2016; Lapidos et al., 2018). Relatedly, peer specialists have been placed on vocational support teams, where they provided vocational services with a modest degree of fidelity (Kern et al., 2013; Swarbrick, Bates, & Roberts, 2009). Qualitative research indicates that peer specialists can be important to vocational interventions because their empathy and absence of judgment are central to helping relationships (Balogun-Mwangi, Rogers, Maru, & Magee, 2019).

As technology-based interventions are increasingly becoming available to psychiatric and vocational services (Fortuna et al., 2019; Nicholson et al., 2017; Nicholson, Wright, Carlisle, Sweeney, & McHugo, 2018; Sohn et al., 2016), one approach to optimizing the work of all employment staff, particularly peer specialists, is to train them to deliver these innovative programs. One area targeted by vocational rehabilitation is training job interview skills, as interviewing can be a barrier to employment, and can be particularly challenging for adults with SMI, who may be prone to anxiety (Braga, Reynolds, & Siris, 2013). To address this barrier, Smith and colleagues (2014) developed Virtual Reality Job Interview Training (VR-JIT) which is a computerized job interview simulator delivered via the internet. VR-JIT was designed to improve interview skills using the Issenberg et al. principles for designing effective simulations (Issenberg et al., 2005) and behavioral learning principles (Cooper, 1982; Cooper, Heron, & Heward, 2007); together promoting the development of sustainable behavioral change (Roelfsema, van Ooyen, & Watanabe, 2010). VR-JIT enables trainees to review an e-learning curriculum about job interview strategies and tips; complete an online job application for a fictional company called “Wondersmart,” and then repeatedly practice interviews with a virtual hiring manager named “Molly Porter.” Trainees choose their responses from scripted options that range from highly effective to highly ineffective and then speak them aloud to “Molly Porter” using the website’s speech recognition function.

Trainees receive four levels of automated feedback during their training that help them retain learned content: (1) real-time nonverbal cues from a job coach; (2) feedback on specific statements made via a color-coded transcript; (3) a qualitative performance assessment on eight learning goals; and (4) a score from 0 to 100 reflecting performance on each learning goal. Moreover, VR-JIT implements hierarchical learning as the interviews with Molly progress through three increasingly more difficult interview types reflecting differences in Molly’s mood (e.g., friendly, strictly professional, and curt/inappropriate).

Training with VR-JIT has been associated with improved interview skills, elevated employment rates, and getting jobs more quickly among individuals with SMI in a series of randomized controlled efficacy trials (Smith, Fleming, Wright, Jordan, et al., 2015; Smith, Fleming, Wright, Roberts, et al., 2015; Smith et al., 2014). VR-JIT is now being evaluated within Individual Placement and Support (IPS) services (Smith, Graham, et al., 2019; Smith, Smith, Graham, et al., 2019), the gold standard supported employment model (Drake, Becker, & Bond, 2020; Frederick & VanderWeele, 2019). Moreover, VR-JIT is being disseminated within community-based and school-based employment services. Although the effectiveness, acceptability, and usability of VR-JIT among individuals with SMI is currently being evaluated (Smith, Smith, Graham, et al., 2019), VR-JIT engagement was found to be acceptable, usable, and was associated with greater employment among transition age youth with disabilities (e.g., autism, emotional disturbance) across 14 special education pre-employment transition service programs (Smith et al., 2021).

Broadly, the role of the VR-JIT instructor is introducing the trainee to this new technology, answering questions, and facilitating engagement. Until now, vocational counselors (e.g., employment specialists within IPS) have served as VR-JIT instructors. However, VR-JIT delivery by peer specialists in a vocational rehabilitation setting may present special facilitators related to their identity as “living exemplars of vocational recovery,” such as the absence of judgment, role modeling, and normalization of life with a psychiatric disability (Balogun-Mwangi et al., 2019). There may also be special barriers related to peer-delivered VR-JIT, given research evidence that peer specialists self-rate their vocational support skills as lower than other frequently-used self-rated skills like crisis support and sharing stories to inspire recovery (Lapidos et al., 2018). As such, adapting VR-JIT for delivery by a new practitioner requires exploration of potential barriers and facilitators prior to implementation. Thus, the current study sought to understand peer specialist beliefs about potential barriers and facilitators influencing peer-delivered VR-JIT as one example of a technological approach to vocational services. We also explored how peer specialists would interpret the role of the VR-JIT instructor in light of their recovery values. As the participants of this study are themselves in recovery from SMI, their perspective also provides insight into the potential acceptability and usability of VR-JIT among individuals with SMI through exploration of implementation domains.

Methods

The Institutional Review Board at the University of Michigan reviewed this study and designated it as exempt human subjects research under exemption #2 of the 45 CFR 46.101; the information obtained from our interviews was not recorded using any identifiers, and the questions were not sensitive in nature. The study team included the principal and co-principal investigators, a master’s degree student, a project coordinator, a doctoral student, and a postdoctoral fellow. The team collaborated with an established five-member Community Advisory Board (CAB) of state-certified peer support specialists that was created the previous year and had collaborated with the PI on a prior study. The CAB consulted on developing the study aims and protocol and coauthored this paper.

Study participants and procedures

We collected data from two sources to explore peer specialists’ beliefs about VR-JIT. First, we invited peer specialists to participate in a focus group to collect qualitative data on their vocational roles and their beliefs about what it would be like to add VR-JIT to their current practice. We recruited peer specialists by distributing flyers at peer specialist conferences and continuing education programs in Michigan, inviting any peer specialists who wished to participate to meet the facilitators in a reserved room. No participants who came to the reserved room were excluded from the study unless they had already participated in one of our prior focus groups. We used this sampling frame because these settings draw large numbers of peer specialists from diverse practices to one place. Five focus groups were held, consisting of 34 participants—52.9% female; 70.6% were 45 years old or older; 56% were Caucasian, and 44% were African American. Peer specialist workplaces were not included in the demographic information that was collected because some peer specialists work in low-frequency settings that would be identifying if reported. Focus group sessions included open-ended questions about peers’ roles in vocational services for individuals with SMI, followed by viewing a VR-JIT video and responding to questions about their views of VR-JIT.

Second, we trained and then interviewed eight peer specialists who were nominated by the CAB to receive VR-JIT training using the same training procedures currently used to train non-peer staff (Smith, Smith, Graham, et al., 2019). The training was provided with the goal of eliciting interview-based feedback on implementing VR-JIT with a peer instructor. Specifically, we asked the peer specialists to complete at least three full interviews at home with the “Molly Porter” virtual hiring manager to complete their training, and then to participate in a semi-structured telephone interview to share their feedback. Of the eight individuals we trained, seven completed three full interviews with “Molly Porter,” and six participated in post-training telephone interviews with a researcher.

Qualitative data processing and analyses

Focus group and interview scripts were developed using (1) feedback from CAB members about key areas to explore; and (2) program implementation areas using The Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009). The CFIR was selected due to its provision of evidence-based implementation domains that help anticipate barriers and facilitators to systems change, and because it has been applied in a mental health focus group context in the past (Lodge, Kaufman, & Stevens Manser, 2017). The CFIR domains of Relative Advantage, Adaptability, Complexity, Patient Needs and Resources, Available Resources, and Knowledge and Beliefs about the Intervention were seen as especially relevant to questions of VR-JIT’s perceived acceptability and feasibility with peer specialist instructors, and informed several stem questions and probes. Both the focus group and interview scripts began with general exploratory questions and then shifted to questions that assessed contextual factors that may influence a peer’s implementation of vocational interventions in general and VR-JIT specifically. Example questions included, “Do you think there is a strong need for this intervention among your consumers? Explain” [Patient Needs and Resources] and, “In your opinion, how complicated would it be to implement this intervention at your site?” [Complexity]. Additional probes included questions about how peer specialists believed they would deliver vocational interventions differently from non-peer staff.

Before starting the interviews and focus groups, participants agreed to be audio-recorded. Additionally, we provided participants with information about the voluntary nature of the interviews and focus groups, potential risks and benefits, and the use and storage of the audio-recordings on secure servers. All interview and focus group discussions were transcribed verbatim. Data were analyzed in four steps utilizing a template approach, which allowed for both deductive coding using the CFIR domains and inductive coding for emerging themes (Crabtree & Miller, 1992). First, three study team members read all transcripts for content and understanding. Second, the three study team members used a constant comparative process to independently code the transcripts into both a template containing CFIR themes and a separate document for emerging themes. Third, the team discussed codes and code definitions until agreement was reached. Fourth, coded areas were translated into final themes. We used Dedoose Version 8.2.14, a web application for managing, analyzing, and presenting qualitative research data (Los Angeles, CA: SocioCultural Research Consultants, LLC www.dedoose.com).

Results

The study’s findings fall into two broad categories. The first category explores peer descriptions of their vocational activities, providing context to where peers are currently situated in vocational services, and discussing how peer support enhances general vocational services. The second category delves into peer specialists’ perceptions of VR-JIT.

Peer support enhancing general vocational services

When peer specialists identified gaps in an individual’s capacity to enter the workforce, they often described providing social support, practical knowledge, and linkages to needed resources. Peer specialist vocational activities entailed not only providing materials and instructions, such as an application form or an address, but also embedding these in real-life settings. For example, peer specialists articulated the importance of validating consumers’ concerns around entering the workforce, accompanying consumers to the social security office, practicing job interviews, finding clothes for an interview, and reminding consumers to follow up with prospective employers. Peer specialists described taking responsibility for, and expanding, tasks often found at the interstices between other roles; especially the informal, day-to-day tasks required to link chronically unemployed consumers to the labor market. When discussing their advantages over non-peer staff, peer specialists emphasized the relevance of their experiences with entering the workforce during their SMI recovery process. In order to relieve their clients’ anxieties around the job interview and instill hope, peer specialists described sharing their own stories of being interviewed, performing as vocational recovery “exemplars” (Balogun-Mwangi et al., 2019). Representative quotations related to these topics can be found in Table 1.

Table 1.

Peer support enhancing general vocational services.

Representative Quotation
We are doing what most everybody else does, I’m sure, but we take it a step farther: We’ll help them fill out the applications, we’ll take them to the place where they fill them application out and sit in on the job interview if we have to. (Focus Group-1)
Really walking them through some of those tough times, and helping them problem-solve how they would get through that, so they won’t react and just leave the job… And if they really don’t want to be there… how to leave the job, so they can be hirable for the next position. (FG-3)
Participant 2: For example, if someone is worried about the interview, “Okay, yeah, most interviews go good, and let me tell you about a bad interview that I had, and tell you that I made it through it, and so you are going to be able to make it through it. I spilled water and blah blah blah… and guess what, I still got the job!”
Participant 5: “I tripped and fell in front of the person that was hiring me and slammed my face!”
Participant 2: “… and that was, by the way, the interview for this job that I am working right now.” (FG-1)

Peer specialist perceptions of VR-JIT

After viewing a video demonstration of VR-JIT, or training to deliver VR-JIT, participants were asked questions to elicit ideas about what it would be like to implement VR-JIT as peer specialist instructors in their organizations.

General beliefs about acceptability and feasibility

When expressing beliefs about queried domains such as complexity, patient needs and resources, and knowledge and beliefs about the intervention, most participants agreed that VR-JIT would be exciting for peers; that they themselves learned something by practicing with it; and had positive experiences with it, which suggests VR-JIT would be an acceptable service for their toolbox. In addition, peer specialists perceived that the VR-JIT implementation process in their organizations would be straightforward and uncomplicated, which reflects the potential feasibility of VR-JIT. However, a few participants had misgivings about whether the VR-JIT would require a technological facility not possessed by certain consumers, especially older consumers. Representative quotations related to this theme can be found in Table 2.

Table 2.

General beliefs about acceptability and feasibility.

Representative Quotation
I think peers would like it. (…) I think people would use it. I think people would sign up to use it because it’s something new. So, you know, just the new aspect. People would want to try it and see what it’s all about. (Interview-2)
Maybe to the millennials that would be cool, right? “Okay, yeah, I’m going to go in and get it over with.” But, to baby boomers, it’s like, “I need to see somebody face-to-face,” right? “My word is my bond. I need to look them in their eyes and not look at a machine!” (FG-4)
In the mental health setting, with everything compounding on top of that, with people with certain mental health issues that are not going to want to be put in a situation where they are seeing a simulated reality vs. the actual reality, I think it is overall just better to do [interview trainings] in person … (FG-5)
I did learn, and I had a positive experience. Ninety-five percent of it was, you know, I wanted to go do another one. You know what I mean? When I got done. (…) For me, to set that in front of him (the client), he’d have to have that spark of hope in his, you know, “Hey, I’m going to do this, so I don’t make a mistake in a real one.” You know what I mean? And to blow that candle out with a glitch, you know? There’s another thing that doesn’t work, you know? And that’s how important I think it is to have that thing, you know, just spot on. (Int-2)

Beliefs about relative advantage of peer delivery

Participants commented that if VR-JIT was made available in their organizations, their lived experience of SMI recovery would be helpful for introducing and coaching consumers to use it and that they would specifically tailor their approach to include self-disclosure of employment-related struggles and successes. Participants mentioned, for instance, that they could make the intervention “a little more warm and a little more personal” (FG-3), or they could connect it to their consumers’ individual needs. They framed their perceived advantages over non-peer staff from a lived experience perspective. Deeper familiarity with individual consumers, the ability to assess consumers’ readiness for practice interviews, and flexibility with time were also brought up as advantages that peer specialists may have over non-peer staff concerning VR-JIT that could generalize to other technology-related tools.

The peer specialist participants anticipated two layers of connection between peers and consumers. In the first layer, participants discussed having similar past job-seeking experiences, and how they share specific stories with their clients in order to provide encouragement or illustrate the importance of interview skills, professionalism, and persistence. In the second layer, participants described having similar experiences of learning to use VR-JIT itself, immediately translating their own learning process into the language of lived experience that their clients will understand. Representative quotations related to this theme can be found in Table 3.

Table 3.

Beliefs about relative advantage of peer specialist delivery.

Representative Quotation
I think with a non-peer they would kind of just stick to the script and I think a peer would add a little more experience: ‘Well, when I interviewed, these are the things that I said and did.’ (Int-6)
I think it will work well with peers doing it … We can assist them and say, “Try it, I am here with you.”
Someone who is coming from the outside may not have that type of empathy. They may not be able to understand what the consumer is going through, and we have that background and relationship to help them move forward and higher in their life if they choose to. (FG-3)
Participant 2: I don’t see many clinicians sitting down to do this. I see them making an electronic babysitter: “You’re going to do this. I’m going to go over here to do paperwork, and I’ll see you when you are done.”
Participant 5: That’s a really good point!
Participant 2: That’s not to shit, I’m sorry, on clinicians at all. It’s just that they are busy; they have stuff to do. We, most of us, have flexible enough schedules at our jobs… that we would say, “Oh, I will dedicate an hour, and we will get through this.” (FG-1)
In my opinion, it’s that people that we work with tend to be highly sensitive. How do I want to say this? If it’s somebody that says, “I know this [VR-JIT] is frustrating. I was really frustrated.” And being able to share the story throughout. You know, “This [VR-JIT] is kind of nerve-wracking. I get that. I was really nervous the first time that I…” and just, you know, being there to be relatable, and share concerns, and coach in a way that can be heard. (Int-4)

Beliefs about suitability of VR-JIT for peer practice

Several peer specialists found that a peer-delivered version of VR-JIT would constitute a higher level of service and an employment opportunity for them. However, a few peer specialists reported an incompatibility between their core mission and the use of VR-JIT. Specifically, they perceived the VR-JIT as a tool that could interfere with the direct, authentic relationship between peer specialists and consumers. One participant, for instance, expressed a concern that such technology-based interventions would “take the human element out of it” (FG-5). Another participant stated a preference for their established job interview preparation process over technology-driven interventions. This concern was often built on the underlying view that using VR-JIT could at first be intimidating and stressful for the consumer. One peer specialist argued that introducing a tool that could potentially provoke anxiety would “violate a confidence that some of [their] consumers will have on them” (FG-3). Meanwhile, another peer specialist explained that she would overcome this potential problem by getting more familiar with the intervention program. Representative quotations related to this theme can be found in Table 4.

Table 4.

Beliefs about suitability of VR-JIT for peer specialist practice.

Representative Quotation
Molly is really scary! I did “scary Molly”, I think, four times out of the seven. I try to do each level at least twice, and then “scary Molly”, Molly scared me enough that I wanted to see what would happen. So, I responded in a certain way or didn’t respond a certain way. … If we’re trained in being able to utilize this program, then we can get in and say, “Look, you know, this person Molly might intimidate you,” and maybe be able to work on that part … Because you’re going to see, in the real world, you’re going to see every type of personality when you’re interviewing. And so, as a peer, we can kind of say, “You know, this person [Molly] is going to be a little bit more intimidating, so maybe we can work on some skills to help you get through that, so you’re not flustered if you’re face to face.” (Int-1)
I think it would be excellent for the peer because the peer would be able to provide a little experience, but also it would enhance their professional development and help them sell themselves, so they can help others, and then they would be able to help the other individual navigate through. I think the peer bringing the lived experiences along with the technology creates the link that is win-win for the peer along with the person that they are serving. (FG-3)
I want them to come into the office, and I want them to sit down, and so I’m going to be interviewing for a job. I don’t want them sitting behind a screen, doing it with somebody that they don’t know. It’s just me. (FG-4)

Discussion

This study locates VR-JIT in a landscape in which peer specialists already perform informal, practical tasks that link chronically unemployed people with a serious mental illness to the workforce. In the focus groups and interviews, peer specialists shared their perspectives on working within the interstices of vocational services by filling gaps left by clinicians or vocational specialists. Our findings are relevant to connecting people with SMI to the workforce, which often involves learning social and behavioral expectations, developing skills, and adapting to the workplace culture. In general, peer specialists reported they support clients by drawing upon their lived experiences of seeking work in ways that are beneficial for clients who are motivated to seek employment.

Relatedly, peer specialists can facilitate the implementation of VR-JIT by using their knowledge of consumer backgrounds and needs, and their lived experiences. Peer specialists viewed themselves as capable of keeping consumers engaged in VR-JIT by either (1) sharing their own experiences with entering the workforce during their mental health recovery, or (2) spontaneously addressing consumers’ questions and anxieties around the intervention itself, especially through self-disclosure of their unique challenges in learning the training intervention.

The peer specialists’ perspective emerging most prominently from this data is that VR-JIT is an acceptable intervention, and that implementation of a peer-delivered VR-JIT in the participants’ various organizations would be feasible as well as could provide a higher level of service and employment opportunity for peer specialists. Although some participants expressed concerns about whether VR-JIT implementation would fit peer specialists’ scope of practice and mental health recovery values, other participants perceived VR-JIT as a tool consistent with the key features of peer services and that they are invested in learning more about how VR-JIT might fit into their practice. Therefore, we suggest that organizations can address concerns about VR-JIT fitting with the values and scope of peer specialist practice by including peer specialists as part of the implementation process; and by anticipating these specific concerns during the training process such that trainers contextualize how VR-JIT fits within the scope of peer specialists’ practice, as opposed to changing it.

Research suggests that when peer specialists are formally asked to provide employment support, “they [peer specialists] must be equipped with requisite knowledge, skills, and resources to deliver that support in service of a specific goal” (Balogun-Mwangi et al., 2019, p. 460). In this respect, VR-JIT adds structured, employment-related intervention skills to the peer specialists’ toolbox. Findings on the use of lived experiences in this context expand prior work on peer-delivered vocational interventions, in that it shows how peer specialists augment a fairly standardized intervention program by including elements of self-disclosure. Moreover, the delivery of VR-JIT by peer specialists might help reach populations with limited access to or trust in formalized care. Equipping peer specialists to deliver VR-JIT also has the potential to task-shift some duties from vocational specialists to peer specialists, without sacrificing quality or effectiveness.

This qualitative study should be understood in the context of certain limitations. Peer specialists’ workplaces were not included in the demographic information that was collected because some peer specialists work in low-frequency settings that would be identifying if reported. Additional focus groups or interviews may have yielded additional themes. Our findings explored the perceptions of peer specialists based on limited exposure to VR-JIT, as the training was delivered with the purpose of eliciting feedback. To fully understand barriers and facilitators to the implementation of VR-JIT, consumers and supervisors’ views would also need to be explored, and the peer specialists would need to experience how VR-JIT feels with real consumers. Given our findings that the VR-JIT and other structured vocational intervention programs are feasible from peer specialists’ point of view, future research can pilot the implementation of a peer-delivered VR-JIT in a naturalistic context.