An RCT of Virtual Reality Job Interview Training for Individuals With Serious Mental Illness in IPS Supported Employment

Abstract

Objective:

Virtual Reality Job Interview Training (VR-JIT) is a computerized interview simulator with efficacy at enhancing interview skills and employment outcomes. A randomized controlled trial assessed VR-JIT effectiveness for participants in individual placement and support (IPS), in which approximately 55% of individuals with serious mental illness obtain employment.

Methods:

Ninety participants with serious mental illness were randomly assigned to IPS+VR-JIT (N=54) or IPS as usual (N=36), completing pretest-posttest assessments and an employment evaluation at 9 months. Intent-to-treat chi-square analysis, multivariable logistic regression, Cox proportional hazards models, and mixed-effects linear regressions were conducted. Fifty-one percent were IPS nonresponders (i.e., no employment within the first 90 days of IPS).

Results:

IPS+VR-JIT participants did not have significantly higher employment rates, compared with IPS-as-usual participants (43% versus 28%). IPS nonresponders (N=46) in the IPS+VR-JIT group had greater odds of obtaining employment (odds ratio [OR]=5.82, p=0.014) and shorter time to employment (hazard ratio=2.70, p=0.044) compared with IPS nonresponders in the IPS-as-usual group. Intent-to-treat mixed-effects linear analyses indicated that IPS+VR-JIT, compared with IPS as usual, significantly improved interview skills (p=0.006), interview confidence (p=0.013), and interview anxiety (p=0.019).

Conclusions:

VR-JIT’s potential benefits (increased employment in a shorter time) appeared to be specific to IPS nonresponders, whereas employment outcomes for recent IPS enrollees were not affected. VR-JIT could be a valuable resource for employment specialists to support IPS nonresponders, because 47% of participants engaged in mock interview training with their specialist. Future research should focus on evaluating the effectiveness and implementation of VR-JIT among IPS nonresponders.

Approximately 10%–20% of individuals with schizophrenia and other serious mental illnesses are competitively employed (1), which contributes to their improved physical and mental health, quality of life, social capital, and reduced poverty (2, 3). The individual placement and support (IPS) model of supported employment focuses on rapid job placement through job development with ongoing supports (4, 5) and is more effective than other supported employment programs at improving competitive work outcomes (6). Still, approximately 45% of IPS clients do not obtain competitive employment (7).

Job interviews are an established factor contributing to successful job attainment in both the general population (8–11) and among individuals with psychiatric or other disabilities (12–15). Job interviewing is anxiety provoking for most people (16) and more so for individuals with serious mental illness, who are prone to anxiety and often have impairments in their interpersonal skills (17, 18). Given that interviewing skills predict employment among individuals with serious mental illness (12, 13, 19, 20), improving such skills could be an important treatment target to enhance IPS outcomes. However, research focused on job interview skills training has not yet occurred within IPS. Although training in interviewing skills is not a core component of IPS (21, 22), it is recommended that employment specialists engage clients in practice interviews when needed (21–24). However, it is not known how frequently employment specialists engage their clients in practice interviews for jobs or whether job interview skills are improved by such practice, nor are the benefits of improving job interviewing skills in IPS known. Thus, tools to facilitate the training of interviewing skills among persons receiving IPS have the potential to improve work outcomes.

To address the need for improved tools to facilitate job interviewing training, a computerized job interview simulator, called Virtual Reality Job Interview Training (VR-JIT; www.simmersion.com), was developed to provide individuals with serious mental illness an opportunity to practice and hone their interview skills while receiving automated feedback. Prior evaluation of VR-JIT in a series of laboratory-based randomized controlled trials (RCTs) for job-seeking individuals with serious mental illness found that VR-JIT improved participants’ interview skills and self-confidence, led to more competitive job offers, and was associated with a shorter time to employment (25–31). Although approximately 30% of the participants in these efficacy studies were engaged in vocational rehabilitation, none of these studies were embedded within IPS (25–31).

The RCT reported here evaluated the effectiveness of VR-JIT when delivered within a high-fidelity IPS supported employment program. The primary hypotheses were that the provision of VR-JIT in addition to IPS would result in better employment outcomes, compared with IPS alone. Secondary hypotheses were that participants receiving VR-JIT would have greater improvements in interview skills, interview self-confidence, and interview anxiety, compared with those receiving IPS as usual.

METHODS

Trial Site

The study was conducted at Thresholds (Chicago), a nonprofit community behavioral health agency that provides comprehensive mental health services, including IPS supported employment (http://thresholds.org).

Trial Design

We conducted an RCT with a type 1 hybrid effectiveness–implementation design (32) to test whether adding VR-JIT to IPS improves employment outcomes, compared with IPS as usual, describe the implementation process, and identify barriers to VR-JIT uptake. This article reports on the RCT data, and a report on the implementation processes and outcomes is forthcoming. The study protocol was approved by University of Michigan and Northwestern University institutional review boards, and all procedures were reviewed, approved, and monitored by a data and safety monitoring board (DSMB).

As detailed in our study protocol (33), employment specialists initially implemented VR-JIT at a centrally located computer lab. However, this approach resulted in low recruitment rates because of transportation and scheduling difficulties. Because VR-JIT is portable (e.g., Internet-based, delivered via computer), the location of VR-JIT delivery was changed to local neighborhood sites where participants received most of their IPS services (which improved recruitment rates). At this time, VR-JIT implementers transitioned from five employment specialists to three trained research staff to avoid significant delays related to the redesign of employment specialists’ workflow to deliver VR-JIT. Using research staff allowed for the development of an optimal process of integrating VR-JIT within IPS without affecting IPS services or fidelity. This process is being evaluated as part of a VR-JIT implementation study. Although the study was originally powered to randomly assign participants at a 1:1 ratio to receive IPS+VR-JIT or IPS as usual (33), recruitment challenges necessitated reducing the target from 160 to 133 participants by using a 3:2 randomization scheme (IPS+VR-JIT:IPS as usual). We initially planned to include only participants who were within 2 weeks of their enrollment in IPS. However, lower-than-expected enrollment necessitated expansion to all clients currently receiving IPS (i.e., with one or more contacts with an employment specialist in the past 30 days) (33).

Participants

Inclusion criteria were being age 18 or older; having a diagnosis of schizophrenia spectrum disorder, bipolar disorder, or major depression via the Structured Clinical Interview for the DSM-5 (SCID-5) (34); having a reading level greater or equal to fourth grade, as assessed with the Wide Range Achievement Test–V (WRAT) (35); being currently unemployed or underemployed; currently being a job seeker (via self-report, “Will you actively be looking for a job in the next 4 weeks?”); and willingness to be video recorded for assessments. Exclusion criteria were having a disability or medical illness that could significantly compromise cognition (e.g., traumatic brain injury), having a documented uncorrected vision or hearing problem that would prevent the participant from using VR-JIT, and being actively suicidal within the past 30 days, as assessed with the Columbia–Suicide Severity Rating Scale (36).

A total of 97 participants met inclusion criteria, provided informed consent, completed baseline study measures, and were randomly assigned to IPS+VR-JIT (N=59) or IPS as usual (N=38) (see CONSORT diagram in the online supplement to this article). Because IPS services were severely affected by COVID-19 pandemic restrictions during spring 2020, the DSMB retroactively stopped enrollment on February 1, 2020. The final intent-to-treat sample included 54 participants in IPS+VR-JIT and 36 participants in IPS as usual (N=90).

Study Measures

Baseline assessments gathered data on demographic, clinical, and cognitive characteristics; participants’ functional ability; and employment history. We assessed the following variables at pretest (baseline) and posttest (approximately 3 months later): interview skills, interview confidence, interview anxiety, and social competence. Comprehensive employment outcomes were obtained monthly throughout the 9-month study period. Because we expanded our inclusion criteria from recent IPS enrollees to also include unemployed clients who had been receiving IPS for a longer time, we ascertained post hoc the number of clients who met criteria for being an IPS nonresponder, as proposed by McGurk and colleagues (37): that is, they received IPS for at least 90 days without obtaining a job or obtained a job in IPS but lost it (were fired or quit) within 90 days.

Participant characteristics.

Participants completed a brief survey about their background (e.g., age and race-ethnicity), work history (e.g., employed full- or part-time within the past 2 years), whether they received vocational support prior to IPS (0, no; 1, yes), and total days in IPS at baseline.

Symptoms were assessed with the Brief Psychiatric Rating Scale (BPRS), with a four-factor solution used to create the following subscale domains: thought disturbance, anergia, depressed affect, and disorganization (38, 39). Participants completed the Alcohol Use Disorders Identification Test (AUDIT), with recoded total scores reflecting hazardous drinking, 1; and nonhazardous drinking, 0 (40). They also completed the Drug Abuse Screening Test (DAST), with recoded total scores reflecting moderate-to-severe use, 1; and low use, 0 (41). We computed a “problematic substance use” variable by combining the AUDIT and DAST, in which problematic substance use was a score of 1 on either the AUDIT or the DAST, and no problematic substance use was a score of 0 on both the AUDIT and the DAST.

Cognition was assessed with the MATRICS Consensus Cognitive Battery (MCCB) (42), which generates T scores for the following six domains: processing speed, attention-vigilance, verbal working memory, verbal learning, visual learning, and reasoning and problem solving. Mean scores were computed across these six indices to reflect overall neurocognition. The MCCB also included a separate T score for social cognition based on the Mayer-Salovey-Caruso Emotional Intelligence Test (43).

Community functioning was assessed by using the total score from the self-report version of the Specific Levels of Functioning (44). This instrument includes 30 items (rated, for example, as 1, highly untypical, to 5, highly typical; Cronbach’s α=0.87) across the domains of interpersonal relationships, social acceptability, activities of daily living, and work skills.

Primary outcomes.

Obtaining a competitive job (i.e., located in an integrated community setting that pays at least minimum wage and was not set aside for a person with a disability) during the 9-month follow-up was coded 1, yes, or 0, no. Competitive employment outcomes were verified by using a combination of self-reports, IPS employment records, and reports from employment specialists. Validation of the employment was required from at least two of these sources. Randomization date and employment start date were used to determine time to employment.

Secondary outcomes.

Interview skills were assessed. Participants completed two interview role plays at the pre-and posttest visits. The two interviews (at each time point) were video recorded and coded by raters masked to study condition and time point and with use of the Mock Interview Rating Scale (MIRS) (25–27, 29, 30). The MIRS uses an anchoring system to rate and score eight job interview skills (e.g., conveying oneself as a hard worker, sounding easy to work with, and sharing things in a positive way) on a 5-point scale. We used a total score across the eight ratings to reflect performance-based job interview skills for each role play and computed an average score for the two role plays at each assessment point to provide a single pretest score and a single posttest score. The MIRS demonstrated strong internal reliability and sensitivity to the effects of VR-JIT in prior studies (25–27, 29, 30) (for additional details on the MIRS, see online supplement).

Interview self-confidence was also rated. Participants self-reported their confidence in interview skills at pretest and posttest by using a nine-item survey, with items rated on a 7-point scale (1, extremely unskilled, to 7, extremely skilled); total scores were computed across all items. This survey demonstrated strong internal reliability and sensitivity to the effects of VR-JIT in prior studies (25–27, 29, 30).

Participants also rated their interview anxiety with an adapted version of the 34-item Personal Report of Public Speaking Apprehension (PRSPA) (45); the phrase “public speaking” was replaced with “job interviewing.” Items were coded from 1, strongly disagree, to 5, strongly agree. Total scores were computed by using the PRSPA’s validated two-step total score procedure, which is described elsewhere (45).

To assess social competence, participants completed the Social Skills Performance Assessment (SSPA) (46) at pretest and posttest. The SSPA includes two brief (3-minute) role plays in which participants engage in a conversation with an unknown confederate who plays the role of a new neighbor (NN) or a landlord (LL). The NN and LL role plays were video recorded at the pre- and posttest visits and were blindly rated by use of an anchoring system. The NN role play was scored on eight items via a 5-point scale (e.g., 5, very interested, to 1, very disinterested), and the LL role play was scored on nine items via a 5-point scale (e.g., 5, very focused, to 1, very unfocused). The item-level means were computed for each role play and then averaged for a single social competence score at pre- and posttest (for additional details on the SSPA, see online supplement).

Interventions

IPS.

All participants received IPS from five Thresholds teams. Independent ratings by the State of Illinois with the IPS Fidelity Scale (4) showed mean IPS fidelity ratings from 111 to 117, reflecting good-to-exemplary fidelity to IPS on all scales. To isolate the impact of VR-JIT on improving interviewing skills and employment outcomes, employment specialists were asked to refrain from conducting practice interview role plays with participants randomly assigned to IPS+VR-JIT, but they could continue discussing interview preparation with them. Employment specialists completed regular surveys reflecting specific interview preparation activities engaged in during each participant contact (for both study groups) over the past week: number of practice interviews completed, estimated time spent on practice interviews, and estimated time discussing job interview preparation.

VR-JIT.

VR-JIT is an interactive, computerized job interview simulator (see figure in online supplement) developed by SIMmersion LLC that uses a virtual hiring manager named Molly Porter, speech recognition, and nonbranching logic components to facilitate repeatedly practicing job interviews across three levels of difficulty (i.e., easy, medium, and hard) that portray different variations in Molly’s personality (e.g., friendly, professional, and inappropriate). Notably, Molly’s personality is selected at random and her mood can change depending on responses to her questions. On the basis of the job interview literature (47), VR-JIT highlights eight interview skills within an e-Learning curriculum designed to convey positive attributes of the applicant (e.g., being a hard worker and being easy to work with) and share job interview preparation tips (33). Prior to interviewing, trainees complete a job application for one of eight positions, which informs the questions asked during the virtual interview. Between Molly’s personality and mood and the unique questions for eight different positions, VR-JIT yields more than 20 hours (approximately 40 interviews) before content will begin to feel repetitive. Also, trainees receive automated feedback in real time and via transcript review and receive a score of 0 to 100 that is anchored to a summary performance assessment of the eight interview skills targeted by the intervention. Participants’ VR-JIT engagement was monitored for the total number of completed virtual interviews, total minutes engaged with the virtual interviewer, total minutes engaged with e-Learning, and mean performance score across all completed interviews.

Participants randomly assigned to IPS+VR-JIT were instructed to attend five, 2-hour VR-JIT sessions over approximately 4 weeks to complete a recommended progression across easy, medium, and hard interviews (see details in the online supplement). This approach was recommended based on the VR-JIT efficacy trials (25–31). However, the VR-JIT implementation in this study was pragmatic and allowed for clients and VR-JIT implementers to modify the number and length of sessions to fit within real-world services. Participants attended an initial face-to-face VR-JIT orientation on how to navigate the simulation (approximately 80 minutes) led by a VR-JIT implementer, who completed an adherence checklist to track whether the implementer had followed the orientation protocol.

The VR-JIT implementers were employment specialists for the first five participants, who used VR-JIT in the lab model, and research staff for the remaining 43 participants, who used VR-JIT in the community model. The checklist completion rate was 92.8%, reflecting exemplary fidelity. After the orientation visit, the remaining visits (~70 minutes each) were coordinated by the VR-JIT implementer (including reminder calls) and consisted of face-to-face meetings in which VR-JIT implementers supported participants to log into the tool (if necessary), review e-Learning content (when requested), complete one or two virtual interviews (lasting approximately 25 minutes each), and then review transcripts and performance assessment feedback. The VR-JIT implementer monitored participant progress across easy, medium, and hard interviews by using a curriculum tracking form (for additional details on the VR-JIT implementation, see online supplement).

Study Procedures

We recruited from active caseloads from five IPS teams supervised by the same IPS administrator but located in geographically separate areas with different employment specialists. After referral, potential participants provided informed consent and then completed pretest visit 1, which included the SCID-V and WRAT (to verify inclusion), self-reported behavioral health status, interview self-confidence, and interview anxiety. At pretest visit 2, participants completed an employment history interview, the BPRS and MCCB, and the MIRS and SSPA role plays. Participants were then randomly assigned to IPS+VR-JIT or IPS as usual. Posttest visits were scheduled for approximately 3 months post-randomization. With use of established methods (48), posttest assessment dates for the IPS-as-usual group were yoked to the IPS+VR-JIT dates to avoid any confounding of group assignment with time elapsed between pre- and posttest assessments (48). The posttest visit was followed by monthly phone calls to check employment status and a 9-month follow-up visit as a final employment status check and review of behavioral health status (e.g., BPRS).

Data Analysis

Chi-square analyses and t tests were conducted to examine differences in baseline characteristics between randomly assigned study groups. To test hypotheses for the primary and secondary outcomes, intent-to-treat analyses were conducted by including all randomly assigned participants. Directional hypotheses regarding the effect of group on the primary and secondary outcomes were evaluated with one-sided tests, given prior research supporting the effects of VR-JIT on these outcomes (25–30, 49–51).

Chi-square, logistic regression, and Cox proportional hazards models evaluated whether IPS+VR-JIT resulted in a greater likelihood of employment and shorter time to employment during the 9-month study period. The proportional hazards assumption was evaluated by using a group-by-time interaction term and visual inspection of log minus log function plotted against time. The intent-to-treat analyses of the primary employment outcomes were conducted adjusting for covariates known to influence employment among individuals with serious mental illness, including prior employment (52), neurocognition (37, 53, 54), social cognition (55–57), negative symptoms (58, 59), community functioning (60–62), and problematic substance use (60, 63, 64). Because external historical events (e.g., changes in local economies in any given year) are associated with obtaining employment (65), we included baseline year of study participation as a covariate. Prior employment and neurocognition covariates were not significant predictors of employment or time to employment. Thus, they were dropped from the final models in the interest of parsimony. Baseline year of study participation, problematic substance use, social cognition, community functioning, and negative symptoms (i.e., anergia) were the final covariates in the employment outcome analyses. (A correlation matrix of the variables considered for the primary intent-to-treat analyses is included in the online supplement.)

Mixed-effects linear regression models with random intercepts and an AR[1] correlation structure (66) were used to test the secondary hypotheses that IPS+VR-JIT would lead to greater improvements in interview skills, interview self-confidence, interview anxiety, and social competence over time by modeling the effects of the group-by-time interaction. The intent-to-treat analyses of secondary outcomes were conducted, and we adjusted for the same covariates as in the primary outcome analyses.

Item-level and measure-level missing data were observed. Sixteen participants did not complete posttest measures. We did not impute measure-level data and analyzed only the raw available data. By using the interquartile range method (67), outliers were observed for interview self-confidence (N=2) and interview anxiety (N=3). Winsorized weighted outlier replacement yielded results similar to those of analyses with trimmed outliers; we thus opted for the latter approach, and the outliers were excluded from the analysis (see online supplement for power analysis details).

RESULTS

Participant Characteristics

The 90 participants’ mean±SD age was 45.6±12.8, 57% (N=51) were male, 61% (N=55) had a schizophrenia spectrum disorder, and 39% (N=35) had a nonpsychotic mood disorder. Fifty-one percent (N=46) of participants met IPS nonresponder criteria. The study groups did not differ with respect to demographic, clinical, cognitive, or employment history variables or IPS nonresponder status (Table 1).

TABLE 1.

Characteristics of the study sample, by group^a

Characteristic	IPS as usual (N=36)		IPS+VR-JIT (N=54)		Test statistic	df	p
	N	%	N	%
Age (M±SD)	43.8±12.8		46.9±12.8		t=1.1	88	.265
Male biological sex	22	61	29	54	χ2=.5	1	.487
Race-ethnicity
Black, African American	24	67	35	65
White	10	28	13	24	χ2=1.2	3	.759
Latinx	2	6	5	9
More than one race	0	–	1	2
Years of education (M±SD)	12.5±1.8		12.5±2.2		t=.1	88	.905
Primary psychiatric diagnosis
Schizophrenia spectrum disorder	19	53	36	67	χ2=1.8	1	.185
Bipolar disorder (no psychotic features)	9	25	6	11	χ2=3.0	1	.083
Major depressive disorder (no psychotic features)	8	22	12	22	χ2=.0	1	1.00
Brief Psychiatric Rating Scale (M±SD subscale score)b
Thought disturbance	6.1±3.7		5.3±2.8		t=−1.3	88	.206
Anergia	5.0±2.4		5.1±2.0		t=.1	88	.905
Depressed affect	9.4±4.2		8.0±3.2		t=−1.8	88	.078
Disorganization	3.7±1.2		3.6±1.2		t=−.2	88	.759
Problematic substance usec	10	28	10	19	χ2=1.1	1	.301
Cognitive ability (M±SD score)
Neurocognitiond	33.8±9.9		34.1±7.9		t=.2	88	.858
Social cognitione	35.0±13.6		36.6±15.8		t=.5	88	.615
Community functioning (M±SD score)f	127.5±14.5		130.4±12.7		t=1.0	88	.318
Employment history
Had competitive employment in past 2 years	19	53	21	39	χ2=1.7	1	.194
Ever employed full-time	23	64	38	70	χ2=.4	1	.519
Ever employed part-time	29	81	48	89	χ2=1.2	1	.271
Days in IPS at baseline (M±SD)g	130.3±106.4		105.4±94.4		t=−1.1	72	.147
IPS nonresponderh	21	58	25	46	χ2=1.3	1	.263
Criminal justice history: prior felony conviction	10	29	18	33	χ2=.2	1	.637

^aIPS as usual, received standard individual placement and support (IPS) services; IPS+VR-JIT, received IPS standard services and Virtual Reality Job Interview Training (VR-JIT).

^bAs measured by the Brief Psychiatric Rating Scale (38, 39). Possible subscale scores range from 4 to 28 for thought disturbance and anergia, 5 to 35 for depressed affect, and 3 to 21 for disorganization, with higher scores indicating more severe symptomatology.

^cAs measured by the Alcohol Use Disorders Identification Test (40) and Drug Abuse Screening Test (41). Problematic substance use is reflected as 1, yes, or 0=no.

^dAs measured by the MATRICS Consensus Cognitive Battery (42). Corrected T score mean±SD is 50±10, with higher scores indicating better neurocognition.

^eAs measured by the Mayer-Salovey-Caruso Emotional Intelligence Test (43). Corrected T score mean±SD is 50±10, with higher scores indicating stronger social cognition.

^fAs measured by the Specific Levels of Functioning scale (44). Possible scores range from 30 to 150, with higher scores indicating better functioning.

^gTwo IPS-as-usual participants and one IPS+VR-JIT participant were enrolled in IPS for longer than 700 days; these individuals were outliers, and they were not included in the mean reporting of days in IPS at baseline.

^hA nonresponder was defined as a person who received IPS services for at least 90 days without obtaining a job or who obtained a job but lost it (was fired or quit) within 90 days.

IPS and VR-JIT Engagement

Table 2 presents data for the two groups on participant engagement in practice interview role plays and job interview discussions with their employment specialists. Per reports from employment specialists, 47% of the IPS-as-usual participants completed at least one practice job interview, and 69% discussed their interview skills with their specialist; 11% (N=4) of participants had missing data from their specialists on these variables. Although employment specialists were instructed to refrain from conducting practice interviews with IPS+VR-JIT participants, 17% (N=9) had such practice interviews. Approximately 65% of IPS+VR-JIT participants discussed their interview skills with their employment specialist, and 54% (N=28) discussed their progress in VR-JIT; 4% (N=2) of participants had missing data from their specialists on these variables.

TABLE 2.

Job preparation practices and IPS services engagement of study participants, by group^a

	IPS as usual (N=36)				IPS+VR-JIT (N=54)
Variable	N	%	Median	Range	N	%	Median	Range
IPS with employment specialistb
Completed mock interview	17	47			9	17
N of mock interviews completed (M±SD)	1.8±1.2		1.0	1–5	1.4 ±.7		1.0	1–3
Minutes of mock interviewing (M±SD)c	24.3±14.3		22.5	5–60	19.4±15.5		15.0	5–45
Discussed interview skills	22	69			34	65
Minutes discussing interview skills (M±SD)d	40.2±26.5		40.0	10–105	38.0±35.8		30	2–165
Discussed VR-JIT interviewing progresse					28	54
Minutes discussing VR-JIT interviewing progress (M±SD)					38.8±41.9		25.5	1–160
VR-JIT engagement (M±SD)f
Sessions completed					4.7±2.1		4.5	1–10
Virtual interviews completed					8.8±4.2		10.0	1–15
Minutes of virtual interviewing					177.8±102.4		174.5	9–432
Virtual interview scoreg					78.7±11.6		80.9	46–96
Minutes of e-Learning					14.4±14.1		13.0	0–80

^aIPS as usual, received standard individual placement and support (IPS) services; IPS+VR-JIT, received IPS standard services plus Virtual Reality Job Interview Training (VR-JIT).

^bEmployment specialist notes regarding job interview preparation were missing for four IPS-as-usual participants and two IPS+VR-JIT participants. Nine IPS+VR-JIT participants completed one to three mock interviews with their employment specialists, which was a minor violation of the protocol meant to isolate the effects of VR-JIT for secondary analyses.

^cMeans do not include data from one IPS-as-usual participant who completed 210 minutes of mock interviewing with the employment specialist, because these data represented an outlier.

^dMeans do not include data from one IPS-as-usual participant who completed 225 minutes discussing interview skills and two IPS+VR-JIT participants who completed 315 and 205 minutes discussing interview skills, because these data were outliers.

^eTwo IPS_VR-JIT participants were missing specialist notes regarding discussing VR-JIT progress.

^fSix IPS+VR-JIT participants did not engage in VR-JIT (i.e., completed no VR-JIT sessions).

^gAs measured by the VR-JIT algorithm. Possible scores range from 0 to 100, with higher scores indicating better interview performance.

Notably, the discussion of VR-JIT happened naturalisti-cally and reflected general discussion about what participants learned from their virtual interview practice. Employment specialists were not instructed or trained to check in on VR-JIT progress, and results were not intentionally shared with them. The IPS-as-usual and IPS+VR-JIT groups did not differ with respect to the number of minutes in which interview skills were discussed with employment specialists. Engagement in VR-JIT showed that 48 (89%) individuals in this group participated in at least one VR-JIT session. Fifty percent (N=24) of these engaged participants completed at least five sessions, 25% (N=12) completed four sessions, and 10% (N=5) completed three sessions over 9.3±6.7 weeks. Notably, 14 (29%) of the 48 IPS+VR-JIT participants who engaged in VR-JIT completed more than five sessions (range six to 10 sessions) as a pragmatic adaptation that was necessary to progress from easy to hard interviews.

Primary Outcomes

Regarding employment, among the participants randomly assigned to IPS+VR-JIT, 43% (N=23) obtained a job during the 9-month study period, compared with 28% (N=10) in the IPS-as-usual group, which was not a statistically significant difference (see figure in online supplement). The multivariable logistic regression model adjusting for baseline year of study participation, problematic substance use, social cognition, community functioning, and negative symptoms (anergia) found that the IPS+VR-JIT group had significantly greater odds of obtaining employment within 9 months (odds ratio [OR]=3.03, p=0.027) (Table 3), compared with the IPS-as-usual group. The multivariable Cox proportional hazards model that adjusted for the same variables as the logistic regression indicated that the IPS+VR-JIT group had a shorter time to employment, compared with the IPS-as-usual group, but the difference was not statistically significant (Table 3) (see figure in online supplement).

TABLE 3.

Intent-to-treat odds of obtaining employment and hazard ratios of time to employment by 9-month follow-up among IPS participants (N=90)^a

Predictor variable	OR	95% CI	pb
IPS+VR-JIT (reference: IPS as usual)	3.03	1.18c	.027
Baseline year	2.25	1.11–4.56	.024
Problematic substance use (reference: no)	3.58	1.03–12.44	.045
Social cognition score	1.06	1.02–1.10	.007
Community functioning score	.95	.91–.99	.010
Negative symptoms score	.76	.56–1.03	.075
	HR	95% CI	p b
IPS+VR-JIT (reference: IPS as usual)	1.84	.96c	.062
Baseline year	1.64	1.03–2.61	.039
Problematic substance use (reference: no)	2.37	1.05–5.32	.037
Social cognition score	1.03	1.01–1.05	.007
Community functioning score	.97	.94–.99	.016
Negative symptoms score	.83	.65–1.06	.136

^aParticipants received standard individual placement and support (IPS) services (N=36) or standard IPS services plus Virtual Reality Job Interview Training (VR-JIT) (N=54). The odds ratios are for the multivariable logistic regression. The hazard ratios (HR) are for the multivariable Cox proportional hazards model. Logistic regression model fit statistics, χ²=28.84, df=6, p=.001; Nagelkerke R²=.375. Cox proportional hazards model fit statistic, χ²=24.37, df=6, p<.001.

^bOne-sided p value for directional intervention hypothesis, two-sided p value for covariates and factors.

^cCI for directional intervention hypothesis uses only a lower-limit value.

Post Hoc Analyses of Primary Outcomes

Of the 90 participants, 46 (51%) met criteria for being IPS nonresponders–46% (N=25) of the IPS+VR-JIT group and 58% (N=21) of the IPS-as-usual group, which was not a statistically significant difference. By definition, IPS nonresponders have lower rates of obtaining employment when engaged in IPS, compared with new enrollees in IPS (37). As noted, the inclusion criteria for this study were modified from an initial focus on recent IPS enrollees to all unemployed IPS clients (i.e., including IPS nonresponders). This inclusion raises the question whether the effects of VR-JIT differed between the recent IPS enrollees and the nonresponders. Thus, we conducted a series of post hoc analyses to evaluate the impact of VR-JIT on employment outcomes separately for the IPS nonresponder subgroup and the recent IPS enrollee subgroup. The first set of analyses were for IPS nonresponders. A chi-square analysis among IPS nonresponders indicated that the IPS+VR-JIT group was significantly more likely than the IPS-as-usual group to obtain employment by 9-month follow-up (52% [N=13] versus 19.0% [N=4]; χ²=5.32, df=1, p=0.021). Subsequently, the addition of the same aforementioned covariates in a multivariable logistic regression within the IPS nonresponder subsample indicated that the IPS+VR-JIT group had greater odds of employment, compared with the IPS-as-usual group (OR=5.82, p=0.014; see table in online supplement). Finally, the adjusted Cox proportional hazards model among the IPS nonresponders (using the same covariates) indicated that the IPS+VR-JIT group had a significantly shorter time to employment, compared with the IPS-as-usual group (hazard ratio=2.70, p=0.044; see table in online supplement) (Figure 1).

FIGURE 1. Cumulative hazard plot of time to employment among IPS nonresponders, by study group^a.

^aTime to employment was measured as days from randomization to employment. Participants were randomly assigned to individual placement and support as usual (IPS as usual) or to IPS combined with Virtual Reality Job Interview Training (IPS+VR-JIT). Data were right-censored after 9 months (274 days), but no new employment occurred after day 235. A nonresponder was defined as a person who received IPS services for at least 90 days without obtaining a job or who obtained a job but lost it (was fired or quit) within 90 days.

The second set of analyses were for recent IPS enrollees. A chi-square analysis for recent IPS enrollees indicated that the IPS+VR-JIT group was not more likely to obtain employment by 9-month follow-up, compared with the IPS-as-usual group (34.5% [N=10] versus 40% [N=6]). Subsequently, the addition of the aforementioned covariates in a multivariable logistic regression for the recent IPS enrollee group indicated that the IPS+VR-JIT group did not differ regarding odds of employment, compared with the IPS-as-usual group (see table in online supplement). Finally, the adjusted Cox proportional hazards model for the recent IPS enrollees (using the aforementioned covariates) indicated that the IPS+VR-JIT group did not have a significantly shorter time to employment, compared with the IPS-as-usual group (see table and figure in online supplement). (The online supplement also includes a correlation matrix of the variables considered for the post hoc analyses among the IPS nonresponders and recent IPS enrollees.)

Secondary Outcomes

Longitudinal intent-to-treat analyses of secondary outcomes are shown in Table 4, including estimated marginal means for each group at pre- and posttest, results of the mixed-effects regression models (that included the same covariates from the primary analyses—baseline year of study participation, problematic substance use, social cognition, community functioning, and negative symptoms [anergia]), and longitudinal effect sizes (d) (66). Three models revealed significant group-by-time interactions that confirmed hypothesized greater improvements in interview skills (estimate±SE=1.62±0.62, p=0.006, d=0.33), interview self-confidence (estimate±SE=4.48±1.99, p=0.013, d=0.42), and decline in interview anxiety (estimate±SE=−8.21±3.90, p=0.019, d=−0.31) in the IPS+VR-JIT group, compared with the IPS-as-usual group. The model evaluating changes in general social competence did not reveal a significant group-by-time interaction.

TABLE 4.

Intent-to-treat change over time in interview skills, interview confidence, interview anxiety, and social competence associated with study condition

Variable and timeb	Estimated marginal score				Regression analysisa
	IPS+VR-JIT (N=54)		IPS as usual (N=36)
	M	SE	M	SE	Model parameter	Estimate	SE	pc	dd
Interview skillse									.33
Pretest	25.6	.6	24.6	.8	Group	.98	1.00	.331
Posttest	27.4	.6	24.8	.8	Time	.17	.47	.723
					Group*time	1.62	.62	.006
Interview self-confidencef									.42
Pretest	43.0	1.5	43.7	1.7	Group	−.68	2.23	.762
Posttest	47.6	1.6	43.8	1.8	Time	.17	1.50	.908
					Group*time	4.48	1.99	.013
Interview anxietyg									−.31
Pretest	100.2	3.6	90.5	4.5	Group	9.69	5.83	.100
Posttest	90.4	3.8	88.9	4.6	Time	−1.63	2.94	.582
					Group*time	−8.21	3.90	.019
Social competenceh									−.08
Pretest	3.71	.1	3.81	.1	Group	−.09	.13	.463
Posttest	3.76	.1	3.90	.1	Time	.10	.09	.294
					Group*time	−.05	.12	.692

^aMixed-effects linear regression models (N=90). For all models, covariates included were the same as in the primary analyses: baseline year, problematic substance use, social cognition, community functioning, and negative symptoms (anergia).

^bPretest, baseline (before randomization);posttest, approximately 3 months later.

^cOne-sided p value for directional intervention by time hypothesis for interview outcomes; two-sided p value for remaining effects.

^dLongitudinal effect size.

^eAs measured by the Mock Interview Rating Scale (25–27). Possible scores range from 8 to 40, with higher scores indicating better interview skills.

^fAs measured by self-reported interview confidence (25–27). Possible scores range from 9 to 63, with higher scores indicating better interview confidence.

^gAs measured by the adapted Personal Report of Public Speaking Apprehension (45). Possible scores range from 34 to 170, with higher scores indicating greater interview anxiety.

^hAs measured by the Social Skills Performance Assessment (46). Possible scores range from 1 to 4, with higher scores indicating better social competence.

DISCUSSION

Job interview skills are a critical contributor to employment (47). IPS supported employment is the most effective psychiatric rehabilitation approach to promote competitive work among individuals with serious mental illness (7). However, the impact of training IPS participants in interviewing skills has not been evaluated. VR-JIT was developed as an efficient approach to interview training by providing clients with the opportunity to practice their interviewing skills and obtain real-time feedback through a computer-based virtual platform. Our primary results were not definitive, because the unadjusted results were nonsignificant and statistically underpowered because of COVID-19 pandemic–related recruitment shortfall. However, adjusting the model for covariates that were theoretically and empirically associated with employment indicated that participants randomly assigned to IPS+VR-JIT were significantly more likely than those in the IPS-as-usual group to obtain employment within the 9-month study period. A similar pattern was noted for the Cox proportional hazards model, in which the unadjusted analysis yielded a nonsignificant effect, whereas the adjusted results indicated a trend that IPS+VR-JIT participants had a shorter time to employment, compared with IPS-as-usual participants. Overall, these results are consistent with prior efficacy trials of VR-JIT among individuals with serious mental illness who were not engaged in vocational rehabilitation (26, 27, 29).

We know that approximately 45% of IPS clients are unable to obtain employment (i.e., IPS nonresponders) (7) and that additional supports or tools may improve employment outcomes for this group. Our post hoc analyses suggested that IPS nonresponders who received VR-JIT had significantly greater odds of employment and reduced time to employment (Figure 1), compared with nonresponders in the IPS-as-usual group. The results demonstrating that IPS nonresponders seem to benefit from VR-JIT are consistent with other lines of research also suggesting that individuals who do not initially benefit from IPS may be able to secure employment with adjunct services integrated within IPS (e.g., cognitive remediation) (37). Overall, our change in eligibility criteria enabled us to detect that VR-JIT may enhance employment outcomes among IPS nonresponders while concurrently providing insight that not all IPS clients may benefit from additional interview training. Notably, the effects of VR-JIT on IPS nonresponders was not hypothesized and requires replication.

We also found that VR-JIT facilitated improved interview skills, interview self-confidence, and interview anxiety for participants, which is consistent with prior trials evaluating VR-JIT among adults with serious mental illness (25–31) and an adapted VR-JIT among autistic transition-age youths (50). These findings have important implications, because self-confidence is known to improve interview performance (68), and high levels of anxiety can disrupt interview performance (16, 69, 70). Although IPS+VR-JIT participants improved their interviewing skills, no improvement was shown for general social competence, which is consistent with research and clinical recommendations that social skills training may improve specific targeted areas of social functioning but has limited generalizability to other areas (71).

Implications for Practice

Given the focus on rapid job search in IPS, employment specialists may or may not work with clients on interview skills preceding real-world job interviews. Further, there is limited information from prior studies about how many IPS clients practice job interview role plays. Notably, we observed that approximately 47% (N=17) of participants in the IPS-as-usual group completed 1.861.2 practice job interviews with their employment specialists. To the best of our knowledge, this is the first study to report on the frequency of mock interviews occurring within IPS. That said, employment specialists are not currently trained in established social skills methods for teaching interview skills. With VR-JIT’s use of behavioral learning principles, including scaffolding and performance feedback, the intervention’s pedagogy is consistent with evidence-based social skills training approaches (71).

As noted in this study, engagement with VR-JIT combined with high-fidelity IPS significantly reduced time to obtaining competitive employment for IPS nonresponders. Further, IPS employment specialists spent 38.8°41.9 minutes discussing virtual interviewing progress with IPS+VR-JIT participants, indicating that employment specialists could naturally integrate VR-JIT into the IPS service structure. Thus, the enhancement of IPS with VR-JIT addresses an area of need—namely, poor interview skills, which could be contributing to a diminished response to IPS for some clients. Improved interview skills could serve to bolster work outcomes, not only in IPS but also in a variety of highly utilized vocational rehabilitation services in the United States and internationally.

Because a large proportion of participants engaged in IPS as usual completed job interview role plays with their employment specialists, providing employment specialists with evidence-based tools to facilitate and standardize this training could bolster their practices. Thus, VR-JIT could potentially offer employment specialists a tool to assess and guide interview skills development for IPS nonresponders without affecting IPS fidelity, serving to enhance the impact of IPS in much the same way as motivational interviewing and cognitive remediation strategies. For example, employment specialists could train clients to use VR-JIT independently and then process transcripts and summary performance assessments with them. Alternatively, employment specialists could review clients’ VR-JIT performance assessments and transcripts prior to sessions with clients and then use the sessions to reinforce their clients’ interviewing strengths and focus in greater depth on interview skills that need improvement.

Although the data are preliminary and future studies are needed, this more naturalistic integration of VR-JIT with employment specialist support generated the best employment rates among all study participants: mock interviewing with the employment specialist plus VR-JIT, 56% (N=9); mock interviewing with the employment specialist alone, 40% (N=15); VR-JIT alone, 40% (N=43); and no mock interviewing with the employment specialist or VR-JIT, 24% (N=17). Finally, Internet-based programs, such as VR-JIT, provide IPS teams and individuals with serious mental illness flexible access and use of tools that can facilitate practice at their convenience and pace and could facilitate engagement in IPS for individuals with serious mental illness who face unexpected barriers to face-to-face meetings (e.g., inclement weather and transportation barriers), although future research is needed to examine potential barriers more carefully.

Limitations

The findings must be considered within the study limitations. First, the observed employment rates in both groups were lower than anticipated when considering other studies evaluating high-fidelity IPS (6). These lower-than-anticipated rates can potentially be explained by two factors. The first was that 51% (N=46) of the main sample included IPS nonresponders (i.e., individuals who did not benefit from IPS). The second was that 49% of the main sample were recent IPS enrollees, and one could estimate that 45% of these participants would become IPS nonresponders during the course of the study (based on historical IPS employment rates) (7). Second, overall recruitment challenges and the COVID-19 pandemic contributed to a smaller sample and significantly lower analytical power than anticipated. Notably, 24% of our sample (N=22) had at least 1 month of follow-up during the COVID-19-imposed quarantine period (March 2020 through October 2020). That said, 50% (N=11) of these participants obtained employment, which was consistent with the 45% employment rate at Thresholds during the quarantine. Thus, the pandemic appears to have had minimal impact on job attainment in this study.

A third limitation is that employment specialists were asked to refrain from mock interview training with IPS+VR-JIT participants and were not masked to study condition. This design may have facilitated bias in how the employment specialists treated participants in both groups. That said, we did not see a difference between groups with respect to the extent that employment specialists discussed job interview skills with the participants. This lack of a difference could reflect minimal bias or a potential increase in job interview skill discussion in the IPS-as-usual group. Fourth, findings regarding interview confidence and anxiety were limited to self-report. Finally, VR-JIT was primarily implemented by researchers, which may limit our understanding of its effectiveness when implemented by IPS staff. However, VR-JIT was delivered by using strategies and procedures recommended by IPS program administrators and in settings where participants typically met with their employment specialists to mirror real-world delivery.

Future Directions

Additional research is needed to help understand why VR-JIT may be effective in helping IPS nonresponders. For instance, research is needed to evaluate differential effectiveness of VR-JIT among IPS nonresponders and its mechanisms of enhanced interview skills and employment. Also, although VR-JIT was primarily implemented in the community by researchers, our hybrid type I design included an initial process evaluation of VR-JIT implementation feasibility, acceptability, and usability and potential barriers to future implementation—and that report is forthcoming. Meanwhile, a future multilevel implementation evaluation of VR-JIT is needed to assess how VR-JIT is delivered by IPS teams to IPS nonresponders while maintaining the core functions of high-fidelity IPS and how VR-JIT fits into the IPS workflow and progress note for services billing to further enhance and sustain implementation. Finally, agencies and interested implementers would benefit from understanding the cost-effectiveness of VR-JIT.

CONCLUSIONS

Vocational rehabilitation in general could benefit from an evidence-based practice targeting job interview skills for individuals with serious mental illness. VR-JIT appears to be a valid option to fill this service gap, given that the combination of high-fidelity IPS and VR-JIT was associated with significantly better employment outcomes among individuals with a history of being nonresponsive to IPS. In addition, VR-JIT trainees more broadly had significantly improved interview skills, interview confidence, and interview anxiety, which speaks to its effectiveness targeting these outcomes. Although more research is needed to understand why VR-JIT is effective and to identify optimal implementation strategies, these results may support the use of VR-JIT in current IPS programs in the United States and internationally.

HIGHLIGHTS.

• Virtual Reality Job Interview Training (VR-JIT) improved job interview skills, interview anxiety, and interview confidence among clients of an individual placement and support (IPS) supported employment program.

• Among IPS clients who were not able to attain employment in 90 days (IPS nonresponders), the group that received IPS plus VR-JIT had a significantly higher employment rate (52% versus 19%) and reduced time to employment, compared with the IPS-as-usual group.

• VR-JIT could offer employment specialists an effective tool to assess and guide interview skills development for IPS nonresponders.