Mechanism of action for obtaining job offers with virtual reality job interview training

Abstract

Objective

Four randomized controlled trials (RCTs) revealed that virtual reality job interview training (VR-JIT) improved interviewing skills and the odds of obtaining a job offer among trainees with severe mental illness or autism spectrum disorder. This study assessed whether post-intervention interviewing skills mediated the relationship between virtual interview trial completion and receiving job offers by 6-month follow-up.

Methods

VR-JIT trainees (n=79) completed pre- and post-test role-plays and a brief survey approximately 6 months later to assess if trainees received a job offer.

Results

As hypothesized, analyses indicated that number of completed virtual interviews predicted greater post-test interviewing skills (β=.20, 95% PCI=.08–.33), which in turn predicted trainee’s obtaining a job offer (β=.28, 95% PCI=.01–.53).

Conclusions

Results suggest that VR-JIT may provide a mechanism of action that helps trainees across various psychiatric diagnoses obtain job offers in the community. Future research can evaluate the community-based effectiveness of this novel intervention.

Introduction

Virtual reality interventions are emerging as evidence-based and highly scalable approaches to enhance skills and outcomes among psychiatric populations. Blinded.Company developed Virtual Reality Job Interview Training (VR-JIT) using recommendations from vocational rehabilitation experts (1) and a theoretical model of job interviewing (2) (Blinded.Website). Moreover, VR-JIT integrates neuroscience-based learning principles that include repeated practice, hierarchical learning across progressive degrees of difficulty, and a reward system to reinforce behavioral change (3).

We conducted four RCTs to evaluate the efficacy of VR-JIT at improving interviewing skills for individuals with psychiatric disorders (e.g., depression, bipolar disorder, schizophrenia, autism spectrum disorder (ASD), or Veterans with posttraumatic stress disorder (PTSD)). We found that completing VR-JIT improved interview performance for trainees between pre-test and post-test for each cohort (4–7). During 6-month follow-up, we found trainees in each cohort had greater odds than controls of receiving a competitive job offer (5, 8) or offer for a competitive volunteer position or job (9). Moreover, we observed a VR-JIT dose effect in two cohorts: completing more virtual interviews was associated with greater odds of getting a job offer (8).

While we hypothesize that more training led to better interviewing skills that resulted in more job offers, it is also possible that the association between training and job offers could be explained by other non-specific mechanisms (e.g., people who persisted in training also persisted in job searching). Understanding the relationship between interviewing skills and job offers in a psychiatric rehabilitation sample is of interest because a thorough literature review found no previous study demonstrating this connection. Job interview training is widely believed to be important for job seekers, which is seemingly common sense. Yet, there are no scientific reports on interviewing as a mechanism of employment.

Mediation analyses have lower statistical power than testing for main effects, so to increase power, we combined the data from four cohorts to assess whether improved interviewing skills among trainees is a mechanism for obtaining a job offer. Our primary hypothesis is post-test interviewing performance will mediate the relationship between the number of completed virtual interviews and receiving a job offer. We also examined whether there was evidence of heterogeneity in the mediated effect across trials using a novel method.

Methods

Participants included 79 individuals (or Veterans) with a diagnosed mood disorder (depression or bipolar disorder), PTSD, schizophrenia or ASD who were assigned to the training group in four RCTs evaluating VR-JIT from January 2012–May 2014. There were n=119 participants across the four studies (n=40 controls). Participants were 43.1±14.8 years old, 56 (71%) were male, 47 (59%) were African-American, 26 (33%) were Caucasian, and 6 (8%) were Latino, Asian, or Biracial. Participants’ mean parental years of education were 13.5±3.1, which served as a proxy of socioeconomic status. Additionally, 8 (10%) participants were never employed or underemployed, 14 (18%) were unemployed 6 months or less, 11 (14%) were unemployed 7–23 months, 32 (40%) were unemployed 24–59 months, and 14 (18%) were unemployed 60+ months.

Mental health and ASD diagnoses were confirmed with the structured clinical interview for the DSM-IV, Mini-International Neuropsychiatric Interview, Social Responsivity Scale 2^nd Edition, and clinical records. All participants were clinically stable. Inclusion criteria included: 18–55 years old; minimum 6^th grade reading level (via Wide Range Achievement Test–IV); willingness to be video-recorded; being unemployed or underemployed; and actively seeking employment. Exclusion criteria included having a medical illness that compromised cognition (e.g., traumatic brain injury), uncorrected vision or hearing problems, or active substance abuse.

We randomized participants (2:1) into the training or control groups. Measures collected at pre-test, post-test, and 6-month follow-up are described below. After 6 months, we approached participants to complete a brief survey in-person, over-the-phone, or via email. All participants provided informed consent and the Blinded.University’s Institutional Review Board approved the protocol. Additional methodological details can be found here (5–7).

VR-JIT is based on 8 learning goals that fit within Huffcutt’s theoretical framework (2) for successful job interviewing that emphasizes job relevant interview content (i.e., conveying oneself as a hard worker, easy to work with, professional, negotiation skills) and interview performance (i.e., sharing things positively, sounding honest, sounding interested in the job, and comfort level) (4, 6, 7). Trainees learned about interviewing skills from online didactics and by practicing these skills during their interactions with Molly Porter, the virtual human resources agent from Wondersmart (a fictional store). VR-JIT scores each virtual interview (0–100) on how well the trainees performed on the 8 learning goals. Trainees also received in-the-moment rewarding feedback on their response’s appropriateness from a virtual job coach. After each virtual interview, trainees reviewed their scores and received automated positive reinforcement and constructive feedback on their performance for each learning goal. Lastly, trainees needed to score 90 or better three times to progress from an easy difficulty level to a medium, and then hard difficulty level. After 5 attempts, trainees automatically advanced to the next stage of difficulty. Training occurred during 5 visits across 5–10 business days.

Research staff recorded the number of completed virtual interviews with Molly. Each trainee received a count of the number of completed virtual interviews as a proxy for their dose of training. Trainees completed 14.8±3.5 virtual interviews.

Participants completed two mock job interviews with a professional actor at pre-test and post-test during the RCTs. The post-test measure is evaluated as the mediator for this study. We assessed performance across nine domains that overlap with the above 8 learning goals, with the ninth domain assessing overall rapport with interviewer. The videos were randomly assigned to two raters, blind to condition, with more than 15 years of experience in human resources. Total scores were computed using 9 domains for each video (scores of 1–5 per domain), and averaged to provide a single score for each time point. Additional methodology (including scoring manuals) on this measure can be found elsewhere (6, 7).

We evaluated neurocognition, measured by the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) total score, and the self-reported number of months since prior employment as covariates as they are associated with vocational outcomes (10, 11). We also evaluated diagnosis as a covariate given the different diagnoses among trainees.

At 6-month follow-up, we surveyed participants and asked them if they received a job offer since completing the study.

We conducted a mediational path model with Mplus 7.2 using a Bayesian estimator because of its robust performance under conditions of smaller sample sizes compared with maximum likelihood estimation and the Sobel Test. Model fit for Bayesian estimation uses the posterior predictive checking method; a posterior predictive p-value (PPP) above .05 indicates a good fit. To assess the statistical significance of mediation using the Bayes posterior credible interval (PCI), the specified indirect effect is significant if the PCI does not contain zero (12). We also examined heterogeneity in mediation across trials (13). Due to the limit on the number of references in brief reports, citations for our assessments and tools discussed in this section are available in an online supplement.

Results

Our mediation model provided a good fit to the data: (PPP=.47) when we accounted for pre-test interview skills (correlated at r=.82, p<.001 with post-test interview skills). Diagnosis, cognition, and months since prior employment were removed as covariates as they did not explain significant variation in the model (all p>.10). We observed a significant mediation effect whereby the number of completed virtual interviews predicted post-test interviewing skills, which in turn predicted receiving a job offer by 6-month follow-up (Product-of-coefficients=.02, SD=.01, p<.05; 95% PCI:.01–.04) (Figure 1). The relationship between number of virtual interviews completed and job offer attainment was fully mediated as indicated by the direct effect between completed interviews and job offer changing from significant (B=.09, p<.05; SD=.046, β=.30, 95% PCI:.01–.54) to non-significant (Path C) in the mediation model (Figure 1). No evidence of heterogeneity across trials was found. We conducted a second mediation analysis while accounting for potential heterogeneity of the meditational paths when combining data from multiple trials (13). The results were very similar (95% CI:–.01–.05) and add to our confidence in a mediated effect.

Figure 1. Mediation analysis of interviewing skills as a mechanism of obtaining a job offer.

Path A: B = 2.66**, SD = .85, β = .20 (95% PCI: .08–.33) Path C: B = .07, SD = .05, β = .23 (95% PCI: −.06–.49) Path B: B = .01*. SD = .01, β = .28 (95% PCI: .01–.53) Path D: B = .68***, SD = .06, β = .79 (95% PCI: .69–.86) Product of coefficients (A * B): B = .02*. SD = .01, β = .02 (95% PCI: .01–.04)

Note. *p < .05. **p < .01. ***p < .001. P-values are two-tailed. B = unstandardized beta; SD = posterior SD; β = standardized beta; PCI = posterior credible interval.

Discussion

Standardized vocational rehabilitation programs such as Individual Placement and Support use several mechanisms (e.g., job development) to help clients obtain employment (14). We conducted a mediation analysis to evaluate if VR-JIT provides evidence of one of these mechanisms, job interview training. The results support our hypothesis that job interviewing skills fully mediated the relationship between completed virtual interviews and obtaining a job offer. Specifically, we observed that performing more virtual interviews enhanced job interviewing skills, which predicted a greater likelihood of receiving a job offer. Although we did not observe diagnosis as a significant covariate, a moderated mediation in a larger sample could test if the observed mediation varies across diagnosis.

Although prior studies suggested VR-JIT trainees from individual cohorts had greater odds of receiving a job offer (trainees across all four cohorts have increased odds of receiving a job offer, see online supplement for details), the current study identified the relationship between the number of completed virtual interviews and improved interviewing skills as the mechanism for getting the job offer. To our knowledge, this is the first study to confirm that job interview training is related to getting job offers. Although we did not evaluate a large enough sample to determine the optimal dose of interview training, we did find that 3–5 sessions was sufficient for most trainees. VR-JIT provides scores, feedback and hierarchical training so that dose can be adjusted depending on individual performance. Perhaps some trainees maximize their benefit with fewer hours of training, while others may need more training to acquire the same skill level.

The results should be interpreted while considering some limitations. The trials were conducted in a laboratory setting; hence, the findings may not generalize to community settings. Also, the samples were small and did not include individuals at an early onset stage of their disorder. Participants needed at least a 6^th grade reading level to use VR-JIT so the findings may not generalize to individuals with reading ability below 6^th grade. Lastly, several other mechanisms of improved skill and outcome (e.g., motivation and anxiety) were identified as barriers to employment (15), but were not measured in these trials. However, future research can evaluate these factors as possible mediators of the relationship between training and receiving a job offer.

Conclusions

VR-JIT improves interviewing skills that lead to receiving a competitive job offer. Future research is needed to evaluate the community-based effectiveness of VR-JIT. Further, evaluating the implementation of VR-JIT in real-world systems, specifically its feasibility, acceptability, and cost-effectiveness, is a necessary step in determining scalability. When implementing under real-world conditions, it will be important to evaluate barriers and facilitators of implementation, and determine how to maintain high participant engagement as achieving a high rate of trial completion relates directly and indirectly to obtaining job offers and securing employment.