Virtual Reality Job Interview Training for Adults Receiving Prison-Based Employment Services: A Randomized Controlled Feasibility and Initial Effectiveness Trial

Abstract

Returning citizens struggle to obtain employment after release from prison, and navigating job interviews is a critical barrier they encounter. Implementing evidence-based interview training is a major gap in prison-based vocational services. We conducted a randomized controlled trial (RCT) to evaluate the feasibility and initial effectiveness of Virtual Reality Job Interview Training within two prisons. Forty-four male returning citizens were randomized to receive service-as-usual (SAU) with VR-JIT (SAU+VR-JIT, n = 28) or SAU (n = 16). Participants reported VR-JIT was highly acceptable and usable. SAU+VR-JIT, as compared to SAU, had significant improvements (with large effect sizes) in interview skills, interview training motivation, and interview anxiety (all p < .05; ηp² > .15), and greater employment by 6-month follow-up (OR = 7.4, p = .045). VR-JIT can potentially help fill a major gap in prison-based services. Future research is needed to validate VR-JIT effectiveness and evaluate VR-JIT implementation strategies within prisons.

Introduction

Every year over 700,000 returning citizens are released from state or federal facilities in the United States (United States Department of Justice, 2021). The Bureau of Justice Statistics reports 44% of people released from state prisons across 30 states were rearrested within the first year (Alper & Durose, 2018). These data suggest justice facilities need to do more to provide rehabilitation and ensure successful reentry into communities. One area of rehabilitation that deserves more attention is employment readiness. Unemployment is among the eight notable criminogenic risk factors related to recidivism (Andrews & Bonta, 2017; Visher et al., 2017).

Gainful employment addresses many criminogenic risk factors by enabling returning citizens to network within their communities, secure housing, and pay their rent and utilities (Petersilia, 2005; Visher & Courtney, 2006). Despite evidence suggesting that employment reduces recidivism, research suggests current employment ranges from 28% to 45% within one to four years after release (Seim & Harding, 2020; Visher et al., 2011). Furthermore, the employment rate was significantly lower among African American as compared to White returning citizens (Seim & Harding, 2020; Visher et al., 2011). Given the increased risk for unemployed returning citizens to recidivate, developing services to increase employment at reentry is a public safety priority.

A meta-analysis of prison-based education, vocation, and work programs found that returning citizens who engage in prison-based educational or vocational services have higher employment rates and are less likely to recidivate upon release (Wilson et al., 2000). However, only half of all state facilities have vocational services (Stephan, 2008). Job interview preparation is one of the most common vocational services provided by state facilities as decades of research suggest job interview skills influence employment in the general population (Macan, 2009). However, despite researchers attempting to develop job interview skills training for returning citizens since the 1970s (e.g., Twentyman et al., 1978), the job interview remains a critical barrier to employment among returning citizens as the field still lacks an evidence-based practice that enhances job interview skills (Flake, 2015). Hence, job interview preparation is highlighted as a critical need by the United States National Institute of Justice (Wells, 2014).

Relatedly, a job acquisition framework suggests that job preparation activities (e.g., job interview training) reduces anxiety which in turn enhances access to employment (Corbiere et al., 2011). This framework is especially relevant for returning citizens who lack adequate interview skills to successfully navigate job interviews (e.g., Pham et al., 2017; Ricciardelli & Mooney, 2018). Furthermore, interviewing for a job provokes anxiety in most people (McCarthy & Goffin, 2004), which is exacerbated for returning citizens who need to effectively discuss their prior conviction but lack the skills to do so (Ricciardelli & Mooney, 2018). Thus, stronger interview skills would likely reduce the anxiety surrounding a job interview, as anxiety itself, can disrupt answering interview questions due to memory lapse, stuttering, and other physical distractions (e.g., excessive sweating) (McCarthy & Goffin, 2004).

Moreover, returning citizens with greater motivation to engage in job preparation have stronger employment outcomes (Hunter & Boyce, 2009). Thus, it is critical that job preparation activities such as job interview training are perceived as acceptable in order to optimize motivation to engage with them. That said, job interview preparation activities typically occur when an educator helps develop job interview skills through mock interviews and class-based didactics. Such mock interviews occur in nearly all vocational programs whether in prisons, mental health services, or special education so that service recipients are better prepared to answer questions regarding their work skills and ability to work hard (Hurtz & Donovan, 2000).

Although mock job interview training is commonly offered by vocational services, few studies have evaluated the effectiveness of these methods when delivered within prisons (Ellison et al., 2017; Newton et al., 2018). Meanwhile, clients receiving community-based vocational services are typically limited to vocational staff conducting one or two mock interviews (due to large caseloads and time required to conduct them) before a client’s real-life interview. Thus, clients may need additional training to practice interviewing and enhance their interview skills. Moreover, agencies seldom train vocational staff on how to conduct mock interview training sessions. For example, vocational staff may not consistently provide feedback to clients regarding their interview responses, interviewing anxiety, or confidence about interviewing. Additionally, vocational staff have existing relationships with clients that likely buffers the feelings of anxiety one has when interviewing with a hiring manager in the community.

Recently, the criminal justice system has advanced the integration of technology (e.g., computer simulated learning, video games) into their practices to alleviate the challenges faced with training and supervising staff and returning citizens (Ticknor, 2019; Cornet & Van Gelder, 2020). Notably, virtual reality (i.e., a 2D or 3D virtual environment accessed using a computing device [non-immersive] or head-mounted display [immersive] (Paes et al., 2017)) has emerged as a versatile technology that can address several novel aspects of correctional services (e.g., substance abuse, aggression regulation, risk assessment) (Ticknor, 2019; Cornet & Van Gelder, 2020). For example, virtual reality has been effective at treating anxiety in clinics and has recently been piloted to treat anxiety within prisons (Peters, 2018). Meanwhile, research suggests virtual reality may be feasible for delivering cognitive behavioral therapy to treat the cognitive distortions of returning citizens (Ticknor, 2019). Moreover, multiple technology-based job interview simulators have emerged to try and fill a gap in evidence-based practice by helping adults with psychiatric and developmental disabilities prepare for job interviews (e.g., Burke et al., 2018 Strickland et al., 2013; Smith, Ginger, Wright, Wright, Humm et al., 2014; Smith, Ginger, Wright, Wright, Taylor et al., 2014). However, most have been limited to small pilot or non-controlled studies. Meanwhile, the non-immersive, Virtual Reality Job Interview Training (VR-JIT) and its adaptation for youth have emerged from these initial studies with the strongest empirical evidence to date (Smith, Sherwood et al., 2021; Smith et al., in press).

Specifically, VR-JIT is a computerized job interview simulator that targets interview skills using SIMmersion LLC’s (www.simmersion.com) proprietary PeopleSim^® technology. PeopleSim^® controls the personality of the virtual hiring manager, Molly Porter, with an algorithm that uses probabilities to facilitate Molly’s emotions, questions she asks, and responses to her questions. The result is that Molly comes across to trainees as realistic, authentic, and engaging during an experience that presents as if you are being interviewed over a remote meeting platform (e.g., Zoom) (Smith, Ginger, Wright, Wright, Humm et al., 2014). This approach creates strong social presence (i.e., sense of being with another person [Aragon, 2003]) for the Molly character that also aligns with best practices for immersive high-fidelity simulations (Motola et al, 2013).

The efficacy of VR-JIT was demonstrated in a series of five randomized controlled trials (RCT) in lab settings, where the VR-JIT groups, as compared to control groups, improved their interview skills and interview confidence, and increased their access to employment (e.g., OR=9.6, OR=8.7; Smith, Fleming, Wright, Jordan et al., 2015; Smith, Fleming, Wright, Roberts et al., 2015). This efficacy was demonstrated among multiple vulnerable adult populations, including individuals with substance abuse histories, serious mental illness, and veterans with PTSD (Smith et al., 2016; Smith, Ginger, Wright, Wright, Humm et al., 2014; Smith, Humm et al., 2015). Additionally, two recent RCTs demonstrated the real-world effectiveness of VR-JIT at improving job interview skills, reducing job interview anxiety, and increasing access to employment. The first study was among adults with serious mental illness in the IPS model of supported employment (OR=2.7; Smith et al., in press). The second study was among transition age youth with autism using an adapted VR-JIT in special education pre-employment transition services (OR=12.4; Smith, Sherwood et al., 2021). However, research has not yet evaluated the potential effectiveness of VR-JIT implemented in the prison system to support returning citizens as they prepare to re-enter their communities.

Based on the Proctor et al. (2011) implementation outcome taxonomy (i.e., a framework to bring consistency to implementation science research domains), the current study tested the feasibility and initial effectiveness of VR-JIT when implemented by local staff in prison-based vocational services. Regarding feasibility, we hypothesized that participants will perceive VR-JIT as acceptable and usable, and they will actively engage the intervention. Regarding initial effectiveness, we hypothesized that trainees receiving VR-JIT in addition to employment services-as-usual (SAU), compared to participants receiving SAU only, would demonstrate improved interview skills, interview training motivation, interview anxiety, and employment rates by a 6-month follow-up. Lastly, we explored the correlates of VR-JIT performance and effectiveness outcomes to inform a future validation trial.

Methods

Trial Design

This randomized controlled trial evaluated SAU+VR-JIT compared to a SAU group at an unbalanced randomized ratio of 2:1, which was done to gather sufficient feasibility data. There were no changes to the methods after the trial commenced. Aligned with best practices of community-based participatory research (Newman et al., 2011), we convened a stakeholder advisory board that included criminal justice scientists, state-level corrections assistant education manager, and a reentry coordinator for a local county sheriff’s office who was a returning citizen. This advisory board was instrumental in developing the overall study design.

Participants

Study participants (ages 26 to 58) were recruited from vocational rehabilitation programs in two prisons (1 minimum security, 1 medium security). Employment readiness counselors at the prisons (that we reference in this paper as prison staff) used study inclusion criteria to pre-screen potential participants who were then invited to a recruitment presentation by research staff. Inclusion criteria included the following: 1) within 3 months of their earliest possible release date, 2) high school diploma or equivalency, or 3) at moderate-to-high risk for reoffending violent crimes (assessed by the Michigan Department of Corrections (MDOC) via the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS; Brennan et al., 2009). Exclusion criteria included the following: 1) the presence of an uncorrected hearing or visual problem that interfered with using VR-JIT, and 2) a medical illness that compromised cognition (e.g., moderate-to-severe traumatic brain injury). The University of Michigan’s Institutional Review Board approved the study protocol, and all participants provided informed consent. The study occurred from September 2019 to June 2020, with follow-up completed by January 2021. Data were remotely collected after prison access was restricted in March 2020 due to the coronavirus pandemic.

Sample Size and Blinding

For this feasibility RCT, a sample size of 60 was determined by budget limitations and reviewed and approved by the study’s data safety and monitoring board (DSMB). A random number generator determined the randomly generated 8 blocks of 9 participants at a 2:1 ratio (n = 6 SAU+VR-JIT to n = 3 SAU only). This block size of 9 was chosen because the prison computer labs housed 6 computers that could be used for VR-JIT. As noted in our CONSORT (Figure 1), 48 participants met inclusion criteria, provided informed consent, and completed pre-test study measures. Given that 4 participants dropped out prior to randomization, we randomized 44 participants to the SAU+VR-JIT (n = 28) and SAU only (n = 16) groups. The DSMB approved the termination of trial enrollment after 44 participants completed the study protocol due to the stoppage of VR-JIT delivery and employment services-as-usual because of the coronavirus pandemic, and after reviewing the study’s feasibility for recruiting and retaining participants in the trial. Twenty-three participants were enrolled at the minimum security prison and n=21 were enrolled at the medium security prison. Participants and VR-JIT implementers (i.e., prison staff) were not blinded to group assignment due to the real-world implementation design.

Figure 1.

CONSORT Diagram

Interventions

Services as usual.

The MDOC Vocational Villages are separate residential programs within the prisons where returning citizens live, work, and study together prior to their release. Returning citizens must meet a high standard of behavioral and academic milestones (e.g., at least 6 months free of Class I misconduct [such as possession of a weapon or assault and battery], must have completed all core and academic programming) for entry into the Vocational Villages. The main components of the Vocational Village curriculum are the 13 vocational trade training programs (e.g., CDL/forklift, carpentry, automotive) that returning citizens can access. Specifically, returning citizens select one primary vocational trade where they earn legitimate trade credentials through coursework and hands-on training that are transferable to the workforce upon release. Returning citizens completed a 15-hour pre-employment preparation workshop designed to enhance employability skills related to the job search, completing job applications, cover letters, resume writing, and job interviewing (Washington, 2018). Returning citizens self-reported they completed approximately 1 to 2 mock interviews (M = 1.22, SD = 1.95) with employment readiness instructors or peers based on their need and an assessment of their skill. One participant completed an outlier of 36 mock interviews (data not included). Data were only available for 24 of 44 participants due to participant attrition (n = 11) and an administrative delay (n = 9) to include the measure. All participants completed 1–2 hours of classroom instruction on interview skills led by employment readiness instructors. Additional Vocational Village details are here: (https://www.michigan.gov/corrections/0,4551,7-119-33218_75514---,00.html).

Virtual reality job interview training.

As noted in the introduction, VR-JIT is a computerized job interview simulator (see Figure 2) and is delivered via the internet (or locally installed, no head mount required). Trainees interview with the hiring manager Molly via speech recognition software across three levels of difficulty (i.e., easy, medium, and hard) that display her different personalities (i.e., friendly, professional, inappropriate). Prior to interviewing with Molly, trainees review an e-learning curriculum that introduces eight job interview skills based on the literature (e.g., hard worker, sounds easy to work with; Huffcutt, 2011). Additionally, VR-JIT helps trainees practice disclosing a prior conviction in a judgement-free environment.

Figure 2.

Interface for Virtual Reality Job Interview Training

During the virtual interviews, trainees receive four levels of feedback: a) real-time non-verbal cues from an on-screen coach (see Figure 2), b) a real-time and a post-interview transcript featuring feedback on each statement that trainees speak to the virtual hiring manager, c) a post-interview performance assessment on how well they addressed each of the eight job interview skills highlighted in the e-learning, and d) a numerical score from 0 to 100 (see Supplemental Figures 1 - 3). A more detailed description of VR-JIT is here (Smith, Ginger, Wright, Wright, Humm, et al., 2014).

Study Procedures

Participants attended pre-test visit 1 where they completed self-report surveys about their background, employment, and criminal justice (i.e., times arrested, duration in jail or prison) histories. During pre-test visit 2 participants completed a mock job interview with a research team member that was recorded on video, followed by a series of self-report measures to assess their job interview skills, motivation, and anxiety. At the end of pre-test visit 2, participants were randomly assigned to SAU+VR-JIT or SAU. At the post-test visit, all participants completed the same self-reports and mock interview. Also, SAU+VR-JIT self-reported their perceptions of VR-JIT acceptability and usability. Due to the coronavirus pandemic, some participants completed their pre-test (n = 4) and post-test (n = 10) mock interviews over the phone.

All prison-based study visits were completed in a semi-private space with a closed door that had a window, and prison staff were in the next room to meet minimal security standards. In conjunction with the pre-test visits, the research team accessed a public MDOC records database (https://mdocweb.state.mi.us/otis2/otis2.aspx) to collect criminal justice history for each participant (i.e., MDOC number, earliest release date, and offense type (e.g., violent or not)). The study team conducted a 6-month post-release follow-up phone interview with participants to establish the feasibility of following participants in order to obtain their employment outcomes. Participants were sent postcard reminders at 3- and 5-months post-release before their 6-month follow-up date. If participants were non-responsive via phone and email after 4 weeks, the study team terminated contact efforts. Concurrently, post-release employment records were collected from MDOC. At the time of release, all participants received a copy of their consent form and an estimated date for their 6-month follow-up phone interview.

Research Visit Training Fidelity

To prepare for data collection at two prisons, the research team completed approximately 10 hours of training: they reviewed study assessments, practiced delivery, and completed at least two mock study visits during which their performance was evaluated by a master’s level research supervisor. Staff members completed approximately 1–2 mock study visits prior to subsequent data collection to prevent drift. To facilitate adherence to research visit fidelity, research staff used a checklist to complete all study visit procedures.

VR-JIT Orientation Training for Research and Prison Staff

Research staff were trained to prepare prison-based employment readiness counselors to implement VR-JIT for returning citizens. The first author led the research team’s VR-JIT orientation using the VR-JIT implementation fidelity checklist that identified each component of the intervention. Then research staff engaged VR-JIT to learn to navigate its interface and completed three virtual interviews. Next, research staff role-played teaching VR-JIT to one another using the fidelity checklist. Concurrently, the first author observed the role-plays and used the checklist to monitor the fidelity of the role-played orientation. If required, feedback was provided, and additional role-plays were conducted until research staff felt prepared to conduct an orientation session with the prison employment readiness counselors. Research staff then followed the aforementioned procedures to prepare the employment readiness counselors to orient study participants on how to use VR-JIT.

VR-JIT Implementation Procedures

After the completion of pre-test visits, one employment readiness counselor at each prison led VR-JIT implementation. During session one (approximately 60 minutes), the counselor used the aforementioned fidelity checklist (monitored by research staff) to teach the various components of VR-JIT. Then the SAU+VR-JIT group practiced using the e-learning, job application, and virtual interviewing components. The two-prison staff led a total of seven cohorts of VR-JIT implementation that ranged from four to nine VR-JIT trainees per cohort. An 8^th cohort was randomized to VR-JIT but never used VR-JIT due to the coronavirus pandemic.

We assessed adherence to the VR-JIT implementation protocol at the staff and participant levels. Regarding staff-level adherence, employment readiness counselors completed all required VR-JIT orientation training requirements. Also, the research team supervised the fidelity of the first two implementation sessions at each prison, which obtained 100% fidelity. Then one prison staff implemented VR-JIT independently for three additional cohorts and provided recordings of these orientations. The staff member’s fidelity ratings (in order) were 95% then 75% then 60%. The staff was retrained due to a drop in fidelity to an unacceptable rating.

Regarding participant-level adherence, we monitored the frequency of participants who progressed from easy-to-medium-to-hard interviews and completed at least four hard interviews. Notably, five of 28 (17.9%) participants randomized to SAU+VR-JIT did not use VR-JIT due to the stoppage of Vocational Village activities during the coronavirus pandemic. Two of the remaining 23 (8.7%) VR-JIT participants dropped out of the study (but did not withdraw their consent). Thus, 21 participants engaged in VR-JIT and required between three and 10 sessions to complete their training. Fourteen of 21 (67.7%) VR-JIT participants successfully progressed through the recommended easy-to-medium-to-hard difficulty levels on the virtual interviews. Meanwhile, six of 21 (28.6%) participants progressed through easy, medium, and three out of four required hard interviews. Data on easy, medium, and hard completions were lost for one participant (4.3%) who completed 15 total interviews.

Study Measures

Background Characteristics

Demographic, employment, and criminal justice histories.

Participants completed a brief survey about their backgrounds (e.g., age, race, education) and employment histories (i.e., prior employment). Participants also completed a survey about their criminal justice involvement (e.g., Number of times arrested? What is the total length of time spent in jail/prison?). The 6-month phone interview asked participants whether they obtained employment and what their employment start date, wages earned, and hours worked were.

Mental health.

Participants self-reported whether they had ever received a DSM-V diagnosis (e.g., depression, bipolar disorder), and we assessed their current psychological distress using the 10-item version of the Symptom Checklist (Rosen et al., 2000) adapted from the Symptom Checklist-90 (Derogatis & Savitz, 1999). The SCL-10 consists of a total score of 10 items summed on a 5-point scale that assessed how often participants had experienced various types of distress (e.g., feeling afraid, difficulty making decisions) over the past 30 days (0 “not at all or 0 days” to 4 “extremely or 20+ days”). Internal consistency was good (α = .87).

Feasibility Outcomes

Feasibility was assessed via VR-JIT acceptability, usability, and engagement. We assessed participant VR-JIT acceptability using an adapted version of the Treatment Acceptability Rating Form (Reimers & Wacker, 1988). The adapted form used a total score from a summed five-item self-report on a 7-point scale (e.g., 1 = “extremely unenjoyable” to 7 = “extremely enjoyable”). Sample items reflected the enjoyability, ease of use, and helpfulness of VR-JIT. Internal consistency was acceptable (α = .70).

We assessed participant VR-JIT usability using an adapted version of the Systems Usability Scale (Brooke, 1996). The adapted form used a total score from a summed six-item self-report scale ranging from 1 = “not at all” to 4 = “very much.” Sample items reflected readiness to use VR-JIT and ability to use VR-JIT. Internal consistency was unacceptable (α = .21) and may reflect a lack of variation in the summed score as prior studies of VR-JIT using this same scale reported good internal consistencies (Smith, Smith et al., 2021).

We assessed VR-JT engagement with variables generated by the intervention. Specifically, VR-JIT automates the total number of completed virtual interviews, the total number of minutes engaged with VR-JIT, and the total number of e-learning minutes. VR-JIT performance was automated via scores from 0 to 100 that participants received for each completed virtual interview.

Effectiveness Outcomes

Employment.

A competitive job (i.e., located in an integrated community setting that pays at least minimum wage and was not set aside for persons with disabilities or other needs) during the 6-month follow-up was coded 1 = “yes,” 0 = “no.” Competitive employment outcomes were provided to the study team by the MDOC Employment and Opportunities Unit. Employment data is validated via a pay stub provided to a parole agent who enters the data into the database.

Job Interview Skills (Performance-Based).

We assessed participant job interview skills at each pre- and post-test visit via a single mock interview using a version of the Mock Interview Rating Scale (MIRS; Smith, Ginger, Wright, Wright, Humm, et al., 2014) that was adapted for justice-involved populations (J-MIRS). The J-MIRS provides eight different job scenarios from which participants can select for their mock interviews with research staff. Participants had 5 minutes to review the scenario prior to research staff asking 14 standard interview questions along with 4–8 random interview questions from a list of 70 available questions.

Prior to rating the J-MIRS, study team raters (blinded to study group assignment) watched the video and then rated a single item ‘likeliness to be hired’ (scaled from 1 = “unlikely” to 5 = “very likely”) to reflect a global job interview skill rating. Then raters re-watched the video and rated 9 job interview skills based on the literature (Huffcutt, 2011), which used an anchored five-point Likert-type scale. Items included the following: 1) comfort level, 2) discuss prior conviction, 3) hard worker, 4) works well with others, 5) communicates in a positive way, 6) sounds honest, 7) sounds interested, 8) sounds professional, and 9) overall rapport. Each item was rated using a set of qualitative anchors to guide the rating (e.g., a rating of 1 for comfort level is anchored by a participant being observed as ‘highly anxious’ and ‘loses train of thought’). A total score was computed by summing the nine job interview skills. The four raters were master degree students with prior experience conducting real-world job interviews. The raters were trained using 10 gold standard role-play videos to learn how to use the rating anchors before independently rating study videos. Raters compared their ratings to the 10 gold standard videos, and were reviewed and discussed by the study role-play trainer. Also, raters jointly coded videos at regular reliability sessions with the role-play trainer to review and discuss rating inconsistencies (i.e., ratings more than one point apart) and reach a consensus score. Raters coded the same videos for 15% of the sample for reliability (one-way random effects ICC across all 9 items = .96).

Job Interview Skills (Self-Report).

Participants self-reported their skill or comfort level across nine job interview skills after completing the mock job interviews. This measure was adapted from prior research and we used a total summed score from nine items using a seven-point Likert-type scale (1 = “extremely skilled” to 7 = “extremely unskilled”). Sample items included: “making a good first impression,” “telling the interviewer about your strong points,” and “asking questions to learn more about the job” (Smith, Ginger, Wright, Wright, Humm, et al., 2014). Internal consistency was strong at pre-test (α = .90) and post-test (α = .94).

Job Interview Training Motivation.

Participants self-reported their motivation to engage in job interview practice and we used a total summed score from the seven-item interest/enjoyment subscale via the intrinsic motivation inventory (IMI; McAuley et al., 1989). The IMI items are based on the theory of self-determinism and are answered using a Likert-type rating scale (1 = “not at all true” to 7 = “very true”) to assess motivation for targeted behaviors. Sample items included: “I enjoyed preparing to find a job very much,” “Preparing to find a job was fun to do,” and “I would describe practicing job interview role-plays as very interesting.” Internal consistency was strong at pre-test (α = .87) and acceptable at post-test (α = .78).

Job Interview Anxiety.

Participants self-reported their job interview anxiety via an adapted version of the Personal Report of Public Speaking Apprehension (PRSPA; McCroskey, 1970). We replaced the phrase “public speaking” with “job interviewing” for all 34 items, whose Likert scale was from 1=“strongly disagree” to 5 = “strongly agree.” Scores were computed using the PRSPA’s validated two-step process reflecting where the total score = ((72 – (total of step 2 items) + (total of step 1 items)). Internal consistency was strong at pre-test (step 1 items, α=0.97 and step 2 items, α = .84) and post-test (step 1 items, α = .96 and step 2 items, α = .88).

Data Analysis

Participants were analyzed in the group to which they were assigned (i.e., intent-to-treat). Independent two-sample t tests and chi-square analyses evaluated potential between-group differences at pre-test related to background characteristics. VR-JIT feasibility measures were evaluated using descriptive statistics: acceptability (total score), usability (total score), and VR-JIT engagement (i.e., total completed virtual interviews, total minutes engaged with virtual interviews, total minutes engaged with e-learning). Meanwhile, VR-JIT performance was measured as the mean (SD) of the virtual interview scores.

Repeated measures analysis of variance evaluated whether VR-JIT was associated with changes in job interview skill (performance- and self-reported), interview training motivation, and interview anxiety. Specifically, we examined whether there was a significant group-by-time interaction and generated partial eta-squared (ηp²) effect sizes. We included the following covariates in the RM-ANOVA models given their prior associations with interviewing and employment: criminal justice history (e.g., time served in prison [or jail], arrest count, and having committed a violent crime as a proxy for risk), age at release, psychological distress, and race (Bushway et al., 2007; Decker et al., 2015; Holzer et al., 2006; Leasure & Kaminski, 2020; Metcalf et al., 2001; Ramakers et al., 2014; Seim & Harding, 2020; Turney et al., 2013). Also, 56.3% of the SAU group and 42.9% of the SAU+VR-JIT group received job offers between pre-test and post-test from community employers due to the SAU model at the Vocational Villages. These offers could have biased participant engagement in SAU or VR-JIT with unintentional effects on interview skills, interview training motivation, and interview anxiety. Thus, we included the receipt of a job offer prior to post-test as a fixed-effect covariate.

For employment outcomes, we first used chi-square analyses to evaluate whether VR-JIT was associated with a higher rate of obtaining competitive employment by 6-month follow-up. Second, we conducted a logistic regression to evaluate the aforementioned relationship while covarying for the above noted factors and pre-test job interview skill given that multiple participants exercised these skills prior to their post-test and not all participants completed post-test assessments. We used a t test to evaluate between-group differences among the employed participants for number of days to obtain employment, hourly wage, and hours worked per week. Notably, all effectiveness outcomes were evaluated with one-tailed tests given the previously demonstrated efficacy and effectiveness of virtual interview training (e.g., Smith, Ginger, Wright, Wright, Humm, et al., 2014; Smith et al., in press). Lastly, we used Pearson correlations (or point biserial with dichotomous variables) to explore whether VR-JIT performance was correlated with the effectiveness outcomes.

Missing Data and Outliers

Data were reviewed and no missing data at the item-level were observed. As noted in Figure 1, n = 11 participants did not complete post-test measures due to early release or the coronavirus pandemic and n = 1 participant’s post-test data was administratively withdrawn due to several outlier values and the participant’s disclosure of anxiety about failing to be released. Two post-test role-plays (one per group) were inaudible and could not be scored. We did not impute these data and only analyzed the raw available data. One participant’s interview anxiety data was outlier and was not analyzed. Outliers were determined using the interquartile range method (Ghasemi & Zahediasl, 2012).

Results

Participant Characteristics

Table 1 displays the characteristics of the study sample (n = 44). Groups did not differ with respect to their demographic characteristics (all p ≥.10). Additionally, groups did not differ regarding their clinical, employment, and justice-involved backgrounds (all p ≥.10).

Table 1.

Background Characteristics of Study Sample

	SAU Group (n = 16)	SAU+ VR-JIT Group (n = 28)	Test Statistic (t or χ2)	p
Mean Age (SD)	39.1 (8.9)	38.1 (6.1)	0.4	.66
Race
Black/African American (%)	56.3	46.4
White (%)	37.5	46.4
Latinx (%)	6.3	0.0	3.3	.35
More than one race (%)	8.0	7.1
Current grade
High school grad/GED equivalent (%)	43.8	46.4
Some college (%)	31.3	14.3	2.8	.43
Technical school graduate (%)	25.0	32.1
Associates degree (%)	0.0	7.1
Prior employment (%)	93.8	96.4	0.2	.68
Self-Reported mental health disorders
Depressive disorder (%)	25.0	17.9	0.3	.57
Anxiety disorder (%)	25.0	7.1	2.8	.10
Bipolar disorder (%)	18.8	7.1	1.4	.24
Posttraumatic stress disorder (%)	12.5	7.5	0.4	.55
Psychological distress (M, SD)	3.7 (6.1)	2.1 (3.3)	1.1	.27
Criminal justice history
Total arrest count (M, SD)	8.3 (9.4)	8.0 (4.9)	0.1	.90
Total years in prison/jail (M, SD)	8.8 (6.6)	6.3 (4.2)	1.5	.13
Primary offense was violent (%)	37.5	39.3	<0.1	.91
Risk for non-violent re-offense
Medium (%)	43.7	46.4	<0.1	.86
High (%)	56.3	53.6
Risk for violent re-offense
Medium (%)	50.0	60.7	0.5	.49
High (%)	50.0	39.3

Feasibility Outcomes

VR-JIT was highly acceptable (M = 31.1; SD = 3.6; range 21 to 35; max = 35) and highly usable (M = 23.6; SD = 0.7; range 22 to 24; max = 24). Regarding VR-JIT engagement, participants completed M = 13.3 (SD = 2.8; range 9 to 18) virtual interviews across M = 5.00 (SD = 1.8; range 3 to 10) sessions. They also spent M = 218.7 (SD = 59.8; range 75 to 350) minutes engaging with the virtual interviewer and M = 22.3 (SD = 18.2) minutes engaging in the e-learning. VR-JIT performance (i.e., means score across all virtual interviews) was M = 92.1 (SD = 3.6; range 86 to 98).

Effectiveness Outcomes

Effectiveness results are displayed in Table 2. We observed a significant time-by-group interaction characterized by a large effect size when evaluating the performance-based job interview skills total score (F_1,18 = 3.4, p = .04; ηp² = .16), where SAU+VR-JIT were rated as having improved overall job interview skills between pre-test and post-test as compared to SAU. The individual job interview skills “communicates in a positive way” and “sounds interested” were both characterized by significant group-by-time interactions (F_1,18 = 10.5, p = 0.005 and F_1,18 = 4.7, p = .02, respectively) and large effect sizes (ηp² = .37 and ηp² = .21, respectively). The remaining interview skills were characterized non-significant (p > .08) though characterized by medium-to-large effect sizes (.06 < ηp² < .15; hard worker; works well with others). The group-by-time interaction for self-reported job interview skills was non-significant (F_1,21 = 0.1, p = .77; ηp² < .01). However, the model revealed a significant time-by-job offer interaction, characterized by a large effect size. Specifically, participants who obtained a job offer between pre-test and post-test self-reported having stronger job interview skills between pre-test and post-test (F_1,21 = 5.3, p = .03; ηp² = .21).

Table 2.

Estimated Means (Standard Error) for Effectiveness Outcomes

Effectiveness Outcomes	SAU Group		SAU+VR-JIT Group		F	p	η p2
	Pre-test M, SE	Post-test M, SE	Pre-test M, SE	Post-test M, SE
Job interview skills (performance-based)	31.8 (2.3)	31.6 (1.7)	29.9 (1.5)	33.2 (1.1)	3.4	.04	0.16
Likeliness to be hired (Global Rating)	3.3 (0.4)	3.6 (0.4)	3.0 (0.3)	3.2 (0.3)	0.1	.71	0.01
Comfort level	3.9 (0.3)	4.2 (0.3)	3.8 (0.2)	3.7 (0.2)	1.9	.09	0.09
Discuss prior conviction	2.9 (0.4)	2.6 (0.3)	2.9 (0.3)	3.2 (0.2)	1.1	.15	0.06
Hard worker	4.0 (0.3)	4.0 (0.4)	3.6 (0.2)	4.0 (0.2)	1.3	.07	0.13
Works well with others	3.9 (0.4)	3.7 (0.4)	3.4 (0.2)	3.8 (0.2)	1.2	.06	0.14
Communicates in a positive way	3.6 (0.2)	3.3 (0.3)	3.1 (0.2)	3.7 (0.2)	10.5	.005	0.37
Sounds honest	3.2 (0.4)	3.5 (0.3)	3.2 (0.3)	3.9 (0.2)	0.4	.53	0.02
Sounds interested	2.4 (0.4)	2.0 (0.4)	2.3 (0.3)	3.0 (0.3)	4.7	.02	0.21
Sounds professional	4.1 (0.5)	4.1 (0.3)	3.9 (0.3)	4.4 (0.2)	1.0	.16	0.06
Overall rapport	3.8 (0.5)	4.2 (0.4)	3.6 (0.3)	3.6 (0.3)	0.5	.24	0.03
Job interview skills (self-report)	48.0 (2.1)	49.8 (2.5)	47.1 (1.5)	49.6 (1.7)	0.1	.77	<0.01
Job interview training motivation	37.9 (2.1)	36.1 (1.7)	40.9 (1.5)	43.0 (1.2)	4.6	.04	0.19
Job interview anxiety	77.7 (6.0)	80.1 (5.6)	81.8 (4.7)	72.4 (4.5)	5.1	.02	0.21

The group-by-time interaction for job interview training motivation was significant (F_1,19 = 4.6, p = .04). The interaction was characterized by a large effect size (ηp² = .19). Specifically, the SAU+VR-JIT group, as compared to the SAU group, had increased enjoyment of practicing job interview training between pre-test and post-test. A significant group-by-time interaction was characterized by a large effect size for job interview anxiety where SAU+VR-JIT had reduced job interview anxiety between pre-test and post-test compared to SAU (F_1,19 = 5.1, p = .02; ηp² = .21). Unless otherwise noted, time-by-race and time-by-job offer interactions within the aforementioned outcome models were non-significant (all p > .10).

Our 6-month follow-up efforts with returning citizens yielded a very low response rate (25%). As a result, we only analyzed the employment outcomes that were provided by MDOC. We observed that SAU+VR-JIT did not differ from SAU with respect to competitive and integrated employment (82.1% vs. 68.8%; χ² = 1.0, p = .15). The logistic regression (Wald = 11.6, p = .001; Nagelkerke R-Squared = .439) revealed that SAU+VR-JIT had significantly better odds of obtaining a competitive job within 6 months compared to the SAU (OR = 7.4, 95% CI = [1.1, 51.4], p = .045), after covarying for age, race, time served in prison (or jail), arrest count, prior violent crime committed, psychological distress, and pre-test interview skill. Significant covariates in this model included psychological distress (OR = 0.7; 95% CI [0.6, 0.98]; p = .04) and total length of time in prison or jail (OR = 1.4; 95% CI [1.0, 1.9]; p = .04). Remaining covariates were non-significant (all p > .10). In addition, the SAU+VR-JIT and SAU groups did not differ with respect to: 1) who interviewed prior to obtaining the job (95.7% vs. 100%; χ² = .5, p = .50; respectively), 2) highest wage earned per hour (M = 15.7 [SD = 4.0] vs. M = 17.1 [SD = 3.1]; t = 1.5, p = .13), 3) hours worked per week (M = 39.6 [SD = 2.1] vs. M = 41.0 [SD = 3.2]; t = 1.0, p = .32), and 4) number of days to obtain their job (M = 42.8 [SD = 26.2] vs. M = 58.0 [SD = 55.2]; t = −0.7, p =.27).

Exploratory Correlations

The exploratory correlations between VR-JIT and effectiveness outcomes are reported in Supplemental Table 1. Higher mean VR-JIT performance correlated with higher performance-based job interview skills (total score) at post-test (r = .50, p = .026). Higher mean VR-JIT performance also correlated with the individual interview skills “discuss prior conviction” (r = .45, p = .045) and “sounds professional” (r = .50, p = .026) at post-test.

Discussion

The findings from this initial study of VR-JIT in prisons revealed feasible implementation by prison staff and was initially characterized as highly acceptable and highly usable by returning citizens. Participants completed a mean of 13 virtual interviews and 3.5 hours of interactions with the virtual hiring manager, and 70% completed the recommended progression from easy to hard interviews (including four hard interviews) with the remaining 30% completing three out of the four required interviews at the hard level. Also, participants reported a strong mean acceptability score of 31.1 (total possible = 35) and strong mean usability score of 23.6 (total possible = 24). Importantly, the acceptability of an intervention is critical to facilitate its effectiveness (Elliott, 2017). Overall, VR-JIT acceptability and usability is consistent with prior research implementing VR-JIT in non-criminal justice settings that include a community mental health agency and special education transition services (Smith et al., in press; Smith, Sherwood et al., 2021; Smith, Smith et al., 2021).

The study results support the initial effectiveness of VR-JIT at enhancing job interview skills among adult male returning citizens. Specifically, SAU+VR-JIT improved their overall mock job interview performance between pre-test and post-test when compared to SAU. Medium-to-large effect size improvements were observed in overall skill, comfort level, discussing a prior conviction, hard worker, works well with others, communicates in a positive way, sounds interested, and sounds professional. Overall, the results are consistent with prior studies of VR-JIT efficacy and effectiveness in non-justice involved populations (e.g., Smith et al., in press; Smith, Ginger, Wright, Wright, Humm, et al., 2014; Smith, Sherwood, et al., 2021). Thus, this study provides initial evidence that VR-JIT could potentially fill a critical gap in evidence-based practices needed to support returning citizens preparing to reenter the community. Also, the SAU+ VR-JIT group had greater motivation to prepare for job interviews, as compared to the SAU group. These results suggests that engaging with virtual interviews could be a more enjoyable method of practice. An alternative interpretation could be that participants using VR-JIT could be more motivated to practice virtual interviews as it enabled them to add something new to their routine.

The SAU+VR-JIT group reported reduced job interview anxiety beyond the SAU group, which was characterized by a large effect size. Although the current study was not fully powered, this finding suggests VR-JIT could potentially serve as an effective exposure therapy given job interviewing is anxiety-provoking among many people (McCarthy & Goffin, 2004), and especially for returning citizens who are preparing for a job search with the added pressure of discussing their prior conviction (Ricciardelli & Mooney, 2018). Notably, this finding is consistent with two recent RCTs that found using VR-JIT in mental health services and its adapted version for youth in special education was associated with reduced job interview anxiety (Smith et al., in press; Smith, Sherwood et al., 2021).

In terms of real-world outcomes, SAU+VR-JIT had a higher unadjusted rate of obtaining competitive and integrated employment by 6-month follow-up when compared to SAU (82.1% vs. 68.8%, p = .15), but the finding did not obtain significance. However, participants randomized to SAU+VR-JIT had greater odds of obtaining competitive employment (OR = 7.4, p = .04) after adjusting for factors associated with obtaining employment among justice-involved adults (e.g., race, psychological distress, time served in prison [or jail], violent crime committed; Bushway et al., 2007; Decker et al., 2015; Holzer et al., 2006; Leasure & Kaminski, 2020; Metcalf et al., 2001; Ramakers et al., 2014; Seim & Harding, 2020; Turney et al., 2013). These results are consistent with recent effectiveness RCTs of VR-JIT (Smith et al., in press; Smith, Sherwood et al., 2021), and suggest a fully powered RCT evaluating VR-JIT within prison-based vocational services is warranted.

Additionally, we observed that the 6-month employment rate for the Michigan Department of Corrections Vocational Villages services-as-usual (i.e., control) group was high (68.8%), which is consistent with their internal records that approximately 67% of returning citizens in 2021 who paroled obtained employment (R. McGeorge, personal communication, October 6, 2021). Although the Vocational Villages themselves have not yet been the focus of a rigorously designed RCT, these are stronger employment outcomes as compared to a comprehensive employment re-entry program that reported peak 12-month employment at 59% (Cook et al., 2015). Meanwhile, the exploratory correlations revealed potential initial mechanisms of action for obtaining employment. Specifically, VR-JIT performance was correlated with multiple post-test job interview skills (i.e., total score, discussing prior conviction, sounds professional). These results suggest that VR-JIT may be an effective intervention for enhancing these job interview skills—findings that are consistent with prior research using VR-JIT and its adapted version (Smith et al., 2017; Smith, Sherwood et al., 2021). Ideally, we would evaluate the correlations between VR-JIT performance, pre-to-post change outcomes (e.g., interview skills), and employment. However, the observed employment rate of 82.1% in the VR-JIT group is near ceiling and lacks variation, and related correlations may not be reliable.

Limitations

The study results should be considered within the context of its limitations. First, this study focused on evaluating VR-JIT within a prison-based vocational rehabilitation program. Notably, returning citizens engaged in the Vocational Villages are required to meet a high thresholds of good behavior targets and completion of internal academic programming to be selected, and most study participants served fewer than 10 years in prison. Thus, it is unclear how our results might generalize to other correctional settings or to returning citizens who served a longer duration, and whose behavior and academic programming accomplishments are not as exemplary. Second, the Vocational Villages conducted some of their typical mock interview training with participants prior to VR-JIT implementation. Thus, we did not capture the potential effects of the usual ‘mock job interview training’ administered within services-as-usual between pre-test and post-test. The study groups received different amounts of mock interview training and future studies could consider controlling the dose of training received. Third, there was a low response rate (25%) when the research team collected data from returning citizens at follow-up. Thus, we analyzed the MDOC-reported employment outcomes that were available for all participants. Fourth, this sample has limited statistical power, and as such, the results should be interpreted conservatively. Lastly, 61% of participants re-entered their communities during the coronavirus pandemic and they all had at least 2 months of their follow-up occur during this period of time. Thus, the effects of the pandemic on study outcomes is unclear, and the results should be interpreted with caution.

Future Directions

Although the current study demonstrated the feasibility and initial effectiveness of VR-JIT in prison-based vocational services, future studies are needed to evaluate the intervention in several areas. First, a fully powered trial using the effect sizes from this study is needed to validate the effectiveness of VR-JIT in prisons and to evaluate whether VR-JIT engagement correlates with employment outcomes and potential mechanisms. Second, future studies will need to engage in research follow-up assessment strategies beyond our methods (i.e., reminder post-cards at 3 and 5 months, outreach to schedule beginning 2 weeks prior to date) to facilitate higher response rates at 6-month follow-up (e.g., outreach to participants 2 to 4 weeks after release followed by regular outreaches, increase incentive payment). Third, the subsequent larger study could consider whether the effects of VR-JIT may be moderated by social and cognitive ability, which were not evaluated in the current study. Fourth, given the evidence of effectiveness for VR-JIT across multiple trials (e.g., Smith et al., in press; Smith, Sherwood et al., 2021; Smith, Smith et al., 2021), an appropriate future study would be a cluster randomized trial design that also tests implementation strategies and uses a multi-level implementation evaluation (e.g., qualitative interviews with returning citizens, staff, and administrators) of VR-JIT within prison-based vocational rehabilitation programs. Such a trial would establish both the effectiveness of VR-JIT, compared to SAU, for obtaining employment and also identify an optimal implementation approach for sustaining the intervention (Damschroder et al., 2009; Owens et al., 2014). Lastly, future studies could consider assessing interview anxiety using physiological measures such as the galvanic skin response.

We offer three lessons learned during this study that may inform the design and implementation of future RCTs in prisons. First, allow sufficient time for research staff to be fully oriented to prison procedures. Second, obtain a physical authorization letter from prison-level (or state-level) correction administrators that research staff can carry to provide tangible authorization for bringing in and removing study materials. Such a letter will expedite potential security concerns upon prison entry and exit. Third, recruit a stakeholder advisory board to inform the study design. Although we included a returning citizen on the panel for this study, future RCTs may consider including a returning citizen from the targeted prison setting as their intimate knowledge of prison procedures will be critical to the success of the study design.

Conclusion

Results suggest VR-JIT is highly acceptable and highly usable among returning citizens. They adhered to and performed well with the training. Prison-based staff implemented VR-JIT with fidelity after being trained by research staff. Results also suggest preliminary effectiveness of VR-JIT for prison-based services as trainees increased their job interview skills, reduced their job interview anxiety, and had greater odds of competitive employment within 6 months of their community re-entry, compared to returning citizens receiving services-as-usual.

Biographies

Matthew J. Smith, PhD, LCSW, is an Associate Professor in the School of Social Work at the University of Michigan and director of the Level Up: Employment Skills Simulation Lab. His research is focused on developing and evaluating the effectiveness and implementation of technology-based interventions to enhance equity in employment for underserved and marginalized groups.

Brittani Parham, MSW, is a PhD student in Social Work and Psychology at the University of Michigan. Her research focuses on evaluating programs and developing interventions that engage vulnerable populations within criminal justice settings. She has experience working as a Juvenile Justice Youth Counselor and Adult Pretrial Probation Officer for the city of Richmond, VA .

Jamie Mitchell, PhD, is an Associate Professor of Social Work at the University of Michigan and assistant director of Clinical Research Participation at Rogel Cancer Center. Her research domain focuses on addressing disparities in health and research participation among Black adults with an emphasis on men.

Shannon Blajeski, PhD, is a research associate in the Level Up Lab at the University of Michigan School of Social Work. Her primary research interest is to develop interventions that improve real-world outcomes for young adults with first episode psychosis who are not engaged in education, employment, or training.

Meghan Harrington, BA, is a clinical subjects coordinator in the Level Up Lab at the University of Michigan School of Social Work. Her role includes data collection, processing, and analysis.

Brittany Ross, MS, manages the Level Up Lab at the University of Michigan School of Social Work. She has worked several years as a research specialist, including research design and data base management, and manages multiple federal and private grant-funded research projects.

Jeffery M. Johnson, LLMSW, builds coordinated care networks as a community engagement manager at Unite Us. He previously served as a community research coordinator in the Level Up Lab where he collected, processed, and analyzed study data.

Daphne M. Brydon, PhD, LMSW, is a clinical social worker and researcher in private practice. Her research focuses on the development of trauma-informed behavioral health interventions and practice for individuals involved in the criminal legal system.

Jennifer E. Johnson, PhD, is the C. S. Mott Endowed Professor of Public Health, Professor of OBGYN, and Professor of Psychiatry and Behavioral Medicine at Michigan State University. She is a licensed clinical psychologist who conducts policy-relevant implementation and effectiveness trials of mental health and substance use interventions for vulnerable populations, including perinatal women and individuals involved in the criminal justice system.

Gary S. Cuddeback, PhD, is a Professor and Associate Dean for Research at the School of Social Work at Virginia Commonwealth University. Dr. Cuddeback’s research is focused on mental health services and intervention research and individuals with mental illnesses, especially those who are involved with the justice system.

J.D. Smith, PhD is Faculty in the Department of Population Health Sciences, Division of Health System Innovation and Research, Spencer Fox Eccles School of Medicine at the University of Utah. He is an implementation scientist and clinical psychologist whose research aims to achieve health equity through delivery of evidence-based interventions to diverse communities and patient populations.

Morris D. Bell, PhD, ABPP, is Emeritus Professor, Yale University School of Medicine, Department of Psychiatry and Senior Research Career Scientist for the Department of Veterans Affairs Rehabilitation Research and Development Service. He is the originator and scientific proponent of the Virtual Reality Job Interview Training program.

Robert McGeorge is the Assistant Education Manager for the Michigan Department of Corrections. He has been highly involved with the creation and implementation of the Vocational Village model now used in 3 locations across Michigan and replicated by several other states.

Kyle Kaminski is the Offender Success Administrator for the Michigan Department of Corrections (MDOC). In this role, he oversees prisoner programming, education, reentry, and education efforts on a statewide basis in Michigan. Kyle has worked for the MDOC for over 7 years after starting his career in the Michigan Legislature.

Aaron Suganuma, MSW is the Reentry Coordinator for the Washtenaw County Sheriff’s Office. He is actively engaged in direct service and advocacy efforts related to criminal justice. He holds a master’s degree in social work from Eastern Michigan University and completed his associate’s degree while serving 4.5 years in prison.

Sheryl Kubiak, PhD, is the Dean of Wayne State University School of Social Work and director of the Center for Behavioral Health and Justice. Kubiak’s 30-year career has focused on improving identification, services access, and intervention focused on the prevention or reduction of consequences associated with behavioral health disorders for those involved in the criminal/legal system.