Abstract
Adults with serious mental illnesses (e.g., Autism Spectrum Disorder [ASD], schizophrenia, post-traumatic stress disorder [PTSD]) often have difficulties obtaining employment. The Job Interview Training System with Molly Porter, developed in collaboration with Yale and Northwestern Universities and vocational rehabilitation specialists with funding from The National Institutes of Health (R43/44MH080496), allows learners to practice job interviews on computers in a stress free environment. The system includes user-driven educational materials, an interactive job application, a practice simulation with a fictional interviewer (Molly Porter), and extensive feedback. SIMmersion’s PeopleSIM™ technology allows each conversation with Molly to provide a unique interview experience, enabling users to gain confidence while building skills. The on-screen coach provides insight during the conversation, and a comprehensive after-action review provides learners with feedback on the entire interview. In a randomized control trial, the system was proven effective at improving participants’ interview skills and confidence. Ninety-six (96) unemployed adults with ASD (n=26), schizophrenia/other (n=37) or PTSD (n=33) were recruited. Participants were randomized into control (n=32) and experimental (n=64) conditions. The control group was “wait-listed” to receive training, and the experimental group used the training system with Molly Porter. Both groups completed pre- and post-intervention role-play interviews and self-assessment questionnaires. Analyses of covariance showed that the simulation provided a highly significant training effect, with experimental group participants scoring better in the role-play interviews and self-assessing higher than control group participants. By increasing skills and confidence, this system may ultimately reduce the length of unemployment for adults with mental illnesses.
Keywords: Simulation, Virtual Reality Training, Job Interview, Serious Mental Illness, Disabilities
Introduction
Obtaining and maintaining employment has been proven to improve quality of life among adults with serious mental illnesses (e.g., Autism Spectrum Disorder [ASD], schizophrenia, post-traumatic stress disorder [PTSD])1, more than two-thirds of whom want to work. Yet these individuals often have difficulties obtaining and maintaining employment. Supported employment (SE), which typically includes job coaching, technical assistance, accommodating work settings, clinician-facilitated role-plays, and other psychosocial supports, is the gold standard for vocational interventions2. However, access to these programs is limited, and only 30% of participants reported that practicing job interviews in SE was helpful3.
To overcome these limitations, researchers developed a virtual-reality role-play utilizing PeopleSim™ technology. The computer-based Job Interview Training with Molly Porter program merges evidence-based principles for both effective job interviewing4 and effective simulation training5. Learners have the ability to repeatedly practice conveying their individual experiences, job-relevant knowledge, and social effectiveness to a simulated human resource manager while receiving in-the-moment feedback. The customizable training package creates an immersive, realistic, and stress-free environment that allows learners to make and address errors across a spectrum of skills and strategies.
1. Job Interview Training with Molly Porter
Job Interview Training with Molly Porter was designed to teach, reinforce, and refresh job-interview skills to adults with mental illness who are actively seeking employment. The system incorporates user-driven educational materials, an interactive role-play simulation, and formative feedback to create a training experience consistent with high-fidelity simulations5, successful job interviews4, and adult learning theory.
The educational component includes approximately five hours of training materials designed to help learners prepare for interviews and complete the other steps necessary to find a job. Some of the topics covered include creating a resume, researching a position, selecting a job that meets individualized needs, deciding what to wear to an interview, selecting appropriate questions to ask, deciding whether or not to disclose a disability, and taking appropriate follow-up steps. The system also includes an interactive job application that allows learners to practice completing an on-line application. These didactic materials adhere to the sixth grade reading level and are arranged in an easy-to-navigate, user-friendly interface. The straightforward presentation allows learners to customize the experience to their individual needs by allowing them to find and access the materials they want, when they want them.
The interactive role-play simulation with Molly Porter includes 10+ hours of practice and was designed to provide a different experience each time it is used. SIMmersion’s patented PeopleSim™ technology combines video, voice recognition software, and non-branching logic to create an interactive environment that allows users to experience complex social cues and realistic interpersonal exchanges, build skills, clarify concepts, and increase retention. Molly personalizes the training experience for each individual learner based on their answers to the job application questions about preferred job (i.e., cashier, stock clerk, customer service, maintenance/grounds, janitorial, food service, inventory, or security); education, employment history, and job-related skills; and optional questions about disabilities (e.g., spinal cord injury, visible disability, hidden disability), history of mental illness, military history, past substance abuse, and criminal history.
Molly has a database of more than 1000 video-recorded questions that range from general inquiries (e.g., “Tell me about yourself?”) to specifics about personal history (e.g., I noticed on your application that there are gaps in your work history. Can you tell me about that?) and job duties (e.g., This position will require you to work closely with other associates. Do you enjoy working as part of a team?). The non-branching logic of PeopleSim creates dynamic links between Molly’s questions and the 2000 available learner responses, allowing learners to try new approaches to answering questions during each interview. Molly’s simulated brain includes memory and a wide range of realistic emotions and personality that allow her to further tailor the interview to each trainee. For example, if someone applies for a customer service position and responds that he prefers to work independently, Molly may say, “That job requires that you work closely with others. Are you still interested in it or would you prefer something else?” The combination of learner options and Molly’s realistic demeanor ensures that trainees experience a new interview each time they talk with her.
The script includes a wide variety of natural choices, so learners have multiple opportunities to provide both appropriate and inappropriate responses during the simulated interview. Because of the variety of responses available to the user at any time, there is no one “right” or “wrong” choice at any point in the conversation. This allows users to move beyond judgment training (i.e., Which of the statements is best?) to learn conversational skills (i.e., What should I say next?).
Every learner response dynamically affects Molly’s confidence that the learner is the right person for the job, just like in real life. Molly may patiently ask for additional information to clarify a vague response given early in the interview but will not be as forgiving when similar mistakes are made later. Throughout the conversation, learners receive real-time support and feedback from an on-screen coach who provides nonverbal cues regarding the learners’ responses. If the learner wants more information, “help” buttons prompt the coach to explain the gesture or provide insights into the kind of response Molly is hoping to receive. Upon the conclusion of each interview, learners receive qualitative and quantitative feedback that evaluates how well they performed each of the learning objectives.
Images of Molly and access to the virtual role-play can be found at http://www.jobinterviewtraining.net.
2. Evaluating Efficacy
To evaluate the efficacy of Job Interview Training with Molly Porter, researchers conducted a randomized control trial with three distinct groups of participants: community dwelling adults with autistic spectrum disorders (ASD; n = 26), community dwelling individuals with schizophrenia and other serious mental illnesses (Schizophrenia/other; n = 37), and Veterans diagnosed with PTSD who received outpatient care at Hines VA in Chicago (n = 33). At recruitment, all participants were unemployed but searching, or planning to search, for work. Participants from each group were randomized to a Wait-list Control (WLC) group or Job Interview Training with Molly Porter Intervention (Molly) group at a ratio of 1:2. Researchers hypothesized that participants in the Molly group would demonstrate improvements in job interview role-play skills and self-confidence about job interview skills compared with participants in the WLC group. Additionally, they hypothesized that the Molly intervention sessions would be rated as easy to use, enjoyable, and helpful.
2.1. Methods
Baseline assessments were conducted for all participants including 1) demographic and vocational interviews, 2) neurocognitive and social cognitive assessments, 3) standardized interview role-plays, and (4) a self-report of self-confidence. Vocational data included prior participation in vocational training programs and amount of time (in months) since any prior employment was held. The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) was administered to assess neurocognitive functioning, and the Bell-Lysaker Emotion Recognition Task (BLERT) was used to assess basic social cognition. There were no significant differences between conditions for any of the three groups.
Interview role-plays were conducted by trained actors with experience working as interviewers. Each 20-minute interview was scored by trained raters (ICC = 0.85) on nine skills that contribute to successful job interviews4: 1) conveying dependability, 2) sounding easy to work with, 3) conveying professionalism, 4) negotiating for individual needs, 5) sharing in a positive way, 6) sounding honest, 7) demonstrating interest in the job, 8) displaying comfort, and 9) establishing overall rapport with the interviewer. Both actors and raters were blind to condition. Participants rated their self-confidence about interviews using a 7-point Likert scale (higher scores reflect more positive opinions) to answer nine questions about comfort and skills.
Following the baseline assessments, the WLC group received no additional training beyond their previously scheduled vocational services. The Molly group was asked to complete 10 hours of training with Job Interview Training with Molly Porter (approximately 20 trials) over the course of 5 visits. Both groups returned after the two-week intervention period to complete the follow-up self-confidence measure and two additional standardized role-plays.
2.2. Results
An analysis of covariance (ANCOVA) was performed with post-intervention role-play scores as the dependent variable, treatment condition and diagnostic group as fixed factors, and pre-intervention role-play scores as the covariate. This analysis showed a highly significant treatment condition effect for the Molly group (F1,86 = 8.3, p < .005). A second ANCOVA was performed with the post-intervention self-confidence measure as the dependent variable, treatment condition and diagnostic group as fixed factors, and pre-intervention self-confidence measure as the covariate. This analysis also showed a highly significant training effect (F1,86 = 12.18, p < .001).
59 out of 63 participants (94%) assigned to the Molly groups completed 3 or more hours of training. All were asked complete the Treatment Experience Questionnaire (TEQ) and rate characteristics of Job Interview Training with Molly Porter. Using a 7-point Likert scale (7 being the most positive), participants scored the system 5 or above (mean) on all characteristics, with the majority receiving mean scores above 6.
3. Conclusions and On-going Research
Job Interview Training with Molly Porter showed a moderate to large effect on improving job interview skills in live role-play interviews and increasing self-perceptions of confidence, comfort, and readiness for job interviews. Differences by diagnostic group were not significant, suggesting that this intervention, like other skills training interventions, may be beneficial to a wide range of disorders. Additionally, the product’s integrated customization options allow it to be used by a general audience.
Follow-up research is being conducted to determine if the increase in skill and confidence demonstrated by participants in the Molly group leads to higher employment rates compared to participants in the WLC group. Preliminary reports from 27 participants contacted 20 weeks post-intervention indicate that of the 16 randomly assigned to the Molly group, 9 (56.3%) had obtained work; while only 3 of the 11 (27.2%) assigned to the WLC had obtained work. 15 of the 16 Molly participants strongly agreed that, looking back, the interview training was helpful, and 13 agreed or strongly agreed that it gave them more confidence in the real job interviews. Researchers anticipate that the larger sample of post-intervention data will follow the same trends as these preliminary reports demonstrating that by increasing job interview skills and confidence, this system may ultimately reduce the length of unemployment for adults with mental illnesses.
Table 1.
Characteristics of the Study Sample
| ASD | Schizophrenia/other | PTSD | ||||
|---|---|---|---|---|---|---|
|
| ||||||
| WLC (SD) (n=10) | Molly (SD) (n=16) | WLC (SD) (n=12) | Molly (SD) (n=25) | WLC (SD) (n=10) | Molly (SD) (n=23) | |
| Demographics | ||||||
| Mean Age | 23.2 (3.0) | 24.9 (6.7) | 44.3 (10.3) | 50.0 (11.6) | 50.8 (10.4) | 51.2 (11.5) |
| Gender (% male) | 80% | 75% | 16.7% | 64.0% | 100% | 95.7% |
| Vocational History | ||||||
| Prior Voc. Training | 20% | 43.8% | 25.0% | 32.0% | 50% | 43.5% |
| Months since employ | 26.5 (24.9) | 32.7 (22.2) | 47.2 (60.5) | 42.1 (43.4) | 38.7 (62.5) | 41.3 (55.9) |
| Cognitive function | ||||||
| Neurocognition, mean | 89.0 (19.2) | 89.8 (21.4) | 91.3 (15.4) | 95.2 (19.9) | 90.7 (4.9) | 88.9 (12.7) |
| Social Cognition, mean | .81 (.11) | .72 (.17) | .75 (.13) | .70 (.16) | .63 (.14) | .66 (.12) |
Table 2.
Mean Score on Training Experience Questionnaire (TEQ), mean (SD)
| ASD (n=16) | Schizophrenia/other (n=25) | PTSD (n=23) | |
|---|---|---|---|
| Ease of use | 5.8 (1.2) | 6.1 (0.9) | 5.8 (1.0) |
| Enjoyable | 5.1 (1.6) | 6.4 (1.0) | 6.5 (0.7) |
| Helpful | 5.4 (1.6) | 6.3 (1.1) | 6.8 (0.4) |
| Instilled confidence | 5.4 (1.7) | 6.0 (1.2) | 6.8 (0.4) |
| Prepared for interviews | 5.8 (1.4) | 6.0 (1.0) | 6.5 (0.6) |
References
- 1.Gold PB, Macias C, Rodican CF. Does competitive work improve quality of life for adults with severe mental illness? Evidence from a randomized trial of supported employment. Journal of Behavioral Health Services & Research. 2014 doi: 10.1007/s11414-014-9392-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Bond GR, Becker DR, Drake RE, Rapp CA, Meisler N, Lehman AF, Bell MD, Blyler CR. Implementing supported employment as an evidence-based practice. Psychiatric Services. 2001;52:313–322. doi: 10.1176/appi.ps.52.3.313. [DOI] [PubMed] [Google Scholar]
- 3.Salyers MP, Becker DR, Drake RE, Torrey WC, Wyzik PF. A ten-year follow-up of a supported employment program. Psychiatric Services. 2004;55:302–308. doi: 10.1176/appi.ps.55.3.302. [DOI] [PubMed] [Google Scholar]
- 4.Huffcutt AI. An empirical review of the employment interview construct literature. International Journal of Selection and Assessment. 2011;19:62–81. [Google Scholar]
- 5.Issenberg SB, McGaghie WC, Petrusa ER, Gordon DL, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Medical teacher. 2005;27:10–28. doi: 10.1080/01421590500046924. [DOI] [PubMed] [Google Scholar]