Simulated Job Interview Skill Training for People with Psychiatric Disability: Feasibility and Tolerability of Virtual Reality Training

Abstract

The job interview is an important step toward successful employment and often a significant challenge for people with psychiatric disability. Vocational rehabilitation specialists can benefit from a systematic approach to training job interview skills. The investigators teamed up with a company that specializes in creating simulated job interview training to create software that provides a virtual reality experience with which learners can systematically improve their job interview skills, reduce their fears, and increase their confidence about going on job interviews. The development of this software is described and results are presented from a feasibility and tolerability trial with 10 participants with psychiatric disability referred from their vocational service programs. Results indicate that this representative sample had a strongly positive response to the prototype job interview simulation. They found it easy to use, enjoyed the experience, and thought it realistic and helpful. Almost all described the interview as anxiety provoking but that the anxiety lessened as they became more skilled. They saw the benefit of its special features such as ongoing feedback from a “coach in the corner” and from being able to review a transcript of the interview. They believed that they could learn the skills being taught through these methods. Participants were enthusiastic about wanting to use the final product when it becomes available. The advantages of virtual reality technology for training important skills for rehabilitation are discussed.

Introduction

Inactivity and loss of productive function commonly accompany severe psychiatric disorders (eg, schizophrenia, other psychotic disorders, mood disorders, post traumatic stress disorder [PTSD]). Yet, surveys¹ indicate that more than 75% of people with these disorders wish to return to some kind of productive activity; however, when they attempt to return to work, they often have no access to appropriate work activity. They become discouraged when they fail to find or maintain a job. As research has shown,¹ unemployment itself can lead to deterioration in mental and physical health in previously healthy individuals, and these consequences are all the more serious for those with severe psychiatric disorders. Moreover, returning to work helps reintegrate individuals with disabilities into their community, a key objective of the Americans with Disabilities Act. For these reasons, improving vocational rehabilitation services has become a national priority.

Over the past 20 years, Supported Employment (SE) has become the preferred model for vocational services for people with psychiatric disabilities, and in particular, the Individual Placement and Support model (IPS) has been developed as the preferred form of SE. IPS was developed by Bond, Becker, Drake, and others² as a standardized version of SE, which is now available in a toolkit from the Substance Abuse and Mental Health Services Administration. This toolkit explains the principals of IPS, provides training modules to staff, and has a fidelity-rating instrument to determine degree of successful implementation. The main tenets of IPS are maximum inclusion, client choice, rapid job finding, competitive employment, integrated mental health care, and continuous support and follow-along services. Even when fully implemented,³ almost half of clients never obtain work even after 2 years of services. Moreover, the average length of the first job is less than 3 months, so that at any one time, between 30% and 40% are employed.

To improve these outcomes, we have been exploring ways to augment work services. Since 1988, our research group has examined a variety of methods including adding cognitive remediation,^4–9 cognitive behavior therapy focused on negative cognitions about self and work,^10,11 and various approaches to providing feedback about work performance.⁷ In the current study, we have targeted job interview skill training for intervention development. We have done so because the job interview is a crucial first step to obtaining competitive employment and a daunting hurdle for many with psychiatric disability.

The skills necessary for a successful job interview can be taught, particularly through role-playing, but there are several barriers to this training being effective. First, vocational specialists who would most likely be responsible for doing this training are not specifically trained in role-playing methods. Second, role-playing is time consuming, particularly because skill development may require multiple repetitions with specific critique and suggestions for improvement. Third, role-playing, which may reveal deficits, may be embarrassing for the client who may therefore avoid the training. Fourth, the client will likely have cognitive impairments that slow the learning process, which makes it important that implicit methods of learning (such as role-modeling and repetition rather than verbal instruction) be utilized. Finally, job interviews are anxiety-producing encounters and as with other feared situations, systematic exposure and mastery are the best methods to reduce those fears.

To address these concerns, our research group teamed up with a company (SIMmersion, LLC) that has successfully produced virtual reality software that creates realistic interview training systems. For example, they have created such software for the Department of Defense to train military chaplains in suicide screening, for the Federal Bureau of Investigation to train culturally sensitive interrogation procedures, for the National Institute on Alcohol Abuse and Alcoholism to train primary care physicians in substance abuse screening and intervention, using their proprietary software that allows users to influence the behavior of the simulated character. Previous work¹² has indicated that the technology is engaging and learners will practice willingly for hours. SIMmersion received a grant (Small Business Innovation Research—phase 1 [SBIR-1]) to produce a prototype of the job interview training software with content provide by our research team. The technology is based on 3 key features to directly address the needs of our clients.

Speech Recognition

Adults with psychological disorders often have difficulties preparing positive answers to difficult questions during an employment interview. Using a headset with a microphone, the user trains the program to recognize the user’s voice. The use of speech recognition technology allows users to practice speaking prescripted positive responses to these difficult questions in a pressure free environment. Then, in a real interview, he or she can utilize the rehearsed answers practiced within the simulation.

Individualized Customization

Completing a job application well takes time and practice. Prior to using the simulated conversation, the user completes an employment application, which includes questions about their employment history, skills, and contact information. This practice better prepares learners to accurately complete future applications. Additionally, the information included within the application is used by the simulation to populate the list of questions from which the simulated character draws. For example, a user may select that he would like to apply for a customer service position on the application and identify that his previous job was for a construction company; the simulated character may ask, “I see from your resume that you have experience in construction and are applying for a customer service position. Why are you looking to make that change?” This innovative application of the PeopleSim™ simulation technology, allows users to personally customize their interview experiences to better prepare themselves for future interviews.

Nonbranching Logic

Deciding whether or not to disclose a psychological disorder to a potential employer can be difficult. The PeopleSim™ technology uses nonbranching technology that allows users to do and say what they want, when they want, within the confines of a rich script. Most conversational simulations utilize branching logic technology that provides users the opportunity to select from a short list of options and terminates when all the options have been exhausted. These systems resemble a multiple-choice test and are only intended to be used once or twice. In addition to giving the users control over what they say about themselves, The PeopleSim™ technology provides for variety in how the simulated interviewer acts. The simulated character (an actress who records hundreds of possible questions and reactions) has memory, emotion, and a personality; this variability encourages repeated practice and makes the exchanges with the simulated character seem more like a real conversation. As described above, the speech recognition feature enables users to create their own scripts and rehearse responses. The individualized customization feature allows them to practice using the appropriate scripts learning from the responses they get. This feature provides hours of self-paced training in a safe and judgment-free environment.

The simulation included 3 main parts: e-learning content, a simulated job interview with a simulated employer (a trained actress), and a multilevel feedback system. The e-learning content provided users with an overview of the technology and guidance on how to interview well. These guidelines included information on preparing for a successful interview, disclosing a disability, and assessing how an interview went. To make the interview as realistic as possible, the simulated interviewer asked the user a wide variety of questions related to employment history and job skills. The user was offered selections from a list of scripted statements for inputting as speech or by mouse-click. Some of the statements were designed to build positive rapport with the character, while other statements impaired rapport. This variety of statements allowed users to conduct a natural conversation, while providing the opportunity for them to make mistakes and learn from them. The feedback system included an on-screen coach who provided feedback throughout the conversation and an after-action transcript and audio recording that allowed users to review their interview. The development process involved focus groups comprised the scientific team, 2 human resource specialists from local businesses who routinely conduct job interviews, 2 vocational specialists from a high fidelity IPS program, and 2 clients receiving vocational services. The focus group included women and minority representation. In the first of 3-day-long focus groups, a scriptwriter from SIMmersion audiotaped numerous job interview role-plays. From these, came sets of typical questions that would be likely asked by the interviewer (eg, gaps in work history, transferrable skills from previous employment) and standards for good and poor responses from applicants. In 2 subsequent focus groups, scripts were reviewed with an increasing range of possible secondary topics and possible responses. The final script was reviewed and approved by the focus group participants and the scientific team. SIMmersion hired an Equity actor to perform the hundreds of iterations that were videotaped and then embedded in the simulation software. Thus a prototype version of the training software was produced for pilot testing.

The full version of the simulation (currently under review) will expand the range of questions and include more nuanced explorations of relevant issues. It will also include a Manual for vocational specialists that instructs them in how best to utilize the software for various types of clients and offers suggestions for integrating the software into their vocational services. Since the software training cannot address special problems related to tone of voice (eg, flat and expressionless) or nonverbal behaviors (eg, eye contact) that are important for successful interviewing, the Manual recommends that when the vocational specialist has a client with these limitations, the specialist use the after-action transcript and audio- recording features to listen to the interview with the client and offer correctives. In some cases, the vocational specialist may wish to reenact the role-play scenario with the client to correct nonverbal behaviors. The full version of the simulation will also include a Client Workbook that reiterates the educational portion and has work sheets to help the client prepare for each interview. These work sheets include learning about the company, writing down employment strengths that the client should emphasize and has sections on what to wear, how to get there, when to arrive, and who to talk with. There are also work sheets for self-evaluation following the interview that ask about what went well, what could have been improved upon, and what might be done differently next time. The Manual instructs the vocational specialist in how to introduce the Client Workbook and how to use it as part of ongoing counseling and rehabilitation.

This approach to training job interview skills is theoretically grounded in the behavioral rehearsal techniques of social skill training. As stated by Robert Liberman in Recovery from Disability,¹³ “A crucially important element of social skills training is having patients practice improved communication in situations that closely simulate real-life situations where they must apply their skills (page 222).” The simulation is not meant to remediate neurocognitive or social cognitive impairments nor is it expected to teach skills that would generalize beyond the specific situation of a job interview. Its purpose is to allow rehearsal of those behaviors that make for a successful job interview. The Manual and Client Workbook (to be included in the full version) are meant to help the client generalize the skills acquired in the simulation to each specific job interview situation that the client encounters. However, there is no expectation that this training will improve social functioning in other domains.

The current study was the initial investigation to test the feasibility and tolerability of the simulated job interview for training clients of vocational rehabilitation programs with psychiatric disability and to gather their impressions of the training procedures. The aim was to obtain responses from 10 participants who would be representative of typical clients in vocational rehabilitation and who would reflect diversity of age, gender, ethnicity, and types of mental illnesses of the population.

Methods

Participants

Recruitment was achieved by contacting several agencies that provide IPS services to clients with psychiatric disability. They referred potential participants who were selected to represent age, gender, ethnicity, and types of mental illness. Participants received a gift card of $60 for their involvement with the study.

Assessments

Participant characteristics were obtained through a semistructured interview that included psychiatric history, work history, current involvement in rehabilitation services, and perceived barriers to employment.

Assessment of the simulated job interview skill training was performed using 17 Likert-scale questions (1 = strongly disagree to 5 = strongly agree) about their reactions to the simulation, 6 questions (1 = poor to 5 = excellent) about their opinion of usability, about their overall opinion of usefulness of the simulation and likelihood that they would use this simulation when fully developed. Participants were asked 5 yes/no questions about the simulation as an alternative to role-plays.(?) The self-report questions were adapted from previous studies of this kind conducted by SIMmersion. Additionally, open-ended questions yielded qualitative responses, which were recorded by the research staff for content analysis of common themes.

Procedures

Following written informed consent procedures as approved by the Institutional Review Board, participants were interviewed by research staff to obtain background characteristics. They were then brought to the computer laboratory where they were oriented to the simulation software. They were shown the special features, including the voice recognition, the help buttons and the replay features and they played the video introduction that features their “Coach,” played by an actress, who remains with them in a corner of the video display throughout the training giving them feedback on their responses during the interview and offering suggestions when asked (by clicking on the help button). The Coach provides further explanation of the program’s special features so that the participant can navigate the program easily. After this orientation, the participant was asked to review the psychoeducational material about job interviewing that is offered as part of the software. This section, which was limited in this prototype version, discusses how to prepare for job interviews, what is expected at an interview, and what are the elements that make for a successful interview. These elements then comprise the scoring system, which is the final feedback that the participant received after each “play.” The participant then engaged in 2 “plays” of the software, each taking about 15 minutes. A member of the research staff remained nearby participants as they carried out the simulated interviews and was available to answer any questions or clarify any concerns that the user may have had. At the conclusion of each play, the participant received a computer-generated qualitative statement (eg, poor, good, excellent) on the following criteria: developing rapport, telling the interviewer about personal strong points for the job, asking questions to learn more about the job, negotiating the best arrangements (eg, schedule), making sure that the interviewer knew what job the interviewee wanted to do, and concluding the interview in a positive way. These qualitative statements were not used for analysis.

Data Analysis

This was an observational study to determine feasibility and tolerability. No formal hypotheses were tested. Analysis was limited to descriptive statistics.

Results

The study was successful in recruiting participants with chronic mental illness who are currently engaged in vocational rehabilitation. They are a representative sample reflecting the diversity of the population.

Participant Characteristics

The sample was comprised 5 males and 5 females between the ages of 24 and 60 (mean = 42.3, SD = 10.0). Six were African-American and 4 were Caucasian; 8 were single, 1 was married, and 1 divorced. They ranged in education from 12 years to 16 years (mean = 13.2; SD = 1.2). Eight were “work experienced” (defined as having had at least 1 full year of competitive employment in their life), and 2 were not. However, typical of this population, in the past 3 years, only 1 had had full-time competitive employment, 6 had held some part-time work, and 3 had not worked at all. Also typical of this population, 6 had been arrested in the past and 3 had been incarcerated, 2 with felony convictions. Eight carried diagnoses of schizophrenia or schizoaffective disorder; 1 was diagnosed with chronic PTSD and 1 with Borderline Personality disorder. Alcohol and substance abuse were common comorbidities for most of the participants with an average lifetime abuse of alcohol of 4.5 years (9.7 years) and drug abuse of 2.1 years (4.6 years). Despite having at least a high school education and most having held a full-time job at one time in their lives, these participants have significant barriers to their returning to full-time employment including serious mental illness, vulnerability to substance abuse, and criminal histories. It is precisely for these reasons that they are appropriate for vocational rehabilitation services and could potentially benefit from job interview training.

Self-Report Scores

Results of the self-report assessment of the simulated job interview training are presented in tables 1–3.

Table 1 shows the scores on the 1– 5 Likert scale (strongly disagree to strongly agree). All means are above 4.0. Especially, encouraging is that their Overall Rating (item 17) had only a range of 4–5 and the mean was 4.8. Ease of using the program (item 14) showed a similar range and a mean of 4.5. Such a high rating on this item indicates that despite cognitive and symptom limitations of these participants, they felt that they could negotiate use of the software. It is also of note that the item with the lowest score was about whether the simulated interviewer treated the respondent fairly (item 5). The mean score was still quite high (4.1), but the range was from 1 to 5. This indicates that participants were willing to use the full range of the scale. It also means that some of them felt considerable discomfort and may have had an attributional bias toward feeling mistreated by authorities. This is precisely the kind of reaction that this training may help to correct.

Table 1.

Responses to Features of the Simulation (Sca1e is 1–5; Strongly Disagree to Strongly Agree)

	N	Minimum	Maximum	Mean	SD
Ease to learn simulation	10	2	5	4.60	0.96
Enjoy playing simulation	10	2	5	4.60	0.96
Able to try new things to say	10	4	5	4.80	0.42
Interviewer look and act real	10	1	5	4.30	1.25
Interviewer treated fairly	10	1	5	4.10	1.29
Choices of what to say to interviewer	10	2	5	4.20	1.03
Choices of realistic responses	10	3	5	4.50	0.71
Helpful introductory screens guidelines	10	3	5	4.50	0.85
Usefulness of help agent feature	10	3	5	4.20	0.92
Usefulness of help on a statement feature	10	3	5	4.50	0.85
Usefulness of help on a response feature	10	3	5	4.40	0.84
Usefulness of view conversation feature	10	3	5	4.30	0.95
Overall usefulness of help feature	10	3	5	4.50	0.85
Ease of using program	10	4	5	4.50	0.53
Prototype useful in improving interviewing skills	10	4	5	4.70	0.48
Likely to use bigger simulation when available	10	3	5	4.50	0.71
Overall simulation rating	10	4	5	4.80	0.42

Items in table 2 cover many of the same areas as those in table 1, but the nature of the scale allows for judgments that have a higher ceiling such as very good or excellent and these items are focused more narrowly on usability. Again, all the scores are very favorable with high agreement that it was easy to use, highly useful for training job interview skills and that the participant would be highly likely to want to use the full simulation when it is available. Indeed, it was to this final item (item 23) that they gave their highest rating.

Table 2.

Usability (Scale 1–5; Poor to Excellent)

	N	Minimum	Maximum	Mean	SD
Ease to use	10	3	5	4.10	0.88
Directions were clear	10	3	5	4.10	0.74
Ease to navigate	10	3	5	4.10	0.88
Interactions seemed real	10	3	5	4.50	0.85
Simulation useful to train job skills	10	3	5	4.40	0.70
Likelihood of using simulation	10	3	5	4.60	0.70

Finally, 9 of 10 found the simulation entertaining (item 1, table 3), which may be important for maintaining interest and engagement with the exercises. Eight of 10 said that they would be curious to try the simulation again, and all 10 agreed that this simulation was a comparable alternative to a live role-play. As reflected above in responses suggesting some discomfort in the questioning, 9 of 10 did not expect the questions that they were asked. This result suggests that the participants were unprepared for the standard interview questions that the simulation used and that they have a great deal to learn about what to expect in a job interview.

Table 3.

Additional Responses (No = 0, Yes = 1)

	N	Minimum	Maximum	Mean	SD
Simulation was entertaining	10	0	1	0.90	0.32
Curious to try simulation again	10	0	1	0.80	0.42
Ever practiced job interviews using role-play	10	0	1	0.40	0.52
Comparable alternative to role-play	10	1	1	1.00	0.00
Expected questions asked	10	0	1	0.20	0.42

Qualitative Responses (Free Response)

Participants made a number of comments that add to our understanding of their experience. They all saw it as helpful overall, although they varied in what they liked most about it. Comments included: “I learned a lot from this simulation about myself and job interviewing.” “It kept me interested and focused.” “It portrayed accurately what might be said in a job interview.” “I felt the interactions were life-like.” “It stimulated my brain. I thought it was very educational.”

Some also commented on how anxiety provoking a job interview can be and that the simulation itself was realistic in causing anxiety. Comments included: “I felt that the interviewer was stoic and unapproachable. Her appearance intimidated me. It was a learning experience.” “Gave tough questions I had to answer.” “Better than role-play. You had the actual feel of being in an interview.” “Felt like an actual interview.” “Didn't hire on the spot, so don't know if I'd be hired.”

Because it felt so realistic to the participants, they also viewed it as a chance to overcome their fear of the situation through practice and to get better at it. Comments along these lines were: “I wanted to take full advantage of the program, so when I go on an interview I can do a good job.” “It was interesting to see how to improve my skills.” “I was not as nervous as I would be in a real-life situation.” “Job interviews are difficult sometimes, I found this one a little more understanding.” “It teaches you how to interact with the person that is interviewing you.”

When asked what would improve the simulation, their comments supported the need for further development. They had suggestions about additional interview questions and wanted a greater variety of possible responses. For example, “More questions about job related issues. “Should be more questions about your resume.” “More variety of interview responses.” “Would like more questions about physical ability.” “Thought it was engaging. Adding more options would make it more realistic and appealing. More interactive like.”

Participants also stated that they really liked the special features of the simulation, especially the Coach and the ability to review the transcript of the dialogue afterwards. For example: “Wanted to see the reaction of the job coach if I answered in a way that was not entirely accurate.” “This training is comparable (to role-play) because it gives you feedback and helps improve interviewing skills.” “Overall, I feel this simulator is excellent in helping people be better interviewers in getting jobs. Thanks a lot.”

Discussion

Virtual reality, of which this simulation is an example, is a promising technology that allows safe and convenient immersion in an experience from which the client can learn. In this study, the aim was to determine whether a job interview could be created in this way that would be realistic to our participants and that would be tolerable and feasible.

We found that a representative sample of clients with mental illness who were engaged in vocational rehabilitation had a highly positive response to the prototype job interview simulation. They found it easy to use, enjoyed the experience, and thought it realistic and helpful. Almost all described the interview as anxiety provoking but that the anxiety lessened as they became more skilled. They saw the benefit of its special features such as the Coach and the transcript and believed that they could learn the skills being taught through these methods. Participants were enthusiastic about wanting to use the final product when it becomes available.

This prototype was created through the National Institute of Mental Health SBIR—phase 1 mechanism and is not a final product. An SBIR-phase 2 has been submitted, and if funded, this would allow the development of a larger simulation system. In phase 2, aspects of the prototype simulation will be used as a model for an expanded simulation, which will include more comprehensive psychoeducational material, a Manual to instruct vocational specialists in how to integrate the software into their vocational planning, and a Client Workbook that contains work sheets to help them prepare for job interviews and to learn from their experience after an interview.

Future studies will include an efficacy trial of the completed version, which if successful will be followed by a dissemination study. The simulation allows consistent behavioral rehearsal with reliable feedback that can be repeated as often as necessary until mastery is achieved. It is likely that illness characteristics such as neurocognitive impairments, social cognitive impairments, and negative symptoms will affect how many learning trials are required to achieve mastery and whether mastery is achieved in all or only some skill areas. We have added measures of these illness characteristics in our efficacy study proposal and hypothesize that their effects on outcome will be smaller for the simulation condition than for the active control, which uses commercially available instructional videotapes and workbooks. The simulation takes advantage of implicit learning which is not true for the control condition, which relies on explicit learning. Moreover, it may be that this software will be useful for disadvantaged or disablility groups beyond those with psychiatric disorders, but who may also benefit from this method of learning. With minor modification, this program might be appropriate for high school students with disabilities entering the work force for the first time, reentering ex-offenders or people in welfare to work programs.

Virtual reality technology is advancing rapidly. More and more our society is becoming comfortable with the interface of personal technology that provides us with information, entertainment, communication, and education. Psychiatric rehabilitation should take advantage of these developments to make skill learning of various kinds easier and more accessible. While this particular software is limited to training the set of skills necessary for successful job interviewing, the methods for its development could be used to create software that address other important areas in psychiatric rehabilitation such as social skills training. Computer-based training increases fidelity and ease of dissemination and these advantages may greatly increase the impact of such an intervention beyond what is currently possible when an intervention must be delivered by specially trained staff. While such software is in no way a replacement for clinical contact, computer-based training may amplify the effectiveness of such contact. Technology is improving physical medicine and rehabilitation, and this study is an early indication of how it may someday improve psychiatric practice and rehabilitation.

Funding

National Institute of Mental Health grant to Dale Olsen, Ph.D. of SIMmersion LLC (#R43 MH080496-01A)