Using community-engaged methods to adapt virtual reality job-interview training for transition-age youth on the autism spectrum

Abstract

Background:

Virtual Reality Job-Interview Training (VR-JIT) is an efficacious Internet-based intervention for adults with severe mental illness (SMI). Evaluations of VR-JIT have shown improved interview skill and access to employment in several cohorts of adults with SMI and with autism spectrum disorders (ASD). VR-JIT trains participants how to fill out job applications and handle job interviews through e-learning content and applied practice. Trainees receive feedback through in-the-moment nonverbal cues, critiques, and recommendations for improving performance. Our study sought to adapt VR-JIT for transition-age youth with ASD (TAY-ASD).

Methods:

We recruited TAY-ASD and adult stakeholders from public and charter schools, transition programs, and community service providers. Participants provided feedback on VR-JIT to enhance its applicability to TAY-ASD. We used community-engaged methods to process and analyze data from TAY-ASD and stakeholders, presented their quantitative and qualitative responses to community and scientific advisory boards for review and recommendations, and adapted the intervention design and content.

Results:

Our adaptations included adding diversity (gender; race/ethnicity) to the virtual hiring manager; shortening the interview by reducing response options; increasing social storytelling to enhance engagement with VR-JIT core components; adding employment opportunities more relevant to younger workers; reducing the reading level; and making the e-learning content more accessible by adding bullet points, voiceover, and imagery/video; and adding new learning goals.

Conclusions:

This study presents a rigorous and innovative community-engaged methodology for adapting VR-JIT to meet the needs of TAY-ASD. We review our engagement with TAY-ASD and stakeholders, and discuss the standardized coding scheme we used to adapt VR-JIT and the usefulness and limitations of employing this methodology in adapting other behavioral interventions.

1. Introduction

Currently, 50,000 youth with autism spectrum disorders (ASD) in the United States turn 18 each year (Shattuck, Narendorf et al., 2012); many of them receive transition services (e.g., life-skill and vocational programming provided in secondary and post-secondary educational settings) mandated by the Individuals with Disabilities Education Act until they reach age 22 (in most states; some states extend services until age 26). These youth have a high unemployment rate (50–75 %) (Liptak, Kennedy, & Dosa, 2011), which may be partially explained by the availability of few evidence-based services (Taylor et al., 2012) to support their transition to employment during high school or postsecondary education programming (Gerhardt & Lainer, 2011; Higgins, Koch, Boughfman, & Vierstra, 2008). The high unemployment rate demonstrates a clear need for evidence-based interventions for transition-age youth with ASD (TAY-ASD) before they age out of services and face additional barriers to employment (Shattuck, Roux et al., 2012).

Thus far, only a few evidence-based interventions for TAY-ASD that focus on employment are implemented in secondary and post-secondary settings. These evidence-based interventions have been successful at helping TAY-ASD obtain higher employment rates (when compared to control groups) and primarily include supported employment (Schall et al., 2015; Wehman, Chan, Ditchman, & Kang, 2014, 2012) and ProjectSEARCH (Persch et al., 2015; Rutkowski, Daston, Van Kuiken, & Riehle, 2006; Wehman et al., 2013, 2019; Wehman, Schall et al., 2014). Specifically, supported employment focuses on rapid job placement with support from a job coach, on-the-job training, job development and follow-up services, while less emphasis is placed on developing specific skills (Wehman et al., 2012). Meanwhile, the ProjectSEARCH model emphasizes a school-to-work transition model where TAY-ASD develop job skills and independent living skills within the context of three community-based internships that are facilitated by a collective effort among the public education system, state vocational rehabilitation services, and community employers (Rutkowski et al., 2006). However, transition services place minimal emphasis on job-interview skills (Gerhardt & Lainer, 2011; Liptak et al., 2011).

Given the importance of the job interview to obtaining competitive, integrated employment, this oversight may be a critical gap in services for TAY-ASD (AutismSpeaks, 2012). During a typical interview, individuals in the general workforce are expected to effectively discuss their strengths (as they relate to particular jobs), their ability to be team players, their strong work ethic, and areas where they see themselves as needing improvement (among other issues) (Huffcutt, 2011). Simultaneously, they are expected to read the interviewer’s social cues to gain insight into how to modify their interview responses in real time. The challenges in social communication and the presence of repetitive behaviors that characterize autism make it challenging for TAY-ASD to engage in effective social interactions and relate to the experiences of others (Paul, Orlovski, Marcinko, & Volkmar, 2009; American Psychiatric Association, 2013). Generally, individuals with ASD often struggle with executive function and central coherence that cause difficulty interpreting relevant social information (Parsons & Mitchell, 2002; Volkmar & Wiesner, 2009). This social-processing problem goes beyond reading facial expressions and inferring emotional states of others to include other aspects of the environment (e.g., body language, social distance, voice tone, irrelevant sensory stimulation) (Bishop-Fitzpatrick, Mazefsky, Eack, & Minshew, 2017; Frazier et al., 2017). These social challenges and the complexity of the job-interview context can make job interviewing a significant barrier to employment (AutismSpeaks, 2012; Geller & Greenberg, 2010).

To narrow this gap, we established the efficacy of an Internet-delivered virtual reality job-interview training (VR-JIT) tool focused on improving interview skills among adults with disabilities, including autism; VR-JIT was designed to meet the needs of adults with severe mental illness (e.g., schizophrenia, bipolar disorder) (Smith, Fleming, Wright, Roberts et al., 2015, 2014a; Smith, Ginger, Wright, Wright, Boteler Humm et al., 2014). VR-JIT has three core components (Virtual Interview, Interview Basics, and Job Application).

First, there is the Virtual Interview, in which trainees repetitively practice interviewing with a hiring manager named Molly Porter. Interviews with Molly typically last approximately 25 min in duration where trainees read from 10 to 15 response options and read-aloud their choice as the VR-JIT uses speech recognition to help facilitate the conversation. During the interview, trainees receive three levels of feedback on their interview responses: feedback from SIMantha (a nonverbal job coach who appears on the screen immediately after the trainee responds to Molly, to reinforce helpful responses with a thumbs up and hurtful responses with a thumbs down); a transcript of the interview that gives feedback on each statement; and overall performance feedback after the interview is complete that centers on eight scoring criteria. Second, trainees learn from a text-based e-learning curriculum (called Interview Basics) that emphasizes strategies for improving interviewing abilities. Finally, trainees complete mock Job Applications, in which their responses inform the virtual interviewer’s questions. Across five randomized controlled trials, we observed that VR-JIT trainees with a range of disabilities or needs (e.g., autism, schizophrenia, mood disorders) were more likely than nontrainees to receive job offers, with enhanced interview skills being significant in the job offers (Smith, Fleming et al., 2015; Smith, Fleming, Wright, Losh et al., 2015, 2015b; Smith et al., 2017). For further details on the design of VR-JIT, see (Smith, Ginger, Wright, Wright, Boteler Humm et al., 2014).

Given the potential for VR-JIT to help different populations with similar job-interviewing needs, we sought to adapt VR-JIT to meet the specific needs of TAY-ASD by using a community-engaged implementation research approach (Holt & Chambers, 2017). Specifically, we integrated input related to adaptations with great potential to improve future implementation. Grounded in principles of and best practices for community-engaged research, we incorporated structural and didactic adaptations based on the input from key expert constituencies – community, scientific, and stakeholder partners (Israel, Schulz, Parker, & Becker, 1998; Viswanathan et al., 2004; Wallerstein & Duran, 2010). This paper presents the methods we used to adapt the intervention, and how, by engaging different groups of stakeholders, we developed a streamlined version of VR-JIT designed specifically for TAY-ASD.

There is a dearth of evidence-based interventions available to support transition services, and little is known about how such interventions, if they existed, would diffuse across educational systems or how they would be implemented by practitioners and teachers. The current paper helps to narrow this gap by introducing an empirically-driven approach for adapting VR-JIT, one with potential to inform intervention adaptation in other areas of inquiry. Herein, we show how intervention and implementation researchers in myriad fields of inquiry can employ a community-engaged approach to test innovative adaptation methodologies (Smith et al., 2018). If successful, these methodologies can be replicated to adapt interventions in the future to help support the needs of TAY-ASD and others.

Specifically, we describe how we used the adaptation coding system developed by Stirman et al. (2013) to characterize precisely our tailoring, adding to, removing, and shortening the content of VR-JIT. That coding system (1) identifies the source of the modifications (e.g., stakeholders, scientists, developers); (2) characterizes modifications by focused areas, including content, context (i.e., strategy to deliver), or training and evaluation (i.e., strategy to train staff on delivery); (3) specifies the levels of delivery at which modifications occur (e.g., individual trainee level, programmatic level, system level); (4) details the context of the modifications (e.g., setting, personnel); and (5) characterizes the nature of the modifications (i.e., tailoring, adding to, removing, or shortening the content of the intervention). We selected this approach because it is grounded in the tenets of implementation science, is intended to be applied retrospectively to code recommended adaptations, and provides terminology consistent with that used in other intervention adaptation studies, which is a noted need for the field (Rabin & Brownson, 2017). Herein, we present a three-stage approach to adapt VR-JIT for TAY-ASD that has implications for the adaptation of similar interventions for different populations.

2. Methods

2.1. Methods overview

Fig. 1 summarizes the community-engaged, three-stage methodological approach we used to adapt VR-JIT for TAY-ASD. In stage 1, we recruited TAY-ASD and adult stakeholders (adults with ASD, parents of TAY-ASD, teachers of TAY-ASD, and potential employers) to review and give feedback on ways to adapt VR-JIT for TAY-ASD. During stage 2, we reviewed the TAY-ASD and adult stakeholder feedback with a community advisory board (CAB) and solicited additional recommendations for adaptations. In stage 3, we reviewed all available feedback and recommendations with a scientific advisory board (SAB) and the intervention developers (SIMmersion, LLC) to prioritize those recommendations that fell within the available budget and that could best meet the intervention’s key learning goals. Here, we characterized the recommended modifications and adaptations to VR-JIT for TAY-ASD post hoc using the categories and dimensions of the system designed by Stirman et al. (Stirman, Miller, Toder, & Calloway, 2013). We used a consensus process in which we retrospectively coded each recommended adaptation using the procedures described in Stirman et al. (2013).

Fig. 1.

Three-stage community-focused approach for intervention adaptation.

2.2. Community-engaged, three-stage methodological approach for intervention adaptation

2.2.1. Procedures and measures

2.2.1.1. Stage 1: TAY-ASD and stakeholder opinion data

2.2.1.1.1. Inclusion and exclusion criteria.

Inclusion criteria: (1) ages 16–21 (for TAY-ASD only) and ages 22 and older (for stakeholders); (2) diagnosis on the autism spectrum (TAY-ASD or stakeholders with ASD [ages 22 and older]); (3) fluency in English with a sixth-grade reading level (reported for TAY-ASD by parents or teachers using Individualized Education Plans); (4) currently receiving transition services (TAY-ASD only); and (5) willingness to provide informed consent. The presence of ASD was determined using the Social Responsiveness Scale-2^nd Edition (SRS-2) with a total score of 60 T or higher (Constantino & Gruber, 2012; Lee et al., 2010). Parents or teachers completed the SRS for TAY-ASD, while stakeholders with ASD completed SRS-2 self-reports. The rationale for using more than one method is that although parents were invited to complete the SRS-2, some parents were not available to complete the ratings and teachers were invited to complete the measure when parents were not available. Meanwhile, adults with ASD were invited to complete the SRS-2 self-report to support their self-determination and maintain their confidentiality as participants. Exclusion criteria: self-reported medical illness that might significantly compromise cognition (e.g., traumatic brain injury) and vision or hearing problems that would impede reviewing VR-JIT.

2.2.1.1.2. Recruitment.

We recruited TAY-ASD (n = 24) to provide experiential opinions and stakeholders (n = 21: five parents, seven teachers, five community employers, and four employed adults with ASD) to provide expert opinions. We used a purposive sample of TAY-ASD from four secondary-school programs representing suburban, urban, and rural areas, and from autism advocacy networks. Stakeholders were recruited through advertisements at community-based vocational service agencies and in our partner school and local advocacy groups. We recruited the first 24 TAY-ASD who were interested to participate. Study participants logged on to the VR-JIT program and completed the corresponding study measures either at school or at home. The institutional review boards at the University of Michigan and the Northwestern University Feinberg School of Medicine approved the study protocol. Participants provided informed parental permission with minor assent, or individual consent.

2.2.1.1.3. Data collection.

A total of n = 20 TAY-ASD completed 5 in-person research visits that lasted approximately 45 min each. Meanwhile, n = 4 TAY-ASD and n = 21 adult stakeholders completed 5 research visits over the internet. During visit 1, participants reviewed the study goals and provided consent. During visit 2, research staff oriented participants on how to use the Virtual Interview, Interview Basics, and Job Application components of VR-JIT. During visit 3, participants completed two virtual interviews and then completed the Virtual Interview questionnaire. During visit 4, participants reviewed the interview basics and then completed the Interview Basics questionnaire. During visit 5, the participants reviewed the job application and then completed the Job Application questionnaire. Community employers, on a sixth visit, held two additional interviews with the virtual interviewer, and then they completed a questionnaire about her.

2.2.1.1.4. Measures.

The study questionnaires were developed by the research team to evaluate the appropriateness of core aspects of VR-JIT’s design and content to be used by TAY-ASD. The questionnaires targeted the core components of VR-JIT: Virtual Interview, Interview Basics, and Job Application. Each questionnaire included Likert-scale survey and yes/no questions, many of which were followed by open-ended prompts asking participants to expand and explain their responses and provide examples (“textual, qualitative data”).

2.2.1.1.4.1. Virtual interview

This domain focused on the virtual interviewer (Molly Porter), the interview interface, and the interview content. Participants provided their feedback on a 27-item questionnaire. We asked participants to identify features that were most helpful, potential barriers (or inappropriate), or recommendations to be added to the virtual interview. We also asked for opinions about the scoring objectives and the types of jobs used as examples. Since a key objective of this intervention was to help TAY-ASD ultimately find employment, we developed an additional (27-item) survey for the employer stakeholders to complete; it evaluated the Molly Porter character on her professionalism and personality, and the length and appropriateness of her questions and responses. We asked for feedback on the realism of the content of job candidates’ responses to Molly’s questions and whether any new targets for skill development should be added (e.g., “What is your opinion of the realism of the negative responses to Molly’s questions?”).

2.2.1.1.4.2. Interview basics

This domain focused on the text-based curriculum used to educate trainees and help them navigate the program. We asked participants to review each section (job application, creating a resume, getting ready, the interview, after the interview) and give feedback on content (e.g., “How hard was it to understand the ‘Getting Ready’ section [and why]?”; “How helpful was the ‘Getting sReady’ section [and why]?”). (Note: “Interview Basics” is text-heavy; therefore, to minimize participant burden, we randomized participants into three groups to review different sections: the job application and creating a resume [40 items], getting ready for the interview [40 items], and interview and after-interview [52 items].) In total, n = 9 TAY-ASD and n = 7 stakeholders reviewed the first set of sections; n = 8 TAY-ASD and n = 6 stakeholders reviewed the second set of sections; and n = 7 TAY-ASD and n = 5 stakeholders reviewed the third set of sections. (N = 3 stakeholders did not complete the Interview Basics review.)

2.2.1.1.4.3. Job application

In this component, trainees practice completing an application, with their answers informing the virtual interview algorithm. Using a 15-item questionnaire, we asked participants to review each section of the Job Application component and identify gaps where content could be added, content or features that could be removed, and ease-of-use of the application (e.g., “Can you think of any parts of the job application that aren’t relevant for students and should be removed? Please explain”).

2.2.1.2. Stage 2: presenting TAY-ASD and stakeholder opinion results to a community advisory board and collecting recommendations to adapt intervention.

We presented an executive summary of the TAY-ASD and stakeholder opinion results from Stage 1 to a community advisory board (CAB) and a scientific advisory board (SAB) to finalize recommendations for adaptation. The executive summary was prepared by the first author and checked for accuracy by co-authors who conducted and supervised the Stage 1 mixed methods data processing and analysis. We elicited input from the CAB in a systematic manner. First, we presented the measures that we used to collect the data during Stage 1. Second, we referenced the executive summary to review key findings (see Results) about each portion of the intervention – Virtual Interview, Interview Basics, and Job Application. For each of these items, we asked for input. We audio recorded the meeting and took notes.

We recruited nine CAB members through networking with local schools, service providers, and advocacy groups. Our diverse CAB included a former state board of education member, a transition manager for a local high school district, a TAY-ASD local transition teacher, a local business owner, three local service providers, and two administrative representatives from a national advocacy group. The CAB included n = 4 male; n = 5 female; n = 6 Caucasian; n = 2 African-American; and n = 1 Latinx participants. Also, the CAB included n = 7 participants who were educators or service providers and served youth with ASD for a mean of 22.3 years (SD = 9.6) each. Representatives from SIMmersion attended the CAB meeting to clarify the technology.

In a three-hour meeting, CAB members were asked to review an executive summary of results, to complete their own personal reviews of VR-JIT (three virtual interviews), and to prepare recommendations for adaptation that would also be reviewed by the SAB.Following the CAB meeting, the SAB reviewed the executive summary, the CAB’s proposed recommendations, and the VR-JIT, and then provided recommendations for adaptation. The six SAB members included scientists with expertise focused on intervention development and TAY-ASD and were recruited based on their expertise in identifying service needs of TAY-ASD, evaluating evidence-based interventions, and developing and evaluating interventions for TAY-ASD. The SAB included the following authors on this paper (MJS, LD, JT, KH, TD, TF).

2.2.1.3. Stage 3: presenting all feedback from community members and experts to a scientific advisory board and intervention developers and collecting recommendations to adapt intervention.

In a four-hour meeting, SAB members presented their unique recommendations for adapting VR-JIT after performing a structured review of all existing recommended adaptations.

Also, the SAB and developers reviewed the final adaptation recommendations and identified which of the recommendations were and which were not within the scope of the present project in terms of content and budget. Lastly, the developers cross-referenced these recommendations within their available budget and prioritized—based on potential impact on the identified learning goals of the program—which adaptations could be implemented.

2.2.2. Analytic approach and data interpretation

2.2.2.1. Stage 1: TAY-ASD and stakeholder opinion data.

Fig. 2 summarizes the analytic approach and coding process. We used SPSS to conduct descriptive analyses of the survey questions. Two master’s-degree-level trained assistants, supervised by the second author, managed the data and independently integrated the qualitative data—text of explanations, clarifications, and examples that followed survey responses—with the descriptive survey data. We independently studied the descriptive survey data and read the opinion data from TAY-ASD and stakeholders. We held five meetings (one hour each) to discuss our initial thoughts, organize the data, and decide how best to report the results. We concluded that TAY-ASD and stakeholders provided similar narratives (explanations and examples) to corroborate their responses to survey questions. Therefore, we agreed that conducting a cross-group analysis was unwarranted. We selected qualitative responses that we inductively (and independently) coded and integrated them with survey responses representing the majority of participants across groups.

Fig. 2.

Summary of analytic approach and coding process.

We agreed on which explanations and examples best embodied the opinions of participants in both groups, and organized our findings according to the three domains of the instrument described above. We reached a consensus on organizing the findings so they could be presented to a community advisory board and a scientific advisory board (see stage 2, below). We included “member checking”—we presented our interpretation to participants to check the authenticity of our findings and the accuracy of the interpretation—to add rigor to the findings (Pinto, Spector, & Valera, 2011; Pinto, Spector, Rahman, & Gastolomendo, 2015).

2.2.2.2. Stage 2: presenting TAY-ASD and stakeholder opinion results to a community advisory board and collecting recommendations to adapt intervention.

This phase did not require data analysis per se; however, all recommendations were discussed and organized based on the degree of importance attributed to each recommendation made by the CAB.

2.2.2.3. Stage 3: presenting all feedback from community members and experts to a scientific advisory board and intervention developers and collecting recommendations to adapt intervention.

This phase did not require data analysis per se; however, all recommendations were discussed and organized based on the degree of importance attributed to each recommendation made by the SAB. After finalizing the recommendations, we retrospectively applied the Stirman et al. coding taxonomy to all recommendations made during Stage 1, Stage 2, and Stage 3 to characterize the nature of the adaptations to VR-JIT (i.e., tailoring, adding to, removing, or shortening the content of VR-JIT) using descriptors consistent with the implementation science field.

3. Results and recommendations

3.1. Stage 1: TAY-ASD and stakeholder opinion data

3.1.1. Demographics

Table 1 summarizes the demographic profile of the n = 24 TAY-ASD participants. The age range for this group was 16–21, and the mean age was 18.

Table 1.

TAY-ASD Sample Characteristics (N = 24).

Race/Ethnicity	N %
Caucasian	13 54.1
Latino/Hispanic	3 12.5
African American	3 12.5
Asian	1 4.2
Native American	1 4.2
More than one race	3 12.5
Gender
Male	17 70.8
Female	7 29.2
Current Educational Level
Freshman (high school)	1 4.2
Sophomore (high school)	11 45.8
Senior (high school)	7 29.2
Undergraduate	5 20.8
Interest in Seeking Employment
Currently looking for a job	6 25.0
Will look for a job in the next six months	6 25.0
Will look for a job more than six months from now	12 50.0
Educational Diagnosis
With primary diagnosis of ASD	23 69.7
With primary diagnosis of ASD and secondary diagnosis of a learning disability	5 15.2
With primary diagnosis of ASD and secondary diagnosis of an emotional disability	4 12.1
With primary diagnosis of “other health impairment” and secondary diagnosis of ASD	1 3.0

Table 2 summarizes the demographic profile of the n = 21 stakeholders. Participants’ ages ranged from 23 to 65 years, and the mean age was 41.

Table 2.

Stakeholder Sample Characteristics (N = 21).

Race/Ethnicity	N %
Caucasian	13 61.9
African American	6 28.5
Latino/Hispanic	1 4.8
More than one race	1 4.8
Gender
Male	8 38.1
Female	13 61.9
Highest Educational Level Attained
High school	1 4.8
Some college	3 14.2
Associate’s degree	1 4.8
Bachelor’s degree	8 38.1
College degree or above (graduate coursework or degree)	8 38.1

3.1.2. TAY-ASD and stakeholder opinion

Below we provide results that represent and illustrate the opinions of the majority of the participants. We collected qualitative and quantitative data concurrently; participants provided brief but specific opinions to guide the adaptation of the intervention.

3.1.3. Virtual interview

Approximately 95 % of participants in the TAY-ASD and stakeholder groups agreed that VR-JIT would be fun and helpful for TAY-ASD. Approximately 75–90 % of participants reported that all aspects of the interview interface were “easy” or “very easy” to navigate and that VR-JIT would help prepare trainees for face-to-face interviews. Participants agreed that the “professional nature” of the virtual hiring manager, Molly Porter, visually prepared them for what to expect in a job interview. Participants reported enjoying Molly as a “person” and not simply as a virtual character. Compared with other tools and interfaces the participants have used, the Molly character appears to be easier to understand.

Community employers, asked to focus on the Molly character, agreed that Molly was professional, and her personality was consistent with those of interviewers in the hiring markets. They agreed that Molly was well spoken and able to maintain good eye contact; the flow of the interview was good; and both the positive and negative responses to Molly’s questions were realistic and appropriate.

The results suggest that, to be more compelling and useful to TAY-ASD, the Molly character needs to be more expressive in her nonverbal cues. Participants felt that the transcript feedback needs to be simplified (by using bullet points). All stakeholders and more than 79 % of TAY-ASD participants reported the need to diversify the hiring manager (currently a Caucasian female) by at least adding a male virtual interviewer. All stakeholders and 41 % of TAY-ASD agreed that diversifying the race/ethnicity of the hiring manager is important.

Participants suggested adding a legend to clarify the color coding of the text-based feedback in the transcript and one to clarify SIMantha’s nonverbal cues. They recommended adding instructions on how to change topics within the interview, and also updated objectives for the interview performance (current objectives include honesty, team player, professionalism, being positive, hard worker, negotiation, interest in job, and overall rapport)—specifically, adding “discussing one’s strengths,” “addressing limited job experience,” and “overcoming limitations.” Data from TAY-ASD suggested that VR-JIT would gain from adding new jobs at Wondersmart, the fictional company where trainees are applying for a job. (Existing Wondersmart jobs include cashier, inventory, food service, maintenance, stock clerk, janitor, customer service, and security.) TAY-ASD agreed that a diverse list of jobs—greeter, website technical support, web development, data entry, automotive or other trade work, and childcare worker—might best meet the needs of both younger and older TAY-ASD. About half of each group suggested reducing the number of responses from 10 to 15 to an average of six per question, to increase the pace of the interview.

3.1.4. Interview basics

Approximately 90 % of both groups reported that the Interview Basics content was helpful. However, 40–50 % of both groups found these sections “somewhat hard” or “hard to understand.” The results suggest accessibility would be improved if the Interview Basics reading level were modified from sixth grade to fourth grade. Moreover, the Interview Basics content could use bullet points, visual imagery, and voiceover rather than presenting information-heavy paragraphs. Both groups suggested adding educational content regarding “discussing one’s strengths,” “addressing limited job experience,” and “overcoming limitations.” The data suggested that changing section titles could improve trainees’ ability to navigate the tool. For example, the “Interview with Molly” section title implies that clicking that button will start the interview. However, clicking the button leads instead to information about the interview scenario and how the interview will be scored. We could rename this section “Before You Interview with Molly” so that it more appropriately reflects its content. And data from both groups of participants suggest that the Interview Basics content could be linked to existing public-domain videos (e.g., YouTube) or online tutorials that emphasize modeling behavior.

3.1.5. Job application

Nearly all (90 % of participants) agreed that the job application should not be modified for TAY-ASD. Specifically, TAY-ASD suggested that adding a feature to help them practice uploading attachments to an online job application would enhance the training. They agreed that choices should be added for different types of jobs for which participants could apply.

3.1.6. Social storytelling

The participants recommended changes to increase overall engagement and accessibility. In response, we created a fourth component, called Social Storytelling, to encompass these changes. First, we identified that we could embed a video to introduce the Interview Basics, SIMantha, Wondersmart, and Job Categories components, which would enhance the tool’s accessibility. Participants suggested that VR-JIT could be made more engaging or fun with the addition of badges, coins, or some other medium of payment to allow trainees to “pay” for fun things or to unlock aspects of the training or enjoyable activities. Others suggested that VR-JIT could include an activity in which they could navigate through the front door of Wondersmart and make their way to the interview room to meet Molly. Finally, the participants suggested adding a video following the interview in which Molly would give her interpretation of how things went.

3.2. Stage 2: presenting TAY-ASD and stakeholder opinion results to a community advisory board and collecting recommendations to adapt intervention

3.2.1. CAB recommendations (Table 3)

Table 3.

Content Adaptations for Virtual Reality Job-Interview Training.

Component 1: Virtual Interview
Adaptation Type	Existing VR-JIT	Adapted VR-JIT	Identified by
Tailoring	Virtual interviewer Molly Porter has subtle expressions	Increase expressiveness for easy level; medium and hard levels remain subtle	TAY-ASD; Stakeholders; CAB; SAB
	Feedback provided during interview uses complete sentences	Simplify feedback into bullet points of information	TAY-ASD; Stakeholders; CAB; SAB
	Three independent levels of difficulty	Integrate a formal progression into VR-JIT’s structure (e.g., require learner to complete three interviews on easy level before moving to medium level)	SAB
	All scoring objectives are introduced before the easy interview	Rename “Scoring Objectives” to “Learning Goals;” Use scaffolding to introduce new learning goals as learner progresses from easy to hard interviews; group learning goals into larger categories	SAB
	Disability and criminal justice statements emphasize adult experiences	Tailor disability and criminal justice responses to be more age-appropriate	CAB
Adding	One female interviewer	One male, one female interviewer	TAY-ASD; Stakeholders; CAB; SAB
	Interviewer is Caucasian	Interviewers will be African American and Latinx	Stakeholders; CAB; SAB
	VR-JIT lacks a legend	Add a legend to interpret color coding for feedback and SIMantha’s nonverbal cues	TAY-ASD; Stakeholders
	VR-JIT instructions	Add instructions on how to change topics with virtual interviewer	TAY-ASD; Stakeholders
	Current scoring objectives include (1) comfort level, (2) hard worker, (3) easy to work with, (4) negotiation, (5) interest in position, (6) honesty, (7) professionalism, (8) sharing things positively	Add (9) discussing strengths, (10) addressing limited job experience, and (11) overcoming limitations as new scoring objectives (using the new ‘Learning Goal’ terminology)	TAY-ASD; Stakeholders; CAB; SAB
	VR-JIT lacks statements for virtual interviewer	Add “Tell me more” or “How” statements for virtual interviewer	CAB
	Molly asks if trainee has questions once per interview	Add more opportunities for Molly to ask if trainee has any questions	SAB
	VR-JIT lacks connection to content in Interview Basics	Add links in feedback to connect learners to content in Interview Basics	SAB
	VR-JIT lacks opportunity to ask for additional time	Add opportunity to ask for additional time before answering a question during interview (e.g., “check notes,” “a moment to think”)	SAB
Shortening	Each question includes 10–15 response options	Each question includes 5–8 response options	TAY-ASD; Stakeholders; CAB; SAB
	Each interview lasts 25–30 minutes	Easy interviews last 10–12 minutes, medium interviews 15 minutes, and hard interviews 16–18 minutes	CAB; SAB
Removing	Includes negotiation scoring objective	Remove negotiation from the learning goals	CAB; SAB
	Includes questions about “dream job”	Remove questions related to “dream job”	CAB
Component 2. Interview Basics
Modification	Existing VR-JIT	Adapted VR-JIT	Adaptations Identified by
Tailoring	Sixth-grade reading level	Fourth-grade reading level	Adult Experts; CAB; SAB
	Content is formatted in sentences and paragraphs	Reformat content using bullet points, visual imagery, and voiceover	TAY-ASD; Stakeholders; CAB; SAB
	Provides basic introduction to the Americans with Disabilities Act (ADA) and provides tips on deciding to disclose that one has a disability; tips on how to share; tips on asking for accommodations	Update ADA language so that it is more applicable to youth in transition: i Recommend talking with a trusted adult ii Provide questions to consider when making the decision to disclose iii iii. Provide suggestions for how to disclose if one chooses to	CAB; SAB
	General titles (e.g., “Talking with Molly” contains introduction to Wondersmart, the Molly character, and scoring objectives)	Modify titles to be more specific (e.g., “Before You Practice with Molly”)	CAB
	Jobs at Wondersmart listed with brief descriptions	Categorize jobs by type of work (e.g., independent, team-based, routine-based); provide visual images of specific job duties being completed	CAB; TAY-ASD (Member Check)
Adding	VR-JIT lacks content connected to scoring objectives (now renamed learning goals)	Add educational content connected to new learning goals: discussing strengths, addressing limited job experience, and overcoming limitations	TAY-ASD; Stakeholders
	VR-JIT lacks links to other trainings	Link content to public videos and trainings, if possible	TAY-ASD; Stakeholders; CAB; SAB
	Content organized in the form of reference materials (i.e., one outline of information that does not require a specific order)	Create modules with specific objectives, targeted content, and comprehension checks	CAB; SAB
	Examples of responses to questions about prior convictions	Expand content related to prior convictions to include when to share, tips on deciding to share, and tips on how to share	CAB; SAB
	VR-JIT lacks integration of positive responses and feedback	Provide examples of positive responses to questions demonstrated by video of SIMantha; negative examples provided in text; tie examples into the feedback at the end of the interview	SAB
Shortening	No recommendations		TAY-ASD; Stakeholders; CAB; SAB
Removing	No recommendations		TAY-ASD; Stakeholders; CAB; SAB
Component 3. Job Application
Modification	Existing VR-JIT	Adapted VR-JIT	Adaptations Identified by
Tailoring	No recommendations		TAY-ASD; Stakeholders; CAB; SAB
Adding	Existing jobs at Wondersmart: cashier, inventory worker, food service worker, grounds worker, stock clerk, janitor, customer service, security	Introduce new jobs at Wondersmart: greeter, tech support, Web development, data entry, fast food worker, auto service, child care	TAY-ASD; Stakeholders; CAB; SAB
Shortening	No recommendations		TAY-ASD; Stakeholders; CAB; SAB
Removing	No recommendations		TAY-ASD; Stakeholders; CAB; SAB
Component 4: Social Storytelling
Modification	Existing VR-JIT	Adapted VR-JIT	Adaptations Identified by
Tailoring	SIMantha provides feedback during the virtual interview	SIMantha is your guide through the training (e.g., introducing the training program, providing feedback after the interview)	SAB
Adding	VR-JIT lacks description of the training	Provide video introduction that describes the training, promotes Wondersmart, and introduces the virtual interviewers	TAY-ASD; Stakeholders; CAB; SAB
	VR-JIT lacks a token economy	Add a token economy where learners earn tokens for completing modules in Interview Basics and virtual interviews; tokens are used to ask SIMantha questions about herself to learn more about her	CAB; SAB
	VR-JIT lacks feedback about scoring objectives	Create videos of SIMmantha providing feedback about each learning goal	SAB
Shortening	No recommendations		TAY-ASD; Stakeholders; CAB; SAB
Removing	No recommendations		TAY-ASD; Stakeholders; CAB; SAB

Note: The level of delivery is at the system level for each adaptation.

Abbreviations. Virtual Reality Job-Interview Training (VR-JIT); Transition-Age Youth with Autism Spectrum Disorders (TAY-ASD); Community Advisory Board (CAB); Scientific Advisory Board (SAB).

The CAB recommended several ways to tailor the Virtual Interview: using scaffolding to reduce the number of interview responses and progressively increase the number of response options; increasing the size of the nonverbal coach, SIMantha; designing the virtual interviewer to be more emotive on the easy level, while progressively being less emotive (and harder to read) on the medium and hard levels; and providing more age-appropriate options to discuss the interviewee’s disability or criminal justice history.

The CAB recommended that the virtual interviewer could say “Tell me more about X” (e.g., “Tell me more about how your education has helped prepare you for this job”) or ask “How” questions more frequently (e.g., “How did your prior job prepare you to work on a team?”). The CAB suggested the addition of a “test” version of the virtual interview that implements a timer, to put pressure on trainees to respond more quickly; that version would also eliminate feedback. The CAB suggested that the virtual interview should be shortened to less than 20 min. It recommended that the interview question about a “dream job” be removed, as the question is not typically asked for jobs for this age group. The CAB also recommended that the “negotiation” scoring objective be removed from the interview content, as it is less relevant for this age group and TAY-ASD would likely ask for work-related accommodations after accepting a job.

The CAB recommended tailoring the introduction of the Interview Basics component to the ADA narrative, to make it more applicable to TAY-ASD. The CAB members suggested adding modules with specific learning objectives, targeted content, and comprehension checks; during the virtual interview, they suggested that SIMantha give cues to improve eye contact. The CAB suggested adding new content to help trainees respond if prior convictions come up during an interview (e.g., educating trainees about when to share, deciding to share, and how to share). The CAB did not have any suggestions for the job application apart from confirming the recommendations offered by the stakeholder groups (working on attachment and adding new job options).

The CAB suggested revising some Interview Basics titles, such as renaming the scoring objectives to “learning goals” and re-organizing the scoring objectives and different jobs at Wondersmart into categories reflecting a common theme (i.e., team-based jobs vs. working independently; routine positions vs. flexible positions). We sought a post-hoc member check from n = 5 of the TAY-ASD; they agreed that they would prefer to have the scores and jobs organized into categories.

The CAB members also suggested adaptations to enhance the social storytelling component of the training. They recommended adding a token economy to reinforce performance by unlocking novel aspects of the training or enabling fun activities. They suggested that we integrate visual images of the jobs (to supplement the text descriptions). The CAB also suggested that VR-JIT could benefit from the addition of a promotional video that trainees could watch, in which SIMantha would introduce them to Wondersmart and the different positions available.

3.3. Stage 3: presenting all feedback from community members and experts to a scientific advisory board and intervention developers and collecting recommendations to adapt intervention

3.3.1. SAB recommendations

After reviewing the CAB’s recommended adaptations, the SAB recommended two strategies to tailor VR-JIT for TAY-ASD. First: modify the levels of difficulty to emphasize a formal progression through the levels embedded in the tool’s structure; second: use scaffolding to introduce new learning goals as the trainee progresses from easy to medium to hard interviews. Also, the SAB recommended several additions to optimize the Virtual Interview component, including adding more opportunities for trainees to ask questions during the interview; adding feedback to new learning goals; adding links to the transcript connecting trainees to videos of SIMantha verbalizing feedback, thus enhancing the social storytelling aspect of the tool; adding links to the summary feedback page after completion of the virtual interview to connect trainees back to the Interview Basics sections where relevant content is presented, and/or have the links connect to videos of SIMantha verbalizing the feedback; adding an opportunity to ask for an accommodation during the interview (allowing trainees to “ask for a moment” or to “refer to their notes to ask a question”); adding more response options for students to request accommodations without disclosing that they have autism; and adding feedback to new learning goals. The SAB refined the suggestion to reduce the length of interviews by targeting the easy interviews to require 10–12 min to complete (with four response options per question), while medium interviews would require 13–15 min (with six response options per question) and hard interviews would require 16–18 min to complete (with eight response options per question).

The SAB also refined the suggestion to add a token economy by suggesting that students could earn points for reading pages and completing mini-quizzes within Interview Basics and that tokens could be used to learn more about SIMantha, who works for Wondersmart and guides trainees throughout their use of VR-JIT. The SAB did not have any additional input about tailoring or removing aspects of the Virtual Interview interface or interview content. It recommended that the Interview Basics component could be further enhanced by adding positive and negative real-world examples of how learned content is demonstrated in an actual interview. The SAB members suggested that the examples could be demonstrated by SIMantha within a video. Finally, they suggested that the summary feedback page could provide links to these examples. The SAB did not have any additional recommendations for the Job Application features.

The SAB and intervention developers considered each item of feedback and provided recommendations through an iterative process, as follows. TAY-ASD suggested that an activity that allowed them to practice walking through the door of Wondersmart and navigating to the interview with Molly would increase social storytelling. The change would require a significant budget and extensive resources and would add a new learning goal to the program (navigating a new environment) rather than enhancing the existing learning goals. Additionally, the new goal would be a generalized skill rather than one unique to the job-interview process. Therefore, the developers identified that this recommendation was out of scope. Subsequently, SIMmersion and members of the SAB developed a plan to adapt (tailor, add to, shorten, or remove content from) the three core components of VR-JIT (Virtual Interview, Interview Basics, Job Application), while introducing social storytelling throughout the intervention. Table 3 summarizes the planned adaptations identified by each group of experts within the framework of the Stirman coding taxonomy (Stirman et al., 2013). Meanwhile, Table 4 summarizes the recommendations that are not being included in this phase of development, and the justification for their exclusion.

Table 4.

Recommendations Excluded from the Current Adaptation of Virtual Reality Job-Interview Training (VR-JIT).

Core Component	Recommendation	Identifier	Justification
Interview Basics	Allow learners to walk through Wondersmart to meet Molly	TAY-ASD; CAB	Significant budget; additional learning goal; not unique to job interviews
Virtual Interview	Have SIMantha provide feedback on learner’s use of eye contact	CAB	Significant budget; additional learning goal; not unique to job interviews
Virtual Interview	Add a “Challenge” level that includes a timer and provides minimal feedback	TAY-ASD; CAB	Significant budget including a technology change
Virtual Interview	Limit the number of available answers (four options at the easy level, six options at the medium level, and eight options at the hard level) to simplify interface and reduce the interview length	TAY-ASD; CAB; SAB	Program usability may be compromised; alternative solutions identified: to reduce interview length, the number of questions asked by the interviewer will be limited by level of play, and to simplify the interface, uncommon errors will be removed, leaving five to eight options available
Virtual Interview	Include audio and/or video to supplement feedback provided in the transcript	CAB	Significant budget; alternative solution identified: add video of SIMantha explaining the scores received after the interview
Job Application	Add ability to practice uploading a resume to the online application	TAY-ASD	Significant budget; additional learning goal

4. Discussion and implications

There are few evidence-based practices to support federally mandated transition services for youth with diagnoses on the autism spectrum. Among available services, job-interview training is largely overlooked and undersupported (Gerhardt & Lainer, 2011; Liptak et al., 2011). As such, the successful deployment of an effective tool to enhance job-interview skills could fill a major gap in the training needs for this group. We sought to address this service gap by adapting VR-JIT (developed for adults with SMI) to meet the needs of TAY-ASD. Once adaptations to VR-JIT were identified through our iterative and systematic process, we used the Stirman et al. (2013) coding system to characterize the specific adaptations that would increase accessibility and acceptability of the adapted VR-JIT for use by TAY-ASD in schools and other settings providing vocational-support services. Several key adaptations emerged through this iterative process that have broader implications for interventions that may serve TAY-ASD.

We learned that there is a need to revise the program to include more-explicit cues (e.g., expressiveness of the virtual interviewer, adding “tell me more” and “how” statements, and adding opportunities for Molly to ask trainees if they have any questions). These revisions will help TAY-ASD focus more directly on learning the interview skills. Other future interventions may similarly need to increase the intensity and salience of some features of the curriculum when it is being adapted for individuals with ASD.

The feedback provided by participants also aligns with literature in the field of instructional design, computer-based learning, and gamification for individuals with ASD (Bol & Garner, 2011; Moore, Cheng, McGrath, & Powell, 2005; Whyte, Smyth, & Scherf, 2015). Instructional-design principles suggest that instructional materials should account for users’ ability to interpret and retain the information presented by ensuring appropriate levels of cognitive load (i.e., storing and processing information in one’s working memory) (Sweller, Van Merrienboer, & Paas, 1998). These principles are particularly important for individuals on the autism spectrum who struggle with following multistep instructions to complete a task. New VR-JIT adaptations will help reduce the cognitive load for TAY-ASD. (See Table 3) This reduction in cognitive load could reduce anxiety and allow users to focus on enhancing their interview skills. The instructional-design and computer-based-learning literature supports integrating tools such as immersive storylines, personable characters, and interactive dialogue, and highlights these tools as some of the most useful for engagement and generalization of skills to real-world environments (Moore, McGrath, & Thorpe, 2000; Whyte et al., 2015).

Thus, a critical and advantageous adaptation of the VR-JIT is the addition of storytelling features, such as using SIMantha as a guide throughout the training, providing a video introduction describing the training, introducing the virtual interviewers and the Wondersmart environment, and creating videos of SIMantha providing feedback. Each of these features involves intuitive learning and usability within a computer-based learning context, particularly in the areas of scaffolding, integrating relevant visual imagery, and auditory feedback. Meanwhile, the reduction of unnecessary text clutter, including using hyperlinks and limiting excessive movement throughout the learning environment, is in sync with the literature on designing virtual environments to optimize executive functioning for individuals with ASD (Bol & Garner, 2011; Whyte et al., 2015).

Our three-stage approach to adapting VR-JIT for TAY-ASD has implications for the adaptation of similar interventions and those for populations other than TAY-ASD. For example, using gamification techniques, such as continuous feedback and the simple reward-token system implemented in this adaptation of VR-JIT, has been shown to motivate learners and enhance attention and engagement (Whyte et al., 2015). As in this study, future adaptations of other interventions for individuals with ASD and others with social cognitive deficits may benefit from careful attention to cognitive load and gamification techniques. Attention to design features will likely be increasingly important as computer-based learning becomes more common in educational settings.

By recruiting TAY-ASD and adult stakeholders (adults with ASD, parents of TAY-ASD, teachers of TAY-ASD, and potential employers) we demonstrated that, in order to acquire useful and feasible adaptations, consultation and collaboration with community members and experts in all phases of the adaptation process are necessary. Such collaboration adds transparency to the process and generates adaptations that reflect those who may most benefit from the intervention. We also demonstrated that our three-stage approach, including the use of a framework for adaptation (Stirman et al., 2013), can be employed by other researchers seeking to modify and adapt similar interventions. As more researchers begin to use specific approaches and frameworks for adaptation, rigorous processes, and both survey and qualitative data, we will be able to further develop the science of intervention adaptation.

Grounded in principles and best practices of community-engaged research, we incorporated input from end-users and stakeholders at each stage of our approach. The procedures we used have been shown to increase the feasibility, acceptability, and utility of behavioral interventions (Dyson, Chlebowski, & Brookman-Frazee, 2019; Pinto, Witte, Wall, & Filippone, 2018; Stirman, Gamarra, Bartlett, Calloway, & Gutner, 2017). In turn we, we applied these practices and procedures to determine adaptations to the virtual job interview tool, now called “Virtual Interview Training for Transition-Age Youth” or VIT-TAY. We acknowledge the need for evaluating VIT-TAY for effectiveness and for comparing the findings with those of previous trials. That said, we feel it is important to address that we designed VIT-TAY to complement available transition services or to serve as a standalone evidence-based practice to support individuals who lack access to transition services.

Available transition- and vocational services are not always standardized or tested directly in the setting of optimal use. Additionally, there is a general lack of research that targets job interview training. Typically, transition services limit interview training to 1–2 mock interviews facilitated by vocational counselors, yet this approach has limited ecological validity due to trainees’ pre-established rapport with the counselors. Moreover, this approach does not use repetitive practice and hierarchical learning methods, which facilitate sustainable behavioral change for individuals with ASD (Cooper, 1982; Cooper, Heron, & Heward, 2007; McClannahan, MacDuff, & Krantz, 2002). Lastly, mock interviewing is time and resource-intensive, which are barriers to accessibility. Thus, the adapted VR-JIT has the potential to fill these gaps in both research and services by 1) studying rigorous methods of adapting interventions, 2) establishing a program that complements existing evidence-based practice, and/or 3) enhancing services that lack empirical support. Moreover, given that this tool is delivered via the internet, there is a strong potential for this tool to be delivered within school-based educational or social service settings, or in the community (e.g., home, the library) in a cost-effective manner. Similar to VR-JIT, SIMmersion will commercially distribute VIT-TAY.

As noted, there are a limited number of evidence-based practices to support transition services (Bishop-Fitzpatrick, Minshew, & Eack, 2013; Roux, Shattuck, Rast, Rava, & Anderson, 2015; Taylor et al., 2012), and few of these programs can be easily administered with fidelity (Schultz, Schmidt, & Stichter, 2011; Seaman & Cannella-Malone, 2016). In addition, the large-scale programs like Project SEARCH are not accessible to all and the field still needs discrete interventions for TAY-ASD that can be easily integrated into services offered in numerous settings, including schools. Thus, we developed VIT-TAY that addresses a critical service gap in the field and will be implemented with TAY-ASD perspectives in mind. By following empirically-driven procedural strategies, we engaged community members and stakeholders, the result of which is a user-friendly intervention with greater potential for diffusion, implementation, and acceptability (Baumann, Cabassa, & Stirman, 2018; Chambers & Norton, 2016; Pinto, Spector, & Rahman, 2019). Although we adapted a highly specific intervention, the methodologies we used to adapt it can inform the development and/or adaptation of interventions for TAY-ASD and other populations.

The development of VIT-TAY also has several research implications. First, future research can evaluate the effectiveness of implementing VIT-TAY in secondary and post-secondary educational settings. Second, although we adapted this tool to meet the needs of TAY-ASD, there is strong potential that this tool could be used to help support pre-employment training for transition-age youth with educational disabilities (e.g., learning disability, emotional disability) receiving services in secondary and post-secondary educational settings. Third, if VIT-TAY demonstrates effectiveness at improving interview skills and increasing access to employment, then future research can evaluate the implementation of VIT-TAY.

The present study was not without limitations. Our sample of TAY-ASD and stakeholders included individuals with a variety of experiences related to autism and employment; nonetheless, we cannot generalize the findings to all TAY-ASD given the small size and homogenous socioeconomic status of the sample. In addition, we focused on adaptations for individuals without intellectual disability during this initial effort. Thus, additional modifications and adaptations likely would be needed for individuals with lower cognitive and reading abilities.

5. Conclusion

This project involved a community-engaged iterative process to adapt an evidence-based, internet-delivered, virtual reality job interview intervention for TAY-ASD, a group with strong unmet needs for efficacious transition services. Key adaptations centered on reducing cognitive load, increasing diversity of the intervention characters, enhancing engagement by including more social storytelling, and increasing relevance for the target population. These efforts may be valuable to consider for future adaptations of other curricula for individuals with ASD, particularly interventions using computer-based instruction. VIT-TAY holds great promise for improving job-interview skills for TAY-ASD, with the ultimate goal of increasing employment during adulthood. Once the adapted intervention is tested, its deployment in schools or postsecondary education transition programs could help fill a critical gap in federally mandated transition services for this population. The multistage process described in this paper provides a methodology for adapting other interventions for use with new populations and delivery settings. Summarizing adaptations using the Stirman et al. (2013) system provides a means of comparing and synthesizing the findings of similar adaptations to contribute to generalizable knowledge for the field of implementation science.