A type 1 hybrid multi-site randomized controlled trial protocol for evaluating virtual interview training among autistic transition-age youth

Abstract

A number of policies mandate that autistic transition-age youth receive employment services to prepare for the workforce before high school graduation. A key limitation to these services is the job interview component, which relies on non-standardized, resource-intensive, staff-led role-plays to help autistic transition-age youth improve their interview skills. The autism community has called for better job interview preparation. To address this gap in services, our team, collaborated with the autism community to adapt the intervention, Virtual Reality Job Interview Training (VR-JIT; effective among adults with serious mental illness), into Virtual Interview Training for Transition Age Youth (VIT-TAY). This adapted intervention was tailored to meet the needs of autistic transition age youth while maintaining the core components of VR-JIT (i.e., an online job interview simulator with four levels of automated feedback and e-learning content). A pilot randomized controlled trial (RCT) demonstrated VIT-TAY's feasibility and initial effectiveness at improving job interview skills, reducing anxiety, and increasing employment rates within six months when added to transition services or pre-employment transition services (Pre-ETS). Thus, the overarching goal of this Hybrid Type 1 effectiveness-implementation study protocol is to conduct a fully-powered RCT of VIT-TAY across 16 schools in various geographical locations. Our specific aims are to 1) Evaluate whether Pre-ETS (or transition services) with VIT-TAY, as compared to Pre-ETS (or transition services) with an active control intervention (i.e., job interview didactics/e-learning with a series of 3–5 min videos of employed autistic adults talking about their career pathways) enhances employment outcomes; 2) Evaluate mechanisms of employment by nine months post-randomization; and 3) Conduct a multilevel, mixed-method process evaluation of the initial implementation of VIT-TAY across settings (e.g., acceptability, feasibility, and barriers and facilitators of implementation).

1. Introduction

According to our estimates using U.S. Census data and autism prevalence rates, approximately 120,000 autistic transition-age youth turn 18 each year [1,2] with 35 % transitioning into college and 50 % not participating in education or employment [3]. Thus, the Interagency Autism Coordinating Committee has prioritized autism research to support lifespan issues, including employment, referring to it as a high-priority area [4]. Notably, research-based programs such as Project SEARCH and supported employment have high employment rates (70 %) for autistic young adults [5,6]. However, access to these services is limited for this population [7], who are typically engaged in school-based transition services (e.g., community experiences, employment and adult living skills) funded via the Individuals with Disabilities Education Act (IDEA [8], or pre-employment transition services (Pre-ETS; i.e., delivery of workplace readiness training, work-based learning experiences, job exploration counseling, postsecondary education counseling, and self-advocacy), funded via the Workforce and Innovations Opportunity Act (WIOA [9]; and delivered by state-level vocational rehabilitation contractors.

The United States Department of Education's Office of Special Education and Rehabilitative Services identified that job interview skills are a critical component of the job readiness training delivered via Pre-ETS or transition services [10]. The job interview is a barrier to employment for autistic youth and adults who have voiced their desire for training to improve their skills; yet experience anxiety and low motivation to interview [[11], [12], [13]]. Importantly, transition services typically rely on 1–2 staff-led role-plays to practice job interview skills which can be considered time intensive. However, there is limited evidence these methods improve interview skills and employment outcomes [12,14]. Thus, the lack of evidence-based practices to improve interview skills is a major gap in services that is compounded by evidence that nearly 90 % of employed autistic transition-age youth who received Pre-ETS interviewed prior to getting their job [15].

To address this gap, our team developed a multi-level adaptation framework where 45 stakeholders from the autism community (i.e., autistic transition-age youth, parents, teachers, service providers, employers) and two advisory boards (i.e., community advisory board [CAB], scientific advisory board [SAB]) reviewed and made recommendations to adapt to an efficacious internet-delivered job interview simulator to support the needs of autistic transition-age youth [16]. The CAB and SAB then gave feedback on the design (and study measures [17,18]) to evaluate the newly adapted Virtual Interview Training for Autistic Transition Age Youth (VIT-TAY). The subsequent pilot RCT evaluating VIT-TAY found that autistic transition-age youth who engaged in Pre-ETS (or transition services) with the addition of VIT-TAY compared to those receiving Pre-ETS (or transition services) as-usual had significantly higher competitive employment (+25 % by six-month follow-up), stronger job interview skills, and reduced job interview anxiety [[14], [19]]. We also learned that the RCT could be feasibly conducted within Pre-ETS (or transition services), VIT-TAY could be implemented by Pre-ETS (or transition services) staff with fidelity, and it was perceived as highly useable and acceptable by youth and staff [20]. Further, we identified a series of initial recommended practices to guide future VIT-TAY implementation efforts [21].

The proposed Hybrid Type 1 effectiveness-implementation study will continue our community-engaged approach where the CAB and SAB reviewed the design from our funded grant and made recommendations to optimize our proposed evaluation of VIT-TAY. First, we will evaluate whether VIT-TAY enhances employment outcomes (e.g., competitive integrated employment, job interview skills, time-to-employment) among autistic transition-age youth. Next, we will evaluate whether job interview skills are a mechanism of employment. Finally, we will evaluate the initial VIT-TAY implementation process outcomes.

2. Methods

2.1. Study design

We will conduct a Hybrid Type 1 effectiveness-implementation study [22] using a two-arm, multi-site, single-blinded, intent-to-treat, RCT (NCT06291233). Hybrid Type 1 studies primarily evaluate the effectiveness of an intervention (i.e., VIT-TAY) while secondarily collecting data on its initial implementation processes. A total of 200 autistic youth (ages 16–26) engaged in high school-based transition services or Pre-ETS will be recruited and randomized to VIT-TAY (Pre-ETS + VIT-TAY) or an Active Control (Pre-ETS + AC; [viewing e-learning job interview tips and set of career pathway videos for autistic adults]) across 16 schools or programs in two states in the United States. Notably, the WIOA defines transition age as individuals with disabilities between the ages of 14 and 24 who have left or aged out of high school (this study will have a lower age limit of 16 due to a focus on obtaining employment) [9]. For this study, we included participants through age 26 as the State of Michigan funds transition services or Pre-ETS through this age [23]. The Institutional Review Board at the University of Michigan reviewed and approved all study procedures and materials for both sites. Study specific aims are:

Aim 1. Evaluate if Pre-ETS + VIT-TAY, compared to Pre-ETS + AC, enhances employment outcomes among 160 autistic youth (after attrition). We hypothesize that Pre-ETS + VIT-TAY, compared to Pre-ETS + AC, will have higher employment rates, greater improvement in job interview skills, job interview preparation knowledge, and get jobs more quickly by 9-month follow-up.

Sub-aim 1. Explore whether VIT-TAY relates to improved hourly wage, work readiness and work motivation, and longer job tenure.

Aim 2. Evaluate mechanisms of employment outcomes. We hypothesize that job interview skills will be a mechanism by mediating the relationship between VIT-TAY and employment.

Exploratory Sub-aim 2. Explore whether job interview anxiety and job interview motivation may be mechanisms of employment outcomes by mediating the relationship between job interview skills and employment.

Aim 3. Conduct a multilevel, mixed-method implementation process evaluation to assess the acceptability, feasibility, and determinants of VIT-TAY implementation. We will use focus groups, interviews, and surveys (with autistic youth, teachers, administrators) within a convergent mixed methods approach to assess implementation process and outcomes variables.

2.2. Study procedures

We hosted Community Advisory Board (CAB) and Scientific Advisory Board (SAB) meetings for stakeholders and scientists to review the design and measures of the proposed study. The CAB suggested two modifications (Table 1) based on the acceptability and potential feasibility of the proposed design. CAB members included two autistic self-advocates (TJG, DT), Director of Post-Secondary Transition Program (JR), Michigan Rehabilitation Services counselor (JH), Coordinator of Psychiatric Services, Illinois Division of Rehabilitation Services (EAO), Staff Services Manager, California Department of Rehabilitation (VD), and a Transition Services Manager (CB). The SAB suggested modifications (Table 1) to improve the scientific rigor of the design. The SAB included a Professor of Social Work (SM), Professor of Counseling, Educational Psychology and Special Education (CS), Associate Professor of Education (KH), and Assistant Professor of Social Work (EW). The below study procedures reflect our final approved procedures after review by our CAB and SAB.

Table 1.

Integrated changes to the study design.

Community Advisory Board (CAB) requested modifications

1. The Autism Diagnostic Observation Schedule 2.0 is time and resource-intensive for a study where an educational record of autism is available. The Social Responsiveness Scale (2nd Edition) was proposed and accepted as a replacement.

2. The MATRICS Cognitive Battery is time and resource-intensive and the CAB requested a shorter battery. The NIH Toolbox was proposed and accepted as a replacement.

3. Advised the study team to collaborate with school partners when devising the suicide risk safety protocol.

4. Autistic board members reviewed and revised study measures for appropriateness and language accessibility.

Scientific Advisory Board (SAB) requested modifications

1. The addition of work readiness and work motivation measures.

2. The addition of implementer perceptions of the training provided as well as their perceptions of the effectiveness attributes of the trainers via the Measure of Effective Attributes of Trainers (MEAT) at post-training.

3. Assessing and ranking barriers prior to implementation and during active implementation, informed by the updated Consolidated Framework for Implementation Research (CFIR).

4. Assessment of implementer behavioral beliefs regarding implementing VIT-TAY, drawing from behavior change theories (Health Action Process Approach; Theory of Planned Behavior).

2.2.1. Eligibility criteria

Study inclusion criteria involve the following: 1) 16–26 years of age; 2) enrollment in pre-employment transition services; 3) Autism Spectrum Disorder diagnosis based upon a cutoff score of at least 60T on the Social Responsiveness Scale (2nd Edition; SRS-2 [24]) or via education record; 4) fluency in English with at least a 4th grade reading and comprehension level measured by the Wide Range Achievement Test, 5th edition (WRAT-5) [25]; and 5) unemployed or currently employed and job-seeking. Participants will be excluded due to: medical illness that may significantly compromise cognition (e.g., moderate or greater traumatic brain injury), uncorrected vision or hearing problems that prevent using the VIT-TAY software, below 4th grade reading level, and either having no verifiable contact information or being unwilling to provide a verifiable collateral contact in order to maintain contact throughout the 9-month follow-up.

2.2.2. Participant recruitment, screening, and randomization

A target sample of 200 autistic transition-age youth will be enrolled and randomized. We will recruit approximately 15–20 autistic transition-age youth from each of the 16 schools or agencies across two states delivering Pre-ETS using an existing infrastructure and approach developed by our Community Advisory Board from Dr. M. Smith's prior RCT (R34 MH111531) [14] and Dr. Baker-Ericzén's pilot RCT (R34 MH111491). Thus, the required sample size will be met based on available autistic youth, a 24-month recruitment period, and the use of existing recruitment infrastructures. Based on the prior experience and success of our teams with some of the same schools, we anticipate our study can use our existing methods (e.g., presentations, mailers, parent email distribution lists, and onsite recruitment at school functions and through teacher support) to inform families and enroll (and randomize) 200 participants with an estimated 20 % attrition by 9-month follow-up. Written informed consent will be obtained from each participant. Eligibility assessments will include screening for reading level and cognition. Study measures will include completing a demographic survey, employment history, mental health, work readiness, and autism traits. To maintain balance and stabilize staffing for role play across time, the biostatistician will create a computer-generated 1:1 randomization scheme for each school/Pre-ETS program site with sealed envelopes protecting the randomization by participant ID number will be prepared prior to administering interview role plays.

2.3. Study interventions

2.3.1. Virtual Interview Training for Transition-Age Youth (VIT-TAY) condition

VIT-TAY is a job interview simulator where trainees practice interviewing with two virtual hiring managers and receive four levels of automated feedback. The process is as follows: 1) trainees review an e-learning curriculum that includes 10 job interview skills based on Huffcutt's job interview framework and focus on job-relevant content and interviewee performance [e.g., rapport, professionalism, dependability, teamwork] [26,27]; 2) trainees complete an online application for one of 14 jobs where the virtual hiring managers will ‘remember’ the application answers; and 3) trainees interview with the virtual hiring managers named Rita Muniz or Travis Bishop. The VIT-TAY platform (accessed via web browser) provides a videoconference-like interview experience where trainees repeatedly practice interviews with Rita or Travis where they speak a response selected from 6 to 8 scripted choices (ranging from very ineffective to very effective). There are four feedback levels. Level 1 reflects trainees receiving real-time nonverbal cues from an onscreen job coach (Kendra; who provides a social story component) as to whether their choice was effective or ineffective. Level 2 is a transcript where Kendra provides feedback on the specific choices spoken or chosen by trainees to Rita or Travis. Level 3 reflects summary feedback (provided by Kendra via video clips) on how the spoken choices address each of the 10 job interview skills. Level 4 reflects a numerical score across all 10 skills. Notably, the 10 job interview skills are scaffolded across easy (4 skills), medium (7 skills), and hard interviews (10 skills). Difficulty levels vary with the personalities of Rita or Travis that are initially randomly selected for easy (friendly), medium (friendly, business professional), and hard (friendly, business professional, unpleasant), but the mood of Rita and Travis can then change in response to trainee answers (e.g., Rita begins unpleasant but becomes friendlier as trainees make positive statements).

2.3.1.1. VIT-TAY implementation plan

2.3.1.1.1. VIT-TAY Orientation and didactics

VIT-TAY trainees will review the approximately 3 h of e-learning job interview didactics and complete their virtual interviews using a hierarchical learning model. They will be asked to complete a minimum of three “easy” interviews (10 min each) until they score at least 85 of 100 points. Then they will complete a minimum of three “medium” interviews (15 min each) until they score at least 85 of 100 points. If a trainee does not score 85 (at easy or medium) after three tries, we recommend that they will be allotted a 4th or 5th attempt before progressing to the next difficulty level. Finally, they will advance to “hard” interviews (25 min each) and complete at least three interviews and score at least 85 before the intervention is concluded. We expect autistic youth to spend approximately 10 min reviewing their transcripts and feedback after each interview. This recommendation was based on our real-world VIT-TAY implementation for 356 youth with disabilities at 32 schools where a dosing analysis identified that approximately three interviews at each difficulty level was the most efficient to gain competitive employment [28]. We anticipate this recommended approach can be tailored for autistic youth who may need more (or less) practice to score 85. This training will occur between pre-test and post-test data collection (see below). Our initial VIT-TAY studies [20,29] revealed that teachers led approximately two sessions (45 min per session) per week over 4–6 weeks (70 % of autistic youth used VIT-TAY independently by session 3). The range in weeks reflected the customizability of VIT-TAY delivery to meet individual autistic youth needs.

2.3.1.1.2. VIT-TAY Manualized support strategies

Manualized strategies [30] will be used to prepare teachers to train autistic youth to use VIT-TAY (e.g., 45 min VIT-TAY orientation, role-plays, using VIT-TAY themselves). Teaching/Pre-ETS staff will spend approximately 10 min helping autistic youth review their VIT-TAY transcripts. They will be trained to monitor VIT-TAY progress through an administrative portal on the VIT-TAY website (access student scores and transcripts) to guide student support. Teaching/Pre-ETS will use the VIT-TAY adherence checklist to ‘orient’ students to: a) use e-learning, b) complete the job application, c) use voice-recognition, d) use VIT-TAY interface, and e) review transcripts. This checklist will serve as written assurance that VIT-TAY was delivered with fidelity. The PI (MJS) or Co-I (MBE) will meet weekly with the site lead to verify that procedures are regularly observed and site leads will receive biweekly supervision to monitor fidelity to teaching autistic youth how to use VIT-TAY.

2.3.2. Active control condition

Teaching/Pre-ETS staff are anticipated to administer 1 to 2 job interview role-plays with feedback (approximately 30 min) as part of standard Pre-ETS (or transition services). The active control will include approximately 3 h of VIT-TAY e-learning job interview tips without practicing VIT-TAY virtual interviews and approximately 3 h of career pathway videos via the YouTube channel Autism Career Pathways (a series of more than one hundred 15-min video conversations with autistic adults about navigating their pathway to a career). Partnering sites will also provide any other existing Pre-ETS elements as usual in both conditions. Given that Pre-ETS will have school-level variation in the depth and breadth of interview preparation content, our design using the VIT-TAY e-learning and YouTube videos will standardize some components of Pre-ETS provided to all students. A separate online portal will be created to deliver the active control content. Time spent using this content will be auto-tracked via the portal and via a paper and pencil form by participants.

2.4. Study measures

The proposed study assessment schedule is displayed in Table 2.

Table 2.

Schedule of assessments.

Study Measures	Instrument	Collection Method	Timing
			T1	T2	T3
Background
Diagnosis	Educational Record, Social Responsiveness Scale (2nd Edition)	Administrative records, Parent/Teacher Report	X
Reading Level &
Comprehension	Wide Range Achievement Test (5th Edition)	Assessment	X
Demographics	Background survey	Self-Report	X
Employment history	Employment History Interview	Interview	X
Neuropsychological function	National Institute of Health Tool Box - Cognition	Assessment	X
Social Cognition	Bell-Lysaker Emotion Recognition Task	Assessment	X
Depressive symptoms	Patient Health Questionnaire – 9 item	Self-Report	X	X
Anxiety	Generalized Anxiety Disorder – 7 item	Self-Report	X	X
Mental Health	DSM-V Cross Cutting Symptom Assessment	Assessment	X	X
	World Health Organization Disability Assessment Schedule 2.0	Self-Report	X	X
Suicidality Assessmenta	Columbia Suicide Severity Rating Scale	Assessment	X	X
Primary Outcomes
Job Interview Skills	Autism Mock Interview Rating Scale	Assessment	X	X
	MOJO - iSkills	Self-Report	X	X
Competitive Employment	Recent Employment Interview	Interview		X	X
Interview Skills Knowledge	Interview Skills Knowledge Check	Self-Report	X	X
Secondary Outcomes
Job Interview Anxiety	MOJO - iAnxiety	Self-Report	X	X
Job Interview Motivation	Intrinsic Motivation Inventory	Self-Report	X	X
Exploratory Outcomes
Work Readiness	Work Readiness Scale	Self-Report	X	X
Work Motivation	Work Motivation Scale	Self-Report	X	X

Abbreviation. MOJO-iSkills: Measure of Job Interview Skills; MOJO-iAnxiety: Measure of Job Interview Anxiety; T1 = Pre-Test; T2 = Post-Test; T3 = Nine-month follow-up.

^aIf needed.

2.4.1. Eligibility and background measures

First, trained research staff will obtain informed consent and then confirm eligibility of autism through educational records or the SRS-2. Next staff will assess participants’ reading level via the WRAT-5 [25]. After eligibility is confirmed, staff will gather demographic and background information through self-report and assessment (e.g., age, race, educational history, neuropsychological function, mental health, parental socioeconomic status). To assess neuropsychological function (i.e., cognitive flexibility, attention, episodic memory, working memory, processing speed, decoding skills, and receptive vocabulary), we will use the NIH Toolbox [31,32], which has previously been used in the study of autism [14].

The mental health self-reports are required for all research projects funded by the National Institute of Mental Health and will be submitted to the National Institute of Mental Health Data Archive (NDA). The required measures included the Patient Health Questionnaire 9 (PHQ-9) [33,34], the Generalized Anxiety Disorder 7 (GAD-7) [35], The Diagnostic and Statistical Manual of Mental Disorders, fifth edition, text revision (DSM-5-TR) Level 1 Cross-Cutting Symptom Measure [36], and the World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0) [37]. The sponsor also required the development of a suicide prevention assessment protocol that was developed (by Co-I LB) using the Columbia Suicide Severity Rating Scale [38].

Lastly, the research team will administer a baseline employment history interview to assess competitive integrated work history and vocational training history (e.g., recent employment, wage, hours worked, prior job interview training). Competitive, integrated employment will be defined by the 2014 Workforce Innovation and Opportunities Act [9] in that the job pays at least minimum wage, occurs in an integrated community setting, and is not set aside for persons with disabilities.

2.4.2. Primary outcome measures

Participants will complete all outcome measures at both pre-test and at post-test (approximately 6–8 weeks after randomization when the intervention or active control will be completed). Trained research staff will administer the employment interview (in case employment status changes). Then the staff will administer the empirically validated Autism Mock Interview Rating Scale (A-MIRS; 7-point Likert scale; 11 items [e.g., professionalism, teamwork]) to assess performance-based job interview skills [17]. Participants will select from 14 job scenarios and complete a job application to guide their role-play. A trained research staff member will perform the job interview role-plays. Role-plays will be video-recorded and rated for fidelity to a standard interview by trained research staff [built into A-MIRS training]. Fidelity to the standard interview requires the role-play actor to ask 100 % of the required questions and at least three follow-up questions. Role-play ratings are highly reliable, stable over time, strongly related to naturalistic social interactions and objective measures of community functioning, and sensitive to the effects of social skills training [39]. The lab manager will regularly meet with the role-play raters, review ratings, and discuss discrepancies to prevent drift. Participants will be instructed not to disclose their randomization group, but such disclosures are possible. If this occurs, interviewers will notify research staff and the video will be edited to mute the disclosure to maintain rater blinding. To maintain fidelity, video raters will be masked to group status.

2.4.3. Secondary and exploratory outcomes

Job interview anxiety will be measured using the Measure of Job Interview Anxiety (MOJO-iAnxiety; 11-item total score; 5-point Likert scale), which has strong psychometric properties and is sensitive to change over time among autistic youth [14,18]. Job interview motivation will be measured with an adapted version of the interest/enjoyment subscale from the Intrinsic Motivation Inventory (IMI; 7-item total score; 7-point Likert scale) [40]. Items focus on motivation to ‘interview for a job’ and to ‘get a job.’ The IMI has previously been adapted for autism research [[41], [42], [43]]. A review of the literature did not generate a brief assessment of general job interview preparation knowledge. Thus, we developed our own 12-item measure based on VIT-TAY's interview basics curriculum. The aforementioned employment history interview will be administered at each timepoint as competitive, integrated employment is a primary outcome measure. Work Readiness will be measured by the 13-item Work Readiness Scale [44] with items capturing stages of motivation to make changes in one's life related to employment including: pre-contemplation, contemplation, preparation, action, and maintenance. Motivation to work will be measured using the 6-item Work Motivation Scale [45].

2.4.4. VIT-TAY and SAU process measures

VIT-TAY will automate process measures (e.g., interview scores, minutes engaged in virtual interviews, number of interviews completed) that can be downloaded in a spreadsheet. Meanwhile, we will use a standardized transition plan scoring rubric to quantitatively assess each participant's services as usual [46]. We will also survey whether staff have been trained on job interview role-playing techniques, the extent of this training (e.g., hours trained, estimated number of role-plays conducted over the past 6 months), their perceived comfort level at facilitating role-plays, their perceived self-efficacy at facilitating role-plays, and how well they feel prepared to give autistic youth feedback on their job interview role-play performance. We will also capture the extent to which staff provided other job interview preparation or training between the pre- and post-test visits. To supplement this teacher-level data, we will survey RCT participants on job interview skill and pragmatic social skill training they received between the pre-test and post-test visits.

2.4.5. Nine-month follow-up measures

Employment outcomes (e.g., obtaining competitive integrated employment, start date) will be tracked during the post-training period between post-test and nine-month follow-up. Additional data will be obtained for descriptive purposes (e.g., job types, hours worked, wage). Job type will be coded via the Dictionary of Occupational Titles [47].

2.5. Implementation evaluation measurement procedures and instruments

Drawing from our pilot RCTs and our Community Advisory Board discussions, we developed our proposed implementation evaluation plan with determinant (barrier/facilitator) measures (Table 3) [[48], [49], [50]] and outcome (see 2.4.1) and process (see 2.4.3) measures [[51], [52], [53], [54], [55]]. For determinant evaluation, n = 16 administrators and n = 16 Pre-ETS/teaching staff (1 each per school) will be interviewed (approximately 45 min) after VIT-TAY implementation and again at 9-month follow-up. Questions will be drawn from the updated CFIR Interview Guide [56,57]. Interviews will be conducted by trained interviewers at the University of Michigan. We will attempt to interview the same person at each time point. Meanwhile, autistic youth will complate a semi-structured interview with questions focused on VIT-TAY characteristics and implementation determinants (i.e., barriers and facilitators of delivery). RCT data will inform youth characteristics (Table 3). In addition to the determinant constructs in Table 3, post-implementation interviews will include questions pertaining to fidelity and adaptation to a prospective delivery plan. Implementation surveys for administrators (n = 16) and Pre-ETS/teaching staff (n = 16) will occur after implementation with a maintenance survey collected at 9 months.

Table 3.

Mixed method implementation evaluation.

Implementation Evaluation Construct	Type of method, source of data, examples
	Quantitative Data	Qualitative Data
Evidence-Based Practice Attitudes and Perceived Need	Pre-Training Questionnaire (sample question: “To what extent do you feel there is a strong need for a new method of teaching job interview skills to your students?”)	Open ended survey question regarding implementing new methods for teaching job interview skills
Behavioral Beliefs (Outcome Expectancies, relative advantage, self-efficacy, intentions)	Post-Training and Ongoing Implementation Questionnaire (sample questions: “VIT-TAY offers significant potential to improve outcomes for my students”; “I have every intention of using VIT-TAY with my students”)	Post-Study semi-structured interviews
Perceptions of Trainer Attributes	Post-Training assessment using the Measure of Effective Attributes of Trainers (MEAT); “please rate your VIT-TAY trainer on the following attributes: engaging, humble, skillful, expert, passionate …”	N/A
Barriers to Adoption and Implementation	Post-Training anticipated barriers to adoption and Ongoing and Post Implementation actual barriers to adoption/implementation (users select from a list of barriers and rank their intensity of impacting implementation (e.g., not enough time, doesn't fit with workflow, isn't effective, etc.).	Open ended survey question regarding additional barriers or expansion and explanation of barriers selected Post-study semi-structured interviews using CFIR guide
Implementation Strategy Acceptability, Appropriateness, Feasibility (AAF)	Ongoing Implementation and Post-Implementation: AAF of Implementation Manual and Admin Center; Frequency of reference to the implementation manual	Open-ended survey question regarding opinions of the VIT-TAY manual and/or use of. Post-study semi-structured interviews
Fidelity and Adaptation	Post-study FRAME assessment (“did you make any of the following modifications to VIT-TAY when you delivered it …”)	Open-ended survey question regarding expansion or further description as to modifications made; Justification for modifications made
Intervention Normalization	Post-study NoMAD questionnaire, 20 items, assesses multiple aspects of technology-based intervention integration within routine ETS delivery	N/A
Maintenance/Sustainment of VIT-TAY	Post-study Program Sustainability Assessment Tool 7 domains (3 items each) (e.g., Workflow Integration; Monitoring and Evaluation; Organizational Context and Capacity)	Post-study semi-structured interviews
Reach	1. Proportion of eligible autistic youth who engage in the intervention (complete ≥7 VIT-TAY interviews)	N/A
	2. Representativeness of autistic youth who engage (e.g., identify potential disparities)	N/A

2.6. Data management and monitoring

In this intent-to-treat RCT, we will collect data on 200 participants, and assuming 20 % attrition by T3, we will have a primary analytic sample of n = 160. Prior to analysis [[58], [59], [60], [61], [62], [63], [64], [65]], quantitative data will undergo standard data cleaning techniques and logic checks, non-normal distributions will be transformed as needed, validated scales will be computed and psychometric properties assessed, multicollinearity of predictors will be assessed, and missing data reviewed. We calculated power using STATA [66], and accounted for multiple testing with a conservative Bonferroni alpha correction factor for the 4 primary tests (.05/4, two-sided p = .0125). All videos will be stored on a HIPAA-compliant server. A Data Safety and Monitoring Board will monitor for adverse events.

2.7. Data analyses

This intent-to-treat trial will analyze all participant data regardless of the dosing received for either intervention arm unless participants withdraw their data from the study.

2.7.1. Analyses to test aim 1

To test whether Pre-ETS + VIT-TAY will have higher competitive, integrated employment rates than Pre-ETS + AC by nine-months post-randomization (H1), logistic regression and a Wald Chi-square test will compare the employment outcomes in the two arms (attained a job = 1 vs. failed to attain a job = 0 between randomization and nine-month follow-up). Given that: 1) our pilot RCT found Pre-ETS + VIT-TAY were competitively employed 25 % higher than Pre-ETS only [[14], [19]]; 2) a national employment rate autistic youth of 50 % [67]; and 3) an analysis sample of n = 80 per study arm; the trial has >80 % power for a two-sided alpha of .0125 (.05/4) level test to detect a significant differences in employment of at least 25 % in bivariate or multivariable adjusted analysis. Covariates will include variables that have been empirically related to obtaining employment (e.g., autistic traits, cognitive ability, vocational history, mental health, parental socioeconomic status, work readiness, work motivation) [13,67,68].

To test whether Pre-ETS + VIT-TAY will have greater improvement in job interview skills than Pre-ETS + AC (H2) we will conduct a repeated measures analysis of variance (RM-ANOVA) with pre and post interview scores as the repeated measures and treatment group as the between groups factor. Our pilot RCT data found r = .7 between pre-test and post-test scores, and a partial eta-squared group-by-time large effect size (ηp² = .15) between pre and post interview role-play scores by group. Even given the large, repeated measures correlation and a corrected two-sided alpha of p = .0125, the trial has >90 % power to detect significance given the expected group by time effect size (ηp² = .15) between pre-test and post-test and >80 % to detect significance given a smaller effect of ηp² = .05. Approach to covariates are noted above.

To test whether Pre-ETS + VIT-TAY will get jobs sooner than Pre-ETS + AC by nine-month follow-up (H3), we will use a Cox proportional hazards regression model [69] to assess the hazard rate on time-to-employment between groups in bivariate and multivariable analyses. For a power analysis, we assume a constant hazard rate for the distribution of time-to-employment or censoring in both groups between randomization and nine-month follow-up, and a Pre-ETS + VIT-TAY to Pre-ETS + AC hazard ratio of HR = 1.80. This analysis is estimated based on a prior study of VIT-TAY and log-rank power tables [70,71]. Thus, 160 autistic youth provide >80 % power for a .0125-level test comparing time-to-employment in the two arms to detect a significant hazard ratio of HR = 1.8 for Pre-ETS + VIT-TAY vs. Pre-ETS + AC. Approach to covariates are noted above.

2.7.1.1. Exploratory analyses for aim 1

We will evaluate if VIT-TAY training (e.g., VIT-TAY scores) is associated with a longer job tenure (total days worked as of right-censored follow-up date) and enhanced social skills using negative binomial models for count data (days) and RM-ANOVA (job interview anxiety, job interview motivation, work readiness, work motivation, and job interview preparation knowledge). Approach to covariates are noted above.

2.7.2. Analyses to test aim 2

Analyses to test Aim 2 are based on the Corbiere et al. [72] model of employment to test whether improved job interview skills will mediate the relationship between VIT-TAY (average VIT-TAY score) and employment outcomes (H6). We will conduct a regression-based mediation analysis [73] assessing the total effect of Pre-ETS + VIT-TAY (X) on employment (Y) by computing both the direct effect (X → Y) and the indirect of Pre-ETS + VIT-TAY on employment outcomes mediated through job interview skills (M) (X → M → Y). An analysis sample of 160 is adequately powered to assess the direct X → Y component (H1) and the direct X → M component (H2). In addition, n = 160 at nine-month follow-up is adequate with a power >.80 to detect a significant relationship between differences in job interview skills of the magnitude anticipated at post-test as posited in H2 and group difference in employment rate at nine-month follow-up posited in H1, the direct M → Y effect. In addition to adequately powered component models, the mediation model will be conducted using bootstrap confidence intervals for indirect effect inference testing, enhancing statistical power.

2.7.2.1. Exploratory analyses for aim 2

We will explore the pathways contributing to employment outcomes by examining separate models of post-test job interview anxiety and post-test job interview motivation serving as mediators of the relationship between VIT-TAY and job interview skills. Evidence of mediation will be followed by probing for conditional model effects moderated by baseline (pre-intervention) anxiety/motivation. As the normality of the distribution of the measured indirect effect is unknown [74] we will identify values of the moderator representing low, medium, and high levels of anxiety or motivation to estimate their conditional indirect effect using bootstrap confidence intervals [75]. We will use the above methods to explore race and autistic traits as possible moderators of VIT-TAY effects on employment.

2.7.3. Analyses to test aim 3- implementation

2.7.3.1. Quantitative analysis

We will adhere to the RE-AIM framework [76] and will use descriptive statistics from validated survey measures administered to all administrators and VIT-TAY teaching staff. Table 3 presents the measures and metrics, data sources, method, and timing of administration.

2.7.3.2. Qualitative analysis

We will audio record and transcribe focus group and open-ended interview data, analyze data iteratively using thematic analysis and the constant comparative approach [77,78] to identify emergent themes. Two research staff will analyze the data using NVivo, a qualitative data analysis software. Staff will independently analyze a subset of transcripts to iteratively develop codes inductively as they emerge, and deductively based on initial topics (e.g., barriers, available resources). After the team agrees on a final codebook and inter-coder reliability is achieved, the codes will be applied to all transcripts [79,80]. To facilitate comparison, framework matrix of themes will be developed to compare autistic youth, teacher, and admin (x-axis) perceptions of barriers and facilitators (y-axis) to VIT-TAY implementation [81]. Matrices will identify y-axis themes common to all groups, and subgroup-specific features (e.g., the experience of implementing VIT-TAY for autistic youth may be related to organizational themes not evident among teachers/administrators) [82].

2.7.3.3. Mixed methods analysis

We will integrate the quantitative and qualitative results by conducting a “Connect the dots” [83,84] convergent mixed methods analysis with sequential-dependent and concurrent-independent data collection. Convergent methods attempt to confirm or disconfirm findings from different data sources. While our design is intended to be fixed (i.e., evaluation schedule and measures are prospectively planned), ongoing analysis of data as it is collected allows for an emergent design in which the design is modified to gather data on themes or findings from the data collected. This mixed methods analysis method was selected to provide greater depth into the findings of quantitative results with the richness of qualitative data.

3. Discussion

Although policy supporting technology-based interventions to support autistic transition-age youth is lacking [85], these interventions are becoming more widely accessible to the general public. Thus research is needed to evaluate the effectiveness and implementation of these technologies on school-centered employment services. This study will be among the first to evaluate the behavioral and the implementation potential of a VIT-TAY to enhance Pre-ETS for autistic transition-age youth in a fully-powered randomized controlled sample. Our use of the Hybrid Type 1 effectiveness-implementation approach is innovative, as we will evaluate both whether VIT-TAY improves job interview skills, interviewing anxiety and employment outcomes. We will also evaluate VIT-TAY implementation by (a) identifying the multilevel barriers and facilitators affecting implementation, (b) determining the acceptability, feasibility, and appropriateness of using an implementation manual and the admin center, (c) determining VIT-TAY's scalability across Pre-ETS programs and generalizability to other vocational rehabilitation settings, as well as (d) its sustainability.

3.1. Potential enhancements to Pre-ETS

Assuming that VIT-TAY effectively improves job interview skills and access to employment as well as reduces interview anxiety and time-to-employment among autistic transition-age youth, the findings from this study have the potential for a widespread impact within the Pre-ETS model of service delivery facilitated by high schools. While Pre-ETS funding is typically distributed to state vocational rehabilitation, and centers for independent living, etc., it's use varies from state to state. Hence, we are evaluating VIT-TAY with teachers in schools under the umbrella of transition services because it is likely to provide a more consistent analysis which can then be adopted by other schools. Thus, our proposed evaluation of VIT-TAY within Pre-ETS has tremendous potential to highlight a strong evidence-based practice to enhance employment outcomes for autistic transition-age youth.

The scientific team also made multiple revisions that were approved by the funding agency. First, the scientific team added work motivation [44] and work readiness [45] as two exploratory outcomes. Third, given recent evidence from over 100 autistic transition-age youth and 60 Pre-ETS staff that VIT-TAY was acceptable, appropriate, and feasible to deliver [20,29], the implementation science team (JM, MJS, JDS) made slight modifications to the implementation evaluation plan. Specifically, existing items were reframed to gather initial implementation outcomes related to a new VIT-TAY implementation manual. The full implementation assessment plan is displayed in Table 3. Given the limited measurement of multilevel determinants of VIT-TAY implementation, we include an assessment of determinants at two time points (pre-implementation and at the mid-point of implementation) to capture anticipated and actual determinants, which was developed using the updated CFIR [86]. Other notable additions to the evaluation plan included a brief pre- and post-training assessment of behavioral beliefs (e.g., teacher and staff perceptions of current practices for teaching job interview skills as well as the need for new methods) to capture individual-level behavioral mechanisms of adoption and implementation among implementers. These items were developed using constructs from two behavior change theories, The Theory of Planned Behavior and the Health Action Process Approach [87,88]. Together, these theories outline motivational and volitional factors that influence behavioral intentions and performance. Finally, we included a brief assessment of participant perceptions of the training they received as well as the attributes of the trainers [89]. These assessments will allow for a more formal evaluation of the process of implementation at post-study, which will inform future implementation trials.

3.1.1. Limitations

Given individual randomization, minor indirect contamination is possible as Pre-ETS/teaching staff are required to practice VIT-TAY themselves. This practice could alter their role-play ability or strategies with students. However, our Community Advisory Board indicated this type of contamination would not increase the number of role-plays students complete with teachers. We will mitigate contamination risk through additional training with teacher and monitoring for signs of contamination via a teacher checklist, which we successfully implemented during our pilot RCT. Risk of contamination is not limited to role plays, but to providing Pre- ETS and job support programming as the staff will be supporting both intervention and active control participants since randomization is at the individual level and both groups of students could be segregated within the same classroom. This contamination level is tolerable for the proposed study design and offset by the use of headphones, the active intervention and control groups work independently in their own spaces with headphones and their own devices to minimize contamination and avoidance of other potential sources of bias and loss of statistical power via cluster randomization [90,91].

3.1.2. Dissemination of findings

Our dissemination plan addresses five goals: 1) expand the Reach (R of RE-AIM) of our results via face-to-face presentations and webinars within the autism and scientific communities; 2) increase the Reach of evidence by targeting diverse stakeholders: autistic youth and young adults, job coaches, and administrators from employment agencies, Pre-ETS programs, and state departments of rehabilitation; 3) increase capacity to leverage study results by developing user-friendly evidence reviews and practice guidelines that stakeholders can understand; 4) leverage the expertise and networks of our CAB and SAB to expand the reach of our dissemination efforts; and 5) expand our website's online research archive (https://leveluplab.org) to share activities (e.g., webinars, trainings) and written materials.

If the hypotheses are supported, at least five scientific publications are likely to emerge: 1) the impact of VIT-TAY on primary employment outcomes; 2) VIT-TAY as a mechanism of change in employment; 3) the barriers and facilitators of VIT-TAY implementation in Pre-ETS, 4) whether VIT-TAY could be more effective in subgroups of participants, and 5) the broad potential of virtual simulation technology to teach important adaptive skills for autistic youth. Lastly, study results will support the dissemination, implementation, and integration of VIT-TAY into Pre-ETS, and inform adoption of VIT-TAY by other Pre- ETS programs and schools as the standalone design of VIT-TAY makes it easily generalizable across different types of transition service settings.

3.2. Conclusion

This is the first fully-powered RCT to assess VIT-TAY in Pre-ETS and transition service programming. The intentional use of the Hybrid Type 1 effectiveness-implementation study evaluation design aims to expedite translational processes [92] by identifying significant delivery facilitators and barriers. Thus, the goal is to assess whether VIT-TAY positively impacts obtaining employment for autistic transition-age youth in Pre-ETS and transition services, with the success of this rapid translation contingent on the intervention's effectiveness and the identification of salient factors through implementation evaluation to support future adoption.

CRediT authorship contribution statement

Matthew J. Smith: Writing – review & editing, Writing – original draft, Project administration, Methodology, Investigation, Funding acquisition, Conceptualization. James L. Merle: Writing – review & editing, Writing – original draft, Methodology, Conceptualization. Mary Baker-Ericzén: Writing – review & editing, Investigation, Funding acquisition, Conceptualization. Kari Sherwood: Writing – review & editing, Funding acquisition, Conceptualization. Lindsay A. Bornheimer: Writing – review & editing, Methodology, Investigation. Brittany Ross: Writing – review & editing, Project administration, Methodology. Meghan Harrington: Writing – review & editing, Methodology. Apara Sharma: Writing – review & editing, Methodology. Cheryl Brown: Writing – review & editing, Conceptualization. Timotheus (TJ) Gordon: Writing – review & editing, Conceptualization. David Telfer: Writing – review & editing, Conceptualization. Jocelyn Reese: Writing – review & editing, Conceptualization. Jennifer Hirst: Writing – review & editing, Conceptualization. Eugene A. Oulvey: Writing – review & editing. Valerie Dignadice: Writing – review & editing, Conceptualization. Ed-Dee Williams: Writing – review & editing, Writing – original draft, Funding acquisition, Conceptualization. Sandra Magaña: Writing – review & editing, Conceptualization. Kara Hume: Writing – review & editing, Funding acquisition, Conceptualization. Connie Sung: Writing – review & editing, Conceptualization. Jane K. Burke-Miller: Writing – review & editing, Writing – original draft, Methodology, Investigation, Formal analysis, Conceptualization. Justin D. Smith: Writing – review & editing, Writing – original draft, Methodology, Investigation, Funding acquisition, Conceptualization.

Funding statements

This study was supported by a grant to Matthew J. Smith (R01 MH132656) from the National Institute of Mental Health. Additional support was provided by the Dissemination and Implementation Science Core (DISC) of the Utah Clinical and Translational Science Institute (UM1 TR004409).

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:

SIMmersion LLC commercially licenses access to Virtual Reality Job Interview Training. Dr. Matthew Smith and his scientific and partners from the autism community adapted this original version into a new iteration called Virtual Interview Training for Transition Age Youth. Royalties from this new version (not the original version) will be shared with the University of Michigan, the University of Michigan's School of Social Work, and Dr. Matthew Smith. The University of Michigan's Conflict Management Office developed a conflict management plan for Dr. Matthew J. Smith that was reviewed and approved by a University of Michigan Conflict of Interest Committee. No other authors report competing interests.

Data availability

No data was used for the research described in the article.