Iterative Redesign of a Caregiver Mediated Intervention for Use in Educational Settings

Abstract

Teachers endorse disruptive behavior as a considerable concern for autistic students, which is compounded by the lack of adequate resources for behavioral intervention planning in the classroom. The RUBI program is an evidence-based, low-intensity manualized intervention, initially developed for parents of autistic children ages 3–14 and co-occurring disruptive behavior. Utilizing the Discover, Design/Build, Test (DDBT) framework, which combines user-centered design and implementation science, RUBI intervention content was collaboratively and iteratively redesigned with elementary school stakeholders (40 school staff from 28 schools) to ensure the feasibility, acceptability, and appropriateness of the redesigned intervention, RUBI in Educational Settings (RUBIES). Iterative quantitative and qualitative methods were conducted with stakeholders to identify targets for RUBI redesign. Conventional content analysis was used to code qualitative data and identify usability issues. Recommendations were provided for modifications to RUBI sessions to address the needs of the school context and end-users to develop RUBIES. Feasibility scores improved following redesign. The use of the DDBT framework to redesign the RUBI intervention may promote greater usefulness and usability in school contexts.

Approximately 50% of autistic children exhibit disruptive behaviors, such as tantrums, aggression, and noncompliance (Hartley et al., 2008; Mazurek et al., 2013), a rate that is higher than neurotypical peers in the school environment (Spaulding et al., 2010). Within the classroom, disruptive behavior is considered one of the most significant barriers to participation for autistic children in general education, as frequent and/or intense problem behavior can impact meaningful engagement in academic tasks and with peers (Horner et al, 2002), as well as placement in more restrictive settings (Sansosti & Sansosti, 2012), higher use of restraint procedures, and suspension, all of which have lasting negative impacts (McIntosh et al., 2014). Child behavior problems also influence teachers’ attitudes toward teaching and job demands, reflected in higher rates of teacher burnout and turnover (Aloe et al, 2014). Finally, peers of children with disabilities may have lower academic outcomes and greater disruptive behavior (Gottfried & Harven, 2015), which may reflect unintended and/or ineffective instructional changes that educators make when school staff lack the resources necessary to meet their students’ behavioral needs.

Autistic children often need highly individualized function-based intervention plans in the school setting. The current usual care model for addressing classroom disruptive behavior in autistic children involves enlisting highly trained staff (e.g., behavioral consultant, school psychologist) to conduct a functional behavior assessment (FBA), which is legally required in instances where a child demonstrates problem behavior that places them at risk for suspension or a more restrictive placement (IDEA, 2004). The FBA identifies the environmental circumstances that may elicit the behavior (antecedents) and the responses that maintain the behavior (consequences) in order to hypothesize its purpose, or function (e.g., escape from unpleasant stimuli, attention from others) as a means to develop an individualized intervention plan. While this model is effective (Martinez, Werch, & Conroy, 2016), the development of function-based intervention plans can be time-consuming and resource-intensive (Sugai et al., 2000), resulting in high costs and delays before the plan can be developed and applied for the student in need. Further, there exists a disparity between service needs of autistic children and the rates of implementation of federally mandated social and behavioral services provided in schools (McDonald et al., 2019; Wei et al., 2014). While there are several behavioral intervention programs in schools and more targeted classroom behavioral management strategies, there may be a need for a paradigm shift in the care model for more severe disruptive behaviors in the classroom. We propose an efficient and pragmatic intervention model that builds internal capacity with direct care providers, which, in turn, streamlines the intervention process, reduces the need for intensive behavioral supports (thus lowering costs), and increases the number of autistic children who can be served.

Over the past decade, the Research Units in Behavioral Interventions (RUBI) Autism Network developed and systematically tested a low-intensity (11-session) manualized intervention for parents of autistic children ages 3 to 14 and co-occurring disruptive behavior (Aman et al., 2009; Bearss et al., 2015; Handen et al., 2015). RUBI is grounded in applied behavior analysis, an evidence-based treatment approach for disruptive behaviors and involves therapists teaching caregivers how to assess the function of their child’s behavior in order to implement behavioral strategies in a targeted manner (i.e., matching strategy to function). RUBI is an outpatient service with weekly 1-hour sessions delivered one-to-one (therapist to caregiver) that utilizes a Behavioral Skills Training (BST) approach, which involves direct instruction, modeling, role-play and practice with feedback, in order to effectively train caregivers in the various RUBI skills (Miltenberger, 2016). This process supports the building of a caregiver behavioral management “toolbox” by: 1) identifying behavioral function to inform strategy choice; 2) decreasing behavioral excess as well as increasing appropriate behaviors; and 3) using positive behavioral supports, such as antecedent management (e.g., use of visual supports), reinforcement, and functional communication as the means to modify behaviors (Reichow & Barton, 2014). Results across two pilot studies and two randomized trials of RUBI demonstrated clinically and statistically significant reductions in child disruptive behavior (within subject effect size ranged from 0.9–2.7; Aman et al., 2009; Bearss et al., 2013; Bearss et al., 2015; Handen et al., 2015). Parents attended a majority (84–93%) of sessions and attrition was low (e.g., 11%; Bearss et al., 2015). Therapist fidelity was high, ranging from 93–97% across the four trials. Further, moderator analyses indicated RUBI is effective with children with average as well as impaired cognitive functioning (IQ < 70) and with children with mild to severe autism symptomatology (Lecavalier et al., 2017), suggesting broad applicability. This research demonstrates that caregivers can be successfully taught to implement a variety of intervention strategies that are grounded in ABA principles (Bearss et al., 2015). However, to date, no studies have evaluated the feasibility or appropriateness of RUBI in schools, the most accessed service system for autistic children (Brookman-Frazee et al, 2009).

Implementation of evidence-based interventions in schools for autistic youth often is fraught with challenges when interventions are not designed with educators in mind and potential barriers to implementation are not mitigated from the start. The literature has documented several barriers to implementation when evidence-based interventions are transported to schools, including: 1) the need for daily use, which requires ample staff that often are not available; 2) extensive data collection requirements, which is difficult for school staff to do; and 3) materials, which may not be available (Locke et al., 2015; Locke et al., 2017). RUBI, as originally designed, does not have these requirements, which may maximize its usability and usefulness in schools once redesigned to fit the needs of the setting (Brookman-Frazee et al., 2010; Kucharczyk et al., 2015; Locke et al., 2015). However, while RUBI addresses salient disruptive behaviors, novel adaptations are a prerequisite to transition the intervention from use in clinics to schools and from parents to school staff to avoid the challenges when interventions are not adapted in the context of implementation, which results in poor intervention-setting fit and low uptake (Chambers & Norton, 2016; Lyon et al., 2013; Gottfredson & Gottfredson, 2002).

Purpose of the Present Study

The purpose of this study was to iteratively redesign RUBI for use in school settings using the Discover, Design/Build, Test (DDBT) framework, which combines user-centered design and implementation science to enhance the usability, contextual fit, uptake, and effective implementation of an intervention (Lyon et al., 2019). The Discover phase targets understanding environmental factors, defining the needs of stakeholders, and identifying problems to solve; the Design phase examines information gathered during the Discover phase and brainstorms concepts for potential solutions; the Build phase involves developing low-fidelity prototypes of concepts, testing prototypes, and refining concepts; the Test phase focuses on the development of high-fidelity prototypes and feasibility testing of the innovation with a larger number of end-users in their actual milieu, with an emphasis on user experience, satisfaction, and benefit over alternative or existing processes (Lyon et al., 2019). In collaboration with elementary school stakeholders (e.g., administrators, teachers, paraeducators) using the iterative DDBT redesign process, this study applied a methodological paradigm shift in the way autism interventions are typically redesigned (Chambers & Norton, 2016). Specifically, this is the first study to use these methods from the user-centered design field to give stakeholders an active role in autism intervention (RUBI) redesign to ensure the end product (RUBI in Educational Settings; RUBIES) is practically relevant, contextually appropriate, and targeted to their specific needs, settings, and priorities.

The study targeted four aims that aligned with the first three phases of the DDBT framework: Aim 1) (Discover) Identify current contextual constraints relevant to behavioral intervention planning for disruptive behavior in the classroom to help provide a “road-map” for required adaptations to RUBI; Aim 2) (Discover) Identify targets for redesign using in-depth intervention demonstrations of RUBI with school staff to inform adaptation and pruning needs, defined as the strategic de-adoption of practices to offset the potential for implementation overload (Cook et al., 2019) related to RUBI content and structure to ensure the redesigned RUBI curriculum would be contextually appropriate for use in schools; Aim 3) (Design/Build) Iteratively and collaboratively redesign RUBI content with school staff based on continual user testing to improve the hypothesized mechanisms of redesign (usability) and perceptual implementation outcomes (feasibility, acceptability, and appropriateness); and Aim 4) (Design/Build) Conduct in-depth intervention demonstrations of the newly developed RUBI in Educational Settings (RUBIES) program to inform final adaptation and pruning needs related to program content and structure to ensure the redesigned RUBIES curriculum would be contextually appropriate for use in schools. Study goals included the development of a redesigned intervention to address relevant disruptive behavior in autistic children in schools (e.g., a RUBIES manual) and identification of targeted end users.

Methods

Participants

Across all aims, we recruited n=40 school staff, including principals, general and special education teachers, paraeducators, and other school personnel (e.g., counselors, speech pathologists, etc.) that support autistic children in general and special education classrooms from 28 elementary schools (see Table 1 for participant demographics by Study Aim). This variety in participant profession allowed for input from unique perspectives (Krueger & Casey, 2009). Unique participants were recruited across aims, which follows the user-centered design approach to have each new group bring in a fresh perspective, thus minimizing biases that could otherwise be carried over from prior contributions to the redesign process. The four aims were initially conducted in a West Coast public school district. After all aims were completed, the final Design/Build Phase (RUBIES Demonstration Study) was replicated with school staff from a Midwest public school district to collect further feedback on RUBIES in a demographically and geographically unique setting compared to the first RUBIES Demonstration. This additional step allowed us to ensure contextual fit and to achieve saturation.

Table 1.

Demographics

Demographics N (%)	Interviews N=8	RUBI Demonstration Study N=15	Collaborative Redesign N=6	RUBIES Demonstration Study N=12
Gender
Female	8 (100)	13 (86.67)	5 (83.33)	11 (91.67)
Male	0 (0)	1 (6.67)	1 (16.67)	1 (8.33)
Nonbinary	0 (0)	1 (6.67)	0 (0)	0 (0)
Race
White	7 (87.50)	14 (93.3)	5 (83.33)	9 (75)
Black	1 (12.50)	1 (6.7)	1 (16.67)	3 (25)
Ethnicity
Hispanic/Latino	1 (12.50)	1 (6.7)	1 (16.67)	1 (8.33)
Age M (SD)	40.86 (14.10)	44.8 (12.6)	43.5 (16.43)	40.58 (12.06)
Highest Education
Tech/Vocational	0 (0)	1 (6.7)	0 (0)	0 (0)
Some college	0 (0)	1 (6.7)	0 (0)	0 (0)
Bachelor’s	1 (12.50)	3 (20)	0 (0)	3 (25.00)
Graduate	7 (87.50)	10 (66.7)	6 (100)	9 (75.00)
Years ASD Experience	9.75 (6.54)	12.53 (8.07)	10.83 (10.57)	7.75 (6.70)
Educational Role
General Ed Teacher	3 (37.50)	2 (13.33)	3 (50.00)	1 (8.33)
Special Ed Teacher	5 (62.50)	4 (26.67)	2 (33.33)	6 (50.00)
(Vice)Principal	0 (0)	3 (20)	0 (0)	0 (0)
Teaching Assistant/Paraeducator	0 (0)	3 (20)	1 (16.67)	1 (8.33)
Other Role	0 (0)	3 (20)	0 (0)	4 (33.33)

Procedures

Discover Phase: Behavior Observation and Interviews

Participants and Procedures.

We first conducted in-class behavioral observations, followed by retrospective cognitive walkthroughs with general and special education teachers (N=8; 3 general education teachers, 5 special education teachers; observed in 4 general education and 4 self-contained classrooms; 1 special education teacher co-taught in a general education classroom) from 4 elementary schools in order to identify the current practices for classroom disruptive behavioral intervention planning, contextual constraints (e.g., classroom rules, school policies), barriers and facilitators, and targeted values and priorities of the school context. After informed consent was obtained, two observers collected Antecedent-Behavior-Consequence (ABC) data on all disruptive behavior episodes exhibited by autistic students in the classroom over the course of a 2-hour observation (Newcomb & Hagopian, 2018). Observers then met with the teacher and completed a retrospective cognitive walkthrough using a “think-aloud” protocol (Kuusela & Paul, 2000), which involved having the teacher talk through their decision-making processes around managing each documented behavioral episode during the observation. The interview guide also asked: 1) What tools are in place to help with classroom behavioral intervention planning?; 2) What would be helpful to have in place to support behavioral intervention planning?; and 3) What are active gaps in staff processes for managing disruptive behavior? Teachers were offered a $50 gift card for their participation.

Discover Phase: RUBI Demonstration Study

Participants and Procedures.

Targets for RUBI redesign were identified with feedback from N=15 school staff (n=2 general education teachers; n=4 special education teachers; n=3 administrators; n=3 paraeducators; n=3 other educational support) from 15 schools over two separate intervention demonstrations studies of RUBI (n=9 in Group 1; n=6 in Group 2), that used behavioral rehearsal, prospective think-aloud procedures, and structured assessments (Duong et al., 2020). Professional diversity provided a range of perspectives on the deconstruction and initial redesign of RUBI for use in schools. After informed consent was obtained, and prior to the in-person meeting, participants received RUBI program materials in their original form. Participants had three weeks to review the materials.

The four-hour, in-person RUBI demonstration study engaged school staff in order to gather detailed, immediate, and actionable feedback on the RUBI program as originally designed. Specifically, participants viewed presentations on each of four segments of RUBI content (Behavioral Principles, Prevention Strategies, Consequences, Teaching Strategies). The presentation included test scenarios and video vignettes, which informed applicability and usability of different components of RUBI (Lyon & Koerner, 2016; Kuusela & Paul, 2000).

After presentation of each segment, participants provided qualitative feedback, guided by a semi-structured interview protocol that emphasized feedback around potential usability issues, along with generation of potential solutions, including: 1) discussion about the relevance and usability of the original RUBI intervention and its overarching objectives in schools; 2) input regarding the feasibility, acceptability and appropriateness of each RUBI component; and 3) brainstorm of effective strategies or modifications. Participants also completed study measures after each segment (Intervention Usability Scale, Feasibility of Intervention Measure, Acceptability of Intervention Measure, Intervention Appropriateness Measure; Lyon, 2021; Weiner et al., 2017). Summary statistics from the quantitative ratings for each RUBI segment were presented to participants to spur discussion and yield insights into score variations, such as qualitative explanations for high/low ratings, as well as to solicit recommendations to revise specific components of RUBI. School staff were provided food and offered a $200 gift card for their participation.

Design/Build Phase: Collaborative Redesign

Participants and Procedures.

As part of the RUBI redesign process, N=6 staff (n=3 general education teachers; n=2 special education teachers; n=1 paraeducator) from 5 schools were invited to attend one of three 2-hour in-person collaborative redesign feedback sessions. The third collaborative redesign feedback session was interrupted by COVID-19 and held via Zoom. Each meeting covered a particular RUBI topic (Prevention, Consequences, or Teaching Skills), where discussion emphasized feedback around potential usability issues for that topic, applications of each section, and generation of new, relevant school-based examples (Xie et al., 2016; Lyon & Bruns, 2019), along with generation of potential solutions, including input regarding the feasibility and appropriateness of the proposed RUBIES components. As an incentive, participants were offered a $75 gift card.

Design/Build Phase: RUBIES Demonstration Study

Two new groups of school staff (N=12; n=6 per group; n=2 general education teachers; n=6 special education teachers; n=1 paraeducators; n=3 other educational support) from 12 schools were recruited for two separate in-depth intervention demonstration studies of the redesigned RUBIES curriculum in order to further refine the manual, enhance intervention-setting fit, and identify targeted end-users (Lyon et al., 2019). Participants were recruited from both a West Coast public school district and a Midwest public school district to optimize the contextual fit of RUBIES implementation. These demonstration studies were conducted virtually due to COVID-19 school closures. This second round of demonstration studies followed the same procedures outlined in the initial Discover phase. Targeted areas of inquiry included potential usability issues of RUBIES along with generation of potential solutions. Summary feedback guided development of the final version of RUBIES, including potential adaptations to fit classroom and end user identification. As an incentive, participants were offered a $200 gift card.

Measures

Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM; Weiner et al., 2017)

The AIM, IAM, and FIM are measures of implementation outcomes that often are considered “leading indicators” of implementation success (Proctor et al., 2011). On each measure, raters score 4 items on a 5-point scale ranging from “Completely Disagree” to “Completely Agree.” Higher scores indicate greater acceptability, appropriateness, and feasibility, respectively. Items can be modified to specify a referent organization, situation, or population.

The AIM, IAM, and FIM can be administered to stakeholders to determine the extent to which they believe an intervention (e.g., RUBI) is acceptable, appropriate, and feasible. The AIM, IAM, and FIM has demonstrated strong psychometric properties: AIM internal consistency α = .89; AIM test-retest reliability α = .83; IAM internal consistency α = .87; IAM test-retest reliability α = .87; FIM internal consistency α = .89; FIM test-retest reliability α = .88 (Weiner et al, 2017). The AIM, IAM, and FIM items also demonstrated good to excellent internal consistency reliability across phases in the current study (AIM α=0.87–0.98; IAM α=0.90–0.94; FIM α=0.89–0.96).

Intervention Usability Scale (IUS; Lyon, 2018)

The IUS is an adapted version of the well-established System Usability Scale (Sauro, 2011), a 10-item measure of the usability of digital technologies. Odd items (1, 3, 5, 7, 9) are totaled and then five is subtracted. Even numbers (2, 4, 6, 8, 10) are totaled and then subtracted from 25 to account for reverse scoring. The two numbers are then added together and multiplied by 2.5 to obtain a final usability score. Scores range from 0–100 with <50 indicating unacceptable usability and >70 acceptable. Used in 500+ studies, the IUS is the best-researched usability measure (Sauro, 2011). It has demonstrated good internal consistency in previous research (α=.83; Lyon et al., 2020). IUS items also demonstrated acceptable to good internal consistency reliability across phases in the current study (α=0.76–0.84).

Data Analysis

Qualitative Data Analysis for all Aims

The classroom interviews, redesign sessions, and demonstration studies were audio recorded and transcribed. Conventional content analysis was used, in which meaning is derived from the content of verbal communications, but no a priori codes are identified prior to reviewing transcripts (Hsieh & Shannon, 2005). Led by the senior author, the research team met to develop a preliminary codebook that could be applied across all qualitative data. Three coders coded all data, and interrater reliability was calculated (MacPhail et al., 2015). The coders met to discuss, clarify, verify, and compare codes; disagreements were discussed with the entire research team to attain consensus. Percent agreement was calculated; average agreement was 99% across all codes. Coders applied the final codebook to all transcripts. Themes were refined throughout the analytic process (Bradley et al., 2007), and data saturation was reached at the point at which no new insights were obtained and no new themes were identified when the codebook was applied across the text segments (Saunders et al., 2018).

Quantitative Analysis for all Aims

All quantitative ratings (IUS, FIM, AIM, IAM) of RUBI and RUBIES were interpreted using the mean of responses on each of the measures to identify targets for RUBI redesign.

Statement of Community Involvement

Community providers (school personnel) were actively involved in each phase of the study; however, autistic people and their family members were not involved given that the purpose of the study was to redesign the intervention for use in elementary schools among educators.

Results

The systematic coding of transcripts across the four study phases resulted in nine themes that contributed to identifying targeted usability issues as well as informing manual modifications and identification of targeted end users: Functions of Behavior, Prevention, Consequences, Teaching Skills, Generalization and Maintenance, School Policies, Barriers and Facilitators, Implementation Decision-Making, and Outcomes/Goals (Supplemental Table 1 provides an outline of the nine themes with examples across the four study phases; Supplemental Table 2 outlines definitions for all themes, the number of coded segments per theme by study phase, and text examples demonstrating how stakeholder feedback informed RUBIES manual modifications and end users).

Intervention Usability Issues

RUBI Modifications were informed by the identification of two primary usability issues:

Time Demands for Creating Visuals

“There’s just a huge barrier to who’s going to make those visuals. And even if you found something online for free, people aren’t sure those are the right thing to use because this is all new to them,” noted a school staff member from the RUBI Demonstration study. A consistent finding across study phases was the value of individualized visual supports, which are a core component of RUBI, compounded with the acknowledgment that they require a significant amount of time to develop. Specifically, school staff do not have the time to find graphics, and then print, laminate, cut, Velcro and organize them into personalized visual cues. To address this practical barrier and increase usability, RUBIES now includes a “visual support toolkit,” which contains a broad set of premade, laminated visual supports to increase ease of use.

Inclusion of Culturally Responsive Text and Pictorial Examples

“[It would be great] to see if we could have some culturally responsive pictorial representation when we talk about daily schedules and things like that,” mentioned staff from RUBIES demonstration study. As designed, staff noted that RUBI is infused with exemplars that illustrate core behavioral principles that represent dominant groups, which reduces the likelihood that RUBIES will be culturally responsive to diverse populations when used in elementary schools. To address this feedback, all RUBIES activity sheets and examples have been co-constructed with stakeholders and modified to include exemplar student names, pronouns (i.e., from primarily “he” to “they”), diverse pictorial representations of students and staff as well as premade visual supports (e.g., visual schedule) that are designed to represent the diversity of student and staff.

Additional RUBI Modifications

Table 2 outlines changes in session content from the original 11-session RUBI intervention to the newly redesigned, 8-session RUBIES intervention. Modifications were informed, in part, by ensuring strategies are broadly applicable across ages and grades as well as congruent with school policies. For example, as several stakeholders expressed concerns about physically prompting students, this was removed as a prompting strategy in the revised manual.

Table 2.

RUBI to RUBIES Redesign

RUBI	Modification	RUBIES
	New Module	Autism 101
Behavioral Principies		Behavioral Principles
Prevention Strategies		Prevention Strategies
Daily Schedules		Daily Schedules
Reinforcement 1	Combined Modules	Reinforcement
Reinforcement 2
Planned Ignoring	Combined Modules	Planned Ignoring + FCT
Functional Communication Training
Compliance Training	Removed Physical Prompting	Compliance Training
Teaching Skills 1	Removed Modules
Teaching Skills 2
Generalization and Maintenance		Generalization and Maintenance

Session Modifications

Inclusion of School-Specific Examples.

Throughout the four study phases, school-specific examples of behaviors and antecedents were derived in partnership with school staff and collated to inform replacement or modification of session content. For example, RUBI Session 1 (Behavioral Principles) reviews antecedents like “passing a playground on the way to the doctor’s office.” In relation to the Teaching Skills content, one teacher from the RUBI demonstration study commented, “So maybe it’s not so much brushing your teeth and putting your shirt on, but how do you sit next to somebody at circle.” Examples throughout the RUBIES manual were reconfigured to be appropriate for the school context (e.g., “passing the playground on the way to the library”).

Addition of an Autism 101 Module.

Stakeholders acknowledged across three study phases that information on neurodevelopmental disorders broadly, and ASD specifically, was lacking in school staff training. This included initial college-level coursework as well as ongoing professional development opportunities. School staff almost uniformly endorsed an interest in more in-depth training on ASD and noted that paraeducators, who often provide the greatest level of support for autistic students, receive little training in this area. For example, a staff member in the first RUBI Demonstration study noted, “I don’t think a lot of the teachers have this training. They don’t have knowledge about autism. They see the chair being thrown by the child as a bad child.” To address this discovery, the newly-designed RUBIES intervention now includes Autism 101 psychoeducation, which is the first session in the program.

Collapsing Reinforcement from Two Sessions to One Session.

A core component of the second Reinforcement session in RUBI involves teaching caregivers how to use praise to “catch the child being good” as well as engage in nondirective child-led play multiple times throughout the week. When this approach was presented in the initial RUBI Demonstration study, comments were recorded on differences in staff availability to integrate this approach into their day. In a collaborative redesign session, one staff noted: “I’m able to do [special time] three or four times a week with them and then the instructional assistants are able to do it a few more times. But that’s with just two kids…so not everyone gets a daily break.” In response to this feedback and based on the goal of ensuring broad applicability of session content, instruction on targeted “special time” (child-led play) was removed while “catch the child being good” and other individualized choices of child-led activities (e.g., going on a walk, special meal with the educator) was integrated into the first Reinforcement session.

Modification of Compliance Training Procedures.

Compliance Training procedures in RUBI follow a 4-step procedure: 1) get the child’s attention; 2) give a specific command; 3) initiate physical prompting; and 4) praise. In the initial RUBI demonstration study, staff expressed concerns about the use of physical prompting with students: “When I watched these videos and when I read the material, that was something that I thought is kind of sticky. So, I am told not to touch students.” In response to this input, and again based on the goal of ensuring broad applicability of session content, compliance procedures were modified based on staff feedback from the collaborative redesign and RUBIES demonstration studies to instead focus on “when-then” verbal prompts: “You can just pull them back to the reinforcement. If the kid’s not going to comply, then you obviously haven’t found a reinforcement that’s going to make him comply in a way,” noted another staff member during the RUBIES demonstration study. Specifically, while staff will continue with getting the child’s attention and giving a simple, specific command, they will pair the command with a “when-then” statement (e.g., “when you put your backpack in your cubby, then you can get a sticker from me.”)

Integration of Planned Ignoring with Functional Communication Training.

Concerns were consistently raised around difficulties working through an extinction burst in a classroom while also trying to continue conducting a lesson. “[Some providers] completely ignore and I felt horrible, but I had to override [them] because that student started to throw out books and hit other kids. And I said, I’m sorry, but this is past the point of safety,” stated a staff member during a RUBI demonstration study. To address this concern, planned ignoring was modified from a stand-alone session in RUBI to a joint session with Functional Communication Training. The emphasis is now placed on teaching students an appropriate (functional) means to communicate needs and using planned ignoring in this context as a way to put the inappropriate communication on extinction.

Elimination of Teaching Skills Sessions.

While school staff did endorse value in the teaching skills sessions for some students (i.e., relevant adaptive skills such as tying shoes, particularly with younger students), there were questions about applicability across all students. “But [these skills] don’t have to be taught as systematically to the majority of my students, it would be more for kids who do have an individualized education program (IEP) or are struggling with some other things that way...It might be the one or two kids who need a little extra help,” stated a staff member in feedback from collaborative redesign session.

Similar to the Core sessions of the RUBI intervention, core RUBIES content was designed to be relevant for all school-age children. As a result, the two RUBI Teaching Skills sessions were removed as Core sessions.

End Users

Targeted Staff End Users

“A lot of times it is paraeducators in both general education and special education settings who are carrying out the plan for kids. And so…there might be a solid team making the plan, but it’s the instructional assistant who is in there, carrying it out,” noted a staff member during a RUBI Demonstration Study. Feedback from stakeholders with regard to end users led to six key findings: 1) special education teachers feel adequately prepared to support the needs of autistic students, including behavioral intervention planning; 2) general education teachers endorsed the desire to successfully support inclusion of autistic children, yet reported having little time to actively intervene one-to-one; 3) challenging behaviors are a key prognostic indicator that disrupts inclusion (i.e., without ongoing support, autistic children and disruptive behaviors often are sent back to their more restrictive classroom placements); 4) paraeducators are best suited to support the transition from special to general education settings due to their consistent access to the child across classrooms; 5) paraeducators are in need of targeted training in classroom behavioral intervention strategies given their minimal autism-specific behavioral intervention training but extensive time with autistic children; and 6) several states in the USA have issued a paraeducators’ certification program, where paraeducators must meet a minimum number of hours of training (e.g., 14 hours in WA State). These findings led to identification of paraeducators serving autistic children with disruptive behavior as the targeted end-user for RUBIES.

Targeted Student End-Users

School staff also noted that the pivotal transition for autistic students from special education to general education is rife with challenges when: 1) the supports that helped the student succeed in a more restrictive setting are not carried over to the general education classroom; and 2) disruptive behavior interferes with instruction in general education classrooms, resulting in autistic students being removed and returned to their self-contained classroom, negating inclusion efforts. As one staffer noted, “We’re placing these kids into pathways in kindergarten. So, the decision between what percentage of the day in general education versus a self-contained classroom falls to me and the biggest deciding factor when I talk to colleagues is behavior that can be dealt with in a classroom.” Given that disruptive behavior in schools is one of the most significant barriers to student participation in inclusive general education settings, RUBIES was identified as being best suited for autistic students and disruptive behavior who are engaging in inclusive placement with their general education peers.

Acceptability, Feasibility, Appropriateness and Usability of RUBI versus RUBIES

Table 3 summarizes changes in intervention acceptability (AIM), feasibility (FIM), appropriateness (IAM), and usability (IUS) ratings (on a 5-point scale) across the entire intervention redesign process. The original RUBI intervention, which was evaluated in the initial demonstration study and collaborative redesign process, was deemed acceptable (4.7 in both phases), appropriate (4.3 and 4.4, respectively) and usable (77.0 and 80.4, respectively), when considered in the context of schools. However, stakeholders endorsed demarked concerns around intervention feasibility (3.9 in both phases involving review of RUBI content). Upon completion of the redesign efforts, intervention feasibility scores on the newly developed RUBIES prototype improved from 3.9 to 4.3. Intervention acceptability (4.6), appropriateness (4.5), and usability (78.4), remained high when evaluating RUBIES.

Table 3.

Intervention Acceptability, Feasibility, Appropriateness and Usability by Study Phase

	Demo Study 1	Collaborative Redesign	Demo Study 2
	M (Range)	M (Range)	M (Range)
Acceptability	4.7 (3.5–5.0)	4.7 (4.3–5.0)	4.6 (1.8–5.0)
Feasibility	3.9 (1.0–5.0)	3.9 (3.3–5.0)	4.3 (2.0–5.0)
Appropriateness	4.3 (2.0–5.0)	4.4 (4.3–5.0)	4.5 (2.3–5.0)
Usability	77.0 (52.5–100.0)	80.4 (60.0–97.5)	78.4 (52.5–97.5)

Discussion

Using the Discover, Design/Build, Test (DDBT) Framework (Lyon et al., 2019), RUBI was redesigned for educational settings (RUBI in Educational Settings; RUBIES). The final version of RUBIES is streamlined in comparison to the original RUBI intervention, informed by stakeholder feedback around removing less relevant, lower value, or specialized content. In addition, target end-users (paraeducators) were identified as they provide direct support to autistic children in both general and special education classrooms and often do not receive specialized training on effective behavioral intervention planning to support the unique needs of autistic children (Carter et al., 2009; Corkum et al., 2014). Additionally, stakeholders identified some broader modifications during the redesign process that could potentially improve feasibility of the original RUBI intervention (e.g., providing a kit of visuals, making visual and case examples more diverse and inclusive). This speaks to the power of iterative design processes and how they also may be useful to adapt interventions across time and contexts. Interestingly, while school personnel deemed the original RUBI intervention as highly acceptable (4.7 out of 5.0) and appropriate (4.3 out of 5.0), feasibility was low (3.9 out of 5.0), suggesting concerns were related primarily to how RUBI could be applied in schools (e.g., data collection procedures were not feasible, the creation of visual supports was not feasible with the time constraints of a school day, etc.). Redesign efforts appear to have improved this issue (RUBIES feasibility = 4.3 out of 5.0). Ultimately, the collaborative redesign process enhanced usability and usefulness of RUBIES and should increase adoption and implementation.

Interventions often are not adapted in the context of implementation (Chambers & Norton, 2016), which results in low uptake (Gottfredson & Gottfredson, 2002). Attempts to transport proven-efficacious interventions into schools without consideration of modifications to ensure the intervention fits with the stakeholders, existing programs, and settings in which it will be used are unlikely to succeed. Iterative redesign efforts may support intervention use within the broader school context and complement existing school-wide programs and practices. For example, many schools use positive behavior intervention supports, a multi-tiered framework that establishes a school culture around behavioral supports to improve social, emotional, behavioral, and academic outcomes for all students. RUBIES has the potential to complement existing school-wide or classroom programs and practices for autistic students who may need more individualized and tailored approaches to address their disruptive behavior. It also is important to consider other implementation drivers at the macro- (e.g., district policies) and organizational-levels (e.g., organizational culture and climate, implementation leadership and climate) that have an impact on evidence-based practice use in schools (Ahlers et al., 2021; Williams et al., 2019). Future studies of RUBIES should examine whether these factors lead to successful adoption and installation of RUBIES.

Moving forward, RUBIES can leverage statewide and/or district mandates to provide specialized training to paraeducators who work with autistic children. In some states and districts, paraeducators also may have the opportunity to obtain certification in special education by completing 20 hours of additional training; although, these policies are relatively rare in US school systems. Where in place, RUBIES can capitalize on this requirement by allowing paraeducators to receive credit as part of their certification for attending RUBIES training, increasing the likelihood of paraeducator willingness to receive instruction.

Similar to the original RUBI program, RUBIES is not designed to support students with profound behavioral challenges (e.g., self-injurious behaviors). Given that disruptive behavior in schools is one of the most significant barriers to students’ participation in inclusive general education settings (Dunlap et al., 2010), RUBIES is redesigned to be best suited for autistic students who are ready to move toward general education settings. This pivotal transition is ripe with challenges when the supports that helped the student succeed in a self-contained setting are not carried over to the general education classroom and disruptive behavior interferes with general education instruction. As a result, oftentimes autistic children are removed and returned to their self-contained classroom, negating inclusion efforts. Based on stakeholder feedback that: 1) special education teachers generally have behavioral management training, 2) general education teachers may not have the time or capacity to consistently use RUBIES with autistic students given the number of students in their classroom; and 3) paraeducators were identified as the least trained provider that spend the most time with autistic children, we believe RUBIES will be most appropriate for school staff who are responsible for supporting students while they bridge classroom settings, potentially improving successful outcomes around mainstreaming efforts. However, we recognize that RUBIES may be more broadly used among all members of the multi-disciplinary team and has the potential to be used for autistic students with varying support needs in any school setting. The school team should work together to determine RUBIES implementation given the different roles they may play (e.g., decision-maker about how behavior management strategies are applied, liaison between special and general education classroom, one-to-one support, etc.).

Limitations

While this study used a novel and innovative framework to redesign a proven-efficacious intervention, certain limitations ought to be noted. First, recruitment was geographically limited. Although enrollment was expanded to a public school district in the Midwest for the final demonstration study and overall themes were consistent across groups, the new school staff did report on specific district policies, school norms, and cultural factors that were unique compared to the West Coast district and would impact RUBI implementation. It stands to reason that other geographically diverse school districts would provide further insights that would lead to further manual adaptations and result in final product that is responsive to contextual fit. Future studies could serve to recruit personnel from a geographically diverse set of schools to ensure RUBIES is universally applicable and useable. Second, data collection was disrupted by COVID-19. One collaborative redesign feedback session and both RUBIES demonstration studies were conducted via Zoom.

Conclusions

A user-centered designed approach combined with implementation science methods has strong utility to ensure proven-efficacious interventions are appropriately redesigned for use in different settings in which they were developed. Ultimately, engaging with the stakeholders for whom interventions are targeted will bridge the research to practice gap. Research around the translation of proven-efficacious interventions into the community considerably lags behind other areas of autism research and is an area of high research priority. Future research should test the intervention mechanisms underlying RUBIES effectiveness and continue to iterate on content and implementation strategies to ensure its utility in schools.