A Tangled Web: The Challenges of Implementing an Evidence-Based Social Engagement Intervention for Children With Autism in Urban Public School Settings

Abstract

There is growing evidence that efficacious autism-related interventions rarely are adopted or successfully implemented in public schools, in part because of the lack of fit between the intervention and the needs and capacities of the school setting. There is little systematic information available regarding the barriers to implementation of complex interventions such as those addressing social engagement for children with autism. The present study used fieldnotes from an implementation trial to explore barriers that emerged during the training of school personnel and subsequent implementation of a social engagement intervention. A number of barriers at the individual (training) and school levels (policies surrounding recess, staffing, prioritization of competing demands, level of respect and support, and availability of resources) interfered with the continued use and sustainment of the intervention. We offer potential strategies to overcome these barriers and provide directions for future research in this critical area.

Social impairment is the most challenging and pervasive core deficit of children with autism, affecting the presence and quality of children’s engagement with peers (American Psychiatric Association, 2013; Bauminger, Solomon, & Rogers, 2010). While inclusion of children with autism in general education classrooms is increasing (Symes & Humphrey, 2010), inclusion is necessary but not sufficient to improve social functioning (Chamberlain, Kasari, & Rotheram-Fuller, 2007; Ferraioli, & Harris, 2011; Kasari, Locke, Gulsrud, & Rotheram-Fuller, 2011). Studies of included children with autism show that they have fewer reciprocal friendships, report more loneliness and poorer friendship quality, are more isolated and less engaged with peers on the playground, and are less socially integrated into their classroom’s social structure compared with their typically developing classmates (Bauminger & Kasari, 2000; Chamberlain et al., 2007; Rotheram-Fuller, Kasari, Chamberlain, & Locke, 2010; Kasari et al., 2011). If left untreated, social impairments may exacerbate with age and result in fewer friendships, greater loneliness, and elevated feelings of anxiety and depression (Rotheram-Fuller et al., 2010; Orsmond, Krauss, & Seltzer, 2004; White & Roberson-Nay, 2009).

Several interventions have been developed to address social impairment in children with autism (McConnell, 2002); however, few of these interventions have been tested in schools. Of those that have, the two models that have emerged as promising are adult-facilitated and peer-mediated interventions (Harper, Symon, & Frea, 2008; Kasari, Rotheram-Fuller, Locke, & Gulsrud, 2012; Rogers, 2000). Adult-facilitated intervention components include transitioning children into a structured activity on the playground, modeling appropriate social behaviors (e.g., good sportsmanship, turn-taking, sharing, being flexible), building interpersonal communication skills, and addressing problematic and challenging social behavior (Kretzmann, Shih, & Kasari, 2015). In contrast, peer-mediated interventions entail training typically developing peers to interact with children with autism using a series of strategies (e.g., understanding differences, using patience, redirecting, sustaining engagement, initiating, responding) that cultivate an environment of understanding, sensitivity, and acceptance that engages children with autism. These interventions typically occur during noninstructional periods of the school day such as recess and lunch on the playground and in the cafeteria (Kasari et al., 2012). Recently, Kasari and colleagues conducted a randomized controlled school-based trial that systematically compared a child-assisted intervention (where the interventionist worked one-to-one with the target student) to a peer-mediated intervention (where the interventionist trained three classmates of the student with autism) and a wait-list control with 60 mainstreamed children with autism from 56 classrooms in 30 elementary schools. They found that children with autism who received a peer-mediated intervention had significant improvements in their social network inclusion and peer engagement on the playground after 12 recess/lunch based sessions over 6 weeks; however, children who received the child-assisted condition did not have those same gains (Kasari et al.). The results of this randomized trial suggested that the social involvement of children with autism within their peer social networks could be improved in a relatively short period of time by changing the environmental context in which children naturally engage. While the results were promising and pointed to a relatively low-cost way for schools to help these children, the Kasari and colleagues study was an efficacy trial, and made no attempt to train school personnel to use the intervention. As would be expected, in the absence of continued support, schools in which the intervention was tested did not continue the intervention once the study ended (Dingfelder & Mandell, 2011).

Although schools are under increasing pressure to incorporate evidence-based interventions to meet the diverse needs of children with autism (Lester & Kelman, 1997), few evidence-based programs have been adopted or successfully implemented in schools. Though social engagement interventions may have the most meaningful results for children with autism if implemented in schools (Locke, Kasari, Rotheram-Fuller, Kretzmann, & Jacobs, 2013), the complexity of these interventions makes them difficult to implement in public schools, given the limited resources and training often available (Dingfelder & Mandell, 2011).

A growing body of research in implementation of other evidence-based mental health interventions in school settings has examined barriers to implementation (Forman, Olin, Hoagwood, Crowe, & Saka, 2009; Forman et al., 2013; Langley, Nadeem, Kataoka, Stein, & Jaycox, 2010; Massey, Armstrong, Boroughs, Henson, & McCash, 2005). These studies have identified a number of important barriers to implementation, including staff’s competing priorities and responsibilities, the lack of administrator and teacher support, difficulty obtaining resources and materials, and finding time for the intervention in the school day (Forman et al., 2013; Langley et al., 2010; Massey et al., 2005). These barriers may affect implementation of social engagement interventions for children with autism as well; however, to date, these issues have not been studied, leaving the question of the best ways to support schools in their use and implementation of evidence-based autism-related interventions. In order to devise strategies that will aid in the implementation process, it is important to first uncover the challenges that may affect implementation of these types of interventions, including those related to infrastructure, administrative and teacher support, school norms and policies, and other needed resources (Atkins, Frazier, Adil, & Talbott, 2003; Langley et al., 2010).

In this paper, we utilized the framework described by Domitrovich and colleagues (2008) to organize the various factors associated with successful implementation of a social engagement intervention for children with autism in urban public schools. According to this framework, multiple factors at different levels (i.e., macro, school, and individual) are critical to successful implementation. Domitrovich and colleagues (2008) posit that factors are interdependent across levels, and have the potential to influence the quality with which interventions are implemented. In this framework, the macro level includes community factors (e.g., district policies) that may influence the quality of implementation within schools. The school level includes organizational factors such as functioning, policies, resources, and climate, and the individual level includes factors associated with the implementer such as professional and psychological characteristics and perceptions of and attitudes about the intervention (Domitrovich et al., 2008). While barriers to implementing social engagement interventions for children with autism exist at multiple levels, for the purpose of this study, we mainly focused on the professional characteristics at the individual level and school-level challenges to implementation during a randomized pilot of an enhanced version of the Kasari and colleagues (2012) social engagement intervention. Our understanding of district-level factors affecting implementation was limited as access to the district in which this intervention was tested was minimal. However, to date, our experiences in schools have shown that, given schools’ autonomy, district-level policies are distally related to implementation (Mandell et al., 2013). It is unclear if district-level policy changes are necessary for successful implementation, but they are not sufficient. The district in which this intervention was piloted has mandated and provided substantial training in other autism-related interventions, yet there is significant variability in implementation and sustainment in these schools. Consistent with Kasari and colleagues (2012), we used the same study design (randomized wait-list control), time frame (6 weeks), and dosage (twice peer week). While previous studies have used different qualitative methods to measure barriers to implementation across levels, we have elected to use fieldnotes rather than other qualitative methods to study barriers to implementation within the Domitrovich and colleagues (2008) model. We did so for two reasons: (a) responses from focus groups or interviews may be affected by social desirability (Callegaro, 2008), especially because the school staff developed such strong relationships with the interventionist; and (b) fieldnotes allow us to examine interactions across levels of the Domitrovich and colleagues framework that illustrate the researcher’s perspective on the individual and school-level challenges and barriers to implementation (Mulhall, 2002). Based on our findings and experiences, we also offer suggestions to researchers and practitioners to address the challenges of implementing evidence-based autism-related interventions in school settings.

Method

PARTICIPANTS

Nine children with autism, 9 school staff members (5 one-to-one assistants, 3 noon-time aides, and 1 bus attendant), and 100 typically developing peers from six classrooms in two schools participated. Both schools were located in a large (149,535 students), urban, ethnically and racially diverse district, where nearly 14% of students have a disability and 82% are considered economically disadvantaged. Children with autism were included in this study if they:

• had a documented diagnosis of autism spectrum disorder from a licensed psychologist;

• were included in a general education classroom for 80% or more of the school day;

• were in a first through fifth grade general education classroom; and

• had an IQ ≥ 65 to ensure they had the verbal and nonverbal abilities to fully comprehend components of the intervention.

Children were excluded from this study if they:

• were not expected to stay in the school or the classroom for the duration of the study;

• had a mental age-equivalent less than 4 years (children less than 4 years do not typically respond well to the proposed intervention procedures);

• had additional diagnoses or sensory or motor impairments; and

• did not have a participating school staff member during the recess period (1 student was excluded from this study because his aide was transferred to a different school).

The average age of students was 8.4 (SD = 1.3) years. One child was in first grade, 3 children were in second grade, 3 children were in third grade, and 2 children were in fifth grade. The majority were male (88.9%); 33% were African American and 67% were Caucasian. The 9 children with autism all met criteria for autism or an autism spectrum disorder using the Autism Diagnostic Observation Schedule (Lord et al., 2000). Children’s cognitive ability also was screened on the Differential Ability Scales–II (DAS-II). The average IQ was 90.8(SD = 13.5) with an average nonverbal IQ (NVIQ) of 89.1 (SD = 11.8) and verbal IQ (VIQ) of 95.7 (SD = 17.0).

School staff members averaged 47 (SD = 9.4) years of age and varied considerably in how long they had worked with children with autism (0–18 years; M = 6.33, SD = 6.87). All participating school staff members were available during the recess/lunch period to engage in the intervention with their respective student with autism and his/her peers.

MATERIALS

Autism Diagnostic Observation Schedule (ADOS; Lord et al., 2000)

The ADOS is a standardized clinician-administered observational measure of social and communication skills used to classify children as meeting criteria for an autism spectrum disorder. For this study, the ADOS was used to confirm a research classification of autism for participating children.

Differential Ability Scales, Second Edition (DAS-II; Elliott, 2007)

The DAS-II is designed to assess cognitive abilities in children ages 2 years 6 months through 17 years 11 months across a broad range of developmental levels. The DAS-II yields a General Conceptual Abilities (GCA) score (M = 100, SD = 15), that is highly reliable, with internal consistency scores ranging from .89 to .95 and a test-retest coefficient of .90. For this study, the DAS-II was used to confirm children had a GCA above 65 (consistent with Kasari et al., 2012).

Playground Materials

The materials required for these interventions are minimal and often involve basic playground equipment (e.g., balls, chalk, hula hoops) and classroom supplies (e.g., markers, paper, construction paper, board games). All materials were provided by the research team.

QUALITATIVE DATA

The first author trained all participating school personnel in the intervention. In order to ensure accuracy of the data, the interventionist reported all events of the session as they occurred. Fieldnotes were immediately written after each intervention session (12 in total) per child/aide case that documented:

• the events of the session,

• successes and/or challenges on the playground and/or in the cafeteria,

• the personal characteristics (e.g., interest level, motivation) of the school personnel implementing the intervention as well as the participating children (i.e., target student and typically developing peers),

• her clinical impressions of the session, school personnel, target student, and peer models,

• school characteristics.

In addition, the interventionist took verbatim note of anything of interest that was said during the sessions and incorporated it in the field notes. To ensure that these quotations were recorded accurately, the first author would jot down notes during the session and would extensively document the session immediately following the intervention session. All fieldnotes (108 entries across 9 participating cases) were analyzed.

PROCEDURE

Intervention implementation and data collection took place in the target students’ schools. Parents of potential participants, identified through the school district, were provided information about the project and asked to contact study personnel for more information. Once families consented, we asked the schools to distribute recruitment materials to staff members who would be potential participants during the child’s recess/lunch period. The research team arranged meetings with interested participants to inform them about the study and their role as a study participant prior to completing the consent process. Subsequently, the ADOS was administered by a doctoral-level assessor who met research reliability with her clinical supervisor and research group, and the DAS-II was administered by two doctoral students in school psychology to ensure children were eligible to participate prior to randomization. Each child with autism and his/her corresponding school personnel were randomly assigned to either an immediate treatment or wait-list control group. Children with autism and school personnel randomized to the immediate treatment group began the 6-week intervention immediately after baseline assessments (see below). Supervision and support were provided only during the 6-week intervention period. Children with autism randomized to the wait-list group began intervention 6 weeks after baseline assessments. Each school staff member participated in 12 hands-on training sessions during their assigned student’s recess/lunch period.

Qualitative fieldnotes were written after each intervention session by the interventionist. The interventionist was a postdoctoral fellow trained in educational psychology with expertise in autism-related intervention development in schools. Independent raters blind to the intervention randomization conducted playground observations of each participating student during baseline, postintervention, and at a 6-week follow-up and once during each week of the intervention/wait-list period for both groups. Blind raters also observed children with autism in the waitlist group on three occasions once they received active treatment and again once the treatment was completed.

INTERVENTION

The Kasari and colleagues (2012) peer-mediated and child-assisted manuals were designed to be implemented by expert university-based researchers with autism intervention training. For this study, that intervention was modified and enhanced to provide relatively inexperienced school personnel with basic training in working with children with autism and specific strategies to facilitate peer engagement during the recess period. The modified manual was adapted for school personnel with less clinical training and additional modules were included that addressed the following: information on children with autism (e.g., autism symptoms and presentation in school); expectations of school personnel during the recess period; the developmental sequence of peer engagement; and matching children’s interest to activities/games for peer engagement during unstructured periods. The first author (JL) provided school personnel with 12 hands-on training sessions (2 sessions per/week for 6 weeks) with the student with autism and his/her peers during the recess/lunch period (approximately 30–45 minutes per session) that included didactics, modeling, and in vivo coaching on strategies to facilitate opportunities for peer engagement.

Content of the Modified Intervention

The training included information on how to: (1) scan and circulate the cafeteria/playground for children who may need additional support; (2) identify children’s engagement states with peers; (3) follow children’s lead, strengths, and interests; (4) provide developmentally and age-appropriate activities and games to scaffold children’s engagement with peers; (5) support children’s social communicative behaviors (i.e., initiations and responses) and conversations with peers; (6) create opportunities to facilitate reciprocal social interaction; (7) sustain children’s engagement within an activity or game; (8) coach children through difficult situations with peers should they arise; (9) provide direct instruction on specific social engagement skills; (10) individualize the intervention to specific children in order to generalize the intervention to other students in their care; (11) work with typically developing peers to engage children with autism; and (12) fade out of an activity/game so children learn independence.

Measurement of Fidelity of Implementation

Two dimensions of program fidelity (i.e., adherence and quality of program delivery) were measured in school personnel’s use of the intervention. Program fidelity measures were developed by the first author and in consultation with the sixth and seventh authors, who have developed fidelity measures in previous studies (Kasari et al., 2012; Mandell et al., 2013). Fidelity measures for each component of the social engagement intervention were based on the modified manual to ensure that school personnel implemented the program in the way that it was designed. There were seven total steps to the interventionist- and observer-rated fidelity measures and both the first author and the observers rated each step during every observation. Adherence was measured via direct observation using an implementation checklist (Yes or No) and coded using criteria specific to the core components of the intervention (e.g., assessing the playground, providing developmentally and age-appropriate activity or game, facilitating conversations with peers, sustaining children’s engagement within an activity or game, providing direct instruction on specific social engagement skills, etc.). Quality of program delivery was measured via direct observation and coded for each core component of the intervention. A score of 1 indicated the school personnel did not use the strategy well during the session or did not implement it correctly, whereas a score of 5 indicated the school personnel implemented the intervention component competently.

Implementation fidelity was measured by the first author after intervention sessions as well as independent observers blind to the intervention randomization once per week. Consistent with Kasari and colleagues (2012), the intervention that was modified in this study, observers comprised research assistants with bachelor degrees who were trained by the first author via live observations and considered reliable with percent agreement = .87 for adherence and κ = .82 for quality of program delivery. Both interventionist and independent observer ratings of fidelity were examined to ensure that school personnel were using the intervention on days when the interventionist was not present and to determine whether fidelity during those days was equivalent to that observed during intervention sessions. Interventionists and independent observers rated adherence (scored as a percentage of completed steps) as well as quality of program delivery on the 1–5 Likert quality scale after viewing an entire recess period. School personnel were given feedback on their implementation each week to ensure that they were adhering to the intervention manual. The interventionist problem-solved with school personnel about relevant issues that arose (as part of their consultation); however, she did not intervene at the school or principal level to increase compliance and reduce barriers to implementation. School personnel were aware that the independent raters observed children’s playground engagement with peers and were members of the research team but were not informed that the independent observers also coded for implementation fidelity to ensure the raters were able to naturally capture implementation if it occurred.

DATA ANALYSIS

The qualitative data were coded by manually reviewing the fieldnotes. Fieldnotes were analyzed in an iterative process and coded using the principles of grounded theory, which provides a systematic approach and inductive process to analyzing qualitative data into recurrent themes and categories (Glaser & Strauss, 1967). In line with this approach, coders avoided forming preconceived expectations of the data and instead allowed ideas and patterns of ideas to emerge from the field notes (Charmaz, 2006; Dey, 1999; Strauss & Corbin, 1998). The first stage of analysis involved reviewing the fieldnotes and engaging in line-by-line coding to identify repeating ideas or patterns as categories. From this first phase, several broad themes regarding barriers to implementation (e.g., individual-, intervention-, and school-level barriers) emerged. During the second stage of analysis, the themes were checked and specific barriers to implementation were defined, described, categorized, and organized. Throughout the process, emerging codes within this theme were discussed with the first author that resulted in a comprehensive coding scheme. Subsequently, the coding scheme was applied to the data to produce a descriptive analysis of each repeating barrier to implementation. A list of codes, definitions, and the frequency of occurrence in all of the fieldnotes (108 in total) is provided in Table 1. Fieldnotes were coded to condense the data into organized units. Chunks of text that ranged from a sentence to a collection of sentences were coded based on meeting criteria within the broader barriers to implementation category, defined as anything that might have impeded or prevented the implementation of the social engagement intervention for children with autism. Although some codes did not frequently occur in the 108 fieldnotes, after discussion with the research team, the decision was made to include these themes into the coding scheme as they may be relevant for successful implementation in schools. The final data codebook contained seven codes (see Table 1). Each coder produced memos that incorporated examples and commentary regarding emergent barriers to implementation. Subsequently, two raters came to consensus on the themes that they abstracted through discussion and the provision of examples from the fieldnotes (Hill et al., 2005). In most cases, examples presented in the text were chosen to best reflect recurrent ideas described by the interventionist. To ensure accuracy of the themes and to reduce potential biases from the research team, member checks were accomplished through consulting with a subset of school personnel (i.e., two trained one-to-one assistants and one noon-time aide) from the two participating schools regarding the interpretation of the data and whether the resulting themes were accurate barriers to implementation (Kvale & Brinkmann, 2009). Consistent with previous research, two raters were trained by the first author until they reached a standard benchmark agreement of 80% of the codes (Hruschka et al., 2004; Landis & Koch, 1977). Each rater coded a subset of the field notes (54 sets each). The raters met on a regular basis to discuss, clarify, and compare emerging categories within the overarching “barriers to implementation” theme to ensure consensus. Any discrepancies that could not be resolved by the two coders were resolved through discussion mediated by the first author. Rater consensus was calculated using Cohen’s Kappa in a subset of 20% of the fieldnotes with rater overlap: agreement was excellent (κ = .80 – .92; Landis & Koch, 1977).

Table 1.

Codes, Definitions, and the Frequency of Occurrence in all of the Fieldnotes

Theme	Frequency of Occurrence (%age of fieldnotes)
Training: Anything that referred to the implementers’ experiences and previous training in working with children with autism	7.14%
Policies Surrounding Recess: Anything that might have interfered with the occurrence of recess (e.g., inclement weather, limited recess time, detention/punishment, etc.)	29.63%
Staffing: Anything related to the number of school personnel available for the social engagement Intervention	21.30%
Prioritization of Competing Demands: Anything that was related to the importance (or lack thereof) of social engagement as a priority within the school	18.52%
Level of Respect and Support: Anything that was related to the amount of administrative/teacher support and regard for the implementers	10.19%
Availability of Resources: Anything related to the materials or physical space required for the social engagement intervention	9.26%

Results

IMPLEMENTATION FIDELITY

Fidelity of implementation was mixed in session notes used to identify barriers to implementation. The average number of completed steps of the intervention as rated by an independent observer (during one recess period when the interventionist was not present) is presented in Figure 1. While school personnel used more components of the intervention as time progressed (adherence), their overall observer-rated quality of program delivery was approximately 50%. The average observer-rated implementation quality of completed steps ranged from 1.50–4.40 for the immediate treatment group and 3.50–4.69 for the wait-list group during active treatment (on a 1–5 scale). The quality of implementation improved as the intervention period progressed for both groups, with the highest ratings at the end of treatment. With regard to interventionist-rated fidelity, the average number of completed steps of the intervention (adherence during the intervention) is presented in Figure 2. These data include school personnel who received active intervention once the waitlist period was completed. Overall, the average interventionist-rated implementation quality of completed steps ranged from 2.80 to 3.45.

FIGURE 1.

Average number of completed steps (out of 7 steps total) of the intervention based on independent observer fidelity for the immediate treatment and wait-list groups. There were five observations in the immediate treatment group and four observations in the wait-list group.

FIGURE 2.

Average number of completed steps of the intervention based on the interventionist’s report of school personnel fidelity.

BARRIERS TO IMPLEMENTATION

Policies Surrounding Recess

School policies surrounding recess varied and often limited opportunities available for implementation. In this district, recess occurred once a day in combination with the lunch period for a total of 45 minutes (30 minutes eating time; 15 minutes playing time). There were frequent, unexpected situations that resulted in canceled recess or limited recess time. In many schools, recess time was shortened because children were dismissed late from the lunchroom: “[The lunch monitor] lets them out really late, so there wasn’t as much time outdoors.” In another entry, “We spent a total of 4 minutes outside. The children were disappointed, as was [the aide].” During inclement weather, recess was usually canceled or held indoors. In one school, all grade levels were expected to watch a movie in the auditorium: “It was too cold to go outside today. The kids were watching a movie in the auditorium.” A different entry for a different target student at that school also noted a similar inclement weather schedule. “It was raining today. When it rains, the children are shuffled into the auditorium and they get to watch 10 to 12 minutes of a cartoon movie.” In another school, all grade levels were expected to remain in the cafeteria because space was an issue: “It was too cold to be outside today and the gym was being used by a different class, so the kids had to stay in the cafeteria and listen to music.” Aside from inclement weather, taking away recess often was used as a punishment. “Every other day it seems as if [the children] are receiving detention … The punishment— recess is taken away. They have to stand in line or on the wall for the entire play time.” In many instances, students were punished for “bad behavior”:

[Peer₁] and [Peer₂] reported that they failed to do their mission yesterday because the entire lunchroom had to stand in line outside … because they were too loud in the cafeteria. The class I am working with did not misbehave, but needed to serve the punishment anyway.

Recess also was frequently canceled during special school-wide activities:

Today was the first day of “Spirit Week” so lunch recess was canceled. The yard was set up with a variety of activities that each class attended during their assigned Spirit Week period—in exchange no classes go out to lunch recess. So, we sat in the auditorium watching movies … roughly a quarter of the year is spent in indoor recess (i.e., testing, assemblies, report card conferences, Spirit Week, etc.).

Staffing

Because of recent fiscal challenges, there were significant staffing shortages, making implementation difficult. Often, few staff were on the playground or in the cafeteria; thus, a staff member was frequently unavailable to facilitate the intervention. Many participating school personnel had dual roles that interfered with implementation, as indicated in the following note: “ [The aide] also is a student support assistant and doubles as a noon-time aide. She is in charge of the lunch room and the entire recess period for the entire school’s different lunch periods. She knows [the target student] but does not interact with her one on one, so getting to her on the yard is problematic.” Staff reassignment and turnover was a challenge as well as evidenced below:

This is the third aide that [the target student] has received in the past 3 months. After 3 weeks of training, [Aide₁] was switched away from [the target student] and put in the autism support classroom. [Aide₂], who was previously with [the target student], resigned – she took a different job with better pay, and now [the target student] is on [Aide₃].

Training

Of staff that was available, all had a variety of experiences, but many were not specifically trained in working with children with autism. In one case, the interventionist noted, “She tries really hard, but just doesn’t have the autism experience …” and in another “ [A one-to-one aide] is an absolute pleasure to work with. She seems grateful for the help.” There was a clear need for additional training specifically surrounding autism-related intervention strategies. One one-to-one aide was proactive and sought out opportunities for professional development specifically in autism; however, many support staff members were not included in the district provided autism-related professional development trainings, which were mostly reserved for and directed at special education teachers. While they spent the majority of the school day working closely with children with autism, many of these support staff members were untrained in evidence-based behavioral, academic, and social strategies in working with children with autism and welcomed the opportunity for additional support in developing social engagement in their students. For example, in one entry the interventionist wrote, “She signed up for the program and seems eager for help in working with children with autism. She expressed that she has not had much training in this area.” The level of training may impact implementation, but once school personnel were equipped with the necessary tools, they were able to make significant changes in the children they served as evidenced by this excerpt: “She said that she’s never been trained in these things before and now that she’s learning some things, it’s changed [the target student’s] whole world.”

Prioritization of Competing Demands

Related to staffing and training, another barrier to implementation was the school leadership’s prioritization of competing demands, as staff members often had a list of responsibilities that changed frequently during the recess and lunch periods, which limited the feasibility of implementation. This was particularly challenging for the “noon-time aides” who participated in the intervention.

[The aide] is in charge of the lunch room and her attention is not solely devoted to [the target student], although she interacts with him and knows him well enough to sign up for the program. [The aide] is a really nice lady — she takes her job seriously but seems overwhelmed at times since she is in charge of so many children, the bathroom schedule, cleaning the lunchroom, and monitoring safety pretty much by herself. … It is clear that safety is her primary concern and unfortunately, her time and energy are limited and most likely will not be devoted to facilitating opportunities for [the target student].

A different noon-time aide was reassigned midway through the intervention: “ [The aide] is doing a great job facilitating. In fact, the principal has reassigned her to only yard duty, which is good in the sense that she is facilitating opportunities for students to play.” Even for one-to-one aides, role assignments changed based on the needs that particular day. “When I arrived, she was not in the teacher’s lounge. During our last session, she informed me that she was no longer allowed to sit with [her student] in the lunchroom per the special education teacher’s and the behavioral specialist consultant’s requests.” Because this one-to-one aide was asked to stay out of the lunchroom, it was difficult to deliver the components of the intervention that occurred in the cafeteria (e.g., building conversations with peers). Implementation also was not possible when school personnel were called into classrooms to shadow a different student during the target student’s recess period, when the intervention was to be delivered. “I wondered where [the aide] was and then she appeared—she told me she was on ‘lunch detention’ duty for the students who are kept in. So, it is not feasible for her to work with [her student today].”

Level of Respect and Support

The amount of administrative and teacher respect and support often impeded implementation. A one-to-one aide (who has been an aide for 20 years) seemed frustrated with the limited support on the playground as she reported that “the teacher and principal [are not aware of what she does as] they’re [rarely] out on [the playground].” A different aide noted differing priorities of support staff and teachers: “Sometimes there’s no communication between the teachers and the yard staff. [The aide] feels like everyone is working on a different page.” In a separate entry for the same case, “The aides feel undervalued, unappreciated, and altogether burnt out. They are responsible for a lot of students, and they are frustrated … there are feelings of being on different pages.” One aide received conflicting messages from her principal who encouraged her to participate in the intervention and later scolded her for participating:

I was informed that [the aide] was reprimanded for playing with the kids. The principal would prefer if the aides did not engage in play activities with the students because that takes them away from doing their other job responsibilities such as monitoring for safety. I get the feeling the aides do not feel supported in their jobs and earn little respect in the school.

The lack of respect and support may lead to feelings of isolation (e.g., I’m all alone in doing this intervention), frustration (e.g., I’m trying so hard, but things keep interfering), and resentment (e.g., Why do I have to do more work?) that may ultimately affect the staff’s use of the intervention, as noted in this excerpt:

I’m not sure what happened, but there was definitely tension in the air today. All of the aides were upset. Morale looked low, and tension was high; everyone seemed disgruntled and no one was doing the intervention. From what I gather, the principal enacted a new policy that the aides perceived as “not being the best for the children” and “making their lives harder.”

In a separate entry 2 days later:

…the principal decided to restructure things—late in the school year. She changed almost every classroom—and reshuffled the entire school (except the autism support classrooms). The aides were not happy about it—they thought it disrupted the children’s learning and created more anxiety and behavioral problems on the playground.

Availability of Resources

Two resources were necessary to successfully implement this social engagement intervention: physical space and playground/activity materials. The structure of many schools in which the intervention was carried out was not conducive to implementation, with some schools lacking a playground, safe equipment, or alternative indoor space. These resources were particularly unavailable during inclement weather, when children had indoor recess. Space was often limited. “[The target student] made up a shape/colors game with running, but the only space we had was on the stage [above the lunch tables] and it was not safe to run around. [The target student] quickly lost interest because we were not able to do the game he selected.” In another example,”It was raining out so we played Heads Up 7-Up at their lunch tables—there was no room to go anywhere else, and it was so noisy in the lunchroom that it was hard to play the game.”Facilitating a social engagement intervention was challenging during indoor recess: “We talked about the possibility of doing indoor activities with the children during rainy/snowy days. [The aide] said it wasn’t possible because of the lack of space and if she were to take them to the stage that would mean one less body in the auditorium watching the other children.” In addition to physical space, schools often may not have the budget to purchase materials such as playground balls, sidewalk chalk, or stickers. In one school, “I noticed there were no balls or playground equipment on the yard.” Although many games require few materials, the intervention has the best outcomes when the activities are centered on the children’s interests. Thus, it is important that the appropriate materials are available in order to maximize the number of potential activities that the implementer could use to facilitate engagement. For example, according to one excerpt, “It is evident that [the aide] has been making efforts to support [the target student] socially—she brought the cones (labeled with numbers) and the ball down with her, so I knew she had the intention of starting something when I arrived. I really believe that half the battle is access to equipment.” Children with highly specified interests or low social motivation to interact with peers may only respond to a limited number of activities with peers that would not be possible without appropriate materials.

Discussion

This study used quantitative and qualitative data to (a) describe the implementation fidelity and quality of a social engagement intervention for children with autism; and (b) identify important barriers to implementation in an urban public school district. The results indicated that average implementation adherence fidelity ranged from 0–4 steps out of 7 total; however, over the course of intervention, quality of implementation fidelity improved. The training of the school personnel over the 6 weeks may explain the increase in fidelity over time. Qualitative data illustrated a number of barriers to implementation, including implementers’ previous training, school policies surrounding recess, staffing, prioritization of competing demands, level of respect and support, and the availability of resources. Some of these barriers were unanticipated and uncontrollable; however, many barriers could be inexpensively addressed to foster a school culture and climate that promotes the use and sustainment of social engagement interventions for children with autism. The discussion focuses on the relationships among these barriers, their effects on fidelity, and potential strategies to overcome them.

This study found several barriers to implementation that were consistent with prior research. In this particular context, the effects of these barriers may have been compounded or exacerbated, because social engagement interventions are most successful if implemented in a natural context with children’s peers and often are limited to highly prescribed periods (i.e., recess). Figure 3 illustrates the different barriers in this study and multiple obstacles that may impede implementation at different stages. Policies surrounding recess were the most common barrier to implementation observed in this study. While behavioral and academic interventions are typically seen as falling within the purview of schools (Adelman & Taylor, 1998), social engagement interventions often are seen as ancillary to schools’ missions, despite their critical importance for children with autism. As a result, policies surrounding recess may limit opportunities for intervention. Even when schools prioritize recess time, a host of other factors may affect implementation, including the availability of trained staff and resources (i.e., intervention materials). If schools have trained staff and materials available during recess time, then the staff’s competing demands are important issues to consider. Staff are often reassigned during the recess period to additional (and often educational) tasks that are given priority over socialization. Last, if implementation is feasible for staff, the lack of administrative and teacher support impedes the use and sustainment of the intervention.

FIGURE 3.

Flow chart of the barriers to implementation.

Although implementation adherence fidelity was generally low, school personnel demonstrated a medium to high level of proficiency and quality of program delivery when they did implement intervention components. This finding suggests that the infrequent use of the intervention may not be a skill deficiency; rather, it may be caused by other factors, such as the implementation barriers described in this study. Upon closer examination, the sessions where we used fieldnotes for analysis had fidelity that was mixed (some sessions had low adherence and quality of program delivery fidelity, whereas others did not). These inconsistent ratings of fidelity may be due to barriers to implementation experienced that day or some other factor (e.g., individual or intervention related). In addition, sessions with high fidelity were of interest as they suggest the implementers were resilient and able to overcome the documented barriers to implementation to deliver the intervention with fidelity.

Several qualitative themes emerged that documented barriers to implementation that are important when translating research to practice. In theory, training school personnel in evidence-based interventions will maximize the potential that children with autism will have access to necessary services to improve core deficits associated with ASD. However, in practice, if barriers to implementation are not resolved or addressed, successful implementation cannot be achieved. We present each identified barrier to implementation from the qualitative data below followed by our recommendations to improve the conditions in which implementation occurs.

TRAINING

At the individual level, the data showed that the lack of training in autism-related interventions was a barrier to effective implementation and sustainment. Consistent with other school-based mental health services, implementation strategies such as high-quality training, ongoing consultation, and booster sessions may be necessary to ensure successful implementation (Forman et al., 2009). This may be achieved through professional development programs and consultative services provided by the district or in partnership with local agencies, parent advocacy groups, or research universities that provide in-service programs, workshops, and other educational seminars. Building internal capacity and support also may contribute to the likelihood of successful implementation and sustainment. In school settings where staff turnover is high, train-the-trainer models may be a potential solution where someone at the administrative level who is less likely to leave (e.g., assistant principal, school psychologist, special education liaison) is trained, who in turn trains new staff. Train-the-trainer models have been successfully used to train supervisors in community mental health agencies (e.g., Southam-Gerow et al., 2014) and may be a promising strategy to support implementation and sustainment in schools.

POLICIES SURROUNDING RECESS

The culture of most schools generally favors academic-related outcomes above other areas of development (Adelman & Taylor, 1998; Massey et al., 2005). Despite its importance for children with autism, socialization often is not prioritized in many schools, and school leadership may not consider socialization as falling under their aegis. School provides an opportune setting to improve social engagement between children with autism and their peers (Locke et al., 2013). Because school practices are generally led by the principal, it is important to involve school leadership in all aspects of implementation to foster the use of these interventions. Training school leadership and staff about the value of recess may be the most important strategy schools could use to support implementation. There are many global benefits of recess for children’s cognitive, emotional, social, and physical development (Centers for Disease Control and Prevention, 1997; Council on School Health, 2013; Henley, McBride, Milligan, & Nichols, 2007; Lee, Burgeson, Fulton, & Spain, 2007); thus, it is important for schools to ensure recess is a protected and meaningful part of the day, where children have the opportunity to socialize and practice their interpersonal skills. During inclement weather, recess should be made available indoors, even if space is an issue (e.g., hallway, back of the auditorium, gym, library, corner of the cafeteria, classrooms).

STAFFING AND PRIORITIZATION OF COMPETING DEMANDS

Principals and other school leadership also can assist in successful intervention implementation by clearly delineating roles, responsibilities, and schedules for all staff. Consistent with findings from other implementation studies in schools, inadequate staffing may prohibit successful intervention implementation (Massey et al., 2005). Building internal capacity to sustain the intervention may ease staff burden, resolve scheduling conflicts, and address issues with sustainment when trained staff members are reassigned or turn over. Training school psychologists, counselors, or teachers may provide substantially more leadership that understands how to implement the intervention. These individuals could potentially take on the role of implementing or assisting with implementation, coordinating intervention procedures and schedules, or providing technical assistance to primary implementers should the need arise (Forman et al., 2009). Some schools also have the opportunity to draw on volunteers, including parents, undergraduates, or doctoral and postdoctoral fellows who need to fulfill service requirements at local colleges and universities as additional resources.

LEVEL OF RESPECT AND SUPPORT

Developing respect and support for implementers from teachers and school administrators, especially from principals, as well as children’s peers, is critical for effective implementation (Forman et al., 2009). It is important to inform and involve multiple agents at the school early on, even if those individuals are not implementing the intervention—they may be instrumental in sustaining the intervention. Literature from other school-based interventions shows that teacher support can be garnered more readily when they can see visible results from use of the intervention (Forman et al.). In one of our cases, the aide reported, “the school was going to remove her from [her student] since he’s been doing so well, but the teacher protested and refused to let the school do that.” Because of this teacher’s support, this aide was able to stay with her student—she executed the intervention with fidelity and her student benefited. Teacher support also could bolster the number of opportunities for intervention. With limited recess time, teachers can use a class-wide reinforcement system that allows children to earn extra recess minutes or free time. In addition, during indoor recess periods, teachers could allow the staff members and selected children to use his/her classroom for a structured activity, if indoor space is unavailable. With regard to administrative support, the organizational literature suggests that it is important for administrators and principals to provide incentives for using the intervention (Klein & Sorra, 1996). Incentives may include both tangible rewards (e.g., small tokens of appreciation) and intrinsic rewards including verbal praise, a pat on the back, an empowering title (e.g., lead play specialist, expert playground associate, lead paraprofessional), or recognition at staff meetings or school-wide morning announcements. Incentives will enhance motivation for delivering the intervention as well as foster feelings of appreciation and gratitude. In addition, administrators could hold regular meetings with staff members implementing the intervention or visit the lunchroom/playground to show the implementers they are equally invested in the intervention and enhancing the wellbeing of students with autism. Last, it is important to establish a routine for classmates and children with autism to expect a structured activity during the recess period. In our most successful cases, “friendship club [became a] part of the culture in the classroom” and as [the aide] said, “the kids [came] to expect recess club and they asked for it every day.” Because the classmates enjoyed the activities, the intervention became a normal part of the school day, and the children provided reminders to the aides to facilitate the program which ensures sustainability.

AVAILABILITY OF RESOURCES

Having the funds necessary to procure the staff and materials necessary for implementation is a challenge for schools, especially in a fiscally unstable climate. Fortunately, social engagement interventions are relatively low cost in the sense that few resources are needed for successful implementation. The social engagement intervention was designed to fit within schools’ existing infrastructure with the constraints and resources in mind. Many activities that are fun and engaging do not require materials (e.g., tag, Red Rover, fantasy games, charades). If playground materials are necessary (e.g., balls, sidewalk chalk) but not readily available, schools could hold fundraisers, utilize their parent-teacher association or home and school funds, apply for small grants, partner with research universities, or seek donations to assist with the costs.

LIMITATIONS

Several study limitations should be noted. First, this study relied on the fieldnotes of a single interventionist (the first author). These data primarily captured the researcher’s perspective on barriers to implementation as well as staff perspectives filtered through the first author’s notes about staff verbalizations. Because this was a pilot study, the number of personnel available to consult on the intervention was limited, and while the interventionist was not a member of the school staff, her perspective as a researcher highlighted the difficulty of implementing these types of interventions in schools. Second, the use of only one mode of data collection limits our understanding of whether the observed barriers were important, salient, and relevant to school staff. Future studies should consider utilizing other qualitative methods such as interviews and focus groups with key informants (e.g., parents, teachers, administrators, district officials) from multiple levels (i.e., individual school, and district) in various schools, who may provide different perspectives on barriers to implementation. This also may allow us to fully understand how the interdependent levels of the Domitrovich and colleagues (2008) model affect implementation of an autism-related intervention. Concurrent use of different methods would help determine whether school personnel’s perceptions of barriers to implementation are similar to or discordant with researchers. Understanding barriers to implementation from both the researchers’ and school personnel’s perspectives may allow researchers to partner with schools to devise strategies that address these challenges. Additional informants from multiple schools also may lead to greater generalizability of these findings to other school settings. Finally, this study focused solely on school-level barriers and did not examine macro-level (i.e., district-level factors), individual-level (at the staff and classroom levels) or intervention-related barriers to implementation. These factors also may be important in understanding how interventions are adopted, adapted, implemented, and sustained over time.

CONCLUSIONS

Despite these limitations, there are important implications related to these findings. This study is one of the first to examine the challenges of implementation of a social engagement intervention for children with autism. The identification of barriers to implementation is an important first step in understanding the many factors that may impede successful implementation of social engagement interventions. The recommendations presented in this study are based on our experiences in partnering with schools. It is important to keep in mind that these suggestions have not been empirically tested. There is a crucial need for school-based research that examines interventions to address implementation challenges particularly at the school level. This study provided the interventionist’s perspective on various barriers to implementation that may point to how we could best support schools with the implementation process, ultimately leading to better outcomes for the growing number of children with autism.