Statewide Interagency Collaboration to Support Evidence-based Practice Scale Up: The California Autism Professional Training and Information Network (CAPTAIN)

Abstract

The California Autism Professional Training and Information Network (CAPTAIN) is a statewide interagency collaboration with the goal of scaling up use of evidence-based practices (EBPs) for individuals with autism spectrum disorder (ASD). CAPTAIN began as a clinical initiative then further developed under the influence of implementation science methodology. The Exploration, Preparation, Implementation and Sustainment framework (EPIS) has impacted targeted strategy use for this statewide scale up of EBPs by informing the development of key partnerships, implementation goals, and collaborative processes within CAPTAIN. Currently, CAPTAIN has over 407 members representing 140 school and community agencies who provide training and coaching in EBP and meet regularly with regional teams. Outcome data indicate 51.9% of members provide training and coaching to more than three direct service providers/programs per year. Primary barriers to implementation of EBP were time for training (25.6%), lack of substitute teachers (16.5%), and staff lacking foundational skills (11.5%). Facilitators of implementation and sustainment of the CAPTAIN model include active participation in effective dissemination practices, creative funding and leveraging of local resources, development of the regional collaboratives with active membership, member commitment to EBP for ASD, and use of implementation science to identify and overcome barriers. The purpose of this paper is to highlight CAPTAIN as a model for statewide scale up of EBP in schools as well as other community agencies. Although these efforts have focused on EBP for ASD, the concepts, partnerships and procedures will likely be transferable to other focal issues and may be generalized across service sectors.

Across child and adolescent behavioral health and educational services, data indicate a significant research-to-practice gap, such that evidence-based practices (EBP) are not routinely used in community practice (e.g., Elmore, 1996; Greenwood & Abbott, 2001). In fact, it can take up to 17 years for only 14% of research-based interventions to reach targeted populations (Balas & Boren, 2000). This delay in implementation leads not only to access issues for students and clients and the providers who support them, but also negatively impacts clinical outcomes.

A review of the literature in implementation science across diverse fields, conducted by the National Implementation Research Network, contains clear recommendations for moving research into practice (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005) and highlights that good outcomes for consumers occur when EBP are implemented effectively. Providers need to have the skills to implement EBP and the support to use these skills in their work environment. Durlak and Dupre’s (2008) meta-analysis of over 500 community prevention and health programs for children and adolescents found that the level of implementation was an important indicator of program outcomes, with better implementation leading to better child outcomes. Programs that are carefully implemented, especially when fidelity or dosage are tracked, lead to mean effect sizes that are two to three times higher than in programs with implementation challenges (Henggeler, 2004). Thus, the research-to-practice gap can be viewed as a dissemination and implementation problem that can be targeted to more effectively move EBP into community care.

Services for Autism Spectrum Disorder

Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by restricted or stereotyped patterns of behavior and impairments in social communication (American Psychiatric Association, 2013). The most recent estimates from the Centers for Disease Control indicate that 1 in 59 children have ASD (Maenner et al., 2020) resulting in the number of children with this diagnosis served by schools growing six-fold across the last two decades, from 93,000 in 2000 to 576,000 in 2015 (Kena et al., 2015). This increase in ASD diagnosis has placed considerable demand on public service systems to quickly develop the scope and quality of services available as well as the size and expertise of the workforce. Furthermore, the issue of slow integration of EBP into community practice is particularly problematic for individuals with ASD, their families and their service providers as their complex and chronic needs are typically lifelong and involve multiple systems of care. For example, individuals with ASD often receive services from education, medical, and mental health systems (Brookman-Frazee, Baker-Ericzén, Stahmeer, Mandell, Haine, & Hough, 2009).

Several systematic reviews have been completed to identify EBP for ASD (National Autism Center, 2009 & 2015; Odom, Collet-Klingenberg, Rogers, & Hatton, 2010; Wong et al., 2014). The National Standards Project identified 14 categories of interventions as “established,” and the National Professional Development Center (NPDC) identified 27 focused intervention practices for ASD (National Autism Center, 2015). The difference in the number of EBPs identified is largely due to methods of classification. The NSP focused on comprehensive interventions, while the NPCD focused on isolated intervention strategies. Therefore, intervention strategies such as prompting, modeling, and reinforcement would be subsumed under the Behavioral Intervention category in the NSP review, but identified separately in the NPCD review. Within this context, 21 of the 27 EBPs identified by the NPDC were considered “Established” practices by the NSP, and these interventions are mostly behaviorally-based. Thus, these independent reviews had great overlap in their respective findings, indicating strong support for efficacious interventions for ASD. The findings have made a significant contribution toward dissemination of EBP for ASD; however, the limited information on community use indicates EBP for ASD are not often incorporated into school-based programs (Hess, Morrier, Heflin, & Ivey, 2008; Morrier, Hess, & Heflin, 2011; Stahmer & Ingersoll, 2004; Suhrheinrich, 2011) or, when used, are implemented with low fidelity or low use of the intervention as designed (Suhrheinrich et al., 2013; Suhrheinrich, Stahmer, & Schreibman, 2007).

In response to this quality gap, there have been urgent calls for the development and testing of implementation interventions to facilitate successful uptake and sustained delivery of EBP for ASD in community programs. The Interagency Autism Coordinating Committee Strategic Plan for ASD Research (IACC, 2013) and the Institute of Education Sciences both have prioritized identifying and targeting mechanisms of successful EBP implementation to maximize public health impact. Multiple factors support successful implementation across individual provider, organizational and systems levels. Research indicates successful training in the use of EBP requires both providing didactic information and competency training, which is “the process of acquiring skills necessary to administer a treatment skillfully and with fidelity” (McHugh & Barlow, 2010, p.74). However, information alone, or basic workshop training, is not an effective implementation method enough to ensure implementation. Incorporating coaching on-site, performance evaluation, program evaluation, facilitative administrative practices, and and methods for systems interventions increase the likelihood of successful uptake of EBP in community programs (Fixen et al., 2005)

Beyond initial implementation of EBP, scaling up interventions across intervention sites, organizations and regions presents an additional challenge. Most state-wide systems have very limited capacity for scaling up interventions in ways that lead to meaningful improvements in outcomes for students (Fixsen, et al., 2013) indicating a clear need for continued development. This purpose of this manuscript is to outline a model for state-wide scale up of EBP for ASD, including the application of an implementation framework and inter-agency collaboration with key partners.

Application of Implementation Frameworks and Practices

The Exploration, Preparation, Implementation and Sustainment (EPIS; Aarons, Hurlburt, & Horwitz, 2011) model is used to highlight factors influencing implementation across specific phases, system levels and activities (see Table 2). EPIS models the implementation process using four phases: Exploration, in which stakeholders become aware of a clinical or public health need, work to identify the best EBPs to address the need, and consider potential system, organizational, and EBP adaptations to support EBP adoption; Preparation, in which the EBPs are selected and support systems are developed; Implementation, in which active training and implementation of the EBPs occurs; and Sustainment, in which the intervention is stabilized and supported through funding systems and ongoing fidelity monitoring (Aarons et al., 2011, Moullin, Dickson, Stadnick, Rabon, & Aarons, 2019). This implementation science framework has been used to guide the implementation process across multiple service sectors, including child welfare, community mental health, and education (Aarons, et al., 2011; Brookman-Frazee, et al., 2019; Moullin, Dickson, Stadnick, Rabon, & Aarons, 2019; Brookman & Stahmer, 2018; Stahmer, Suhrheinrich, Schetter, & Hassrick, 2018). We propose the California Autism Professional Training and Information Network (CAPTAIN) as a model to support state-wide cross-agency scale up of EBP. Although these efforts have focused on EBP for ASD, the concepts, partnerships and procedures will likely be transferable to other focal issues and may be generalized across service sectors. The following sections describe implementation strategies (Powell et al., 2015) presented as key activities and outcomes and lessons learned across each phase of implementation (see Table 1). Multiple methods for data collection were used throughout implementation activities, including field notes, review of operational procedures and a structured survey.

Table 2.

Key Stakeholder Groups

Key Stakeholder		Description	Reason for Inclusion	Participation Rate
UCEDD	University Center for Excellence in Developmental Disabilities	Authorized under the Developmental Disabilities and Bill of Rights Act of 2000 (PL 106–402), UCEDDs are resource for people with disabilities in every state.	Part of UCEDD functions are to coordination and collaboration around disability issues, community outreach and training and translation of research to practice.	2 out of the 3 in the state
	• University of California at Davis (CEDD at UC Davis)
	• University of Southern California (USC UCEDD)
CA-DDS	CA Department of Disability Services	Regional centers are nonprofit private corporations that contract with the Department of Developmental Services to provide or coordinate services and supports for individuals with developmental disabilities.	Regional Centers have offices throughout California to provide a local resource to help find and access the many services and evidence based treatments available to individuals and their families.	19 out of 21 Regional Centers
	• Regional Centers
CDE	CA Department of Education	SELPAs are the special education intermediary that may be comprised of a group of many small districts or a large single district, but each SELPA must be of sufficient size and scope to provide the full continuum of Special Education supports and services for students residing within the region boundaries.	As the primary intermediary for special education services, SELPAs are responsible for insuring their school district provide high quality services and use EBPs	124 out of 132 SELPAs
	• Special Education Local Planning Area (SELPA)
	• Diagnostic Centers
FRCNCA	Family Resource Center Network of California Family Resource Center (FRC)	FRCs provide parent to parent support, outreach, information and referral services for families of children with disabilities and the professionals who serve them.	Since FRCs are often a point of first contact and a trusted resources for families, FRCs have been an important partner with dissemination of information about EBPs to families.

Table 1.

Activity Alignment with Exploration, Preparation, Implementation, and Sustainment (EPIS) Framework

Dates
2008–2012	2012–2013	2013–2014	2014-present
Stages of Implementation
EXPLORATION	PREPARATION	IMPLEMENTATION	SUSTAINMENT
• Development of the California IAPGS	• Establishment of leadership teamS	• Annual training summitS,R
• Implementation of the NPDC-ASD grantS,R,A	• Building interest and recruiting cadreS,A
	• Accessing funding and resourcesS,A	• Post-summit supportS	• Regular communication with agency leadershipS,A
		• Annual cadre surveyS
			• Increasing awareness with professional associationsS
			• Regionalized collaborationsR

Note: IAPG = Interagency Autism Planning Group, IES = Institute for Education Sciences, NPDC-ASD = National Professional Development Center for Autism Spectrum Disorders, SELPA = Special Education Local Plan Area

^SState: State-level activities;

^RRegional: Regional-level activities and collaborations;

^AAgency: Agency-level activities (one SELPA, Regional Center, or Family Resource Center)

Exploration Phase (2008 – 2012): State-level Collaboration and Research Participation

Key Activities.

With the goal of evaluating needs and program fit, key activities of the Exploration phase involved development of the California Inter-agency Autism Planning Group (IAPG) and participation as a demonstration site for the National Professional Development Center for ASD technical assistance project. Discrete implementation strategies (Powell et al., 2015) employed during Exploration included: building a coalition, involving patients/consumers and family members, engaging champions, and developing academic partnerships.

California Inter-agency Autism Planning Group.

In 2007 the California Legislative Blue Ribbon Commission on Autism and the California Superintendent of Public Instruction Autism Advisory Committee (California Legislative Blue Ribbon Commission on Autism, 2007) outlined recommendations for improving services for individuals with autism that included: identifying and using a core set of EBPs across agencies so that consistent information is provided to families, providing training and professional development in the EBPs so that educators are using them appropriately, and developing a clearinghouse of vetted information and resources related to ASD. To address these goals the Center for Excellence in Developmental Disabilities (CEDD) at the UC Davis MIND Institute initiated the development of an Inter-agency Autism Planning Group (IAPG) in January of 2008. The group members represented multiple service sectors, including special education, developmental disabilities services, public health and managed care, and other key stakeholders including various parent groups (see Table 2 for descriptions of each group). The IAPG met quarterly to evaluate options for moving this agenda forward. The group established a mission and vision statement as well as a logic model outlining goals and action plans and set into action a multi-year plan to bring this vision to fruition.

National Professional Development Center for ASD.

One of the primary objectives of the IAPG was to create inter-agency awareness and collaboration by developing a common training curriculum for EBP for ASD to be used by all the above key stakeholders. To address this need, in 2009, the IAPG participated in a technical assistance project through the National Professional Development Center on Autism Spectrum Disorder (NPDC-ASD). The NPDC-ASD had already identified 26 EBP for ASD, created a training curriculum, and was funded to develop a model of professional development that states could employ to increase use of EBP for ASD in public school programs (Cox et al., 2013). The goal of the project was adoption and use of the NPDC-ASD model in at least six school sites and sustained use across each participating state. Selected EBP identified by the NPDC-ASD were implemented at these demonstration sites and trainers received coaching and feedback on their use of the NPDC-ASD model and resources. The NPDC-ASD model consisted of assessment, implementation and measured outcomes within the context of a reflective coaching model. The training tools and resources included a web-based Foundations of Autism Spectrum Disorders course through the University of North Carolina, web-based training modules with summaries and implementation checklists on each of the EBP and a coaching and procedures manual. Goal Attainment Scaling (Ruble, McGrew & Toland, 2012) was used to measure student progress in three individually developed annual goals. The Autism Program Environmental Rating Scale (APERS), an instrument designed to assess the overall quality of program environments for students with ASD, was used to measure program change associated with the NPDC-ASD model intervention (Odom et. al, 2018).

As part the IAPG’s two-year participation period, state and local trainers and technical assistance providers were identified from two divisions of the California Department of Education, State Diagnostic Centers (n = 6), which provide diagnostic and program development support to local education agencies and the Special Education Local Plan Areas (SELPAs; n = 6), the intermediary agencies who oversee the implementation of special education services throughout the state. Intentional activities to increase “buy-in” from key stakeholders included: 1) selecting representatives from key stakeholder groups; 2) selecting demonstration sites from various geographic regions of the state and 3) selecting demonstration sites that together covered training for the full continuum of ages offered by the NPDC-ASD (Pre-Kindergarten to High School). By focusing on these issues, IAPG members were all directly represented and impacted by the state-wide training efforts, which provided critical support for the next phase of planning.

Outcomes and Lessons Learned.

Positive outcomes from the NPDC-ASD project at the national and local levels were used to identify needs at next steps. At a national level, significant improvements were found across the 58 school programs in nine states that participated in the NPDC-ASD project (Odom et al., 2013). In California specifically, positive fidelity and clinical outcomes (Proctor et al., 2011) were reported (Figure 1). Fidelity of implementation of the target EBP increased by at least 44% (range: 44.4% - 84.6%, mean = 62.63%) and exceeded 80% fidelity for four specific EBP: peer-mediated instruction/interventions, self-management, naturalistic interventions, and video modeling. In addition, all participating students (n = 18) made progress on all three of their identified annual goals based on Goal Attainment Scaling. Forty-four percent of participating students exceeded expected progress on all three of their goals and 72.2% exceeded expected progress on at least one goal. Overall program quality, as measured by the APERS, also increased for participating California programs (Figure 2). Based on these outcomes, the IAPG determined that the NPDC-ASD model was a good fit for scale-up of EBP for ASD based on the identified needs in California.

Figure 1.

Change in the average fidelity of implementation of seven Evidence-Based Practices in California during the NPDC-ASD Project.

Note: FCT = Functional Communication Training, NAT = Naturalistic Interventions, PII = Parent Implemented Intervention, PMII = Peer Mediated Instruction/Intervention, SM = Self-Management, VM = Video Modeling, VS = Visual Supports.

Figure 2.

Change in program quality in California programs during the NPDC-ASD project.

Note: AM = Assessment, APERS = Autism Program Environment Rating Scale, AVG = Average overall APERS score, CE = Classroom Environment, CI = Curriculum and Instruction, COM = Communication, CS = Classroom Structure, FB = Functional Behavior, FI = Family Involvement, PCC = Positive Classroom Climate, PI = Personal Independence, TM = Teaming, TP = Transition Planning (only assessed in middle and high school programs)

Preparation Phase (January 2012 – June 2013)

Key Activities.

With the goals of gaining internal and external support and planning for implementation, several key activities were targeted during the Preparation phase. These included development of the California Autism Professional Training and Information Network (CAPTAIN); establishment of a leadership team; building interest and recruiting trainers; and accessing funding. Discrete implementation strategies (Powell et al., 2015) employed during Preparation phase included: starting a dissemination organization, promoting network weaving, using an advisory board, facilitation, development of a formal implementation blueprint, development of educational materials, resource sharing, and accessing new funding.

Establishment of the CAPTAIN Leadership Team.

The IAPG was expanded to include additional service system agency members (including developmental disability services and family resource and support agencies) and was renamed the “CAPTAIN leadership team.” This expanded team (n=20) carried forward the mission of “developing a statewide training and technical assistance network with a focus on EBP for individuals impacted by ASD inclusive of stakeholder agencies who will disseminate information at a local level.” Leadership decisions were determined through consensus among members. Workgroups were formed to address specific issues such as membership, website/social media, summit planning, quality improvement/research and development/ongoing management of the online Foundations of Autism class.

Building interest and recruiting cadre.

To build awareness and interest in CAPTAIN, leadership team members provided descriptive presentations about CAPTAIN to key professional organizations across service sectors during the 2012–2013 school year. Based on input from these groups and outcomes from the NPDC-ASD project, the leadership team developed selection criteria and expectations that the stakeholder agencies could use for selection of their agency trainers, called CAPTAIN cadre. In addition to trainers from special education, additional trainers were needed to work with families to build their awareness of EBP and to work with other allied health providers outside of the school system to ensure consistency and cohesion among the various service agencies. A key component for CAPTAIN membership was the requirement of each cadre to actively participate in the process of implementing and/or disseminating information on EBP and ASD. Specific requirements were assigned to each of the above groups based on their skill sets and role definition (see Table 3) and the following key stakeholders were actively recruited:

Table 3.

Cadre Requirements

Foundations in Autism online course	All Cadre
Provide at least one training on ASD and EBPs
Complete annual internal survey
Develop an inter-agency regional plan
Provide 3 trainings on specific EBPs and coaching to 3 teachers/staff	SELPA Cadre
Provide 1 overview training on EBPs for vendors and contracted providers	Regional Center cadre
Provide information on ASD and EBPs to agency staff	Family Support Agency Cadre

Special Education.

Given children with ASD spend a significant portion of the day in a school setting, special education practitioners were recruited to be part of the CAPTAIN cadre. In California, special education services are funded through regional special education local plan areas (SELPAs). SELPA’s provide compliance monitoring as well as training and technical assistance to local education agencies within their catchment area. Each of the 132 SELPAs was offered a designated number of CAPTAIN cadre positions based on the number of students they served who were educationally identified as having Autism (1 cadre member per every 500 identified students with ASD). CAPTAIN cadre representing special education services were selected based on their ability to fulfill the training and coaching requirements and to participate in annual CAPTAIN summits and quarterly inter-agency regional collaboration meetings (see Table 3). Selected cadre were all individuals within the special education system who had the capacity to train others, who demonstrated a strong base of knowledge about ASD and who could support implementation of learned skills and disseminate information to a broad audience.

Developmental Disabilities Services.

In California, services for individuals with developmental disabilities are funded through Regional Centers. Regional Centers are nonprofit private corporations that contract with the California Department of Developmental Services (DDS) to provide Part C/Early Intervention Services and to administer the provision of California’s Lanterman Developmental Disabilities Act, which entitles individuals with developmental disabilities and ASD to services that allow them to live more independent lives within the community. Given that these professionals are responsible for the clinical support, information dissemination and training of Regional Center service coordinators and clinical staff, it is important for them to be up to date on best clinical practices, including EBP for their target population, individuals with Autism. Furthermore, they perform and conduct quality assurance and act as the liaison with contracted community based allied health providers; having knowledge in EBP to make the best decisions in these roles is essential. Autism Coordinators from the state’s 21 Regional Centers were invited to participate as cadre. Regional Center cadre were selected by agency leaders based on their ability to fulfill the training requirements and to participate in annual CAPTAIN summits and quarterly inter-agency regional collaboration meetings (see Table 3).

Family Support Agencies.

Children spend most of their time with their families, an often-forgotten training partner. Frequently, skills carefully taught at school are not taught to parents, thus, greatly reducing generalization and successful clinical outcomes. To minimize this potential gap, participants from the nine California Early Start Family Resource Center (FRC) Regions and fourteen California Family Empowerment Centers (FECs) were invited to participate. FRCs are funded under the Individuals with Disabilities Education Act (IDEA) Part C/Early Intervention to provide parent to parent support to families receiving early intervention services (ages birth to 3). Likewise, FECs are funded under IDEA/Part B to provide parent information and advocacy training to families who have children with disabilities between ages 3 to 22. Cadre from the FRCs and FECs were identified by their agency leaders based on their abilities to disseminate information to their agency staff and to families within their regions about ASD and EBP learned through participation in the CAPTAIN summits and activities (see Table 3).

Funding and Resources.

Initial funding for CAPTAIN activities was very limited. However, to emphasize the importance of how funding has been linked to sustainment of this model, a description of resources is provided below.

In-kind Funding for Leadership Team Participation.

Leadership team members’ time and travel for CAPTAIN activities was provided through in-kind support by their employing agencies. In addition, leaders volunteered their time to ensure that this initiative was successful. The CEDD at the UC Davis MIND Institute provided 10% dedicated time for a staff member to act as the CAPTAIN Project Coordinator. In addition, the CDE, Diagnostic Center committed 10% dedicated time to support CAPTAIN and develop and sustain the CAPTAIN website.

Participating Agency Support for Cadre Participation.

To cover the costs of the annual training summits and to provide funding for some ongoing leadership activities, nominating agencies were asked to pay a $125 registration fee for each Cadre to attend the annual training summit. Due to the vast size of California, a northern summit and southern summit were held to minimize travel and time challenges, with the same program repeated at each site. Fees were collected by the hosting agencies (two of the State SELPAs), who also acted as the fiscal agents for the CAPTAIN project.

CAPTAIN Website.

The CAPTAIN website, www.captain.ca.gov, was developed through in-kind support by the CDE, Diagnostic Center North. The website contains useful links to the NDPC – ASD resources, the National Autism Center, and California state agencies including the Department of Developmental Services, the California Department of Education and the University Centers for Excellence in Developmental Disabilities (UCEDDs). In addition, the website contains all of the presentations from the annual training summits, contact information for the CAPTAIN leadership team and CAPTAIN Cadre who support training at the local level, and additional resources such as infographics and regional brochures.

Funding the Foundations of Autism Online Class.

Seed funding of $1000 was provided by the California Services for Technical Assistance and Training program, which is part of the CDE, Special Education Division to support professional development of special educators and their families. This one-time funding supported enrollment for an online Foundations of Autism class to all newly nominated CAPTAIN Cadre. Completion of this class was a requirement to insure a common foundation of knowledge about ASD and EBP. This 10-hr class was originally developed and offered by the University of North Carolina and the NPDC-ASD. Through a collaboration with UC Davis Extension and the CEDD at the UC Davis MIND Institute, CAPTAIN updated and revised the Foundations of Autism Class in 2015. The current version of this class now addresses lifespan issues and is available for no charge through the Coursera learning platform. All new CAPTAIN cadre are required to complete the class and are also expected to share this class as a resource to other community members to assist with building awareness about ASD.

Outcomes and Lessons Learned.

CAPTAIN Cadre were selected and nominated by their respective agencies. Presentations to the agency leaders outlined the qualifications and requirements for cadre (see Table 3). Nominated members were contacted in June 2013 by CAPTAIN leadership team (n = 20) to finalize their participation and enroll them in the Foundations of Autism Class. During the 2013–2014 academic year there were a total of 341 Cadre nominated by their respective agencies (277 of the members from SELPAs, 38 from Regional Centers, 13 from Family Support Agencies, 5 from Universities and 8 from state agencies including CDE and DDS). Approximately 275 cadre completed this Foundations of Autism online class during the 2013–2014 year and an additional 60 completed it during the 2014–2015 year.

Implementation Phase (June 2013 –July 2014)

Key Activities.

With the goal of actively putting the program into place, key activities of the Implementation phase were identifying cadre current knowledge and practices, providing an initial training summit and post-summit support. Discrete implementation strategies (Powell et al., 2015) employed during Implementation phase included: identifying barriers and facilitators, conducting educational meetings, distributing educational materials, providing local technical assistance, using train-the-trainer strategies and organizing implementation team meetings.

Training Summits.

The first annual CAPTAIN Summits were held on October 3–4, 2013 in Riverside, CA and on October 17–18 in Stockton, CA. The goals of the initial summit were to provide resources to assist cadre with dissemination of information about ASD and EBP, train SELPA Cadre to implement the NPDC-ASD model, and facilitate regional collaborations between schools, Regional Centers and family support agencies to better leverage local resources.

Summit content was developed by CAPTAIN leaders based on the NPDC-ASD model and identified “Hot Topics” in ASD policy, treatment and research. Showcase presentations by the original six NPDC-ASD demonstration sites were given to illustrate real world examples of how the NPDC-ASD model can be used in public schools. Regional Center and family support agency staff were provided with tools and resources for information dissemination and “Overview of ASD and EBP” trainings. Presenters modeled how to access the training materials and implementation resources on the CAPTAIN website.

In addition, time was provided for regional cross-agency teams to meet and have conversations that lead to the development of regional implementation plans. Due to the size of California, we broke the state up into 17 regions, with each region being comprised of at least one Regional Center, at least one family support agency and as many SELPAs as operate within that same boundary. The regional planning was designed to be the key format to increase inter-agency collaborations. The regional planning time was facilitated by a CAPTAIN leader; however, teams were encouraged to identify internal mechanisms for ongoing communication and facilitation. All regional plans were captured by the teams using Goal Attainment Scaling for each of their regional goals. Goals ranged from plans to host regional inter-agency conferences to simply agreeing to meet again as a region in the same physical space to share information and resources and build relationships within the region.

Post Summit Support.

Throughout the first year of implementation of the CAPTAIN network, leaders supported cadre through quarterly conference calls or face-to-face meetings, and content sharing using the CAPTAIN website and informational emails. More recently, these communication efforts have expanded to include social media as well (including Facebook, Instagram and Twitter), where information can be shared with and between cadre and more broadly.

Survey of Cadre.

Beginning in 2013, cadre have been surveyed annually to gather demographics, assess baseline knowledge about ASD and EBP and to gather information about needs related to the CAPTAIN goals. The information has been used by the CAPTAIN leadership team to determine content for the annual summits and has been used as a tool for quality evaluation and continuous program improvement. In 2016, we received approval from the UC Davis Institutional Review Board to use survey data for research purposes and all cadre henceforth had the option to consent for their responses to be used for research purposes or to “opt out” and complete the survey solely for internal quality improvement purposes. All data reported in this paper have been approved for research use and are drawn from the 2017 survey.

The 128-item survey, including subsections on demographics, organizational and regional practices, and goal fulfillment. The survey was distributed to cadre via an email with an embedded link using the Qualtrics web-based survey application. For the purposes of this manuscript, a subset of questions was analyzed.

Outcomes and Lessons Learned.

Cadre Membership.

A total of 407 Cadre (89% of the membership) completed the survey, with 317 members returning from previous years. The majority of members represented SELPA or school districts (78.1%), followed by Regional Centers (10.1%), Family Support Agencies (4.2%), Department of Developmental Services (.7%), California Department of Education (1.7%), Universities (3.7%) and other organizations (1.5%). The CAPTAIN membership represents 94% of SELPAs, 90% of Regional Centers and 100% of Family Resource Center Network regions statewide. This group was a highly educated and experienced group. The majority of members had a Master’s Degree (76.4%), followed by a Bachelor’s Degree (13.5%), and a Doctorate Degree (8.1%). The most reported credential amongst members was Education Specialist/Special Education Teaching Credential (43.5%), followed by a BCBA/BCaBA certificate (26.0%), and CA School Administrators Credential (19.2%). The average number of years in the field of Autism Services was 15.85 (SD = 10.09). The average number of years in their current position was 5.83 (SD = 7.74).

Of the cadre members representing SELPAs/schools, job classifications included Behavior Analysts/Specialists (22.7%), Program Specialists (21.8%), and Administrators/Program Managers (18.9%). Over 75% of Regional Center cadre were either Behavior Specialists/Behavior Analysts (29.3%), Case Management Supervisors (24.4%), and Autism Specialists (22.0%). The majority of Family Support Agency cadre were Advocates for People with Autism (64.7%).

Regional Practices/Activities.

A major component of CAPTAIN is the follow through by cadre on required activities (see Table 3) between summits. Over half of the members reported attending their Regional Collaboration meetings quarterly (56.9%) as required. Three out of four members are using their region’s Goal Attainment Scaling goals developed at the annual summit to guide their work during regional meetings (75.4%). Cadre members believe the regional collaboration meetings are extremely important (42.5%) and are moderately valuable (40.3%) to the mission of CAPTAIN. In addition, many cadre met with their agency leaders (36.4%) and direct supervisors (45.0%) to discuss CAPTAIN efforts more than three times during the year. See Table 4 for additional data on regional collaboration.

Table 4.

Regional Collaboration and Goal Fulfillment

Regional Collaboration	n (%) of cadre
Regional Collaboration Meeting Attendance
Quarterly	178 (56.9%)
Once	72 (23.0%)
Did Not Attend	62 (19.8%)
Meeting w/ Agency Leader about CAPTAIN activities
More than 3 times	114 (36.4%)
3 times	37 (11.8%)
2 times	45 (14.4%)
1 time	55 (17.6%)
Never	61 (19.5%)
Meeting w/ Direct Supervisor about CAPTAIN activities
More than 3 times	141 (45.0%)
3 times	35 (11.2%)
2 times	50 (16.0%)
1 time	50 (16.0%)
Never	34 (10.9%)
Goal Fulfillment	n (%) of cadre
Providing Awareness Training on EBP/ASD
More than one	216 (69.0%)
One	78 (24.9%)
Did not provide	19 (6.1%)
Provided Training on Specific EBP
More than three	120 (49.4%)
Three	40 (16.5%)
Two	39 (16.0%)
One	35 (14.4%)
Did not provide	9 (3.7%)
Provided Coaching to Teachers/Programs
More than three	126 (51.9%)
Three	31 (12.8%)
Two	41 (16.9%)
One	25 (10.3%)
Did not provide	20 (8.2%)

Note: EBP = Evidence-based practices

The majority of cadre met or exceeded expectations for training and coaching requirements. Sixty-nine percent met the training requirement of providing awareness training on EBP and ASD and almost half (49.4%) met the requirement of providing trainings on specific EBP. Over half (51.9%) of the cadre are providing coaching to more than three teachers/programs. See Table 4 for additional data on goal fulfillment.

Barriers and Resources.

Our survey sought to identify both facilitators and barriers to implementation. The top 3 barriers to implementation of EBP identified by the cadre were: Time for Training (25.6%), Lack of Substitutes (16.5%), and Staff Lacking Foundational Skills (11.5%). See Table 5 for additional data on barriers. The top three most used resources were the Autism Focused Internet Resources and Modules (64.4%), the CAPTAIN website (59.5%), and Autism Internet Modules (50.1%), all being used by at least half of the cadre.

Table 5.

Barriers Reported by CAPTAIN Cadre

	n (%)
Time for Training	80 (25.6%)
Time for Coaching	33 (10.5%)
Staff Buy-in	26 (8.3%)
Administrative Support	20 (6.4%)
Budgets/Funding	34 (10.9%)
Foundational Skills	36 (11.5%)
Lack of Substitutes	40 (16.5%)
Role Does Not Allow for Training/Coaching	19 (6.1%)
Rate of Staff Turnover	4 (1.3%)
Lack of Resources	9 (3.7%)
Other	17 (5.4%)

Sustainment Phase: Ongoing Training Activities (2014 – present)

Key Activities.

With the goal of continuing to nurture and improve an embedded program, key activities during the Sustainment phase involve ongoing communication with cadre and agency leaders, increasing awareness through professional conference presentations, annual recruitment of new cadre, annual summit training activities and regional collaborations. Discrete implementation strategies (Powell et al., 2015) employed during Sustainment included: capturing and sharing local knowledge, conducting ongoing training and educational outreach and ongoing pursuit of new funding.

Regular Communication with Agency Leadership.

To facilitate their institutional support of CAPTAIN, CAPTAIN leaders present to agency leaders twice a year. To encourage their participation in the important process of recruitment and selection of leaders in EBP for ASD from their respective agencies, each fall CAPTAIN leaders review the nomination procedures and selection criteria. To provide ongoing interest in CAPTAIN, each spring, CAPTAIN leaders present an overview of the summit content and outcomes of the annual survey of cadre. Successes are shared as well as specific barriers to training and coaching so that agency leaders can provide support as needed. Additionally, beginning in 2016 CAPTAIN cadre received the additional requirement of meeting with their agency leaders immediately following the annual summit to share back information, the regional plan, and to work with leaders to secure the needed time and resources to fulfill their CAPTAIN requirements. The addition of this step has decreased several of the perceived barriers, and cadre outcome data will be analyzed to determine the impact of administrator contact on training and coaching outcomes.

Increasing Awareness with professional associations.

CAPTAIN Leaders also conduct ongoing information dissemination by presenting at state-level professional conferences. These conferences include annual or semi-annual meetings of the California Association of School Psychologists, the California Association of Resource Specialists and Special Education Teachers, the Association of California School Administrators, the California Speech-Language-Hearing Association, the California Association for Behavior Analysis, the California Council on Teacher Education, and the California Psychological Association. Presentations focus on providing overview information about ASD and EBP, how to access information on the CAPTAIN website, and how to access local cadre for additional trainings and support. CAPTAIN leaders also present implementation outcomes at academic and scientific meetings at state, national and international levels, including the International Society for Autism Research, the Association of University Centers on Disabilities, and the Council for Exceptional Children’s Division on Autism and Developmental Disabilities.

Annual recruitment of new cadre.

Routine staffing changes necessitate annual recruitment of new Cadre. As more agencies become aware of CAPTAIN, there has also been an increased interest in the network, with more agencies requesting trained Cadre. Since its inception 1028 Cadre have been trained, and CAPTAIN annual membership has increased by 20.8% (n=412). We intentionally limit annual membership to ensure a high level of fidelity by providing close supervision of our cadre and their training activities. Previously, nominations were closed by the end of the academic year in June, but invariably there would be staffing changes over the summer, particularly with the SELPA cadre, which led to another round of decision making for the replacements. To minimize this problem, agency leaders now nominate new cadre from August-October each year. Presently 28.7% of the original cadre who joined in 2013 are still active members.

Annual training summit activities.

Prior to each annual summit, new cadre complete the online Foundations of Autism class and attend a three-hour “Bootcamp” training that orients them to how to be a CAPTAIN Cadre, implement the NPDC-ASD model and use the CAPTAIN resources to fulfill their cadre requirements. All cadre then participate in the following training activities: “Hot Topics” sessions covering ASD updates, new resources on the CAPTAIN website, and current ASD policy issues; “Job Alike” breakout sessions where cadre meet with their peers in similar roles and share how they implement and disseminate EBP at their agency as well as help each other problem solve challenges; “Showcase Presentations” where members share their successes and lessons learned; and “Regional Planning” time to develop annual regional goals that cadre will then implement during the interval period between summits. We also include a feedback loop to cadre called “Year in Review” to share the results of the annual cadre survey and showcase the many successes in training and implementation accomplished by cadre over the previous year. It also reinforces the importance of completing the survey and highlights the importance of data informed decision making. This activity highlights the positive impact implementing EBP for ASD with fidelity can make.

Regionalized collaborations.

Since one of the primary goals of CAPTAIN has been to improve inter-agency collaboration, we have been careful to support this across the 17 unique and diverse regions in this state. Prior to CAPTAIN, some regions had established relationships between schools, Regional Centers and family support agencies; however, others had limited contact or even adversarial relationships with historical barriers to overcome. As indicated by the regional collaboration meeting participation data, frequency of regional collaborations has increased in all of the 17 CAPTAIN regions. In addition, when the network started, only one of the regional groups had an established interagency regional conference that they held each year. Current data from the 2018–2019 school year indicate 10 of the 17 regional teams have collaborated to conduct cross-agency regional conferences to support dissemination. These data indicate improved collaborations that may also improve alignment of services and family supports between agencies. Having a shared vision and definition of EBP can support families in seeking services and can support continued access across grade levels and service systems as children move from early intervention, to school services and into Regional Center supported adult services.

Facilitators of Success and Recommendations

Although CAPTAIN was created to address implementation and dissemination of EBP for ASD, we hope that this model and the alignment with implementation science frameworks can support reduction of the research-to-science gap for other content issues that can benefit from interagency collaborations.

Commitment to EBP and effective dissemination and implementation practices.

A core value of CAPTAIN is that all cadre must be involved in the implementation and dissemination of EBP for ASD, and guiding requirements were created to align with cadre professional roles and respective agencies to support these efforts (see Table 3). CAPTAIN success is also directly related to the personal investment of individual cadre. Regional groups and agency leaders recruit local cadre who have demonstrated a commitment to best practices when working with individuals with ASD. Given time and service requirements needed to facilitate EBP training, participation in the regional cadre requires commitment from the individual as well as the agency they represent. Although cadre participate with “in-kind” support from their respective agencies, many volunteer on their own time. These dedicated cadre share experiences and resources with each other and learn from other regional collaboratives. Given the critical roles of cadre and leader commitment to both evidence-based interventions for individuals with ASD and evidence-based dissemination and implementation strategies, we recommend these considerations be integrated into selection of clinical and academic partners.

Active multi-level interagency collaboration.

Interagency collaboration has been a key feature of CAPTAIN since initial exploration activities. Additionally, multi-level engagement of leaders has facilitated support and continuous improvement. The centralized leadership team is comprised of representatives from the key stakeholder groups from across the state. This group coordinates all CAPTAIN activities and engages with state-level agency leaders bi-annually. Leadership team members also guide regional activities to focus on the overall CAPTAIN mission and support cadre engagement. Since the summit is only held annually, regional communities of practice and implementation support is critical. Regional collaboratives allow for CAPTAIN to more uniquely address these specific needs of each region. California is a large state with agency and provider diversity and regional nuances impacting implementation. Innovative strategies developed through regionalized interagency relationships address issues such as rural and underserved areas, ethnic and cultural values and emergency procedures necessitated by a recent wave of natural disasters. We recommend strategic development of interagency collaboration or networks of implementation.

Creative funding and leveraging of local resources.

Because the work of the IAPG and CAPTAIN has not been tied to consistent funding, the involved agencies leveraged their own internal resources in order to support the project. Reassigning staff, collaborating with other agencies in local trainings and awareness events and sharing resources between and among members of the CAPTAIN network has allowed the model to continue to grow even without dedicated funding. One such example has been the donation of space for annual summits. The agencies with the largest meeting facilities have provided physical space for summits so that the cost of renting a conference center was not a financial barrier to our success. By leveraging what is available, rather than relying on new sources of funding, CAPTAIN has been able to grow and establish itself as a valuable resource within the state. With this in mind, we recommend a creative approach to funding and resource sharing.

Data-based program improvement.

Survey data have been used to identify barriers to CAPTAIN cadre implementation. Barriers included lack of time for training, lack of substitutes, and staff lacking foundational skills as well as administrative support. These data have been used to inform the required practices of CAPTAIN cadre and agency leaders. This use of implementation science practices and continuous improvement cycles is an essential component of our success, therefore, we recommend investment in meaningful data sharing to inform improvement efforts.

Summary and Next Steps

Research has demonstrated that EBP for ASD, when applied with fidelity, can greatly enhance the desired outcomes of individuals (Wong et al., 2015). However, research on implementation of EBP for ASD in community services indicates low to moderate fidelity. A primary goal of CAPTAIN was to create an interagency collaborative to advance the dissemination and implementation of EBP for ASD across service sectors throughout California. The CAPTAIN model employs a statewide annual training summit and regional expertise to provide ongoing training, support and technical assistance to direct service providers. Drawing on research supported frameworks from implementation science, CAPTAIN has developed from a clinical collaboration into a strategic model for state-wide implementation and scale-up of EBP.

After five years of successful implementation with participating agencies with very limited funding, in February 2019, the California Department of Education, through a newly appropriated program improvement grant, awarded CAPTAIN and Marin County SELPA $1.1 million dollars per year for 5 years to further develop the CAPTAIN model and capacity to assist underperforming school districts in California to improve educational outcomes for students with ASD. The aim of this initiative is to help school districts develop their capacity to use EBP and improve student outcomes. This will require bringing the NPDC-ASD model to scale at the practitioner level and will require bringing the CAPTAIN implementation model to scale to support the organizational capacity at the systems level. Additionally, funding through Institute of Educational Science is supporting exploration of mechanisms related to CAPTAIN implementation outcomes and targeted educational outcomes at district and teacher levels (Stahmer et al, 2018).

The data presented in this manuscript describe the CAPTAIN model for statewide scale up EBP through inter-agency collaboration and facilitative leadership at the state and regional level. We discuss how identified barriers can be used to inform future directions for further dissemination efforts, through targeted data collection and feedback to both providers and organizational leaders.