Evaluating a Learning Collaborative to Implement Evidence-Informed Engagement Strategies in Community-Based Services for Young Children

Attendance at community-based child mental health services is a longstanding concern. Studies have found no-shows rates at intake appointments to be 30–60%, and similarly, 40–60% of children receiving outpatient services drop out quickly after few sessions (Gopalan, Goldsteing, Klingerstein, Sicher, Blake, & McKay, 2010). Several attendance engagement interventions have been designed to address barriers to initial and ongoing attendance and have shown positive effects on attendance outcomes (Gopalan et al., 2010; Ingoldsby, 2010). While a number of interventions with promising effects on attendance exist, very few have been implemented beyond experimental trials overseen by the treatment developers. Given the dismal state of attendance in community-based care, it is critical that evidence-informed attendance engagement strategies be implemented within the systems that face this great challenge to serving children and families.

There is growing research on effective methods for training in evidence-based practices (EBPs), and the literature strongly suggests that stand-alone workshops are not sufficient for successful uptake of an intervention into a service system (Herschell, Kolko, Baumann, & Davis, 2010; Lyon, Stirman, Kerns, & Bruns, 2011). One model that shows promise for training providers on EBPs, including evidence-informed attendance engagement interventions, is the Institute for Healthcare Improvement’s (IHI) Breakthrough Series Collaborative method for implementing systematic quality improvement efforts (IHI, 2003; Kilo, 1998). This method includes bringing together teams from multiple programs who participate in learning sessions and briefer check-ins facilitated by one or more experts in the focused area of change. Action periods take place between learning sessions and check-ins, allowing for time for each team to implement change ideas identified during meetings and study the effectiveness of those change ideas using the Plan-Do-Study-Act (PDSA) cycle of learning (IHI, 2003; Kilo, 1998). A unique hallmark of the Breakthrough Series Collaborative method is its focus on shared learning among members both within and across programs. The method also emphasizes evidence-based decision making, the need for transparency, and collaboration among providers and organizations to enact change at both levels (Institute of Medicine [IOM], 2001).

The Breakthrough Series Collaborative method has been used extensively within physical health care systems to enhance quality improvement (Mittman, 2004; Schouten, Hulscher, van Everdingen, Huijsman, & Grol, 2008), and is starting to be used more widely with some success within the adult mental health field to enhance attendance (Roosa, Scripa, Zastowny, & Ford, 2011; Rutkowski et al., 2010), implement EBPs (Roosa et al., 2011), and increase overall quality (Solberg et al., 2001; Strating, Broer, van Rooijen, Bal, & Nieboer, 2012). However, very few studies have examined this implementation method within the field of child mental health. One recent effort evaluated the use of the method to implement Trauma Focused-Cognitive Behavioral Therapy (TF-CBT) (Ebert, Amaya-Jackson, Markiewicz, Kisiel, & Fairbank, 2012) and focused on several important implementation outcomes. Implementation outcomes assess the effects of actions designed to facilitate the implementation of an innovation and can include acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability (Proctor et al., 2011). Implementation outcomes are not only indicators of implementation success but are also proximal indicators of the implementation process and can serve as intermediate outcomes that may relate to service system or clinical outcomes (Proctor et al., 2011). Ebert et al.’s (2012) results indicated that: 1) all participating agencies were implementing TF-CBT by the end of the collaborative (adoption); 2) providers were providing TF-CBT as intended based on provider and supervisor ratings (fidelity); and 3) at one-year follow-up the majority of agencies were providing TF-CBT supervision and fidelity monitoring (sustainability). Ebert and colleagues (2012) also reported implementation outcomes of the collaborative method itself, such as high acceptability and overall fidelity to requirements (e.g., submitting PDSA plans, creating monthly metrics to monitor improvements). In addition, (Epstein, Langberg, Lichtenstein, Kolb, & Stark, 2010; Epstein et al., 2008) found that the collaborative method was successful in significantly increasing primary care physicians’ adoption and sustainability of evidence-based practice guidelines for the diagnosis and treatment of ADHD. Adult mental health studies of the collaborative method have also examined implementation outcomes of the collaborative method and found positive effects on outcomes such as adoption, fidelity, and acceptability (e.g., Duffy, 2008; Vannoy et al., 2011; Versteeg, Laurant, Franx, Jacobs, & Wensing, 2012).

Looking specifically at the collaborative method and the implementation of evidence-informed attendance engagement efforts in child mental health services, two studies by Cavaleri, and colleagues (Cavaleri et al., 2006; Cavaleri et al., 2010) examined whether an adaptation of the Breakthrough Series Collaborative method was effective in enabling mental health clinics to utilize McKay and colleagues’ evidence-informed attendance engagement strategies to improve treatment attendance (McKay, McCadam, & Gonzales, 1996; McKay, Nudelman, McCadam, & Gonzales, 1996; McKay, Stoewe, McCadam, & Gonzales, 1998). Both studies found that attendance increased following the completion of a learning collaborative (LC) based on the method (Cavaleri et al., 2006; Cavaleri et al., 2010), indicating that LC’s are a successful implementation approach for evidence-informed engagement strategies. A third study examined sustainability of the evidence-informed strategies following completion of the LC (Cavaleri et al., 2007). The authors found that almost all participating agencies had continued one or more of the strategies implemented through the LC. Factors that enhanced sustainability included the effectiveness of the strategies, the ease of implementation, and staff buy-in, while challenges to sustainability included time, adequate staffing, and lack of technology. Taken together, the Ebert et al. (2012) and Cavaleri et al. (2007; 2006; 2010) studies of the Breakthrough Series Collaborative method provide preliminary support for this implementation method in children’s mental health service systems.

It is not known whether this implementation strategy is feasible in other children’s services systems that focus on child mental health such as early childhood intervention. Although there is considerable variability across states, young children with recognized developmental or behavior problems receive services through Parts C (ages 0 to 3 years) and B (ages 3 to 4 years) of the Individuals with Disabilities Education Act (IDEA, 2004). They may also receive services through other state or local programs (e.g., First 5 Commission of San Diego) that focus on early intervention for children with more mild developmental delays or broader school readiness. Although services and implementation research in the area of early intervention for autism is growing (e.g., Brookman-Frazee, Stahmer, Lewis, Feder, & Reed, 2012; Stahmer & Aarons, 2009; Stahmer, Collings, & Palinkas, 2005), less attention has been paid to the broader population of young children at risk for social-emotional and developmental problems. Providers in the early intervention system likely differ from those in mental health. For example, research suggests that providers in early intervention versus mental health systems differ in attitudes towards EBPs, such that early intervention providers exhibit more favorable attitudes towards adopting EBPs than mental health providers (Stahmer & Aarons, 2009). Early intervention systems likely differ from the traditional mental health systems in a number of ways, including the target populations (age and clinical focus on developmental problems or risk factors), provider background and education, and funding/organizational structure (special education vs. behavioral health) as well as different attitudes towards evidence-based practices (Stahmer & Aarons, 2009). Importantly, there is requirement for parent involvement for these services, including the creation of individualized family service plans, the implementation of family interventions, and attention to family outcomes and satisfaction with services (see IDEA, 2004, §§1470–1482).

In addition to a lack of knowledge regarding whether the Breakthrough Series Collaborative method is feasible in early intervention systems, important implementation outcomes have not been examined using this method. For example, the implementation field has paid close attention in recent years to one element of acceptability, provider attitudes about EBPs (Aarons, 2005; Proctor et al., 2011). Research in the adoption of innovations suggests that provider attitudes towards EBPs will predict the likelihood of adopting such practices (Nelson & Steele, 2007; Rogers, 2003). Studies have begun to examine the effects of EBP training on provider attitudes about EBPs and have found mixed results regarding improvement in attitudes following training (Borntrager, Chorpita, Higa-McMillan, & Weisz, 2009; Lopez, Osterberg, Jensen-Doss, & Rae, 2011). To date, no study has examined whether collaborative-based training efforts improve attitudes about EBPs.

The current study examines implementation outcomes of a Breakthrough Series Collaborative-based LC to implement McKay and colleagues’ evidence-informed attendance engagement strategies in a set of community-based early intervention programs. This study expands the existing literature in several ways. First, this study examines the collaborative method in a different and critically important context - early intervention programs that serve young children with socio-emotional and developmental needs and their families across both the low-income and child welfare populations. Attendance is particularly important for early intervention services and for families involved in the child welfare system. Second, this study utilizes a train-the-trainer implementation model; the LC was not facilitated by the treatment developers but rather by a trainer certified on the attendance engagement intervention (Herschell et al., 2010). Third, this study assesses several implementation outcomes that have not been widely examined, including whether attitudes about EBPs improve following completion of the LC.

We apply the Proctor et al. (2011) taxonomy of implementation outcomes to frame the outcomes examined in this study, namely feasibility, acceptability, adoption, fidelity, and sustainability. Based on the limited extant literature on efforts to improve implementation outcomes more broadly as well as specifically linked to LC’s, we hypothesize the following : 1) the LC implementation method will be feasible to support the implementation of evidence-informed attendance strategies with early intervention providers (feasibility); 2) both LC participants’ perceptions of the desired outcome (improved client attendance) and attitudes towards evidence-based strategies will improve after participating in the LC (acceptability); 3) LC participants will report adopting evidence-informed engagement strategies to promote attendance and these strategies will be consistent with the intervention (adoption and fidelity); and 4) LC participants will identify many planned strategies to sustain practice changes following completion of the LC (sustainability).

Method

Participants

Participants were employees from four community-based programs within the Developmental Services Division at a large regional children’s hospital in Southern California (see Table 1). The mission of the participating programs is to prevent, identify, and address social-emotional and developmental needs in young children. All programs receive funding from the First 5 Commission of San Diego, funded by a state-wide tobacco tax (first5sandiego.org). The programs requested and paid for training on evidence-informed attendance engagement strategies and the accompanying LC. A total of 29 participants started the LC. One team for each program was created, and program teams ranged from 5 to 11 and the average was 7.5. Teams were not separated into control and intervention groups; all teams received the LC intervention. Three participants dropped (two left their program and one dropped due to time demands) and one participant was added after the start of the LC. Participants were selected by their program leader. No standard selection criteria were applied, but program leaders were encouraged to select participants who were enthusiastic about quality improvement, motivated to improve client attendance, and represented a range of professional roles within the program.

Table 1.

Program Descriptions.

Program	Description & Target Population	Services Provided
Program A	Serves children with complex mental health, developmental, medical and/or family needs through a trans-disciplinary approach to assessment and treatment Serves children in the child welfare system Services on-site and community- based 33 staff hired for time equivalent to 19 full-time positions	Triage Comprehensive assessment Care plan development with integrated clinical team meetings, including parents and caregivers, to discuss and refine care plan Treatment (mental health and developmental ) Intensive case management Parent and caregiver support
Program B	Serves children identified with social-emotional/behavioral and developmental concerns who are receiving services as part of an early childhood prevention/intervention network Most services provided via phone 11 full-time staff	Case management
Program C	Identifies and addresses social- emotional/behavioral and developmental concerns in children to promote kindergarten readiness and facilitate healthy development Services on-site and community- based 27 staff hired for time equivalent to 19 full-time positions	Screening Assessment Developmental treatment classes Specialized behavior treatment classes Individual developmental and behavior treatment consultations
Program D	Identifies and addresses the social- emotional and developmental needs of children entering the child welfare system Services are community-based 40 staff hired for time equivalent to 30 full-time positions	Social-emotional/behavioral and developmental screening Facilitated linkage to services Developmental enrichment for children Behavioral coaching for caregivers Intervention for children admitted to a local temporary placement

Note: All programs serve children ages 0 to 5 and their families.

Procedures

The intervention training and the LC implementation method are part of a training package based on McKay and colleagues’ attendance engagement intervention (McKay, McCadam et al., 1996; McKay, Nudelman et al., 1996; McKay et al., 1998). The training package, known as Training Intervention for the Engagement of Families or TIES (www.tiesengagement.com), follows a train-the-trainer model whereby a group of engagement researchers/trainers became certified in providing the training and LC to community agencies. The first author is a certified trainer and a paid consultant of Danya International, which owns the rights to the TIES intervention and LC implementation protocol and is the entity that coordinates TIES trainings throughout the country. In the current study, the first author co-conducted the training with Dr. McKay (October 2010) and independently facilitated the LC (January-September 2011).

Intervention: TIES evidence-informed attendance engagement strategies

TIES has a strong conceptual and empirical basis. In particular, the extant literature on barriers to attendance in child mental health services has indicated that perceptual barriers such as parents’ attitudes about mental health care and the extent to which care matches their expectations are more important in hindering attendance than concrete barriers such as child care and transportation (Bannon and McKay 2005; McKay et al. 2001). TIES is designed to focus on these perceptual barriers through the first telephone contact with a family as well as during the first clinical interview, and has been successful in significantly increasing intake and subsequent appointments compared to usual care (McKay, McCadam et al., 1996; McKay, Nudelman et al., 1996; McKay et al., 1998).

The TIES training was conducted over one full day at a community conference center and consisted of didactics, interactive exercises, and handouts designed to facilitate participants’ learning of telephone and first-interview strategies. The telephone engagement strategies include clarifying the need for mental health services, increasing parent or caregiver investment and efficacy, identifying attitudes about previous experiences with mental health services and service delivery institutions, and problem-solving around concrete obstacles to care. The first clinical interview strategies include clarifying the helping process for the client, developing the foundation for a collaborative working relationship, focusing on immediate, practical concerns, and identifying and problem-solving around barriers to utilizing the program’s services.

Participation in initial TIES training

The initial training was open to all employees from the four programs; LC teams were selected by program leaders after the training. Attendance at the training was not an LC eligibility requirement. A total of 58% of LC participants reported that they attended the full-day training. The remaining participants had reviewed training slides, asked a colleague who had attended the training about the content, and/or reviewed materials in a staff meeting.

Implementation method: TIES LC

The first phase of the TIES LC was to engage program leaders in the LC process. The LC facilitator met with program leaders prior to the start of the LC to provide information about the LC process and to encourage leaders to select participants for their team who were most likely to benefit from participation. Leaders were provided with paperwork to complete, including a description of the specific requirements of the LC that leaders had to sign. Each program selected one team for a total of four teams participating in the LC.

The second phase of the TIES LC, the implementation curriculum, consisted of nine monthly meetings to encourage systematic implementation of TIES strategies. Meetings were held at the hospital that administers the four programs (although many participants worked at satellite locations and had to travel to attend meetings). The LC curriculum for the current study consisted of three 3-hour learning sessions with two 1-hour check-in meetings between each learning session (this is somewhat shorter than the typical collaborative but was necessary given funding constraints). These nine sessions are conceptualized as three action cycles (one learning session and two check-ins per cycle). Participants were required to attend learning sessions in person and had the option to attend check-in meetings either in person or via telephone. Learning session content included: 1) didactics on continuous quality improvement strategies and methods for implementing and sustaining organizational change; 2) opportunities for programs to examine their current engagement process, identify measurable attendance-related targets, and develop change ideas and a plan to implement change ideas; and 3) opportunities for programs to provide feedback to each other about the selection, implementation, and evaluation of change ideas. Integrating consumer feedback into the development and implementation of change ideas was emphasized throughout the LC. Check-in meetings followed a set structure with a representative from each program reporting on progress with opportunities for all participants to provide feedback and troubleshoot any challenges. Participants had access to the facilitator via email over the course of the LC, and on two occasions the facilitator met with separate programs’ participants at the program leaders’ request. In addition, a Google Group was formed to share materials and encourage cross-program sharing and feedback. Each team was asked to select at least one target related to attendance (e.g., attendance at the intake, increased overall attendance, program completion). Teams submitted monthly Plan-Do-Study-Act (PDSA) reports for each of their selected change ideas to the LC facilitator, as well as a long-range planning report at the end of the LC.

LC evaluation procedures

At the start of the initial LC learning session, participants completed a Pre-LC Survey and a measure on attitudes about EBPs. A total of 26 of 29 participants completed these assessments. At the end of the LC, participants completed an LC Perceptions Survey as well as the attitudes measure again. A total of 22 of 27 participants completed these assessments. Both surveys were developed to evaluate this LC. All participant-level data were collected anonymously as part of an evaluation of the LC for quality improvement purposes (i.e., ID numbers were generated by participants rather than generated by the facilitator) and IRB approval was later obtained to analyze the unidentifiable data. Participants were not compensated for completion of evaluation data.

Measures

Background information

Participants were asked at the start of the LC to report their age, gender, education level, racial/ethnic group, professional role, and previous TIES training participation. The remaining measures are organized according to the constructs outlined in the Proctor and colleagues’ (2011) implementation outcome taxonomy. The results are organized similarly.

Feasibility of the LC implementation method

Attendance: sign-in sheet

Participant attendance was tracked through a sign-in sheet at each session (participants who called in were added to the sign-in sheet by the facilitator).

Participant perceptions of the LC process: satisfaction

As part of the LC Perceptions Survey, participants were asked to rate on a four-point scale one item related to their satisfaction with the LC and the degree to which the LC was worth their time.

Acceptability of TIES Strategies and EBPs Generally

Participant perceptions of impact on client attendance: Perceived improvement on attendance

As part of the LC Perceptions Survey, participants rated on a four-point scale one item related to the degree to which their participation in the LC resulted in improved family attendance.

Participant attitudes towards EBPs: Modified Practice Attitudes Scale (MPAS)

To measure their attitudes towards EBPs, participants completed an adaptation of the Modified Practice Attitudes Scale (Borntrager et al., 2009). This measure was selected given its observed sensitivity to change when examining attitudes about evidence-based interventions that are not manualized, such as TIES. Adaptations were made to make the measure more applicable to a wide range of providers (rather than just therapists) and to reflect that TIES is a set of strategies to encourage attendance rather than a specific treatment. The measure consists of eight questions with response options on a five-point Likert scale. Higher scores reflect more positive attitudes. Cronbach’s alpha for the baseline MPAS was .72 (which is similar to other studies using the MPAS; e.g., Borntrager et al., 2009).

Adoption of TIES strategies and fidelity

Participant adoption of TIES strategies: self-reported practice changes

As part of the LC Perceptions Survey, participants rated on a four-point scale one item related to the degree to which their participation in the LC resulted in changes to their approach to interacting with families. In addition, participants were asked to list three ways their approach to interacting with families had changed since the LC.

Team fidelity to TIES strategies: Plan-Do-Study-Act Form (PDSA)

Teams completed PDSA forms each month (for a total of up to eight months). The PDSA form has sections for the plan, what the LC team will do, the LC team’s study of the data, and the LC team’s plan to act in the next cycle.

Sustainability

Participant plans to sustain practice changes

As part of the LC Perceptions Survey, participants were asked to rate on a four-point scale one item related to how likely they were to continue the changes made in their approach to interacting with families. In addition, participants were asked to list two challenges to sustaining changes they made and ways to address those challenges.

Team plans to sustain practice changes

At the conclusion of the LC, each program was asked to submit a long-range plan to maintain changes and continue to implement new change ideas.

Analysis Plan

All quantitative analyses were performed using SPSS. Frequencies and descriptives were used to analyze sample descriptive data and multiple-choice responses. A paired samples t-test was used to test whether attitudes about EBPs improved at the end of the LC based on the 17 respondents who completed the MPAS at both pre- and post-LC and provided matching ID numbers.

All qualitative analyses were conducted by coding open-ended responses to survey questions, the long-range plan, and the PDSAs. Regarding the survey questions and long-range plan, responses were coded using an open-coding process whereby the first and third authors developed a set of codes based on the responses and then independently coded each response. Disagreements on codes were discussed and a final disposition was assigned. For the question about self-reported practice changes, six codes were created and the two coders independently agreed on 87% of responses. For the question about challenges to sustaining changes, seven codes were created and independent agreement between coders was 100%. For the question about ways to address identified challenges, four codes were created and agreement was 100%. For the long-range plan, seven codes were developed and independent agreement between coders was 96%. Regarding the PDSA’s, change ideas as written in the PDSA’s were coded by the first author and treatment developer Dr. McKay for consistency with TIES strategies (code options were “Recommended” or “Consistent, but Not Specifically Recommended”). Agreement between coders was 100%.

Results

Unless indicated, results reported were quantitative in nature. Any qualitative results are labeled as such. As indicated earlier, participants were not separated into control and intervention groups; all participants received the LC intervention. Participants were recruited into the LC between November and December 2010. All data collection took place during the LC implementation between January and September 2011.

A total of 26 participants provided demographic information. All 26 were female and ranged from 23 to 49 years old (mean was 33). A total of 42% were Caucasian, 35% Hispanic/Latino, and 23% African American, Asian American, mixed, or other. Half of the participants reported having their Master’s degree, while 12% had their Doctorate and 38% had their Bachelor’s degree or less. Participants had spent a mean of 3.6 years (SD = 3.6) in their current professional role and were involved in various aspects of their programs: 54% had administrative/evaluation duties, 38% scheduled families, 35% provided services to families, and 23% supervised staff.

Feasibility of the LC Implementation Method

Attendance

Attendance across the LC was high. The average number of sessions attended was 7.1 (SD=1.5; range 4 to 9). A total of 78% attended 7 out of 9 sessions, and 65% attended all three learning sessions where the majority of the didactics and cross-program sharing and problem-solving took place.

Participant Perceptions of the LC Process

Overall, participants reported positive perceptions of the LC. A total of 95% of participants reported being “very” or “somewhat” satisfied with the LC, and 96% reported that it was “very” or “somewhat” worth their time.

Acceptability of TIES Strategies and EBPs Generally

Participant Perceptions of Impact on Client Attendance

Overall, participants reported perceived improvement in family attendance, with 82% indicating that family attendance improved “very much” or “somewhat” as a result of the LC.

Participant Attitudes Towards EBPs

Significant improvement in participant-reported attitudes about EBPs was found. MPAS scores were higher at the end of the LC compared to the start (pre-LC M=23.7, SD=3.8; post-LC M=26.2, SD=3.6; t(16) = 2.59, p < .01).

Adoption of and Fidelity to TIES Strategies

Participant Adoption of TIES Strategies

Almost all participants reported that their approach to engaging families improved as a result of the LC, with 91% indicating their approach changed “very much” or “somewhat.”

Qualitative results of how participants’ approach to engaging families changed as a result of the LC were analyzed. Participants provided a total of 45 open-ended responses regarding changes. Both individual practice and organizational level changes were reported. The largest proportion of responses (29%; n=13) indicated an increased focus on collaboration and alliance with parents and caregivers (e.g., encouraging parent and caregiver participation in decision making and goal setting, tailoring services to the family, having patience and focusing on the individual), with a similar number of responses (27%; n=12) indicating improvements in participants’ clarity of communication with families (e.g., providing more detail about the program, being more open to questions) and being receptive to parent and caregiver feedback and trying new strategies (22%; n=10). The remaining responses focused on organizational level changes such as improving data collection and reporting (13%; n=6) and changes to policies and procedures (e.g., consistency across staff/sites, streamlining processes) to support parent and caregiver engagement (9%; n=4).

Team Fidelity to TIES Strategies

Teams developed a total of 11 change ideas (see Table 2). Ideas ranged from creating and distributing printed materials such as reminder cards, brochures, and flyers to adding engagement questions to telephone contact and intake procedures. Nine of the 11 ideas were implemented during the LC (Program A’s brochures had to obtain several approvals before being finalized).

Table 2.

Description of Change Ideas.

Program	Change Idea	Implemented during LC?	Incorporated Parent/Caregiver Feedback?	Recommended TIES Change Idea
Program A	Create first-contact script that includes program description and two engagement questions.	Yes	Yes	Recommended
	Create brochure for parents/caregivers that provides general information and service pathways.	No	No	Recommended
	Create brochure for providers that provides general information about the program and service pathways.	No	No	Consistent, but Not Recommended
Program B	Add five engagement questions to intake phone call.	Yes	Yes	Recommended
	Create and mail personalized appointment reminder card that includes how the program can support families.	Yes	Yes	Recommended
	Create script for intake phone calls that describes the program and the next steps in the registration process.	Yes	Yes	Recommended
Program C	Create and distribute informational flyer about classes when referral to class is made.	Yes	No	Recommended
	Provide one-on-one consults before or after class sessions; after consultation, follow-up offered via telephone.	Yes	Yes	Recommended
Program D	Add three engagement questions at initial home visit.	Yes	No	Recommended
	Create phone scheduling script for scheduling services that informs families what to expect and asks three engagement questions.	Yes	Yes	Recommended
	Create and hand out business/reminder cards at first visit with contact information and future appointments.	Yes	Yes	Recommended

Qualitative coding of fidelity to both TIES’ emphasis on parent and caregiver consumer feedback and TIES’ recommended change ideas was examined. Seven of the change ideas incorporated parent and caregiver consumer feedback through written surveys and interview questions. Ten of the ideas were recommended TIES change ideas; the eleventh was consistent with TIES but not a specific recommended change idea.

Sustainability

Participant Plans to Sustain Practice Changes

All 100% of participants reported being either “very” or “somewhat” likely to continue the changes they had made in their approach to interacting with families.

Qualitative results of participants’ reported challenges to sustaining changes made over the course of the LC were analyzed. Participants provided 26 responses describing challenges. The largest proportion of responses (27%; n=9) focused on staff buy-in and motivation, followed by challenges related to time (19%; n=5) and consistency across employees/sites (19%; n=5). Additional responses focused on challenges related to data collection and reporting (12%; n=3), program changes (12%; n=3), and funding (4%; n=1).

Qualitative results of participants’ reported ways to address identified challenges were also analyzed. Participants provided 12 responses for ways to address the identified challenges. The largest proportion of responses (50%; n=6) were related to enhanced communication through check-in meetings and communicating with program leaders and other programs, with additional responses focused on creating data tracking systems and processes (25%; n=3), allowing team members to share ideas and adapt change strategies (17%; n=2), and keeping parents and caregivers informed (8%; n=1).

Team Plans to Sustain Practice Changes

A total of 24 planned actions were reported across the LC teams’ long-range planning reports. Analysis of the qualitative coding of planned actions indicated that responses fell into two general categories. The first category reflected plans to sustain/implement change ideas and included: 1) developing new change ideas and implementing ideas started during the LC (21%; n=5), 2) continuing to implement current change ideas already in place (17%; n=4), and 3) discontinuing change ideas that were not working (8%; n=2). The second category reflected methods to sustain/implement change ideas and included: 1) check-ins among staff (25%; n=6); 2) plans to incorporate change ideas and TIES strategies into policies, procedures, and employee goals (17%; n=4); 3) budgeting for materials and ongoing training (8%; n=2); and 4) sharing what had been learned with other agencies (4%; n=1).

Discussion

The results of this study indicate that: 1) using an LC implementation method with early intervention providers was feasible; 2) participants rated TIES strategies as acceptable based on perceived improvements in client attendance and increased attitudes towards EBPs; 3) the method supported successful self-reported adoption of and fidelity to TIES based on participants’ reported implementation of many change ideas consistent with TIES; and 4) the method supported sustainability as evidenced by the wide range of strategies identified to sustain practice changes. It is encouraging that, overall, early intervention providers engaged in and had positive responses to the LC. This finding is consistent with Ebert and colleagues’ (2012) study of TF-CBT implementation and suggests that the LC method is a feasible implementation method for child-focused providers.

Regarding changes in attitudes towards adopting EBPs, the observed means indicate moderately high positive attitudes about EBPs overall given the range of possible scores. It is also notable that the observed pre-LC average score on the MPAS is consistent with a previous study that reported provider scores on the MPAS prior to a clinical training (M=21.1; Borntrager et al., 2009). We had insufficient power to test any potential covariates of this association (whether the participant had attended the TIES training, racial/ethnic minority status, education level, age). While this is the first known study to look at changes in attitudes towards adopting EBPs following the use of a collaborative implementation method, the observed improvement in attitudes about EBPs is consistent with a study that found improvements in attitudes for those who received training in a modular approach to implementing EBPs (Borntrager et al., 2009). It is notable that both the TIES LC and the modular approach incorporate participant choice in how to use evidence-informed strategies rather than promoting strict adherence to a manualized EBP protocol. Further, this study adds to the small but important literature examining attitudes among early intervention providers given attitudes towards adopting EBPs can differ across types of child-focused providers (Stahmer & Aarons, 2009). It is possible that the significant attitude shift found in this study may be due in part to the content focus of TIES on engaging parents in services, which is consistent with the parent involvement requirement in many early intervention services. Thus, these providers may be primed to respond positively to an EBP that is consistent with the services they provide.

This study provided more detail about the change ideas generated by LC teams than previous studies of the LC method to implementing evidence-informed engagement strategies (Cavaleri et al., 2006; Cavaleri et al., 2010). The large majority of ideas were clearly recommended based on TIES and cut across different points of service (intake phone call, first in-person contact, ongoing clinical contact) and varied delivery methods (written, phone, in-person). The variability in these ideas indicates that the LC methods support the flexible application of TIES across different programs and contexts, which may be driving the improvement in attitudes about EBPs. Furthermore, all of the ways that participants described changes in their approach to engaging families as a result of the LC were consistent with TIES strategies.

This study was the first among the small number of studies examining the Breakthrough Series Collaborative implementation method in child services, and specifically early intervention, to collect data on ways to address sustainability challenges. Overall, the sustainability plans reported in this study emphasized communication and enhancing motivation among staff. Some strategies to sustain practice changes included encouraging consistency across staff and incorporating changes into program policies and procedures, which are consistent with some of the institutionalization of policies to support TF-CBT implementation cited in Ebert et al. (2012). Some identified challenges to sustainability were consistent with findings from the one-year follow-up of an earlier LC to implement McKay and colleagues’ evidence-informed attendance engagement strategies (time, technology to support ongoing data collection, staff buy-in) (Cavaleri et al., 2007). Interestingly, a recent review of determinants of successful collaboratives within medical care that examined predictors of sustainability did not find links between these types of challenges and maintenance of changes (Hulscher, Schouten, Grol, & Buchan, 2013).

The sustainability plans reflect changes to both the provider and organizational levels (Strating, Nieboer, Zuiderent-Jerak, & Bal, 2011). Leadership support of quality improvement efforts and implementation of evidence-based practices has been identified as a key variable in the success of these efforts (Aarons, Hurlburt, & Horwitz, 2011). Dedicated leadership was cited as a facilitator to sustainability in Cavaleri and colleagues’ LC one-year follow-up study (2007). It is important to note that leadership was highly engaged in the LC process in the current study. Also, in addition to requesting the TIES training and LC, senior leadership of the participating LC programs exhibited a very strong commitment to quality improvement as evidenced by an annual required quality process improvement course, which programs have been participating in for the last ten years. Clearly the organizational context within which this LC took place highly valued and facilitated the testing of innovations, which is associated with the intensity and success of quality improvement efforts (Deo et al., 2009; Strating et al., 2011).

The TIES LC represents one implementation approach to increasing and enhancing engagement in early intervention services. Future research may examine a more comprehensive implementation intervention, in which the LC is one component. Other components may include use of additional training methods (Herschell et al., 2010) or organizational-level interventions (Aarons, Ehrhart, Fahranak, & Hurlburt, 2013; Glisson, Hemmelgarn, Green, & Williams, 2013). Combining provider and organizational implementation strategies has potential to facilitate adoption and sustained EBP delivery (Glisson et al., 2010).

This study has several strengths, including collecting both participant-level and team-level data as well as both quantitative and qualitative data. This study is also the first to examine changes in attitudes about EBPs following completion of an LC and to collect detailed information about plans for sustainability.

Limitations

Without a control or comparison condition, the study was unable to assess whether another implementation method would result in more positive changes than the LC method. Further, without follow-up it cannot be determined whether planned sustainability efforts were implemented. In addition, the programs that participated in this initiative were not selected at random but requested to receive the TIES training and participate in the LC, which limits generalizability. Generalizability is also limited by the small sample size and the fact that all participating programs operated under the same division within one hospital.

Given the data were not collected for research purposes, the evaluation team was unable to control the quality of data collection, which contributed to missing data. The small sample size of repeated measures, in particular on the MPAS, due to missing data precluded the ability to examine potential covariates of that significant association. The evaluation team was also unable to collect and analyze client-level attendance data or observer-rated fidelity or to look at team-level effects.

Conclusions

Unfortunately, the existence of evidence-based and evidence-informed practices is necessary but not sufficient to ensure that such practices are implemented by frontline providers. Moving EBPs and evidence-informed strategies into existing service settings such as early intervention programs requires detailed and well-specified efforts to build a supportive infrastructure to adopt and maintain practice innovations. Breakthrough Series Collaborative-based methods such as learning collaboratives are starting to be utilized in the mental health services field to address this need for systematic and intensive efforts to enhance implementation (e.g., Cavaleri et al., 2006; Cavaleri et al., 2010; Ebert et al., 2012; Roosa et al., 2011). The current study extends this small body of research to examine the feasibility and acceptability of a learning collaborative with early intervention providers who serve children with behavioral and developmental problems to implement evidence-informed attendance engagement strategies, which are critically needed in community-based care. Results support the LC as a useful method for implementing attendance engagement strategies (and perhaps other EBPs) in the community early intervention context, suggest that the method allows for flexibility of implementation, and indicate that the method can result in rich sustainability plans to enhance attendance engagement. Future research is warranted to continue learning about how this implementation method can improve the quality of child and family focused services.