Intervention Integrity: New Paradigms and Applications

This special series seeks to summarize the status of integrity monitoring in school-based intervention research and practice. The four articles included in this issue highlight several innovative research projects to design, implement, and evaluate integrity across diverse settings, and outline future research directions and practical implications for school-based professionals. Although the concept of understanding the degree to which an intervention has been implemented as planned has long been recognized as critical within the prevention and intervention literature (Basch, Sliepcevich, Gold, Duncan & Kolbe, 1985), research and conceptual models related to treatment integrity have surprisingly lagged behind. For instance, a series of recent reviews of the literature suggests that treatment integrity is only recorded in a small minority of studies, ranging from 3.5% to 28% of studies reviewed (e.g., Gresham, MacMillan, Beebe-Frankenberger, & Bocian, 2000; McIntyre, Gresham, DiGennaro, & Reed, 2007; Perepletchikova, Treat, & Kazdin, 2007).

Intervention integrity is an important construct to understand and evaluate in the school-based mental health literature given its possible positive association with intervention success (e.g., Gresham, 1989; Peterson, Homer, & Wonderlich, 1982). Thus, effective programs implemented with a high degree of fidelity are expected to produce the most consistent and positive effects. Further, systematically monitoring the integrity with which treatments are implemented provides insight into interventions that do not produce positive effects. Specifically, intervention integrity data can determine if a program was ineffective because it was not a strong program (e.g., poorly conceptualized/designed) or because it was implemented poorly (e.g., the main components of the program were not implemented as intended). Monitoring how programs are carried out can also provide information on components that are critical to intervention success (Perepletchikova, & Kazdin, 2005) and the feasibility of implementing the intervention (Peterson & McConnell, 1993). Finally, given the recent reauthorization of the Individuals with Disabilities Education Improvement Act in 2004, increasingly school staff are being called upon to utilize a response to intervention (RTI) framework in identifying children with learning disabilities (Klotz, & Canter, 2006). Within the RTI movement, there is a clear mandate that the school provide continuous monitoring of evidence-based interventions to determine treatment adherence and success for individual students. For all of these reasons, it is critical to develop comprehensive, practical, and community-responsive treatment integrity procedures within the school-based mental health literature base.

Historical Context of Treatment Integrity

Over the last 25 years, clinical research trials have emphasized the importance of using treatment manuals, and implementing consistent practices, trainings, and supervision (e.g., Drozd, & Goldfried, 1996; Leff, Power, Manz, Costigan, & Nabors, 2001; Luborsky, & DeRubeis, 1984). Traditionally, treatment integrity or fidelity has referred to therapist adherence to the intervention manual (Gresham, Gansle, Noell, Cohen, et al., 1993). In other words, has a therapist or program implementer covered the main or crucial steps stipulated for a particular intervention session? This is often referred to as procedural or content integrity. While procedural integrity is an important part of integrity monitoring, this current special series draws upon recent research suggesting that there are a number of other important ways in which to conceptualize integrity that expand upon this definition and provide future directions to the field of prevention and intervention science. For one, intervention integrity has expanded to include measurement of therapist ability to effectively carry out the treatment, i.e., therapist competence. Thus, the extent to which the therapist effectively manages child behavior, responds to participant questions, and encourages active involvement is considered critical to intervention success. Intervention integrity has also been expanded to address the issue of treatment differentiation, such that evaluating the extent to which different program components are implemented allows researchers to compare the relative impact of different treatment components on participant outcome. Finally, recent literature on treatment fidelity includes participant response to the intervention, suggesting that participants who enjoy and are engaged with the intervention are more likely to experience a positive impact (Dane, & Schneider, 1998).

Controversies and Conceptual Models

As a field, there are a number of areas in which research has helped to further our understanding and appreciation of integrity monitoring over the past few decades. These areas include (a) the recognition that there must be “flexibility” within the delivery of manualized treatments (Kendell, & Beidas, 2007); (b) that models for treatment integrity must include not just procedural or content integrity, but also indices for determining the quality of treatment delivery and measures of participant engagement (Perepletchikova, & Kazdin, 2005); and (c) that we consider partnership-based approaches to help ensure the cultural sensitivity and participant engagement aspects of the interventions (Power et al., 2005).

Flexibility in Implementation Procedures

Over the last 10 years, the core ingredients for determining whether a program is efficacious (proven) or possibly efficacious (promising) have been clearly articulated (Chambless, & Hollen, 1998). Programs are designated as efficacious if they employ an experimental group design (e.g., random assignment procedures), have well-documented and clearly delineated treatment procedures including treatment manuals, contain uniform therapist training procedures, employ multi-method outcome assessments, have demonstrated longitudinal effects, and have an independent replication by another team of researchers. If all but the independent replication has occurred, then the program is possibly efficacious. In recent years, there has also been growing recognition that treatment adherence or procedural integrity is also critical to identifying successful programs. Specifically, providers can enhance program development by delineating and measuring the core or critical components of each intervention session. Parallel with this is the recognition that program facilitators can effectively develop their own “voice” or style, and that these features can be understood and monitored in terms of quality indicators (e.g., Kendall, & Beidas, 2007).

Multi-Component, Multi-Method Indices of Intervention Integrity

As the definition of intervention integrity has expanded, the need for utilizing multiple methods and multiple reporters has been highlighted. For example, integrity monitoring systems should include both participant and therapist reports on key program processes. Including participant report will not only serve as a check on the reliability of therapist perceptions, but will also provide unique insight into critical intrapsychic factors, such as participant engagement. In addition, for interventions with more than one participant, integrity monitoring systems should include both individual and group-level indices. Doing so will allow for a more nuanced assessment of the mediating effect of program integrity (e.g., evaluation of differential intervention impact for participants with high vs low engagement). Finally, diversity in evaluation methodology will also provide a more accurate understanding of key intervention components. As such, intervention integrity should be monitored using multiple methods, including self-report, observer-report, direct observations, and permanent products.

In their article entitled, “An analysis of teacher investment in the context of a family-school intervention for children with ADHD,” Power, Soffer, Mautone, Jones, and Clarke, (2009) described the reliability and validity of the Teacher Investment Questionnaire (TIQ), an instrument designed to assess participant (i.e., teacher) investment in intervention delivery. This measure contributes to the knowledge base related to intervention integrity in that it was designed to assess the important yet relatively under-developed construct of “participant engagement” in intervention delivery.

Partnership-Based Approaches to Integrity Monitoring

Traditionally, the development of assessment tools places the emphasis upon the research team to identify critical constructs, develop and fine-tune items or subscales, and to establish the psychometric properties of the tools generated. These approaches are generally top-down and are directed by researchers without the involvement of research participants (see Nastasi et al., 2000; Power et al., 2005). While these certainly are strengths to these traditional approaches, there also are some limitations which include that the resultant measures may not be reflective of the social context and/or may not be meaningful for the local community (Leff et al., 2006). Recently, Power et al. (2005) extended the notion of using partnership-based approaches to the development and validation of intervention implementation procedures. This allows for a much better understanding of what aspects of treatment are applied and effective, how participant engagement impacts treatment delivery, and a recognition that deviations from treatment protocols provide an ample opportunity to better understand treatment feasibility.

In their article entitled, “Using participatory action research to design an intervention integrity system in the urban schools,” Gullan, Feinberg, Freedman, Jawad, and Leff, (2009) described partnership-based procedures used to develop and refine an integrity monitoring system designed to evaluate multiple dimensions of implementation integrity, namely, the extent and quality of intervention delivery and participant responsiveness. This research is one of the few studies to examine the effect of implementation fidelity on participant outcomes, and it may be the only study to have so extensively explored the mediating effect of integrity measures on intervention effects.

Supports for Monitoring Integrity

Monitoring intervention integrity is a best practice and in certain instances a mandated requirement (e.g., Iowa Department of Education, 2007). Implementing interventions with (e.g., Noell et al., 2005) and collecting implementation integrity data consistently (e.g., Burns, Peters, & Noell, 2008; Cochrane, & Laux, 2008), however, can be challenging. For example, efforts to increase the acceptability and feasibility of interventions and integrity measurement in time- and resource-limited school settings are often at odds with proper monitoring of intervention implementation (Evans, Schultz, & Serpell, 2008). Two articles in this special issue describe supports for families to facilitate the implementation of interventions (Swanger-Gagne, Garbacz, & Sheridan, 2009) and supports for school staff to monitor implementation integrity (Brown, & Rahn-Blakeslee, 2009).

In their study entitled, “Intervention implementation integrity within conjoint behavioral consultation: Strategies for working with families,” Swanger-Gagne et al. (2009) described procedures that consultants used to support parents’ implementation of home-based interventions in the context of a conjoint behavioral consultation model. Those authors presented data from a large study that described parents’ implementation integrity assessed as per two methodologies: self-report and permanent products. They also presented a case study that described in rich detail the types of supports one family in the “at-risk” group received. This study extends the literature by initiating a line of research assessing the integrity of interventions being implemented within the home setting with a particular focus on families whose life circumstances may make it particularly difficult to implement interventions consistently.

In their article entitled, “Training school-based practitioners to collect intervention integrity data,” Brown and Rahn-Blakeslee (2009) described the Heartland Area Education Agency’s model of assessing intervention integrity, which includes providing professional development and supports to help school staff monitor intervention integrity and use these data when making educational decisions in an RTI framework. The article provides specific, practical information about the logistics and content of staff training and monitoring tools that can be broadly applied in school settings.

Summary

While the field of intervention and prevention science has in recent years seen an increase in the recognition and importance of providing empirically supported and well-defined prevention and intervention programs, researchers are now advocating for the need for multi-faceted intervention implementation monitoring systems. This special series highlights new methods and innovative approaches for accomplishing this task, while identifying challenges in this new area of research. We hope that the special series will expand the current definitions of treatment integrity, identify gaps in the literature-base, and outline practice implications and future research directions for the monitoring of interventions across diverse settings.