IMPACT OF TREATMENT INTEGRITY ON INTERVENTION EFFECTIVENESS

Abstract

Treatment integrity has cogent implications for intervention effectiveness. Understanding these implications is an important, but often neglected, undertaking in behavior analysis. This paper reviews current research on treatment integrity in applied behavior analysis. Specifically, we review research evaluating the relation between integrity failures and the efficacy of behavioral interventions. Avenues for future research are provided.

A hallmark of applied behavior analysis (ABA) is the demonstration of functional relations between target behaviors (dependent variables) and environmental manipulations (independent variables). The extent to which such relations can be understood depends, in part, on the precise measurement of target behaviors and the accurate implementation of independent variables. The precise measurement of independent variable integrity has been identified as an overlooked area within ABA (McIntyre, Gresham, DiGennaro, & Reed, 2007; Peterson, Homer, & Wonderlich, 1982; Wheeler, Baggett, Fox, & Blevins, 2006). However, integrity has begun to receive specific attention from researchers in recent years.

Although research has examined interventions aimed at improving treatment integrity in natural settings (e.g., DiGennaro, Martens, & Kleinmann, 2007), a number of other issues have emerged within the area of integrity. Most notable is the nature of the relation between treatment integrity and intervention effectiveness. This issue seems to be especially pertinent to applied researchers and clinicians in behavior analysis, because integrity may be compromised when behavioral interventions are employed in natural settings and treatment is carried out by individuals with little or no specific training in behavior analysis. The purpose of this review is to provide an overview of the recent literature that has included evaluations of the impact of integrity on intervention effectiveness. We will begin with correlational research and finish with research that has involved both the manipulation and comparison of multiple levels of integrity and their impact on intervention effectiveness. Finally, we will provide some implications and recommendations for further research and development in this area.

Descriptive Analysis of Treatment Integrity and Intervention Outcomes

Recent research has reported on the correlation between treatment integrity and intervention effectiveness with caregivers in natural settings. Arkoosh et al. (2007) evaluated the long-term relation between treatment integrity and intervention effectiveness with five child–parent dyads in their home settings. Parents were taught to implement function-based treatment packages for problem behavior. The packages included differential reinforcement of mands, reinforcement of other social behavior, and the withdrawal or withholding of reinforcement for problem behavior (i.e., guided compliance or response cost). The experimenters identified a relation between higher levels of treatment integrity and improved intervention effectiveness, and the two children who demonstrated the least amount of progress had parents with relatively poor integrity. Interestingly, all parents implemented extinction with much lower integrity than reinforcement. Thus, the extent to which integrity failures affect intervention efficacy might be somewhat procedure specific, perhaps especially when interventions involve multiple components. However, because this was a descriptive study, inferences regarding this possibility are difficult to make and require systematic evaluation.

DiGennaro et al. (2007) demonstrated a relation between improved integrity and intervention effectiveness when they compared procedures aimed at improving special educators' integrity in classrooms. Problem behavior decreased for all students when an effective intervention for integrity was in place, supporting the general notion that improved integrity leads to intervention effectiveness. Moreover, the relation between increased integrity and decreased problem behavior was statistically significant for three of the four teacher–student dyads.

Although integrity levels were correlated positively with treatment effectiveness in both of the above-mentioned studies, neither manipulated integrity levels in a controlled manner to measure their effects on treatment outcomes. A complete understanding of the effects of integrity failures on treatment effectiveness can be understood only by experimental evaluation. The following studies involved such experiments.

Experimental Analysis of Treatment Integrity and Intervention Outcomes

Wilder, Atwell, and Wine (2006) evaluated different levels of treatment integrity (100%, 50%, and 0%) of a three-step prompting procedure on the compliance of two typically developing children. A multielement design was used to evaluate the effects of integrity level on child compliance. Compliance systematically varied according to the level of treatment integrity. That is, 100% integrity was associated with the most compliance, 50% with moderate compliance, and 0% with very little compliance.

More recently, Stephenson and Hanley (2010) conducted two experiments on the assessment and treatment of preschoolers' compliance in a semianalogue classroom setting. In the second experiment, during which the experimenters manipulated integrity levels, participants received an additive antecedent intervention (AAI), a noncompliance intervention (three-step prompting), and a combination of the two (full-antecedent plus noncompliance intervention; FANI). After the experimenters found that two of the four participants required the FANI intervention to show improvements in compliance, they manipulated the integrity level of the intervention. Specifically, the researchers included a FANI challenge condition in which one compliance trial was selected randomly to have the AAI component removed and one to have the noncompliance intervention removed, progressively, until both children complied with four or fewer of the 10 compliance trials. Compliance remained high for both children until the experimenters implemented procedures correctly during just two of the 10 trials (20% integrity). As the authors noted, this finding suggests that teachers and caregivers can implement an intervention with low integrity and still maintain treatment outcomes if they initially implement the intervention with high levels of integrity. This is an important possibility that requires further experimental evaluation.

Most research on integrity has focused on errors of omission, which involve failing to implement treatment (e.g., not reinforcing a target during acquisition training). However, the impact of errors of commission, for which treatment is implemented at the wrong time (e.g., reinforcing an incorrect response), is equally important. St. Peter Pipkin, Vollmer, and Sloman (2010) conducted a series of experiments on treatment integrity failures that included errors of both omission and commission during differential reinforcement of alternative behavior (DRA) procedures. In the first study, 22 undergraduate students interacted with a computer program designed to model DRA conditions. Red and black circles were presented on the computer screen. Clicking on the red circle was an analogue for appropriate behavior, and clicking on the black circle was an analogue for problem behavior. Points were given as reinforcers and were programmed to be delivered on schedules that varied for each circle and participant. Participants were assigned randomly to one of four groups. Participants in the first group experienced varying levels (20%, 40%, 60%, and 80%) of omission errors, participants in the second group experienced varying levels of commission errors, and participants in the third group experienced covarying levels of both omission and commission errors (e.g., 40% chance that appropriate behavior would receive reinforcement, 60% chance that problem behavior would receive reinforcement). The fourth group experienced one level of combined errors (50% integrity). The sequence in which participants were exposed to the 50% level of integrity was changed systematically to evaluate the effects of recent history on treatment integrity manipulations. Specifically, 50% integrity was implemented after both 100% integrity and 0% integrity.

Results for Group 1 (omission errors; i.e., failure to reinforce appropriate behavior) were interesting in that varying the level of integrity did not affect levels of problem behavior. Furthermore, this type of integrity failure was associated with lower rates of the alternative behavior, which, as the authors noted, could be preferred in some situations. For the second group (commission errors; i.e., reinforcing problem behavior), integrity failures did not become problematic (i.e., decrease appropriate behavior and increase problem behavior) until integrity fell to 40% and 20%. For the third group (combined errors), responding seemed to match the specific schedule of reinforcement. That is, when problem behavior had a greater than 50% chance of producing reinforcement, problem behavior occurred more frequently than appropriate behavior, and vice versa. Finally, the fourth group, which was exposed to 50% integrity after baseline and after 100% integrity, engaged in two patterns of responding. One pattern indicated some carryover from the most recent condition such that 50% was most effective after 100% and less so after 0% integrity; the other indicated that the conditions exerted stimulus control over responding, such that carryover was not observed.

The second and third experiments involved translational efforts, in which the basic human operant research in Experiment 1 was replicated in applied circumstances. Specifically, the second and third experiments attempted to replicate the results obtained for Groups 3 and 4 of Experiment 1. Results of Experiment 2 largely replicated those for Group 3, for whom responding was allocated to the response associated with the highest levels of reinforcement during integrity failures. Results of Experiment 3 largely replicated those for Group 4, for whom 50% integrity was less effective after 0% (baseline) conditions than after 100% (DRA) integrity conditions. Interestingly, participants showed some bias towards problem behavior in the 50% conditions that followed 100% integrity, raising questions about extraexperimental history.

Finally, DiGennaro Reed, Reed, Baez, and Maguire (2011) examined various levels of treatment integrity that involved errors of commission during discrete-trial training. Therapists reinforced varying levels of incorrect responses while teaching three children with autism to identify nonsense shapes receptively. Each shape was associated with either 0%, 50%, or 100% errors of commission. Results demonstrated that the highest level of treatment integrity (0% errors of commission) produced the most acquisition. Interestingly, however, the 50% and 100% errors conditions produced similar outcomes for two of the three children. Thus, an important conclusion from this study is that some commission errors (in this case 50%) may be just as detrimental as 100% commission errors.

Implications for Future Research

A number of possibilities exist for further research on the relation between treatment integrity and intervention effectiveness. Additional research is needed on different levels of integrity failures. First, a descriptive study aimed at the identification of common integrity failure levels in the real world would be important for determining those integrity levels in need of evaluation. In addition, the field is just beginning to understand how various types of integrity failures (e.g., errors of commission or omission) might have distinct influences on intervention outcomes (e.g., DiGennaro Reed et al., 2011; St. Peter Pipkin et al., 2010). Further research also should include a variety of behavioral interventions in various contexts, because some may be more or less sensitive to integrity failures. For example, research is needed on common interventions such as noncontingent reinforcement, the high-probability instructional sequence, differential reinforcement of other behavior, response-cost procedures, and treatment packages that combine multiple interventions. Although Arkoosh et al. (2007) did not control specifically for this, their results highlighted the possibility that some interventions might be implemented at less than full integrity without compromising treatment effects, especially when they are combined with other interventions that are implemented at relatively high levels of integrity.

In addition, more research is needed to understand the role of reinforcement history on the effects of integrity failures. The recent studies by St. Peter Pipkin et al. (2010) and Stephenson and Hanley (2010) raise a number of interesting possibilities in this regard. As both of the aforementioned studies discovered, it is possible that treatment integrity failures are less detrimental if they are preceded by a history of high integrity levels. Thus, it is possible that treatment integrity is most critical during the early stages of treatment and becomes less critical over time. Related to these issues, more studies are needed to explore the long-term impact of integrity failures. For example, although some studies suggest that integrity failures might not always be detrimental (e.g., St. Peter Pipkin et al.; Stephenson & Hanley), it is unclear how these failures affect responding over time. Indeed, it is well known that intermittent reinforcement has the effect of strengthening behavior; thus, researchers should consider the possibility that integrity failures might result in the long-term strengthening of problem behavior or, conversely, in the further strengthening of appropriate behavior. As these studies highlight, integrity failures occur in historical, multifactored fields (Kantor, 1958), and surely such contextual issues warrant extensive investigation.

Integrity is an important topic in need of exploration. Continued research in this area will have direct implications for the practice of behavior analysis. Furthermore, the area seems to be ripe with opportunities for researchers to pursue.