On the Topic of Treatment Integrity

Abstract

Treatment integrity, also known as treatment fidelity, is integral for empirical testing of intervention efficacy as it allows for unambiguous interpretations of the obtained results. Assuring treatment integrity is also important for dissemination of evidence-based practices and quality improvement of services. However, in the examination of the relationship between treatment integrity and treatment outcome it is important to consider that treatment integrity may be a proxy variable for other variables impacting therapeutic change (e.g., characteristics of intervention, clients, setting, and therapist). Considerations on examining the association between integrity and outcome are discussed. Further, recommendations on the level to which treatment integrity needs to be addressed in psychotherapy research and clinical practice are provided.

Treatment integrity, also known as treatment fidelity, refers to the extent to which treatment was implemented as intended (Vermilyea, Barlow, & O’Brien, 1984; Yeaton & Sechrest, 1981). Treatment integrity is essential for empirical testing of intervention efficacy, as it allows for unambiguous interpretations of the obtained results. Indeed, obtained results may be related to the implemented intervention (i.e., treatment as delivered) and not to the intended intervention (i.e., treatment as designed). Therapists may implement all, some or none of the procedures specified in a treatment protocol, or may supplement prescribed tasks with procedures not in a manual. Having information on treatment integrity allows for judgments regarding how closely the implemented intervention approximates the intended intervention.

Bond, Becker, and Drake (this issues) highlighted that assuring treatment integrity is also important for dissemination of evidence-based practices and quality improvement of services. As stated in this article, once an intervention is validated, its utilization with high integrity may increase chances for outcomes similar to those in the original efficacy research. However, the relationship between treatment integrity and treatment outcome is not straightforward and warrants further discussion.

Treatment integrity and treatment outcome

Multiple studies have been conducted to examine whether levels of treatment integrity relate to treatment outcome, but with mixed results (see Perepletchikova & Kazdin, 2005 for details). There may be multiple reasons for conflicting findings, as the relationship may be a function of intervention efficacy, as well as characteristics of clients, setting, treatment and therapists. High levels of integrity per se do not increase treatment strength. Significant effects can be obtained with low integrity levels (e.g., Watson & McCurdy, 1999). Low integrity does not mean that an intervention is weak, but that it is different from the originally intended. If an intervention is not effective, high integrity cannot be expected to lead to better outcomes, while low integrity can actually improve results, as added procedures may better address patients' needs. Further, high integrity levels do not guarantee that an intervention found efficacious in one setting will demonstrate efficacy under different conditions. For example, intervention validated in an experimental setting, with highly trained therapists, and carefully selected homogenous population may be less effective in a community mental health center with more heterogeneous clientele, and high levels of treatment integrity may actually attenuate results.

There were several attempts to evaluate empirically the relationship between integrity and outcome (e.g., Gansle & McMahon, 1997; Holcombe, Wolery, & Snyder, 1994; Noell, Gresham, & Gansle, 2002). However, obtained results were inconclusive. Even an investigation that uses empirically supported intervention and random assignment of subjects to high vs. low integrity conditions may yield ambiguous results. The problem lies in the very nature of treatment integrity - when integrity is low, interpretations of the obtained results are obscured. Researchers are left with no answers as to what was the intervention in the low integrity condition.

Treatment integrity is not conceptually related to treatment outcome, and may be just a proxy variable for other influences that account for therapeutic change. Associated variables (e.g., characteristics of treatment, setting, patients, and therapists) may function as moderating or mediating variables in the relationship, or may just be the confounding influences that may serve as alternative explanations of the obtained outcome. It may be difficult to formulate a question on the relationship between integrity and outcome to obtain information that incrementally adds to data collected on other variables. Indeed, a question of whether high treatment integrity relates to improved outcomes basically translates into a question of whether a particular intervention implemented as designed is effective in addressing a particular problem in a particular population. When an intervention is found efficacious and treatment integrity levels are high, an additional examination of the association between integrity and outcomes may be redundant. In a situation when changes on the dependent measures are in an expected direction but treatment integrity levels are low, data on the association between integrity and outcomes may not offer much insight. Instead, it may be advisable to evaluate all available information (e.g., sessions tapes, therapist's self-reports, clinical notes, written homework assignments, data collection sheets) to establish what was actually done by therapists. Information on the employed procedures may provide clues for developing a more promising treatment. Similarly, when a treatment validated in one setting fails to demonstrate efficacy in a different setting even when implemented with high integrity, examination of the relationship between integrity levels and outcome may not be informative. In such cases, variability that contributed to discrepant findings most likely rests in characteristics of a setting, patients and therapists, rather than intervention implementation.

It may be difficult to draw inferences on the association between treatment integrity and treatment outcome due to multiple influences that normally occur in the context of clinical trials and clinical services. When approaching on this line of investigation, researchers may need to consider how the obtained data may be informative, as well as take into account other variables at play and affecting this association.

Considerations on Implementing Treatment Integrity Procedures

Regardless of its relationship with outcome, treatment integrity is imperative for the interpretation of the obtained results. This objective demands attention to the way treatment integrity is established, assessed, evaluated, and reported. Recommendations on treatment integrity procedures (e.g., Perepletchikova, 2009; Perepletchikova, Hilt, Chereji, & Kazdin, 2009; Perepletchikova & Kazdin, 2005) are summarized in Table 1.

Table 1.

Guidelines on Implementation of Treatment Integrity Procedures

Treatment Integrity Procedure	Description
	Therapist Treatment Adherence	Therapist Competence	Treatment Differentiation
Definitions of treatment integrity aspects	Degree of utilization of the specified procedures and avoidance of proscribed procedures	Level of therapist’s skill and judgment in intervention delivery	Ensuring that treatments under investigation differ from each other along critical dimensions
Operational definition of intervention	Specific treatment manual provides information about a theoretical basis of the intervention, specifies rationales for adherence, spells statements to be made by treatment agents, describes characteristics of the treatment process and sequencing the techniques, provides examples of treatment operations, and procedures for handling deviations	Description of procedures encompasses guidelines on competent implementation and provides venues for flexible administration of tasks	Treatment manual describes procedures, tasks and activities to be avoided
Training of therapists	Indirect training procedures: didactic instructions about the intervention and written materials describing the rationale, scripts, tasks, and activities Direct training procedures: opportunities to role-play utilization of techniques, model competent administration of tasks, rehearse and trouble-shoot difficult situations, treatment simulation, and provide periodic booster sessions
Supervision of therapists	Weekly supervision session may include viewing therapy tapes to provide feedback on adherence and competence, role-playing approaching difficult situation, discussion of cases and additional procedures to be incorporated in treatment manual. Ongoing clinical supervision may be provided through direct observation of live treatment sessions via a TV monitor or a two-way mirror. Further, supervisor may examine adherence and competence checklists that are filled out by therapists after each session to provide performance feedback and adjust implementation of tasks.
Assessment of treatment integrity	Direct assessment: observations, video/audio taping Indirect assessment: therapists fill out therapy checklists after each session to indicate procedures that were or were not implemented as prescribed in a manual (checklists should allow for explanations on any protocol deviations)	Direct assessment: observations, video/audio taping (with considerations on the number of sessions completed, extent of client progress, client difficulty, and sensitivity of approach in a manner consistent with the prescribed procedures Indirect assessment: therapists fill out checklists on their competency of intervention delivery (checklists should allow for explanations for low levels of competence)	Assessment of implementation of prescribed procedures and avoidance of proscribed tasks is sufficient to demonstrate treatment differentiation
Psychometric properties of measures	Provide data on the validity and reliability of the utilized adherence and competence measures (can be published separately from primary report)
Accuracy of data	A random sample (usually 20% to 40%) of collected tapes is examined to assess integrity. To ensure adequate data representation, sample across therapists, treatment phases, situations, cases and sessions.
Rating treatment integrity	Raters are trained on major and minor treatment components, subtle aspects of the treatment and treatment manual Raters are trained on treatment adherence and competence measures Interater reliability is assessed Rater bias is reduced (blinding, multiple raters per target, consensus ratings, statistical correction of distortions)
Control for measure reactivity	All sessions are video/audiotape ‘Spot checks’ of treatment implementation are performed on a variable-time schedule
Report of data	Specific definitions of treatment adherence are provided Procedures for establishing, assessing and evaluating adherence are reported Treatment adherence is reported in terms of overall integrity (i.e., integrity of treatment components across sessions); component integrity (i.e., integrity of implementing each treatment component across sessions) session integrity (i.e., integrity of all treatment components within each session) Numerical data are informative of treatment adherence levels (e.g., percent adherence)	Specific definitions of therapist competence is provided Procedures for establishing, assessing and evaluating competence are reported Overall therapist competence is reported Numerical data are informative of competence levels (e.g., percent competence)	If proscribed procedures were monitored, than data on treatment adherence is sufficient to demonstrate treatment differentiation

The provided recommendations are exhaustive. It may be unrealistic to expect optimal utilization of these strategies due to cost constraints, considerable labor demands, and lack of editorial requirement. Further, in a recent examination of the degree to which integrity is addressed in treatment outcome research, only 3.5% of the evaluated trials adequately implemented treatment integrity procedures (Perepletchikova et al., 2007). It may take some time before adequate attention to treatment integrity in psychotherapy research becomes standard practice. Improvements in scientific methods are gradual, thus, it may be more realistic to consider a continuum of procedures in addressing treatment integrity (see Figure 1).

Figure 1.

A Continuum of Procedures in Addressing Treatment Integrity

Although presented levels are arbitrary, they reflect current variability in attention to treatment integrity (Perepletchikova et al., 2007). Optimal adequacy in addressing independent variables may be attained by employing all of the specified strategies. Yet, any attention to treatment integrity is quite scarce, so even minimal improvements are welcome and necessary to remedy the situation. At the minimum, integrity needs to be established by operationally defining intervention, training therapists in treatment delivery via direct methods, and monitoring implementation by ongoing supervision. Other procedures can be built on this foundation and the ease with which these procedures can be implemented largely depends on the specificity of a treatment manual. Clear specifications of tasks and procedures simplify training of therapists, monitoring of treatment delivery, assessment strategies and training of raters, and may affect the accuracy of data representation.

The question of what represents a satisfactory assurance that treatment was implemented as designed may not have a straightforward answer. Multiple considerations may affect how treatment integrity is addressed, including available funding, study design, setting, and the level of risk of treatment inaccuracies. The degree to which each treatment integrity procedure is implemented may also be dictated by the nature of an intervention. Some interventions, particularly process-oriented treatments (e.g., humanistic, existential) do not permit strict manualization of tasks and activities. Thus, some integrity procedures may not be applicable. Ambiguity in definition of variables, however, produces ambiguity in interpretation of findings. Current research methods of testing intervention efficacy demand standardization and precision of randomized clinical trials (RCT). Literature suggests that Level 4 procedures on the presented continuum (see Figure 1) may be the recommended level of rigor for RCTs (e.g., Carroll & Nuro, 2002; Gresham et al., 2000; Schlosser, 2002; Waltz et al., 1993).

Treatment development may necessitate Level 3 attention to integrity, as the main objective of Stage 1 research is to develop elements required for testing intervention efficacy (Rounsaville, Carroll, & Onken, 2001). Attention to treatment integrity may facilitate refinement of a treatment manual via recognition and retention of elements that are unique to an intervention, identification of elements to be excluded from a treatment, and identification of procedures that therapists consistently fail to implement as well as techniques therapists add to the protocol to better suit the population. Further, therapist training may be aided via selection of sessions exemplifying successful and unsuccessful attempts of implementing prescribed procedure.

Small pilot studies may not have funds to implement all treatment integrity procedures, but presumably no clinical research study should be published if it does not meet at least Level 2 criteria. Interventions conducted in clinical settings may require less rigorous monitoring and evaluation of treatment integrity (Level 1 procedures) as opposed to empirical efficacy trials that demand precision. Yet, interventions that may be at high risk for treatment inaccuracies (e.g., interventions delivered by parents or teachers) may need more rigorous procedures for ensuring treatment integrity, such as relying on direct assessments, ascertaining representative sampling of integrity data, and examining overall, component and sessions adherence (for further discussion on levels of risk for treatment inaccuracies see Peterson et al., 1982). Further, as noted by Bond and colleagues (this issue), successful dissemination of evidence-based interventions and improvement of services necessitate assurance of the high quality of the employed procedures, pointing to a need for implementing more rigorous treatment integrity procedures in clinical settings.

Addressing treatment integrity is expensive and labor intensive. Ways to attain a satisfactory balance between cost and adequacy of treatment integrity procedures need to be examined. This is an empirical question to which an answer is urgently needed. Cost constrains may have impeded adequate attention to treatment integrity in the past, and this situation may continue if this concern is not addressed. Further, the issue of validation of treatment integrity measures requires further elaboration. Treatments differ in the operational definition of an intervention, treatment components and requirements for competent implementation. This may mean that each time a new treatment is tested, a new integrity measures has to be developed. Consequently, valid measures of treatment integrity may not be readily available. This represents a problem for cost containment. Future research may evaluate ways to create more general, already validated measures of integrity and define acceptable limits within which these measures can be altered to fit specificities of each treatment under investigation.