Development of an Audio-Tape Review System for Supporting Adherence to an Evidence-Based Treatment

Abstract

In bridging the science to service gap, effective, yet practical, strategies are needed for supporting practitioner implementation of evidence-based treatments. The development and preliminary evaluation of an adherence monitoring system to support clinician fidelity to an evidence-based treatment for substance-abusing adolescents was tested for community-based practitioners. Session tapes were monitored for adherence to a family-based approach to CM (CM/FB) for 27 practitioners during baseline, post-workshop, and follow-up periods. Approximately half of the practitioners were randomized to receive intensive quality assurance following a CM/FB workshop as part of a larger study. Findings supported the clinical feasibility of the developed system as well as the face and content validity, reliability, and concurrent validity. Future directions are discussed in light of these results, including instructions for the use of the developed system to efficiently train clinicians to adequate fidelity.

To address the widely acknowledged gap between science and service, major federal initiatives have promoted the adoption of evidence-based treatments (EBTs¹) by community providers, especially in the area of substance use treatment (e.g., National Institute on Drug Abuse [NIDA], 1999; Substance Abuse and Mental Health Service Administration [SAMHSA], 2007). In a recent review, Miller and colleagues (2006) concluded that practitioners are receptive to and interested in research-based practices, but effective methods for training clinicians in EBTs are rarely employed. Further, typical training mechanisms do not effect substantive change in clinician behavior; that is, providing treatment manuals, presentations, and workshops do not appear to be sufficient to enhance clinicians’ fidelity to treatment (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005; Hoge & Morris, 2002). These training methods continue to be promulgated, however, most likely because they offer a convenient means for community-based clinicians to obtain exposure to EBTs for substance use.

To address the science to service gap and achieve success in preparing clinicians to implement EBTs with fidelity, creative training and monitoring methods that are both effective and practical must be developed. The primary purpose of this article, therefore, is to present a feasible approach that can be used to improve treatment adherence. Specifically, we detail the development and preliminary evaluation of an adherence monitoring system to support clinician adherence for an EBT (a family-based contingency management intervention for adolescent substance use), and describe how the system can be used to promote clinician fidelity and improve training.

Why Monitor Adherence?

Practitioners often wonder why treatment disseminators care so much about fidelity. Findings from community-based research across several EBTs suggest treatment adherence may be positively associated with improved clinical outcomes (see Weisz, Weersing, & Henggeler [2005] for a summary). If achieving desired outcomes relies, in part, on treatment fidelity, then attempts to transport EBTs require effective methods for monitoring and improving treatment fidelity. Given the need to also track other important aspects of treatment (e.g., service hours, clinical outcomes), these methods must be practical, efficient, and readily integrated into existing therapeutic structures, as well as being reliable and valid.

Within highly controlled clinical research, investigators typically ensure integrity of treatment implementation through intensive and ongoing oversight that relies on training therapists to high adherence levels and continued monitoring. For example, in their university-based adolescent substance use intervention research, Azrin and colleagues (1994; 1996; 2001) included: (1) extensive training procedures with modeling and role-playing; (2) ongoing audio-tape coding with corrective feedback; (3) observation of sessions, again with feedback; (4) detailed in-session checklists that contained prompting for the clinician, reviewed weekly; and (5) weekly written documentation by clinicians of techniques used and progress toward treatment goals. Notably, fidelity data from independent raters of audio-tapes and in-session checklists showed greater than 95% adherence (Azrin et al., 1994), indicating that the intensive monitoring and quality assurance system was highly effective. Similar methods of assessing fidelity have been used in other types of research. These include, for example, the examination of clinical records (e.g., Farkas, Cohen, & Nemec, 1988), progress notes (e.g., Sexton, Alexander, & Harrison, 1998), and checklists (e.g., Jerrell & Hargreaves, 1991), as well as employment of manuals, staffing reports, client ratings of clinician style, ethnographic observations, abstracts of program records, and ratings of staff-client process (see Hargreaves, 1990). For more than a decade, Hogue and colleagues have studied the effectiveness of straightforward methods of treatment fidelity that have good face validity and complement staff routines (Faw, Hogue, Liddle, 2005; Hogue, Liddle, & Rowe, 1996; Hogue, Liddle, Rowe, Turner, Dakof, & LaPann, 1998; Hogue, Liddle, Singer, Leckrone, 2005). For instance, in residential settings, Faw, Hogue, and Liddle (2005) obtained an overall score reflecting the average percentage of prescribed services provided (e.g., recreational services, psychiatric consultations, basic living services) based on daily activity logs completed by staff. Clinician or client ratings of adherence also have been employed (e.g., Azrin et al., 2001; Henggeler, Melton, Brondino, Scherer, & Hanley, 1997).

Although the intensive methods used by treatment researchers to ensure adherence within clinical trials are effective, these methods also are time consuming and costly. Such cost and the concomitant expertise needed to implement the methods make them difficult to use in community contexts, and some EBT purveyors (e.g., Multisystemic Therapy[MST]; Henggeler, Schoenwald, Borduin, Rowland, & Cunningham, 1998) have used relatively inexpensive means such as client reports to monitor adherence. Such methods can miss important aspects of the treatment, however, and direct (i.e., in vivo observation) or indirect (i.e., examination of tapes) monitoring of sessions by independent raters generally are considered the most accurate methods for measuring treatment fidelity (Carroll et al., 2002; Madson, Campbell, Barrett, Brondino, & Melchert, 2005). Indeed, most EBTs of adolescent substance abuse rely on video- or audio-tape assessments of clinician behavior to assess fidelity (e.g., Liddle et al., 2001; Santisteban et al., 2003), albeit in well-funded clinical trials.

Thus, the tension is between validity (i.e., observational methods are the best indicators) and cost (i.e., observational methods are also the most expensive) in developing effective and efficient measures of therapist adherence in community-based settings. Resolving this tension (i.e., validating a low cost observational method) is the focus of the present article. A structured tape review system for monitoring adherence was developed in the context of a controlled experimental study regarding contingency management (CM) techniques used in a family-based intervention. Before describing the system and study, however, a brief overview of the EBT that was the focus of the experimental study is provided to give the reader an understanding of the techniques being monitored.

Contingency Management (Family-Based) Techniques

Considerable evidence supports the effectiveness of CM programs for treating substance abuse (see Petry & Simcic [2002] for a summary). Specific CM protocols for adolescent substance abusers include a family-based approach researched by Azrin and his colleagues, with results demonstrating relatively high rates of abstinence as well as other positive outcomes (e.g., decreased depression and conduct problems, improved employment/school attendance; Azrin et al., 1996; Azrin et al., 1994; Azrin et al., 2001). Further, CM is capable of being integrated into other types of psychosocial treatment programs, such as Henggeler and colleagues’ (2006) successful use of CM techniques within a juvenile drug court program that included MST.

The present study used a family-based approach to CM (CM/FB; Cunningham et al., 2003) that is described in more detail on the web at http://_____. The four primary intervention protocols are: Level System, Functional Assessment or Antecedent-Behavior-Consequence (ABC) Assessment, Self-Management, and Drug Refusal Skills Training. Each primary intervention protocol includes subcomponents with specified action steps that guide clinicians in treatment implementation. Primary intervention protocols can be implemented independently or in combination, and implementation of the protocols varies based on family needs. Level System consists of 69 itemized actions across 7 protocol components. ABC Assessment consists of 38 itemized actions across 6 protocol components. Self-Management consists of 15 actions across 3 protocol components. Drug Refusal Skills Training consists of 15 actions across 2 protocol components.

Developing an Efficient Observational System for Assessing Therapist Adherence

The overriding aim of the present study was to develop an adherence monitoring system that would provide (a) efficient methods for supervisors to monitor clinician adherence and for clinicians to conduct self-checks on their CM/FB implementation, and (b) data to assess the need for more in depth training in CM/FB. The initial purpose of this monitoring system was to provide quantifiable data for assessing changes in therapist adherence to CM for a larger study(described subsequently). The developers of the adherence monitoring system, however, also undertook the larger goal of identifying a system that could eventually be used in real world practice settings. To enhance the likelihood that the system would be transportable and relatively unobtrusive, audio-taping was used rather than “in-person” or videotape observation. The remainder of this section describes the adherence monitoring system, and the evaluation of the system is detailed in the subsequent section.

In light of the large number of components and action steps (i.e., 137) required across the four main intervention protocols of CM/FB (i.e., Level System, ABC Assessment, Self-Management, Drug Refusal Skills Training), checklists depicting each of the specified intervention action steps were utilized to record implementation. These included a page for each of the four major protocols, with clear headings for components and succinct descriptions for each action step. This checklist format was developed through an iterative process in which various formats were tested, resulting in a rating form that could be quickly referenced by a rater while listening to session content, even if protocols or the respective action steps were implemented out of order. The checklists were similar to the ones utilized by therapists that depicted actions expected during the administration of the respective CM/FB protocol components.

Even though rating 137 action steps would appear to be comprehensive for assessing adherence to a treatment, there were further detailed behaviors that should be present for full adherence. For example, “explaining that urine tests will be conducted at random” seems rather straightforward to rate, but in actuality involves distinct molecular behaviors (e.g., fully defining “random” to the youth and family - that the youth will not know when the screens will happen, that screens can be conducted at a session, home, or unannounced office visit, that the caregiver will determine when a screen is conducted and may conduct tests during high-risk times such as breaking curfew, as well as ensuring both youth and caregiver comprehend the information and demonstrating enthusiasm and willingness to assist families in understanding this step). Although the measurement of each molecular behavior within the 137 action steps might have been preferred, this level of detail was found to be impractical for raters during tests of various audio-tape rating strategies. Therefore, raters were trained to first determine if the specified action from the therapist checklist was, in general, attempted, and then trained to subsequently lower or raise their rating to accommodate the clinician’s comprehensiveness in performing the molecular behaviors making up the more inclusive action step. Raters were trained in many exemplars to promote accuracy in making such adjustments.

A brief description of the adherence monitoring system follows: (1) Clinicians audio-record all sessions with the youth and family, and one tape from each family per month is selected for review. To minimize selection bias in assessing the clinicians’ implementation of CM/FB protocol, this tape is selected randomly. Of course, more tapes could be reviewed at the discretion of a supervisor or therapist, but these tapes were not included in the present validation study. (2) Raters listen to audio-tape recordings of the sampled therapy sessions and use the rating form to record the extent to which specific steps in each of the four CM/FB intervention protocols were performed. (3) For each component of the four intervention protocols in which at least one action step was judged to be performed, a percentage score is computed (i.e., percentage of prescribed steps that were performed within each component). For example, if two action steps in a component containing four steps were deemed performed, a percentage score of 50% is given for that component. (4) The following 7-point scale is then utilized to convert the percentage scores to preliminary rating scores so adherence scores can be adjusted for comprehensiveness in which a given component is implemented (described previously): a rating of 1 is awarded to a component if the percentage score is 1–13%; 2 = 14–27%; 3= 28–41%; 4= 42–55%; 5= 56–69%; 6= 70–83%; 7= 84–100%. To accommodate comprehensiveness, the rater can increase or decrease each final component rating by 1 unit. Therefore, if a clinician performed steps in the component in a relatively deficient manner (i.e., erroneously implemented or omitted molecular behaviors underlying the respective action), the clinician’s rating is reduced by one (to a minimum of 1), and vice versa (to a maximum of 7).

Evaluating the Adherence Monitoring System

The adherence monitoring system was developed in conjunction with a controlled experimental study that assessed clinician implementation of CM/FB before and after a workshop, as well as during a follow-up period in which some clinicians received intensive quality assurance (IQA) focused on CM/FB. As described subsequently, this methodology allowed for assessment of the feasibility and reliability of the tape coding system as well as preliminary evaluation of the system’s validity.

The Study Methods

Main findings of the larger study, directly comparing training strategies, have been reported elsewhere (Henggeler et al., 2007). The present evaluation examined the reliability and validity of the tape coding system developed in conjunction with the study. Feasibility, reliability, and initial validity evaluations used data across all timeframes of the larger study. Concurrent validity was evaluated using data separated by baseline, immediate post-workshop, and follow-up time periods.

Participants

Clinician participants (n = 27) were employed by three community-based MST programs and were treating antisocial youth who also were marijuana abusing or dependent. MST clinicians were used in this study to reduce the possibility of negative attitudes toward EBTs by the therapist or organization being a confound to adoption of CM. One hundred percent of clinicians consented to participate, although not all consented clinicians were able to identify cases to refer to the study (e.g., did not have cases with suspicion of substance use). Three clinicians did not provide tapes, but all other clinicians with families in the larger study were included in the present study. Most clinicians were female (82%), with ages ranging from 24 to 55 years (M = 33, SD = 8). The sample was primarily Caucasian (63%), but had substantial minority representation (15% African American, 11% Hispanic). Most clinicians held at least a master’s degree (82%), although 18% had only a bachelor degree. Almost half (48%) of clinicians had fewer than 3 months of experience with MST at the beginning of the study, and 30% had 3 to 12 months of MST experience at study start.

After providing informed consent, clinicians could begin referring families they were treating and continued referring families throughout the study. Specifically, clinicians were asked to refer cases where there was suspicion of substance use. The researcher recruited the family, obtaining consent from the legal guardian and assent from the youth. Youth meeting diagnostic criteria for marijuana abuse or dependence based on the Structured Clinical Interview for DSM-IV were admitted for the study. Session tapes were collected for participating therapists and their families throughout the length of the study, which included a baseline period, a 4-month post-workshop period, and a sustainability 6-month follow-up period. Each family participated for 4 months after their recruitment or the end of their MST treatment.

The 52 youth with audio-taped sessions ranged in age from 12 to 17 years (M = 15 years, SD = 1 year) and 69% were male. The racially diverse sample was 39% Caucasian, 27% African American, and 23% Hispanic, with annual income below $10,000 for 29% of families, $10,000 to $20,000 for 17%, $20,000 to $40,000 for 23%, and greater than $40,000 for 31%. Half of households were single-parent families. Youth and families were MST participants prior to their being recruited for this study, so all families in the study received MST regardless of when they entered the study or whether their clinician received IQA for CM/FB (described subsequently).

Training strategies

The baseline data collection period was conducted prior to clinician exposure to CM/FB techniques. At the end of the baseline period, clinicians were provided CM/FB training materials (treatment manuals and forms) and attended a CM/FB workshop. The workshop, conducted by two clinical psychologists with extensive experience in training and delivering the CM/FB protocol, reviewed each of the steps within the CM/FB components and their integration into MST. The workshop also included demonstrations and experiential exercises (e.g., small group exercises and role-plays) to facilitate skill acquisition, and clinicians received checklists for each component of the intervention protocols that would serve as reminders for the necessary action steps. The CM/FB experts, one of whom also was an MST expert, reviewed potential difficulties that might arise during delivery of the CM/FB protocol, such as engaging caregivers and other key stakeholders (e.g., juvenile probation officers) in the CM/FB intervention and refusal of youth to perform role-plays or perform therapy assignments. Immediately following the conclusion of the workshop, MST teams were randomized into a Workshop Only (WSO) condition (n = 10 clinicians for the present study) or an IQA condition (n = 17 clinicians for the present study). Because of (a) teams (as opposed to individual therapists) having to be randomized and (b) therapist turnover during the study, the group sizes are unequal. New therapists entering the study were given a workshop immediately after their MST training.

Following the workshop, teams in the WSO condition had telephone/email access to one of the CM/FB experts. Clinicians randomized to the IQA condition received (1) an additional half day of training for their supervisor; (2) booster trainings in CM/FB; (3) group supervision weekly with the onsite MST+CM/FB trained supervisor; (3) weekly consultation with an MST+CM/FB expert; and (4) goals for CM/FB skill development incorporated into clinician development planning.

Audio-tape collection

As described previously, clinicians were instructed to audio-tape all sessions, although telephone and emergency sessions sometimes could not be taped. One tape from each family per month was selected randomly by the researcher, but if that tape contained fewer than 2 hours worth of taped session time, either the next tape chronologically was coded or, if the selected tape was the last tape of the month, the immediate prior tape was coded. In some cases, 2 full hours of tape were unavailable for a given month (based on therapists’ ability to tape sessions). In total, 215.3 hours of audio-tape were sampled (132.4 = WSO, 82.9 = IQA) across 52 substance abusing youth (WSO = 26, IQA = 26). After training, a primary coder who was blind to study hypotheses and conditions coded all tapes. In addition, secondary coders reviewed 47.2% (101.6 hours) of the tapes.

Feasibility, Reliability, and Validity

This evaluation had three aims: 1) to examine the clinical feasibility of the adherence monitoring system; 2) to examine its psychometric properties (i.e., face and content validity, inter-rater reliability); and 3) to examine concurrent validity.

Clinical feasibility

To ensure the adherence monitoring system was easy to learn, replicable, and relatively inexpensive, the raters used in this initial examination of the system had no previous training in psychotherapy and received fewer than 8 hours of training. In spite of these restrictions, estimates of their inter-rater reliability were high, as discussed subsequently. Thus, clinicians and supervisors should be able to be trained quickly to adequate reliability, which will facilitate integration of the review system into their monitoring and supervision structure.

Face and content validity

The steps identified in the CM/FB adherence monitoring system were based directly on those action steps clinicians were trained on and were listed in handouts provided during trainings. Thus, the system clearly was a face valid translation of the construct of interest and accurately operationalized CM/FB implementation. Notably, the interventions included in CM/FB are prescribed and well specified, making observation of therapist behavior relatively straightforward. Consequently, even minimally trained reviewers can identify implementation.

Reliability

Inter-rater reliability was assessed using the Kappa coefficient of agreement, which takes into consideration expected agreement (Cohen, 1960), for the tapes rated by both coders. A full table of results can be found at http://_____. Inter-rater reliability of protocol adherence was generally excellent overall for the intervention protocols, as well as for the components within each: Level System (for within ±1 rating: κ = 0.937; for absolute agreement: κ = 0.913), ABC Assessment (for within ±1 rating: κ = 0.956; for absolute agreement: κ = 0.954), Self-Management (for within ±1 rating: κ = 0.945; for absolute agreement: κ = 0.952). Notably, the Drug Refusal protocol was attempted with such low frequency, that accurate agreement coefficients could not be computed.

Concurrent validity

Data from the three time periods of the trial comparing the WSO and IQA methods were used to evaluate concurrent validity, or the capacity of the adherence monitoring system to distinguish between clinician tapes during which very little or no CM/FB should have occurred versus tapes in which CM/FB should have been delivered. The first 5 months served as a baseline assessment of the degree to which clinicians implemented CM/FB prior to any exposure to CM/FB training or techniques. As seen in the Table, only the Level System protocol was documented as being attempted during the baseline period. Specifically, 38.5% of tapes in the WSO condition had observable Level System actions, which was anticipated given (a) the behavioral intervention techniques normally implemented within the MST approach and (b) that many youth referred to MST are on probation and undergo urine drug screens as part of juvenile probation requirements. Thus, the adherence monitoring system indicated that tapes collected during this time period evidenced low implementation of the Level System protocol and no implementation of the ABC Assessment, Self-Management, or Drug Refusal protocols of CM/FB.

Table.

Tapes with Intervention Protocol Attempts and Mean Ratings Received by Groups on Each Protocol for Different Time Periods

		Pre-Workshop		Immediate Post-Workshop		Follow-up
		WSO	IQA	WSO	IQA	WSO	IQA
Intervention Protocol		13 tapesa	15 tapes	20 tapes	20 tapes	63 tapes	35 tapes
Level System	% of tapes with attemptsc	38.5%	0%	75%	55%	46%	54%
	Rating M (SD)d	3.20 (1.10)	e	4.00 (1.82)	3.88 (1.86)	3.39 (1.49)	3.86 (1.91)
ABC Assessment	% of tapes with attempts	0%	0%	15%	10%	2%	14%
	Rating M (SD)	e	e	3.86 (1.68)	4.20 (2.05)	5.00 (1.41)	4.10 (1.73)
Self-Management	% of tapes with attempts	0%	0%	5%	5%	5%	14%
	Rating M (SD)	e	e	7.00 (f)	5.00 (f)	4.67 (1.51)	4.88 (1.81)
Drug Refusal	% of tapes with attempts	0%	0%	0%	0%	0%	0%
	Rating M (SD)	e	e	e	e	e	e

Note. WSO = Workshop Only; IQA = Intensive Quality Assurance

^aThe number of tape segments available for review for this column (i.e., on a monthly basis, one tape segment of up to 2 hours per participating family was randomly selected, if tapes were available.)

^bThe number of clinicians submitting tapes for this time period. Therapist turnover and not having current youth clients in the study resulted in the therapist cohort not being consistent over time.

^cAttempts is defined as obtaining a rating >0 for that intervention protocol.

^dMean and standard deviation for rating of tapes in which the intervention protocol was attempted.

^eNo intervention protocol attempted.

^fCannot be computed with n = 1

After clinicians were provided CM/FB training materials (treatment manuals and forms) and attended a CM/FB workshop, the number of tapes identified by the adherence monitoring system as having CM/FB techniques attempted and the percentage of steps observed in those tapes increased. Inferential statistics on this data were precluded by sample size and an unstable sample (i.e., the same clusters of families and clinicians are not represented across all time periods given the short duration of MST treatment). Therefore, visual estimation (see Hurlburt, 1994) for program data was used to examine the magnitude of time and condition effects. As seen in the Table, a fairly consistent pattern of findings is apparent across three of the four CM/FB protocols. Data from the 4 months after the workshop indicated that the immediate response of clinicians to the workshop was to increase use of most CM/FB techniques, with a particularly sharp increase in Level System implementation. Further, this qualitative pattern generally was sustained during the follow-up period, with relatively greater sustainability among the clinicians receiving IQA focused on CM/FB techniques. Notably, the tape review system identified no instances of Drug Refusal protocols being conducted.

Future Directions and Implications

Although practical and efficient, traditional methods for training and monitoring therapists (e.g., providing manuals, presentations, and workshops) do not generate sustained change in clinician behavior. The system described here provides a creative, yet practical, solution with preliminary evidence for usefulness. That is, the adherence monitoring system, used here for an adolescent substance use intervention, proved to have feasibility and ease of use, as well as face and content validity. Training for the system was particularly straightforward and efficient, with reliability easily attained. Further, a preliminary test of the system’s validity was afforded by the controlled trial, with the concurrent validity of the adherence monitoring system receiving some support.

Integrating the System into Training

Currently, there is great variability in training and supervision within evidence-based substance abuse treatment programs (Madson et al., 2005), and little is known as to the type and extent of supervision necessary to bring about satisfactory implementation within community settings (Miller, Sorensen, Selzer, & Brigham, 2006). Specialized training procedures with ongoing supervision have been recommended to promote implementation fidelity(Andrzejewski, Kirby, Morral, & Iguchi, 2001).

The system described here can likely be integrated into the training and supervision systems employed by community providers, with therapists, supervisors, or other onsite staff being trained to conduct the reviews. For example, the experience of this research showed that learning the adherence monitoring system is not overly complex(i.e., individuals without graduate training and with no previous training in psychotherapy underwent fewer than 8 hours of training). Therefore, therapists and supervisors should be able to be efficiently trained in reliably reviewing tapes. Further, protocol checklists are straightforward and succinct enough that therapists can complete the checklists as they conduct their sessions (i.e., therapist indicates with a check mark if each protocol step in the respective intervention was completed or not). Supervisors can select tapes of therapy sessions (randomly or otherwise) to review, independently completing protocol checklists that can be compared with the therapist’s self-report for that session. In addition to providing an ongoing check to ensure therapists are adhering to intervention protocols, this process also can pinpoint areas to focus on in additional, individualized training.

Within their clinical trials, Azrin and colleagues (e.g., Azrin et al., 2001) have piloted some of these tape review feedback methods for improving CM protocol adherence, albeit using independent reviewers rather than clinical supervisors to provide feedback. For example, trained reviewers listened to a percentage (e.g., 10% randomly selected) of the audio-tape sessions that were conducted by a therapist during the previous week and utilized a similar checklist system. The tape reviewers’ lists were compared to the treatment session lists completed by the therapists, and specific feedback was provided to the therapist during weekly group supervision. Therapists also reported that, aside from the initial clinical training in the intervention, this method significantly improved adherence because (a) they became better trained on accurate identification of and conduct of each component and (b) they knew they would be held accountable for accuracy in reporting. Thus, use of such a system appeared to be beneficial from multiple perspectives.

Anecdotally, therapists with relatively high levels of protocol implementation in these trials reported that they preferred focusing more on the protocol checklists rather than the treatment manual for training. They reported including the therapist protocol checklists in functionally operative “notebooks” created for use within treatment sessions. Although stylistically unique across therapists, these notebooks included tabs indicating folders where the protocol checklists and worksheets were stored for each component and set of steps. Thus, upon deciding to implement a particular aspect of the CM/FB protocol, the therapist had ready access to the materials. This unique arrangement of the materials is important given the flexible nature of CM/FB and other “non-cookbook” interventions in which the therapist may need to quickly draw on intervention material in response to what occurs during the session (Kendall & Beidas, 2007). In retrospect, it may be important to provide pre-organized notebooks to therapists during training and instruct them in the effective use of such materials within treatment sessions.

In addition to considering the use of the developed system to support therapist adherence and inform development of clinical materials, such an adherence monitoring system could be utilized by community agencies or educational institutions to train clinicians prior to contact with clients. Specifically, a library of scored tapes could be generated that includes exemplars, as well as tapes with selected subcomponents missing or poorly conducted. After receiving introductory training in the intervention and being trained to conduct the adherence monitoring system, the trainee would code the template tapes. Their checklists and scoring would then be evaluated for accuracy, and trainees could receive feedback before moving on to other samples in the library. Advantageously, this protocol could be conducted with or without an on-site trainer and at the pace and convenience of the trainee.

Relevant to dissemination, an identified barrier to implementing EBTs in community settings is clinicians’ attitudes regarding the particular intervention (e.g., McGovern, Fox, Xie, & Drake, 2004). Similarly, the effectiveness of a fidelity monitoring system can be influenced by clinician attitudes toward the system they are being asked to implement. Future investigations should attempt to systematically determine consumer (i.e., therapist, supervisor) satisfaction with the developed system.

Limitations

The present study was conducted using data from a pre-existing design intended to investigate a related, but different, set of questions. Thus, the design did not provide a full test of the developed adherence monitoring system. For example, the sample size was suboptimal. However, this number of clinicians is greater than or commensurate with the relatively few existing studies evaluating training effects on adherence (e.g., Andrzejewski, Kirby, Morral, & Iguchi, 2001; Madson et al., 2005; Multon, Kivlighan, & Gold, 1996; Rounsaville, Chevron, Weissman, Prusoff, & Frank, 1986). In addition, predictive validity would be a useful supplement to the evaluation presented here. Further, the adherence monitoring system was not included in the IQA system used in the larger study of therapist training that provided the present data. Obviously, including the system as part of the larger study’s training procedures would have provided an opportunity for further evaluation.

Although the adherence monitoring system for CM/FB was useful and efficient, it may not be feasible for other EBTs. CM/FB consists of four intervention protocols that are well described, and therefore, easily identified when reviewing session audio-tapes. Thus, content validity for CM/FB might have been easier to attain than for more complex interventions because each step of the CM/FB protocol contains observable behaviors. Some EBTs will have greater difficulty in this respect as their intervention methods are less behaviorally defined.

Reliance upon the percentage of therapeutic actions administered was shown to be a reliable and valid method of protocol adherence in this study, but percentage scores were converted to rating scores. While providing parsimony for summing data across units (e.g., therapists or programs) or for statistical analyses, conversion into rating scores may add greater complexity to the interpretation of adherence. The procedures used, however, were an attempt to enhance precision, while developing a system that was not so molecular that it made the system impractical. Rating data, however, can be reported on the broader adherence rating scale to maintain parsimony, or in a more detailed format of specific percentages and precision adjustments for review between a specific therapist and supervisor (e.g., for individualized feedback that does not necessarily need to be parsimonious.)

Also of note, the occurrence rates for CM/FB interventions were lower than would be expected for therapists solely conducting CM/FB or if therapists were treating adolescents only for substance use. Although CM/FB is relatively comprehensive for treating substance use, other MST-based interventions were occurring during session tapes. These interventions were not monitored by the CM/FB adherence monitoring system. The low occurrence rates of CM/FB interventions highlight the practical challenges of monitoring implementation in community-based settings. As the number of distinct interventions implemented increases or becomes more eclectic, it becomes more difficult for the individual conducting the supervision and quality assurance to identify and subsequently monitor the respective interventions. Monitoring adherence to multiple interventions being implemented with the same client is particularly relevant when clinicians are treating psychological conditions that may co-occur with substance abuse (e.g., internalizing disorders).

Notably, Drug Refusal Skills Training did not appear to be used by clinicians across the project time periods. When implementing CM/FB, this protocol should occur at frequencies similar to the other three protocols. Possible reasons for this lack of use include (a) community counselors of antisocial youth may not find value in drug refusal skills (necessitating additional clarification of the rationale) and/or (b) community counselors might perceive they are incapable of implementing the intervention effectively (necessitating additional training). More simply, perhaps the instruction in Drug Refusal Skills Training, as the final component of the workshop, was not as competently delivered by the trainers or the clinicians were fatigued by that point in training. Any of these conclusions would suggest additional training strategies are needed and that a creative, practical, and flexible training system like the one described previously might be effective.

Conclusion

The intensive and ongoing oversight employed within controlled clinical research to ensure treatment integrity may not be transportable when investigators are ready to disseminate EBTs. Instead, practical methods must be developed and evaluated, methods that can be integrated efficiently into training and supervision mechanisms in the community. The adherence monitoring system described here proved to be reliable and valid in preliminary tests, and could feasibly be incorporated into a quality assurance protocol for training and monitoring clinician adherence.

Biographies

ASHLI J. SHEIDOW received her PhD from the University of Illinois at Chicago. She is Assistant Professor of Psychiatry and Behavioral Sciences at the Family Services Research Center and a Faculty Scholar at the Center for Health Economic and Policy Studies, Medical University of South Carolina. Her research interests include the development, prevention, and treatment of adolescent psychopathology and juvenile delinquency from an ecological perspective, with a concentration in both co-occurring disorders and the effective dissemination of evidence-based practices.

BRADLEY C. DONOHUE received his PhD from Nova Southeastern University in 1995. He is Associate Professor and Director of Achievement Center in the Psychology Department at the University of Nevada, Las Vegas. His research interests include the development, controlled evaluation, and dissemination of interventions for substance abuse and its associated problems, e.g., child maltreatment, conduct disorders.

HEATHER H. HILL received her MA in psychology from the University of North Carolina at Wilmington. Currently she is a graduate student in the Clinical Psychology PhD program at the University of Nevada, Las Vegas (UNLV) and the Program Coordinator of the Achievement Center at UNLV. She is the editorial assistant of the Journal of Child and Adolescent Substance Abuse. Her research and clinical interests include child maltreatment, substance abuse, and treatment outcome research.

SCOTT W. HENGGELER received his PhD in clinical psychology from the University of Virginia in 1977. Currently, he is Professor of Psychiatry and Behavioral Sciences at the Medical University of South Carolina and Director of the Family Services Research Center (FSRC). The mission of the FSRC is to develop, validate, and study the dissemination of clinically effective and cost effective mental health and substance abuse services for children presenting serious clinical problems and their families.

JULIAN D. FORD received his PhD in clinical psychology from the State University of New York at Stony Brook. He is Associate Professor of Psychiatry at the University of Connecticut School of Medicine and Director of the University of Connecticut Health Center Child Trauma Clinic. His areas of research include adult, adolescent, and child traumatic stress disorders, psychotherapy and family therapy process and outcomes, health service utilization, and psychiatric epidemiology.