Abstract
Background
Teaching community program therapists to use motivational interviewing (MI) strategies for addictions treatment with sufficient frequency (i.e., adherence) and skill (i.e., competence) is a priority and challenge for the field. The development of psychometrically valid MI integrity measures that can be used for supervision and evaluation and be both sensitive and robust across clinical situations is needed.
Objective
This article examines the performance of the Independent Tape Rating Scale (ITRS) (1) when used to evaluate the delivery of MI within a one-session assessment intake (2).
Methods
Audiotapes of 315 sessions of therapists in MI and counseling-as-usual conditions were rated according to the ITRS by raters blind to treatment condition.
Results
Results indicate that community therapists were successfully trained and supervised to use MI within an assessment intake session, with MI adherence and competence that was discriminable from counseling-as-usual practices. Increased therapist MI adherence and competence was associated with increases in an index of client motivation for change, though unrelated to treatment outcome.
Conclusions and Scientific Significance
The ITRS appears to be a valid instrument for measuring therapist MI adherence and competence within an assessment intake.
Keywords: motivational interviewing, substance abuse treatment, therapist adherence and competence, therapist training and supervision
INTRODUCTION
Motivational interviewing (MI) (3) is efficacious treatment for addictions and other problem behaviors (4) characterized by a combination of client-centered counseling techniques and strategies to directly elicit clients' motivation for positive behavior change (5). Teaching therapists to perform empirically supported treatments, including MI, with integrity has become a priority for the field (6). Treatment integrity refers to the degree to which therapists deliver interventions as intended with sufficient frequency (i.e., adherence), skill (i.e., competence), and discrimination from other treatments (7). The development of psychometrically valid MI integrity measures that can be used to supervise therapists' MI performance, discriminate it from other counseling methods, and be robust when used with diverse samples of therapists, treatment settings, and applications of MI is needed.
This article examines the psychometric properties of a MI adherence and competence measure, called the Independent Tape Rating Scale (ITRS) (1), when used within a National Institute on Drug Abuse (NIDA) Clinical Trial Network (CTN) protocol that incorporated MI into a one-session assessment intake (2). The protocol compared the effectiveness of a MI assessment (MIA) to a counseling-as-usual (CAU) intake. MIA, in comparison to the CAU, resulted in significantly better 4-week client retention and reduced days of primary substance use in clients whose primary substance was alcohol rather than drugs (2).
The ITRS has been found to be reliable and valid in two previous CTN studies of motivational enhancement therapy (MET), one delivered in English (8) and the other in Spanish (9). In both protocols the ITRS demonstrated excellent inter-rater reliability. In the English MET protocol, the scale revealed two MI skills factors—fundamental skills (client-centered counseling strategies) and advanced skills (multiple methods for evoking change talk). It also discriminated between MET and CAU, and was related to clients' motivation and some treatment outcomes (8). In the Spanish MET protocol, MI integrity was assessed among bilingual therapists who used MET with monolingual Spanish-speaking clients (9). The ITRS showed strong support for the fundamental skills factor in this protocol, but only weak support for an advanced skills factor. It effectively discriminated between MET and CAU.
The ITRS served as the foundation for one of the five NIDA-Substance Abuse and Mental Health Services Administration (SAMHSA) Blending Initiative's products (10), called Motivational Interviewing Assessment: Supervisory Tools for Enhancing Proficiency (11). This product contains ITRS rating items for community program supervisors to evaluate their therapists' use of MI, provide feedback to them, and coach therapists to perform MI in a manner discriminable from usual practice assessments. The validity of the ITRS for assessing the adherence and competence of therapists using MI during an intake has not previously been established, and is paramount to such training initiatives.
In this study we examine the performance of the ITRS in measuring community therapist MI integrity within an assessment intake (2). We hypothesized that: 1) the MI consistent adherence items would converge to form two factors that reflect fundamental and advanced MI skills, 2) the ITRS would discriminate higher integrity among therapists trained and supervised to deliver MIA compared to those conducting CAU (i.e., greater MI-consistent and lower MI-inconsistent adherence; higher MI-consistent competence), and 3) MI adherence and competence scale scores would be positively associated with increases in measures of client motivation within sessions and better client retention and substance use outcomes at follow-up.
METHODS
Participants
A full description of study participants and inclusion/exclusion criteria can be found in the main outcome paper (2). Participants included therapists, clients, and tape raters. Thirty-seven therapists employed in one of four outpatient nonmethadone substance abuse treatment programs delivered MIA or CAU. Client participants included 423 outpatients, 89% (n = 377) of whom completed an assessment intake session. Fifteen independent tape raters were trained to evaluate the session audiotapes generated within the protocol. These raters were the same as those used in the English MET study of the ITRS (8).
Assessment of Therapist Adherence and Competence
All recorded and audible sessions from the protocol were rated (84% of all sessions), resulting in 315 rated sessions (155 MIA, 160 CAU). Therapist adherence and competence was evaluated using the ITRS. The scale has been described elsewhere (1, 8), and consists of 39 items, all of which are rated on a 7-point Likert scale. Lower ratings indicate a general absence of behavioral indicators and higher scores indicate more extensive occurrence. The scale has been demonstrated to have excellent interrater reliability (8, 9). This report focuses on the 10 MI consistent items (those prescribed and encouraged in MI), and the 10 MI inconsistent items involving interventions antithetical to MI (e.g., unsolicited advice), or typical of treatment approaches different from MI (e.g., skills training). With the exception of items involving rating other treatment approaches, these items are the same as those used in the NIDASAMHSA Blending Initiative MI supervision toolkit. Each item is rated for adherence (extent to which the therapist delivered the intervention based on frequency counts, with 1 = not at all to 7 = extensively) and competence (skill of delivery, with 1 = very poor to 7 = excellent). Competence ratings were anchored with guidelines that reflected higher and lower levels of skill for each item and were only provided if the therapists had performed that item (i.e., item adherence rating ≥2).
Assessment of Client Motivation, Retention, and Substance Use Outcomes
Change in client motivation was measured using independent 7-point global ratings of the first and last five minutes of the session. Each scale point reflected a relative balance of client change talk and resistant statements, such that 1 represented no motivation for changing primary substance use (very little change talk and very strong resistance), and 7 represented extremely strong motivation for change (almost all change talk and very little resistance). We subtracted motivation at the beginning of the session from motivation at the end session to obtain a change in motivation score (range = −6 to 6). Client retention data (days of program enrollment) was based on self-reports and confirmed with client records. Substance use was assessed with the Timeline Followback method (12, 13). Self-reports were compared to urine and breath screens for accuracy; these comparisons indicated high correspondence (2).
Therapist Training, Certification, and Supervision Procedures
Therapists were randomized to either continue administering the intake assessment according to their usual practice, or to be trained and supervised to use MI as part of the intake session. The MI assessment intake session, described elsewhere (2), involved the formal use of MI during the first and last 20 minutes of the session, hence, “sandwiching” the traditional intake with MI practice. Across all programs, the traditional intake was characterized by the detailed assessment of the clients' substance use and related psychosocial factors (14). MI therapists and program-based supervisors received a 16-hour intensive MI workshop training, followed by audio-taped practice cases supervised by MI experts until they demonstrated minimal performance certification standards (i.e., at least half of the MI-consistent items rated at a minimum of “4” or above in terms of adherence and competence) in three sessions. The MI experts also reviewed with the supervisors how to use the ITRS after the workshop and continued to support their use of the scale with monthly consultation calls, which included comparison of adherence and competence ratings from common tapes of trial sessions. After therapists were certified in MI, they began to treat randomized clients in the protocol and receive biweekly supervision from program-based supervisors who provided the therapists with MI adherence and competence rating-based feedback and coaching after reviewing audio-taped client sessions. Therapists in both conditions audio-taped all protocol sessions for independent treatment integrity assessment (2). Across conditions therapists delivered an identical number of sessions.
Independent Tape Raters and Training
Characteristics of the raters and their training have been described in detail elsewhere (8). In brief, 67% of the raters had Master's degrees in a clinical profession, an average of 6.9 years (sd = 9.7) of substance abuse treatment experience, 8.3 years (sd = 7.9) of general psychotherapy experience, and 5.6 years (sd = 5.3) of clinical research experience. Sixty percent of the raters had served as independent raters in prior clinical trial studies testing the efficacy of behavioral treatments, and 53% had reported prior MI workshop training (on average 9.0 hours [sd = 5.9]). Training involved the use of didactics, review of the rating manual, practice rating exercises, performance feedback, and rating calibration tape samples to evaluate inter-rater reliability via estimates of intraclass correlation coefficients (ICC) (15). Raters received about 44 hours of ITRS training.
Statistical Analyses
To test for the two hypothesized MI consistent factors (fundamental and advanced skills), we conducted confirmatory factor analysis using structural models with AMOS (6.0) software (16), using maximum likelihood estimation and several indices to determine the acceptability of model fit (17–19)1 We calculated the mean score of the MI inconsistent items that occurred on average at least once per rated session in either condition, as was done in the prior ITRS studies (8, 9). To test for the predicted differences in fundamental MI, advanced MI, and MI inconsistent strategy ratings (i.e., six comparisons), we conducted ANOVAs using a Bonferroni-corrected α of .0084 (.05/6) with the two mean CFA-derived MI consistent factors and the mean MI inconsistent adherence and competence scores as the separate dependent variables (i.e., six contrasts), treatment condition, and program site as the fixed factors, and therapists (nested within condition) as a random factor. Multivariate ANOVAs were used to compute estimates (Roy's theta) of the proportion of variance accounted for by treatment condition, program site, and therapist (within condition) effects, with the respective mean adherence and competence scores entered simultaneously in separate analyses (21). Finally, we calculated Pearson correlations to test for the predicted positive associations between therapist MI adherence and competence and client outcomes (in-session change in client motivation, program retention, primary drug abstinence, and percent drug negative urine screens).
RESULTS
ITRS Scale Validity
Overall, MI consistent items occurred on average about 2–4 times within sessions (Table 1). The most frequently occurring MI consistent items were fundamental MI skills: reflections, open-ended questions, and MI style. MI inconsistent items seldom occurred. The sample sizes for the mean competence ratings for the items varied considerably across conditions from n = 133 for open-ended questions in MIA to n = 2 for heightening discrepancies in CAU. Overall, mean competence ratings across items and treatment conditions suggested an “average” therapist skill level.
TABLE 1.
MI consistent and MI inconsistent item ratings: adherence and competence means, standard deviations, and intraclass correlation coefficient reliabilities.
| Adherence |
Competence |
|||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| MIA |
CAU |
MIA |
CAU |
|||||||
| Items | M | SD | M | SD | ICC | M | SD | M | SD | ICC |
| MI Consistent (Fundamental Skills) | ||||||||||
| 1. Open-ended questions | 5.8 | 1.2 | 4.2 | 1.8 | .89 | 5.1 | .9 | 4.1 | .9 | .81 |
| 2. Reflective statements | 5.8 | 1.5 | 3.4 | 1.9 | .94 | 4.9 | 1.1 | 4.7 | .9 | .88 |
| 3. Affirmations of strengths/self-efficacy | 2.9 | 1.6 | 1.8 | 1.3 | .96 | 4.6 | .8 | 4.2 | .8 | .84 |
| 4. Fostering a collaborative relationship | 2.9 | 1.5 | 2.0 | 1.3 | .66 | 4.6 | .9 | 4.2 | .9 | .69 |
| 5. Motivational interviewing style | 4.6 | 1.6 | 2.6 | 1.8 | .86 | 4.8 | 1.1 | 4.2 | 1.1 | .82 |
| MI Consistent (Advanced Skills) | ||||||||||
| 1. Client-centered problem discussion and feedback | 4.4 | 1.6 | 3.7 | 1.6 | .93 | 4.7 | .8 | 4.2 | .9 | .90 |
| 2. Pros, cons, and ambivalence | 3.4 | 2.0 | 1.5 | 1.1 | .99 | 4.6 | 1.0 | 4.0 | .8 | .97 |
| 3. Heightening discrepancies | 1.6 | 1.2 | 1.0 | .4 | .91 | 4.7 | .9 | 4.5 | .7 | .91 |
| 4. Motivation for change | 3.6 | 1.7 | 1.8 | 1.1 | .87 | 4.6 | .8 | 3.8 | .9 | .79 |
| 5. Change planning | 3.5 | 1.9 | 1.5 | 1.0 | .93 | 4.8 | 1.0 | 3.8 | .8 | .90 |
| MI Inconsistent | ||||||||||
| 1. Emphasis on abstinence | 1.3 | .8 | 1.4 | .9 | .89 | 4.0 | .8 | 3.9 | .8 | .93 |
| 2. Powerlessness and loss of control | 1.2 | .7 | 1.1 | .5 | .74 | 4.0 | .7 | 4.0 | .8 | .78 |
| 3. Unsolicited advice or direction Giving | 2.3 | 1.9 | 2.6 | 1.9 | .96 | 4.3 | .9 | 4.2 | 1.0 | .94 |
| 4. Confrontation of denial/defensiveness | 1.3 | 4.1 | 1.3 | .8 | .78 | 4.7 | 1.0 | 4.3 | .7 | .71 |
| 5. Therapeutic authority | 2.0 | 1.6 | 2.5 | 1.9 | .97 | 4.3 | .9 | 4.1 | .9 | .96 |
| 6. Self-help group involvement | 2.5 | 1.2 | 2.4 | 1.1 | .98 | 4.1 | .8 | 4.0 | .7 | .96 |
| 7. Reality therapy principles | 1.1 | .5 | 1.1 | .4 | .85 | 4.2 | .8 | 3.7 | 1.0 | .81 |
| 8. Skills training | 1.1 | .4 | 1.0 | .1 | .55 | 4.3 | 1.5 | 4.0 | .0 | .57 |
| 9. Cognitions | 1.2 | .6 | 1.0 | 2 | .86 | 3.9 | .9 | 3.8 | .5 | .88 |
| 10. Psychodynamic interventions | 1.1 | .5 | 1.1 | .5 | .57 | 4.0 | .7 | 4.1 | .4 | .67 |
Note: MIA = motivational interviewing assessment; CAU = counseling-as-usual; ICC = intraclass correlation coefficients based on the Shrout and Fleiss (15) two-way mixed model. Adherence and competence ratings are on a 7-point Likert scale from 1 = not at all/very poor, to 7 = extensively/excellent.
Adherence rather than competence ratings were used in the CFA because competence is predicated on the occurrence of the items, the high variability in completed competence items, and the insufficient number of several competence items. We predicted 5 items would form an independent fundamental MI skills factor and 5 items would form an independent advanced MI skills factor. Table 2 reports the fit indices of the two models from the confirmatory factor analysis of our two predicted models. The fundamental MI skills factor had a reasonable fit, surpassing three of six thresholds for goodness-of-fit. The hypothesized advanced MI skills factor did not meet threshold fit criteria for any index, indicating poor fit. Nonetheless, for hypothesis testing purposes and to compare findings with the two prior psychometric reports about the ITRS, we derived separate factor scores of therapist MI adherence and competence in subsequent analyses.
TABLE 2.
MI fundamental and advanced confirmatory factor analysis fit indices.
| CFA Fit Indices |
||||||||
|---|---|---|---|---|---|---|---|---|
| Items | χ 2 | df | p | χ2/df | NFI | IFI | CFI | RMSEA |
| MI Consistent (Fundamental Skills) | 27.33 | 5 | .00 | 5.47 | .95 | .96 | .96 | .12 |
| MI Consistent (Advanced Skills) | 44.55 | 5 | .00 | 8.91 | .83 | .85 | .84 | .16 |
Note: In confirmatory factor analysis, the goodness-of-fit of any predicted latent structure is determined by the preponderance of several indices suggesting a well-fitted model. These fit indices include a nonsignificant chi-square value, chi-square degrees of freedom ratios < 2, a normed fit index (NFI), incremental fit index (IFI), and comparative fit index (CFI) < .90, and the root mean square error of approximation (RMSEA) ≤.10 degrees of freedom (17–19). Statistics meeting these thresholds are in bold face.
Adherence and Competence Analyses between Treatment Conditions, Program Sites, and Therapists
Multivariate ANOVAs examining adherence and competence in fundamental MI, advanced MI, and MI inconsistent strategies displayed significant differences by treatment condition, site, and the interaction of treatment and site (p < .001), as well as a significant difference by therapist (Table 3). The univariate breakdown of these findings revealed that therapists in the MIA condition employed more fundamental and advanced MI interventions (adherence) more skillfully (competence) than therapists in the CAU condition, with no differences between the groups in MI inconsistent strategy adherence or competence. The amount of variance accounted for by treatment condition was considerably higher than the amount of variance accounted for by site, condition by site, or therapist for adherence (thetas 48% vs. 19%, 9%, and 9%, respectively) and competence (thetas 18% vs. 9%, 6%, and 9%, respectively).
TABLE 3.
Fundamental and advanced MI skills, and MI inconsistent strategies adherence and competence ratings by treatment condition, program site, and therapist (nested in condition): univariate and multivariate ANOVAs.
| ANOVA | MIA | CAU | Treatment Condition | Program Site | Treatment by Site | Therapist (in condition) |
|---|---|---|---|---|---|---|
| Univariate | ||||||
| Fundamental MI Skills | ||||||
| Adherence | M = 4.43 | 2.83 | F = 125.98 | 9.64 | 1.34 | 9.52 |
| SD = 1.03 | 1.13 | df = 1,232 | 3,232 | 3,232 | 2,232 | |
| N = 154 | 150 | p < .001 | p < .001 | p=.26 | p < .001 | |
| Competence | M = 4.84 | 4.05 | F = 22.00 | 10.93 | 1.70 | 9.91 |
| SD = .79 | .83 | df = 1,196 | 3,196 | 3,196 | 2,196 | |
| N =154 | 145 | p < .001 | p < .001 | p = .39 | p < .001 | |
| Advanced MI Skills | ||||||
| Adherence | M = 3.32 | 1.90 | F = 196.96 | 6.92 | 5.47 | 1.74 |
| SD = .95 | .60 | df = 1,232 | 3,232 | 3,232 | 2,232 | |
| N = 154 | 150 | p < .001 | p < .001 | p < .001 | p =.18 | |
| Competence | M = 4.69 | 4.05 | F = 14.80 | 7.93 | 5.66 | 7.84 |
| SD = .75 | .88 | df = 1,196 | 3,196 | 3,196 | 2,196 | |
| N = 153 | 143 | p < .001 | p < .001 | p = .01 | p < .001 | |
| MI Inconsistent Strategies | ||||||
| Adherence | M = 2.26 | 2.52 | F = 3.43 | 8.33 | 1.58 | 1.14 |
| SD = 1.26 | 1.35 | df =1,232 | 3,232 | 3,232 | 2,232 | |
| N = 154 | 150 | p = .07 | p < .001 | p = .20 | p = .32 | |
| Competence | M = 4.19 | 4.00 | F = 1.99 | 3.29 | 1.66 | 2.41 |
| SD = .75 | .76 | df = 1,196 | 3,196 | 3,196 | 2,196 | |
| N = 126 | 134 | p = .06 | p=.11 | p=.38 | p=.11 | |
| Multivariate | ||||||
| Adherence | i = .48 | .19 | .09 | .09 | ||
| F = 70.62 | 18.39 | 8.00 | 8.03 | |||
| df = 3,230 | 3,232 | 3,232 | 3,231 | |||
| p < .001 | p < .001 | p < .001 | p < .001 | |||
| Competence | i = .18 | .09 | .06 | .09 | ||
| F = 14.60 | 6.70 | 4.28 | 6.31 | |||
| df = 3,194 | 3,196 | 3,196 | 3,195 | |||
| p < .001 | p < .001 | p < .001 | p < .001 |
Note: For univariate ANOVAs, p-values ≤.0084 (Bonferroni-correction of .05/6) are significant. MIA = motivational interviewing assessment; CAU =counseling-as-usual. Univariate ANOVAs (for tests of adherence and competence differences) and multivariate ANOVAs (for deriving Roy's theta) included two MIA and CAU therapists from each site who had five or more unique client sessions that had been independently rated (MI =242/315). Roy's theta θ provides an estimate of the amount of variance accounted for by each effect involving adherence and competence ratings, with each set of ratings entered simultaneously in separate multivariate models (21).
Adherence and Competence, In-session Change in Motivation, and Client Outcomes
Adherence and competence for fundamental and advanced MI skill factors were positively related to in-session increases in client motivation (r ranged from.14 to.34, p < .01). The use of MI inconsistent strategies was negatively associated with in-session increases in client motivation (r=−.15, p < .01). Significant associations between MI adherence and competence and client treatment outcomes were not present.
DISCUSSION
Training therapists in the delivery of empirically supported treatments is a priority for the field and relies on the development of high quality training initiatives. Such training procedures depend on the ability of supervisors and programs to evaluate therapist performance. The current study examined the psychometric properties of the ITRS when used in an assessment intake. The results of this study replicated the majority of the previous findings (8, 9) about the use of the ITRS as an MI integrity measure. The findings indicate that the scale is a valid instrument for measuring therapist MI adherence and competence within a single intake session. The study also indicates that community therapists can be successfully trained and supervised to use MI within an assessment intake session, with MI adherence and competence that is discriminable from CAU practice. Finally, increased therapist MI adherence and competence was associated with increases in client motivation for changing their substance abuse during the intake, though unrelated to treatment outcome.
Consistent with the two previous studies (8, 9), a subset of the MI consistent items converged to form a fundamental MI skills factor that captured the client-centered component of the approach. Contrary to the English MET trial, but consistent with the Spanish MET trial, there was less support for an advanced factor tapping strategic aspects of MI. This may be due to limited opportunities to use these skills within intake sessions that require substantial assessment of the clients' substance use and related factors. Therapists had a relatively limited amount of time to work with clients motivationally in one session, and needed to balance these efforts with gathering extensive factual information within the intake. In addition, unlike in the MET protocols where specific client-centered feedback and decisional balance activities were prescribed (22), these interventions were not required in MIA.
The ITRS also displayed sufficient sensitivity to detect differences in MI performance across treatment conditions, therapists and sites. Consistent with our hypotheses, MIA therapists displayed greater MI adherence and competence than CAU therapists for fundamental and advanced MI skills, lending support for the ability of the ITRS to discriminate among key MI skill sets. Discriminability is especially important in settings that integrate MI into an assessment procedure, which has many components potentially antithetical to MI (e.g., closed questions, education/advice). This finding also suggests that community program therapists can learn to deliver MI with integrity within an intake session when provided with sufficient training and supervision and a clear model about how to integrate MI with other treatments, a challenging task for many therapists as they learn MI (23). Treatment condition accounted for a substantially larger amount of variance in adherence and competence than non-specific effects of therapists within conditions or the sites in which they worked. This finding suggests that the model of training and supervision used in the original protocol holds promise as a means to prepare community program clinicians in MI and supports the use of NIDA-SAMHSA Blending Initiative MI supervision toolkit that is based on this model for developing adherent and competent MI practice in the United States (10).
Therapists with greater fundamental and advanced MI adherence and competence were significantly more likely to have clients who expressed in-session increases in their motivation to reduce or stop substance use. This finding was more pronounced in the MIA condition, and is consistent with previous findings that therapists who received feedback and coaching after MI workshop training had significantly better client responses in their sessions (24), as well as with recent research that has shown how therapists' use of strategies consistent with MI predicted more client motivation for change (25). It is also possible that therapists were able to use MI with greater adherence and competence as their clients expressed more motivation for change rather than arguing against it, a likely easier circumstance in which to demonstrate skill (26, 27). Disentangling the sequential nature of therapists' MI integrity and their clients' in-session responses and treatment outcomes is an important area of future research (22).
Contrary to the English MET trial (8), there were no significant relationships between MI adherence/competence and treatment outcome as measured by patient retention in treatment or days abstinent from substance use at any of the follow-up points, despite the levels of discrimination achieved between therapists trained in MI and those in CAU. This lack of association might be due to the generally good treatment outcomes achieved by clients in both conditions (2), limiting the range in which associations might be established. These findings, nonetheless, raise the question of what levels of MI adherence and competence are needed to significantly influence clients' treatment outcomes. Possibly, therapists did not use strategies that elicit and support the clients' motivations for change (i.e., advanced MI skills factor) sufficiently in the assessment intake, a presumed mediator in MI (5), and that more use of these strategies might have resulted in the expected relationships between MI integrity and treatment outcomes. More research is needed to isolate the active ingredients of MI and to establish empirically derived training criteria or benchmarks linked to client outcomes.
One limitation of the study is the potential nonrepresentativeness of CTN-affiliated programs and therapists (e.g., may be more open to using empirically supported treatments) (28). In addition, the extent to which community program supervisors can use the ITRS with levels of reliability and validity similar to highly trained independent raters is unknown, though an initial evaluation of rating correspondence between these two groups suggests promise in this area (29). Nonetheless, this study replicated most of the previously reported psychometric properties of the ITRS and implies that the measure is robust across diverse samples of programs, therapists, and applications of MI. The ability of the ITRS to differentiate the practice of community therapists trained in MI from those who did not receive this training in three CTN trials (2, 22, 30) lends support for the use of the scale to supervise therapists in MI.
ACKNOWLEDGMENTS
National Institute on Drug Abuse grants (P50-DA09241, U10 DA13038, DA1025273, K05-DA00457 and K05-DA00089) supported this study. The authors are grateful to the individual who helped develop the rating manual (Joanne Corvino, MSW, Jon Morgenstern, Ph.D.), the MET/MI expert trainers and supervisors (Ken Bachrach, Ph.D., Jacqueline DeCarlo, Chris Farentinos, M.D., Melodie Keen, M.A., L.M.F.T., Terence McSherry, M.S.P.A., Jeanne Obert, M.S., Doug Polcin, Ed.D., Ned Snead, M.S., Richard Sockriter, M.S., M.B.A., Deborah Van Horn, Ph.D., Paulen Wrigley, R.N., M.S., Lucy Zammarelli, M.A, and Charlotte Chapman, Ph.D.) who provided valuable feedback while piloting the rating items, the independent tape raters (Luis Anez Nava, Psy.D., Theresa Babuscio, B.A., Declan Barry, Ph.D., Natalie Dumont, MSW, Lynn Ferrucci, M.A., Francis Giannini, LCSW, Rachel Hart, M.A., Karen Hunkele, B.A., Susan Kasserman, RN, M.Div., Brian Kiluk, B.A., Demetrios Kostas, LCSW, MBA, Mark Lawless, LCSW, Manuel Paris, Psy.D., Jane Stanton, LCSW, and Mary Ann Vail, LCSW) whose ratings provided the basis for this study's findings, and Julie Matthews, B.A. and Monica Canning-Ball, BA who administratively managed the tape rating project. The ITRS scale and rating manual are available from Dr. Martino.
Footnotes
Declaration of Interest The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article and the views they have expressed do not necessarily represent the official views of National Institute on Drug Abuse.
Nonsignificant (p > .05) chi-square goodness of fit index, a χ2/degrees of freedom ratio < 2, normed fit index (NFI), incremental fit index (IFI), and comparative fit index (CFI) > .9, and root mean square error of approximation (RMSEA) <.05. Because in larger models (n > 200), the chi-square test usually is significant and often detects trivial differences between sample covariance and fitted covariance matrices (20), we relied on the preponderance of evidence from the other indices in determining the best fitting model.
REFERENCES
- 1.Ball SA, Martino S, Corvino J, Morganstern J, Carroll KM. Independent tape rater guide. Unpublished psychotherapy tape rating manual. 2002 [Google Scholar]
- 2.Carroll KM, Ball SA, Nich C, Martino S, Frankforter TL, Farentinos C, Kunkel L, Mikulich-Gilbertson S, Morgenstern J, Obert JL, Polcin D, Snead N, Woody GE. Motivational interviewing to improve treatment engagement and outcome in individuals seeking treatment for substance abuse: A multisite effectiveness study. Drug Alcohol Depend. 2006;81:301–312. doi: 10.1016/j.drugalcdep.2005.08.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Miller WR, Rollnick S. Motivational Interviewing: Preparing People for Change. 2nd ed Guilford Press; New York: 2002. [Google Scholar]
- 4.Lundahl BW, Kunz C, Brownell C, Tollefson D, Burke B. Meta-analysis of motivational interviewing: Twenty Five years of empirical studies. Res Soc Work Pract. 2010;20:137–160. [Google Scholar]
- 5.Miller WR, Rose GS. Toward a theory of motivational interviewing. Am Psychol. 2009;64:527–537. doi: 10.1037/a0016830. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Miller WR, Sorensen JL, Selzer JA, Bringham GS. Disseminating evidence-based practices in substance abuse treatment: A review with suggestions. J Subst Abuse Treat. 2006;31:25–39. doi: 10.1016/j.jsat.2006.03.005. [DOI] [PubMed] [Google Scholar]
- 7.Waltz J, Addis ME, Koerner K, Jacobson NS. Testing the integrity of a psychotherapy protocol: assessment of adherence and competence. J Consult Clin Psychol. 1993;61:620–630. doi: 10.1037//0022-006x.61.4.620. [DOI] [PubMed] [Google Scholar]
- 8.Martino S, Ball SA, Ball SA, Nich C, Frankforter TL, Carroll KM. Community program therapist adherence and competence in motivational enhancement therapy. Drug Alcohol Depend. 2008;97:37–48. doi: 10.1016/j.drugalcdep.2008.01.020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Santa Ana EJ, Carroll KM, Anez L, Paris M, Jr, Ball SA, Nich C, Frankforter TL, Suarez-Morales L, Szapocznik J, Martino S. Evaluating motivational enhancement therapy adherence and competence among Spanish-speaking therapists. Drug Alcohol Depend. 2009;103(1–2):44–51. doi: 10.1016/j.drugalcdep.2009.03.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Condon TP, Miner LL, Balmer CW, Pintello D. Blending addiction research and price: Strategies for technology transfer. J Subst Abuse Treat. 2008;35:156–160. doi: 10.1016/j.jsat.2007.09.004. [DOI] [PubMed] [Google Scholar]
- 11.Martino S, Ball SA, Gallon SL, Hall D, Garcia M, Ceperich S, Farentinos C, Hamilton J, Hausotter W. Motivational interviewing assessment: Supervisory tools for enhancing proficiency. Northwest Frontier Addiction Technology Transfer Center, Oregon Health and Science University; Salem, OR: 2006. [Last accessed on May 24, 2010]. Available at http:/www/attcnetwork.org/explore/priorityareas/science/blendingintiative/index.asp. [Google Scholar]
- 12.Fals-Stewart W, O'Farrell TJ, Freitas TT, McFarlin SK, Rutigliano P. The timeline followback reports of psychoactive substance use by drug-abusing patients: Psychometric properties. J Consult Clin Psychol. 2000;68:134–144. doi: 10.1037//0022-006x.68.1.134. [DOI] [PubMed] [Google Scholar]
- 13.Sobell LC, Sobell MB. Timeline followback: A technique for assessing self-reported alcohol consumption. In: Litten RZ, Allen J, editors. Measuring Alcohol Consumption: Psychosocial and Biological Methods. Humana Press; New Jersey: 1992. pp. 41–72. [Google Scholar]
- 14.Santa Ana EJ, Martino S, Ball S, Nich C, Frankforter T, Carroll KM. What is usual about “treatment-as-usual”? Data from two multisite effectiveness trials. J Subst Abuse Treat. 2008;35:369–379. doi: 10.1016/j.jsat.2008.01.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychol Bull. 1979;86:420–429. doi: 10.1037//0033-2909.86.2.420. [DOI] [PubMed] [Google Scholar]
- 16.Arbuckle JL. AMOS. Version 6.0. SPSS; Chicago, IL: 2005. Computer Program. [Google Scholar]
- 17.Kline RB. Principles and Practice of Structural Equation Modeling. Guilford Press; New York: 1998. [Google Scholar]
- 18.Marsh HW, Balla JR, McDonald RP. Goodness-of-t indexes in confirmatory factor analysis: The effect of sample size. Psychol Bull. 1988;103:391–410. [Google Scholar]
- 19.Yadama GN, Pandey S. Effect of sample size on goodness-of-t indices in structural equation models. J Soc Serv Res. 1995;20:49–70. [Google Scholar]
- 20.Hu LT, Bentler PM. Evaluating model fit. In: Hoyle RH, editor. Structural Equation Modelling: Concepts, Issues and Applications. Sage; Thousand Oaks, CA: 1995. pp. 77–99. [Google Scholar]
- 21.Harris RJ. A Primer of Multivariate Statistics. 2nd edition Academic Press; New York: 1985. [Google Scholar]
- 22.Ball SA, Martino S, Nich C, Frankforter TL, Van Horn D, Crits-Christoph P, Woody GE, Obert JL, Farentinos C, Carroll KM. Site matters: Motivational enhancement therapy in community drug abuse clinics. J Consult Clin Psychol. 2007;75:556–567. doi: 10.1037/0022-006X.75.4.556. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Miller WR, Moyers TB. Eight stages in learning motivational interviewing. J Teach Addict. 2006;5:3–17. [Google Scholar]
- 24.Miller WR, Yahne CE, Moyers TE, Martinez J, Pirritano M. A randomized trial of methods to help clinicians learn motivational interviewing. J Consult Clin Psychol. 2004;72:1050–1062. doi: 10.1037/0022-006X.72.6.1050. [DOI] [PubMed] [Google Scholar]
- 25.Moyers TB, Martin T, Houck JM, Christopher PJ, Tonigan JS. From in-session behaviors to drinking outcomes: A causal chain for motivational interviewing. J Consult Clin Psychol. 2009;77:1113–1124. doi: 10.1037/a0017189. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Gaume J, Gmel G, Faouzi M, Daeppen J. Counselor skill inuences outcomes of brief motivation interventions. J Subst Abuse Treat. 2009;37:151–159. doi: 10.1016/j.jsat.2008.12.001. [DOI] [PubMed] [Google Scholar]
- 27.Thyrian JR, Freyer-Adam J, Hannöver W, Röske K, Mentzel F, Kufeld C, Bischof G, Rumpf HJ, John U, Hapke U. Adherence to the principles of motivational interviewing, clients' characteristics and behavior outcome in smoking cessation and relapse prevention in women postpartum. Addict Beh. 2007;32:2297–2303. doi: 10.1016/j.addbeh.2007.01.024. [DOI] [PubMed] [Google Scholar]
- 28.Roman PM, Ducharme LJ, Knudsen HK. Patterns of organization and management in private and public substance abuse treatment programs. J Subst. Abuse Treat. 2006;31:235–243. doi: 10.1016/j.jsat.2006.06.017. [DOI] [PubMed] [Google Scholar]
- 29.Martino S, Ball SA, Nich C, Frankforter TL, Carroll KM. Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychoth Res. 2009;19:181–193. doi: 10.1080/10503300802688460. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Carroll KM, Martino S, Suarez-Morales L, Ball SA, Miller WR, Añez L, Paris M, Nich C, Frankforter TL, Matthews K, Farentinos C, Szapocznik J. A multisite randomized effectiveness trial of motivational enhancement therapy for Spanish-speaking substance users. J Consult Clin Psychol. 2009;77:993–999. doi: 10.1037/a0016489. [DOI] [PMC free article] [PubMed] [Google Scholar]