Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2013 Jun 1.
Published in final edited form as: J Subst Abuse Treat. 2011 Nov 25;42(4):429–437. doi: 10.1016/j.jsat.2011.09.006

Therapist Competence and Treatment Adherence for a Brief Intervention addressing Alcohol and Violence among Adolescents

Stella M Resko 1, Maureen A Walton 2, Stephen T Chermack 2,3, Frederic C Blow 2,3, Rebecca M Cunningham 4,5,6
PMCID: PMC3290756  NIHMSID: NIHMS329561  PMID: 22119182

Abstract

This study examines therapist competency and treatment adherence for a brief intervention addressing alcohol misuse and violent behaviors among adolescents aged 14–18. Three observational measures of fidelity were used by independent raters to evaluate 60 therapist-delivered sessions (m=32.5 minutes). Individual items from the Content Adherence scale, the Global Rating of Competence (GROMIT) and Self Exploration and Change Talk (SECT) demonstrated fair to excellent inter-rater reliability (intraclass correlations ranged from .40 to 1.0). Principal components analysis (PCA) was utilized to identify the underlying factor structure of the Content Adherence and GROMIT. Parallel analysis suggested the extraction of three components for the Content Adherence reflecting the three distinct goals for each segment of the intervention. Two components were identified for the GROMIT representing the general spirit of motivational interviewing and empowerment. Findings provide support for the fidelity instruments adapted for this study and offer direction for future training and clinical supervision.

Keywords: treatment fidelity, treatment integrity, implementation fidelity, therapist competence, motivational interviewing

With the growing emphasis on disseminating evidence-based treatments in the substance abuse field, it is critical that clinicians and researchers assess the fidelity of behavioral interventions (Madson & Campbell, 2006; Mowbray et al, 2003). Treatment fidelity refers to the extent to which an intervention is implemented as intended and is critical for the successful translation of evidence based treatments into practice (Baer et al., 2007; Breitenstein et. al, 2010; Carroll et al 2007; Mihalic 2004; Mowbray et al., 2003; Perepletchikova, Treat, & Kazdin, 2007). It is an essential, yet often overlooked, component of intervention research. Perepletchikova and Kazdin (2005), for example, reviewed psychiatry and clinical psychology journals that frequently publish treatment outcome research (ie. ≥100 articles on treatment outcomes research from 2000–2004) and found that treatment fidelity was adequately addressed for only 3.5% of the evaluated psychosocial interventions.

Treatment adherence and therapist competence are two key components of treatment fidelity that are conceptually related (Perepletchikova, & Kazdin, 2005; Perepletchikova, Treat, & Kazdin, 2007). Treatment adherence is the degree to which the therapist implements procedures prescribed by the intervention protocol and avoids those that are proscribed (Hogue et al., 2008; Perepletchikova, Treat, & Kazdin, 2007). Adherence measures focus on the presence or quantity of components that are specific and essential to the defined intervention (Breitenstein et al., 2010). In contrast, therapist competence addresses qualitative aspects of the therapists’ skills and assesses how well prescribed procedures are implemented (Breitenstein et al., 2010; Forgatch et al 2005; Perepletchikova and Kazdin 2005; Perepletchikova, Treat, & Kazdin, 2007; Stein, Sargent, & Raphaels, 2007). Competence in delivering an intervention includes qualities related to communication, technical abilities, and skills in responding to the participants receiving the intervention (Breitenstein et al., 2010).

Monitoring treatment fidelity is important for implementing interventions consistently and at a high level of quality throughout the course of a study, particularly when different therapists with different levels of expertise are implementing the intervention in multiple settings (Breitenstein et al., 2010; Carroll et al., 2000; Glasgow, Lichtenstein & Marcus, 2003). In cases of negative or ambiguous findings, fidelity data can provide insight that helps researchers determine whether the outcomes reflect a failure of the therapeutic model or a failure to implement the model (Chen, 1990). Fidelity monitoring can provide additional support for the validity of results in a study and inform the wider dissemination of empirically based treatments by providing guidelines for larger scale dissemination of interventions (Breitenstein et al., 2010; Baer, et al., 2007; Mowbray, et al., 2003).

Research on treatment fidelity can also contribute to our understanding of the mechanisms of behavior change, ways interventions can be improved, and strategies to overcome barriers to implementation (Madson and Campbell, 2006; Moyers et al., 2005; Orwin, 2000; Perepletchikova and Kazdin, 2005). Fidelity assessments provide a method for determining critical ingredients in treatment models that are most strongly associated with client outcomes. To this end, it is critical that researchers develop rigorous tools that will yield reliable and valid data on treatment fidelity and can also assist in clinical supervision (Carroll et al., 2000; Hogue et al., 2008; Garland, Hurlburt, & Hawley, 2006; Madson & Campbell, 2006; Moyers et al., 2005). Reliable and valid rating scales become essential when attempting to standardize treatments across multiple treatment sites. Carefully developed measures of fidelity may help researchers to determine whether particular components are more strongly associated with a desired outcome and can help to identify promising leads for research on mechanisms of behavior change (Perepletchikova and Kazdin, 2005).

The purpose of the present study is to examine the treatment fidelity for a brief intervention (BI) addressing alcohol use and violent behaviors among adolescents. The combined adapted motivational interviewing and skills training intervention was recently tested in a larger randomized controlled trial (See Cunningham et al., 2010; Walton et al 2010). Using a subset of data drawn from this larger study, the therapists’ implementation of the intervention was evaluated for two aspects of treatment fidelity. These are the degree to which an intervention is conducted competently (competence) and according to protocol (adherence) (Breitenstein et al., 2010; Carroll et al 2007; Dusenbury Brannigan Hansen Walsh & Falco, 2005; Perepletchikova and Kazdin, 2005; Perepletchikova, Treat, & Kazdin, 2007).

Treatment fidelity and the quality with which an intervention is implemented has been a particular concern for motivational interviewing (MI) interventions (Madson Campbell, Barrett, Brondino, and Melchert; 2005; Madson & Campbell, 2006; Martino et al., 2008; Moyers et al., 2005). Several researchers have cautioned that MI is sometimes implemented in a manner that is inconsistent with the intended spirit of the approach (e.g. Madson et al 2005; Miller & Rollnick, 2002; Moyers, Martin, Catley, Harris & Ahluwalia, 2003). Although there has been a growing number of studies that have empirically examined the treatment fidelity of MI interventions (Madson & Campbell, 2006; Moyers et al., 2005), there has been less attention to MI-based interventions with adolescents. Those studies that have examined treatment fidelity of MI interventions with adolescents have focused primarily on treatment adherence (e.g. Dennis et al., 2002; Dennis et al., 2004; Smith et al., 2009) or client language within MI sessions (Baer et al., 2008). This study contributes to the small but growing body of research assessing treatment fidelity in evidence-based treatments for adolescent substance misuse. In addition, this article presents novel data as it is the first published data reporting on psychometric properties of the two instruments: the Global Rating of Motivational Interviewing Therapist (GROMIT) (Moyers, 2004) and the Self Exploration and Change Talk (SECT).

2. Methods

2.1 Participants

Data for this study were drawn from a larger and ongoing randomized controlled trial designed to compare the effectiveness of different delivery mechanisms for a tailored brief intervention addressing alcohol use and violence among adolescents (See Cunningham et al., 2010; Walton et al 2010;). Participants were recruited during the afternoon and evening shifts (12-11pm) from a Level 1 Trauma Center located in Flint, Michigan. Patients who endorsed past year alcohol use (e.g., drinking beer, wine or liquor more than 2 times in the past year) and engaging in one or more violent behavior (e.g., pushed/shoved, hit/punched, serious physical fight, group fighting, using a knife or gun) in the past year were eligible for the study. Patients were excluded if they had unstable vital signs, were actively suicidal, being treated for a sexual assault, in police custody, or unable to provide consent due to impaired cognitive function. All eligible and consenting participants were randomized into one of three treatment groups: a brief intervention (BI) administered by a therapist, a BI administered by a computer, and an informational pamphlet control condition. Consent for participation was obtained directly from those age 18 years and older. Parental consent and adolescent assent was obtained for participants age 17 years and younger. All procedures were approved and conducted in compliance with the University of Michigan’s and Hurley Medical Center’s Institutional Review Boards for Human Subjects guidelines. A Certificate of Confidentiality was obtained from NIH for this study.

For the current analyses, we focus on a randomly selected subset of 60 therapist delivered BI sessions (33.3% of therapy sessions conducted from September 2006 through December 2008). The random selection was stratified according to therapist and time (i.e. month) in order to account for differences between clinicians and to capture any seasonal variation throughout the course of the study (Perepletchikova & Kazdin, 2005). Each of the six therapists conducted between 2 and 20 of the sessions that were analyzed (mean = 10)1. On average, adolescents from these sessions were 16.7 years old (Range 14–18; SD=1.15) and 54.9% of participants were female. With regard to race, 51.6% of the adolescents self identified as African American, 41.8% identified as Caucasian and 6.3% percent identified as another race (e.g. Asian). Following the larger study inclusion criteria, all adolescents in this sample self-reported past year alcohol use and engaging in one or more violent behaviors in the past year.

2.2 Overview of Treatment

The adapted motivational interviewing (MI) and skills training intervention is an empirically-derived, brief intervention addressing alcohol misuse and violent behaviors among adolescents. The intervention is tailored to the participant’s gender and level of involvement in alcohol use, binge drinking, fighting and weapon carrying. The one session intervention was administered onsite in the ED by clinical research staff. Tablet computers were utilized during the therapist-led sessions to guide the process and provide feedback on the participant’s alcohol use and involvement with violence. Sessions lasted an average of 34.1 minutes (SD = 12.3 minutes) and were digitally recorded with the permission of the participants.2

The initial focus of the session was for the therapist to establish rapport with the participant and set the stage for an open discussion about their goals, values, and behaviors. Therapists encouraged the adolescent to reflect on those things that are important to them and how their current or future behaviors (i.e., alcohol use, fighting, and weapon carriage) might impact future goal attainment. Participants were then given feedback concerning their current drinking and involvement in violence in a non-confrontational, nonjudgmental manner consistent with motivational interviewing. This discussion attempted to enhance the participant’s awareness of the consequences of their involvement in these behaviors and encouraged the adolescent to think about reasons to change their alcohol use, fighting, and weapon carrying or to remain uninvolved. The final segment of the intervention used skill building scenarios and role plays. The messages in the scenarios were tailored to the adolescents’ gender, specific skill deficits (e.g., refusal skills for drinking, anger management, conflict resolution), risk factors (e.g., illicit drug use, weapon carriage, contextual factors), and experiences with alcohol and violence. The therapists drew on fundamental (e.g. open-ended questions, affirmations and reflections delivered with MI spirit) and more advanced MI skills (e.g. techniques for evoking change talk and handling client resistance) while avoiding techniques that were inconsistent with the spirit of MI (e.g. direct confrontation or unsolicited advice) (Miller and Rollnick, 2002; Rollnick and Miller, 1995). Throughout the intervention, there was an emphasis on collaboration with participants to identify potential options that may be helpful with regard to their unique situation. The process of adolescents identifying and evaluating a menu of options was designed to increase a sense of personal responsibility and investment regarding choices and to enhance self-efficacy.

2.3 Fidelity Procedures

Efforts were made to ensure that training for the therapists was uniform in content; therapists hired after the trial began received the same training as those hired at the beginning of the project. Therapists were provided with a therapist manual and treatment fidelity manual that had been developed specifically for the project. The therapist manual included a detailed description of the intervention, screen shots, and sample scripts for each section of the intervention. The fidelity manual was provided to all therapists delivering the intervention as well as all independent raters that coded the audio-taped therapy sessions. The treatment fidelity manual and training included general information on treatment fidelity, scoring procedures for each item on the measures used, and specific examples of ratings.

The therapist training began with a 3-day (24 hours) training workshop that reviewed the rationale for the study, use of the ED setting, and the adapted motivational interviewing approach. During this training, therapists were taught to adapt therapeutic tasks and their responses to participants’ cultural background (See Hammond and Yung, 1993) and level of interest in change (See Monti el al., 2001). Therapists were instructed on a range of therapeutic strategies including goal setting, weighing the pros and cons of making behavioral changes, developing a discrepancy between the participant’s behaviors and their goals, reflective listening, avoiding argumentation and ‘rolling with resistance’, and assisting participants in identifying strategies to help achieve goals. The training workshop included audio and videotaped examples of intervention techniques and role playing.

Following an approach similar to Project MATCH (Carroll et al., 1998; Kadden et al., 1995), therapists were required to demonstrate proficiency via role play with practice sessions prior to being approved to meet with participants. These practice sessions were reviewed in detail by the clinical supervisor. Throughout the study, digital recordings of the therapy sessions were monitored to ensure acceptable adherence and competence levels. Therapists received regular supervision (initially weekly, then monthly) in group and one-on-one sessions to ensure accuracy of implementation, to reduce therapeutic drift, provide therapists with feedback, and clarify any questions pertaining to implementation of the intervention.

Each of the 60 BI sessions was rated by two independent coders. This allowed for comparisons of inter-rater agreement and helped avoid potential problems of over-reporting with self ratings of fidelity (Carroll et al., 2000; Santacroce, Maccarelli, and Reid, 2004). The independent raters included one African American female and one white female, both of whom had completed doctoral degrees (in either psychology or social work). The raters coded each session in their entirety during a single uninterrupted time slot in order to maintain a sense of the overall gestalt of the session. Raters were assigned tapes from this study’s data pool only after completion of training in the fidelity procedures and group practices where each rater demonstrated acceptable levels of reliability (Moyers, et al., 2008).

2.3 Measure Description

Treatment fidelity was examined using three observational measures. The Content Adherence scale consists of 15 items focusing on faithfulness to content. Although this scale was developed specifically for use on this study, it builds on the Yale Adherence and Competence Scale developed by Carroll and colleagues (2000). Items on the Content Adherence scale focus on the extent to which specific elements of the intervention (e.g. setting the agenda and reviewing goals) are covered or not covered. The Global Rating of Motivational Interviewing Therapist (GROMIT) is a 16 item tool that broadly measures therapist skill, responsiveness, and overall competence (Moyers, 2004). The GROMIT evaluates 15 elements of motivational interviewing that either should or should not be present when delivering the brief intervention. Each item is rated on a 7-point Likert scale ranging from 1 “Very Poor” to 7 “Expert Mastery” after listening to the session one time (Moyers, 2004). The Self Exploration and Change Talk (SECT) is a 5 item tool adapted for this study from the Motivational Interviewing Skill Code’s (Miller & Mount, 2001) global and self exploration items. These items measure the clients’ use of change talk and their commitment to avoiding alcohol and violent behaviors and are rated on a 7-point Likert scale ranging from 1 “Not At All” to 7 “Extensively” for the frequency item and 1 “Unwilling to Consider Change” to 7 “Highly Committed to Change” for the commitment items. The SECT also includes an item that provides a global rating of the participant’s behavior that was scored with ratings that ranged from 1 “No personally relevant material is revealed or discussed by the client during the session” to 7 “The client engages in active interpersonal exploration, openly exploring values, feelings, relationships, fears, turmoil, life choices and perceptions of others. Clients experience a shift in perception.”

2.4 Statistical Analyses

To assess inter-rater agreement, intraclass correlations (ICC) were computed to compare the scores of the two raters on each of the items on the Content Adherence, GROMIT, and SECT measures. Means and standard deviations were calculated for each item on the three measures and one-way ANOVAs were performed to assess the variability in scores that is attributable to the therapist. Principal Components Analysis (PCA) with varimax rotation was then used to identify the underlying data structure of the Content Adherence and GROMIT measures. The number of components to be retained was determined by the parallel analysis procedure (Horn, 1965)3. Parallel analysis is a simulation technique that is considered one of the more robust and statistically valid approaches for determining the number of components to retain in principal components analysis (Fabrigar et al., 1999; Hayton, Allen and Scarpello, 2004; Lance, Butts, and Michels, 2006; Patil, McPherson, and Freisner, 2010; Velicer et al., 2000; Zwick and Velicer, 1986). Observed eigenvalues extracted from the correlation matrix for a data set are compared with the average eigenvalues obtained from a simulated data set with the same sample size and number of variables as the original data set (Hayton, Allen and Scarpello, 2004). Components with eigenvalues in the original data set that are greater than those obtained for the simulated data should be retained (Zwick and Velicer, 1986; Hayton, Allen and Scarpello, 2004). Finally, correlations were calculated between the SECT summary scores (measuring the adolescents’ use of change talk and their commitment to avoiding alcohol and violent behaviors) and the components of the Content Adherence and GROMIT that were identified through Principal Components Analysis.

3. Results

Content Adherence

As shown in Table 1, items from the Content Adherence scale exhibited fair to excellent interrater reliability (ICC’s ranged from .56 to .87) with the exception of one item (“Summarized Session” ICC = .12). Means for each individual item on the Content Adherence scale were calculated using the ratings from the two independent raters (120 ratings based on 60 BI sessions) and overall clinicians adhered to the brief intervention. Specific areas in which there were high levels of treatment adherence (mean >= 4.0) were identifying goals (item 1), reviewing alcohol use, norms and consequences (item 2), reviewing fighting, norms and consequences (item 3), and reviewing the key message of scenario one (item 8). Areas in which there were lower levels of adherence to the intervention (mean < 3.8) were the use of the key message for the final scenario (item 11), summarizing the session (item 12), identifying appropriate resources (item 13) and identifying one next step in avoiding drinking, fighting or weapon carriage (item 14). Results of the one-way ANOVA indicate that Content Adherence scores differed significantly across therapists, F (5, 115) = 5.65, p < 0.001, r2=0.19.

Table 1.

Mean and Intraclass Correlation (ICC) scores for each item

Content Adherence
Mean (SD) Intra Class Correlation
Identified goals 4.58 (0.81) 0.70
Alcohol use, norms and consequences 4.39 (0.98) 0.73
Fighting, norms and consequences 4.24 (0.90) 0.80
Reasons to Stay Away from Drinking 3.91 (2.03) 0.87
Reasons to Stay Away from Fighting 3.84 (1.94) 0.77
Introduced Scenarios 0.81 (0.25) 1.00
Key Message of Scenario 1 4.03 (1.97) 0.75
Key Message of Scenario 2 3.99 (1.94) 0.67
Key Message of Scenario 3 3.81 (1.96) 0.73
Key Message of Scenario 4 3.73 (2.28) 0.67
Key Message of Scenario 5 3.17 (3.00) 0.82
Summarized Session 2.61 (3.96) 0.83
Identified Appropriate Resources 1.91 (4.54) 0.91
Identified one next step 1.16 (4.13) 0.90
Global Rating of Competence (GROMIT)
Mean (SD) Intra Class Correlation
Provided feedback in objective fashion 4.64 (0.89) 0.69
Avoided power struggles 5.07 (0.76) 0.74
Understanding of client’s perspective 4.85 (0.85) 0.65
Confidence in client’s ability to make changes 4.70 (0.95) 0.82
Guided the client toward change talk 4.37 (0.98) 0.78
Expressed interest in client’s goals and values 4.86 (0.83) 0.77
Did not attempt to persuade client 4.72 (0.82) 0.65
Did not argue with client 5.16 (0.73) 0.79
Did not assume expert role 4.60 (0.91) 0.72
Encouraged client to contribute ideas: alcohol 4.49 (0.95) 0.81
Encouraged client to contribute ideas: violence 4.57 (1.04) 0.67
Expressed approval of the client 5.06 (0.80) 0.52
Directed attention toward client’s strengths 4.47 (0.77) 0.41
Seemed genuine 4.98 (0.83) 0.60
Recognized and responded to change talk 4.41 (0.96) 0.71
Did not steam roll the client 4.89 (0.97) 0.64
Self Exploration and Change Talk (SECT)
Mean (SD) Intra Class Correlation
Use of Change Talk 4.03 (0.89) 0.57
Commitment to Avoid Drinking 4.59 (1.06) 0.87
Commitment to Avoid Fighting 4.69 (1.06) 0.83
Commitment to Avoid Weapon Carrying 4.34 (1.07) 0.85
Overall Client’s Behavior 4.81 (1.12) 0.63
Open in a new tab

Principal Components Analysis with varimax rotation was utilized to identify the underlying data structure of the Content Adherence scale. The critical cut values identified through parallel analysis (Horn, 1965) suggested the extraction of three components (parallel analysis plots are available from authors upon request). The first principal component explained 38.7% of the variance, the second for 19.4%, and the third for 12.4% (total variance explained is 70.5%). Component 1 represented the beginning of the BI which focused in reviewing their personal feedback (e.g., goals, behaviors, norms). Component 2 represented the middle of the session which focused on a decisional balance exercise and a ‘menu’ of options for addressing future risk situations. Component 3 focused on a final summary, community resource options, and elicited commitment talk for a next step.

GROMIT

Individual items from the GROMIT demonstrated fair to excellent inter-rater reliability (GROMIT ICC’s mean = 0.69 and range from .42 to .89) (See Table 1). The means for individual items on the GROMIT indicated that the clinicians were generally competent in delivering the brief interventions (mean scores range from 4.37 to 5.16). Specific areas in which there was high competency (μ > 4.9) were avoiding power struggles (item 2), not arguing with the client (item 8), expressing approval of the client (item 12), and seeming genuine (item 14). Results of the one-way ANOVA indicate that overall GROMIT scores differed significantly across therapists, F (5, 115) = 9.77, p < 0.001, r2=0.33.

Principal components analysis was utilized to identify the factor structure of the GROMIT and determine whether items loaded on a single dimension, or multiple dimensions. Parallel analysis suggested retention of two components whose actual eigenvalues are above the line representing the randomly generated eigenvalues (parallel analysis plots are available from authors upon request). The first principal component explained 31.7% of the variance and the second explained 30.4% (total variance explained is 62.1%). For factor one, seven items loaded on the first factor (ie factor loadings >0.6 on the component and <0.40 on other two components). These items included: understanding the client’s point of view (item 3), showing confidence in the client’s ability to make change (item 4), expressing interest in the client’s values and goals (item 6), actively encouraged client to contribute ideas (items 10 and 11), and expressed approval of the client (item 15). Six items loaded on factor two: providing objective feedback (item 1), avoiding power struggles (item 2), not persuading the client of the seriousness of the problem (item 7), therapist did not assume the expert role (item 9), therapist seemed genuine (item 15), therapist did not steamroll client (item 16). Conceptually, factor one represents an empathic counseling style consistent with the MI spirit while factor two represents empowerment and the therapist’s ability to negotiate power issues.

SECT

As shown in Table 1, individual items from the SECT demonstrated fair to excellent inter-rater reliability4 (SECT ICC’s ranged from .40 to .93). Means for each individual item on the SECT were calculated using the ratings from the two independent raters (120 ratings based on 60 BI sessions). As indicated by mean scores from the individual SECT items, adolescents in this sample generally used moderate amounts of change talk throughout the brief intervention sessions (μ= 4.03 and sd = 0.89) and indicated some commitment to avoiding drinking, fighting and weapon carriage (μ = 4.59, 4.69, and 4.34 and sd = 1.06, 1.06, and 1.07, respectively). Results of the one-way ANOVA indicate that SECT scores were not significantly different across therapists, F (5, 115) = 1.02, p < 0.41.

Scores for the SECT, which measured the adolescents’ use of change talk and their commitment to avoiding alcohol and violent behaviors, were positively correlated with the two principal components identified within the GROMIT (r = 0.70 and r = 0.68 respectively). These two factors represented an empathic counseling style consistent with the MI spirit and empowerment and the therapist’s ability to negotiate power issues. The SECT also correlated positively with the three components of the Content Adherence scale (r = 0.67, r = 0.65, and r = 0.57 respectively). These components represented the three segments of the intervention including the beginning of the BI where personalized feedback was reviewed, the middle of the session which focused on a decisional balance exercise and a ‘menu’ of options for addressing future risk situations and the final section which included a summary of the session, a discussion of community resource options, and elicited commitment talk for a next step.

4. Discussion

With the growing emphasis on disseminating evidence-based treatments in the substance abuse field, it is critical that clinicians and researchers use reliable measures to assess the fidelity of behavioral interventions (Madson & Campbell, 2006; Mowbray et al, 2003). Although many researchers utilize scales to rate adherence to the clinical process, empirical validation for many of these scales is lacking (Allsop, 2007; Baer et al., 2007). This study contributes to the evidence-base for this brief motivational interviewing intervention and advances the empirical base for the therapist competence and adherence scales. The current findings provide evidence that one can reliably measure therapist behavior for MI-based interventions addressing multiple risk behaviors (i.e. alcohol and violence) in ED settings. This is particularly noteworthy, given the challenges of a busy ED setting where outside noise (e.g. crying babies, patients moaning in pain, screaming, and overhead systems) and interruptions for medical care are common and have hampered other researcher’s efforts to assess fidelity for ED-based interventions (e.g. Monti et al., 2007). The use of tablet computers providing tailored prompts for the therapists throughout the session may have enhanced the levels of fidelity noted in this study. The portions of the intervention with lower levels of competence and adherence often did not have corresponding computer screens to prompt therapist behavior. Although further study is needed, this type of technology may be especially useful for interventions delivered in the ED and other busy settings.

Although the inter-rater reliability of the 5-item SECT was variable, the current findings provide preliminary support for the inter-rater reliability of the GROMIT and Content Adherence scale. Results from the one way ANOVAs indicated that while a significant proportion of the variability in GROMIT and Content Adherence scores was attributable to the therapist (r2=0.33 and r2=0.19 respectively), a large proportion of the variance in these scores was unexplained and resided within the error term of these models. The Content Adherence scale’s focus on coverage of specific elements of the intervention and the GROMIT’s focus on global measures of therapists’ skills, may contribute to the fact that a larger proportion of the variance in GROMIT scores was attributable to the therapist in comparison to the Content Adherence scores. Global ratings (e.g. GROMIT scores) may have a stronger relationship with a therapist’s ability in comparison to treatment adherence ratings (e.g. Content Adherence scores). Assessments of treatment fidelity and effectiveness of MI have been limited by a paucity of reliable, valid and practical assessment tools (see Madson and Campbell, 2006 for discussion) and the GROMIT may potentially be useful for monitoring treatment fidelity as well as clinical supervision. Although further research is needed, findings from the current study are promising given the user-friendly nature of the GROMIT which uses a one-pass rating system.5 While the Content Adherence scale is specific to the SafERteens intervention, it may provide a model for future studies that require project-specific ratings of treatment adherence.

Results from this study also highlight how fidelity measures can be used to assess strengths and weaknesses of the therapists’ performances. In the current sample, therapists’ performances tended to be weaker for items in the last segment of the intervention. This may result from therapist or participant fatigue, as well as the challenges specific to the ED setting (e.g., imminent discharge, fatigue due to noise and health issues of the participant, family or medical staff interruption). This may also reflect the lack of computer-based prompts for these sections of the intervention and highlight how technology may be used to improve treatment fidelity for interventions delivered in busy clinical settings. Further examination is warranted to determine whether these items that received lower fidelity scores are unnecessary to the intervention or appear to be viewed this way by the therapists. These findings highlight how the length of an ED-based intervention should be carefully considered, and how technology (ie computer-based prompts) can provide an additional safeguard to ensure the delivery of critical components of an intervention. Although patients frequently experience lengthy waits in the ED, our therapists reported in clinical supervision that they often felt rushed when dealing with tired participants, those in pain, and participants and/or parents who wanted to leave immediately following discharge.

Findings from this study also provide insight into the underlying factor structure of the Content Adherence and GROMIT. For the content adherence, three components were identified in the PCA reflecting the three distinct goals for each of the segment of the intervention. Future research should examine the extent to which each of these components moderates intervention effectiveness. Correlations between the three components of the Content Adherence scale and the SECT suggest there is a relationship between higher content adherence ratings and the level of change talk and commitment among adolescents in our sample. For the GROMIT, two components were identified in the PCA reflecting the general MI spirit and empowerment of the participant. The second component of the GROMIT may be indicative of how issues related to power between the therapist and client may have a particular salience given the developmental position of adolescents. Correlational analyses indicated that both of the components of the GROMIT were positively associated with change talk and a commitment to avoid alcohol and violence. Future studies are needed to assess these underlying components of the Content Adherence and GROMIT and the extent to which they are associated with in-session change talk and clinical outcomes. Studies with larger samples and using different analytic techniques can and should also be used to simultaneously examine patterns of fidelity between therapists across sessions.

This study has a number of limitations including: the use of a small sample size, and the setting, an emergency department visit. The nature of principal components analyses is inherently descriptive and sample bound. These limitations restrict causal inferences and indicate the need for replication in studies with larger samples and in other settings. Future research should further examine the utility of the factors identified in this study to determine the differential relationships with client outcomes and change talk. Because content adherence measures are, by definition, specific to the intervention being tested, the measure used in this study would require modification for use in other studies.

Despite these limitations, these findings have important implications and provide further evidence of the need to assess therapist competence and treatment adherence of behavioral interventions. Findings from this study support the reliability of the GROMIT in measuring fidelity for MI-based interventions. Measures of content adherence, by definition, should be specific to the particular intervention and may provide important information on which specific components of a brief intervention are actually delivered. Evidence from this study suggests that the GROMIT and Content Adherence scale can help identify strengths and weaknesses of therapists in clinical practice settings and research trials. Further refinement and study of the therapeutic process measures may increase understanding and refinement of the essential elements of brief interventions in order to increase the effectiveness of these approaches.

Table 2.

Factor Loadings for Factor Analysis and Principal Components Analysis

Content Adherence Principal Components Analysis (varimax rotation) Three Factor Solution
Factor 1 Factor 2 Factor 3
Identified goals −0.06 0.01 0.64a
Alcohol use, norms and consequences −0.01 0.13 0.82a
Fighting, norms and consequences 0.11 0.09 0.80a
Reasons to Stay Away from Drinking 0.91a 0.24 0.11
Reasons to Stay Away from Fighting 0.93a 0.18 .07
Introduced Scenarios 0.94a 0.17 −0.13
Key Message of Scenario 1 0.93a 0.19 −0.03
Key Message of Scenario 2 0.94a 0.19 −0.06
Key Message of Scenario 3 0.93a 0.17 0.02
Key Message of Scenario 4 0.79a 0.31 −0.09
Key Message of Scenario 5 0.47 0.68 −0.18
Summarized Session 0.24 0.89a 0.02
Identified Appropriate Resources 0.18 0.82a 0.11
Identified one next step 0.21 0.90a 0.07
GROMIT Principal Components Analysis (varimax rotation)Two Factor Solution
Factor 1 Factor 2
Provided feedback in objective fashion 0.30 0.78a
Avoided power struggles 0.30 0.80a
Understanding of client’s perspective 0.71a 0.27
Confidence in client’s ability to make changes 0.77a 0.16
Guided the client toward change talk 0.69 0.44
Expressed interest in client’s goals and values 0.75a 0.22
Did not attempt to persuade client 0.30 0.68a
Did not argue with client 0.40 0.75a
Did not assume expert role 0.16 0.85a
Encouraged client to contribute ideas: alcohol 0.76a 0.25
Encouraged client to contribute ideas: violence 0.71 0.39
Expressed approval of the client 0.70a 0.24
Directed attention toward client’s strengths 0.51a 0.23
Seemed genuine 0.40 0.72
Recognized and responded to change talk 0.65a 0.34
Did not steam roll the client 0.24 0.75a
Open in a new tab
a

Marker item for the factor (i.e. loading > 0.50 on the factor and < 0.35 on other factors.

1 ICC= Intra-class Correlation

Note: All items scored on a 7-point Likert type scale

Acknowledgments

This project was supported by National Institute on Alcohol Abuse and Alcoholism (NIAAA) grants 014889 and 07477. An earlier version of this work was presented at the 31st Scientific Meeting of the Research Society on Alcoholism, June 2008, Washington, D.C.

Footnotes

1

More specifically, the therapists conducted either 2, 4, 8, 12, 14, or 20 sessions analyzed as part of the current study.

2

All of the brief intervention sessions in the larger trial were recorded in order to limit the influence of measure reactivity and fluctuation in fidelity due to the presence or absence of an audio-recorder (Perpletchikova, Treat, and Kazdin, 2007).

3

As Zwick and Velicer (1986) note, the Kaiser criterion is more liberal and typically extracts more components relative to parallel analysis.

4

Cicchetti’s (1994) criteria for classifying the utility of ICC magnitudes are as follows: <.40 = poor, .40–.59 = fair, .60–.74 = good, .75–1.00 = excellent.

5

Rating systems such as the Motivational Interviewing Skill Code (Miller and Mount, 2001), for example, have shown to be reliable tools for measuring therapist competence, but also require that the rater listen to the session three times for three separate coding passes (Moyers, 2003; de Jonge, 2005).

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Baer JS, Ball SA, Campbell BK, Miele GM, Schoener EP, Tracy K. Training and Fidelity Monitoring of Behavioral Interventions in Multi-Site Addictions Research: A Review. Drug and Alcohol Dependence. 2007;87(2–3):107–118. doi: 10.1016/j.drugalcdep.2006.08.028. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Baer JS, Beadnell B, Garrett SB, Hartzler B, Wells EA, Peterson PL. Adolescent change language within a brief motivational intervention and substance use outcomes. Psychology of Addictive Behaviors. 2008;22(4):570–575. doi: 10.1037/a0013022. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Bentler PM. EQS Structural Equations Program Manual. Los Angeles: BMDP Statistical Software; 1989. [Google Scholar]
  4. Breitenstein SM, Gross D, Garvey CA, Hill C, Fogg L, Resnick B. Implementation fidelity in community based interventions. Research in Nursing & Health. 2010;33(2):164–173. doi: 10.1002/nur.20373. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Browne MW, Cudeck R. Alternative ways of assessing model fit. Sociological Methods and Research. 1992;21(2):230–258. [Google Scholar]
  6. Carroll KM, Connors GJ, Cooney NL, DiClemente CC, Donovan DM, Kadden RR, Zweben A. Internal validity of Project MATCH treatments: Discriminability and integrity. Journal of Consulting and Clinical Psychology. 1998;66(2):290–303. doi: 10.1037//0022-006x.66.2.290. [DOI] [PubMed] [Google Scholar]
  7. Carroll KM, Nich C, Sifry RL, Nuro KF, Frankforter TL, Ball SA, Rounsaville BJ. A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug and Alcohol Dependence. 2000;57(3):225–238. doi: 10.1016/s0376-8716(99)00049-6. [DOI] [PubMed] [Google Scholar]
  8. Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implementation Science. 2007;2:40. doi: 10.1186/1748-5908-2-40. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Chen HT. Theory-driven evaluations. Newbury Park, CA: Sage; 1990. [Google Scholar]
  10. Cunningham RM, Walton MA, Goldstein A, Chermack ST, Shope J, Bingham CR, Blow FC. Three-month follow-up of brief computerized and therapist interventions for alcohol and violence among teens. Academic Emergency Medicine. 2010;16(11):1193–1207. doi: 10.1111/j.1553-2712.2009.00513.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. de Jonge J, Schippers GM, Schapp CPDR. The Motivational Interviewing Skill Code: Reliability and a critical appraisal. Behavioural and Cognitive Psychotherapy. 2005;33(3):285–298. [Google Scholar]
  12. Dennis M, Godley S, Diamond G, Tims F, Babor T, Donaldson J, Funk R. The Cannabis Youth Treatment (CYT) study: main findings from two randomized trials. Journal of Substance Abuse Treatment. 2004;27(3):197–213. doi: 10.1016/j.jsat.2003.09.005. [DOI] [PubMed] [Google Scholar]
  13. Dennis M, Titus J, Diamond G, Donaldson J, Godley S, Tims F, Scott C. The cannabis youth treatment experiment: rationale, study design and analysis plans. Addiction. 2002;97(s1):16–34. doi: 10.1046/j.1360-0443.97.s01.2.x. [DOI] [PubMed] [Google Scholar]
  14. Dusenbury L, Brannigan R, Hansen WB, Walsh J, Falco M. Quality of implementation: developing measures crucial to understanding the diffusion of preventive interventions. Health Education Research. 2005;20(3):308–313. doi: 10.1093/her/cyg134. [DOI] [PubMed] [Google Scholar]
  15. Forgatch MS, Patterson GR, Degarmo DS. Evaluating fidelity: predictive validity for a measure of competent adherence to the Oregon model of parent management training. Behavior Therapy. 2005;36(1):3–13. doi: 10.1016/s0005-7894(05)80049-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Garland AF, Hurlburt MS, Hawley KM. Examining psychotherapy processes in a services research context. Clinical Psychology: Science and Practice. 2006;13(1):30–46. [Google Scholar]
  17. Glasgow RE, Lichtenstein E, Marcus AC. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. American Journal of Public Health. 2003;93(8):1261–1267. doi: 10.2105/ajph.93.8.1261. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Hammond WR, Yung B. Psychology’s role in the public health response to assaultive violence among young African-American men. American Psychologist. 1993;48(2):142–154. doi: 10.1037//0003-066x.48.2.142. [DOI] [PubMed] [Google Scholar]
  19. Hogue A, Dauber S, Chinchilla P, Fried A, Henderson C, Inclan J, Liddle HA. Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment. 2008;35(2):137–147. doi: 10.1016/j.jsat.2007.09.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling. 1999;6(1):1–55. [Google Scholar]
  21. Kadden R, Carroll K, Donovan D, Cooney N, Monti P, Abrams D, Hester R. Project MATCH Monograph Series 3, NIH Pub #94-3724. NIAAA; 1995. Cognitive Behavioral-Coping Skills Therapy Manual. [Google Scholar]
  22. Lance CE, Butts MM, Michels LC. The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods. 2006;9(2):202–220. [Google Scholar]
  23. Madson MB, Campbell TC. Measures of fidelity in motivational enhancement: A systematic review. Journal of Substance Abuse Treatment. 2006;31(1):67–73. doi: 10.1016/j.jsat.2006.03.010. [DOI] [PubMed] [Google Scholar]
  24. Madson MB, Campbell TC, Barrett DE, Brondino MJ, Melchert TP. Development of the Motivational Interviewing Supervision and Training Scale. Psychology of Addictive Behaviors. 2005;19(3):303–210. doi: 10.1037/0893-164X.19.3.303. [DOI] [PubMed] [Google Scholar]
  25. Martino S, Ball SA, Nich C, Frankforter T, Carroll KM. Community program therapist adherence and competence in motivational enhancement therapy. Drug and Alcohol Dependence. 2008;96(1–2):37–48. doi: 10.1016/j.drugalcdep.2008.01.020. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Mihalic S. The importance of implementation fidelity. Emotional and Behavioral Disorders in Youth. 2004;4(4):83–105. [Google Scholar]
  27. Miller WR, Mount KA. A small study of training in motivational interviewing: Does one workshop change clinician and client behavior? Behavioural and Cognitive Psychotherapy. 2001;29(4):457–471. [Google Scholar]
  28. Miller WR, Rollnick S. Motivational Interviewing: Preparing People to Change. The Guilford Press; New York: 2002. [Google Scholar]
  29. Miller WR, Moyers TB, Ernst D, Amrhein P. Manual for the Motivational Interviewing Skill Coding (MISC) Version 2.1. 2008 Retrieved from: http://casaa.unm.edu/download/misc.pdf.
  30. Monti PM, Barnett NP, Colby SM, O’Leary TA. Motivational enhancement of alcohol-involved adolescents. In: Monti PM, Colby SM, O’Leary TA, editors. Adolescents, Alcohol and Substance Abuse. The Guildford Press; NY: 2001. pp. 145–182. [Google Scholar]
  31. Mowbray CT, Holter MC, Teague GB, Bybee D. Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation. 2003;24(3):315–340. [Google Scholar]
  32. Moyers TB. The Global Rating of Motivational Interviewing Therapists. 2004 Retrieved from: http://casaa.unm.edu/download/GROMIT.pdf.
  33. Moyers TB, Martin T, Catley D, Harris KJ, Ahluwalia JS. Assessing the integrity of motivational interviewing interventions: Reliability of the motivational interviewing skills code. Behavioural and Cognitive Psychotherapy. 2003;31(2):177–184. [Google Scholar]
  34. Moyers TB, Martin T, Manuel JK, Hendrickson SML, Miller WR. Assessing competence in the use of motivational interviewing. Journal of Substance Abuse Treatment. 2005;28(1):19–26. doi: 10.1016/j.jsat.2004.11.001. [DOI] [PubMed] [Google Scholar]
  35. Moyers TB, Manuel JK, Wilson P, Hendrickson SML, Talcott W, Durand P. A randomized trial investigating training in motivational interviewing for behavioral health providers. Behavioural and Cognitive Psychotherapy. 2008;36(2):149–162. [Google Scholar]
  36. Orwin RG. Assessing program fidelity in substance abuse health services research. Addiction. 2000;95(S3):S309–S327. doi: 10.1080/09652140020004250. [DOI] [PubMed] [Google Scholar]
  37. Patil VH, McPherson MQ, Freisner D. The use of exploratory factor analysis in public health: A note on parallel analysis as a factor retention criterion. American Journal of Health Promotion. 2010;24(3):178–181. doi: 10.4278/ajhp.08033131. [DOI] [PubMed] [Google Scholar]
  38. Perepletchikova F, Kazdin AE. Treatment Integrity and Therapeutic Change: Issues and Research Recommendations. Clinical Psychology: Science and Practice. 2005;12(4):365–383. [Google Scholar]
  39. Perepletchikova F, Treat TA, Kazdin AE. Treatment Integrity in Psychotherapy Research: Analysis of the Studies and Examination of Associated Factors. Journal of Consulting and Clinical Psychology. 2007;75(6):829–841. doi: 10.1037/0022-006X.75.6.829. [DOI] [PubMed] [Google Scholar]
  40. Resnick B, Inguito P, Orwig D, Yahiro JY, Hawkes W, Werner M, Magaziner J. Treatment fidelity in behavior change research. Nursing Research. 2005;54(2):139–143. doi: 10.1097/00006199-200503000-00010. [DOI] [PubMed] [Google Scholar]
  41. Smith DC, Hall JA, Jang M, Arndt S. Therapist adherence to a motivational-interviewing intervention improves treatment entry for substance-misusing adolescent with low problem perception. Journal of Studies on Alcohol and Drugs. 2009;70(1):101–105. doi: 10.15288/jsad.2009.70.101. [DOI] [PubMed] [Google Scholar]
  42. Santacroce SJ, Maccarelli LM, Grey M. Treatment fidelity. Nursing Research. 2004;53(1):63–66. doi: 10.1097/00006199-200401000-00010. [DOI] [PubMed] [Google Scholar]
  43. Stein KF, Sargent JT, Rafaels N. Intervention research: establishing fidelity of the independent variable in nursing clinical trials. Nursing Research. 2007;56(1):54–62. doi: 10.1097/00006199-200701000-00007. [DOI] [PubMed] [Google Scholar]
  44. Walton MA, Chermack ST, Shope J, Bingham CR, Zimmerman MA, Blow FC, Cunningham RM. Effects of a brief intervention for reducing violence and alcohol misuse among adolescents: A randomized controlled trial. The Journal of the American Medical Association. 2010;304(5):527–535. doi: 10.1001/jama.2010.1066. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES