Reliability and Validity of an Observational Measure of Client Decision-Making: The Client Language Assessment - Proximal/Distal (CLA-PD)

Abstract

Overview

The Client Language Assessment – Proximal/Distal (CLA-PD) is a language rating system for measuring client decision-making in interventions that target a specified behavior change (e.g., alcohol or other drug use). In the CLA-PD, there are five dimensions of change language (Reason, Ability, Commitment, Taking Steps, Other) adapted from the client portion of the Motivational Interviewing Skill Code (MISC). For the CLA-PD, language codes are sub-divided to discriminate statements regarding the primary, or target behavior change (distal change) from the intermediate coping activities (proximal change) that are prescribed to facilitate that target behavior change. The goal of the CLA-PD is to allow for higher specificity than existing client language measures, when process studies consider interventions that are multi-session and skill-based (e.g., Cognitive Behavioral Therapy).

Method

Three raters received 40 hours of training on the use of the CLA-PD. The data were a sample of therapy session audio-files from a completed clinical trial (N = 126), which enabled examination of client language across four sessions (i.e., first three and final attended) of three evidence-based alcohol interventions (Cognitive Behavioral Therapy, Twelve-Step Facilitation Therapy, Motivational Enhancement Therapy).

Results

Inter-rater reliability results for summary scores showed “excellent” reliability for the measure. Specifically, two-way mixed Intraclass Coefficients ranged from .83 to .95. Internal consistency reliability showed alphas across sessions that ranged from “fair” to “good” (α = .74 to.84). In convergent and discriminant validity analyses using data independently measured with MISC-based ratings, the pattern of results was as would be expected. Specifically, convergent correlations, by valence (i.e., change and sustain talk), between CLA-PD Distal and MISC-based language scores were moderate (r = .46 to .55, p < .001) while discriminant correlations by valence for CLA-PD Proximal and MISC-based language scores were small (r = .22 to .24, p < .05). Finally, proportion Change Talk Proximal predicted subsequent session coping behaviors (i.e., Processes of Change) as well as 3-month Alcoholics Anonymous involvement and attendance (ps < .05 – .005), but not 3-month alcohol abstinence self-efficacy. Further, analyses of criterion predictive validity showed that proportion Change Talk Distal predicted 3- and 12-month drinking frequency and quantity measures (ps < .05 – .005).

Conclusions

When behavior change treatments are multi-session and/or skill-based, the present analyses suggest the CLA-PD is a promising, psychometrically sound observational rating measure of client verbalized decision-making.

1 Introduction

Clinical research has begun to emphasize the importance of identifying therapeutic ingredients and behavior change mechanisms most predictive of client outcomes. The underlying rationale is that empirically-based knowledge on key predictive processes has the potential to streamline existing treatments, and therefore enhance treatment efficacy and efficiency (Longabaugh, Magill, Morgenstern, & Huebner, 2013). Process research can also complement efforts to develop new treatments, an area of clinical science that also strives to produce outcomes superior to those currently achievable. To date, some studies have suggested that therapeutic factors common to many nominally distinct treatment modalities are actually as much or more important than factors unique to each modality (Imel, Wampold, Miller, & Fleming, 2008; Wampold, 2001). Indeed, research often supports outcome equivalence across varying treatments, despite very different proposed theories of change (Longabaugh, 2007). Having research tools that can identify within-session processes predictive of outcome may accelerate the identification of effective vs. ineffective therapist behaviors and/or therapeutic components, and thus facilitate the development of improved treatment modalities that target a specified behavior change.

A particularly promising area of process-outcome research focuses on motivational interviewing (MI) with alcohol and other drug use disorders (e.g., Moyers, Martin, Houck, Christopher, & Tonigan, 2009), as well as with other behavioral domains such as gambling (Hodgins, Ching, & McEwen, 2009), diet and nutrition (Pirlott et al., 2012), and sexual risk reduction (Flickinger et al., 2013). Moreover, we can conceptualize MI as a vehicle for studying common factor ingredients and mechanisms since it aligns with a number of foundational theories on behavior change, including: the trans-theoretical model of change (Prochaska & DiClemente, 1983), self-perception theory (Bem, 1967), and self-efficacy theory (Bandura, 1969). In MI, the primary therapeutic mechanism of intervention effect is client decision-making as operationalized by statements for or against engaging in the behavior and/or behavior change (Miller & Rollnick, 2013; Miller & Rose, 2009). In MI process research, these statements are measured by the Motivational Interviewing Skill Code ([MISC] Houck, Moyers, Miller, Glynn, & Hallgren, 2010).

1.1 Behavior change theory and the study of verbalized decision-making

The literature has a number of theories for understanding behavior change and behavior change decisions that inform the constructs of interest to the present work. A contribution of the trans-theoretical model (TTM; Prochaska & DiClemente, 1983) is the characterization of motivation along a continuum where more to less ambivalence will be observed. This is important because motivation to take action is no longer inferred simply by the presence of the client in the therapy room, and this recognition has led to better clinical tailoring to an individual’s presenting readiness for change. Early on, the MI literature directly incorporated Prochaska and DiClemente’s (1983) model, framing the MI dialogue as movement through change stages (Miller & Rollnick, 1991/2002). The authors emphasized “eliciting self-motivational statements” and subsequent to this, such statements were shown to predict outcome (Amrhein, Miller, Yahne, Palmer, & Fulcher, 2003). This finding of a direct effect of in-session client language on subsequent substance use outcomes was grounded in Bem’s (1967) self-perception theory, where a statement of intent is hypothesized to bring about a belief in that intent (cf. Miller & Rollnick, 1991/2002). As the state-of-the-art measure in addictions process research, the MISC provides a communication rating system of behavior change decisions, focused on this direct effect. However, a commitment statement regarding use reduction or cessation is an example of an initial decisional step, but change maintenance is ongoing. Successful behavior change involves ongoing decisions as to whether to engage in the types of coping behaviors that are proposed to help initiate and maintain change (DiClemente, 1986). The effect of client language on change maintenance behaviors, or what Prochaska and DiClemente (1984) refer to as processes of change, has never been studied.

1.2 Empirical support for client language as a predictor of behavior change

Research on client language as a mechanism of behavior change has typically been conducted as treatment-specific analyses of MI effectiveness. An early study conducted by Miller, Benefield, and Tonigan (1993) examined client resistance (e.g., instances of arguing or interrupting behaviors) in relation to a directive-confrontational versus client-centered style of counseling. That study found that client in-session resistance was lower in the client-centered condition and that resistance predicted more drinking at 12 month follow-up (Miller et al., 1993). In the seminal work by Amrhein and colleagues (2003), five types of change language (Reasons, Desire, Need, Ability, Commitment) were coded in a sample of adult illicit drug users. The authors found that a positive slope of commitment strength over the course of an MI session was associated with optimal 12-month outcomes (i.e., status as “changer” or “maintainer”). Moyers, Martin, Christopher, Houck, and Tonigan (2007) extended this research to consider client language as a common, rather than MI-specific, mechanism of behavior change. Here, 45 Project MATCH (Project MATCH Research Group, 1997) session one recordings were coded using the first version of the MISC (four language types: Change Talk, Resist Change, Follow, Ask; Miller, 2000). This study found that frequency of change and resist change talk were independent predictors of 12-month drinks per drinking day across MATCH treatment modalities (i.e., Cognitive Behavioral Therapy [CBT], Twelve Step Facilitation Therapy [TSF], Motivation Enhancement Therapy [MET]). When the sample was grouped by good and poor outcomes (consistent with the methods of Amrhein et al., 2003), both change and resist change talk distinguished groups in the expected directions.

1.3 In session decision-making – is it a direct or indirect effect?

Recently, Magill and colleagues (2014) conducted a meta-analysis of 12 MI process studies and found that certain measures of client language show better predictive validity than others. Specifically, a frequency measure of sustain talk (“the person’s own arguments for not changing, for sustaining the status quo”; Miller & Rollnick, 2013, p. 7) and a composite measure of change and sustain talk (e.g., proportion change talk) were significantly related to outcome, while a frequency measure of change talk (“any self-expressed language that is an argument for change”; Miller & Rollnick, 2013, p. 159) was not. In light of these results, client sustain talk can be conceptualized as having a direct effect on outcome. On the other hand, change talk may be part of a more complex causal chain, a part better characterized as an indirect effect (i.e., involving not only initiation, but also maintenance). This might be particularly true when behavior change interventions have a skill-based, rather than strictly motivational, focus. For example, a study of CBT with cocaine users found that early session commitment strength, along a negative to positive continuum, predicted treatment retention and drug use outcome (Aharonovich, Amrhein, Bisaga, Nunes, & Hasin, 2008). In another CBT process study, sustain talk, but not change talk, was associated with treatment effect among generalized anxiety patients (Lombardi, Button, & Westra, 2014). From the studies reviewed, a couple conclusions may be considered. First, it appears that the predictive validity of change talk is less consistent than sustain talk or than change language, as measured along a negative to positive continuum. This finding appears to be the case in process studies of motivational and skill-based interventions. Second, if change talk has mixed predictive validity to date, then further development of measures to examine client verbalized decisions is needed. When interventions view change as occurring via intermediate behaviors, this need may be even more critical.

1.4 Purpose

In summary, a measure of client decision-making that considers both direct and indirect pathways to change is needed, and this may be particularly important when behavior change treatments are multisession and/or skill-based. We have developed and tested such: the Client Language Assessment – Proximal/Distal (CLA-PD). Distal language refers to verbalized decision-making that may have a direct effect on behavior change (e.g., alcohol or other drug use), while proximal language refers to decisions about indirect effects. Indirect effects are coping activities or life-style changes (e.g., attending a self-help meeting or enlisting the support of a concerned significant other) that are prescribed to facilitate that change. The current study reports on the psychometric properties of this new system. Specifically, we examine inter-rater reliability, internal consistency reliability, and convergent, discriminant, and predictive validity in a sample of therapy session files from a Northeast, Project MATCH clinical research unit (CRU). Within this design, client proximal and distal change language across three evidence-based, multi-session interventions for adult alcohol use disorders could be considered.

2 Material and Methods

2.1 The CLA-PD measure

2.1.1 Overview

The CLA-PD is a novel communication rating system for measuring client verbalized decision-making in interventions that target a specified behavior change (e.g., alcohol or other drug use). In the CLA-PD, there are five dimensions of change language (Reason, Ability, Commitment, Taking Steps, Other), which are adapted from the client portion of the MISC (2001; 2008; 2010). A small departure from the MISC was to subsume Reason-need and Reason-desire codes under the overall Reason category. This decision was made due to typically low observed frequency for these two indicators. A major departure from the MISC was to sub-divide language codes. Further, CLA-PD exemplar ratings for language codes are written to represent a wide cross-section of behavior change interventions. Table 1 summarizes procedures, strengths, and limitations of commonly used client language measures, along with an overview of the CLA-PD.

Table 1.

A Comparison of Client Change Language Measures

Measure (Authors)	Coding Procedure Overview	Language Items	Intended Use	Most Useful When	Less Useful When
Karno, Longabaugh, & Herbeck (2010)	Four 4-point items assessing client Change and Sustain Talk, each scored once for every twenty minutes of an interview.	I A TS C R1	Assessing the strongest instance of client language categories in a recorded session.	Limited resources are a concern; count data is not required; strength of language is of primary interest.	Utterance-level data is desired; count data is required; sequence analyses are a research interest.
CLEAR (Glynn & Moyers, 2012)	Client dialogue is segmented first into utterances. Client utterances which represent attitudes favorable or unfavorable to change are categorized accordingly.	CT CCT2	Assessing relative levels of change/counter-change talk in MI sessions.	Researching MI-based interventions; ease of training and data collection and/or limited resources are a concern; only frequency of CT/CCT are of interest.	Detailed categorizations of client language are desired.
MI-SCOPE (Martin, Moyers, Houck, Paulette, & Miller, 2005)	Dialogue is segmented into utterances. Each utterance is then assigned one code from the available categories.	Ask FN D± A± N± TS± C± O±3	Collecting data with emphasis on preserving temporal sequence for testing relationship between MI skills, therapy process, and client outcome.	Sequence analyses or temporal effects during MI-based interventions are a primary research focus.	Global assessment of client language is desirable; sequence analyses are not a concern.
CLAMI /MISC 2.1 (Miller, Moyers, Ernst, & Amrhein, 2008)/MISC 2.5 (Houck, Moyers, Miller, Glynn, & Hallgren, 2010)	Client’s global level of self-exploration (Truax & Charkhuff, 1967) is assessed. Client dialogue is then segmented into utterances and coded into one of fifteen language categories	R± R±d R±a R±n TS± C± O± FNA4	Assessing therapist adherence to MI principles; assessing the processes hypothesized to underlie MI efficacy.	Researching MI-based interventions; more detailed descriptions of client language are needed.	Researching interventions with a skills-based component; the ability to distinguish between main outcomes and proximal coping skills is desired.
Client Language Assessment - Proximal/Distal (Magill & Apodaca, 2011)	Dialogue is segmented into utterances. Each utterance is assigned one code from the available categories.	RD± RP± AD± AP± TSD± TSP± CD± CP± OD± OP± FN±5	Assessing client language regarding change as it occurs across behavior change treatments.	Comparing client language across different treatments; researching skill-based or multi-session behavioral interventions.	Distinction between main outcomes and proximal coping is not of interest; a broad cross-section of behavior change interventions is not of interest.

¹I = Importance of Change; A = Ability to Change; TS = Taking Steps to Change; C = Commitment to Change; R = Resistance to Change;

²CT = Change Talk; CCT = Counter-Change Talk (Counter Change Talk is now more commonly referred to as Sustain Talk in reports on MI therapy process);

³FN = Follow/Neutral; C = Commitment; D = Desire; A = Ability; N = Need; TS = Taking Steps; O = Other;

⁴R = Reason; d = Desire; a = Ability; n = Need; TS = Taking Steps; C = Commitment; O = Other; FNA=Follow/Neutral/Ask;

⁵R = Reason; A = Ability; TS = Taking Steps; C = Commitment; O = Other; FN = Follow/Neutral/Ask; D = Distal; P = Proximal.

Notes. The CLA-PD rating manual (Magill & Apodaca, 2011) is available, by request, from the first author. Measures included above are a selection of those recently reviewed by Dobber and colleagues (2014).

2.1.2 The targeted behavior change

Commonly used client language measures typically focus on a pre-determined behavioral outcome such as achieving nicotine abstinence or moderating alcohol intake. Critical to reliability, validity, and sensitivity of measurement, this Targeted Behavior Change (TBC) is: a) well defined, b) likely to be impacted by the intervention, and c) involves both a behavioral outcome (e.g., smoking, drinking) and a direction of change (e.g., to eliminate or reduce versus maintain or increase) (Houck, Moyers, Miller, Glynn, & Hallgren, 2010). Thus far, TBC measurement has not discriminated distal from proximal statements. These two types of outcome, therefore, have carried equal weight within a causal chain. In contrast, the CLA-PD frames behavior change as involving two independent decision-making processes, proximal and distal, and we believe making this distinction is particularly important when studies of treatment process involve interventions that are multi-session, didactic, and/or skill-based (e.g., CBT).

2.1.3 Rater training

For the present psychometric report, three bachelor’s level raters received approximately 40 hours of training on the use of the CLA-PD by the first author. Rater training followed standard procedures, including the use of audio-recorded pilot sessions from a training library (N = 7). These training sessions have exemplar ratings of client codes with narrative justification. Observational rater training involved three phases: 1) didactic overview, including treatment- and coding-related readings (i.e., Kadden et al., 1992; Magill & Apodaca, 2011; Miller, Zweban, DiClemente, & Rychtarik, 1992; Nowinski, Baker, & Carroll, 1992), 2) group coding practice with corrective feedback, and 3) individual coding practice with group corrective feedback. Rater proficiency was defined by Intraclass Correlation Coefficient (ICC) agreement with training library exemplar ratings (i.e., ICC = .75 or above; Cicchetti, 1994). Project observational raters were masked to participant outcomes.

2.1.4 Rating procedure

Data collection for the project occurred in two steps. First, all audio session files were parsed, which involved dividing all speaker statements into utterances (distinct units of meaning) using the Coding Application for Client Therapist Interactions (CACTI). The CACTI is a software program for parsing and coding therapy session audio-files that is easily adaptable to varied measurement systems (Glynn, Hallgren, Houck, & Moyers, 2012). Second, each parsed utterance was assigned a client language code. Third, the CACTI produces a text file of utterance-level language codes, at which point a range of scoring procedures can be employed within a separate statistical software program. For the CLA-PD, summary scores are consistent with those described in the MISC, that is, the primary summary measure of client language is a proportion change talk score (sum frequency of change talk items/sum frequency of change and sustain talk items).

2.2 Study sample

Observational rating data were derived from a sample of session files from a Northeast, Project MATCH aftercare site. Project MATCH (1997) tested 21 matching variables, across three multi-session, alcohol treatments (CBT; TSF; MET) at 10 research sites among 1,726 participants with alcohol use disorders. The study demonstrated significant main effects, across treatment conditions, over three-year follow-up (Project MATCH Research group, 1998). Participants were treatment-seeking adults meeting DSM III-R criteria for alcohol abuse or dependence. The average age of the current sample was 45 (SD = 13.3) years old, the sample majority was male (69.8%) and Caucasian (94%). The majority of participants were employed (64.2%), unmarried (59.5%), and their average years of education was 13 (SD = 2.1). This was a primarily alcohol dependent sample (69.6%).

2.3 Study session data

The selected sample of therapy sessions enabled an examination of client proximal and distal change language across the three treatments tested in MATCH. CBT and TSF involved 12 weekly sessions, while MET included four sessions, conducted at the first, second, sixth, and twelfth weeks of treatment. For the current study, the first three and final sessions attended were rated. Each treatment had a well-specified theoretical foundation and corresponding manualized protocol. First, CBT was based on a social learning model with intervention strategies targeting prescribed coping activities related to internal and external risks for relapse (e.g., managing urges/cravings, managing negative affective states, drink refusal skills, social skills training; Kadden et al., 1992). Second, TSF was based on a disease framework and focused on involvement in Alcoholics Anonymous prescribed coping activities (e.g., acceptance of disease, meeting attendance, sponsorship, engaging in the 12-steps; Nowinski, Baker, & Carroll, 1992). Third, MET was grounded in a theoretical integration of motivational psychology and client-centered therapy and emphasized therapeutic skills that activate client internal capacities for change (e.g., efficacy support, exploration of ambivalence, personalized feedback on alcohol use, change planning; Miller et al., 1992). Treatment adherence was assessed through weekly therapist ratings as well as supervisor monitoring of a 25% random selection of session files. Project MATCH achieved high treatment adherence, integrity, as well as discriminant validity (Carroll et al., 1998).

2.4 Measurement

The Client Language Assessment – Proximal/Distal (CLA-PD) was used to code therapy session audio-files. See full description of the measure above.

2.4.1 Convergent and discriminant validity measure for Distal and Proximal Change and Sustain Talk

For convergent and discriminant validity analyses, we compared CLA-PD change and sustain talk with client language data from a study conducted by Karno, Longabaugh, and Herbeck (2010). The authors measured client language in audiotaped Project MATCH sessions from the same CRU as the present study. Using MISC-based ratings (i.e., Amrhein et al., 2003), the authors assessed the first two sessions of each treatment modality, divided sessions into segments, and segments were scored by two independent raters. Language categories included Importance (i.e., Reason, Desire, Need), Ability, Commitment, and Taking Steps; a score represented the strongest expression of that category within a given segment. In the present study, summary scores (i.e., Importance, Ability, Commitment, Taking Steps) were averaged to the session level, and then tested by valence (i.e., change and sustain talk) as convergent (sum Change Talk Distal and sum Sustain Talk Distal) and discriminant (sum Change Talk Proximal and sum Sustain Talk Proximal) criterion measures for CLA-PD session one data. Session two data were examined as sensitivity analysis.

2.4.2 Predictive validity measures for Proximal Change Language

Predictive validity for proximal language was examined using three separate indicators. The first criterion was coping activities/behavior measured via weekly self-report ratings from an eight-item version of the Processes of Change scale (POC-8; Prochaska, Velicer, DiClemente, & Fava, 1988). The POC-8 assesses coping behaviors along two sub-scales of experiential (cognitive processing; two items: seeking information about drinking, get upset when I think about my drinking) and behavioral (relational and behavioral; six items: self-commitment for abstinence, talking to others, avoid high risk situations, avoid high risk people/places, replace response to tension/urges, reward self for not drinking) processes (Prochaska & DiClemente, 1984). The psychometric properties of the 20-item version of the POC are “excellent” with sub-scale alphas ranging from .90 to .91 (DiClemente, Carbonari, Addy, & Velasquez, 1996; cf. DiClemente, Carbonari, Zweben, Morrell, & Lee, 2001). Because there were only two experiential items included in the POC-8, only the total score was used.

Given the extent of self-help involvement among Project MATCH participants (Connors et al., 2001), the Alcoholics Anonymous Involvement (AAI) scale (Tonigan, Connors, & Miller, 1996) was used as the second criterion measure of coping activities/behavior (at 3-month follow-up). The AAI is a 13-item self-report inventory that examines both attendance and involvement in Alcoholics Anonymous. The third proximal language criterion was self-efficacy. The 3-month Alcohol Abstinence Self-Efficacy Scale (AASE; DiClemente, Carbonari, Montgomery, & Hughes, 1994) served as the self-efficacy criterion measure. The AASE is a 5-point Likert-rated scale of temptation to use and confidence to abstain across 20 different high-risk alcohol use situations; the confidence sub-scale was used in the present report.

2.4.3 Predictive validity measures for Distal Change Language

The current study used the following drinking criterion variables in predictive validity analyses: past 30-day percent abstinent days (PDA) and number of drinks per drinking day (DDD) from the Form-90 (Miller, 1996). The Form-90 is an established measure with sound test-retest reliability (Del Boca & Brown, 1996) and concurrent validity with secondary biochemical alcohol measures (Project MATCH Research Group, 1997). Alcohol outcomes were arcsine and square root transformed, respectively, to improve distributional properties and maintain consistency with other studies reporting Project MATCH data. Outcomes at 3- and 12-month follow-ups were used. For consistency with established client language scoring methods (see e.g., Table 1) and due to recent findings on client change language in relation to follow-up outcomes (Magill et al., 2014), all analyses of predictive validity were conducted with a proportion change talk score. This score can be conceptualized as a combined measure of the positive (change talk) and negative (sustain talk) sides of client ambivalence.

2.5 Data-analysis

Analyses for the current psychometric report targeted examination of the inter-rater and internal consistency reliability as well as the convergent, discriminant, and predictive validity of the CLA-PD coding system. For inter-rater reliability, a 10% random sample of treatment sessions (N_sessions = 47) was double-coded. Analyses were specified as a two-way mixed effects (rater as random; measure as fixed), single measure ICC (McGraw & Wong, 1996). Internal consistency analyses were computed with Cronbach’s alpha (1970). Finally, Pearson bivariate correlations were used for analyses of convergent, discriminant, and predictive validity in relation to within treatment, 3- and 12-month criteria. For these analyses, sum change talk and sum sustain talk or proportion change talk scores were used as summary variables.

3 Results

Of the original sample (N = 168), session report data were available for 89.9% of participants (N = 151). Of these cases, recorded treatment sessions were available for 99.3% (N = 150). Finally, we selected only those cases where at least three treatment sessions were available (final N = 126; 106 four-session and 20 three-session cases).

3.1 Inter-rater reliability

For reliability estimates, summary scores by valence (i.e., change talk and sustain talk) were calculated by summing the frequency of relevant individual client language items. For example, Change Talk Distal is comprised of the sum of Reason: Positive Distal, Ability: Positive Distal, Taking Steps: Positive Distal, Commitment: Positive Distal, Other: Positive Distal. Table 2 shows generally very good reliability for the measure. Specifically, ICC values ranged from .83 to .95 for CLA-PD proximal and distal change and sustain talk scores. Further, only four of the 21 individual language items showed inter-rater reliability in the “fair” or “poor” range (Cicchetti, 1994).

Table 2.

Reliability and Session-level Descriptive Information

Frequency/Summary Scores	ICCa	Minimumb	Maximumb	Meanb	(SD)b
Change Talk Distal	.89	0	133	35.96	(23.26)
Change Talk Proximal	.94	1	157	41.24	(25.63)
Sustain Talk Distal	.95	0	94	14.40	(15.26)
Sustain Talk Proximal	.83	0	105	12.03	(12.03)
Follow/Neutral	.99	34	468	168.33	(70.51)
Individual Client Behavior Counts
Reason: Positive Distal	.96	0	100	22.39	(17.78)
Ability: Positive Distal	.87	0	28	5.17	(4.79)
Taking Steps: Positive Distal	.51	0	20	1.92	(2.39)
Commitment: Positive Distal	.69	0	10	0.83	(1.42)
Other: Positive Distal	.54	0	44	5.65	(6.17)
Reason: Positive Proximal	.97	0	104	19.67	(15.42)
Ability: Positive Proximal	.72	0	21	2.80	(3.14)
Taking Steps: Positive Proximal	.94	0	56	9.79	(7.86)
Commitment: Positive Proximal	.81	0	19	3.63	(3.58)
Other: Positive Proximal	.62	0	43	5.37	(5.81)
Reason: Negative Distal	.84	0	51	6.01	(8.30)
Ability: Negative Distal	.90	0	43	4.99	(5.80)
Taking Steps: Negative Distal	.88	0	35	1.14	(3.36)
Commitment: Negative Distal	.70	0	5	0.15	(0.61)
Other: Negative Distal	.01	0	19	2.11	(3.13)
Reason: Negative Proximal	.84	0	51	5.41	(6.18)
Ability: Negative Proximal	.74	0	15	2.63	(3.00)
Taking Steps: Negative Proximal	.75	0	22	1.97	(2.66)
Commitment: Negative Proximal	.65	0	4	0.25	(0.65)
Other: Negative Proximal	.43	0	30	1.75	(3.23)

Notes.

^aReliability estimates based on N = 47_sessions double-coded. Cicchetti (1994) suggests the following guidelines for assessing reliability of observational coding systems: ICC of .75 or above = excellent; .60–.74 = good; .40–.59 = fair; below .40 = poor. Fair or poor items are shown in bold.

^bSession-level descriptive data based on N = 484_sessions.

3.2 Internal consistency reliability

Analyses of internal consistency reliability showed acceptable internal consistency for CLA-PD change and sustain talk summary scores (Nunnally, 1978). For drinking language across the four treatment sessions, Cronbach’s alpha was α = .74 for Change Talk Distal and α = .84 for Sustain Talk Distal. For proximal language across the four treatment sessions, Cronbach’s alpha was α = .80 for Change Talk Proximal and α = .81 for Sustain Talk Proximal.

3.3 Convergent and discriminant validity

Table 3 shows results from convergent and discriminant validity analyses with change language measures from the work of Karno and colleagues (2010). The pattern of results was as would be expected. Specifically, convergent correlations, by valence (i.e., change and sustain talk), between CLA-PD Distal and MISC-based language scores were moderate (r = .46 to .55, p < .001) while discriminant correlations by valence for CLA-PD Proximal and MISC-based language scores were small (r = .22 to .24, p < .05).

Table 3.

Convergent and discriminant validity measure for Proximal and Distal Change and Sustain Talk

	Distal Change Talk - session 1	Distal Sustain Talk - session 1	Proximal Change Talk - session 1	Proximal Sustain Talk - session 1	Change Talk – session 1 (Karno et al., 2010)	Sustain Talk – session 1 (Karno et al., 2010)
Distal Change Talk - session 1	1	.301***	.326***	.001	.455***	−.122
Distal Sustain Talk - session 1		1	.226*	.446***	.021	.547***
Proximal Change Talk - session 1			1	.417***	.218*	−.019
Proximal Sustain Talk - session 1				1	.001	.244**
Change Talk – session 1 (Karno et al., 2010)					1	−.027
Sustain Talk – session 1 (Karno et al., 2010)						1

N = 126.

^†<.10,

^*< .05,

^**<.01,

^***<.005.

Notes. Correlation between session 2 data were run as sensitivity analyses and these results were consistent with those presented for session 1 (above).

3.4 Predictive validity

For analyses of predictive validity, proportion, rather than frequency/summary scores were examined. This was to remain consistent with predominant scoring methods in the change language literature.

3.4.1 Proximal criteria

For Change Talk Proximal proportion scores in relation to selected coping and self-efficacy criteria, predictive validity analyses showed mixed results. For processes of change coping during the course of treatment (POC-8), proportion scores for Change Talk Proximal at each session showed positive and moderate correlations with coping behavior in each subsequent session (session one to two r = .198, p < .05; session two to three r = .295, p < .005; session three to four r = .401, p < .005). Change Talk Proximal scores in relation to 3-month self-help coping behavior (i.e., AA involvement and attendance) and self-efficacy are shown in Table 4. The proportion of Proximal Change Talk showed generally positive, significant, and small to moderate correlations with both self-help coping criteria (ps < .05 – .005). Surprisingly, Change Talk Proximal proportion score in session four was unrelated to self-reported AA attendance at 3-month follow-up. Finally, and contrary to expectations, Change Talk Proximal proportion scores were not associated with self-reported abstinence self-efficacy at 3-months (see Table 4).

Table 4.

Predictive Validity Results for CLA-PD Proximal Summary Measures – Self-Help Coping and Self-Efficacy Outcomes

	Alcoholics Anonymous Involvement - 3 month	Alcoholics Anonymous Attendance - 3 month	Abstinence Self- Efficacy - 3 month	Proximal Change Talk - Proportion score - session 1	Proximal Change Talk - Proportion score - session 2	Proximal Change Talk - Proportion score - session 3	Proximal Change Talk - Proportion score – final session
Alcoholics Anonymous Involvement - 3 month	1	.740***	.092	.283**	.215*	.328***	.340***
Alcoholics Anonymous Attendance - 3 month		1	.047	.333***	.315***	.324***	.122
Abstinence Self-Efficacy - 3 month			1	.147	.141	.066	.174†
Proximal Change Talk - Proportion score - session 1				1	.336***	.293***	.421***
Proximal Change Talk - Proportion score - session 2					1	.525***	.268***
Proximal Change Talk - Proportion score - session 3						1	.447***
Proximal Change Talk - Proportion score – final ses.							1

N = 126.

^†<.10,

^*< .05,

^**<.01,

^***<.005.

Notes. ses. = session.

3.4.2 Distal criteria

Analyses examining predictive validity for Change Talk Distal proportion scores showed very promising performance for the CLA-PD (see Table 5). Distal Change Talk at each session predicted participant alcohol use at 3- and 12-month follow-ups. These small to moderate correlations were relatively stable over time (ps < .05 – .005).

Table 5.

Predictive Validity Results for CLA-PD Distal Proportion Scores - Alcohol Outcomes

	% Days abstinent – 3 month	% Days abstinent - 12 month	Standard Drink Per Drinking Day – 3 month	Standard Drink Per Drinking Day – 12 month	Distal Change Talk - Proportion score - session 1	Distal Change Talk - Proportion score - session 2	Distal Change Talk - Proportion score - session 3	Distal Change Talk – Proportion score – final session
% Days abstinent – 3 month	1	.619***	−.751***	−.453***	.290***	.344***	.220*	.239*
% Days abstinent - 12 month		1	−.397***	−.777***	.292***	.283***	.278*	.292*
Standard Drink Per Drinking Day – 3 month			1	.346***	−.324***	−.433***	−.276***	−.298***
Standard Drink Per Drinking Day – 12 month				1	−.326***	−.306***	−.287***	−.190*
Distal Change Talk - Proportion score - session 1					1	.605***	.535***	.398***
Distal Change Talk - Proportion score - session 2						1	.543***	.336***
Distal Change Talk - Proportion score - session 3							1	.547***
Distal Change Talk - Proportion score - final ses.								1

N = 126 (12 month drinking outcomes include 5% attrition).

^†<.10,

^*< .05,

^**<.01,

^***<.005.

Notes. ses. = session

4 Discussion

This study presents psychometric findings on a novel communication rating measure of client verbalized decision-making in interventions that specify a targeted behavior change. The CLA-PD is an adaptation of the client portion of the Motivational Interviewing Skill Code (MISC; 2001; 2008; 2010) where client language can be assessed both in relation to primary (e.g., alcohol use) and secondary (e.g., coping activities/behaviors) outcomes of interest. The CLA-PD additionally provides exemplars for client language that may occur broadly across behavior change interventions. For research on mechanisms of change in the addictions, this study suggests the CLA-PD is a psychometrically promising measure that can facilitate the study of a more comprehensive causal model of treatment-facilitated behavior change, particularly when multi-session, skill-based interventions are the subject of study. In other words, the CLA-PD affords future addictions process research the opportunity to model within-session pathways to behavior change that are direct (via decisions about the target outcome) and indirect (via decisions about how to achieve that outcome).

4.1 CLA-PD reliability

CLA-PD change and sustain talk summary scores ranged well above the standard .75 ICC threshold of “excellent” reliability for distal behavior change language as well as for our new dimension, language about proximal behavior change. This compares favorably with studies using the MISC where ICCs for change talk have ranged from .66 (Gaume, Bertholet, Faouzi, Gmel, & Daeppen, 2010) to .88 (Moyers et al., 2009), and ICCs for sustain talk have ranged from .54 (Apodaca, Magill, Longabaugh, Jackson, & Monti, 2014) to .84 (Vader, Walters, Prabhu, Houck, & Field, 2010). This suggests that the CLA-PD can be used to code therapy sessions with similar reliability to the MISC while also providing greater detail (i.e., distal and proximal change) for measuring within-session statements about behavior change. This kind of detail (i.e., sub-division of the TBC) will result in a reduced frequency of individual language items, which often negatively affects ICC magnitude (Xu & Lorber, 2014). Therefore, our findings have demonstrated that greater specificity of measurement can be achieved without significant sacrifices to reliability. This result should be of particular interest to observational process researchers studying behavior change interventions. Finally, internal consistency reliability analyses showed summary measures by valence met the recommended .70 threshold for acceptability when indices are novel or exploratory (Nunnally, 1978).

4.2 CLA-PD validity

4.2.1 Convergent and discriminant validity

In the present research, we had access to independently-rated data from a previously conducted process study (Karno et al., 2010) that used measurement methods consistent with those described by Amrhein and colleagues (2003). While these ratings were available for sessions one and two only, correlations between measures were positive, moderate, and statistically significant in convergent analyses (i.e., Distal Scores) and positive, small, and less significant in discriminant analyses (i.e., Proximal Scores). These kinds of analyses are typically more art than science, as the construct or method in question will typically drive what constitutes a determination of “good” versus “bad” construct validity (Campbell & Fiske, 1959). That said, our results suggest an expected pattern of correlations among measures. However, because our coding data collection methods differed from those of Karno and colleagues (2010; i.e., behavior counts versus a segment highpoint, respectively), this study cannot isolate the methodological origin of correlation magnitude between convergent and discriminant measures (i.e., sub-divided TBC and/or counts versus a highpoint).

4.2.2 Predictive validity

Our findings suggest promising predictive validity for the CLA-PD. The proportion Change Talk Distal summary scores were consistently associated with drinking outcomes, such that higher proportions of change language in each of the four coded sessions were associated with more percent days abstinent and fewer drinks per drinking days at both the 3- and 12-month follow-up points. Correlation coefficients were moderate in magnitude, which exceeds the small effects recently found for a composite change and sustain talk measure in meta-analytic research (Magill et al., 2014). Predictive validity for Change Talk Proximal in relation to short-term, coping outcomes was also encouraging. Specifically, proximal language at each treatment session showed positive, moderate correlations with client self-reported coping measured at the subsequent session. This finding is important because the behavioral processes measured by the POC have demonstrated empirical support in relation to alcohol use among non-treatment seeking hospital patients (Freyer et al., 2006), as well as drinking outcomes in analyses of Project MATCH data (DiClemente et al., 2001). In addition to Change Talk Proximal predicting subsequent coping behaviors during treatment, this language variable was associated with greater involvement in AA and higher levels of attendance at AA meetings at 3-month follow up (with the exception of session four). Finally, Change Talk Proximal summary scores were not associated with self-reported abstinence efficacy. This is surprising given that proximal language was associated with greater engagement in coping activities throughout treatment, which one might expect would be related to an increase in clients’ subsequent sense of self-efficacy as they engage in new behaviors to support their sobriety. However, the abstinence self-efficacy measure specifically evaluates how confident individuals are that they will not drink in specific high-risk situations (DiClemente et al., 1994). Engagement in coping activities may not be linearly related to success, and in fact, greater confidence may lead to less need for specific coping behaviors. Such speculation will require further study. However, our findings suggest that proximal client language is associated with subsequent change behaviors, but not expectancies about abstinence from drinking.

4.3 Limitations

This study has some limitations to consider. First, the use of archival MATCH data can be considered a strength given the availability of three behavioral treatments for process analysis. Yet, with these rich data comes limitations inherent to any secondary data-analysis. In our case, the study sample was relatively homogeneous with respect to age, gender, and race. Further, these were aftercare participants and it is unclear how our results would replicate among MATCH outpatients. Given our sample size was relatively small, we were also unable to test variation in psychometric performance by treatment condition. To date, observationally-rated process studies of MATCH data have not achieved this important goal. Finally, we were constrained by the available criterion measures in the MATCH dataset, and this would not have been the case if we had conducted an original study upon which to validate the CLA-PD measure.

4.4 Conclusions

The present psychometric report summarizes promising properties of a novel observational coding system for the study of verbalized decision-making in multi-session, skill-based interventions. The CLA-PD offers sound reliability and validity while offering a level of detail that may result in a more nuanced understanding of treatment-facilitated behavior change. Specifically, to change an addictive, or other ingrained behavioral, pattern is a process; there are decisions not only related to the primary change, but also to change initiating and maintaining behaviors. Further, the vast majority of behavior change interventions include intermediate behavioral prescriptions intended to positively impact the primary change of interest. The CLA-PD can aid future psychotherapy process research in this area.

HIGHLIGHTS.

• The Client Language Assessment – Proximal/Distal (CLA-PD) is a communication rating system for measuring client decision-making in interventions that target a specified behavior change.

• Interrater agreement reliability for the CLA-PD was “excellent”.

• Internal consistency reliability for the CLA-PD was “fair” to “moderate”.

• Convergent and discriminant construct validity analyses showed a pattern of correlations with Motivational Interviewing Skill Code-based ratings (Karno, Longabaugh, & Herbeck, 2010) that were consistent with expectations.

• Predictive validity was generally very good, but contrary to expectations, Proximal Change Talk within sessions did not predict subsequent Alcohol Abstinence Self-Efficacy at 3-months.

• When behavior change treatments are multi-session and/or skill-based, the present analyses suggest the CLA-PD is a psychometrically promising observational rating measure of client verbalized decision-making (i.e., the balance of pro- and anti-change statements).