Development of a Scale to Measure Practitioner Adherence to a Brief Intervention in the Emergency Department

Abstract

Brief intervention (BI) can reduce harmful and hazardous drinking among emergency department patients. However, no psychometrically-validated instrument for evaluating the extent to which practitioners correctly implement BIs in clinical practice (e.g., adherence) exists. We developed and subsequently examined the psychometric properties of a scale that measures practitioner adherence to a BI, namely the Brief Negotiation Interview (BNI). Ratings of 342 audio-taped BIs in the emergency department demonstrated that the BNI Adherence Scale (BAS) has: 1) excellent internal consistency and discriminant validity; 2) good to excellent inter-rater reliability, and 3) good construct validity, with an 8-item, 2-factor structure accounting for 62% of the variance, but 4) no predictive validity in this study. The BAS provides practitioners with a brief, objective method to evaluate their BNI skills and give feedback to them about their performance.

1. Introduction

A growing body of evidence documents the efficacy of brief interventions (BI) for harmful and hazardous alcohol drinkers in emergency departments and acute care settings (D'Onofrio, Fiellin, Pantalon, Chawarski et al., 2012; Gentilello, Ebel, Wickizer, Salkever & Rivara, 2005; Neumann, Neuner, Weiss-Gerlach, Tønnesen et al., 2006) and the substantial reduction in health-care related costs resulting from such interventions (D'Onofrio & Degutis, 2002; Havard, Shakeshaft & Sanson-Fisher, 2008; McDonald, Wang & Camargo, 2004). Further, emergency department practitioners can be successfully trained to implement BI in “real-world” medical settings in approximately 2 hours (D'Onofrio, Pantalon, Degutis, Fiellin & O'Connor, 2005). BI also has promise for illicit drug abuse (Bernstein, Edwards, Dorfman, Heeren, Bliss & Bernstein, 2009; D'Onofrio & Degutis, 2010; Madras, Compton, Abula, Stegbauer, Stein, & Clark, 2009) and several national agencies have endorsed BI for use in emergency care settings to address problematic public health behaviors (D'Onofrio, Goldstein, Denisco, Hingson, Heffelfinger & Post, 2009).

If BI is to be used in “real-world” emergency and acute medical settings, it is critical to understand the degree to which practitioners can implement the intervention as intended (i.e., practitioner adherence). Adherence to BI in such settings may vary widely and thus, accurately measuring and monitoring performance could lead to identifying and addressing ongoing training and supervision needs. This is especially true for medical practitioners initially learning BI, as they would likely benefit from adherence rating feedback and coaching to improve their performance, as has been demonstrated for addiction counselors (Miller, Yahne, Moyers, Martinez & Pirritano, 2004; Sholomskas, Syracuse-Siewert, Rounsaville, Ball, Nuro, & Carroll, 2005). This type of adherence measurement could also aid efforts to determine the relationship between BI implementation and patient outcomes in clinical trials.

Prior studies or descriptions of BI implementation efforts in emergency departments, however, have not used psychometrically established instruments to document the degree to which practitioners have adhered to essential components of BI. In fact, no psychometrically validated instrument for evaluating the extent to which practitioners correctly implement BIs (e.g., adherence) in medical settings exists. To address this shortcoming, the current study describes the development and initial psychometric properties of an adherence scale for a BI called the Brief Negotiation Interview or the BNI (D'Onofrio et al., 2005).

Our group developed the BNI, a 10-minute intervention to reduce harmful and hazardous drinking among emergency department patients. The BNI includes strategies of motivational interviewing (Miller & Rollnick, 2002), physician advice and behavioral contracting. Our group tested the BNI's efficacy in a randomized controlled trial of harmful and hazardous alcohol drinkers (D'Onofrio et al., 2008) and found that the mean number of drinks in the past 7 days and binge-drinking episodes (greater than 4 drinks for men and greater than 3 drinks for women and all individuals older than 65 years) in the past 30 days were significantly reduced in both the BNI and control group (Discharge Instructions) at 1-, 6-, and 12-month follow-ups, but there were no statistically significant differences between the two groups. These findings were consistent with a 2008 meta-analysis, which found that brief interventions for alcohol problems in the ED did not significantly reduce drinking (Havard, Shakeshaft & Sanson-Fisher, 2008). However, in a subsequent and recently completed study of 889 harmful and hazardous drinkers in an emergency department, we found that the reductions in both mean number of standard drinks in the past 7 days and mean number of binge-drinking episodes in the past 30 days from baseline to 6 and 12 months were greater in the BNI group as compared to the standard care group (p=0.03 and 0.045, respectively). In addition, reductions in rates of driving after drinking ≥ 3 drinks from baseline to 12 months were greater in the BNI (38% to 29%) than in a standard care group (43% to 42%); (p=0.040) (D'Onofrio et al., 2012). In the course of these trials, we developed a practitioner adherence measure called the BNI Adherence Scale (BAS). The BAS assesses the degree to which practitioners correctly use BNI components. Differences in drinking outcomes between the BNI and the control conditions in the second trial, as compared to the original one, likely are threefold: (1) Participants in the second trial drank greater quantities and did so more consistently than those in the first study. This was most likely due to our more intensive screening procedure, which aimed to confirm that all participants, in fact, met criteria for high-risk drinking. These procedures, which included a brief screener and a more detailed time-line follow-back method, resulted in baseline means of weekly drinks and binge drinking days of 20.4 and 7.2, respectively, for participants in the second study, compared with 13 and 5.7, respectively, in the first study; (2) The control conditions in the second trial were closer to “treatment-as-usual,” as they did not include a script or training as did the DI condition in the first trial; and (3) We severely reduced the number of assessments in the control-assessment condition of the second trial and completely eliminated those that measured participants' motivation to change, given that such an assessment is actually part of the BNI, which may have led to a greater contrast between the treatment conditions.

In the current study we report on the development and psychometric properties (i.e., reliability and validity) of the BAS in the context of the first trial (D'Onofrio et al., 2008) described above. Consistent with previous research in the area (Carroll, Nich, Sifry, Frankforter, et al., 2000; Perepletchikova & Kazdin, 2005), we examined the internal consistency of the items, inter-rater reliability, and scale construct, discriminant and predictive validity. We hypothesized that the BAS items would 1) have good internal consistency, 2) show good rating agreement among independent raters, 3) converge to form independent factors corresponding to the key components of the BNI, 4) discriminate between recordings of BNI intervention sessions vs. those that involved sessions in which practitioners gave discharge instructions only, and 5) be positively associated with improved drinking outcomes at 1-, 6- and 12-month follow-ups.

2. Materials & Methods

2.1 Overview of the BNI Trial

A randomized clinical trial of the BNI's efficacy was conducted in an urban hospital emergency department (see D'Onofrio et al., 2008 for a full description). Adult patients who presented with hazardous/harmful drinking were randomly assigned to receive either the BNI or Discharge Instructions (DI), where patients were simply told to follow-up on their drinking with their primary care physician, by emergency department practitioners trained to deliver both interventions. The 494 participants enrolled in the study were similar in baseline characteristics across conditions. The mean number of drinks in the past 7 days and binge drinking episodes over the past 30 days were collected at 1-, 6- and 12-months.

2.2 Interventions

The same practitioners implemented both interventions in the study. The interventions were the Brief Negotiation Interview (BNI) and Discharge Instructions (DI). All interventions were audiotaped.

2.2.1 The Brief Negotiation Interview

The BNI (D'Onofrio et al., 2005) is a brief adaptation of motivational interviewing (Miller & Rollnick, 2002) for use in health care settings, combined with physician advice and behavioral contracting. The components of the BNI include: 1) raising the subject of and discussing alcohol use in a patient-centered manner; 2) providing feedback (e.g., make a connection between drinking and the emergency department visit/other illness or injury); 3) identifying and eliciting motives for change (e.g., having patients self identify their readiness to reduce or stop drinking on a scaled ruler and enhancing their motivation for change); and 4) negotiating a change plan and advising as needed. The BNI takes 10 minutes to complete.

2.2.2 Discharge Instructions

DI, the control condition in the trial, required the practitioner to: 1) hand the patient a pamphlet with information about and referral sources for getting help for a variety of health-risk behaviors (e.g., alcohol), including the NIAAA definitions of harmful and hazardous drinking; 2) encourage the patient to follow-up with his/her primary care physician regarding their engagement in any of these health-risk behaviors; and 3) thank the patient for his/her time. These three elements were also part of the BNI. When conducting DIs, the practitioners were asked not to use other BNI items. The DI took 1-2 minutes to complete.

2.3 Participants

Participants in the current study of BNI adherence were emergency department practitioners, emergency department patients, and independent tape raters, all of whom were part of the aforementioned first randomized controlled clinical trial of the BNI (D'Onofrio et al., 2008). All of the patients gave consent for participation in the study according to the rules and policies of the Yale University School of Medicine Human Investigation Committee.

2.3.1 Emergency department practitioners

Forty-seven emergency practitioners administered the BNI, where 16 (34%) were attending physicians, 22 (47%) were third- and fourth year emergency medicine residents and 9 (19%) were physician associates. Forty-seven emergency practitioners administered the DI, where 19 (40%) were attending physicians, 19 (40%) were third- and fourth year emergency medicine residents and 9 (19%) were physician associates. On average, each practitioner completed 5.34 BNIs (SD = 4.89; median = 4; mode = 1; range = 1–28) and 5.04 DIs (SD = 5.02; median = 3; mode = 3; range = 1-22) over the course of the study.

2.3.2 Emergency department patients

Patients 18 years or older who reported alcohol consumption that exceeded the National Institute on Alcohol Abuse and Alcoholism's low-risk limits (hazardous drinkers) or whose index emergency department visit was related to an injury associated with alcohol use (harmful drinkers) were eligible to participate in the study (see D'Onofrio et al., 2008 for details). Patients were excluded if they were non–English speaking, likely to be alcohol dependent (defined as having an Alcohol Use Disorders Identification Test [Babor, Higgins-Biddle, Saunders & Monteiro, 2002] score greater than 19) or drug dependent (as determined by self-reported daily use of any illicit drug), currently enrolled in a substance abuse treatment program, seeking treatment for an acute psychiatric complaint or hospitalized for a psychiatric problem in the past year, or critically ill, injured, or cognitively impaired.

2.3.3 Independent tape raters

Three tape raters rated the brief intervention encounters. All three were substance abuse clinicians who had previously served as independent treatment adherence raters in at least three prior addiction treatment studies. They were blinded to the study protocol, randomization of patients, selection of emergency department practitioners, hypotheses and outcome measures. All three raters had Master's degrees and worked in medical or mental health settings.

2.4 Assessment Measures

2.4.1 BNI Adherence Scale (BAS)

The development of the BAS mirrored prior work by Carroll et al. (2000) in the area of treatment adherence and competence with substance abuse psychotherapies, where the current investigators 1) generated multiple candidate items for each of the four BNI components (see BNI description above), and 2) piloted this initial, longer version of the scale in the emergency department by scoring other practitioner's BNI implementations (n=5) to evaluate the usefulness and clarity of each of the items. Following these ratings, items that were unclear, difficult to rate or redundant were removed from the scale, resulting in the final version with 21, “yes/no” items (see Table 1). For each item, raters indicated whether or not the prescribed procedure for the BNI occurred, checking a box marked “Yes” or “No,” respectively. They also assessed if two proscribed procedures inconsistent with the philosophy of the BNI and DI occurred, such as approaching the patient in a confrontational manner (BAS item #9) or referring to the patient as an “alcoholic” (BAS item #11). Three DI consistent items overlapped with the BNI items in that they were expected to occur in the BNI also (i.e., giving the patient an information sheet (BAS item #16), encouraging primary care follow-up (BAS item #17) and thanking the patient for his/her time (BAS item #18).

Table 1. BAS inter-rater reliability.

BAS Item Number/Description	Intraclass Correlation Coefficient
1/Ask Permission to discuss alcohol use	.92
2/Review drinking patterns	.93
3/Ask about connection between alcohol use & ED visit and/or other medical problems	.69
4/Make connection between alcohol use & ED visit and/or other medical problems	.03
5/Inform about NIAAA guidelines	.98
6/Use readiness ruler strategy	1.0
7/Ask why not a lower number	.97
8/Use strategy to build motivation for patient with no readiness to change	.99
9/Confront patient to cut down or quit drinking (R)	.00
10/With permission, offer suggestion about reducing or stopping alcohol use	.78
11/Refer to patient as an “alcoholic” (R)	.00
12/Negotiate a drinking goal	.89
13/Discuss benefits/problems of staying/not staying within NIAAA guidelines	.67
14/Complete a Drinking Agreement	1.0
15/With permission, offer other advice as needed	.25
16/Give patient ED information sheet	.86
17/Encourage follow-up with primary care provider	.96
18/Thank patient for discussing alcohol use	.91
19/Warn patient about alcohol use (R)	.00
20/Reflect statements that favor change	.77
21/Redirect statements that disfavor change	.33

Note: BAS=Brief Negotiation Interview Adherence Scale; ED=Emergency Department; (R)=Reverse-scored.

2.4.2 Self-reported alcohol use

Alcohol consumption was assessed via self-report using the Time-line Follow-Back (TLFB) method (Sobell & Sobell, 1992), a calendar-based assessment where patients are asked to recall their drinking behavior over the past 30 days by looking at a calendar, recording the number of drinks consumed daily starting with the previous day and working back, using holidays or special occasions as memory triggers. The reliability and validity of this instrument has been well documented (Fals-Stewart, O'Farrell, Freitas, McFarlin, & Rutigliano, 2000; Sobell & Sobell, 1992). Alcohol use outcomes derived from the TLFB included mean number of standard drinks in the past 7 days and mean number of binge drinking episodes over the past 30 days at 1-, 6- and 12-month follow-ups.

2.5 Procedures

2.5.1 Practitioner training, certification and supervision

Between March 2002 and August 2003, 58 practitioners were trained on the BNI and DI during any one of eleven, 2-hour, training sessions offered, representing 97% (58/60) of the total number of practitioners in the emergency department eligible to participate. Training procedures are described in detail in D'Onofrio et al. (2005).

Those trained were then tested in a role-play pilot case scored with the BAS. Fifty-three practitioners (91%) passed the initial testing (i.e., achieved a score of 75% adherence or higher to the BAS items), while five required remediation. Three of the five ultimately passed subsequent testing following additional training. Thus, 56 practitioners were “certified” to deliver the interventions in the randomized trial. However, only 47 of these 56 worked during the shifts when patients were recruited for the study. Thus, 47 practitioners delivered all of the BNIs and DIs for this trial.

Monitoring and supervision of BNI and DI intervention encounters during the trial included a review of that week's audiotaped BNI encounters and weekly meetings with the practitioners who had conducted a BNI that week, in which supervisors (GD or MVP) gave constructive feedback on adherence, reinforcing good BNI skills and correcting any missed or incorrectly implemented procedures. Practitioners rehearsed correct skill usage in experiential learning activities (e.g., role-plays).

2.5.2 Independent tape rater training and rating

Based on rater training methods established by Carroll and colleagues (2000), raters received a didactic seminar in which they reviewed the BAS rating manual and participated in guided rating practice of BNI and DI sessions randomly selected from the trial. Randomization was done by using a table of random numbers. A participant's intervention tape was selected for rating when their 3-digit number was closest to the last 3 digits of a random number, starting with the first value in the table of random numbers. When the last 3 digits of the random number were higher than the highest participant number, the next random number in sequence was selected until its last 3 digits were within the range of participant numbers. When more than one tape qualified for the rating, then all of the qualifying tapes were selected (e.g., random number 114283166 meant that participant number 166 was selected for tape review. Had there been no recording for participant 166, then both 165 and 167 would have been selected for review). Raters received a total of 5 hours of training. Practice ratings conducted during training (n=5) were compared against the trainer's ratings (MVP), which served as the “expert criterion.” Inter-rater agreement on all BAS items reached 100% by the fifth rating, with a mean of 85% agreement across all tapes. Next, the raters each rated another 30 randomly selected audiotapes (15 BNI and 15 DI) from the trial to calibrate the BAS's reliability. Thereafter, raters rated 77% of all the recorded sessions from the trial (165 BNI and 177 DI). We were unable to rate all sessions due to logistical constraints. Nonetheless, the final percentage rated compares favorably with other studies evaluating provider adherence scales (Carroll et al., 2000; Gibbons, Nich, Steinberg, Roffman, et al., 2010; Martino et al., 2008; Santa Ana et al., 2009).

2.5.3 Statistical analyses

We used the Kuder-Richardson alpha to determine internal consistency among the items, as suggested by Streiner (2003) for use with dichotomous (i.e., “Yes/No”) data. Based on previous work (Carroll et al., 2000; Martino et al., 2010; Santa Ana et al., 2009), the Shrout and Fleiss (1979) intraclass correlation coefficient (ICC) two-way mixed model, with item ratings as the fixed effect and raters as the random effect, was used to estimate inter-rater reliability in the calibration sample of 30 tapes.

Given its frequent use for initial scale development (Cureton & D'Agostino, 1983), exploratory factor analysis (EFA) was used to empirically derive a measurement structure using only the tape ratings from BNI encounters. The number of components retained was based on scree plot analysis and Eigen values greater than 1 (with the components accounting for more of the total variance than any single variable), as well as, the following model selection criteria based on the work of Hu & Bentler (1998) and Yu (2002): Root Mean Square Error of Approximation (RMSER) < 0.06, Root Mean Square Residual (RMSR) < 0.06, Comparative Fit Index (CFI) > 0.95, Tucker Lewis Index > 0.95, Chi-square p > 0.05, and the interpretability of the final structure. Factor loading — the geomin rotated tetrachoric correlation (a measure of linear association) between an observed dichotomous variable and an underlying factor — was used to interpret the factor structure. Loadings are equivalent to tetrachoric correlation coefficients, with a higher loading indicating a stronger relation between a factor and an observed variable. We defined factor loadings above 0.4 as indicating strong correlation (Cureton & D'Agostino, 1983). The items “Review patient's drinking…” (BAS item #2), “Inform patient about the exact NIAAA guidelines…” (BAS item #5), and “Ask patient to select a number on the Readiness Ruler” (BAS item #6) were eliminated from the factor analyses due to their strong multicolinearity. The following items were also excluded from the factor analysis: “Make a connection between alcohol use & emergency department visit and/or other medical problems” (BAS Item #4), “With permission, offer other advice as needed” (BAS item #15) and “Redirect statements that oppose change” (BAS Items #21) due to very poor inter-rater reliability (i.e., intra-class correlation coefficients), and the three reverse-scored items given that their use was prohibited in the BNI.

Given its frequent use for initial scale development (Cureton & D'Agostino, 1983), exploratory factor analysis (EFA) was used to empirically derive a measurement structure using only the tape ratings from BNI encounters. The number of components retained was based on scree plot analysis and Eigen values greater than 1 (with the components accounting for more of the total variance than any single variable), as well as, the following model selection criteria based on the work of Hu & Bentler (1998) and Yu (2002): Root Mean Square Error of Approximation (RMSER) < 0.06, Root Mean Square Residual (RMSR) < 0.06, Comparative Fit Index (CFI) > 0.95, Tucker Lewis Index > 0.95, Chi-square p > 0.05, and the interpretability of the final structure. Factor loading — the geomin rotated tetrachoric correlation (a measure of linear association) between an observed dichotomous variable and an underlying factor — was used to interpret the factor structure. Loadings are equivalent to tetrachoric correlation coefficients, with a higher loading indicating a stronger relation between a factor and an observed variable. We defined factor loadings above 0.4 as indicating strong correlation (Cureton & D'Agostino, 1983). The items “Review patient's drinking…” (BAS item #2), “Inform patient about the exact NIAAA guidelines…” (BAS item #5), and “Ask patient to select a number on the Readiness Ruler” (BAS item #6) were eliminated from the factor analyses due to their strong multicolinearity. The following items were also excluded from the factor analysis: “Make a connection between alcohol use & emergency department visit and/or other medical problems” (BAS Item #4), “With permission, offer other advice as needed” (BAS item #15) and “Redirect statements that oppose change” (BAS Items #21) due to very poor inter-rater reliability (i.e., intra-class correlation coefficients), and the three reverse-scored items given that their use was prohibited in the BNI.

A confirmatory factor analysis (CFA) was carried out to verify the structure implied by EFA and to derive factor scores, a standard psychometric approach (Cureton & D'Agostino, 1983). To test for discriminant validity, we compared percent occurrence of each individual BAS item and mean CFA factor scores between the BNI and DI recordings, using Chi-Square analyses and the Wilcoxon test for significance. For predictive validity, only the BNI group's final CFA factor scores (i.e., BNI adherence) were correlated with changes in drinking outcomes (i.e., mean number of standard drinks in the past 7 days and mean number of binge-drinking episodes in the past 30 days) at 1-, 6- and 12-month follow-ups using Spearman rank correlation. Given the dichotomous scaling of items, all factor analyses were estimated using Weighted Least Square Measurement Adjusted for Mean and Variance in M-Plus 5.0 (Muthén & Muthén, 2010).

3. Results

3.1 Reliability

Kuder-Richardson's alpha for the overall scale was 0.94, indicating excellent internal consistency reliability among the BAS items (Streiner, 2003). Table 1 presents the ICCs for the BAS items. As a general rule, ICCs below .40 are poor, .40–.59 are fair, .60–.74 are good, and . 75 or above are excellent (Cicchetti, 1994).

The results show that 13 of 21 items showed excellent inter-rater reliability. Two items (BAS item #s 3 and 13) showed good reliability. Three items (BAS item #s 4, 15, and 21) showed poor reliability (make connection between alcohol use and a medical issue, offer advice with permission, redirect statements that disfavor change). ICCs could not be calculated for the three proscribed items (BAS item #s 9-confront, 11-label alcoholic, 19-warn) because they seldom or never occurred (across conditions).

3.2 Factor Structure

The exploratory factor analysis resulted in the following 3-factor solution: Factor 1: patient-centered discussion of alcohol use; Factor 2: identifying motives and plans for change; and Factor 3: goal-setting (see Table 2 for factor loadings). However, item #10 demonstrated strong cross-loading with both Factor 2 and Factor 3 and was removed, leaving only one item in Factor 3. As a result, a 2-factor solution was selected. Whenever a practitioner implemented a BAS item with a positive loading, that action indicated adherence with its factor. However, whenever a practitioner implemented a BAS item with a negative loading, that action indicated non-adherence with its factor. Confirmatory factor analysis results, conducted to confirm the initial, exploratory structure, suggested that the data best fit a 2-factor solution for an 8-item final BNI Adherence Scale. These 2 factors, which accounted for 62% of the variance in the model, were: Factor 1: patient-centered discussion of alcohol use (BAS Item #s 1 [Ask Permission to discuss alcohol use], 3 [Ask about a connection between alcohol use and the emergency department visit and/or medical problems], 8 [Use strategy to build motivation for patients with no readiness to change, i.e., “What would make your drinking a problem for you?”], 13 [Discuss benefits/problems of staying/not staying within NIAAA guidelines] and 18 [Thank patient for discussing alcohol use]; and Factor 2: identifying motives and plans for change (BAS Item #s 7 [Ask the motivational question, “Why didn't you pick a lower number on the readiness scale?”], 14 [Complete a drinking agreement] and 17 [Encourage follow-up with a primary care provider]). Fit indices for both exploratory and confirmatory factor analysis models are presented in Table 3 and suggest that the final 2-factor structure was a very good fit. Finally, the correlation between these two factors was .50 (p < .001), which suggests the expected positive association, but not one that is completely overlapping.

Table 2. BAS factor structure.

BAS Item Number/ Short Description	EFA Loadings Factor			CFA Loadings Factor
	1	2	3	1	2
1/Ask Permission	.582			.787
3/Ask about alcohol-ED connection	.451			.621
7/Ask why not a lower number		−.667			.461
8/Use motivation strategies	1.023			.716
10/Offer suggestions		−.849	−.949
12/Negotiate a drinking Goal	.472		.682
13/Discuss benefits of reducing alcohol use	.579			.571
14/Complete a Drinking Agreement		−.870			.971
17/Encourage primary care follow-up		−.658			.708
18/Thank patient	.671		.692

Note. BAS=Brief Negotiation Interview Adherence Scale; EFA=Exploratory Factor Analysis; CFA=Confirmatory Factor Analysis; ED=Emergency Department; EFA Factor 1=Patient-Centered Discussion of Alcohol Use; EFA Factor 2=Identifying Motives & Plans for Change; EFA Factor 3=Goal-Setting; CFA Factor 1= Patient-Centered Discussion of Alcohol Use; CFA Factor 2= Identifying Motives & Plans for Change.

Table 3. BAS fit indices for exploratory and confirmatory factor analyses.

Model				Model Statistics
	X2	df	p	X2/df	CFI	TLI	RMSEA	SRMR / WRMSR
3-factor EFA	41.0	33	.160	1.24	.98	.960	.038	.074
2-factor CFA	17.3	14	.241	1.27	.97	.970	.038	.677

Note. BAS=Brief Negotiation Interview Adherence Scale; EFA=Exploratory Factor Analysis; CFA=Confirmatory Factor Analysis. In confirmatory factor analysis, the goodness-of-fit of any predicted latent structure is determined by the preponderance of several indices suggesting a well-fitted model. These fit indices include a nonsignificant chi-square (X²) value, chi-square degrees of freedom (df) ratios <2, a comparative fit index (CFI) >.90, a Tucker Lewis Index (TLI)>0.95, a root mean square error of approximation (RMSEA) ≤.0.06, a Root Mean Square Residual (RMSR) < 0.06 for EFA and a Weighted Root Mean Square Residual (WRMR) < 0.9 for CFA (Yu, 2002; Hu & Bentler, 1998). Statistics meeting these thresholds are bolded.

3.3 Intervention discriminability

Table 4 shows that the BAS individual items and factors significantly differentiated between the BNI and the DI based on differences in mean percentage occurrence of individual BAS items and in mean BAS factor scores. Almost all of the BNI consistent items occurred in significantly higher proportions of the BNI compared to the DI sessions (ps < .001), with the exception of giving the patient an information sheet (wherein DI > BNI), a prescribed DI item. Given that BNI-inconsistent items (confront, label “alcoholic” and warn) rarely occurred in either condition, no group differences were present. The factor score differences between the two conditions were also significantly different, with higher mean factor scores in BNI relative to DI (ps < .001).

Table 4. BAS scores by treatment condition.

BAS Item Number/ Short Description or Factor	Percent Recordings with Item or Mean (SD) Factor Score		Statistic (df)	p-value
	BNI	DI
1/Ask Permission	72.6	20.7	94.21	.0001
2/Review drinking patterns	94.0	2.2	293.51	.0001
3/Ask about alcohol-ED connection	87.5	0.6	267.81	.0001
4/Make alcohol-ED connection	57.1	3.4	120.82	.0001
5/Inform about NIAAA guidelines	92.8	0.6	296.99	.0001
6/Use readiness ruler strategy	96.4	1.1	315.85	.0001
7/Ask why not a lower number	65.5	0.6	186.74	.0001
8/Use motivation strategies	81.8	0.6	291.09	.0001
9/Confront patients (R)	1.8	0.0	3.22	.073
10/Offer suggestions	70.8	0.6	189.18	.0001
11/Use the term “alcoholic” (R)	0.0	0.0	--	--
12/Negotiate a drinking goal	73.2	0.6	199.21	.0001
13/Discuss benefits of reducing use	48.8	1.1	107.44	.0001
14/Complete a Drinking Agreement	94.6	0.6	308.72	.0001
15/Offer other advice as needed	30.4	0.0	63.70	.0001
16/Give patient ED information sheet	64.9	93.9	45.18	.0001
17/Encourage primary care follow-up	58.7	21.2	50.81	.0001
18/Thank patient for discussion	67.9	36.9	33.33	.0001
19/Warn patient about drinking(R)	7.1	3.4	2.53	.112
20/Reflect statements that favor change	56.0	0.0	136.75	.0001
21/Redirect statements that disfavor change	39.5	0.0	86.99	.0001
Patient-Centered Discussion of Alcohol Use Factor	3.15 (1.44)	.60 (.80)	13.8	.0001
Identifying Motives & Plans for Change Factor	2.10 (.87)	.226 (.43)	15.2	.0001

Note. BAS=Brief Negotiation Interview (BNI) Adherence Scale; DI=Discharge Instructions; ED=Emergency Department.

3.4 Relationship of BAS Factor Scores to Alcohol Use

None of the correlational analyses between BAS factor scores in the BNI group (i.e., BNI adherence) and drinking outcomes (i.e., mean self-reported number of standard drinks in the past 7 days and binge-drinking episodes in the past 30 days) at 1-, 6- or 12-month follow-ups reached statistical significance.

4. Discussion

In this study, we report on the psychometric properties of a newly developed measure of BI adherence in the emergency department, called the BNI Adherence Scale or BAS, which was used in the context of a randomized clinical trial evaluating the efficacy of BNI for harmful and hazardous drinking compared to a control condition where only discharge instructions were offered. Results demonstrated that the BAS has excellent internal consistency and good to excellent inter-rater reliability (ICCs) for more than two-thirds of its items. The ICCs for the BAS items compare favorably with those of other studies (Martino et al., 2008; Santa Ana et al., 2009) evaluating psychotherapist adherence to motivational interviewing, an approach from which many BNI techniques derive. These findings suggest that raters can be trained in five hours to reliably identify techniques used within the BNI. This amount of training time is substantially less than the 40 hours or more reported by developers of other integrity rating scales of adaptations of motivational interviewing (e.g., Martino et al., 2008), hence, learning how to use the BAS may be more acceptable and feasible to emergency department-based training supervisors. Alternatively, providing expert, rather than department-based, supervision of the BNI using the BAS may be a preferable approach for busy medical professionals who may not have the time to learn the BAS or supervise their staff with it (Ostbye, Yarnall, Krause, Pollak, Gradison, & Michener, 2005). Recent research suggests that this type of supervision for motivational interviewing may be more cost effective than establishing program-based supervisors in clinical settings (Olmstead, Carroll, Canning-Ball, & Martino, 2011).

When considering only the BNI sessions, the BNI-consistent BAS items converged to form two independent factors that appeared to overlap with the main components of the BNI, namely, discussing with patients their perspectives on their alcohol use and identifying their motives and plans for changing their drinking. In most brief interventions, the practitioner's task is to balance discussing how patients view their drinking, while simultaneously trying to enhance their motivations and plans to drink less or not at all. These two factors were significantly and positively correlated, though not completely overlapping, suggesting that the efficient balancing of both skill sets may be important in BIs such as the BNI, just as it is in the broader approach of motivational interviewing (Miller & Rose, 2009) upon which most BIs are based.

The study also showed that BAS was able to discriminate among emergency practitioners when they implemented BNI-specific procedures versus DI ones. The study included a large number of different medical practitioners (physicians, residents, physician assistants) who were trained in the BNI and supervised with the BAS. Consistent with prior studies (D'Onofrio et al., 2005; D'Onofrio et al., 2008; Tetrault, Green, Martino, Thung, et al., 2012; Ryan, Martel, Pantalon, Martino et al., 2012), the findings suggests that emergency department practitioners can learn to deliver the BNI or other BIs with sufficient adherence.

Higher BNI adherence in the BNI group was not associated with better patient outcomes (i.e., mean number of standard drinks in the past 7 days and mean number of binge-drinking episodes in the past 30 days) at any of the follow-up points in this study. This finding may be due in part to the restriction in the ranges of the outcome variables given the relatively similar treatment effects found in both conditions. Additional factors potentially moderating BAS adherence-patient outcome relations, and not measured in our study, include important third variable confounds such as therapeutic alliance, different levels of practitioner responsiveness to patients using the BNI depending on how they present in the emergency department, and unknown processes that affect outcome post-intervention that more directly affect follow-up outcomes (Webb, DeRubeis, & Barber, 2010). Future studies attempting to establish BI adherence-outcome relations are thus needed, especially those that examine the above issues. Establishing criterion-based performance benchmarks empirically linked to patient outcomes for addiction psychotherapies, including BIs, remains a challenge to the field.

4.1 Limitations

The study has several limitations. First, the study was conducted in a health-care setting where brief intervention research on alcohol problems is frequently conducted, thereby limiting the generalizability of the findings. Second, for efficiency and ease of administration, our scale only assessed adherence (i.e., whether or not a prescribed BNI action was done) and not competence, which addresses how well a given action was implemented. Though competence presupposes adherence, competence is a separate construct that may separately influence the BI effects (Perepletchikova, Treat, & Kazdin, 2007). The BAS does not address this dimension of practitioner performance. Third, and again, for the sake of efficiency and ease of administration, we assessed adherence dichotomously (yes/no) rather than dimensionally as in a Likert-type scale. This approach limited our capacity to determine the degree to which practitioners adhered to the BNI (e.g., a little, somewhat, a lot, etc.) and may have affected our findings. Fourth, while the BAS focused on a BNI for alcohol use in an emergency department setting, how well the BAS would work if modified for drug interventions awaits further study. Fifth, to the extent that BIs differ from the BNI, the generalizability of our study's findings and the applicability of the BAS would be reduced. However, a recent review of BIs concluded that current models of BI that are being taught to resident physicians nationally are more similar than different (Pringle, Kowalchuk, Adams-Meyers, & Seale, 2012).

4.2 Conclusions

This study provides a critical first step toward developing a psychometrically sound measure of BNI adherence, one that demonstrated the BAS's good to excellent reliability and construct and discriminant validity. The findings suggest that emergency department practitioners can learn to perform the BNI as it was intended to be implemented. The BAS provides an objective method for feedback during skills-based teaching sessions and evaluation of BNI skill acquisition, and for establishing the integrity of BNI interventions within clinical trials. Future studies of the BAS are needed to examine BNI adherence-patient outcome relations to establish the scale's predictive validity. In addition, the feasibility and acceptability of training supervisors based in emergency departments to use the BAS and support highly adherent BNI needs more investigation.