An interventionist adherence scale for a specialized brief negotiation interview focused on treatment engagement for opioid use disorders

Abstract

Background:

No psychometrically validated instrument for evaluating the extent to which interventionists correctly implement brief interventions designed to motivate treatment engagement for opioid use disorders has been reported in the literature. The objective of this study was to develop and examine the psychometric properties of the Brief Negotiation Interview (BNI) Adherence Scale for Opioid Use Disorders (BAS-O).

Methods:

In the context of a randomized controlled trial evaluating the efficacy of 3 models of emergency department care for opioid use disorders, the authors developed and subsequently examined the psychometric properties of the BAS-O, a 38-item scale that required raters to answer whether or not (“Yes” or “No”) each of the critical actions of the BNI was correctly implemented by the research interventionist. BAS-O items pertained to the BNI’s 4 steps: (1) Raise the Subject, (2) Provide Feedback, (3) Enhance Motivation, and (4) Negotiate and Advise. A total of 215 audio-recorded BNI and 88 control encounters were rated by 3 trained raters who were independent of the study team and blind to study hypotheses, treatment, and assignment.

Results:

The results indicated the BAS-O has fair to excellent psychometric properties, in terms of good internal consistency, excellent interrater reliability, discriminant validity, and construct validity, and fair predictive validity. A 13-item, 2-factor solution accounted for nearly 80% of the variance, where factor 1 addressed “Autonomy and Planning” (7 items) and factor 2 addressed “Motivation and Problems” (6 items). However, predictive validity was found for only one of the BAS-O factor items (i.e., Telling patients that treatment will address a range of issues related to their opioid use disorder).

Conclusions:

This study suggests that the BAS-O is a psychometrically valid measure of adherence to the specialized BNI for motivating treatment engagement in patients with opioid use disorders, thus providing a brief (13-item), objective method of evaluating BNI skill performance.

Introduction

In clinical trials, researchers have demonstrated that brief interventions, which usually include elements of motivational interviewing, autonomy enhancement, patient education, advice, and behavioral contracting,¹ can motivate individuals with harmful and hazardous drinking seen in the emergency department to reduce their alcohol consumption^1–3 and health care–related costs.^4–6 A more recent study has also demonstrated that emergency department patients with opioid use disorders who received a specialized brief intervention called the Brief Negotiation Interview, or BNI, in combination with buprenorphine/naloxone, engaged in addiction treatment at a significantly greater rate than patients who received the BNI and a facilitated referral or simply a referral to community-based treatment.⁷

Historically, researchers who have demonstrated the efficacy of a psychosocial intervention have often developed an accompanying adherence and competence scale to measure the degree to which practitioners implement the intervention as intended (adherence) and how well they do so (competence). This instrument helps to guide training prior to the commencement of the study, and intervention implementation and supervision during the study. Notable examples of such scales in the substance abuse literature are the Yale Adherence and Competence Scale,⁸ the Medical Management Adherence and Competence Scale,⁹ and the Behavior Change Counseling Index.¹⁰ Being able to measure the degree to which a practitioner adheres to an intervention protocol is as important to the training, implementation, and supervision of the BNI as it is to other addiction-focused psychosocial interventions.^11,12 However, no such adherence measure exists for the BNI for opioid use disorders—the only brief intervention that has been found to be efficacious with patients with these conditions. Although it might seem that existing measures that focus on addiction, motivational interviewing, and/or medical management broadly could be used for this purpose, this specialized BNI is so tailored to the brevity of the encounter in the emergency department and the specific needs of patients with opioid use disorder that no combination of these would serve the purpose satisfactorily.

Thus, we adapted the psychometrically validated BNI Adherence Scale (BAS) for harmful and hazardous drinking in the emergency department.¹³ The BAS has been used in the emergency department and assesses adherence to BNI techniques that have been specially tailored to be utilized in a highly efficient manner (e.g., a partial decisional balance). The original BAS has also been shown to have (1) excellent internal consistency and discriminant validity, (2) good to excellent interrater reliability, and (3) good construct validity, with an 8-item, 2-factor structure accounting for 62% of the variance, but (4) no predictive validity. The current study describes the revision and initial psychometric properties of BAS for use with patients with opioid use disorders (BAS-O) seen in the emergency department. This work was conducted in the context of a randomized controlled trial evaluating the efficacy of 3 models of emergency department care for opioid use disorders.⁷

Specifically, we report on the development and psychometric properties (i.e., reliability and validity) of the BAS-O. Consistent with previous research in the area,^8,14 we examined the internal consistency of the items, interrater reliability, and scale construct, discriminant, and predictive validity.

We hypothesized that the BAS-O items would (1) have good internal consistency, (2) show good rating agreement among independent raters, (3) converge to form independent factors corresponding to the key components of the BNI, (4) discriminate between recordings of BNI versus control encounters, and(5) be positively associated with addiction treatment engagement and negatively associated with drug use at the 1-month follow-up.

Methods

Overview of the emergency department–initiated BNI+buprenorphine trial

A randomized controlled trial was conducted in an urban teaching hospital emergency department evaluating the efficacy of 3 models of emergency department care for opioid use disorders, namely, (1) screening and referral (Referral); (2) screening, brief intervention (i.e., a specialized Brief Negotiation Interview, or BNI, focused on treatment engagement) and facilitated referral to community-based treatment services (BNI); and (3) screening, BNI, emergency department–initiated treatment with buprenorphine/naloxone, and referral to primary care for 10 weeks of follow-up buprenorphine/naloxone maintenance (BNI+buprenorphine). See the report of this trial’s outcomes for a full description of the design.⁷

Interventions

The specialized BNI focused on treatment engagement for opioid use disorders

This BNI, which has been previously described,¹⁵ is based in part on a brief adaptation of motivational interviewing¹⁶ for use in health care settings, combined with autonomy enhancement, patient education about opioid use disorder diagnoses and treatment, advice, and behavioral contracting.

The components of the BNI include (1) raising the subject of and discussing opioid use in a patient-centered manner, including the reinforcement of autonomy, and reviewing diagnosis and treatment options, much like is done in typical patient education in medical settings; (2) providing feedback on problems related to opioid use and how engaging in treatment could help (e.g., ask about and make a connection between opioid use and the emergency department visit/other illness or injury, including human immunodeficiency virus/acquired immunodeficiency virus [HIV/AIDS] risk); (3) eliciting and reflecting on motives for engaging in formal addiction treatment in the emergency department and post emergency department (e.g., having patients self-identify their readiness to engage in emergency department and primary care buprenorphine [for BNI+buprenorphine patients] or a variety of treatment options [BNI patients] on a scaled ruler); and (4) negotiating a change plan and advising as needed (to follow-up with the relevant addiction treatment referral; BNI training manual available from the first author). The BNI takes 10–15 minutes to complete. Patients in the BNI and BNI+buprenorphine groups received the same BNI protocol, except that in the former, the research interventionist secured a treatment referral for the patient based on his/her preferences, insurance, and availability, whereas in the latter, patients were referred to the Yale New Haven Hospital Primary Care Center for buprenorphine maintenance. All of the BNI encounters were audio-recorded.

Referral

The control condition in the trial, Referral, required the research interventionist to (1) introduce him/herself; (2) provide patients with a handout providing names, locations, and telephone numbers of addiction treatment services in the area and telephone access to call a clinician or facility of their choice, which were categorized according to preference and the insurance plans in which they participated; (3) ask if the patient had any questions; and (4) thank the patients for their attention. No motivational or patient education components were included in the Referral condition. The addiction services offered to Referral patients included a range of treatments with varying intensity and duration, including opioid treatment programs, inpatient or residential treatment, and outpatient services, including intensive outpatient programs and office-based physicians who prescribe buprenorphine or other forms of medication-assisted treatment. When conducting Referral, the research interventionists were asked not to use other BNI techniques. The Referral was designed to take 2 minutes to complete. All of the Referral conversations were audio-recorded.

Participants

Participants in the current study were emergency department research interventionists, emergency department patients, and independent audio-recording raters, all of whom were part of the aforementioned randomized controlled clinical trial of the 3 brief interventions for moderate to severe opioid use disorders.⁷ All of the patients gave consent for participation in the study according to the rules and policies of the Yale University School of Medicine Human Investigation Committee, which approved the study.

Emergency department research interventionists

The same 20 research interventionists implemented both psychosocial interventions in the study, namely, the BNI and Referral. None of the interventionists were licensed health care providers. Instead, they were emergency department technicians, counselors, social workers, case managers, and public health or physician assistant graduate students. All of the interventionists had the same level of exposure to motivational interviewing (i.e., a brief introductory workshop of 1.5–3 hours), and none had any experience using it in previous clinical work or employment.

Emergency department patients

Adult patients 18 years or older were screened with a 20-item Health Quiz. Those who reported nonmedical use of prescription opioids or any heroin use in the past 30 days were further evaluated with the Mini-International Neuropsychiatric Interview¹⁷ to evaluate for opioid dependence using Diagnostic and Statistical Manual of Mental Disorders (Fourth Edition, Text Revision; DSM-IV-TR)¹⁸ criteria. Patients with a urine sample that tested positive for opioids (opiates or oxycodone) and a Mini-International Neuropsychiatric Interview score 3 or higher were considered to have met criteria for opioid dependence and were eligible for inclusion. Research associates reviewed the study procedures and protocol and obtained signed informed consent from those interested in participation.

Independent audio-recording raters

Three raters rated the audio-recorded brief intervention encounters. They were considered “independent” raters because they were blinded to the study protocol, randomization of patients, selection of emergency department interventionists, hypotheses, and outcome measures. All 3 raters had master’s degrees and worked in medical or mental health settings. Additionally, all raters were substance abuse clinicians who had previously served as independent treatment adherence raters in at least 6 prior addiction treatment studies.

Assessment measures

BNI Adherence Scale for Opioid Use Disorders (BAS-O)

The development of the BAS-O mirrored prior work by Carroll and colleagues in the area of treatment adherence and competence with substance abuse psychotherapies⁸ and extended the work previously reported on the original BAS.¹³ Specifically, the current investigators (1) generated 2–3 candidate items for each of the BNI techniques (see BNI description above) based on the BAS and the BNI for opioid use disorders manual; and(2) piloted this initial, longer version of the scale in the emergency department by scoring the research interventionist’s BNI implementations (n = 5) to evaluate the usefulness and clarity of each of the items. Following these ratings, items that were unclear, difficult to rate, or redundant were either revised or removed from the scale by author consensus, resulting in the next version of the BAS-O. The BAS-O at this stage was a 38-item scale that required raters to answer, for all but 2 of the items, whether or not (“Yes” or “No”) each of the critical actions of the BNI for opioid use disorders was correctly (i.e., with at least fair to good competence, as defined in the manual) implemented by the research interventionist (see Table 1). Items pertained to each of the 4 steps of the BNI: (1) Raise the Subject, (2) Provide Feedback, (3) Enhance Motivation (i.e., to engage in addiction treatment post emergency department visit), and (4) Negotiate and Advise. The exceptions to the Yes/No response set were as follows. Item 16 asked the rater to indicate the number patients selected on a 1–10 scale measuring their readiness to engage in addiction treatment, and item 37 asked raters to rate, on a 1–7 Likert-type scale, the degree to which research interventionists used reflective listening and/or open-ended questions in response to motivational statements made by the patient, with higher numbers indicating higher readiness and frequency, respectively. Raters also assessed if 6 proscribed procedures inconsistent with the philosophy of the BNI and Referral occurred, such as approaching the patient in a confrontational manner (BAS-O items 21 and 33) or referring to the patient as an “addict” (BAS-O item 27). Four Referral consistent items overlapped with the BNI items in that they were expected to occur in the BNI also (i.e., introducing oneself as a research interventionist [BAS-O item 1a], giving the patient an information sheet on referrals [BAS-O item 35], asking if the patient has any questions [BAS-O item 36], and thanking the patient for his/her time [BAS-O item 36]).

Table 1.

BAS-O interrater reliability.

BAS item number/description	Intraclass correlation coefficient
1a. Introduce him/herself	1.0
1b. Ask Permission to discuss use of opioids	1.0
2. Explain that the RI was there to help with opioid	.93
problems
3. Ask about physical discomfort related to opioid use	1.0
4. Review patterns of opioid use	1.0
5. Ask about other problems related to opioid use	1.0
6. Express concerns about patient’s opioid use	.94
7. Ask about the connection between opioid use and ED visit	1.0
8. Make a connection between opioid use and ED visit	.80
or other medical issues
9. Ask about connection between opioid use and HIV/AIDS	1.0
10. Explain how opioid use increases risk of HIV/AIDS	.94
11. Explain what a “facilitated referral” (BNI) entails	1.0
12. Explain what “standard care” (Referral) entails	.91
13. Explain what buprenorphine treatment (BNI+buprenorphine) entails	1.0
14. Tell the patient that if he/she accepts the referral, it would help with many problems related to opioid use	1.0
15. Ask the patient to select number on readiness ruler about motivation to engage in addiction treatment post ED discharge	1.0
16. What was the number the patient selected on the readiness ruler?	.89
17. Ask why the patient did not pick a lower number on the ruler	1.0
18. Ask why the patient did not pick a higher number (R)	1.0
19. Ask why the reason the patient gave to #17 is important to him/her	.99
20. Ask the patient what it would take for a “1” on the readiness ruler to turn into a “2”	.40
21. Tell the patient in a confrontational way that he/she has to stop using opioids or to enroll in treatment (R)	1.0
22. Make suggestions regarding how to the patient should stop or reduce opioid use (R)	1.0
23. Ask the patient what his/her next step regarding the referral might be	1.0
24. Summarize the patient’s reasons for entering into treatment?	1.0
25. Tell the patient that it is up to the him/her whether or not to engage in treatment	.94
26. Ask the patient to describe the types of treatment(s) that have been or would be helpful to him/her?	.48
27. Refer to the patient as an “addict”, “junkie” etc. (R)	1.0
28. Explain why buprenorphine was better than methadone? (R)	1.0
29. Address the patient’s upset over not getting their preferred treatment	.70
30. Tell the patient that if he/she starts treatment they will have begun their recovery?	.92
31. Provide and have the patient fully complete a treatment agreement sheet?	1.0
32. Add the RI’s advice regarding enrolling in treatment?	.10
33. Offer confrontational warnings regarding future opioid use? (R)	1.0
34. Provide “Project ED Health” Referral sheet?	.57
35. Ask if the patient had any questions?	.89
36. Thank patient for his/her time?	.89
37. How well did the RI reflect or ask questions about the patient’s motivational statements	.33

Note. N = 303. BAS-O = Brief Negotiation Interview Adherence Scale for Opioid Use Disorders; RI = research interventionist; (R) = reverse-scored.

Treatment engagement

The primary outcome, engagement in post–emergency department addiction treatment, was defined as enrollment and receiving formal addiction treatment on the 30th day following randomization, assessed by direct contact with the facility, clinician, or both. Formal addiction treatment included any of a range of clinical settings, including an opioid treatment program, inpatient or residential treatment, or outpatient services, including intensive outpatient programs or office-based physicians who prescribe buprenorphine or other forms of medication-assisted treatment.

Procedures

Research interventionist training, certification, and supervision

Between March 2009 and June 2013, 20 interventionists were trained on the BNI and Referral during any one of twelve, 2-hour, training sessions offered, representing 100% (20/20) of the total number of interventionists in the emergency department eligible to participate. Training procedures have previously been described in detail.¹⁵

Once trained, research interventionists were then tested in a role-play pilot case scored with the BAS-O. All 20 interventionists (100%) passed the initial testing (i.e., achieved a score of 75% adherence or higher on the BAS-O items, a score that was associated with positive outcomes in a previous BNI trial¹). Thus, 20 research interventionists were “certified” to deliver the interventions in the randomized trial. These 20 research interventionists delivered all of the BNIs and Referrals for this trial. The difference in number of BNIs conducted by each research interventionist throughout the trial (mean = 11.2, SD = 7.7) was not statistically significant (P = .83).

Monitoring and supervision of BNI and Referral intervention encounters during the trial included a review of that week’s audio-recorded BNI and Referral encounters and weekly meetings with all of the research interventionists, in which their supervisor gave them constructive feedback on adherence, reinforcing good BNI skills, and correcting any missed or incorrectly implemented procedures, as well as instructing on how to avoid using BNI techniques in Referral encounters. Research interventionists rehearsed correct skill usage in experiential learning activities (e.g., role-plays).

Independent audio-recording rater training and rating

Based on rater training methods established previously,⁸ raters received a didactic seminar in which they reviewed the BAS-O rating manual and participated in guided rating practice of BNI and Referral sessions randomly selected from the trial, using a table of random numbers. Raters received a total of 5 hours of training. Practice ratings conducted during training (n = 5) were compared against the trainer’s ratings, which served as the “expert criterion.” Interrater agreement on all BAS-O items reached 100% by the fifth rating, with a mean of 85% agreement across all tapes. Next, the raters each rated another 10 randomly selected audio recordings (5 BNI and 5 Referral) from the trial to calibrate the BAS-O’s reliability. Thereafter, raters rated 92% of all the recorded sessions from the trial (215 BNI and 88 Referral). We were unable to rate all sessions due to logistical constraints, patient refusals, and technical malfunctions. Nonetheless, the final percentage rated compares very favorably with other studies evaluating provider adherence scales.^8,19,20

Statistical analyses

Reliability

We used the Kuder-Richardson alpha to determine internal consistency among the items, as suggested by other researchers for use with dichotomous (i.e., Yes/No) data.²¹ Although the Cronbach alpha might in general be an appropriate measure of internal consistency for the 2 continuous BAS-O items (items 16 and 37), in this case it is not because these items were not expected to correlate, given that one is a rating of patient motivation and the other is a rating of how frequently the interventionist used reflective listening or open-ended questions. Thus, there was no measure of internal consistency for these 2 items. Based on previous work,^8,19,20 the Shrout and Fleiss²² intraclass correlation coefficient (ICC) 2-way mixed model, with item ratings as the fixed effect and raters as the random effect, was used to estimate interrater reliability in the calibration sample of 30 recordings (i.e., 10 encounters × 3 raters).

Validity

Given its frequent use for initial scale development,²³ exploratory factor analysis (EFA) was used to empirically derive a measurement structure (i.e., construct validity) using only the audio-recording ratings from BNI encounters. The number of components retained was based on scree plot analysis and Eigen values greater than 1 (with the components accounting for more of the total variance than any single variable), as well as the following model selection criteria based on previous work^24,25: root mean square error of approximation (RMSEA) < .06, root mean square residual (RMSR) < .06, comparative fit index (CFI) > .95, Tucker-Lewis index > .95, chi-square P > .05, and the interpretability of the final structure. Factor loading—the geomin rotated tetrachoric correlation (a measure of linear association) between an observed dichotomous variable and an underlying factor—was used to interpret the factor structure. Loadings are equivalent to tetrachoric correlation coefficients, with a higher loading indicating a stronger relation between a factor and an observed variable. We defined factor loadings above .4 as indicating strong correlation.²³ A confirmatory factor analysis was carried out to verify the structure implied by EFA and to derive factor scores, a standard psychometric approach.²³ Given the dichotomous scaling of items, all factor analyses were estimated using weighted least square measurement adjusted for mean and variance in M-Plus 5.0.²⁶

To test for discriminant validity, we compared percent occurrence of each individual BAS-O item and mean CFA factor scores between the BNI (combining BNI and BNI+buprenorphine ratings) and Referral recordings, using chi-square analyses and the Wilcoxon test for significance. For predictive validity, we evaluated differences between BNI patients who were engaged in treatment at 30 days versus those who were not on BAS-O factor scores, using independent t tests, and on each of the individual items within those 2 factors, using chi-square analyses.

Results

Reliability

Kuder-Richardson alpha for the overall scale was .82, indicating good internal consistency reliability among the BAS-O items.²¹ Table 1 presents the ICCs for the BAS-O items. As a general rule, ICCs below .40 are poor, .40–.59 are fair, .60–.74 are good, and .75 or above are excellent.²⁷ The results show that 33 of 38 items showed excellent interrater reliability. One item (item 29: Address the patient’s upset over not getting his/her preferred treatment) showed good reliability. Three items showed fair reliability (item 20: Ask the patient what it would take for a “1” on the readiness ruler to turn into a “2”; item 26: Ask the patient to describe the types of treatment(s) that have been or would be helpful to him/her; and item 34: Provide the patient with “Project ED Health” Referral sheet), and one item had poor reliability (item 32: Add the research interventionist’s advice regarding enrolling in treatment).

Factor structure

Before running the exploratory factor analysis, we removed 18 items because they either (1) lacked of variability (e.g., only 3 out of 215 BNI recordings had a rating of “Yes” for item 8); or(2) perfectly or nearly perfectly correlated with each other so that they basically conveyed same information (e.g., items 8 and 9). Thus, we performed the exploratory factor analysis with the remaining 19 items. A 3-factor exploratory factor analysis achieved a satisfactory model fit. We removed 6 additional items in confirmatory factor analysis because they (1) cross-loaded on multiple factors, (2) had a loading of <0.4 on all factors, or (3) were the only item in a given factor (e.g., item 32).

The exploratory factor analysis resulted in the following 3-factor solution for the BNI recordings only (N = 215): factor 1: “Autonomy and Planning”; factor 2: “Referral Advice”; and factor 3: “Motivation and Problems” (see Table 2 for factor loadings). Whenever a research interventionist implemented a BAS-O item with a positive loading, that action indicated adherence with its factor. Confirmatory factor analysis results (on the same 215 recordings), conducted to confirm the initial, exploratory structure, suggested that the data best fit a 2-factor solution for a 13-item final BNI Adherence Scale for Opioid Use Disorders, or BAS-O.

Table 2.

BAS-O factor structure.

	EFA loadings Factors			CFA loadings Factors
BAS-O item number/short description	1	2	3	1	2
1a. Introduce	.61			.58
1b. Ask permission	.71			.80
2. RI there to help	.91			.77
7. Ask about connection			.90		.90
8. Make a connection			.66		.62
10. Explain HIV risk			.66		.65
14. Help with other aspects	.61			.64
17. Why not lower number			.58		.54
23. Next step	.66			.70
24. Summarize reasons			.55		.76
25. Patient’s choice	.62			.73
26. Preferred treatment		.85
30. Recovery has begun	.61			.62
32. RI’s advice			.57
35. Questions			.76		.80

Note. N = 215. BAS-O D Brief Negotiation Interview Adherence Scale for Opioid Use Disorders; EFA = exploratory factor analysis; CFA = confirmatory factor analysis; EFA factor 1 = Autonomy and Planning; EFA factor 2 = Referral Advice; EFA factor 3 = Motivation and Problems; CFA factor 1 = Autonomy and Planning; CFA factor 2 = Motivation and Problems; RI = research interventionist.

These 2 factors, which accounted for nearly 80% of the variance in the model, were factor 1, “Autonomy and Planning” (BAS-O items 1a [Introduce him/herself], 1b [Ask the patient for permission to discuss opioid use and pause for his/her response], 2 [Explain that the research interventionist was there to help with the patient’s opioid problems, including patient education regarding diagnosis and treatment options], 14 [State that if the patient accepts the referral, treatment would help with many of his/her problems related to opioid use], 23 [Ask, “What’s the next step, if any,” in reference to the treatment referral], 25 [Tell the patient that, while the research interventionist believes it would be very helpful, it is up to the patient whether or not to enter treatment], and 30 [Tell the patient that if he/she starts treatment he/she will have begun their recovery]) and factor 2, “Motivation and Problems” (BAS-O items 7 [Ask the patient what connection he/she sees between opioid use and the emergency department visit?], 8 [Make or reflect a specific connection between opioid use and emergency department visit or other medical issues (e.g., hepatitis, overdose], 10 [Explain how the patient’s opioid use puts him/her at a higher level of risk for contracting HIV/AIDS], 17 [Ask the patient why he/she did not pick a lower number on the readiness ruler], 24 [Summarize the patient’s reasons for entering into treatment, with an emphasis on reducing severity of DSM-IV-TR symptoms, if identified by the patient], and 35 [Ask if the patient had any questions]. Fit indices for both exploratory and confirmatory factor analysis models are presented in Table 3 and suggest that the final 2-factor structure was a very good fit.

Table 3.

BAS-O fit indices for exploratory and confirmatory factor analyses.

	Model statistics
Model	x2	df	P	x2/df	CFI	TLI	RMSEA	RMSR/WRMSR
3-Factor EFA	145.1	117	.04	1.24	.97	.95	.03	.11
2-Factor CFA	31.5	25	.17	1.26	.94	.96	.04	.79

Note. N = 215. BAS-O = Brief Negotiation Interview Adherence Scale for Opioid Use Disorders; EFA = exploratory factor analysis; CFA = confirmatory factor analysis.

In confirmatory factor analysis, the goodness-of-fit of any predicted latent structure is determined by the preponderance of several indices suggesting a well-fitted model. These fit indices include a nonsignificant chi-square (X²) value, chi-square/degrees of freedom (df) ratios <2, a comparative fit index (CFI) >.90, a Tucker-Lewis index (TLI) >.95, a root mean square error of approximation (RMSEA) <.06, a Root mean square residual (RMSR) <.06 for EFA and a weighted root mean square residual (WRMSR) <.90 for CFA.^24,25 Statistics meeting these thresholds are bolded.

Finally, the correlation between these 2 factors was .72 (P < .0001), which suggests the expected positive association, but not one that is completely overlapping. Additionally, Kuder-Richardson alphas for factor 1, factor 2, and the final 13-item scale (N = 215 for all) were .82, .75, and .87, respectively, indicating excellent internal consistency.

Intervention discriminability

Table 4 shows that the BAS-O individual items and factors significantly differentiated between the BNI and the Referral recordings based on differences in mean percentage occurrence of individual BAS-O items and in mean BAS-O factor scores. As predicted, almost all of the BNI-consistent items occurred in significantly higher percentages of the BNI compared with the Referral recordings (Ps < .0001), with the exception of item 12, explaining standard care or Referral (wherein Referral > BNI), the only prescribed Referral item that did not overlap with prescribed BNI items. Additionally, asking about a patient’s preferred type of treatment referral occurred at a similar rate in Referral as in the BNI (without buprenorphine), as they were prescribed in both conditions. As expected, and given that BNI-inconsistent items (confrontation, focusing on opioid use reduction vs. engagement, use of the term “addict,” discussing buprenorphine vs. methadone, and warnings) rarely occurred in either condition, no group differences were found. The factor score differences between the 2 conditions were also significantly different, with higher mean factor scores in BNI relative to Referral (Ps < .0001).

Table 4.

BAS-O scores by treatment condition.

	Percent recordings with item/Mean (SD) factor score
BAS item number/Short description or factor	BNI (n = 215)	Referral (n = 88)	Statistic (df)	P value
1a. Introduce	88.4	46.6	62.3 (2)	.0001
1b. Ask Permission	90.2	11.4	179.6 (2)	.0001
2. RI there to help	93.5	4.5	229.3 (2)	.0001
3. Physical discomfort	94.9	0.0	263.4 (2)	.0001
4. Patterns of opioid use	95.3	0.0	267.4 (2)	.0001
5. Other problems related to use	94.0	0.0	255.6 (2)	.0001
6. Concern about opioid use	91.6	0.0	237.3 (2)	.0001
7. Ask about connection	93.0	0.0	228.0 (2)	.0001
8. Make a connection	81.9	0.0	189.1 (2)	.0001
9. Connection between use and HIV	97.2	0.0	289.9 (2)	.0001
10. Explain HIV risk	90.2	0.0	230.5 (2)	.0001
11. Explain “facilitated referral”a	99.0	0.0	64.8 (2)	.0001
12. Explain “standard care”a	0.0	92.0	274.7 (2)	.0001
13. Explain buprenorphinea	98.2	0.0	68.9 (2)	.0001
14. Help with other aspects	84.7	0.0	191.3 (2)	.0001
15. Readiness ruler number	97.2	0.0	288.9 (2)	.0001
17*. Why not lower number	91.2	0.0	278.9 (2)	.0001
18. Why not higher number (R)	2.3	0.0	66.7 (2)	.0001
19. Reason why item 17 is important	90.2	0.0	284.4 (2)	.0001
20. What would make “1 ” a “2”	3.7	0.0	8.8 (2)	.032
21. Use confrontation (R)	0.0	0.0	.79 (2)	.45
22. Suggestions to stop. reduce use (R)	0.0	0.0	.824 (2)	.36
23. Next step	84.2	1.1	192.6(2)	.0001
24. Summarize reasons	84.7	0.0	202.7 (2)	.0001
25. Patient’s choice	77.2	0.6	150.9 (2)	.0001
26. Preferred treatmenta	50.5	20.5	18.6 (1)	.32
27. Use “addict,” “junkie” etc. (R)	0.0	0.0	1.2 (2)	.27
28. Buprenorphine vs. methadone? (R)	0.0	0.0	1.66 (2)	.20
29. Address patient’s upset	7.9	12.5	3.9 (2)	.27
30. Recovery has begun	74.9	0.0	145.0 (2)	.0001
31. Agreement sheet	95.3	0.0	275.7 (2)	.0001
32. RI’s advice	20.5	26.1	5.0 (2)	.082
33. Confrontational warnings (R)	0.0	1.1	3.1 (2)	.21
34. Provide referral sheet	13.5	81.8	131.8(2)	.0001
35. Questions	85.6	70.5	12.6 (2)	.002
36. Thank patient	60.9	42.0	13.4 (2)	.001
37. Reflections of motivation	98.1	30.6	184.1 (2)	.0001
Autonomy and Planning	3.1 (1.44)	.60 (.80)	13.8	.0001
Motivation and Problems	2.1 (.87)	.23 (.43)	15.2	.0001

Note. BNI = Brief Negotiation Interview. BAS-O = Brief Negotiation Interview (BNI) Adherence

Scale for Opioid Use Disorders; RI = research interventionist.

^aWhere applicable.

^*Item 16 is omitted given that it simply asks what number the patient selected on the readiness ruler versus an indication of research interventionist adherence.

Relationship of the BAS-O to treatment engagement

Only one of the planned analyses evaluating the association between BAS-O factor, and items scores, and treatment engagement (i.e., engaged in treatment at the 1-month follow-up, Yes/No) reached statistical significance. Specifically, participants whose BNI audio recordings included BAS-O item 14 (part of factor 1, “Autonomy and Planning,” which required the research interventionist to inform the patient that if he/she accepts the referral, treatment would help with many of his/her problems related to opioid use) were engaged at a significantly higher rate (65.4%) than participants whose recordings did not include this item (41.4%; X² = 6.41, df = 2, P < .05).

In summary, the results supportedDall of our hypotheses except with regard to predictive validity. They indicated that the BAS-O has sound psychometric properties, namely, good internal consistency and excellent interrater reliability, discriminant validity, and construct validity. A 13-item, 2-factor solution accounted for nearly 80% of the variance, where factor 1 addressed “Autonomy and Planning” (7 items) and factor 2 addressed “Motivation and Problems” (6 items). However, predictive validity was only fair, as we found an association between the BAS-O and treatment engagement with only one of the BAS-O factor items.

Discussion

In this study, we report on the psychometric properties of a newly developed measure of adherence to a specialized BNI, called the BNI Adherence Scale for Opioid Use Disorders, or BAS-O. The BAS-O was used to assess the implementation of the BNI, which was designed to motivate engagement in treatment generally, and in buprenorphine specifically, for patients with moderate to severe opioid use disorders seen in the emergency department. The results of the current study demonstrate that the BAS-O has good internal consistency and excellent interrater reliability coefficients for all but 5 of its items. These findings that compare very favorably with those of other studies of interventionist adherence to motivational interviewing.^13,19,20

When considering only the BNI encounters (in both the BNI and BNI+buprenorphine conditions, N = 215), the BNI-consistent BAS-O items converged to form 2 independent factors that appeared to overlap with the main components of the BNI, namely, discussing with patients their autonomy, or freedom-of-choice, regarding what their next steps might be, and their planning of those next steps (factor 1), and discussing their motivations for treatment engagement, as well as the problems they have suffered because of their opioid use disorder (factor 2). In most brief interventions, the interventionist’s task is to balance discussing the patient’s wish to address their opioid use disorder, focusing on a future that will bring improvements in a number of areas related to opioid use, while simultaneously trying to elicit and reflect their motives for engaging in treatment, which in an emergency department setting often revolve around the negative consequences of the disorder. These 2 factors were significantly and positively correlated, although not completely overlapping, suggesting that the efficient balancing of both skill sets may be important in this specialized BNI, just as it is in the broader approach of motivational interviewing,²⁸ upon which many brief interventions are based.

The study also showed that BAS-O was able to discriminate among emergency department research interventionists when they implemented BNI- versus Referral-specific procedures. Consistent with prior studies,^{1,13,15,29–31} the findings suggest that emergency department research interventionists can learn to deliver this specialized BNI with sufficient adherence. Although not the focus of the current investigation, the high adherence in Table 4 (i.e., the 13 items of the final BAS-O were implemented in 86.2% of the BNI recordings) also suggests that interventionists can be effectively trained in 2 hours to implement the BNI. This is substantially less than the 40 or more hours reported previously for other adaptations of motivational interviewing,¹⁹ which may make it a more feasible intervention for use in busy medical settings.³² Further, all of the above results indicate that raters too can be trained efficiently and effectively, in this case, in less than 5 hours.

Higher BNI adherence was only associated with better patient outcomes (i.e., engagement in addiction treatment at the 1-month follow-up) on only one item (item 14), where greater engagement was noted for patients who were told that treatment would help with a range of problems related to the opioid use disorder versus those who were not. Although it is important not to overstate the significance of this single predictive validity finding, it is notable that an item that reassures patients that all of their concerns will be taken into account by a treatment provider is the item that is associated with treatment engagement. This may be because it is more motivating when patients believe all or most of the issues they (vs. treaters) are concerned with will be addressed, as would be predicted by the highly client-centered motivational interviewing.³³ Further, this may also be the case given that of the nearly 90% of the 25 million US citizens with an addiction who are not in treatment report that the primary reason for this is that they are not ready to completely stop their use,³⁴ which may be based on the often correct expectation that treatment providers will focus on an complete cessation of drug use (versus help to address the negative consequences of use that are important to the patient).

The fact that there were no other such significant findings is not unexpected, as most adherence-outcome studies show no such significant relationship.³⁵ The lack of additional significant predictive validity findings may be due in part to the restriction in the ranges of the predictor variables (BAS-O items), given that there was a very high rate of implementation of all of the BAS-O items, leading to little power to detect a significant difference between recordings that included versus did not include BAS-O items. Other factors that may have affected BAS-O adherence–patient outcome relationships, which were not measured in our study, are therapeutic alliance and unknown processes that affect outcome post intervention.³⁵ Future studies attempting to establish BNI adherence-outcome relations are thus needed, especially those that examine or control for the above issues.

Limitations

The study has several limitations. First, the study was conducted in a health care setting where brief intervention research on substance-related problems is frequently conducted, thereby limiting the generalizability of the findings.

Second, for the sake of efficiency and ease of administration, for the most part, we assessed adherence dichotomously (Yes/No) rather than dimensionally as in a Likert-type scale. This approach limited our capacity to determine the degree to which interventionists adhered to the BNI (e.g., a little, somewhat, a lot, etc.) and may have affected our findings. However, it is worth reiterating that one of the Likert-type BAS-O items had poor interrater reliability.

Third, for brevity sake, we assessed adherence and competence in a combined fashion, where raters could only indicate a “Yes” for the presence of a BAS-O item if it exceeded the threshold of fair-good competence. However, different ratings, and hence different results, may have been found had we asked raters to assess the adherence and competence dimensions separately.

Fourth, the BAS-O focused on patients with moderate to severe opioid use disorder seen in an emergency department setting, approximately half of whom also received buprenorphine in that setting and subsequently in primary care. How well the BAS-O would work if modified for mild opioid use disorder awaits further study. Fifth, the generalizability of our findings is limited by the degree to which the BAS-O focused on a specialized BNI aimed at motivating treatment engagement. Thus, it cannot be used with other, either more generic brief interventions or brief interventions that do not specifically focus on moderate to severe opioid use disorders, including physical dependence, as defined by the DSM-IV-TR, and motivational strategies for treatment engagement (vs. use reduction), or do not include the type of patient education around opioid use disorder diagnosis, treatment, and course that is typically offered for other chronic medical diseases.

Conclusions

This study provides a critical first step toward developing a psychometrically sound measure of adherence to a specialized BNI focused on motivating opioid addiction treatment engagement among emergency department patients with moderate to severe opioid use disorders, namely, the BAS-O. The BAS-O demonstrated good internal consistency, excellent interrater reliability, construct validity, and discriminant validity, and fair predictive validity. The findings suggest that emergency department research interventionists can learn to perform this BNI as it was intended. The BAS-O provides a brief (13-item) and objective method for feedback during skills-based teaching sessions and evaluation of BNI skill acquisition, and for establishing the integrity of BNI interventions within clinical trials. Future studies of the BAS-O are needed to more fully examine its psychometric properties, in general, and its predictive validity, in particular, which would serve to more clearly elucidate any BNI adherence–patient outcome relationships.