Development and Initial Evaluation of the Brief Addiction Monitor (BAM)

Abstract

This project developed and tested a 17-item monitoring instrument covering important substance use related behaviors to support measurement-based care and outcomes assessment. The study consisted of two phases, an instrument development phase and an initial study to examine its psychometric properties. Participants were 175 patients entering VA outpatient substance abuse treatment. The findings revealed that this Brief Addiction Monitor (BAM) exhibited acceptable characteristics. Exploratory factor analysis yielded three summary factors; Recovery Protection, Physical & Psychological Problems, and Substance Use & Risk. The RMSEA estimate was acceptable and the factors had alpha values exceeding or approaching 0.70. All three factors were sensitive to change and had excellent test-retest reliability. Predictive validity was demonstrated for two factors, Recovery Protection and Substance Use & Risk. At the item level, there was little indication of inappropriate response patterns. Change over time was significant for most items, and test-retest reliability was acceptable for nearly all items. Additional research is warranted to further establish the BAM’s reliability, validity and usefulness.

1. INTRODUCTION

Substance dependence (SD) is increasingly conceptualized as a chronic disease (Compton, Glantz, & Delany, 2003; McKay, Lynch, Shepard, & Pettinati, 2005; McLellan, Lewis, O’Brien, & Kleber, 2000; McLellan, McKay, Forman, Cacciola, & Kemp, 2005; Scott & Dennis, 2009). As currently treated it has a high relapse rate and is characterized by periods of problematic use and/or functional impairment alternating with periods of remission or less problematic use (Anglin, Hser, & Grella, 1997; McKay & Weiss, 2001; Vaillant, 2003). Measurement-based care (Valenstein et al., 2009) is an essential component of treatment for many chronic physical and mental health conditions, but it is not typically used in the treatment of substance use disorders (SUDs) (McKay, 2009; Murphy, Lynch, Oslin, McKay, & TenHave, 2007). For example, hypertension, asthma/COPD, and diabetes have standard and objective tests that are routinely administered during clinical visits and used to provide a readily accessible summary of symptom/disease status to help guide treatment (Institute of Medicine, 2006; Valenstein et al., 2009). In general psychiatry, it has been demonstrated that when clinicians repeatedly measure and monitor patients’ during-treatment status and change over time, the effects of outpatient mental health treatment are enhanced (Lambert et al., 2003). One way to promote individualized, adaptive and continuous care for individuals with SUDs is to systematically monitor patient progress during substance abuse treatment (SAT).

The Institute of Medicine (IOM) has explicitly recommended the development and implementation of patient monitoring systems in SAT (Institute of Medicine, 2006) Moreover, SAT programs have been repeatedly called upon to implement outcomes monitoring systems (OMSs) as a way to justify their effectiveness and improve their performance. In the U.S., federal agencies have required and assisted states in large scale efforts to develop and implement OMSs and report outcomes data [e.g., Substance Abuse and Mental Health Services Administration’s Treatment Episode Data Set (TEDS) and National Outcome Measures (NOMs); and Center for Substance Abuse Treatment’s Government Performance and Results Act (GPRA) data collection tool, and Treatment Outcomes Performance Pilot Studies (TOPPS I & II)]. Likewise, the U.S. Department of Veterans Affairs (VA) implemented a system in 1997 for clinical staff to monitor outcomes of all new substance abusing patients using the Addiction Severity Index (ASI; McLellan, Cacciola, Alterman, Rikoon, & Carise, 2006; Tiet, Brynes, Barnett, & Finney, 2006). While all admirable efforts, with varying degrees of successful implementation, most have involved a pre-treatment or admission assessment, followed by a single follow-up assessment (e.g., discharge, 6-months post admission). With these approaches, the information (i.e., outcomes data) is generally used to evaluate the system of care not to guide care on the individual patient level. Moreover, extant OMSs have been difficult to successfully implement because they often involve substantial effort (e.g., extensive training, lengthy assessment, additional “paperwork”, burdensome data entry, distal post-discharge follow-up) that compete with rather than inform clinical care. The challenge for the field is to develop a set of items, clinical and administrative procedures and reporting mechanisms that are easy to implement, have clear clinical value and address administrative/management needs.

Currently available instruments for the assessment of SUD and related problems consist largely of: 1) brief screening measures (e.g., CAGE; Ewing, 1984; AUDIT; Reinert & Allen, 2007) used in non-SAT healthcare settings to determine whether patients have a substance abuse problem that warrants further assessment (Bradley, Bush, McDonell, Malone, & Fihn, 1998); 2) psychiatric diagnostic tools (e.g., PRISM; Hasin et al., 1996; SCID; Kranzler, Kadden, Babor, Tenne, & Rounsaville, 1996; Spitzer, Williams, Gibbon, & First, 1992), most often used in research settings; and 3) comprehensive multi-domain intake and follow-up assessments (e.g., GAIN; Dennis, Scott, & Funk, 2003; Dennis, Titus, White, Unsicker, & Hodgkins, 2003; ASI; McLellan et al., 2006; ASI-Lite: Cacciola et al., 2007) for initial treatment planning and program evaluation. These instruments are useful for their intended purposes but unsuitable for frequent during-treatment monitoring. Three monitoring instruments have been developed, the Brief Treatment Outcome Measure (BTOM; Lawrinson, Copeland, & Indig, 2005), the Addiction Severity Assessment Tool (ASAT; Butler et al., 2005) and the Treatment Outcome Profile (TOP; Marsden et al., 2008). Unfortunately, inherent limitations render these instruments less than ideal for use in outpatient drug-free treatment programs, the predominant form of SUD treatment.

The Brief Treatment Outcome Measure (BTOM; Lawrinson et al., 2005), developed in Australia, is a structured interview for patients in opioid maintenance pharmacotherapy (OMP) treatment. It is administered at intake and at follow-up intervals of ≥3 months. The BTOM yields data on frequency of drug, alcohol, and tobacco use, needle sharing, overdose, and polydrug use, as well as social functioning, arrest frequency, current and previous SAT treatment, and physical health. Although reliability and validity estimates are acceptable, the prescribed time between administrations, duration (15-20 minutes), and limited generalizability to non-OMP patients pose limitations to its use in outpatient drug-free treatment programs.

The Addiction Severity Assessment Tool (ASAT; Butler et al., 2005) is a 27-item self-report instrument designed to provide a multidimensional profile of current problems. Although the ASAT was tested with a more heterogeneous patient group of SAT patients than the BTOM and is potentially more broadly applicable, it has important shortcomings. For example, the Likert-scale ratings for the individual items (1=“Not at All True” to 4=“Very True”) and the subscale scores may not be meaningful to clinicians. Specifically, the “arbitrary metrics” (Blanton & Jaccard, 2006) employed make the measure difficult to use to assist in guiding clinical decisions as the item and scale scores may not be easily linked to experiences in a patient’s everyday life (Blanton & Jaccard, 2006; Kazdin, 2006). Also, discerning whether changes over time, at the item and scale levels, are clinically meaningful may be difficult for clinicians (and researchers) (Blanton & Jaccard, 2006; Kazdin, 2006). An instrument comprised with at least some non-arbitrary behavioral count items (Blanton & Jaccard, 2006) could remedy these issues.

Finally, the Treatment Outcome Profile (TOP; Marsden et al., 2008) is another multi-dimensional assessment instrument that was developed in the United Kingdom. This instrument covers 28-day status across four domains (substance use, health, crime and social functioning) and has demonstrated preliminary reliability, validity and sensitivity to change. The TOP includes 23 items, most with multiple components/questions. Inspection of the items reveals an emphasis on crime and substance use (6 items and 10 items, respectively) and no attention to, for instance, family or protective factors associated with recovery. Limitations also include its method of administration (interview only) that requires support with a calendar for TLFB procedures. Hence, the TOP requires more than minimal staff training and has an administration time of approximately 15 minutes.

In general psychiatry, multidimensional monitoring and outcomes instruments with demonstrated reliability and validity such as the Behavior and Symptom Identification Scale (BASIS-32; Eisen, Dill, & Grob, 1994; Eisen, Wilcox, Leff, Schaefer, & Culhane, 1999) and the Outcome Questionnaire-45 [OQ-45; Lambert et al., 1996, and its very brief alternative, the single dimenision Outcome Rating Scale (ORS; Miller et al. 2003)] do exist. Unfortunately, these instruments are not ideally suited for use in SAT without significant modification because they were designed for the general psychiatric, not SAT, populations. This is also true for widely used instruments such as the Short-Form Health Survey (SF-12; Ware et al., 1996) that measures both physical and mental health.

Our position is that a successful monitoring instrument should balance brevity and ease of administration, to allow for routine use, with comprehensiveness in order to capture multiple domains of import. Items should reflect constructs or variables (e.g., psychiatric distress, family/social relationships, substance use, abstinence supports) that have been empirically linked with patient problems and service needs, or SAT outcomes. Additionally, items should be included that assess both negative or risk factors, and positive or protective factors. Finally, the instrument should provide both a ‘clinical snapshot’ that depicts the patient’s status at a particular moment in time, and with each subsequent administration, create a cumulative record of the patient’s status over time.

Within this context, the objective of the research described herein was to develop a brief, reliable, and valid monitoring instrument for outpatient SAT to support the VA’s move toward measurement-based care as recommended by their clinical practice guidelines.

2. METHODS and MATERIALS

The study consisted of two phases, an instrument development phase and an initial study to examine the psychometric properties of the brief monitoring instrument. The project was approved by the Philadelphia Veterans’ Administration Medical Center’s (PVAMC) Internal Review Board (IRB).

2.1. Phase 1: Instrument Development

Our guiding perspective was a multidimensional monitoring instrument consisting of no more than 20-25 items, appropriate for periodic re-administration, and able to be completed in less than 10 minutes as either a structured interview or patient self-administered questionnaire. While some items would assess substance use and related issues (i.e., risk and protective factors), others would address areas of functioning that have been found to be particularly relevant for SUD patients (e.g., family/social and financial status, as well as psychiatric and medical problems) in terms of documenting treatment needs, status, progress, or outcomes. The measurement of multiple areas of functioning to evaluate substance abusing patients and SAT has been widely accepted and expected within the field. This is consonant with the IOM’s recommendation for client monitoring systems in SAT that “validly assess response to treatment… and… are practicable for routine use…. Including a set of… substance use vital signs… a brief set of indicators measurable at the patient level… for repeated administration during and following treatment to monitor symptoms and functional status.” (IOM 2006, p. 15; Recommendation 4.5). Thus, the set of items should have value for both guiding individual patient level care and evaluating the system of care as data can be used to predict and monitor individual patient progress as well as evaluate program outcomes. Clinically, the items, and the instrument in its entirety, should elicit core information that can be followed with selective probing to elucidate the specifics of the issue(s) at hand to further individualize clinical discussion of the patient’s progress and current status.

In order to maintain contact with the substantial patient assessment literature, most items were culled from existing instruments and OMSs. Accordingly, the initial instrument development phase included a review of relevant instruments and OMSs [e.g., ASI, TEDS, NOMS, GPRA (complete list available from 1^st author)]. We reviewed and considered the empirical support for reliability/validity of the individual constructs measured by the candidate items in our selection process (also see, Description of the Instrument 2.2.3., below). To ensure that the general content and specific items were adequate and appropriate within the VA system of care, we consulted with eight local PVAMC SAT bachelor’s/master’s level practicing clinicians (some also had supervisory responsibilities), as well as with our national VA Scientific Advisory Group (i.e., doctoral level, researcher/clinician members of the national VA SUDs Quality Enhancement Research Initiative, Continuity of Care Work Group). More specifically, we generated a list of domains and items and presented these to clinicians within the PVAMC SAT system for review and feedback. The items were produced to be administered largely as written and most were adapted from reliable and valid items within other instruments. Meetings were held with individual and small groups of clinicians where the investigators explained the goals and purposes of the proposed instrument. The clinicians were handed the preliminary set of items and were queried by one of the first two authors regarding the choice, acceptability, utility, and wording of the items, and the overall content coverage; suggestions for revisions, deletions and additional items were elicited. Notes were taken by a research technician. Similar feedback from multiple clinicians was given greater weight in creating a draft instrument. The investigators then sent this draft to each Scientific Advisory Group member for written expert feedback. The reviews were then discussed in a conference call with the Advisory Group, and consensus-driven revisions lead to a beta version.

These steps resulted in a 25-item instrument that was then administered to 25 PVAMC SAT outpatients who had been in treatment for varying durations (0-3 months). Following the administration of the instrument, feedback was obtained from the participants concerning their understanding and perceived value of each item, the extent to which they felt they could answer each item honestly, and what other items they might recommend. Administration time of the assessment was documented. Patients were paid $10 for participation.

Participant data were evaluated. Items were generally well understood by and acceptable to the participants; modifications or deletions were made in the few cases when this was not so. Items which correlated too highly or minimally with each other within a nominal area, or that had very low endorsement rates, were considered for deletion. Meetings were then held with the PVAMC clinicians and the Advisory Group. Items ultimately retained for the final instrument were those determined rationally, based on these consultations, the data analyses, and the goals of the monitoring system. As a result, the final instrument consisted of 17 items (see below) that queried the prior 30-day time period. A brief Instrument Administration and Study Procedures Manual for SAT clinicians was developed. It included item clarifications and probes; patient contact procedures; strategies for using the data clinically (e.g., identification of referral needs, problem solving, support); and information for handling emergency situations.

2.2. Phase 2: Assessment Study

This study was designed to provide initial data on the psychometric characteristics and feasibility of the instrument henceforth designated as the Brief Addiction Monitor (BAM). Baseline and 3-month administrations by clinicians provided the following information: 1) response characteristics of the instrument, factor structure, and internal reliability; and 2) the instrument’s sensitivity to change. In addition, independent administration of the instrument at 3-months by research technicians was built into the study design. This aspect of the test-retest reliability study provided information on whether patients provide different information on the severity of their problems to independent technicians than they do to their clinicians. Finally, to measure predictive validity, we reviewed clinical charts and coded whether or not each participant had successfully completed their first stage of outpatient SAT [typically intensive outpatient treatment (IOP)].

Prior to the onset of Phase 2, two separate one-hour meetings were held with the PVAMC’s outpatient SAT staff (clinicians, program director, etc.) to go over issues related to the administration of the instrument and study procedures, as described in the Manual. Research technicians were also available during the course of the study to answer questions and facilitate the conduct of the study for clinical staff. Data collection procedures were based in part on prior meetings with treatment staff which delineated the program structure and process regarding intake, treatment and discharge. As a result, the initial baseline assessment was collected within a 1-2 week patient intake and disposition process, and was administered by one of the program’s four disposition counselors as part of standard operating procedures. Once the instrument was administered, the disposition counselor communicated this to one of the study’s research technicians who attempted to meet with the patient within one week to obtain informed consent for study participation. Once consent was obtained, the technician collected patient locator information for follow-up on a study form which was also made available to the clinical staff.

In addition to the baseline assessment, the study incorporated a 3-month follow-up evaluation by both clinicians and researchers using the BAM. These two administrations were planned to be conducted within one week of each other in counterbalanced order. Regarding the clinicians, 50% of the follow-up evaluations were assigned to the SAT counselor with the most recent clinical responsibility for the patient, while the remaining 50% were assigned to a nurse who was a member of an ongoing PVAMC SAT clinical outreach effort. This approach thereby included types of personnel resources for follow-up that might be available at different VA facilities. Research technicians alerted appropriate staff when the due date for a follow-up evaluation was approaching. Both the clinical and research staff were to make at least three contact attempts (e.g., phone calls, scheduled visits) to administer the follow-up assessment. When the follow-up was scheduled to be conducted first by the clinician, and was completed, the patient was referred to the research technician. A corresponding approach was implemented when the follow-up evaluation was completed first by a technician, that is, referral to the clinician. Patients were compensated $15 upon completion of both follow-up assessments.

2.2.1. Participants

Over a 4-month period, 150 patients participated in the study. Their mean age was 51.4 years (SD = 7.1). The large majority (78.7%) were African American, 20.7% were white, and only 0.6% Hispanic. Nearly all participants were male (96.7%), and 12.7% were currently married.

2.2.2 Follow-up

Clinical staff was able to obtain a 3-month follow-up evaluation for 84 of the 150 participants (56.0%). For those still in treatment at this time point, clinicians conducted 36 of a possible 37 (97.3%) assessments. In contrast, they completed 48 out of 113 (42.5%) assessments for participants who were no longer in treatment. Research staff completed 78 assessments (52.0%). A total of 88 participants (58.7%) were assessed at follow-up by either clinical or research staff; 74 (49.3%) were assessed by both clinical and research staff.

2.2.3. Description of the Instrument

The items of the Brief Addiction Monitor (BAM) and their descriptive statistics are summarized in Table 1. Of the 17 items, four, [#s 4, 5, 6, 7 (i.e., drug type)] are concerned specifically with alcohol or drug use. Two of these items are contingent items (#s 5, 7), dependent upon the response to the immediately preceding item, and provide more specific information on alcohol and drug use if acknowledged by the respondent. If any alcohol use is reported (#4), item #5 queries days of heavy alcohol use. If any drug use is reported (#6), item #7 (a checklist of drug categories), queries which drugs were used. The remaining items address aspects related to substance use, recovery and treatment that span a number of life areas considered important for a multidimensional assessment of substance abusing patients and include interpersonal relationships, psychological/medical problems and finances (McLellan et al., 2006). Moreover, many of the items can be considered risk or protective factors for poor treatment response or substance use (e.g., retention in treatment, continued problematic use, relapse after abstinence). We reviewed the empirical support (e.g., concurrent/predictive validity) for items that were considered risk or protective factors. For example, frequency of substance use, is a proxy for severity, which has been shown to predict treatment attrition and other poor outcomes (Laudet, Stanick, & Sands, 2009; McKellar, Kelly, Harris, & Moos, 2006). Psychiatric difficulty and negative affect have also shown to predict poor outcomes (Carroll, Power, Bryant, & Rounsaville, 1993; Mertens & Weisner, 2000; Schiffman, Paty, Gnys, Kassel, & Kickcox, 1996; Zywiak, Connors, Maisto, & Westerberg, 1996). Exposure to risky situations and severity of craving have has been implicated in relapse (Freedman, Lester, Roth, McNamara, & Milby, 2004; Schiffman et al., 1996; Weiss et al., 2003; Zywiak et al., 1996). The protective function of 12-Step attendance has been well documented (e.g., Cacciola, Dugosh, Foltz, Leahy, & Stevens, 2005; Ferri, Amato, & Davoli, 2006; Hser, Polinsky, Maglione, & Anglin, 1999; McKay & Weiss, 2001). Social support has also been associated with positive outcomes (Dobkin, Civita, Paraherakis, & Gill, 2002; McKay et al., 2005). Self-efficacy for abstinence has predicted early relapse and is associated with long-term recovery (Laudet et al., 2009; Maisto, Connors, & Zywiak, 2000; Schiffman et al., 2000).

Table 1.

BAM Item Response Characteristics at Baseline (N = 150)

Item # and Content	Mean	SD	Median	Skewness
#1 Rating of physical healtha	2.33	1.09	2.00	-0.23
#2 Days trouble sleeping	16.26	11.41	15.00	-0.07
#3 Days psychological problems	14.06	10.47	15.00	0.22
#4 Days alcohol use	6.71	9.63	1.00	1.38
#5 Days heavy alcohol use	6.24	9.12	0.00	2.10
#6 Days drug use	3.76	6.88	0.00	2.26
#8 Rating of cravingb	1.64	1.16	2.00	0.29
#9 Rating of abstinence confidenceb	2.90	1.24	3.00	-0.85
#10 Days self-help group attendance	8.76	11.14	2.00	0.98
#11 Days in risky situations	6.98	9.82	2.00	1.38
#12 Rating of religion/spirituality supportb	2.62	1.37	3.00	-3.17
#13 Days structured activities	6.51	9.44	0.00	1.24
#14 Adequate income (% yes)	42.0	_	0.00	0.33
#15 Rating of arguments with family/friendsb	0.95	1.15	1.00	1.14
#16 Days with supportive family/friends	16.47	12.69	15.00	-0.08
#17 Recovery satisfactionb	2.60	1.24	3.00	-2.94

^a--0 = excellent; 4 = poor

^b--0 = not at all; 4 = extremely

Note: Data from Item #7, a 6 category checklist of Drugs taken in the past 30 days is included in the text (See 3.1. Item Baseline Response Characteristics).

2.2.4. Data analysis

Using baseline data, descriptive statistics (e.g., means, standard deviations) were calculated for the items. Also, in order to identify the factor structure, an exploratory factor analysis (EFA; MPlus; Muthen & Muthen, 1998) was undertaken using maximum likelihood extraction and the varimax and promax rotations. Prior to undertaking this analysis, the directionality of 7 of the items was reversed in order to establish uniform directionality for interpreting the findings. The two contingent items were not included in the analysis so that 15 items were ultimately subjected to EFA. EFA was undertaken rather than confirmatory factor analysis (CFA) since there was no prior model that explicitly delineated the underlying dimensionality of the instrument.

Paired t-tests were used at the item and factor levels to compare the baseline versus 3-month clinician assessments and measure sensitivity to change [McNemar χ² was used for the income item (#15) which had a dichotomous response option]. At the 3-month assessment point, short-term test-retest reliability between the counselors’ and research technicians’ assessments was determined using a parallel series analyses (i.e., paired t-tests, McNemar χ²) as well as with intraclass correlations (ICCs) (except for the income item where kappa was used). As patients may be more candid under research conditions where the counselor is not privy to their responses, and in turn, responses do not result in consequences, these short-term test-retest analyses were also considered to inform validity. As the sample is small and the factor analysis exploratory, the change and test-retest analyses were conducted and reported on all 15 noncontingent items offering a more comprehensive evaluation of the BAM.

Finally, a logistic regression was performed using the baseline factors derived with the EFA (after converting to T-scores to standardize the variance) to predict successful completion of the first stage of outpatient SAT.

3. RESULTS

The BAM instrument took an average of 6 ± 4 minutes to administer at the baseline assessment. Due to clinician preference the majority (72.8%) of the assessments were completed via counselor interview rather than patient self-report.

3.1. Item Baseline Response Characteristics

The most frequently used substances were alcohol (53.5%), cocaine (35.7%), marijuana, (18.1%), and opiates (7.8%); no other drugs were reported in >5% of the sample. Descriptive findings for the remaining BAM items are shown in Table 1. Participants reported using alcohol a mean of 6.71 (± 9.63) days in the past 30 days, and other drugs 3.73 (± = 6.86) days. They also reported experiencing frequent problems with sleep (16.26 ± 11.41 days) and psychological distress (14.06 ± 10.47 days). Participants spent about half of the past 30 days with supportive family or friends (16.47 ± 12.69 days) and reported being slightly bothered by arguments with family or friends. Over half (58%) reported inadequate income for necessities. Although participants reported being in risky situations almost twice a week during the past month (6.98 days ± 9.82), they attended self-help groups more than twice a week (8.76 ± 11.14), reported ‘slight’ to ‘moderate’ craving for alcohol or drugs, and felt ‘considerably’ confident they could maintain abstinence over the next 30 days.

There was little indication of inappropriate response patterns. Of the 15 items, 8 had zero or one missing response (i.e., <1% of the cases), and 6 items had two to six missing responses (1 - 4%). Only one item, days of drug use, had greater than 5 percent missing responses (11 cases, 7.3%). Skewness was unremarkable for 12 of the 15 items (i.e., <2; Tabachnick & Fidell, 1996). The skewness of 2.26 on the days of drug use item was consistent with 56% of the responding participants indicating that they had not used in the prior 30 days. All other items regarding frequency (‘days’) of past 30-day behaviors/experiences had response rates <50% for all response options, including those at the extremes of 0 or 30 days. The skewness of -3.17 for the religious support item was consistent with 63% of the respondents indicating that religion or spirituality ‘considerably’ or ‘extremely’ supported their recovery. Similarly, 57% of the participants indicated that they were ‘considerably’ or ‘extremely’ satisfied with their recovery progress; skewness on this item was -2.94.

3.2. EFA Findings

The sample size for this analysis was 120 due to missing data. The varimax and promax rotations yielded similar solutions. We report the results of the varimax rotation as it maximizes the variance of factors across the variables (i.e., the distinction among the factors), which produces a somewhat simpler solution. Although four dimensions were found to yield eigenvalues >1, the fourth factor included only one item and was therefore not considered a viable factor. The first factor was comprised of four items that appeared to reflect a protective dimension (see Table 2) including abstinence confidence, self-help group attendance, religion/spirituality supporting recovery, and recovery satisfaction. The second factor included three items related to medical and psychological status; a rating of physical health, and the frequency of days with sleep and psychological problems. The third factor included four items describing or related to substance use, and may be considered risk factors for continued use or poor treatment response; days of alcohol use, days of drug use, days in risky situations, and a rating of craving. Thus, 11 of the 15 BAM items in the analysis loaded on the three derived factors. The percent of the variance explained by the factors was 14.1% (Factor 1: Recovery Protection), 9.0% (Factor 2: Physical & Psychological Problems), and 14.1% (Factor 3: Substance Use & Risk), for a total of 37.2%. Additionally, the standardized alpha values for Factors 1, 2 and 3 were 0.78., 0.67., and 0.71, respectively. Using a RMSEA estimate of ≤0.05 as excellent and <0.08 as acceptable (Browne & Cudeck, 1993), the RMSEA of 0.056 achieved for this analysis indicates a very adequate model fit.

Table 2.

Exploratory Factor Analysis Findings^* (N = 120)

BAM Items	Factor
	1	2	3
Rating of physical health	0.07	0.44	0.26
Days trouble sleeping	-0.07	0.46	0.20
Days psychological problems	0.08	0.71	0.34
Days alcohol use	0.25	-0.12	0.68
Days drug use	0.27	0.20	0.50
Rating of craving	0.18	0.28	0.43
Rating of abstinence confidence	0.80	0.09	0.10
Days self-help group attendance	0.48	-0.26	0.37
Days in risky situations	0.12	-0.01	0.73
Rating of religion/spirituality support	0.62	0.05	0.35
Days structured activities	0.09	0.27	-0.11
Adequate income	-0.08	0.19	-0.09
Rating of arguments with family/friends	0.11	0.14	0.32
Days with supportive family/friends	0.09	0.23	0.02
Recovery satisfaction	0.79	0.23	0.26
Eigenvalue	4.01	1.83	1.37
% Variance	14.1	9.0	14.1

^*--Maximum likelihood extraction; varimax rotation

3.3. Sensitivity to Change

These analyses evaluated the responses of participants with both baseline and 3-month follow-up clinician assessments (N = 84; Table 3). The results reveal reductions in problem frequency and severity, and improvements in protective factors or prosocial behaviors on all the items as well as on the three factor scores. These changes were statistically significant (p<.05) for 11 of the 15 items and for all three factors.

Table 3.

Baseline vs. 3-Month Follow-Up Values (N=84) - Paired t-test

BAM Items	Baseline Mean (SD)	3-Month F-U Mean (SD)	t	p
Rating of physical healtha	2.36 (1.10)	2.05 (1.11)	2.42	0.018
Days trouble sleeping	15.69 (11.42)	11.42 (10.86)	3.11	0.003
Days psychological problems	13.75 (10.56)	9.17 (10.02)	4.51	<.001
Days alcohol use	6.80 (9.61)	2.35 (5.22)	4.29	<.001
Days drug use	4.04 (6.68)	0.59 (2.96)	4.19	<.001
Rating of cravingb	1.68 (1.13)	0.96 (0.99)	4.88	<.001
Rating of abstinence confidenceb	2.93 (1.27)	3.13 (1.12)	-1.37	0.17
Days self-help group attendance	8.62 (10.97)	11.88 (11.52)	-2.54	0.013
Days in risky situations	6.73 (9.37)	3.83 (8.00)	2.59	0.011
Rating of religion/spirituallity supportb	2.54 (1.86)	2.87 (1.34)	-1.64	0.104
Days structured activities	5.72 ((9.40)	8.55 (10.76)	-2.42	0.018
Adequate income (% yes)	40.48	54.76	4.32*	0.038
Rating of arguments with family/friendsb	1.17 (1.30)	1.04 (1.07)	0.93	0.356
Days with supportive family/friends	15.83 (12.71)	18.54 (11.77)	-1.83	0.071
Recovery satisfactionb	2.46 (1.79)	3.23 (0.91)	-4.26	<.001
BAM Factors
1: Recovery Protection	49.66 (8.29)	46.02 (9.44)	3.57	<.001
2: Physical & Psychological Problems	50.07 (9.86)	45.44 (10.29)	4.26	<.001
3: Substance Use & Risk	50.55 (10.74)	43.80 (7.82)	5.16	<.001

^a--0 = excellent; 4 = poor

^b--0 = not at all; 4= extremely

^*-McNemar χ²

3.4. Test-Retest Reliability

These analyses were undertaken on the 3-month follow-up assessment data. As noted earlier, a total of 88 patients were assessed at follow-up by clinical or research staff but 74 were assessed by clinical and research staff which was necessary for these analyses. We used guidelines of ≥0.75 as indication of excellent reliability, 0.60 to 0.74 good reliability, 0.40 to 0.59 fair reliability, and <0.40 as poor reliability (Cicchetti, 1994). Accordingly, test-retest reliability was excellent (5 items), good (4 items), or fair (4 items) for all but 2 of the 15 items (Table 4). The ICCs indicated poor reliability for the days of ‘drug use’ (ICC = 0.24) and the ‘risky situations’ (0.25) items. As days of drug use was not normally distributed (i.e., >70% reported no use at follow-up under both clinical and research conditions), this item was dichotomized (use/no use) and reanalyzed using the Kappa statistic. These results revealed a value of .070. Test-retest reliability was excellent for each of the three factors. The paired t-tests revealed significant (p<.05) differences between participants’ independent responses to clinicians and researchers on only 3 items. Specifically, participants reported more days of self-help group attendance, less severe craving, and more recovery satisfaction to clinicians than to researchers. Additionally, the score on the Recovery Protection Factor was significantly lower (more positive) using clinician collected data compared to researcher collected data.

Table 4.

Test-Retest Reliability at 3 Months (N = 74)

BAM Items	Counselor Mean (SD)	Researcher Mean (SD)	t	p	ICCx	p
Rating of physical healtha	2.07 (1.13)	2.05 (1.00)	0.13	0.90	0.66	<.001
Days trouble sleeping	11.23 (10.84)	10.47 (10.00)	0.77	0.44	0.67	<.001
Days psychological problems	8.61 (9.63)	9.38 (10.49)	-1.09	0.28	0.82	<.001
Days alcohol use	2.42 (5.37)	2.39 (6.43)	0.09	0.94	0.88	<.001
Days drug use	0.54 (1.69)	1.21 (3.22)	-1.73	0.09	0.24	0.21
Rating of cravingb	1.04 (1.00)	1.22 (1.01)	-2.08	0.046	0.73	<.001
Rating of abstinence confidenceb	3.11 (1.13)	2.88 (1.24)	1.79	0.078	0.56	<.001
Days self-help group attendance	12.49 (11.57)	11.24 (11.50)	2.09	0.040	0.90	<.001
Days in risky situations	4.09 (7.95)	3.69 (6.99)	0.38	0.70	0.25	0.015
Rating of religion/spirituality supportb	2.92 (1.28)	2.57 (1.93)	1.93	0.057	0.53	<.001
Days structured activities	8.55 (10.95)	9.31 (10.99)	-1.08	0.28	0.85	<.001
Adequate income (% yes)	55.4	51.4	0.36*	0.55	0.70x	<.001
Rating of arguments with family/friendsb	1.03 (1.02)	1.20 (1.18)	-1.39	0.17	0.51	<.001
Days with supportive family/friends	19.11 (11.60)	18.45 (11.85)	0.50	0.62	0.55	<.001
Recovery satisfactionb	3.25 (0.92)	3.07 (1.12)	2.50	0.015	0.86	<.001
BAM Factors
1: Recovery Protection	45.59 (9.50)	47.67 (9.88)	-3.37	0.001	0.84	<.001
2: Physical & Psychological Problems	45.20 (10.28)	45.27 (10.65)	-0.09	0.930	0.80	<.001
3: Substance Use & Risk	43.49 (8.11)	44.82 (8.42)	-1.88	0.064	0.75	<.001

^a--0 = excellent; 4 = poor

^b--0 = not at all; 4= extremely

^*-McNemar χ²

^x-Kappa value

3.5. Predictive Validity

These analyses included 121 participants [n=45 (37.2%) completed vs. n=76 (62.8%) did not complete IOP treatment]. The baseline factor scores for Factor 1: Recovery Protection (Wald Chi-square = 4.377; df = 1; p<.05) and Factor 3: Substance Use & Risk (Wald Chi-square = 4.261; df = 1; p<.05) were significant predictors of treatment dropout with the model accounting for 16.9% of the variance. The odds ratios indicate a 5% increase (CI = 1.003 - 1.100) in the likelihood of treatment dropout for each 1 point increment in the Factor 1: Recovery Protection score (Note: Items are reverse scored so that higher scores indicate fewer protective factors.) and a 5.2% increase (CI = 1.003 - 1.103) in the likelihood of treatment dropout for each additional point in the Factor 3: Substance Use & Risk score.

4. DISCUSSION

This study developed and tested a brief 17-item monitoring measure covering important substance use related behaviors which could be used at the beginning of treatment to determine the severity of a patient’s problems and at subsequent time points to chart patient status over time. The findings revealed that this instrument, Brief Addiction Monitor (BAM), exhibited acceptable characteristics.

Exploratory factor analysis of the BAM resulted in the derivation of three summary factors; Recovery Protection, Physical & Psychological Problems, and Substance Use & Risk. These results that reveal a positive, Protection factor, and negative Use & Risk and Problems factors are consistent with the findings of positive and negative dimensions of health (Keyes, 2005). The RMSEA estimate for this analysis was acceptable; the factors had alpha values exceeding or approaching 0.70 and explained 37.2% of the variance. All three factors were sensitive to change and had excellent test-retest reliability. Finally, predictive validity was demonstrated for two of the factors, Recovery Protection and Substance Use & Risk.

At the item level, there was little indication of inappropriate response patterns. Change over time was significant for most items, and test-retest reliability was acceptable for all but 2 items (days ‘drug use’ and ‘risky situations’). The absolute magnitude of days of drug use was quite low for both the test and retest which did not differ significantly. Moreover, when the response was dichotomized to indicate any use, the Kappa was found to be good. Days in risky situations also did not differ significantly between the two assessments. Participants did report significantly more days of self-help group attendance, less severe craving, and more recovery satisfaction to clinicians than to researchers. Nonetheless, reliability was acceptable for these items. Taken as a whole, there was only limited evidence in this study that patients were less likely to reveal problematic status to treatment staff.

The study provides support for the robustness of the BAM. This is in addition to the strengths of the BAM not found in other instruments such as a 5-10 minute administration time, many non-arbitrary behavioral count items, multidimensional coverage, an option of interviewer or self-administration, and minimal training requirements. Although the findings were relatively encouraging it is also important to emphasize limitations as well as additional steps that need to be taken to further establish the reliability, validity and usefulness of the instrument. First, the sample size was modest and the characteristics of the patients were somewhat homogeneous vis-à-vis the large population of veterans treated in the U.S. and the overall population of SAT patients more generally. Clearly, the acquisition of a larger number of assessments in a more heterogeneous sample would strengthen confidence in the psychometric strength and practical value of the BAM. In that regard, the BAM is currently being administered and used in a variety of ways at a number of VA Medical Centers throughout the country. These data will eventually be available for analyses. It is likely that the psychometric characteristics and underlying structure could be determined more definitively with this larger sample. In this regard, a limitation of the relatively small sample size is that smaller samples tend to yield less reliable correlation estimates (Tabachnick and Fidell, 2007). Various recommendations have been offered as to the desired sample size for an EFA, ranging from a minimum of 100 (Gorsuch, 1983) to 300 (Rouquette & Falissard, 2011) participants. When researchers limit the number of expected factors as was the case in the current study, smaller sample sizes may produce reliable estimates (Preacher & MacCallum, 2002). Future research will examine the generalizability of our findings by determining the extent to which the current factor structure is replicated in a new, larger sample. Since the factors are preliminary, individual items may also be useful for clinical and evaluation purposes. Data on the BAM from use in systems of care outside the VA would also be informative both in terms of the BAM’s psychometric properties and its content for non-veterans and women. Nonetheless, while the BAM was developed cognizant of the needs of the VA and its population, the item construction and selection relied largely on the general substance abuse treatment, assessment and population research.

Second, the 56% follow-up rate in this study compares favorably against those obtained by clinical or research personnel within large VA SAT outcomes initiatives (Tiet et al., 2006). Nonetheless, the rate is too low. Thus, even though the instrument itself is brief and easy to administer, the amount of attrition highlights the importance of other methodological issues as well as organizational issues in obtaining high follow-up rates. At the same time, our rate may have been enhanced, relative to what may have been achieved under a standard program evaluation initiative, by the existence of auxiliary research staff associated with the project who provided indirect support to clinical staff, i.e., reminders of follow-up evaluation due time, etc. Again, more light may be shed on the effectiveness of various approaches to implementation and data collection from the ongoing project mentioned earlier. It also should be noted that measurement-based care and outcomes assessment have somewhat different goals with regard to follow-up rates. In measurement-based care, it is important to assess all patients at regular intervals while they are still in treatment (here clinicians assessed >95% of patient still in treatment at 3 months); whereas in outcomes assessment, the goal is to achieve a reasonably high follow-up rate (e.g., ≥70%) on all patients who initiated treatment, whether they are still in treatment or not at the follow-up point.

Implementing adaptive treatment requires an appropriate patient monitoring protocol. The BAM can potentially serve as a tool within such a protocol. Important additional steps in the implementation of the BAM or other such instruments include the development of electronic versions which can facilitate the assessment process by providing clinician prompts concerning when assessments are due, and latest patient contact information. Additionally, automated clinical status and progress reports that provide current status and longitudinal, individual patient-level data would support the clinical utility of the measure. Similarly, automated aggregate reports would be an enhancement for management and executive staff.