Factor Structure of the Brief Addiction Monitor in a Non-Veteran Substance Use Disorder Outpatient Treatment Sample

Highlights

• •We examined the validity of BAM factor structure in non-veteran SUD population
• •Structural validity of BAM subscales could not be confirmed
• •BAM factor loadings inconsistent across racial, referral, and primary SUD subgroups
• •Implications for tracking recovery over time across different populations

Abstract

Background

The Brief Addiction Monitor (BAM) was developed as a comprehensive substance use disorder (SUD) outcome metric to fill a gap in quality measurement. Research to date has only examined the psychometric performance of this measure in veteran SUD populations. The purpose of the current research is to examine the factor structure and validity in a non-veteran SUD population.

Methods

Non-veteran patients admitted to a SUD treatment program (N = 2,227) completed BAM at intake. After confirmatory factor analysis (CFA) was performed to evaluate the measurement model validity of previously defined latent structures, exploratory factor analysis (EFA) was used to assess the factor structure and psychometric properties of the BAM within the full sample and within subgroups, specifically racial, referral source (mandated vs. not), and primary SUD diagnosis.

Results

Exploratory factor analyses in the full sample supported a 4-factor model (representing Stressors, Alcohol Use, Risk Factors, and Protective Factors) derived from 13 items. Subsequent EFAs conducted separately in each subgroup revealed variability in the number of resulting factors and pattern matrices. The internal consistency also varied among factors and between subgroups; in general, reliability was greatest for the Alcohol Use scale and either poor or questionable for pattern matrices resulting in scales reflecting Risk or Protective Factors.

Conclusion

Findings from our study suggest that the BAM might not be a reliable and valid instrument for all populations. More research is needed to develop and validate tools that are clinically meaningful and allow clinicians to track recovery progress over time.

1. Introduction

To fill the need for a brief, yet comprehensive SUD outcome measure, the Department of Veterans Affairs (VA) commissioned and developed the Brief Addiction Monitor (BAM). The VA continues to use the BAM to measure substance use-related outcomes (DeMarce, 2021). This need for validated scales to monitor SUD outcomes is well supported outside of the VA (Alter et al., 2021). The BAM has been endorsed by the Kennedy Forum as an evidence-based SUD outcome measure (Wrenn & Fortney, 2015), and it is listed by the Joint Commission (2020) as a measurement-based care instrument for adults seeking various levels of SUD speciality care. Given its recommended use, non-VA SUD speciality programs may consider implementing the BAM as a measurement-based outcome tool. However, studies have yet to examine the factor structure and internal reliability of the BAM in non-veteran samples. Research has also not evaluated whether the BAM demonstrates a consistent factor structure across different racial identities, individuals who are mandated to SUD treatment versus those who are not mandated, and primary SUD diagnosis.

1.1. Psychometric performance of the BAM in US Veteran samples

Although the clinical scoring guidelines for the BAM propose three subscales: Substance Use, Risk Factors, Protective Factors (Cacciola et al., 2013; Department of Veterans Affairs, 2009), studies among US veterans have mostly not supported this three factor model. Cacciola and colleagues (2013) conducted an exploratory factor analysis among 150 US veterans outpatients. They found support for three factors: Abstinence Confidence and Recovery Satisfaction, Psychological and Medical Problems, and Substance Use. Several additional studies of the BAM failed to replicate a three factor model (cf. Gaddy et al., 2018; Hallinan et al., 2020; Nelson et al., 2014). Nelson and colleagues (2014) used principal components analysis in a sample of 700 US veterans receiving SUD treatment. They did not find support for a consistent factor structure across inpatients versus outpatients, although the findings partially supported a unitary factor structure. A difference between these studies was that the former used the continuous response version of the BAM (i.e., item responses included the number of days out of 30 days) whereas the latter used the discrete scoring for the BAM (i.e., item responses included a 5-point scale representing ranges of days of occurrence in the past month).

Subsequent studies have supported a four factor structure, although three of the four factors have shown questionable to poor internal reliability. Gaddy and colleagues (2018) examined the 5-point item response version of the BAM in a national US veteran sample. They used structural equation models and found that a four factor model showed structural invariance over time with the factors being: Alcohol Use, Stressors, Risk, and Stability. Internal reliabilities were excellent for Alcohol Use (α = .96 and .94). Other subscale internal reliabilities were questionable (α ≤ .68), and the Stability subscale was poor (α = .40 and .42). Hallinan and colleagues (2020) examined the factor structure and longitudinal invariance of the continuous item response version of the BAM in a national sample of veterans. Using confirmatory factor analysis, they found support for a four factor solution that demonstrated structural invariance over time. Internal reliability was excellent for Alcohol Use (α = .94 and .92) but questionable (i.e., α ≤ .68) for the other subscales, with the Stability subscale showing very poor reliability (α = .35 and .28). In summary, studies of the BAM in national veteran samples support a four factor structure but only the Alcohol Use subscale shows high internal reliability.

1.2. Racial differences in SUD outcomes

Although studies of the BAM have included samples from multiple racial groups (Gaddy et al., 2018; Hallinan et al., 2020; Nelson et al., 2014), research has yet to examine its performance across different racial identities. Understanding the performance of the BAM across different racial identities is important because ethnic and racial differences have been observed in the psychometric performance of substance use measures and other related outcomes (Montgomery et al., 2019; Lopez-Vergara et al., 2021).

To draw conclusions about cross-racial comparisons of substance-related outcomes, it is first necessary to establish that an instrument exhibits cross-racial psychometric equivalence. Lopez-Vergera and colleagues (2021) recommend first examining whether metric invariance exists in the factor loadings of items across racial groups. If non-equivalent factor loadings are found, then models should be iteratively computed and compared to determine whether partial invariance exists and where cross-racial differences in factor loadings may be found. Because cross-racial differences on the BAM have yet to be examined, a starting point is to test for configural invariance and explore cross-racial differences on factor loadings.

1.3. Assessing SUD outcomes in mandated versus non-mandated individuals

An important metric in understanding SUD outcomes is the referral source for beginning treatment. These include non-mandated (e.g., self-referral), versus mandated (e.g., from the criminal justice (CJ) system) (Coviello et al., 2013; Rivera et al., 2021). Yet, evidence is mixed with regards to treatment outcome differences (for review see Werb et al., 2016). Generally, CJ mandate to SUD treatment appears to increase treatment retention and completion compared to self-referred individuals (Coviello et al., 2013; Kelly et al., 2005) . In randomized controlled treatment trials, those who were referred by the CJ system had similar treatment outcomes to those self-referred for cocaine use (Kiluk et al., 2015), whereas completion rates were higher for individuals mandated by CJ for OUD compared to non-mandated individuals (Lucabeche & Quinn, 2022). Thus, treatment outcome differences in referral source may be related to specific SUD diagnoses, and represent another important area of consideration.

1.4. Primary SUD differences

Among individuals who exhibit primary (non-alcohol) drug-related diagnoses, polysubstance use appears to be common, with reports indicating 25-50% of patients with OUD also use other substances (Cicero et al., 2020; Xu et al., 2021). In these cases, employing an outcome measure that captures non-alcohol drug use behaviors would be desirable. However, studies have shown that the BAM produces a subscale measuring alcohol but not drug use (Cacciola et al., 2013; Gaddy et al., 2018; Hallinan et al., 2020; Nelson et al., 2014). Instead, studies have found that the BAM drug use items either load onto a factor that captures substance use risk (Gaddy et al., 2018; Hallinan et al., 2020; Nelson et al., 2014) or that it fails to load onto any factors (Nelson et al., 2014). Research is needed to determine if the BAM produces a scale that assesses drug use among individuals with primary drug use disorders. Furthermore, it is important to investigate whether there is measurement invariance and factor structural differences for those with different primary substance use disorders.

1.5. Current study purpose and aims

The purpose of this research is to examine the factor structure and reliability of the BAM in a non-veteran, SUD treatment program. We sought to explore the consistency of the factor structure compared to previous research (Cacciola et al., 2013; Gaddy et al., 2018; Hallinan et al., 2020; Nelson et al., 2014), as well as between subsamples (i.e., primary SUD diagnosis, racial identity, and referral source) within a non-veteran SUD population. Because prior studies have varied in their support for one (Nelson et al., 2014), three (Cacciola et al., 2013), and four (Gaddy et al., 2018; Hallinan et al., 2020) factor solutions, we expected an inconsistent factor structure in our non-veteran population. Specifically, we expected inconsistent factor structure and reliability from previously proposed factor structures across our whole population, and within each of the subsamples.

2. Materials and method

We conducted a retrospective study of patients admitted to a SUD treatment program using electronic health record data. The study was approved by the WCG IRB.

2.1. Treatment setting

This BAM was collected at intake from patients who were admitted to an outpatient SUD treatment program located in a mid-sized city in Southwestern Ohio that primarily serves individuals with Medicaid. The outpatient program offers multidisciplinary services including individual and group psychotherapy, care coordination, peer support services, medication-based treatment of SUD, and treatment of co-occurring psychiatric disorders. Individuals can be referred into the treatment program from local medical facilities and social service agencies, by calling to schedule an intake assessment, or as a walk-in. In this study, individuals were considered to be “mandated” for treatment if they were recommended to receive a drug-and-alcohol assessment or to complete a treatment program as a part of pretrial services, a court or probationary sentence, or an open case with the local children's services agency. Patients completed the BAM on a tablet or by a clinician orally administering the BAM.

2.2. Cohort sample and eligibility criteria

Participants in this study were adults who completed a BAM during a SUD outpatient intake between October 2019 and December 2021 (see Table 1 for patient characteristics). Seventy percent of outpatient intakes had completed the BAM. Compared to those who had completed the BAM, patients who did not complete the BAM were older, more likely to be homeless, unemployed, diagnosed with primary OUD, and self or medically-referred (see Appendix 1).

Table 1.

Patient demographics (N = 2,227)

	N (%)
Age (years), median (IQR)	33 (26-40)
Female (vs. Male)	1091 (49)
Hispanic (vs. non-Hispanic)1	39 (1.8)
Race1
White	1307 (58.7)
Black/African-American	858 (38.6)
Other	127 (2.7)
Homeless (vs. Housed)1	113 (5.1)
Unemployed (vs. Employed)1	1184 (53.2)
Primary SUD
Alcohol	627 (28.2)
Opioid	438 (19.7)
Cannabis	879 (39.5)
Other Drug	283 (12.7)
Referral Source
Self (non-mandated)	719 (32.3)
Medical / Behavioral (non-mandated)	91 (4.1)
Criminal Justice (mandated)	800 (35.9)
Social Programs (mandated)	617 (27.7)

Note. Data shown are N (%) unless otherwise indicated.

¹Missing data ranged from 0.3-0.9%

Appendix Table 1.

Differences between patients who completed BAM and those who did not

	Overall (N = 3202)	Completed BAM (N = 2227)	Did Not Complete BAM (N = 975)	p
Age (years), median (IQR)	33 (27-41)	33 (26-40)	35 (29-43)	< 0.001
Female (vs. Male)	1548 (48.3)	1091 (49.0)	457 (46.9)	0.287
Hispanic (vs. non-Hispanic)1	54 (1.7)	39 (1.8)	15 (1.5)	0.789
Race1
White	1995 (62.4)	1307 (58.7)	688 (70.6)	< 0.001
Black/African-American	1120 (35.0)	858 (38.6)	262 (26.9)	< 0.001
Other	84 (2.6)	60 (2.7)	24 (2.5)	0.796
Homeless (vs. Housed)1	220 (6.9)	113 (5.1)	107 (11.0)	< 0.001
Unemployed (vs. Employed)1	1755 (54.9)	1184 (53.2)	571 (58.7)	0.005
Primary SUD
Alcohol	878 (27.4)	627 (28.2)	251 (25.7)	0.172
Opioid	855 (26.7)	438 (19.7)	417 (42.8)	< 0.001
Cannabis	1044 (32.6)	879 (39.5)	165 (16.9)	< 0.001
Other Drug	425 (13.3)	283 (12.7)	142 (14.6)	0.171
Referral Source
Self (non-mandated)	1245 (38.9)	719 (32.3)	526 (53.9)	< 0.001
Medical/Behavioral (non-mandated)	164 (5.1)	91 (4.1)	73 (7.5)	< 0.001
Criminal Justice (mandated)	1008 (31.5)	800 (35.9)	208 (21.3)	< 0.001
Social Programs (mandated)	785 (24.5)	617 (27.7)	168 (17.)	< 0.001

Note. Data shown are N (%) unless otherwise indicated.

¹Missing data ranged from 0.3-0.9%

2.3. Measures

The version of the BAM used in this study consists of 16 items that require a single response and one multi-response item. Ten items use a 5-point Likert scale to represent the number of days patients engaged with targeted SUD-related behaviors (response choices: 0, 1-3, 4-8, 9-15, 16-30), five items use a 5-point Likert scale rating from ‘Not at All’ to ‘Extremely,’ one item with a 5-point Likert scale rating from ‘Poor’ to ‘Excellent,’ and a single yes/no item. One item (item 7) is a multi-response item asking about the frequency of drug use over the past month from various categories. Consistent with prior studies (c.f., Cacciola et al., 2013; Nelson et al., 2014; Gaddy et al., 2018), only the 16 items requiring a single response were included in analyses.

2.4. Statistical analysis

All analyses had a p < 0.05 unless otherwise specified and were conducted using Python (v3.9.5). Factor analysis was performed using the Python factor-analyzer (v0.4.0) and semopy (v2.3.9) packages.1

Initially, an exploratory analysis of the data was performed to evaluate data quality and to check that requirements of statistical tests were met. Before submitting the BAM to factor analysis, the directionality of six questions was reversed so that higher scores on all items would indicate increased substance-related problem severity (see Table 2 for descriptive statistics). Bartlett's test of sphericity (Bartlett, 1950) was used to test that the correlation matrix was not random and the Kaiser-Meyer-Olkin (KMO) measure (Kaiser, 1974) was used to assess the level of factor stability, with values above 0.7 considered to be desirable (Hoelzle & Meyer, 2013). Next, confirmatory factor analyses (CFAs) were performed to evaluate the measurement model validity of previously defined latent structures. Because the version of the BAM employed in this study uses categorical (binary or ordinal) responses, all CFA models used diagonally weighted least squares (DWLS) procedures which are more robust to nonnormality due to categorization (Li, 2015). First, a 3-factor model was specified in which 3 items loaded on the Substance Use factor, 6 items loaded on the Risk Factors factor, and 6 items on the Protective Factors factor, corresponding with the clinical scoring guidelines resulting from early studies (Cacciola et al., 2013). We also specified a 4-factor model in which 2 items loaded on an Alcohol Use factor, 4 items loaded on a Risk factor, 3 items loaded on a Stressors factor, and 4 items loaded on a Stability factor, which correspond to subsequent studies on the discrete BAM version resulting in a model that was found to be reliable across time points (Gaddy et al., 2018).

Table 2.

BAM item response characteristics at intake (N = 2,227)

BAM item description	Mean (SD)	Median	Skew	Kurtosis
1. Rating of physical health	1.82 (1.13)	2.00	0.14	-0.71
2. Days trouble sleeping	1.64 (1.71)	1.00	0.34	-1.62
3. Days psychological problems	1.57 (1.65)	1.00	0.42	-1.51
4. Days alcohol use	0.81 (1.30)	0.00	1.45	0.73
5. Days heavy alcohol use	0.58 (1.18)	0.00	1.98	2.59
6. Days drug use	1.24 (1.65)	0.00	0.81	-1.09
8. Rating of craving	1.11 (1.39)	0.00	0.96	-0.47
9. Rating of abstinence confidence1	2.76 (1.23)	3.00	0.59	-0.63
10. Days of self-help group attendance1,2	0.29 (0.87)	0.00	-3.22	9.55
11. Days in risky situations	1.24 (1.55)	0.00	0.81	-0.97
12. Rating of religion/spirituality support1	1.34 (1.63)	0.00	-0.67	-1.24
13. Days structured activities (e.g., work, school)1,2	1.86 (1.84)	2.00	-0.10	-1.85
14. Adequate income (% yes)1	72.2%	0.00	0.99	-1.02
15. Rating of arguments with friends/family	0.80 (1.18)	0.00	1.43	0.99
16. Days with supportive family/friends1,2	3.14 (1.32)	4.00	1.39	0.56
17. Recovery satisfaction	2.63 (1.19)	2.00	0.36	-0.78

¹Directionality was reversed to establish uniform directionality for interpretation of results in factor analysis

²Excluded in final model

To determine model fit, given there are no clear guidelines for when DWLS is applied to ordered categorical data, we relied on multiple conventional CFA indices, including root mean square error of approximation (RMSEA) estimates, the comparative fit index (CFI), and the Tucker-Lewis index (TLI). DWLS indices tend to show a better model-data fit evaluation than conventional maximum likelihood indices and often surpass conventional cutoffs for acceptable fit (Xia & Yang, 2019), namely, RMSEA < 0.06, and CFI and TLI > 0.95 (Hu & Bentler, 1999). For the current CFA models, we used these conventional cutoffs as guidance and to inform model improvement.

After finding a lack of measurement invariance of the 3- and 4-factor structures from prior studies, exploratory factor analysis (EFA) was performed to identify the psychometric properties and dimensionality of the BAM in a non-veteran population. Because of its tolerance of nonnormality and demonstrated ability to recover relatively weak factors (Briggs & MacCallum, 2003; De Winter & Dodou, 2012), an iterated principal axis (IPA) estimation method was used. This estimation method makes no distributional assumptions, producing an ordinary least squares solution. Parallel analysis, the visual scree test, and theoretical convergence were used to determine the appropriate number of factors to retain for rotation. Because factors were assumed to be correlated, we employed an oblimin rotation.

Considering parsimony and simple structure, we established a set of a priori criteria for determining factor adequacy. While many statistical programs use a default Eigenvalue of 1 as a cut-off score to retain factors, the present analyses aligned with statistical best practices and employed several other criteria instead (c.f., Costello & Osborne, 2005; Watkins, 2018). The practical usefulness of pattern coefficients often lie in the 0.3 to 0.4 range (Bandalos & Gerstner, 2016; Hair et al., 2010; Costello & Osborne, 2005), thus pattern coefficients ≥ 0.35 were considered salient. Complex loadings (i.e., where the loading is ≥ 0.35 on more than one factor) were rejected in favor of a simple structure, unless there was a clear theoretical reason to believe that the measured variable was influenced by more than one latent factor. We considered factors with a minimum of two salient pattern coefficients to be adequate as long as the two items are highly correlated (i.e., r > .70) and relatively uncorrelated with other variables (Worthington & Whittaker, 2006). Finally, items identified in the EFA were standardized, and reliability was assessed on the standardized items. Factors with an internal consistency reliability of 0.7 or higher and that were theoretically meaningful were considered to be adequate.

2.5. Subgroup analysis

Subgroup analyses were performed based on primary SUD (opioid, alcohol, cannabis), racial identity (White, Black/African-American), and referral source (mandated, not mandated) to assess the potentially differential factor structures that were derived from EFA.

3. Results

3.1. Patient descriptive statistics

A total of 2,227 patients were included in the analysis. The majority (59%) self-identified as White, while 39% identified as Black/African-American and less than 3% identified as other. Over a third of patients were diagnosed with primary cannabis use disorder (CUD; 39%), followed by alcohol use disorder (AUD; 28%), OUD (20%), and other substance use disorders (13%). Nearly two-thirds were mandated treatment (i.e., referred from criminal justice or social programs), while the remaining were either self- or medically-referred (see Table 1).

3.2. Item baseline response characteristics

There were no missing BAM responses. The most frequently used substance was cannabis (43.5%), followed by alcohol (36%), opiates (15.4%), other stimulants (12.7%), sedatives (9.3%), cocaine (8.6%), other drugs (6.2%) and inhalants (0.4%). Forty-one percent reported no drug use, and most (64%) reported no alcohol use in the past 30 days. Over two-thirds (72%) reported having adequate income for necessities. Skewness and kurtosis for 15 of the 16 items were appropriate for EFA, (i.e., ≤ 2 and ≤ 7, respectively; Curran et al., 1996). The skewness and kurtosis of one item (self-help group attendance) fell outside the appropriate range (skewness = -3.22, kurtosis = 9.55) but this was consistent with 88% of the patients reporting zero days of attending self-help groups. Because IPA estimation accounts for nonnormality, no transformations were conducted for this item.

3.3. Confirmatory factor analyses

For both the 3- and 4-factor models, we observed negative factor loadings for the self-help groups and religion support items. For the 3-factor model, standardized factor loadings for items that loaded onto the Substance Use factor were 0.45-0.46, 0.43-0.80 for Risk Factors items, and absolute values of 0.16-0.59 for Protective Factors items, all p<0.05). For the 4-factor model, all items for the Alcohol Use and Stressors factors displayed substantial factor saturation, as indicated by high standardized factor loadings (0.59-0.92). Loadings for the Risk factor were 0.44-0.85, while loadings for the Stability factor were all <0.1. Model fit for both the 3- and 4-factor models were not acceptable per RMSEA, CFI and TLI values using conservative conventional criteria (Table 3).

Table 3.

Confirmatory factor analysis model fit of previously defined models

	Fit indices
Model	χ2	df	p	CFI	TLI	RMSEA
Three factors	1,678	87	< .001	0.881	0.857	0.091
Four factors	973	59	< .001	0.922	0.896	0.083

Note: see Cacciola et al. (2013) for 3-factor model and Gaddy et al. (2018) for 4-factor model

3.4. Exploratory factor analyses

Bartlett's test of sphericity indicated the correlation matrix was not random (p < 0.001) and the KMO measure was 0.81, indicating that the correlation matrix was appropriate for factor analysis.

An initial EFA was performed on the full cohort using all BAM items except the multi-response item (item 7). Three items (self-help groups, structured activities, and social support contact) failed to exhibit any loadings of at least 0.35. EFAs including two of these three produced similar results, and thus, these items were excluded from subsequent EFA. Following removal of the three items, scree plots and parallel analysis suggested that three factors should be retained. The resulting pattern correlation matrix derived factors that were difficult to interpret theoretically. For example, risky situations loaded onto a factor along with abstinence confidence and recovery satisfaction.

We attempted a four-factor solution, as demonstrated in (Gaddy et al., 2018; Hallinan et al., 2021), with the remaining 13 items, which resulted in a more interpretable finding, with the four factors explaining 50.6 percent of the variance (see Table 3). The first factor was saliently loaded by items representing physical and psychological stressors and accounted for 19.1% of the variance. A second Alcohol Use factor accounted for 13.5% of the variance. A third factor, representing stabilizing and protective variables, accounted for 9.1% of the variance. The fourth factor, representing drug use and risky situations, accounted for another 8.9% of the variance. Weighing overfactoring (too many factors included) against underfactoring (too few factors included), as well as interpretability, we chose to retain the 4-factor solution, since overfactoring alters the solution less than underfactoring (Watkins, 2018).

3.4.1. Subgroup exploratory factor analyses

Bartlett's test of sphericity was significant (p < 0.001) and the KMO statistic was > 0.7 for all subgroup EFAs, suggesting that the correlation matrices were appropriate for subsequent factor analysis.

3.4.1.1. Primary substance use disorder

When EFA was repeated separately for patients with primary OUD, AUD, and CUD, the number of factors and items that were saliently loaded onto each factor varied. For example, while the alcohol use and heavy alcohol use items loaded onto its own factor for the OUD and CUD subgroups, the two alcohol use items loaded onto a factor along with drug use and risky situations in the AUD subgroup. While drug use and risky situations loaded onto a separate factor for the CUD subgroup, similar to the whole sample EFA, these items either cross-loaded or were not saliently loaded onto any factor in the OUD subgroup. The pattern matrix resulting from the OUD subgroup was unique in that it yielded a 4-factor solution, including two distinct factors that both represented protective and stability factors. The variance explained by the factor structures was 39.3% in the OUD group, 47.3% in the AUD group, and 49.8% in the CUD group (see Figure 1 and Appendices 2, 3 4).

Figure 1.

Factor characteristics and factor distributions by primary SUD.

Note: Protective Factors II refers to the unique factor structure that appeared only in the opioid use disorder subgroup, whereas Protective Factors I was evident across all subgroups.

Appendix Table 2.

BAM factor loading in the primary Opioid Use Disorder population (N = 438)

	Stressors	Alcohol Use	Protective Factors I	Protective Factors II	Communality
Physical health	0.506	0.013	0.186	0.034	0.292
Sleeping problems	0.745	-0.023	-0.102	-0.009	0.566
Psychological problems	0.758	0.018	-0.094	0.026	0.584
Alcohol use	0.031	0.841	0.01	-0.043	0.71
Heavy alcohol use	-0.027	0.936	-0.019	0.022	0.878
Drug use	0.432	0.046	0.383	-0.041	0.338
Cravings	0.51	0.081	0.263	0.126	0.352
Abstinence confidence	-0.08	0.006	0.58	0.038	0.345
Support group	0.064	-0.051	0.424	-0.316	0.286
Risky situations	0.345	0.045	0.278	0.09	0.206
Religion / spirituality	-0.257	0.002	0.385	-0.123	0.229
Structured activities	-0.058	-0.014	0.089	0.474	0.236
Adequate income	0.17	-0.037	0.006	0.47	0.251
Arguments with family / friends	0.532	0.027	0.046	-0.031	0.286
Recovery satisfaction	0.036	0.034	0.534	0.206	0.33
% variance explained by factor	15.6	10.7	8.7	4.2

Note. Bolded loadings are those at or above 0.35.

Appendix Table 3.

BAM factor loading in the primary Alcohol Use Disorder population (N = 627)

	Alcohol Use and Risk	Stressors	Protective Factors	Communality
Physical health	-0.062	0.481	0.281	0.314
Sleeping problems	-0.001	0.821	-0.055	0.678
Psychological problems	-0.006	0.862	0.039	0.744
Alcohol use	0.879	0.003	0.039	0.774
Heavy alcohol use	0.933	-0.033	0.001	0.871
Drug use	0.405	0.06	0.011	0.168
Cravings	0.459	0.411	0.07	0.385
Abstinence confidence	0.065	-0.025	0.715	0.516
Risky situations	0.482	0.041	0.08	0.24
Religion / spirituality	-0.129	-0.189	0.432	0.239
Arguments with family / friends	0.311	0.459	-0.124	0.323
Recovery satisfaction	0.063	0.082	0.646	0.428
% variance explained by factor	19.8	17.3	10.2

Note. Bolded loadings are those at or above 0.35.

Appendix Table 4.

BAM factor loading in the primary Cannabis Use Disorder population (N = 879)

	Stressors	Alcohol Use	Protective Factors	Risk	Communality
Physical health	0.338	-0.047	0.165	0.137	0.163
Sleeping problems	0.71	-0.04	-0.115	-0.072	0.525
Psychological problems	0.826	-0.028	-0.005	0.021	0.684
Alcohol use	0.018	0.839	-0.04	0.061	0.71
Heavy alcohol use	-0.018	0.889	0.023	-0.052	0.793
Drug use	-0.001	-0.004	-0.049	0.881	0.778
Cravings	0.497	0.133	0.058	0.242	0.326
Abstinence confidence	-0.048	-0.009	0.71	0.042	0.508
Risky situations	-0.02	0.03	0.158	0.663	0.465
Religion / spirituality	-0.197	-0.063	0.448	-0.01	0.244
Adequate income	0.409	0.02	0.047	-0.085	0.177
Arguments with family / friends	0.631	0.096	0.112	-0.051	0.422
Recovery satisfaction	0.057	0.007	0.818	-0.005	0.673
% variance explained by factor	16.6	11.8	11.2	10.1

Note. Bolded loadings are those at or above 0.35.

3.4.1.2. Referral source

In the subgroup analysis by referral type, the number of factors and pattern matrices significantly varied. The EFA for non-mandated referrals yielded 3 factors, representing Stressors, Alcohol Use, and Protective Factors, accounting for 50.2% of the variance. Items measuring drug use and risky situations failed to load onto any factor and only 10 total items were retained. The mandated referrals subgroup yielded a 4-factor solution that resulted in a pattern matrix that was identical to the whole sample EFA (i.e., Stressors, Alcohol Use, Stability, and Risk factors). The 4 factors accounted for 48.9% of the variance (see Figure 2 and Appendices 5-6).

Figure 2.

Factor characteristics and factor distributions by referral source

Appendix Table 5.

BAM factor loading in patients self or medically referred (N = 719)

	Stressors	Alcohol Use	Protective Factors	Communality
Physical health	0.5	-0.061	0.18	0.50
Sleeping problems	0.796	-0.012	-0.043	0.796
Psychological problems	0.837	-0.009	-0.046	0.837
Alcohol use	-0.005	0.944	0.002	-0.005
Heavy alcohol use	0.007	0.903	-0.004	0.007
Cravings	0.503	0.164	0.208	0.503
Abstinence confidence	-0.061	0.025	0.647	-0.061
Adequate income	0.347	-0.13	0.123	0.347
Arguments with family / friends	0.508	0.09	0.022	0.508
Recovery satisfaction	0.036	-0.013	0.724	0.036
% variance explained by factor	22.2	17.6	10.4

Note. Bolded loadings are those at or above 0.35.

Appendix Table 6.

BAM factor loading in criminal justice or social program referrals (N = 719)

	Stressors	Alcohol Use	Protective Factors	Risk	Communality
Physical health	0.381	0.041	0.231	0.038	0.202
Sleeping problems	0.739	-0.021	-0.097	-0.076	0.562
Psychological problems	0.84	-0.001	0.031	-0.025	0.708
Alcohol use	-0.019	0.922	-0.035	0.023	0.852
Heavy alcohol use	0.019	0.891	0.032	-0.034	0.797
Drug use	-0.012	-0.069	-0.042	0.751	0.571
Cravings	0.505	0.093	0.057	0.296	0.354
Abstinence confidence	-0.056	0.027	0.697	0.008	0.489
Risky situations	-0.008	0.086	0.082	0.687	0.486
Religion / spirituality	-0.216	-0.052	0.443	-0.028	0.247
Adequate income	0.415	-0.039	-0.009	0.006	0.174
Arguments with family / friends	0.591	0.052	0.004	0.064	0.357
Recovery satisfaction	0.072	-0.016	0.741	0.017	0.555
% variance explained by factor	17.2	12.9	10.1	8.8

Note. Bolded loadings are those at or above 0.35.

3.4.1.3. Race

When EFA was conducted separately among patients who identified as Black/African-American and in those who identified as White, scree plots and parallel analysis indicated 3-factor solutions. However, similar to the whole sample EFA, the 3-factor solution yielded a factor that did not theoretically converge (e.g., drug use with abstinence confidence), thus we retained the 4-factor solution. The pattern matrix in the Black/African-American subgroup was identical to that resulting from the whole sample EFA, accounting for 51.2% of the variance, and the pattern matrix in the White subgroup was nearly identical to that resulting from the Black/African-American subgroup except cravings cross-loaded on 2 factors and was consequently deleted. The 4 factors resulting from the White subgroup EFA accounted for 49.6% of the variance (see Figure 3 and Appendices 7-8).

Figure 3.

Factor characteristics and factor distributions by race

Appendix Table 7.

BAM factor loading in the self-identified Black/African-American population (N = 858)

	Stressors	Alcohol Use	Protective Factors	Risk	Communality
Physical health	0.398	0.029	0.235	0.047	0.217
Sleeping problems	0.847	-0.05	-0.018	-0.069	0.726
Psychological problems	0.837	-0.005	0	0.011	0.70
Alcohol use	0.012	0.911	-0.023	0.004	0.831
Heavy alcohol use	-0.013	0.92	0.016	-0.019	0.848
Drug use	0.004	-0.071	-0.025	0.817	0.673
Cravings	0.525	0.199	0.084	0.236	0.378
Abstinence confidence	0.009	0.002	0.723	-0.003	0.522
Risky situations	-0.026	0.139	0.1	0.598	0.388
Religion / spirituality	-0.331	-0.081	0.43	-0.011	0.301
Adequate income	0.459	0.007	0.053	-0.05	0.216
Arguments with family / friends	0.57	0.118	-0.001	0.085	0.346
Recovery satisfaction	0.054	0.007	0.708	0.027	0.504
% variance explained by factor	19.2	13.6	9.9	8.5

Note. Bolded loadings are those at or above 0.35.

Appendix Table 8.

BAM factor loading in the self-identified White population (N = 1,307)

	Stressors	Alcohol Use	Protective Factors	Risk	Communality
Physical health	0.466	-0.012	0.131	0.189	0.27
Sleeping problems	0.826	0.001	-0.081	-0.044	0.691
Psychological problems	0.857	0.002	-0.012	0.036	0.736
Alcohol use	0.004	0.899	-0.014	0.025	0.81
Heavy alcohol use	-0.01	0.954	-0.003	-0.015	0.91
Drug use	0.009	-0.091	0.709	0.032	0.513
Cravings	0.414	0.163	0.41	0.034	0.367
Abstinence confidence	-0.04	0.02	0.021	0.707	0.502
Risky situations	-0.064	0.07	0.67	0.052	0.461
Religion / spirituality	-0.217	-0.032	-0.186	0.417	0.257
Adequate income	0.395	-0.09	0.193	-0.017	0.202
Arguments with family / friends	0.502	0.082	0.186	-0.029	0.294
Recovery satisfaction	0.122	0.021	0.058	0.641	0.43
% variance explained by factor	17.6	13.7	9.6	8.7

Note. Bolded loadings are those at or above 0.35.

3.5. Internal consistency

3.5.1. Four-factor solution from current study

In the whole sample, the internal reliability varied greatly. Cronbach's alpha values for the 4-factor solution were 0.82, 0.92, 0.58, and 0.67 for the Stressors, Alcohol Use, Stability, and Risk factors, respectively. In the subgroup analyses by primary SUD, the internal consistency was acceptable or good for both the Stressors and Alcohol Use factors (i.e., α ≥ 0.75) and poor or questionable for the Stability factor (e.g., α = 0.53 - 0.69) for all subgroups (George & Mallory, 2003). The additional Stability factor resulting from the OUD EFA exhibited even poorer internal consistency (α = 0.44). Among the primary SUD subgroups, the Stability and Risk factors exhibited the greatest internal reliability in the CUD subgroup (α = 0.69 and 0.77, respectively). In both referral source subgroups, internal consistency was acceptable or good for the Stressors and Alcohol Use factors (α = 0.76 - 0.92) and questionable for the Stability and Risk factors (i.e., 0.6 ≤ α < 0.7), exhibiting similar internal reliability to that in the whole sample. Finally, alpha values exhibited similar patterns in both racial subgroups as that in the whole sample. In the whole sample and across all subgroups, Cronbach's alpha was greatest for the Alcohol Use scale and either poor or questionable for pattern matrices resulting in Risk or Protective Factors scales.

3.5.2. Three-factor solution from original study

The BAM structure suggested by clinical scoring guidelines (Cacciola et al., 2013; Department of Veterans Affairs, 2009) included three factors: Use, Risk Factors, and Protective Factors. Additional internal consistency analyses assessed the reliability of the proposed item clusters among the whole sample and within each subgroup. Internal consistency for the Use Factor ranged from poor to questionable for all groups except the AUD group, which produced an adequate internal reliability (α = 0.77). The Risk Factor produced acceptable or good reliability for all groups except for the CUD subgroup. Internal consistency was unacceptable for the Protective Factors across all subgroups (see Appendix 9).

Appendix Table 9.

Internal consistency of clinical scoring guidelines item clusters

Group	Use	Risk	Protective Factors
Overall	0.61	0.82	0.29
Opioid	0.64	0.78	0.41
Alcohol	0.77	0.82	0.34
Cannabis	0.68	0.69	0.21
Self/Medical	0.52	0.78	0.40
Criminal Justice/Social Program	0.61	0.73	0.24
Black/African-American	0.67	0.79	0.24
White	0.58	0.81	0.30

Note. Bolded values indicate fair to good internal consistency.

4. Discussion

This study evaluated the factor structure and psychometric performance of the BAM in a non-veteran population to assess the instrument's validity among a more heterogeneous SUD treatment-seeking group than the tool originally had been developed to evaluate. Several findings suggest that the BAM might not be a reliable and valid instrument for all populations. First, we excluded self-help groups, structured activities, and social support contact because of the poor factor loadings. However, previous research has defined these items as strongly associated with recovery (White & Kurtz, 2005; Laudet et al., 2006; Kaskutas, 2002), which suggests that we lost valuable recovery capital elements.

Patient-reported outcomes used in measurement-based care (MBC) need to be valid across health states in order for the point-in-time interpretation to be accurate, and the distance between health states to be precise when measured over time (Ding, 2005). If the BAM cannot demonstrate multidimensional scaling, then it does not have test-retest reliability and cannot be used to evaluate recovery progress (Anselmi et al., 2015). The BAM scoring guidelines suggest that changes are observable on the subscales across multiple administrations, but our study questions the use case. The recovery science field requires valid instruments that enable measurement of treatment quality and clinical outcomes over time.

Secondly, across the subgroups, we found that the BAM had inconsistent factor structures and loading patterns. The number of factors and item loadings varied depending on primary SUD diagnoses, suggesting that the BAM might not be an appropriate tool for non-veteran populations without primary AUD. The different factor structures by referral source also suggest that the BAM inconsistently performs based upon an underlying factor, perhaps recovery motivation, because items loaded differently when comparing the self or medical (non-mandated) referral group to the social programs or criminal justice (mandated) referred group. We posit that motivation to change impacts the effectiveness of the BAM to accurately capture psychometric properties related to measuring recovery. Motivation is not tested in the BAM, despite being a predictor of SUD outcomes (Coviello et al., 2012; Kelly et al., 2014); and, as a result, might be missing an important indicator that explains variation in recovery for different populations (see also Rivera et al., 2021).

The subgroup results also have implications for future analyses using the BAM. The lack of equivalent latent structures, or configural invariance, as described by Lopez-Vergara et al. (2021), suggests that the same latent constructs are not being measured across subgroups, implying a lack of equivalent instrument functioning between groups and thereby compromising the validity of inferences based on observed group similarities or differences on BAM responses.

On the other hand, we observed similar factor structure and item loadings comparing the Black/African-American and White subgroups, with the exception of cravings. Previous research has called for the need to validate tools across all groups, because unintended consequences can result when tools have not been validated across all groups, leading to health disparities among minoritized groups that were not adequately represented (Liu et al., 2019). Testing the sensitivity and specificity of clinical instruments is critical for assuring MBC returns value to all patients.

Third, we were not able to find support of measurement invariance of previously proposed factor structures, including the BAM's original three subscales. Subscales are useful in clinical practice because they provide interpretive value to clinicians and patients who often use them in developing treatment plans. The care team might identify areas of risk and focus on the domains that will mitigate these factors. However, the lack of subscale validity does not support targeted clinical practice interventions. Furthermore, the inability to confirm the subscales and replicate the factor structure suggests that the BAM might not be a multidimensional measure, or that the dimensions are not defined well by the items.

4.1. Limitations

This study has some limitations. The generalizability of our results might be limited due to data collection from a single treatment program. Patients in the dataset likely live in the same geographic area, and have similar environmental exposures which impacts treatment outcomes in the same way. Further, the sample was limited to outpatients, so we cannot assess whether the level of care during BAM administration might have changed the results. Finally, patients who completed the BAM differed from those who did not complete the BAM on demographic variables, SUD diagnosis, and referral source, thereby limiting the generalizability of the findings.

4.2. Conclusion

The use of MBC has demonstrated effectiveness in improving patient outcomes (Aboraya et al. 2018), and MBC, which relies upon patient reported outcomes that assess health states from the patient's point of view, is critical to evaluating quality of care and services in SUD recovery which is multidimensional and personally defined. Yet, few instruments measure SUD recovery progress, despite the value that measuring recovery has for patients (Cuperfain, 2021), providers, and health systems (Valenstein, 2009). Despite the need for more SUD quality and performance measures, it is critically important for the addiction recovery field to be made aware of the inability to replicate previous findings. Furthermore, existing measures might not reliably and validly measure recovery across different subpopulations, which can contribute to unequal outcomes based on sociodemographic and clinical factors. Therefore, we believe that more research is needed to develop and validate tools that are clinically meaningful and that track recovery progress over time. Table 4, Table A1, Table A2, Table A3, Table A4, Table A5, Table A6, Table A7, Table A8, Table A9

Table 4.

BAM factor loading in full patient cohort (N = 2,227)

	Stressors	Alcohol Use	Protective Factors	Risk	Communality
Physical health	0.459	-0.005	0.203	0.094	0.26
Sleeping problems	0.839	-0.013	-0.035	-0.087	0.712
Psychological problems	0.855	0.003	0.027	-0.019	0.732
Alcohol use	-0.004	0.898	-0.002	0.006	0.807
Heavy alcohol use	-0.001	0.946	0.005	-0.021	0.895
Drug use	0.018	-0.088	0.01	0.726	0.536
Cravings	0.493	0.153	0.048	0.331	0.378
Abstinence confidence	-0.032	0.025	0.704	0.015	0.498
Risky situations	-0.051	0.085	0.052	0.677	0.47
Religion / spirituality	-0.275	-0.049	0.434	-0.114	0.279
Adequate income	0.448	-0.068	-0.001	0.106	0.216
Arguments with family / friends	0.546	0.087	-0.028	0.15	0.329
Recovery satisfaction	0.109	0.007	0.675	0.044	0.469
% variance explained by factor	19.1	13.5	9.1	8.9

Note. Bolded loadings are those at or above 0.35.

Role of Funding Source

Financial support for this analysis was provided by Verily. Employees of Verily have participated in the conception, analysis, and dissemination of results from this study, and collaborated on writing/editing the article with employees of OneFifteen and Wright State University.

Contributors

Jeremiah Schumm, Elisabeth Okrant, Jordan Thorpe, and Jared Embree contributed to the conceptualization and writing. Celeste Wong conducted the analyses and contributed to the writing. Natalie Lester contributed to the conceptualization.

Funding

Support for this work was provided by OneFifteen, Inc., Samaritan Behavioral Health, Verily, LLC, and Wright State University.

Conflict

No conflict declared.