An early evaluation of implementation of brief intervention for unhealthy alcohol use in the US Veterans Health Administration

Abstract

Aims

The US Veterans Health Administration [Veterans Affairs (VA)] used performance measures and electronic clinical reminders to implement brief intervention for unhealthy alcohol use. We evaluated whether documented brief intervention was associated with subsequent changes in drinking during early implementation.

Design

Observational, retrospective cohort study using secondary clinical and administrative data.

Setting

Thirty VA facilities.

Participants

Outpatients who screened positive for unhealthy alcohol use [Alcohol Use Disorders Identification Test Consumption (AUDIT-C ≥ 5)] in the 6 months after the brief intervention performance measure (n = 22 214) and had follow-up screening 9–15 months later (n = 6210; 28%).

Measurements

Multi-level logistic regression estimated the adjusted prevalence of resolution of unhealthy alcohol use (follow-up AUDIT-C <5 with ≥2 point reduction) for patients with and without documented brief intervention (documented advice to reduce or abstain from drinking).

Findings

Among 6210 patients with follow-up alcohol screening, 1751 (28%) had brief intervention and 2922 (47%) resolved unhealthy alcohol use at follow-up. Patients with documented brief intervention were older and more likely to have other substance use disorders, mental health conditions, poor health and more severe unhealthy alcohol use than those without (P-values < 0.05). Adjusted prevalences of resolution were 47% [95% confidence interval (CI) = 42–52%] and 48% (95% CI = 42–54%) for patients with and without documented brief intervention, respectively (P = 0.50).

Conclusions

During early implementation of brief intervention in the US Veterans Health Administration, documented brief intervention was not associated with subsequent changes in drinking among outpatients with unhealthy alcohol use and repeat alcohol screening.

INTRODUCTION

Multiple meta-analyses have confirmed the efficacy of brief interventions for reducing drinking among primary care patients with unhealthy alcohol use identified by routine alcohol screening [1–5]. Therefore, routine alcohol screening and brief intervention for outpatients who screen positive is a top prevention priority [6] and widely recommended [2,7,8].

However, brief intervention has been extremely challenging to implement into routine care [9–12]. Studies of primary care providers have described multiple knowledge, attitudinal and logistical barriers to implementation of brief intervention [13–18]. In healthcare settings without successful implementation, providers offer brief intervention primarily to patients with medical conditions related to alcohol use or to those with severe problems due to drinking [19–25], for whom brief intervention may be inadequate [2,26,27].

The Veterans Health Administration (VA) is the largest integrated healthcare system in the United States, caring for more than 5 million patients per year [28]. The VA screens more than 90% of regular users of outpatient care annually for unhealthy alcohol use [29], and uses national performance measures coupled with ‘clinical reminders’ (decision support tools embedded in the electronic medical record for prompting and documenting care) to incentivize recommended care [28,30,31]. On 1 October 2007, the VA implemented a national performance measure incentivizing brief intervention for patients who screened positive for unhealthy alcohol use [32] and, in January 2008, made an electronic clinical reminder for brief intervention available to all VA facilities nationally [32]. Manual medical record reviews revealed that documented brief intervention increased in association with both of these initiatives [32].

The VA has been recognized as a leader in implementation of brief intervention [11,33], and other systems— both within and outside the United States—are currently employing similar ‘systems-level’ [34] strategies to implement brief intervention [35,36]. Additional US healthcare systems are likely to follow suit due to recent healthcare reform, which established brief intervention as a standard benefit [37,38]. Therefore, VA's experience implementing brief intervention using systems-level strategies may inform other systems during implementation efforts.

The present study used an existing secondary data set to obtain an initial view of the reach and effectiveness [39,40] of VA's performance measure for brief intervention among a sample of patients who screened positive for unhealthy alcohol use in the 6 months after implementation of the performance measure and had follow-up screening 9–15 months later. Specifically, this study sought to describe patient characteristics associated with receipt of brief intervention, and to evaluate whether documented brief intervention was associated with resolution of unhealthy alcohol use at follow-up screening in this sample.

METHODS

Data sources and study sample

This observational, retrospective cohort study used an existing secondary data set, which was constructed for another study [41] with data from two national VA data sources: the Corporate Data Warehouse (CDW) and National Patient Care Databases (NPCD). The data set included clinical and administrative data for all Veteran outpatients who received care in VA Region 1 (30 medical centers in the northern and western United States) between 1 January 2004 and 31 December 2008 and were ‘regular users of care’, defined as having two documented alcohol screens at least 270 days apart [41]. Patients were eligible for the present study if they: (i) screened positive for unhealthy alcohol use on an initial screen in the first 6 months of the brief intervention performance measure (1 October 2007–4 April 2008); and (ii) had follow-up alcohol screening documented ≥270 days later but before the study end (31 December 2008). The VA Puget Sound Health Care System Institutional Review Board reviewed and approved the study, including waivers of informed consent and Health Insurance Portability and Accountability Act 1996 (HIPAA) authorization.

Measures

Unhealthy alcohol use

Unhealthy alcohol use was defined based on documented alcohol screening with the Alcohol Use Disorders Identification Test Consumption (AUDIT-C) questionnaire [29]. Although AUDIT-C scores of ≥3 for women and ≥4 for men optimize sensitivity and specificity for unhealthy alcohol use [42–44], the VA's performance measure requires documented brief intervention only among patients with AUDIT-C ≥ 5 in order to minimize burdening providers with follow-up of false-positive screens. Therefore, this evaluation of VA's brief intervention was conducted among patients targeted by the performance measure—those with initial AUDIT-C scores ≥ 5.

Documented brief intervention

Documented brief intervention was measured using text data elements called ‘health factors’ that are generated when care is documented using electronic clinical reminders [45]. VA facilities often develop their own clinical reminders to meet the requirements of VA performance measures, and health factors can be developed and edited locally [45]. For these reasons, and because the national clinical reminder for brief intervention— which included nationally standardized (but editable) health factors indicating alcohol-related advice—was not disseminated until January 2008, we abstracted and reviewed all alcohol-related health factors documented during the study period for each facility. Those indicating advice to reduce and/or abstain from drinking between initial and follow-up AUDIT-C screens were combined into a single dichotomous measure of documented brief intervention. While this measure is less stringent than the VA's brief intervention performance measure, which requires both advice to reduce and/or abstain from drinking and feedback linking alcohol use to health [32], feedback is not captured using health factors and is not available in VA's CDW. However, because the clinical reminder typically required documentation of both advice and feedback, alcohol-related advice was considered a good proxy for care consistent with VA's brief intervention performance measure.

Outcomes—resolution of unhealthy alcohol use

Resolution of unhealthy alcohol use was defined, consistent with previous studies [46,47], as screening negative on the follow-up AUDIT-C screen with a score ≤ 5 and at least a 2-point score reduction.

Facility

Patients were assigned to a VA healthcare facility based on the facility (n = 30) where they received initial AUDIT-C screening.

Covariates

Covariates, reflecting demographics, severity of unhealthy alcohol use, other substance use and physical and mental health comorbidity, were selected based on known associations with both brief intervention and resolution of unhealthy alcohol use [19,23,24,48–53]. Demographics included age in years (categorized into 25–34, 35–49, 50–65, >65 years), gender, marital status (married/widowed versus other) and VA eligibility status (exempt from mandatory co-payment versus not). Four measures of severity of unhealthy alcohol use were derived using alcohol screening and diagnostic data. Because higher AUDIT-C scores indicate greater severity [54–56], initial AUDIT-C scores were used to create baseline AUDIT-C severity categories (scores of 5, 6–7, 8–9 and 10–12, representing mild, moderate, severe and very severe unhealthy alcohol use, respectively). Patients were classified as having past-year addictions treatment if they had any visit for VA addictions treatment documented in the year prior to the initial AUDIT-C. International Classification of Diseases, Ninth Revision Clinical Modification (ICD-9 CM) diagnosis codes documented in the year prior to the initial AUDIT-C were used to define any alcohol use disorder (ICD-9 CM for alcohol abuse, dependence, intoxication or withdrawal) and any alcohol-specific medical condition (ICD-9 CM for alcoholic liver disease, alcoholic cardiomyopathy, alcoholic polyneuropathy or peripheral neuropathy, alcoholic gastritis or alcoholic psychosis or dementia). Patients were considered to have past-year tobacco use if, in the year prior to the initial AUDIT-C, they had a tobacco diagnosis or health factor indicating current smoking [45]. ICD-9 CM codes documented in the year prior to the initial AUDIT-C were used to identify any non-alcohol substance use disorder and any mental health condition (including major depression, anxiety or other mood disorders), as well as to derive the validated Deyo Comorbidity Index [57]. Deyo scores were dichotomized with scores ≥ 3 representing ‘high’ physical comorbidity [57].

Statistical analyses

Characteristics of the analytical study sample were described overall. The prevalence of brief intervention documented during the study period was described overall and by facility. To assess whether brief intervention was distributed equitably, patient characteristics were described and compared across documented brief intervention status using χ² tests of independence. To assess the potential for bias in the sample, characteristics of the analytical study sample were compared to regular users of care who screened positive during the first 6 months of brief intervention implementation but did not have a follow-up screen ≥ 270 days later and prior to the study end (31 December 2008).

Main analyses used multi-level logistic regression models to assess the association between documented brief intervention and resolution of unhealthy alcohol use at follow-up. Models were first unadjusted and then adjusted for all measured covariates, and included both random intercepts for facility and random slopes for brief intervention to account for correlation of patient outcomes at the level of the facility and to allow the association between documented brief intervention and resolution of unhealthy alcohol use to vary across facilities. The Delta method was used to obtain standard errors [58]. The main results are presented as the average adjusted predicted prevalence of resolution for patients with and without documented brief intervention, based on recycled predictions [58]. All analyses were performed using Stata version 12 [59].

RESULTS

Among 269 937 regular users of VA care with an initial AUDIT-C during the first 6 months of brief intervention implementation, 22 214 (8.2%) screened positive (AUDIT-C ≥ 5). Of patients with positive screens, 6210 (28%) had a follow-up AUDIT-C documented ≥270 days later but before 31 December 2008 and were included in the analytical sample. Time between initial and follow-up screens ranged from 270 to 457 days (mean = 350).

Included patients were mostly male, white and aged ≥ 65 years (Table 1). The average number of eligible patients per facility was 207 (range 2–529 across 30 facilities). Compared to the 16 004 patients with positive initial, but no follow-up, alcohol screens, the study sample had slightly higher proportions of patients who were married or widowed, in the youngest and middle age groups and who had a mental health diagnosis (Supporting information, Table S1). No other significant differences between samples were observed.

Table 1.

Sample characteristics among Veterans Affairs (VA) outpatients who screened positive for unhealthy alcohol use on an initial screen and had follow-up screening at least 270 days later: overall and compared across documented brief intervention (BI)^a.

	Total (n = 6210)		No BI (n = 4459)		BI (n = 1751)
Characteristics	n	(%)	n	(%)	n	(%)	P-value
Age (years)							<0.001
25–34	169	(3)	135	(3)	34	(2)
35–50	523	(8)	406	(9)	117	(7)
50–64	1281	(21)	874	(20)	407	(23)
≥65	4237	(68)	3044	(68)	1193	(68)
Female	161	(3)	118	(3)	43	(3)	0.671
Married	3018	(49)	2190	(49)	828	(47)	0.195
Exempt from mandatory VA co-payment	4674	(75)	3302	(74)	1372	(78)	<0.001
Tobacco use (past year)	2849	(46)	1915	(43)	934	(53)	<0.001
Any mental health	2452	(40)	1592	(36)	860	(49)	<0.001
High physical comorbidity (Deyo ≥ 3)	531	(9)	353	(8)	178	(10)	0.004
Baseline AUDIT-C severity category							<0.001
Mild (AUDIT-C 5)	2051	(33)	1590	(36)	461	(26)
Moderate (AUDIT-C 6–7)	1854	(30)	1323	(30)	531	(30)
Severe (AUDIT-C 8–9)	1119	(18)	771	(17)	348	(20)
Very severe (AUDIT-C 10–12)	1186	(19)	775	(17)	411	(24)
Any alcohol use disorder	2193	(35)	1442	(32)	751	(43)	<0.001
Any addictions treatment	618	(10)	381	(9)	237	(14)	<0.001
Any non-alcohol SUD diagnosis	729	(12)	440	(10)	289	(17)	<0.001
Any alcohol-specific medical conditions	173	(3.0)	115	(3)	58	(3)	0.144

^aBI = brief intervention documented in the electronic medical record with a clinical reminder; AUDIT-C = Alcohol Use Disorders Identification Consumption Test; SUD = substance use disorder.

Among the 6210 patients who screened positive for unhealthy alcohol use and had a follow-up alcohol screen, 1751 (28%) had documented brief intervention. The prevalence of documented brief intervention ranged from 0 to 68% across the 30 facilities; seven facilities had prevalences < 10%.

Patients with documented brief intervention were older and more likely to be exempt from a mandatory VA co-payment, to use tobacco and to have a high level of physical comorbidity, mental health conditions and non-alcohol substance use disorders than those without (Table 1). Patients with documented brief intervention also had more severe unhealthy alcohol use, as indicated by a higher prevalence of: AUDIT-C scores indicating severe and very severe unhealthy alcohol use; alcohol use disorders; and past-year addictions treatment (Table 1).

Overall, 2922 (47%) patients resolved unhealthy alcohol use at follow-up. The average change in AUDIT-C score from initial to follow-up screening was a decrease of 2.63 (range −12 to +7). In both unadjusted and adjusted analyses, no significant differences in the prevalence of resolution across documented brief intervention were identified (Table 2).

Table 2.

Association between documented BI^a and resolution of unhealthy alcohol use^b among Veterans Affairs (VA) outpatients who initially screened positive.

		Documented BI		Resolved unhealthy alcohol use		Unadjusted prevalence of resolved unhealthy alcohol use			Adjusted prevalence of resolved unhealthy alcohol use (RE)
						No BI	BI		No BI		BI
	n	n	(%)	n	(%)	%	%	P-valuec	%	(95% CI)	%	(95% CI)	P-valued
Main results in overall sample	6210	1751	(28%)	2922	(47%)	46%	49%	0.06	47%	(42–52%)	48%	(42–54%)	0.50
Results of secondary post-hoc analyses in subsamples defined by severity of unhealthy alcohol use
AUDIT-C < 8	3905	992	(25%)	1874	(48%)	47%	51%	0.02	48%	(43-53%)	51%	(45–56%)	0.19
AUDIT-C ≥ 8	2305	759	(33%)	1048	(45%)	45%	46%	0.66	45%	(40–50%)	45%	(39–51%)	0.90
Sample without AUD/TX	3979	987	(24%)	1861	(47%)	46%	48%	0.26	47%	(43–53%)	48%	(42–54%)	0.57

^aBI = brief intervention documented in the electronic medical record with a clinical reminder.

^bPrimary study outcome defined as screening negative on the follow-up Alcohol Use Disorders Identification Test Consumption (AUDIT-C) with at least a 2-point reduction in score.

^cχ² test for difference in proportion of patients with resolution.

^dWald test for multi-level regression model coefficients for documentation of brief intervention (yes/no).

CI = confidence interval; AUD = alcohol use disorder; TX = receipt of specialty addictions treatment in the year prior to initial positive alcohol screening.

Because brief intervention has unclear efficacy for patients with the most severe unhealthy alcohol use [2], post-hoc exploratory analyses examined whether severity influenced the association between documented brief intervention and resolution. Patients’ baseline AUDIT-C scores were categorized into two groups (scores <8 and ≥8) based on increased probability of alcohol use disorder among patients with scores ≥8 [55,56,60]. Main analyses were repeated, stratified by this dichotomous measure of severity, and a multiplicative interaction between this measure and documented brief intervention was tested. No significant interaction between severity and documented brief intervention (P = 0.35) was detected, and no significant differences in resolution of unhealthy alcohol use were observed in either severity subgroup after appropriate adjustment for covariates (Table 2). Subsequently, main analyses were repeated only among patients without a documented alcohol use disorder or attendance at VA addictions treatment in the year prior to initial screening. Again, no significant differences in resolution of unhealthy alcohol use across brief intervention status were identified (Table 2).

Finally, because the conservative main outcome definition may have masked resolution for patients with an initial positive screen near the cut-point, main analyses were repeated to assess whether documented brief intervention was associated with screening below the cut-point at which VA incentivizes brief intervention (score < 5) without the requirement of a ≥2-point decrease. Results mirrored the main results with no identified differences in resolution based on documented brief intervention (data available upon request).

DISCUSSION

This observational retrospective cohort study of Veteran outpatients with unhealthy alcohol use and follow-up screening found limited reach of documented brief intervention during early brief intervention implementation. Specifically, only slightly more than one-quarter of patients had documented brief intervention, and rates of brief intervention varied substantially across facilities. Further, there were meaningful differences in patient characteristics between those with and without documented brief intervention. Similar to studies conducted in settings without widespread implementation practices, brief intervention was more likely among patients with severe unhealthy alcohol use and greater comorbidity than those without [19–25]. These findings suggest that systems-level implementation strategies may take time to achieve their full impact, initial reach may vary by site and early implementation efforts may not overcome biases regarding who is usually offered brief intervention. These findings highlight the need for evaluations in later stages of implementation. Although rates of brief intervention at the time of this study were only 28%, national rates of brief intervention reached 77% in 2010 [61], which may have resulted in a more even distribution of brief intervention across facilities and patient subpopulations.

This study also found that patients with documented brief intervention were not more likely to resolve unhealthy alcohol use at follow-up screening than those without. These findings held when we assessed a more sensitive outcome measure. Further, although unadjusted post-hoc analyses identified a slightly higher prevalence of resolution among patients with documented brief intervention than those without among patients with AUDIT-C scores < 8, after appropriate adjustment, null findings again held in this subgroup as well as in a subgroup that excluded patients with the most severe unhealthy alcohol use for whom more intensive interventions may be required [26,62]. However, these findings were in contrast to those of a pilot study conducted prior to implementation of the brief intervention performance measure at a single multi-clinic VA facility, which found that patients with documented brief intervention had a small but significant increase (3% difference) in the adjusted prevalence of resolution compared to those without [46]. Differences in findings between the two studies could be due to several factors. Most notably, due to a lower-than-expected rate of follow-up screening, the present study had only 62% power to detect the previously observed difference (3%) in resolution across brief intervention status. While the present study's lack of power may be the primary explanation for differences between studies, differences could also be due to differences in rates of documented brief intervention across studies (71% in the pilot versus 28% in the current study) and the fact that, different from the pilot [46], patients with documented brief intervention in this study were more likely to have severe unhealthy alcohol use than those without.

Because the present observational study was underpowered to detect a difference in the rate of resolution across documented brief intervention, further effectiveness research at later implementation stages is clearly needed. However, if replicated, negative findings could reflect the need for improvements to the quality of brief intervention offered to patients and documented in the VA. While electronic clinical reminders have been associated with increased provision of recommended care for multiple conditions [63–71], one study found that 59% of internists reported ‘questionable’ documentation practices, including documenting clinical information in the medical record that was not observed [72]. Moreover, findings from two recent studies evaluating drinking outcomes associated with brief intervention delivered in regular clinical care suggest the possibility that the efficacy of brief intervention demonstrated in randomized trials diminishes [73] or may even have a negative effect when delivered in practice [74,75]. It is possible that the use of top–down quality improvement initiatives to implement brief intervention [35,36] may increase documentation of alcohol-related care, but be insufficient to address barriers to implementation of brief intervention described previously [13–18]. Rigorous mixed-methods implementation evaluations will be needed to assess these issues in the VA and other systems implementing brief intervention using systems-level strategies [35,36,76].

The present study's findings that nearly half of patients resolved unhealthy alcohol use at follow-up (48%), regardless of whether or not brief intervention was documented, may be hypothesis-generating for VA and other systems implementing routine alcohol screening [36,76]. While within the range described in randomized trials of brief intervention [77], the identified prevalence of resolution is high relative to that identified in the pilot (48 versus 32%) [46] and may reflect low-quality repeat screening. It is also possible that repeat screening has an assessment effect, whereby patients are decreasing their drinking as a result of screening [78] or that patients are learning the ‘right’ answers over time [30]. As systems are increasingly implementing routine screening in response to policy initiatives [79–83], evaluations of screening quality and/or assessment effects over time may be warranted.

This study has several limitations. First, this study was observational and may be biased by residual confounding. Secondly, as above, the follow-up rate was low, and analyses of resolution of unhealthy alcohol use were underpowered, which may have limited our ability to identify the true effect of documented brief intervention on resolution. While the follow-up rate was comparable to that observed in previous VA studies with larger windows for follow-up [46,47,84], sensitivity analyses found that eligible patients with follow-up screening had a slightly higher proportion of married individuals, those with a mental health diagnosis and those in the younger and middle-aged groups than those without follow-up screening. These findings suggest that follow-up screening data were not missing completely at random, which may have biased results [85]. Further, generalizability of findings may be limited to patients who are more frequent VA users and those with a higher likelihood of being married and having greater mental health comorbidity than a more general VA outpatient sample. However, because regular use of care is a common tenant of primary care and consistent with the model for outpatient prevention and management of unhealthy alcohol use [7,8], applicability of the results to regular users of care may be reasonable. Thirdly, this study assessed documentation of alcohol-related advice with an electronic clinical reminder, which would have missed brief intervention documented outside a clinical reminder [32]. This measure was selected because it is common for care recommended by performance measures in VA to be documented using clinical reminders [28,30,31] and because conducting manual medical record reviews would have been prohibitively costly. However, research is needed to validate whether the brief intervention health factors from clinical reminders capture most documented brief intervention, and whether results are consistent with patient report, as has been conducted with smoking data documented in VA clinical reminders [45]. Fourthly, although two previous evaluations of documented brief intervention used the same outcome measure as the present study [46,47], and two studies in other settings have used similar outcomes [73,74], no published study has validated changes in alcohol screening scores for use as outcomes. AUDIT-C-based measures were used in this study because the AUDIT-C is administered routinely to a vast majority of Veteran outpatients, which enabled a real-world evaluation of brief intervention implementation in a relatively unbiased sample compared to studies that recruit and consent samples of willing patients. However, in systems where routine repeat clinical alcohol screening data are unavailable, evaluations of brief intervention implementation could also be conducted using alcohol-related clinical outcomes [86–88]. Other systems conducting this type of research may also wish to use a shorter follow-up time-frame, depending on the systems’ recommended interval for screening [84]. This study required at least 270 days between screens because VA requires annual screening and most sites have electronic prompts for re-screening 9 months (270 days) after a prior screen. Finally, these secondary VA data do not capture the intensity of brief interventions and do not enable linking documented brief intervention to specific providers, both of which are likely to influence effectiveness.

Despite these limitations this study is the first, to our knowledge, to describe the reach and effectiveness of brief intervention documented in a real-world clinical setting in which population-based alcohol screening occurs and brief intervention is incentivized via a performance measure and facilitated with electronic clinical decision support. The findings offer an initial view of implementation of brief intervention in the largest integrated care system in the United States and suggest that, during early implementation, documentation of brief intervention did not yet overcome biases in who is offered brief intervention and was not associated with resolution of unhealthy alcohol use. Findings may be useful for informing future research and quality improvement efforts in the VA, as well as other systems implementing brief intervention [37,38,40,89–91]. While future well-powered evaluations of drinking outcomes at later stages of implementation will be required to understand whether improvements to the effectiveness of VA's brief intervention are needed, other healthcare systems implementing brief intervention [35,36] should be aware that during early implementation stages, the reach and effectiveness of brief intervention may be limited.