High test-retest reliability of the Alcohol Use Disorder Identification Test-Consumption version (AUDIT-C) completed by primary care patients in routine care

Abstract

Background:

The Alcohol Use Disorder Identification Test-Consumption (AUDIT-C) is a three-item screening measure of unhealthy alcohol use that is widely used in healthcare settings. Evidence suggests high test-retest reliability of the AUDIT-C in research samples, but most studies had limited external validity and used small samples that could not evaluate reliability across demographic subgroups and/or screening modalities. This study evaluates test-retest reliability of the AUDIT-C completed in routine care in a large primary care sample, including across demographic subgroups defined by age, sex, race, ethnicity, and screening modality (i.e., completed in-clinic or online).

Methods:

This study used electronic health record (EHR) data from Kaiser Permanente Washington. The sample included 18,491 adult primary care patients who completed two AUDIT-C screens 1–21 days apart as part of routine care during 2021. Test-retest reliability was evaluated for AUDIT-C total scores (0–12) and for a binary measure indicating unhealthy alcohol use (scores ≥3 women, ≥4 men). Using previously established cutoffs, we interpreted reliability coefficients >0.75 as indicating “excellent” reliability.

Results:

AUDIT-C screens completed in routine care and documented in EHRs had excellent test-retest reliability for total scores (ICC=0.87, 95% CI: 0.87–0.87) and the binary indicator of unhealthy alcohol use (kappa=0.79, 95% CI: 0.78–0.80). Reliability coefficients were good-to-excellent across all demographic groups and for in-clinic and online modalities. Higher reliability was seen when both screens were completed through online patient portals (ICC=0.93, 95% CI: 0.93–0.93) vs in-clinic (ICC=0.81, 95% CI: 0.79–0.82) or when one screen was completed using each modality (ICC=0.83, 95% CI: 0.82–0.83). Lower reliability was seen in those who were American Indian/Alaska Native (ICC=0.82, 95% CI: 0.75–0.87) or multiracial (ICC=0.82, 95% 0.80–0.84).

Conclusions:

In real-world routine care conditions, AUDIT-C screens have excellent test-retest reliability across demographic subgroups and modalities (online and in-clinic). Future research should examine why reliability varies slightly across modalities and demographic groups.

Introduction

Unhealthy alcohol use is a leading cause of preventable death; more than 140,000 people die of alcohol-related causes in the United States annually (CDC, 2022). Unhealthy alcohol use, defined as use that increases the risk of health consequences or that may have already led to health consequences, may be identified via routine screening with standardized validated measures. The United States Preventive Services Task Force recommends screening all adults for unhealthy alcohol use in primary care (O’Connor et al., 2018). The three-item Alcohol Use Disorders Identification Test-Consumption (AUDIT-C) is valid and increasingly used to screen for unhealthy alcohol use (Bradley et al., 2003, Bush et al., 1998). Studies show patient self-administered screening with the AUDIT-C on a confidential survey is more likely to detect unhealthy alcohol use than when screening is documented in the EHR as part of clinical care in veterans especially in some minoritized groups (Bradley et al., 2011).

Despite growing evidence of the validity of the AUDIT-C as a screening tool for unhealthy alcohol use, its test-retest reliability has not been evaluated in routine-care conditions with adult primary care patients. While European samples suggest good-to-excellent test-retest reliability of the longer 10-item AUDIT in adolescents (Källmén et al., 2019) and adults (Selin, 2003, Bergman and Kallmen, 2002, Dybek et al., 2006), few studies have evaluated the test-retest reliability of the briefer 3-item AUDIT-C screen, which may be more feasible for general medical settings. Extant test-retest reliability studies suggest that the AUDIT-C has high test-retest reliability when administered as part of a research study, including for Korean first responders (Jeong et al., 2017), Japanese healthcare workers (Osaki et al., 2014), male military trainees (Robinson et al., 2010), college students (Erford et al., 2021, Miller et al., 2002), and older male veteran patients (Bradley et al., 1998). However, the external validity of these findings is limited given that these test-retest findings were not conducted under real world conditions where patients complete alcohol screening as part of population-based clinical care. No studies have evaluated the test-retest reliability of the AUDIT-C when it is self-administered as part of routine clinical care with results entered in EHRs and used by clinicians to identify unhealthy alcohol use, related health risks, and/or monitor changes in drinking over time. Additionally, due to small sample sizes of previous studies, test-retest reliability has not been thoroughly examined across demographic subgroups or across two screening modalities that are commonly used in routine care (paper-based screening forms completed in clinic vs. online patient portals). The aims of this study were to (1) evaluate test-retest reliability of the AUDIT-C completed in real-world routine care conditions in a large primary care patient sample, and (2) evaluate test-retest reliability within subgroups defined by patient demographics and the modality through which they completed the AUDIT-C.

Materials and Methods

Setting and sample.

This study used electronic health record (EHR) data from Kaiser Permanente Washington (KPWA), an integrated health system that provides health insurance and medical care to about 700,000 patients annually. During the year of this study, approximately 371,729 patients had a primary care visit at one of KPWA’s 31 primary care clinics. Starting in 2015, KPWA implemented annual alcohol screening for all adult primary care patients using the AUDIT-C as part of a larger effort to integrate behavioral health into primary care (Glass et al., 2018). At that time, patients were asked to self-report their alcohol use in the past-year on a paper-based AUDIT-C screening measure completed in primary care clinics before clinical encounters if screening had not been completed in the past year. As part of the process of implementing alcohol screening, primary care clinic staff developed standard processes and received training to help ensure that screenings were administered in a standardized, non-stigmatized and comfortable manner. In mental health clinics, the AUDIT-C was included on a behavioral health monitoring tool that was typically completed prior to every mental health visit. Other specialty settings also had the option to screen for alcohol using the AUDIT-C; however, procedures for screening in other settings typically were not standardized. Starting in 2020, screening became widely available through the online EHR portal where patients had the option to complete screenings before a scheduled visit in lieu of completing them in clinic on paper forms.

Patients were included in this study if they (1) completed at least one primary care encounter at KPWA during the study period (01/2021–12/2021), (2) were age 18 or older at the time of the encounter, and (3) fully completed two AUDIT-C screens 1–21 days apart during the study period, with each AUDIT-C screen being linkable to a clinical encounter. The 1–21 day test-retest window was chosen to be consistent with other test-retest reliability studies of alcohol and drug-related measures (Hallgren et al., 2022)^,(Hasin et al., 2020).

Measures.

The AUDIT-C (Bradley et al., 2003, Bush et al., 1998) is a three-item screening measure that asks about typical drinking frequency, drinking quantity, and frequency of heavy drinking in the past year (see Supplement). Each item is measured on a five-point Likert scale ranging from 0 to 4. Responses are summed, yielding total scores ranging from 0 to 12, with scores ≥3 (women) or ≥4 (men) often used to indicate unhealthy alcohol use (Bradley et al., 2007). Patient age, sex, race, ethnicity, and the modality in which the AUDIT-Cs were completed were obtained using EHR data.

Analytic approach.

Test-retest reliability of the AUDIT-C was evaluated for total scores (0–12) and for a binary measure indicating unhealthy alcohol use. These two measures are the primary focus of the current analysis because they reflect summary measures that have clinical utility, with total scores offering a scaled index that is correlated with total alcohol consumption and risk for alcohol use disorder (Dawson, 2000) and the binary measure of unhealthy alcohol use often being used to identify patients who could potentially benefit from brief interventions. Test-retest reliability was also measured for each of the three individual items comprising the AUDIT-C. Reliability coefficients were estimated using one-way, single-measures, agreement intraclass correlation coefficients (McGraw and Wong, 1996) for total scores and individual item responses, and using Cohen’s kappa (Cohen, 1960) for the binary measure of unhealthy alcohol use. Interpretation of reliability coefficients was based on previous guidelines (Cicchetti, 1994) where coefficients 0.75 to 1.00 were deemed to indicate “excellent” reliability, 0.60 to 0.74 “good” reliability, 0.40 to 0.59 “fair” reliability, and <0.40 “poor” reliability. The study procedures were approved by the KP Washington Health Research Institute’s Institutional Review Board, with waivers of consent and HIPAA authorization to use existing EHR data.

Results

There were 240,017 patients who completed one or more AUDIT-C screens that were linked with clinical encounters during the study period. Of them, 18,491 met inclusion criteria for the current study and were retained in the analytic sample. Patients in the analytic sample were 69.0% women, 72.4% White (including Hispanic and non-Hispanic ethnicity), and 93.2% non-Hispanic (any race). Just under half (48.0%) completed both AUDIT-C screens via the online patient portal, 18.1% completed both in-clinic, and 33.8% completed one AUDIT-C screen using each of the two modalities (Table 1). Most patients (55.8%) completed both AUDIT-C screens in primary care, 27.5% completed both in mental health clinics, 11.1% completed one in primary care and one in a mental health clinic, and 5.6% completed one or both AUDIT-C screens in a different setting (e.g., obstetrics and gynecology, urgent care, etc.). Compared to all patients who completed an AUDIT-C during the study period, those retained in the analytic sample were more likely to be younger, female, and White, and were more likely to complete the AUDIT-C in a mental health clinic and via the online portal.

Table 1.

Descriptive Statistics for Test-Retest Reliability Sample (N = 18491)

		n	(%)
Age (y)	18–24	2122	(11.5%)
	25–44	8056	(43.6%)
	45–64	4770	(25.8%)
	65+	3543	(19.2%)
Sex	Female	12767	(69.0%)
	Male	5724	(31.0%)
Race	Asian or Asian American	1317	(7.1%)
	Black or African American	719	(3.9%)
	American Indian or Alaskan Native	121	(0.7%)
	Native Hawaiian or Pacific Islander	151	(0.8%)
	White	13379	(72.4%)
	More than one race	768	(4.2%)
	Another race not listed	595	(3.2%)
	Unknown race	1441	(7.8%)
Ethnicity	Hispanic	1257	(6.8%)
	Not Hispanic	13752	(74.4%)
	Unknown ethnicity	3482	(18.8%)
Setting for AUDIT-C's	Both completed in primary care clinics	10319	(55.8%)
	Both completed in mental health clinics	5086	(27.5%)
	One in primary care, one in mental health	2055	(11.1%)
	One or both completed in other settings	1031	(5.6%)
Modality for AUDIT-C's	Both completed via online portal	8884	(48.0%)
	Both completed via in-clinic entry	3350	(18.1%)
	One completed via both modalities	6257	(33.8%)
AUDIT-C score (T1), M (SD)		1.80	(2.00)
Unhealthy alcohol use (T1, AUDIT-C score ≥3/4 for women/men)		4716	(25.5%)
AUDIT-C item 1 (T1), M (SD)		1.34	(1.29)
AUDIT-C item 2 (T1), M (SD)		0.18	(0.52)
AUDIT-C item 3 (T1), M (SD)		0.27	(0.66)
AUD diagnosed by healthcare provider in past two years		775	(4.2%)

In the full sample, AUDIT-C screens completed in routine care and documented in EHRs had excellent test-retest reliability for total scores (ICC=0.87, 95% CI: 0.87–0.87) and the binary indicator of unhealthy alcohol use (kappa=0.79, 95% CI: 0.78–0.80) (Table 2). Test-retest reliabilities were good to excellent for each of the demographic subgroups and screening modalities (in-clinic vs online patient portal). Slightly lower test-retest reliability was observed in patients who were American Indian/Alaska Native (ICC=0.82, 95% CI: 0.75–0.87) or reported more than once race (ICC=0.82, 95% 0.80–0.84). Higher test-retest reliability was observed when both screens were completed through online patient portals (ICC=0.93, 95% CI: 0.93–0.93) compared to in-clinic (ICC=0.81, 95% CI: 0.79–0.82) or when one AUDIT-C was completed using each modality (ICC=0.83, 95% CI: 0.82–0.83). Higher test-retest reliability was also observed when both screens were completed in mental health clinics (ICC=0.89, 95% CI: 0.88–0.89), when both were completed primary care clinics (ICC=0.87, 95% CI: 0.86–0.87), or when one screen was completed in mental health and one was completed in primary care (ICC=0.86, 95% CI: 0.85–0.87), compared to when one or both screens were completed in other non-primary care and non-mental health settings (0.76, 95% CI: 0.74–0.79) (Table 2).

Table 2.

Test-Retest Reliability Coefficients (and 95% CI's) for AUDIT-C's Completed in Routine Care, for Full Sample and by Subgroup

		AUDIT-C score (0–12)	Unhealthy alcohol use (score ≥3/4 for women/men)	Item 1 (drinking frequency)	Item 2 (drinking intensity)	Item 3 (heavy drinking freq.)
Full sample		0.87 (0.87, 0.87)	0.79 (0.78, 0.80)	0.88 (0.88, 0.88)	0.74 (0.73, 0.75)	0.77 (0.77, 0.78)
Age group	18–24	0.84 (0.83, 0.86)	0.74 (0.71, 0.77)	0.84 (0.82, 0.85)	0.70 (0.68, 0.72)	0.73 (0.71, 0.75)
	25–44	0.87 (0.86, 0.87)	0.78 (0.76, 0.79)	0.86 (0.85, 0.86)	0.75 (0.74, 0.76)	0.78 (0.77, 0.79)
	45–64	0.87 (0.86, 0.88)	0.81 (0.79, 0.83)	0.89 (0.88, 0.90)	0.73 (0.71, 0.74)	0.77 (0.75, 0.78)
	65+	0.89 (0.89, 0.90)	0.85 (0.83, 0.87)	0.92 (0.91, 0.92)	0.69 (0.67, 0.70)	0.68 (0.66, 0.70)
Sex	Female	0.86 (0.86, 0.87)	0.79 (0.78, 0.81)	0.87 (0.87, 0.88)	0.73 (0.72, 0.74)	0.76 (0.75, 0.76)
	Male	0.87 (0.87, 0.88)	0.79 (0.78, 0.81)	0.88 (0.88, 0.89)	0.74 (0.73, 0.75)	0.79 (0.78, 0.80)
Race	Asian or Asian American	0.86 (0.84, 0.87)	0.77 (0.72, 0.82)	0.86 (0.84, 0.87)	0.72 (0.69, 0.75)	0.74 (0.71, 0.76)
	Black or African American	0.87 (0.85, 0.89)	0.74 (0.68, 0.80)	0.83 (0.81, 0.85)	0.76 (0.73, 0.79)	0.76 (0.73, 0.79)
	American Indian or Alaskan Native	0.82 (0.75, 0.87)	0.69 (0.54, 0.84)	0.84 (0.78, 0.89)	0.75 (0.65, 0.82)	0.73 (0.63, 0.80)
	Native Hawaiian or Pacific Islander	0.91 (0.88, 0.94)	0.86 (0.76, 0.95)	0.89 (0.85, 0.92)	0.84 (0.79, 0.88)	0.77 (0.69, 0.83)
	White	0.88 (0.87, 0.88)	0.80 (0.79, 0.82)	0.89 (0.88, 0.89)	0.74 (0.74, 0.75)	0.79 (0.78, 0.79)
	More than one race	0.82 (0.80, 0.84)	0.76 (0.71, 0.81)	0.83 (0.80, 0.85)	0.70 (0.66, 0.73)	0.71 (0.67, 0.74)
	Other race	0.84 (0.82, 0.87)	0.69 (0.62, 0.76)	0.84 (0.81, 0.86)	0.73 (0.68, 0.76)	0.70 (0.65, 0.74)
	Unknown race	0.85 (0.83, 0.86)	0.79 (0.75, 0.82)	0.85 (0.84, 0.87)	0.71 (0.68, 0.73)	0.75 (0.73, 0.77)
Ethnicity	Hispanic	0.86 (0.84, 0.87)	0.72 (0.67, 0.77)	0.84 (0.83, 0.86)	0.72 (0.69, 0.75)	0.75 (0.72, 0.77)
	Not Hispanic	0.87 (0.87, 0.87)	0.80 (0.79, 0.81)	0.88 (0.88, 0.89)	0.74 (0.73, 0.75)	0.77 (0.76, 0.78)
	Unknown ethnicity	0.87 (0.86, 0.88)	0.79 (0.77, 0.82)	0.86 (0.86, 0.87)	0.74 (0.73, 0.76)	0.79 (0.77, 0.80)
Setting for AUDIT-C's	Both completed in primary care clinics	0.87 (0.86, 0.87)	0.80 (0.78, 0.81)	0.88 (0.88, 0.89)	0.72 (0.71, 0.73)	0.76 (0.75, 0.77)
	Both completed in mental health clinics	0.89 (0.88, 0.89)	0.81 (0.79, 0.83)	0.89 (0.89, 0.90)	0.78 (0.77, 0.79)	0.80 (0.79, 0.81)
	One in primary care, one in mental health	0.86 (0.85, 0.87)	0.78 (0.74, 0.81)	0.86 (0.85, 0.87)	0.73 (0.71, 0.75)	0.79 (0.77, 0.81)
	One or both completed in other settings	0.76 (0.74, 0.79)	0.59 (0.51, 0.66)	0.67 (0.63, 0.70)	0.68 (0.64, 0.71)	0.67 (0.64, 0.70)
Modality for AUDIT-C's	Both completed via online portal	0.93 (0.93, 0.93)	0.85 (0.83, 0.86)	0.93 (0.93, 0.93)	0.85 (0.84, 0.85)	0.85 (0.84, 0.85)
	Both completed via in-clinic entry	0.81 (0.79, 0.82)	0.72 (0.69, 0.75)	0.80 (0.79, 0.81)	0.66 (0.65, 0.68)	0.70 (0.68, 0.72)
	One completed via both modalities	0.83 (0.82, 0.83)	0.75 (0.73, 0.77)	0.84 (0.83, 0.85)	0.65 (0.64, 0.67)	0.72 (0.71, 0.73)

Note. Test-retest reliability coefficients were estimated using a one-way single-measures agreement intraclass correlation coefficient (ICC) for scaled measures (summed scores, item-level responses) and using Cohen's kappa binary indicators (unhealthy alcohol use).

Because of the large difference in test-retest reliability associated with screening modality (online patient portal vs. in-clinic), we hypothesized (post hoc) that this difference could be attributable to the online screening modality increasing the extent to which patients were comfortable with self-reporting higher levels of alcohol use on healthcare screens. Post hoc analyses were consistent with this hypothesis: controlling for age, sex, race, ethnicity, and clinical setting, initial AUDIT-C screens completed via the online patient portal had higher total scores (M=1.88, SD=1.94) compared to screens completed in-clinic (M=1.61, SD=2.10, p<0.001) and were more often indicative of unhealthy alcohol use (26.8% vs. 22.7%, p<0.001). The same trend was seen for the second AUDIT-C screen: those completed via the online patient portal had higher total scores (M=1.84, SD=2.01) compared to screens completed in-clinic (M=1.59, SD=1.92 p<0.001) and were more often indicative of unhealthy alcohol use (26.4% vs. 22.6% p<0.001). Test-retest reliability was consistently higher for the online portal when both screens were completed in primary care settings, both were completed in mental health settings, or one screen was completed in primary care and the other in mental health (Supplemental Table 1).

Discussion

In routine care conditions, AUDIT-C screens had excellent test-retest reliability for total scores and the binary indicator of unhealthy alcohol use. Small increases in test-retest reliability were observed with increasing age. Slightly lower test-retest reliability was observed in patients who were American Indian/Alaska Native or who were multiracial. Reasons for these differences are not known and could include increased stigma or negative consequences for reporting drinking for members of these subgroups, small sample sizes (0.7% and 4.2% of sample respectively, for American Indian/Alaska Native and multiracial patients, respectively), differences in drinking patterns, or other factors. Additional research is needed to understand what factors contribute to differences in screening reliability for these racial minority groups.

Although test-retest reliability was excellent across screening modalities, higher reliability was observed in patients who completed both screens via the online patient portal. A post hoc hypothesis for this finding could be that completing screens outside of the clinic leads to reduced hesitancy about reporting drinking to healthcare providers, which was consistent with the finding that drinking levels were higher on screens completed via the online patient portal. However, it is important to qualify this with the possibility that patients who completed screens outside of clinic had higher levels of drinking (not just reduced hesitancy reporting drinking on healthcare screens). Future research should examine AUDIT-C screening modalities to better understand reasons for this variability in test-retest reliability (e.g., whether online screening leads to a greater sense of comfort or confidentiality) and explore strategies to reduce concerns about reporting drinking to healthcare providers in settings where online screening is not possible (e.g., emergency departments).

Higher test-retest reliability was also observed for patients who completed both screens in primary care and/or mental health clinics compared to patients who completed the screens in other settings (i.e., non-primary care and non-mental health settings). This suggests that the setting in which AUDIT-C screen is completed may impact how patients report drinking behaviors. For example, the primary care clinics in this study had automated electronic prompts, standardized workflows, and staff training to ensure consistency in population-based alcohol screening (Williams et al., 2015). In mental health clinics, the AUDIT-C was completed as part of a mental health monitoring tool that was typically completed prior to every mental health encounter. It is possible that the standardization of these screening processes contributed to higher test-retest reliability in these settings, and that screening procedures varied more considerably in other settings leading to lower test-retest reliability in those settings (Hallgren et al., 2022). It is also possible that patients felt differing levels of comfort reporting their drinking in non-primary care and non-mental health settings, which could potentially account for the lower test-retest reliability when one or both screens were completed in other settings.

Limitations of this study include a lack of information on why patients in the study sample completed two AUDIT-C screens in the 1–21 day window. This may have led to sampling bias, such that the study sample did not represent the larger primary care population. It is also unknown if patients included in the sample changed their drinking during the 1–21 day window or were trying to change their drinking, in which case actual changes in drinking would have been modeled as measurement error, which would yield lower estimates of test-retest reliability. Patients who completed multiple AUDIT-Cs in the 1–21 day window may have had ongoing clinical monitoring for alcohol use or other concerns included on behavioral health screening forms (e.g., cannabis use, other drug use, depression symptoms). We were not able to obtain information about whether patients were undergoing clinical monitoring or the reasons for healthcare encounters.

The study also has noteworthy strengths, including the large sample size, which yielded statistical precision and the ability to examine test-retest reliability within specific racial/ethnic minority subgroups and within specific settings. Test-retest reliability was examined for AUDIT-C total scores and a binary indicator of unhealthy alcohol use which have meaningful clinical interpretations. External validity was high from utilizing AUDIT-C measurements completed in real world conditions (vs. in research contexts where screens are not completed in routine care or documented in EHRs).

The current study’s findings indicate that AUDIT-C total scores and the binary indicator of unhealthy alcohol use are reliable measures across demographic subgroups and screening modalities in real-world routine care conditions. These findings support the reliability of the AUDIT-C as a reliable scaled measure of drinking risk and for identifying unhealthy alcohol consumption in routine care settings.