Assessment of the validity of the AUDIT factor structure in parents involved with child protective services

Abstract

Background:

Identification of hazardous alcohol use is a critical step in connecting individuals to treatment and child protective services (CPS) is a treatment entry-point for parents if hazardous use is identified. The Alcohol Use Disorders Identification Test (AUDIT) is a common screening tool but prior research identifies factor structures ranging from one to three factors in different populations, indicating variation in the perception and/or impact of alcohol use. Determining the factors of the AUDIT for CPS-involved parents is important for its relevance and use in CPS.

Objectives:

This analysis examines the type and number of factors present in a sample of parents involved with CPS.

Methods:

Using confirmatory factor analysis (CFA), this study compares the one-, two-, and three-factor structures of the AUDIT in a large sample of CPS-involved parents (N=4009, 90.8% female, 9.2% male) and a sub-sample who admitted alcohol use (N=1950). This analysis used data from Waves I and II of the National Survey of Child and Adolescent Well-Being II.

Results:

In the main sample, the two-factor (RMSEA=.044, 90% CI: 0.039–0.048; CFI=0.967; TLI=0.956) and three-factor (RMSEA=.045, 90% CI: 0.041–0.050; CFI=0.966; TLI=0.952) fit better than the single factor model (RMSEA=.072, 90% CI: 0.067–0.076; CFI=0.908; TLI=0.881). In the three-factor model two of the factors had a correlation of 0.99; parsimonious models are usually preferable. Sub-sample results were similar.

Conclusions:

The two-factor AUDIT is appropriate for screening CPS-involved parents. Screening with the AUDIT should improve early identification and referral to treatment for CPS-involved parents with hazardous alcohol use.

1. Introduction

Many adults receiving public treatment services for a substance use disorder (SUD) are referred by the child welfare system, and their involvement with child protective services (CPS) impacts their treatment receipt and outcomes (1–3). While statistics generally suggest high levels of need within the child welfare population (4), valid and reliable alcohol screening tools, which are crucial for assessment, are not consistently used. Child welfare caseworker perception is a common method of determining who should be screened for an Alcohol Use Disorder (AUD) in child welfare, but caseworker perceptions are often inaccurate and may be biased (5, 6). One study found that of CPS-involved parents who met the threshold for an AUD on a self-report measure, case managers only identified 17.7% of them as engaging in hazardous alcohol use (7). Examining the reliability and validity of available screening tools within this population is an essential step towards improving screening and service delivery.

The AUDIT is a promising measure for screening child welfare-involved parents for hazardous alcohol use. Developed as part of a six-country World Health Organization collaborative project, the Alcohol Use Disorders Identification Test (AUDIT) is a 10-item measure originally intended to assess for hazardous and harmful alcohol consumption over the past year in primary health care settings (8, 9). The measure contains three questions about alcohol consumption (Items 1-3), three about drinking behavior that indicate dependence (Items 4-6), and four about alcohol-related problems and adverse reactions (Items 7-10).

The factor structure of the AUDIT has been examined in other populations. Most studies find empirical support for and recommend the two-factor structure with a consumption factor (items 1-3) and a consequences factor (items 4-10) (10, 11). A major strength of the AUDIT is the ability to detect the hazardous use of alcohol, i.e., use that does not yet meet diagnostic criteria for AUD but may result in negative outcomes for the individual (8). More stringent assessments of AUD may under-identify hazardous use (8). This is particularly important in working with parents because hazardous alcohol use may result in unsafe parenting practices even in the absence of the level of disorder that would qualify someone for an AUD (12).

Among parents involved with CPS, mothers are overrepresented (13). However, the AUDIT factor structure has been tested in predominantly male samples (10, 11, 14–16). Alcohol misuse, problem identification, and treatment access and entry have gendered dimensions. Women are less likely to have an alcohol use disorder than men and less likely to enter treatment, and women are more susceptible to health and life-functioning problems than men with similar levels of substance use (17). The AUDIT may also perform more poorly for women than for men in identifying hazardous use and some studies have suggested a lower cut-point for women (18). In a review of the literature, sensitivity was consistently lower for women than for men, indicating that many women with hazardous use were not identified (18).

Furthermore, CPS-involved mothers with a SUD are a specific sub-population of women who have unique characteristics and specific needs. Mothers involved with CPS are more likely to be single parents, younger, and more economically disadvantaged than mothers in the general population (13). Research indicates that mothers referred for SUD treatment by CPS are unique from mothers referred by other sources. Among mothers receiving SUD treatment, mothers involved with CPS are younger, have more children, have more financial problems, and are more likely to have experienced physical abuse in the past (19). These characteristics create a unique profile that must be considered when choosing an alcohol use measure. It is possible that these differences may result in different patterns of alcohol use and in a different relationship between use and adverse consequences than has been identified in other populations.

The purpose of this study was to examine the factor structure of the AUDIT in a sample of CPS-involved adults. The inclusion of the AUDIT into the National Survey of Child and Adolescent Well-Being II (NSCAW II) allows us to examine the AUDIT as a stronger assessment tool for AUD within the broader population of families that come into contact with CPS and is not limited to those families where children are taken into foster care. The NSCAW II is a probability sample of children and families with an investigated report of maltreatment. This is important as the majority of children who are reported for maltreatment do not enter the foster care system, and this sample is representative of families who engage with the child welfare system at some point.

Considering the impact an alcohol use disorder can have on parenting and the family (20–23), it is important to determine what the appropriate factor structure, and consequently assessed subscales, is for this population in order to guide research and future intervention studies on treatment for parents with a SUD. Using confirmatory factor analysis (CFA), this study compared the one-, two-, and three-factor structures of the AUDIT with a sample of CPS-involved adult parents.

2. Methods

2.1. Participants and data

This analysis utilized data from Wave 1 of the National Survey of Child and Adolescent Well-Being II (NSCAW II). The NSCAW II is a national probability sample of children, 0 to 17.5 years of age at baseline, and their families who were investigated for child maltreatment between February 2008 and April 2009. The NSCAW II sample was obtained with a two-stage stratified sampling procedure. A total of 81 primary sampling units (PSUs) were randomly selected from eight sampling strata. PSUs typically corresponded with a CPS agency or a group of smaller CPS agencies. All participants in the original NSCAW II study gave informed consent. Further details about the NSCAW II survey methods are available through the National Data Archive on Child Abuse and Neglect (24). The current study was approved in an expedited review by the Institutional Review Board of the first author.

This study included two sub-samples of NSCAW II cases. The first sub-sample is families in which the child remained in the home following the baseline maltreatment report (n=4009, referred to as the ‘in-home sample’). The AUDIT was administered at baseline to parents if the target child remained in the home following the investigation. The child’s primary caregiver serving in the parenting role (biological mother if involved) was interviewed in-person. The second sub-sample was drawn from within the in-home sample and included all families where the child’s primary caregiver responded on the first question of the AUDIT that they had consumed alcohol in the past year (n=1950, referred to as the ‘in-home and consumes alcohol sample’). Primary caretakers were mostly women; the in-home sample was 90.8% women and 9.2% men and the in-home and consumes alcohol sample was 88.9% women and 11.1% men. For reference, the other demographics of these two samples and the demographics of the total NSCAW II sample from which these samples were drawn (n=5872) are reported in Table 1.

Table 1.

Demographics of Total, In-home, and In-home and consume alcohol samples in NSCAW II

	Total NSCAW II Wave I	Families with Child In-home at Wave I	Families with Child In-home and Parent Consumes Alcohol at Wave I
	n (%)a	n (%)a	n (%)a
N	5872	4009	1950
Age in Yearsb	34.3 (32.0)	33.3 (32.0)	32.9 (32.0)
Gender
Male	462 (9.4)	352 (9.2)	205 (11.1)
Female	5316 (90.6)	3657 (90.8)	1745 (88.9)
Race
White/Non-Hispanic	2501 (48.9)	1743 (48.8)	920 (50.2)
Black/Non-Hispanic	1629 (20.3)	1070 (20.0)	509 (20.0)
Hispanic	1300 (24.5)	939 (24.8)	395 (23.6)
Other	336 (6.3)	249 (6.5)	123 (6.3)
Highest Level of Education
Beyond High School Degree	1906 (28.0)	1050 (27.3)	586 (30.7)
High School Degree	2479 (45.0)	1776 (44.7)	900 (46.1)
Less than High School Degree	1385 (27.1)	1177 (28.0)	464 (23.2)
Poverty
Above poverty line	2684 (42.9)	1457 (40.5)	834 (46.9)
At or below poverty line	2639 (57.1)	2235 (59.6)	1005 (53.1)

Note.

^aValues reflect weighted percentages;

^bValues reflect weighted mean (and median);

NSCAW=National Survey of Child and Adolescent Well-Being

2.2. Measures

The AUDIT assesses for the respondent’s self-reported level of alcohol use in the past 12 months. The 10-item AUDIT was administered to parents using Audio Computer-Assisted Self-Interviewing (ACASI) technology. The measure contains three questions about alcohol consumption, three about drinking behavior, and four about alcohol-related problems and adverse reactions. Items one through eight can receive zero to four points, and items nine and ten can receive either zero, two, or four points each for a total possible score of 40 points.

2.3. Statistical Analyses

One-, two-, and three-factor CFA models of the AUDIT were conducted. The one-factor model included all ten items as a single factor. The two-factor model examined the hazardous consumption factor (Items 1-3) and the consequences factor (Items 4-10). The three-factor model included hazardous consumption (Items 1-3), dependence (Items 4-6), and harmful consequences (Items 7-10). Models were run on the in-home sample and on the in-home and consumes alcohol sample.

Data management was conducted in SAS version 9.2. Weighted descriptive analyses were obtained using STATA 10.0. The CFA models were conducted in Mplus version 7.0 and were unweighted. The robust Weighted Least Squares (WLSMV) estimator was used due to the ordinal nature of the response items. All Mplus analyses utilized full information maximum likelihood (FIML). Model fit was assessed using fit indices and factor loadings. The fit indices examined were the Chi-Square Test of Model Fit, Comparative Fit Index (CFI), Tucker-Lewis Index (TLI), and Root Mean Square Error of Approximation (RMSEA) (25). Although reported, the Chi-Square Test of Model Fit is biased with large samples and more likely to be significant even when model fit is strong (26). Respectively, indicators of good model fit on the CFI, TLI, and RMSEA are ≥ .90, ≥ .90, < .05 and non-significant (25, 26). Due to the use of the WLSMV estimator, model comparisons were conducted with the DIFFTEST procedure in the Mplus software.

3. Results

Model results for the in-home sample are presented with standardized estimates in Table 2. The two- and three-factor models provided a better fit to the population than a single factor model. For all models, the chi-square test of model fit was significant. The single factor model had poor fit for RMSEA but good fit for CFI and TLI (RMSEA=.072, 90% CI: 0.067–0.076; CFI=0.908; TLI=0.881). Both the two-factor (RMSEA=.044, 90% CI: 0.039–0.048; CFI=0.967; TLI=0.956) and three-factor models provided a better fit for CPS-involved parents (RMSEA=.045, 90% CI: 0.041–0.050; CFI=0.966; TLI=0.952) than the single factor model. All factor loadings were significant (p < .0001) in each model. In the two-factor model, the correlation between the factors was 0.67 indicating reasonable discrimination between factors. In the three-factor model, the correlation between the first and second factor was 0.65, and first and third factor was 0.67, and the second and third factor correlation was 0.99. In each model, AUDIT item one had the lowest standardized estimates followed by item 9. In general, the factor loadings were high and in the two- and three-factor models, seven of the factor loadings were greater than or equal to 0.84. Item-level R²s were highest in the two- and three-factor models. Item-level results are presented in Table 2. Using a series of DIFFTEST procedures, the fit of the one-factor model was compared to the two-factor model, the fit of the one-factor model was compared to the three-factor model, and the fit of the two-factor model was compared to the three-factor model. The results of the DIFFTEST for the in-home sample indicated that the two-factor model was superior in fit to the one-factor model (χ² (1)= 193.17, p < .001), the three-factor model was superior in fit to the one-factor model (χ² (3) = 297.26, p < .001), and the three-factor model was not superior in fit to the two-factor model (χ² (2)= 1.72, p = .42).

Table 2.

Factor Analysis with In-home Sample in NSCAW II

AUDIT item	One Factora	Two Factorsa		Three Factorsa
1	.30 (.09)	.35 (.12)		.35 (.12)
2	.58 (.34)	.68 (.46)		.68 (.46)
3	.70 (.50)	.88 (.78)		.88 (.78)
4	.84 (.71)		.87 (.75)		.87 (.76)
5	.86 (.73)		.87 (.76)		.88 (.77)
6	.85 (.72)		.86 (.74)		.86 (.75)
7	.87 (.77)		.89 (.80)			.90 (.80)
8	.82 (.68)		.85 (.71)			.85 (.71)
9	.54 (.29)		.56 (.31)			.56 (.31)
10	.83 (.69)		.84 (.71)			.84 (.71)
Estimated correlations among factors
1 and F2/F3		0.67
F1 and F2				0.65
F1 and F3				0.67
F2 and F3				0.99
x2 (df)	747.29 (35), p < .001	290.63 (34), p < .001		292.62 (32), p < .001
RMSEA	.072, p < .001	.044, p = .99		.045, p = .94
CFI	.908	.967		.966
TLI	.881	.956		.952

^aStandardized estimates (item-level R²);

F1=Consumption, F2=Dependence, F3=Problems; NSCAW=National Survey of Child and Adolescent Well-Being; AUDIT=Alcohol Use Disorders Identification Test; F=Factor; RMSEA=Root Mean Square Error of Approximation; df=degrees of freedom; CFI=comparative fit index; TLI=Tucker Lewis Index

Among the in-home and consumes alcohol sample, item-level standardized estimates and R² were higher for AUDIT item one but remained similar for the remaining items. These results indicate that the poorer fit of item one in the in-home sample may be due to the inclusion of individuals who did not consume alcohol in the past year. However, by including the larger NSCAW II sample, the models confirm the fit of the AUDIT for the wider child welfare population. Results are presented in Table 3. The results of the DIFFTEST for the in-home and consumes alcohol sample indicated that the two-factor model was superior in fit to the one-factor model (χ² (1)= 200.75, p < .001), the three-factor model was superior in fit to the one-factor model (χ² (3) = 304.61, p < .001), and the three-factor model was not superior in fit to the two-factor model (χ² (2)= 1.48, p = .48).

Table 3.

Factor Analysis with In-home and Consumes Alcohol Sample in NSCAW II

AUDIT item	One Factora	Two Factorsa		Three Factorsa
1	.54 (.29)	.62 (.38)		.62 (.38)
2	.59 (.35)	.69 (.48)		.69 (.48)
3	.70 (.49)	.88 (.77)		.88 (.77)
4	.85 (.71)		.87 (.76)		.87 (.76)
5	.86 (.74)		.87 (.76)		.88 (.77)
6	.85 (.72)		.86 (.74)		.86 (.74)
7	.88 (.77)		.89 (.80)			.90 (.80)
8	.83 (.68)		.85 (.71)			.85 (.72)
9	.53 (.28)		.56 (.31)			.56 (.31)
10	.82 (.67)		.84 (.70)			.84 (.71)
Estimated correlations among factors
1 and F2/F3		0.66
F1 and F2				0.65
F1 and F3				0.67
F2 and F3				0.99
x2 (df)	634.60 (35), p < .001	179.07 (34), p < .001		180.16 (32), p < .001
RMSEA	.094, p < .001	.047, p = .77		.049, p = .60
CFI	.921	.981		.980
TLI	.898	.975		.972

^aStandardized estimates (item-level R²);

F1=Consumption, F2=Dependence, F3=Problems; NSCAW=National Survey of Child and Adolescent Well-Being; AUDIT=Alcohol Use Disorders Identification Test; F=Factor; RMSEA=Root Mean Square Error of Approximation; df=degrees of freedom; CFI=comparative fit index; TLI=Tucker Lewis Index

4. Discussion

This first CFA of the AUDIT in a representative sample of CPS-involved parents indicates the measure is stable within this population and confirms that latent constructs are present within the AUDIT. Although empirical support is available for both the two- and three-factor models, the results of the model comparisons indicated that the two-factor model was more parsimonious than the three-factor model. Thus, the parsimony principle supports the two-factor structure for the child welfare population. However, as explained in Confirmatory Factor Analysis for Applied Research, factors often determine the subscales of the measure that are used in practice (27). Child welfare case managers are burdened by highly demanding jobs, limited resources, time constraints, and frequently, a lack of training on SUDs (28, 29). The three-factor model separates out the concepts of alcohol dependence from alcohol-related problems which may be particularly relevant to this population. In the child welfare system, substance use alone is not considered child maltreatment and, thus, often not enough to initiate services for the family. Parental substance use must result in harm or the threat of harm to the child in order for it to be considered child maltreatment (30). The impact of substance use is best captured in the AUDIT in the alcohol-related problems factor. When alcohol dependence and alcohol-related problems are combined into a single factor or subscale, important aspects of alcohol-related problems may be missed; some parents with lower levels of dependence may still present with higher levels of alcohol-related problems that impact their role as parents. Until future research examines the impact of the three separate subscales in this population, the two-factor model is recommended due to parsimony.

The AUDIT contains items related to alcohol-related problems that may be highly relevant for parents reported to CPS. In particular, item nine states, “Have you or someone else been injured as a result of your drinking?”. In other factor analyses of the AUDIT, item nine tends to load weakly onto the factors, with studies finding loadings as low as 0.24 (14). While item nine had one of the lower factor loadings on all six models in this study, the standardized loading was still 0.56, suggesting that item nine is more relevant to the latent constructs in this population than in others. One reason that item nine may be more important in this population is because CPS-involved parents may have a smaller or weaker support network (31), leaving them and their children more vulnerable to the effects of drinking, such as drinking-related injuries.

This population reported lower rates alcohol use than the general population, with only half of parents reporting any alcohol use compared to 67% of women and 73% of men in the general population (32). However, alcohol use varies widely across demographic groups. Black and Latina women in particular have lower rates of use with only 55% of Black women and 54% of Latina women reporting any alcohol consumption in the last year (32). As the majority of this sample were women, and approximately half were Black, Latinx or other races/ethnicities, the actual prevalence of alcohol use is likely lower than the general population.

The model fit both samples well (in-home sample and in-home and consumed alcohol sample) suggesting that the AUDIT is a robust measure that is appropriate for the population. For some parents, experiencing a CPS investigation may result in a temporarily heightened awareness of the impact of their alcohol consumption on their children or partners and that may cause them to under-report alcohol use due to social desirability. However, research supports the validity of self-report measures of substance use in child welfare (33) and other populations (34–36).

This analysis utilizes a national probability sample of parents involved in the United States child welfare system. Approximately 90% of the parents in the sample were female which is indicative of the overrepresentation of female-headed households within the child welfare system. Thus, interpretation of the results to fathers-only samples within the child welfare population should be cautioned. Additionally, while this sample had a larger percentage of Black and Latinx persons than many other studies of the AUDIT (e.g. in Kelly and Donovan (37) the sample was 80% white), additional research is needed to better understand the factor structure and validity of the AUDIT in diverse populations.

Given the important impact that maternal substance use has on children (20, 21) and the large number of mothers involved with child welfare, the results have strong implications for SUD assessments within the CPS system. The fit and brevity of the AUDIT suggests that incorporating it as an assessment for AUD in child protective services may be feasible and useful for identifying hazardous use and connecting parents to effective services. However, the AUDIT alone does not provide sufficient information to understand the elements impacting an individual’s use of substances and additional measures may be supportive. For example, a measure obtaining information about social support, specifically information about the parents’ social network support for abstinence, could provide more information to guide referrals for assessment (38, 39). In addition, social support is an important aspect of parenting that is often limited in families involved with child welfare (40). Obtaining more information about social networks for non-drinking can inform the development of plans to support the family and children during treatment .

Implementation of the AUDIT in the field of child welfare should be accompanied by additional research. For example, there are different points of contact between CPS and families, including the initial investigation or family assessment process, voluntary services for families who are determined to be low-risk for harm and on-going case management once a formal case has been opened. The best juncture for assessment and engagement of parents, where they are most likely to disclose and be connected to services, needs additional study. Additionally, the conditions of the study under which the data were collected, with parent responses collected without their results disclosed to their CPS agency, are different than regular CPS practice. It is a limitation of the study that parents’ disclosure in usual CPS practice is unknown. However, while CPS-involved parents may not always fully disclose their alcohol use, consistently using a measure that assesses for different aspects of AUD is likely to accurately identify more parents with hazardous use than the current practice of case workers relying on their own judgment without systematic assessment (7).

5. Conclusions

These results indicate the AUDIT is a valid measure, in research and practice, for assessing alcohol use in the CPS population that captures key components of alcohol use including consumption, dependence, and problems due to alcohol use. At only ten items, the AUDIT is a brief self-report screener that could be universally conducted with a child welfare population to facilitate appropriate referrals for further assessment. Using the AUDIT can provide case managers and other service providers with a more accurate and nuanced picture of a parent’s alcohol use, therefore allowing for more appropriate and targeted treatment options.