Psychometric Properties of Brief Screening Tests for Alcohol Use Disorders during Pregnancy in Argentina

Abstract

Background Considering the physical, mental and behavioral problems related to fetal alcohol exposure, prenatal clinical guides suggest a brief evaluation of alcohol consumption during pregnancy to detect alcohol intake and to adjust interventions, if required. Even if any alcohol use should be considered risky during pregnancy, identifying women with alcohol use disorders is important because they could need a more specific intervention than simple advice to abstain. Most screening tests have been developed and validated in male populations and focused on the long-term consequences of heavy alcohol use, so they might be inappropriate to assess consumption in pregnant women.

Objective To analyze the internal reliability and validity of the alcohol screening instruments Alcohol Use Disorders Identification Test (AUDIT), Alcohol Use Disorders Identification Test – Consumption (AUDIT-C), Tolerance, Worried, Eye-Opener, Amnesia and Cut-Down (TWEAK), Rapid Alcohol Problems Screen – Quantity Frequency (RAPS-QF) and Tolerance, Annoyed, Cut-Down and Eye-Opener (T-ACE) to identify alcohol use disorders in pregnant women.

Methods A total of 641 puerperal women were personally interviewed during the 48 hours after delivery. The receiver operating characteristics (ROC) curves and the sensitivity and specificity of each instrument using different cut-off points were analyzed.

Results All instruments showed areas under the ROC curves above 0.80. Larger areas were found for the TWEAK and the AUDIT. The TWEAK, the T-ACE and the AUDIT-C showed higher sensitivity, while the AUDIT and the RAPS-QF showed higher specificity. Reliability (internal consistency) was low for all instruments, improving when optimal cut-off points were used, especially for the AUDIT, the AUDIT-C and the RAPS-QF.

Conclusions In other cultural contexts, studies have concluded that T-ACE and TWEAK are the best instruments to assess pregnant women. In contrast, our results evidenced the low reliability of those instruments and a better performance of the AUDIT in this population.

Introduction

From the first descriptions of serious developmental problems in children of alcoholic mothers in the 1970s,1 2 the literature in the field grew quickly, evidencing a wide spectrum of damages associated with heavy prenatal alcohol exposure.3 More recent studies have pointed out that moderate consumption is not harmless either.4 5 6 Prenatal alcohol exposure can lead to a wide spectrum of physical, mental, and conduct problems, encompassed in the term Fetal Alcohol Spectrum Disorders (FASD). Fetal Alcohol Syndrome (FAS), characterized by pre- and postnatal growth retardation, neurodevelopment problems evident in intellectual disabilities, and a set of facial abnormalities, is the most severe consequence of prenatal alcohol exposure.7

The effects of alcohol exposure on the fetus depend on several factors, from the nutritional condition of the mother8 9 to the genetic characteristics of the mother and the child.10 11 Thus, to this day, a safe dose of alcohol consumption during pregnancy has not been established. Therefore, official recommendations about drinking during pregnancy are changing globally and progressively from moderate consumption to abstinence. Nevertheless, and despite public efforts to reduce the prevalence of alcohol consumption in women of childbearing age and during pregnancy, many women continue to drink during pregnancy, and some of them are heavy drinkers. Alcohol consumption during pregnancy in the United States ranges from 10 to 12%.12 In some Latin-American countries like Uruguay and Argentina, the prevalence is much higher, ranging from 4013 to 75%.14

To articulate massive information campaigns on the topic with individual counseling and intervention, an accurate screening tool is fundamental. Indeed, prenatal alcohol consumption screening is recommended in prenatal care clinical guides.15 16 Some studies suggest that screening itself could reduce alcohol consumption,17 but knowing tightly the level of consumption and related problems of each woman would enable the administration of adequate interventions. Although during pregnancy any level of consumption is risky, identifying alcohol use disorders (AUDs) among pregnant women is important because women with AUDs could need targeted interventions, such as the referral to a specialized treatment, or close monitoring.16

Screening tools have some advantages over directly inquiring the quantity and the frequency of alcohol consumption. By focusing on the indirect effects of drinking, the defensiveness and the consumption underreport, which are common among pregnant women, could be reduced.18 However, most of the available alcohol screening tools have been developed and tested in male populations, and focused on the social consequences of heavy and long-term alcohol use.18 Considering the biological19 20 and cultural21 differences related to alcohol consumption between men and women, a screening tool developed for male populations could not be accurate for female populations. Hence, the cut-off points used to distinguish a positive from a negative result in a screening tool made to be used in male populations could not be appropriate to identify alcohol problems in female populations.

The need for an appropriate assessment tool to use in prenatal care led to the development of some specific instruments for this population. The first of these instruments was the Tolerance, Annoyed, Cut-Down and Eye-Opener (T-ACE), designed to detect risky drinking during pregnancy, which is defined as the consumption of 1 ounce (∼ 30 mL) or more of alcohol per day.22 The second one was the Tolerance, Worried, Eye-Opener, Amnesia and Cut-Down (TWEAK), designed to detect alcoholism or heavy drinking,23 which proved useful in identifying risky drinking among pregnant women.24 Both instruments have been found to be particularly sensitive and specific for pregnant women when compared with other screenings tools.18 25 Nevertheless, psychometric studies are scarce, and they are mostly concurrent validity studies, and studies performed in the United States.26 Besides, to our knowledge, the use of other screening tools like the Alcohol Use Disorders Identification Test (AUDIT),27 the Alcohol Use Disorders Identification Test – Consumption (AUDIT-C)28 and the Rapid Alcohol Problems Screen – Quantity Frequency (RAPS-QF)29 for identifying alcohol use or AUDs during pregnancy has not been established; furthermore, we have found no evidences of the reliability of any of these screening tools for pregnant women.

Thus, the present article aims at exploring the validity, internal reliability and item-level statistics of the following brief alcohol screening tests: AUDIT,27 AUDIT-C,28 TWEAK,23 RAPS-QF29 and T-ACE,22 to identify AUDs among pregnant women in Argentina.

Methods

Participants

A probabilistic sample of puerperal women (N = 641) was personally interviewed in a private maternity unit and in the largest public (state-owned) maternity unit in the city of Santa Fe, Argentina. The participants were between the ages of 13 and 44 years (mean [M] = 25.6); most were living with theirs partners, either cohabiting (59%) or married (24.6%), and almost 1/3 were primiparas (35.7%), with the mean of previous children of 2.4 (standard deviation [SD] = 1.5).

Over half of the participants (53.7%) reported being housewives, and among those who were employed (33.6%), most had a part-time job (19.3%). The formal schooling level of the sample was low: 62% had less than 12 years of formal education, and only 10% had either completed or started graduate programs.

Instruments

The AUDIT is a 10-item instrument that aims at assessing: alcohol consumption (items 1 to 3), dependence (items 4 to 6) and alcohol consumption-related consequences (items 7 to10). It has a score ranging from 0 to 40, and it is used to identify AUDs and hazardous drinking. The AUDIT-C is a shortened version of the AUDIT that consists of items to assess alcohol consumption (items 1 to 3), and it is used to identify hazardous drinkers. Its score ranges from 0 to 12. The TWEAK consists of 5 items that evaluate: tolerance, worries about consumption, morning consumption, amnesia, and the need to cut down the consumption. Its score ranges from 0 to 7. The RAPS-QF is a 6-item short questionnaire that assesses remorse, amnesia, performance, morning consumption and quantity and frequency of consumption. Its score ranges from 0 to 6. Finally, the T-ACE is a 4-item instrument that assesses tolerance, annoyance, need to cut down drinking and morning consumption. Its score goes from 0 to 5.

The gold standard used for the diagnosis of AUDs was obtained from the Alcohol section of the Composite International Diagnostic Interview (CIDI). The CIDI30 is the most widely used structured diagnostic interview.31 It has been translated by an editorial committee and validated in different countries,32 showing very good performance in different contexts.33

Procedure

Data were obtained through probabilistic systematic cluster sampling. The face-to-face interviews took place in the perinatal centers during the 48 hours after labor, from October 2010 to February 2011. Prospective participants were invited to participate in the study, and informed consent was obtained either from them or from their parents/legal guardians, when the women were underage. After the interview, a brochure on “breastfeeding and alcohol” containing relevant information was given and commented with each woman. The study obtained the ethical approval from the ethics committee of one of our institutions.

Data Analysis

Exploration of the receiver operating characteristic (ROC) curves, using the Statistical Package for the Social Sciences (SPSS, SPSS Inc., Chicago, IL, US) software, version 12.0, and the sensitivity and specificity for each scale at various cut-off points were used to determine the optimal cut off-point for each scale.

Sensitivity refers to the percentage of true positives, that is, participants who screened positive and were diagnosed with an AUD, while specificity refers to the percentage of true negatives, that is, participants who screened negative and were not diagnosed with an AUD. In addition, we included the predictive values and likelihood ratios, both measures of diagnostic accuracy. These estimates and their 95% confidence intervals (95%CIs) were obtained using the online site Vassar Stats.34

The internal consistency was estimated as a measure of reliability using Cronbach's alpha for the AUDIT, and the Kuder–Richardson Formula 20 (KR-20) for the other screening instruments. The confidence intervals for the internal consistency coefficients were estimated using ICalfa, a Microsoft Excel (Microsoft Corporation, Redmond, WA, US) module to estimate CIs for Cronbach's alpha.35

Results

Areas under the ROC curve are presented in Fig. 1. All screening instruments had areas under the curve of over 0.80.The highest area under the curve was from the AUDIT, which was closely followed by the TWEAK.

Fig. 1.

ROC Curve for Alcohol Use Disorders Screening Tests.

The optimal cut off-point for each scale was determined by exploring the sensitivity and specificity at each cut-off point, along with their sensitivity and specificity, predictive values and likelihood ratios for the optimal cut-off scores and their 95%CIs, their mean and standard deviation. That information, along with the internal consistency coefficients (and their 95%CIs), the reliability for the cut-off scores used, and the correlation of the brief screening tests with the AUDIT are presented in Table 1. Weighted by prevalence, the likelihood ratios indicated that a positive result on the tests increased the probability of an AUD as follows: AUDIT - 56%, AUDIT-C - 38%, TWEAK - 36%, RAPS-QF - 39% and T-ACE - 35% (data not shown in tables).

Table 1. Psychometric properties of the screening tests.

	Cut-off point	Sensitivity and specificity for alcohol use disorder		Predictive values
				Positive result		Negative result
		Sensitivity (95%CI)	Specificity (95%CI)	True (95%CI)	False (95%CI)	True (95%CI)	False (95%CI)
AUDIT	4	0.87 (0.74-0.94)	0.86 (0.83-0.89)	0.36 (0.28-0.45)	0.64 (0.55-0.72)	0.99 (0.97-0.99)	0.01 (0.00-0.03)
AUDIT-C	3	0.90 (0.78-0.96)	0.79 (0.76-0.82)	0.28 (0.21-0.35)	0.72 (0.65-0.79)	0.99 (0.97-0.99)	0.01 (0.00-0.03)
TWEAK	2	0.96 (0.86-0.99)	0.77 (0.73-0.80)	0.27 (0.21-0.34)	0.73 (0.66-0.79)	0.99 (0.98-0.99)	0.01 (0.00-0.02)
RAPS-QF	1	0.79 (0.65-0.88)	0.82 (00.79-.85)	0.28 (0.21-0.36)	0.72 (0.64-0.79)	0.98 (0.96-0.99)	0.02 (0.01-0.04)
T-ACE	2	0.96 (0.86-0.99)	0.76 (0.72-0.79)	0.26 (0.20-0.33)	0.74 (0.67-0.80)	0.99 (0.98-0.99)	0.01 (0.00-0.02)
	Likelihood Ratio			M (SD)	Reliability (95%CI)	Cut-off point reliabilityc	Correlation with the AUDITd
	Positive (95%CI)		Negative (95%CI)
AUDIT	6.37 (5.06–8.02)		0.16 (0.08-0.31)	2.30 (2.85)	0.76 (73–79)a	0.82	—
AUDIT-C	4.36 (3.63–5.22)		0.12 (0.05-0.28)	1.95 (1.92)	0.76 (73–79)a	0.81	0.93**
TWEAK	4.13 (3.53–4.83)		0.05 (0.01-0.20)	0.82 (1.38)	0.49 (42–55)b	0.70	0.79**
RAPS-QF	4.38 (3.51–5.47)		0.26 (0.15-0.44)	0.35 (0.74)	0.56 (0.50-.61)b	0.75	0.75**
T-ACE	3.98 (3.42–4.65)		0.05 (0.01-0.20)	0.72 (1.13)	0.33 (24–41)b	0.71	0.60**

Abbreviations: 95%CI, 95% confidence interval; AUDIT, Alcohol Use Disorder Identification Test; AUDIT-C, Alcohol Use Disorder Identification Test – Consumption; M, mean; SD, standard deviation; RAPS-QF, Rapid Alcohol Problems Screen – Quantity Frequency; T-ACE, Tolerance, Annoyed, Cut-Down and Eye-Opener; TWEAK, Tolerance, Worried, Eye-Opener, Amnesia and Cut-Down.

Notes: ^aCronbach's Alpha. ^bKuder-Richardson Formula 20 index. ^cLivingston's K2 coefficient. ^dAlcohol Use Disorders Identification Test. ** Significant at 0.01 (two-tailed).

The optimal score for cut-off for the AUDIT was 4, while for the AUDIT-C it was 3. The highest sensitivity was obtained for the TWEAK, and the T-ACE, followed by the AUDIT-C, while the AUDIT and the RAPS-QF had a higher specificity. The mean endorsement was somewhat similar for the AUDIT and the AUDIT-C, and lower for the other brief screening tests. The reliability (internal consistency) was somewhat low for the instruments, especially for the TWEAK and the T-ACE. The differences among internal consistency coefficients were statistically significant (AUDIT and TWEAK: X²= 66.47, df =1, p < .0001; AUDIT and RAPS-QF: X²= 43.33, df =1, p < .0001; AUDIT and T-ACE: X² = 116.74, df =1, p < .0001). However, the reliability increased when it was estimated for the optimal cut-off scores, with the AUDIT, AUDIT-C and RAPS-QF showing the highest values. The correlation of the brief screening tests with the AUDIT (as the longest measure) was very high for the AUDIT-C, and high for the TWEAK and the RAPS-QF. The discrimination index for each item of the AUDIT is presented in Table 2, and the proportion of the indicator presence, the item discrimination index, and the decimal reduction time (D-value) are presented in Table 3.

Table 2. Item discrimination index^a for the Alcohol Use Disorder Identification Test (AUDIT).

Item 1	Item 2	Item 3	Item 4	Item 5	Item 6	Item 7	Item 8	Item 9	Item 10
0.42	0.650	0.77	0.51	0.42	0.35	0.30	0.55	0.33	0.20

Note: ^aCorrected item-test correlation.

Table 3. Item analyses for brief screening tests.

Item		TWEAK	RAPS-QF	T-ACE
1	Proportion of the indicator presence	30	4	30
2		6	6	5
3		1	4	6
4		6	1	1
5		7	20	−
1	Item discrimination indexa	0.37	0.26	0.33
2		0.35	0.48	0.28
3		0.22	0.34	0.39
4		0.45	0.22	0.19
5		0.43	0.45	−
1	Item discrimination indexb	0.72	0.19	0.72
2		0.37	0.36	0.22
3		0.11	0.29	0.31
4		0.36	0.11	0.11
5		0.31	0.55	−

Abbreviations: RAPS-QF, Rapid Alcohol Problems Screen – Quantity Frequency; T-ACE, Tolerance, Annoyed, Cut-Down and Eye-Opener; TWEAK, Tolerance, Worried, Eye-Opener, Amnesia and Cut-Down.

Note: ^aCorrected item-test point-biserial correlation; ^b D-value.

The items of the AUDIT with the highest performance were items 2 (number of drinks), 3 (frequency of heavy episodic drinking), and 8 (frequency of amnesia), while items 7 (remorse), 9 (harm from drinking), and 10 (concern of other people) had a poor performance. Regarding the frequency of endorsement for the brief screening tests, the most endorsed items were the first items of the TWEAK and the T-ACE (tolerance), and the quantity frequency (QF) item of the RAPS-QF. Less endorsed were the items that evaluate drinking in the morning (starter, or eye-opener), namely item 3 of the TWEAK, and item 4 of the RAPS-QF and the T-ACE. Regarding performance, items 4 and 5 of the TWEAK, 2 and 5 of the RAPS-QF, and 3 of the T-ACE had the highest item discrimination indexes. The tolerance item of the TWEAK and the T-ACE, the QF item of the RAPS-QF, and the amnesia item of the TWEAK had the best discrimination indexes. The ones with the poorest performance were those indicating morning drinking: the eye-opener item of the TWEAK and the T-ACE, and the starter item of the RAPS-QF.

Discussion

Areas under the Curve

Our results indicate that all instruments have satisfactory areas under the ROC curve, meaning that they all have a good overall ability to discriminate the participants with AUDs from those without it. The areas under the ROC curve found in this study were notably higher than those reported for the T-ACE,36 and similar to those found for the AUDIT-C in pregnant women from the US.37 We could not find previous studies reporting ROC curve values for the TWEAK or the RAPS-QF in pregnant women that used AUD as a criterion.

Optimal Cut-Off Scores

The optimal cut-off point for the AUDIT was 4. This is a low value compared with other values reported in the literature for different populations,38 and it is likely due to the characteristics of the sample, considering that alcohol consumption and the problems related to it are expected to be minimal in pregnant women. Although the use of a lower cut-off point for the AUDIT when screening women is often suggested, most intervention studies tend to use a cut-off point of 7.38 Studies evaluating the optimal cut-off point for pregnant women are scarce.

Interestingly, the cut-off point for the AUDIT-C was very close to that of the complete version of the AUDIT. This is probably explained by the characteristics of the population, who present frequent high consumption, but not in the more severe end of the spectrum of alcohol problems.

The optimal cut-off points for the other screening tests were like the ones reported in the literature,25 probably because the TWEAK and the T-ACE were designed especially for this population.

Sensitivity and Specificity

The highest sensitivity was obtained for the TWEAK and the T-ACE, followed by the AUDIT-C, while the AUDIT and the RAPS-QF had higher specificity. The AUDIT also had the lowest probability of false positives, while false negatives had a low probability in all tests. The high sensitivity of the TWEAK and the T-ACE could be due to the fact that they consider as positive the question about tolerance with two standard units, and the studied population has shown a high level of consumption.14 Conversely, their lower specificity could be due to the same fact: Argentinean pregnant women tend to drink while not presenting other indicators of an AUD. The high specificity of the AUDIT and the RAPS-QF for AUDs in women was already found in prior studies with Hispanic emergency patients,39 and the AUDIT itself has shown high specificity with pregnant women.36 In this study the sensitivity of the AUDIT was also high, but surprisingly lower than that of the AUDIT-C, a finding probably due to the lower cut-off score of the latter.

Instrument Endorsement

The mean of the scores was rather similar for the AUDIT and the AUDIT-C, likely due to the tendency of this population to endorse items referring to risky drinking, and the lower probability to endorse items evaluating alcohol-related problems. Nevertheless, it is not possible to know whether this aspect relates to their willingness to endorse a given item or criteria, or to their drinking practices and lesser degree of alcohol-related problems. As expected due to their brevity and reduced answer scale, the scores were lower for the other screening tests compared with the AUDIT.

Reliability

Reliability was somewhat low for all the instruments, but especially for the TWEAK and the T-ACE, showing an important degree of measurement error. However, the reliability improved when it was estimated for the optimal cut-off scores it improved, with the AUDIT, the AUDIT-C and the RAPS-QF having the highest scores. Similar results indicating the highest reliability for the AUDIT and the RAPS-QF were found in a previous study in Argentinean Emergency Department patients.40

Concurrent Validity

The correlation of the brief screening tests with the AUDIT (as the longest measure) was very high for the AUDIT-C, and high for the TWEAK and the RAPS-QF, indicating a better performance of these three short instruments over the T-ACE .

Properties of the Items

The items evaluating the number of drinks, the frequency of heavy episodic drinking, and the frequency of amnesia (namely items 2 and 3 of the AUDIT, item 5 of the RAPS-QF and 8 of the AUDIT, and item 4 of TWEAK and 2 of RAPS-QF respectively) showed the best overall performance (discriminating power) in all instruments. Heavy episodic drinking may indicate in pregnant women a higher degree of AUD severity than in other populations, as the condition of pregnancy often leads to a lower consumption.14 For its part, alcoholic amnesia (blackouts) has also been related to rapid consumption of high quantities, and in other Argentinean populations, such as university students or emergency patients, it has been closely associated with AUDs.41

The tolerance question of the T-ACE and the TWEAK (first item of both tests) was among the most endorsed items, probably because both instruments use two drinks as a cut-off point in this question. The consumption characteristics of Argentinean pregnant women might have resulted in a high number of positive responses to the item. The less-endorsed items, which also showed poor performance, were the items evaluating morning consumption (item 6 of the AUDIT, 3 of the TWEAK, and item 4 of the T-ACE and the RAPS-QF). This is one of the most severe consequences of AUDs,42 usually indicating physiological dependence, and it may not be representative of the population of pregnant women, as they usually do not show the characteristics of the most severe end of the AUD spectrum.43

We have analyzed the psychometric performance of the most used alcohol screening tests for AUD in Argentinean pregnant women, providing data that could be useful for the design of local prenatal screening strategies. Screening for prenatal alcohol consumption has proved to reduce alcohol intake during pregnancy,17 particularly if it is combined with a brief intervention.44 Although it is still necessary to discuss the place of prenatal alcohol assessment in relation with national health systems, implementing the evaluation is mandatory, considering the high prevalence of alcohol consumption during pregnancy reported in Argentina14 and other countries of the region.45 To our knowledge, this is the first report evaluating the performance of screening tests for pregnant women in a Spanish-speaking South American country.

Furthermore, the results presented here could be useful to the discussion about the cross-cultural validity of the analyzed screening tools to assess AUDs in prenatal care settings, particularly regarding the less broadly used instruments, like the T-ACE, the TWEAK or the RAPS-QF. Even though the T-ACE and the TWEAK have been used and recommended for screening pregnant women in other cultural contexts,46 our results show that the AUDIT has the best performance in screening for AUDs among Argentinean pregnant women.

Despite those contributions, the present study has some limitations that should be considered. First, the sample was from a single city, and although it was probably representative of the centers it was drawn from, the conclusions could not be extended to other populations. Although the overall evidence suggests that self-reporting is reliable for measuring alcohol consumption, pregnant women in other cultural contexts have shown to underreport their intake,47 undermining the reliability of the measures. However, considering the wide acceptance of drinking in Argentina, even among pregnant women,48 it is probably unlikely that this could result in a serious bias. The order of administration of the screening tests might also have an effect,49 and more research is certainly needed to determine whether it could have affected the results, and in which manner.

Conclusions

Although any level of alcohol consumption in pregnant women should be considered of high risk and targeted, identifying pregnant women with AUDs is especially important, since these women might require more than simple advice to quit drinking, namely a brief intervention or a referral to a specialized treatment. Nevertheless, most screening tests have been developed and validated in male populations, and validation studies in pregnant women are scarce.

In prior studies, mostly from the US and Europe, the T-ACE and secondly the TWEAK were the most used and recommended screening tools in the target population,46 probably because they were designed specifically for pregnant women. In that context, they both showed good sensitivity and specificity for detecting risk consumption and AUDs.25 However, in the local context, the psychometric performance of the AUDIT has proved to be superior for identifying AUDs in pregnant women. This result in not unexpected, since the AUDIT has proved to be the best screening tool in a variety of contexts and populations, especially in low-income countries.42 50 51 52