Evaluating Recall Bias in a Case-Crossover Design Estimating Risk of Injury Related to Alcohol: Data from Six Countries

Abstract

Introduction and Aims

Prior work suggests recall bias may be a threat to the validity of relative risk estimation of injury due to alcohol consumption, when the case-crossover method is used based on drinking during the same 6-hour period the week prior to injury as the control period. This work explores the issue of alcohol recall bias used in the case-crossover design.

Design and Methods

Data were collected on injury patients from emergency room studies across six countries (the Dominican Republic, Guatemala, Guyana, Nicaragua, Panama, and Canada), conducted in 2009–11, each with n ≈500 except Canada (n=249). Recall bias was evaluated comparing drinking during two control periods: the same 6-hour period the day before vs. the week before injury.

Results

A greater likelihood of drinking yesterday compared to last week was seen using data from the Dominican Republic, while lower likelihood of drinking yesterday was found in Guatemala and Nicaragua. When the data from all six countries were combined, no differential drinking between the two control periods was observed.

Discussion and Conclusions

These findings are in contrast to earlier studies showing a downward recall bias of drinking, and suggest it may be premature to dismiss the last week case-crossover method as a valid approach to estimating risk of injury related to drinking. However, the heterogeneity across countries suggests there may be some unexplained measurement error beyond random sampling error.

INTRODUCTION

Over the last decade the case-crossover method [1] has become the primary approach for estimating relative risk of injury associated with alcohol consumption in studies of emergency room (ER) patients [2, 3]. Unlike case-control studies that use either non-injury ER patients or community samples as controls, the case-crossover method compares the transient exposure before the event (e.g., drinking before injury) with injury patients’ own exposure during prior control period(s). This design, using case subjects as their own control, costs less and overcomes the problem of finding an appropriate control group encountered in case-control studies [1, 4]. Among several choices such as the previous day [5, 6] or usual frequency of drinking over the past 12 months [7–9], the same period one week prior has become the most commonly used control period for ER case-crossover studies. It has been used in a number of ER studies, [see, for example, 10, 11], including the WHO Collaborative Study on Alcohol and Injury [12, 13].

Since alcohol consumption often varies by day of the week, drinking measured during the same time last week is considered a better match with exposure at the time of injury than other control periods which may not be matched to day of the week, but has been criticized as a valid approach because of its potential recall bias [4, 14]. Since the case time period is more recent and easier to recall than the control time period, recall has become a concern for the case-crossover method in general, regardless of the control period used. Two studies found larger relative risk estimates for last-week in contrast to yesterday as the control time interval [15, 16], suggesting potential downward bias in recall of drinking based on the previous week, although neither study tested differences between the two control periods, or controlled for day of the week.

One ER study in Switzerland evaluating last-week drinking recall [17] found, after controlling for day of the week alcohol use decreased by length of the recall period, with average consumption 0.9 drinks lower 7 days prior compared with yesterday, but recall bias was only significant for sporadic drinkers (drinking no more than 4 days a week), and not for regular drinkers (5 or more days a week). While the Swiss study provides convincing evidence of last-week drinking recall bias, at least for non-regular drinkers, drinking was assessed for total consumption per day over the previous 7 days, using a retrospective diary; a design generally different from that applied in case-crossover analysis in other ER studies, in which the control period is matched to an exact hazard period prior to the injury event.

It thus remains unclear whether recall bias is a real problem associated with the use of the same 6-hour period the week prior to injury. Several recent ER studies, each using the WHO study instrument which included “the same 6-hour period last week,” added a second control period of drinking “the same 6-hour period the day before injury”. Examining data from these studies across six countries, we aim to evaluate (1) the potential recall bias comparing drinking between the same 6-hour period the week prior to injury and the day prior, after controlling for day of the week, and (2) whether a patient’s usual drinking frequency is related to differential recall.

METHODS

Data

Data come from 10 ERs in five countries (Dominican Republic, Guatemala, Guyana, Nicaragua, Panama) comprising the Pan American Health Organization (PAHO) Collaborative Study on Alcohol and Injury (2010–2011) (N’s ranged from 485–518 across countries) and two ERs in Vancouver, Canada (2009) (N=249). Probability samples of injured patients 18 years and older (with equal sampling of each shift for each day of the week) consecutively arriving within six hours of the event were approached with informed consent and administered a 25-minute structured questionnaire. Completion rates averaged 93% for the PAHO studies and 69% for the Canadian study. All studies were approved by ethic review boards prior to their implementation.

Measures

All patients were interviewed with a modified version of the instrument used in the WHO Collaborative Study on Alcohol and Injury [13], which obtained data on, among other items, frequency of usual drinking and alcohol consumption a) within six hours prior to injury, b) during the same six-hour period the previous day, and c) during the same 6-hour period the previous week. For the two control periods, patients were probed on where they had been and what they had been doing at the time.

For the three 6-hour periods evaluated (before injury, yesterday and last week), patients who reported drinking were asked a series of questions regarding the type of alcohol beverages, beverage container size and number of containers consumed, with questions in each country adapted to accommodate local beverages. Volume of consumption was then transformed to standard drinks for analysis, defined as 12 grams of pure ethanol per drink.

Analyses

To examine whether alcohol consumption is differently reported during the same 6-hour period the week prior to injury and yesterday, two approaches are used: between-and within- persons. Using the between-person approach, standard logistic regression or ordered logistic regression is fitted for dichotomous drinking (any vs. none) or consumption volume (categorized as none, 1–2, 3–5 and 6+ drinks) as the dependent variable, predicted by the day for which drinking was assessed (yesterday vs. the week before). The within-person approach is essentially a case-crossover analysis with yesterday treated as the case period and last week as the control. Conditional logistic regression is fitted with the dependent variable indicating cases vs. control periods (i.e., yesterday vs. last week) and drinking measures entered as independent variables. For both approaches, an OR larger than one indicates respondents are more likely to report drinking the day prior compared to the week prior.

To examine cross-country variation, the test of heterogeneity is carried out across the six countries on the log ORs and their associated standard errors, using the Q-statistics [18]. Last, the modifying effect of usual drinking frequency is evaluated, repeating the analysis above using dichotomous drinking separately for different frequency levels (i.e. 3–4 times a week, 1–2 times a week, 2–3 times a month, once a month, and less than once a month).

One advantage of the within-person approach compared to the between-person approach is that it controls all individual characteristics stable over time. The within-person estimation, however, can be much less efficient than the between-person approach. In the simplest form of the within-person approach of a 2×2 table cross-classifying yesterday’s and last week’s drinking (any vs. none), the conditional logistic regression OR is equivalent to ratios between two discordant pairs with all concordant pairs discarded, an estimate which can be highly unstable. Given the pros and cons of either approach, results from both methods are shown demonstrating the robustness of findings.

RESULTS

Table 1 shows the distributions of gender, age and alcohol use during the three 6-hour periods for the six countries separately and combined. While the prevalence of drinking before injury was consistently higher than drinking the same time yesterday or last week, there was no clear difference between the two control periods, nor did volume differ significantly between the two periods.

Table 1.

Emergency Room Injury Patients from 5 PAHO Countries and Canada: Demographics and Prevalence of Drinking Six-hour before Injury and the same Time Yesterday and Last Week(%)

N	Dominican Republic	Guatemala	Guyana	Nicaragua	Panama	Canada	Combined
	501	513	485	518	490	249	2,756
Gender male	80.8	69.4	72.4	69.1	68.8	68.4	71.1
Age
18–29	53.1	46.4	43.0	53.2	41.6	35.1	46.4
30–49	33.0	35.3	44.9	31.6	40.2	39.5	37.1
50+	13.9	18.3	12.1	15.2	18.2	25.4	16.5
Alcohol 6-hr before injury	19.3	21.1	21.0	21.5	20.8	22.2	20.9
Alcohol same time yesterday	9.1	4.7	7.5	8.6	10.4	10.8	8.3
Alcohol same time last week	6.3	7.5	7.9	11.6	8.7	10.5	8.6
Volume 6-hr before injury1
1–2 drinks	15.0	14.8	27.8	4.1	14.9	29.6	16.8
3–5 drinks	18.7	28.7	20.6	10.2	26.6	31.5	22.2
6+ drinks	66.3	56.5	51.6	85.7	58.5	38.9	61.0
Volume same time yesterday1
1–2 drinks	25.0	12.5	26.5	5.3	9.5	61.5	23.5
3–5 drinks	25.0	12.5	14.7	15.8	23.8	23.1	18.4
6+ drinks	50.0	75.0	58.8	78.9	66.7	15.4	58.2
Volume same time last week1
1–2 drinks	31.8	13.2	24.2	6.3	13.2	60.0	21.1
3–5 drinks	4.6	26.3	24.2	10.4	13.2	12.0	17.2
6+ drinks	63.6	60.5	51.5	83.3	73.7	28.0	61.8

¹Only among those who reported any alcohol use during the period

The effect of day of the week was then examined between drinking yesterday and last week. Table 2 shows, for the combined data, the proportion drinking during the two control periods, when yesterday or the day last week fell on a specific day of the week. For example, when last week was a Monday, 8.36% reported drinking during that control period, while 12.62% drank during the control period yesterday, which was a Sunday. Drinking was more likely to occur during weekends than weekdays. Table 2 also shows the distribution of day of the week for ER admission. Injury patients were more likely to arrive on Saturday and Sunday than on weekdays. The results suggest that, compared to the 6-hour period yesterday, the same control interval last week is more likely to fall onto a weekend day and injury patients are more likely to drink during this period.

Table 2.

Distribution of day of the week for ER admission and its association with drinking for the total combined sample

Day of the week	Drinking Yesterday (row %)		Drinking last week (row %)		ER admission
	Any drinking 6-hr period	Chi-square statistics	Any drinking 6-hr period	Chi-square statistics	Distribution	Binomial test1
Monday	7.01%	27.91 (df=6), P<0.001	8.36%	34.68 (df=6), P<0.001	12.45%	P=0.006
Tuesday	5.16%		5.03%		11.98%	P<0.001
Wednesday	5.56%		10.14%		13.18%	P=0.102
Thursday	6.06%		2.19%		12.09%	P<0.001
Friday	8.46%		8.95%		15.07%	P=0.241
Saturday	12.12%		10.04%		17.66%	P<0.001
Sunday	12.62%		12.85%		17.58%	P<0.001

¹Binomial test tests each proportion equals to 1/7; Chi-square statistic is 71.81 (df=6, p<0.001) when testing all proportions across days of a week equal to 1/7

Table 3 shows the results from logistic regression (for between-person comparisons) and conditional logistic regression (for within-person comparisons) models comparing dichotomous drinking (any vs. none) between the two control periods. A greater likelihood of drinking yesterday vs. last week (i.e. OR larger than one) was seen for the Dominican Republic. The unadjusted OR from the between-person approach was 1.49 (p<0.05) and the OR adjusting for gender, age and day of week was 1.45 (p<0.10). From the within-person approach, the unadjusted OR was 1.86 (p<0.10), dropping to 1.57 (not significant) when adjusting for day of the week. These results also demonstrate the within-person approach is less efficient, with wider confidence intervals seen than the between-person approach. In contrast, a lower likelihood of drinking yesterday (vs. last week), was seen for Guatemala and Nicaragua. For Guatemala, the adjusted OR estimate was 0.61 (p<0.10) from the between-person approach and 0.56 (p<0.10) from the within-person approach, while for Nicaragua, the adjusted OR was 0.78 (not significant) from the between-person approach and 0.49 (p<0.05) from the within-person approach. No clear evidence of differential drinking was seen for the other three countries. As a formal test of differences across the six countries, Table 3 shows the Q-statistics for the test of heterogeneity. For both between- and within-person approach, tests are significant (p<0.05) for unadjusted estimates, and marginally significant for adjusted estimates, indicating variation across countries. For the combined data, there was no evidence of differential drinking between the two control periods using either the between- or within-person approach.

Table 3.

Odds Ratios (95% Confidence Interval) of any drinking yesterday versus last week, separately from between-and within-person approach

	Dominican Republic	Guatemala	Guyana	Nicaragua	Panama	Canada	Combined	Test of heterogeneity3
Between-person
Unadjusted	1.49 (1.02, 2.18)*	0.61 (0.37, 1.00)†	0.95 (0.64, 1.40)	0.71 (0.53, 0.97)*	1.22 (0.86, 1.73)	1.04 (0.62, 1.72)	0.96 (0.82, 1.12)	Q=14.1 (p=0.015)
Adjusted1	1.45 (0.98, 2.16)†	0.61 (0.36, 1.02)†	0.97 (0.65, 1.47)	0.78 (0.55, 1.11)	1.24 (0.85, 1.83)	1.03 (0.60, 1.76)	1.00 (0.85, 1.18)	Q=10.0 (p=0.075)
Within-person
Unadjusted	1.86 (0.97, 3.56)†	0.58 (0.33, 1.01)†	0.82 (0.44, 1.53)	0.41 (0.21, 0.81)*	1.38 (0.79, 2.42)	1.00 (0.51, 1.96)	0.89 (0.70, 1.14)	Q=14.8 (p=0.011)
Adjusted2	1.57 (0.80, 3.07)	0.56 (0.31, 1.01)†	0.83 (0.44, 1.56)	0.49 (0.24, 0.99)*	1.34 (0.73, 2.48)	1.01 (0.51, 1.99)	0.93 (0.72, 1.19)	Q=9.7 (p=0.083)

¹Controlling for gender, age and day of the week

²Controlling for day of the week

³Test of heterogeneity reports Q-statistics and relevant p-values based on χ² test across 6 countries (df=5)

^†p<0.10,

^*p<0.05

To examine differences in number of drinks (none, 1–2, 3–5 and 6+) between the two control periods, sites were combined to obtain sufficient data for all exposure levels. For the between-person approach using ordered logistic regression, the adjusted OR (95% confidence interval) was 0.98 (0.83, 1.17), while for the within-person approach, the adjusted OR was 0.95 (0.85, 1.05). It should also be noted that no significant difference in drinking was observed between the two control periods when the drinking measure was treated as multinomial (results not shown).

The modifying effect of patients’ usual frequency of drinking was also examined, using both between- and within-person approaches across individuals with different drinking frequencies for the combined data. For the five drinking frequency groups separately (3–4 times a week, 1–2 times a week, 2–3 times a month, once a month, and less than once a month), the adjusted ORs for any drinking yesterday (vs. last week) were 0.92 (0.67, 1.26), 1.03 (0.73, 1.45), 1.00 (0.62, 1.61), 1.58 (0.93, 2.69) and 0.54 (0.29, 1.00), respectively, using the between-person approach. The corresponding adjusted ORs (95% CI) using the within-person approach were 0.80 (0.46, 1.39), 0.93 (0.59, 1.46), 1.00 (0.57, 1.73), 1.86 (0.84, 4.10) and 0.52 (0.26, 1.05), respectively. Only those who reported drinking “once a month” showed signs of a greater likelihood of drinking yesterday (vs. last week), although the results were not statistically significant. This upward trend was further offset by the lower estimate (ORs<1, not significant) for the “less than once a month” group.

DISCUSSION

Using ER data from six countries, this study compared injury patients’ self-reported drinking during the 6-hour control periods for the day prior to injury and one week before injury, and found that for all data combined across sites there was no significant difference between the two time intervals. Nor was a difference in drinking between the two periods found across patients’ usual drinking frequency levels. On the other hand, cross-country variation was observed in this study, with a greater likelihood of drinking yesterday than last week seen in the Dominican Republic, but a lower likelihood in Guatemala and Nicaragua. Tests of heterogeneity were marginally significant even after adjusting for day of the week, suggesting there may be some unexplained measurement error beyond random sampling error.

One key difference between the Swiss study, noted in the introduction [17], and the present study is the manner in which drinking was assessed. In the earlier study, a retrospective diary applying the timeline followback method was used to record ER patients’ alcohol consumption for each full day across the seven days before injury. The current study, in contrast, first asked whether injury patients drank alcohol during the 6-hour period before injury, then whether they drank in the same 6-hour window on the day before injury and one week prior. In a study of ER injury patients in Missouri, Vinson et al. [19] showed a significant decay of average consumption per day for the last 8 days based on total daily consumption, in comparison to a much smaller and non-significant recall bias for drinking during the 6-hour window each day, corresponding to the same 6-hour period before injury. The 6-hour window, the authors argued, had personal meaning for injury patients, leading to greater engagement in the interview and accordingly less recall bias. In the present study, each injury patient was also asked about the context (where they were) and activities (what they did) during the same 6-hour window, both the day and week prior, before alcohol consumption was assessed. Such contextual cueing likely helps further improve the respondents’ reports of earlier consumption and may serve to alleviate potential recall bias.

Instead of comparing directly the control periods, as in this and the Swiss study [17], recall bias can be examined indirectly, estimating the risk estimates for the two control periods separately [15, 16]. Using yesterday as the control, the OR of injury related to any drinking was 3.32 (95% CI: 1.99, 5.54) for the Dominican Republic, 12.71 (5.89, 27.44) for Guatemala, 4.44 (2.67, 7.41) for Guyana, 7.44 (3.71, 14.93) for Nicaragua, 3.71 (2.17, 6.33) for Panama, and 3.27 (1.67, 6.43) for Canada. In contrast, ORs using last week as the control were: 9.00 (4.12, 19.65), 4.58 (2.79, 7.52), 3.32 (2.12, 5.19), 4.15 (2.27, 7.61), 4.00 (2.38, 6.70) and 4.12 (1.91, 8.93), respectively. One interesting note is that although both the direct and indirect approaches should show bias in the same direction, their estimates are not directly comparable. Using the Dominican Republic as example, the estimated magnitude of bias from indirect method is 9.00/3.32=2.71, versus the OR=1.86 using the direct approach (Table 2, unadjusted within-person approach). The incomparability of the two estimates is the result of the matched case-control design, in which the estimate uses only subjects from the two discordant cells of the 2×2 table. This is in contrast to the standard case-control design in which all 4 cells are used in estimating the OR, in which case the direct estimate can be obtained from the indirect estimates simply by changing the reference category. While it is hard to argue which method gives a more valid estimate of bias, the direct comparison allows formal hypothesis testing, as well as controlling for important confounders such as day of the week and is thus the approach used for this study.

The purpose of this study was to compare drinking between two control periods to examine whether earlier studies suggesting recall bias related to last week as the control period could be replicated, and is thus not a full assessment of all potential biases related to respondents’ underlying cognitive and communicative processes during the survey. One potential design issue is related to the common method bias such as the priming effect and consistency motif [20], when questions on drinking during two control periods are asked sequentially. The priming effect may occur when cognitive processes involving the first question (yesterday’s drinking) carry over and influence a subsequent similar question (last week’s drinking), while the consistency motif is related to the potential bias arising from respondents’ desire to maintain consistency in their responses to similar questions. The common method bias leading to suspiciously similar answers, however, may be less a concern here. As observed above, drinking varied between yesterday and last week if one period was on a weekday while the other was on a weekend, suggesting that respondents were not just repeating their previous answers. Given the importance of the cognitive process in behavior survey research, however, future research is needed on these design issues, for example, using a counterbalancing design to examine the priming impact [20].

Last, all studies in the current analysis are from countries in the Americas. Despite regional variation in drinking, irregular heavy drinking is the prevailing pattern in the Latin America and Caribbean (LAC) area [21, 22]. One limitation of the current study is that the results may be population-specific, applying only to drinking cultures primarily dominant in the LAC region, and thus cannot be generalized to other countries.

Overall, results suggest no evidence supporting existence of strong recall bias, and the last week case-crossover method cannot be ruled out as a valid approach to estimating risk of injury related to drinking. However, the heterogeneous findings across countries here, as well as results from earlier investigations [15–17], suggest that it may be premature to accept the last-week approach as an established method. Future surveys may consider including yesterday’s drinking as a routine check for potential recall bias in the last week case-crossover approach. When no strong recall bias is observed, multiple control periods can be used to produce more efficient and stable estimates, as errors from different control periods may cancel one another. Nevertheless, results from this study, together with recent findings showing strong recall bias using the usual frequency case-crossover method [23], argue for more methodological research on the case-crossover design in general, as used in the alcohol-injury risk estimation.