The effect of survey sampling frame on coverage: the level of and changes in alcohol-related mortality in Finland as a test case

Abstract

Aim

Exclusion of e.g. the homeless and institutionalized from survey sampling frames has been suggested to be one important reason for why surveys cover only a modest proportion of all alcohol sold. We assess to what extent this might be the case, using cross-sectional and longitudinal mortality data from Finland, where in 2004 alcohol taxes were lowered by one-third, and surveys were unable to capture the 10% increase in per capita consumption.

Design

We compared the level of and the change in alcohol-related mortality in 2001-2003 and 2004-05 in (1) the whole population, (2) the population included in the sampling frame of many Finnish surveys and (3) the population excluded from the sampling frame.

Data

Finns aged 15 and above, individually linked to cause of death data.

Results

The population outside survey sampling frames constituted 1.4% of the whole population and had a high rate of alcohol-related deaths. E.g., among men the rate of directly alcohol-attributable causes was 3.7 times higher than in the survey population. Among women the rate ratio was 4.6. The exclusion of the non-survey population reduced the estimated level of alcohol-related mortality by 1-4%. Similarly, the non-survey population had only a marginal effect on the estimates of temporal change.

Conclusions

Alcohol-related mortality, and hence probably also alcohol consumption, is on the average much higher in the subgroups of population excluded from survey sampling frames. Due to the small size of the excluded group in the Finnish context, this has only a small effect on population-level estimates.

INTRODUCTION

It is well established that general population surveys manage to cover only a modest part of all alcohol sold – or consumed – in a society [1]. What is less well known is to what extent different factors contribute and explain this under-coverage. Even if sales data are hardly error-free, it may be assumed that biases in survey estimates are the main source of under-coverage. These biases or errors can be divided into two [2]. First, there is (systematic) error in the responses obtained. This covers errors due to inadequate measurement instrument, inadequate interviewer performance and failure of the respondent to report the correct value, whether intentionally or unintentionally. These problems can be partly addressed by improving interviewer training and the questions on consumption [3].

The second type of error in surveys is non-observation error, which is caused by the fact that we cannot measure every individual in the population [2]. Also this error causes under-coverage, as for various reasons heavier drinkers tend to be under-represented in respondent samples. The first stage where systematic non-observation error may occur is that the sampling frame of the survey fails to cover the entire target population, i.e. the entire population the researcher is interested in. Typically, the non-household population including the homeless and the institutionalized – groups that may contain a relatively large proportion of problem drinkers [4] – are excluded even when population lists are used as the sampling frame, and often the entire age range of the general population is not covered, either.

Another important type of non-observation error is caused by non-response that results when people who are either too busy to respond, are hard to locate or actively refuse are not in the data, and again heavy drinkers (but also abstainers) may be over-represented among these groups [5, 6]. Indeed, surveys with register-based follow-up have shown that non-respondents have a higher rate of alcohol-related hospitalization than respondents [5]. In order to be able to draw justified conclusions on the basis of their data, survey researchers need to know more about the reasons for under-coverage. Particularly, as rates of non-response continue to increase nearly universally, more information on the importance and nature of non-observation error is called for.

Gmel and Rehm [7, p.505, 527-8] have hypothesized that one central reason for why survey studies cover only a small proportion of recorded alcohol consumption is that survey sampling frames exclude heavy drinking parts of the population, e.g. the homeless and the institutionalized. They conclude “more research is needed, especially on the question of coverage of high-drinking groups by the sample frame. The distribution function of alcohol suggests that very small subgroups could be responsible for a considerable portion of overall consumption, and thus the identification of groups with high consumption missed by usual sample frames would be of primary importance here (e.g., institutionalized, homeless).” (p. 527-8). It is not possible to get reliable information on this from surveys [5], but register-based studies are needed.

In this paper, we assess what impact the exclusions from sampling frames have on one alcohol-related outcome, namely alcohol-related mortality, and its changes. We compare register data on the population inside and outside the survey sampling frame, or the survey and non-survey population, for short. The cross-sectional data will show us the non-survey population’s impact on the population estimates of alcohol-related mortality.

The longitudinal part of the study relates to the question of what happened to alcohol consumption when alcohol taxes were lowered in Finland by one-third, on the average, in March 1^st, 2004. Estimated total consumption (recorded and estimated unrecorded consumption) increased by 10% from 2003 to 2004, and another 2% in 2005 [8]. These estimates are held to be relatively reliable: recorded sales, which in 2003 were estimated to make four-fifths of the total, increased by 6.5%, and the more uncertain unrecorded consumption, estimated regularly by smaller surveys and the majority of which is imported alcohol, was estimated to increase by nearly one-fourth [9]. The increase in consumption is also supported by the fact that alcohol-related mortality increased considerably in 2004 [9,10]. Some survey results based on both panel and repeated cross-sectional designs, on the other hand, were unable to show any increase in consumption [11,12]. One of the hypotheses that was put forward to explain this discrepancy was that heavy drinkers, who are often not well represented in surveys, accounted for a major part of the increase in consumption. Gmel and Rehm’s [7] supposition that non-survey population has an important role in biasing survey studies is relevant here and worth assessing with longitudinal register data on alcohol-related mortality.

We study this issue by using the good and - compared to many other countries – relatively reliable [13, 14] Finnish register data on alcohol-related mortality, which have been linked to census data that include information on whether the person belongs to the sampling frame often used with surveys in Finland. Hence, our aim is to compare the level of and the changes in alcohol-related mortality in the whole population and in the survey and non-survey population. These results contribute to our knowledge of the importance of exclusions in survey sampling frames as the source of undercoverage in surveys.

DATA AND METHODS

The register data was obtained from Statistics Finland (permission CS-53-457-06). The study population was taken from employment statistics for 2000 and 2003 and includes all Finns aged 15 and above. These data were linked individually by means of personal identification codes to records from the death register for 2001-2003 (“before”) and 2004-2005 (“after”), respectively.

In Finland, survey samples are often drawn using the same population list as that which is covered by the employment statistics, but the so called “900 group” is typically excluded. Here, “population outside survey frame” was defined as a membership in this group. This group was here divided into three subgroups: “the homeless”, i.e. people without a permanent address; “the institutionalized”, i.e. incarcerated people (who were too small a group to be analysed alone) and people living in hospitals and other institutions, and “others”, i.e. temporarily abroad, conscripts, absent from municipality due to studies or temporary work, parliament members, diplomats and those whose information is restricted due to safety issues.

Causes of death were coded using the Finnish edition of the Tenth Revision of the International Classification of Diseases (ICD). We used three different categories of alcohol-related causes of death, which can be assumed to be linked to alcohol in different extents. (1) Directly alcohol-attributable causes, i.e. those in which the underlying cause of death was an alcohol-attributable disease or fatal alcohol poisoning. In these, the link to alcohol is the closest and most certain. (2) Deaths from accidents and violence where alcohol intoxication (ICD10 code F100) has been coded as a contributory cause. In these cases, the physician writing the diagnosis has, on the basis of blood alcohol level measurement and other information available to him, believed alcohol intoxication to be a contributory cause, but in some cases the death would have occurred even without alcohol. (3) Other deaths where contributory cause included either an alcohol-attributable disease (e.g. alcohol dependence) or alcohol intoxication (e.g. when coronary heart disease was the underlying cause). These cases are the most heterogeneous and in them, causality could be less certain than with the other categories. Alcohol-attributable diseases were: alcohol dependence syndrome (ICD10 code F102), other mental and behavioral disorders due to alcohol (F101, F103-109), alcoholic cardiomyopathy (I426), alcoholic liver disease (K70), alcoholic diseases of the pancreas (FCD K860), and additionally a few rarely occurring categories (K292, G312, G4051, G621, G721).

The different groups to be compared had quite different age distributions. The mean ages among men were 40.3, 53.9, 40.8 and 44.7 among the homeless, institutionalized, “other 900” and other men, respectively. Among women the mean ages were 38.2, 73.0, 37.7 and 47.7. These differences partly explain differences in alcohol-related mortality. Hence, age standardization was needed. This was done using direct standardization, with the whole Finnish population in 2001-05 as the standard population. We calculated 95% confidence intervals (CI) for the age-standardized rate ratios using the formula by Clayton and Hills [15]. CI’s for the changes in standardized rates were calculated using the formula for the difference between two age standardized rates [15], and dividing this by the estimate for the first time point in order to turn it into a percentage figure.

RESULTS

From 2001-03 to 2004-05, alcohol-related mortality increased among both men (Table 1) and women (Table 2). The proportional increase was the largest among women in all three cause-of-death categories and among men for the directly alcohol-attributable causes. Among men, mortality from injuries with alcohol intoxication as an underlying cause did not increase.

Table 1.

Person years, number of alohol-related deaths and rate per 100 000 among men in the whole population and in the population inside and outside survey sampling frame in 2001-03 and 2004-5

		Person years	(1) Directly alcohol attributable		(2) Injury and violence with alcohol intoxication		(3) Other alcohol- related
		(1000’s)	Rate per		Rate per		Rate per
			N	1000 000	N	1000 000	N	1000 000
All men	2001-03	6 191	3 573	58	2 344	37	2 800	47
	2004-05	4 192	3 045	71	1 620	38	2 166	52
	Change, %			24 [18, 29]		2 [−4, 9]		10 [5, 16]
Men in survey frame	2001-03	6 094	3 422	56	2 239	36	2 667	45
	2004-05	4 125	2 889	68	1 557	37	2 057	50
	Change, %			22 [17, 28]		3 [−4, 10]		10 [4, 16]
Men outside survey frame
	2001-03	97	151	185	105	105	133	179
	2004-05	67	156	254	63	90	109	197
	Change, %			37 [6, 48]		−14 [−51, 18]		10 [−16, 35]
	Rate ratio (ref=survey pop.)1			3.7 [3.1, 4.4]		2.4 [1.9, 3.1]		4.0 [3.2, 4.9]
- Without permanent residence	2001-03	30	62	310	52	158	34	172
	2004-05	25	66	294	37	124	34	198
	Change, %			−5 [−55, 44]		−22 [−79, 24]		15 [−44, 69]
	Rate ratio (ref=survey pop.)1			4.3 [3.2, 5.8]		3.3 [2.4, 4.7]		4.0 [2.6, 6.1]
- Institutionalized	2001-03	29	53	171	17	66	60	176
	2004-05	23	63	260	11	71	56	220
	Change, %			51 [1, 67]		8 [−69, 83]		25 [−16, 56]
	Rate ratio (ref=survey pop.)1			3.8 [2.9, 5.0]		1.9 [1.0, 3.5]		4.4 [3.3, 5.9]
- Other outside survey frame	2001-03	38	36	103	36	95	39	119
	2004-05	20	27	154	15	73	19	98
	Change, %			49 [−20, 86]		−23 [−109, 51]		−18 [−89, 46]
	Rate ratio (ref=survey pop.)1			2.2 [1.4, 3.6]		2.0 [1.1, 3.5]		2.0 [1.2, 3.2]

¹Ratio between the given subgroup’s mortality rate in 2004-05 and that of the population in the survey frame.

²Age-standardized mortality rate per 100 000 inhabitants

Table 2.

Person years, number of alohol-related deaths and rate per 100 000 am ong women in the whole population, in population inside and outside survey sampling frame in 2001-03 and 2004-5

		Person years	(1) Directly alcohol attributable		(2) Injury and violence with alcohol intoxication		(3) Other alcohol- related
		(1000’s)	Rate per		Rate per		Rate per
			N	1000 000	N	1000 000	N	1000 000
All women	2001-03	6 605	941	14	346	5	443	7
	2004-05	4 455	818	18	293	7	414	9
	Change, %			27 [16, 38]		26 [8, 43]		37 [22, 52]
Women in survey frame	2001-03	6 522	898	14	339	5	428	6
	2004-05	4 398	789	18	289	7	396	9
	Change, %			28 [17, 39]		26 [8, 44]		36 [19, 52]
Women outside survey frame
	2001-03	83	43	85	7	14	15	27
	2004-05	57	29	82	4	11	18	46
	Change, %			−4 [−55, 48]		−18 [−159, 114]		74 [−20, 105]
	Rate ratio (ref=survey population)1			4.6 [3.1, 6.8]		1.7 [0.6, 4.5]		5.3 [3.1, 9.1]
- Without permanent residence	2001-03	8	17	232	3	25	4	72
	2004-05	7	11	208	3	42	6	92
	Change, %			−10 [−107, 84]		67 [−96, 177]		27 [−97, 139]
	Rate ratio (ref=survey population)1			11.7 [5.4, 25.7]		6.3 [2.0, 19.7]		10.6 [4.7, 23.7]
- Institutionalized	2001-03	48	16	50	1	3	7	12
	2004-05	35	11	40	1	4	7	19
	Change, %			−19 [−134, 89]		33 [−220, 270]		51 [−108, 175]
	Rate ratio (ref=survey population)1			2.3 [1.1, 4.9]		0.6 [0.1, 4.5]		2.2 [0.7, 7.2]
- Other outside survey frame	2001-03	27	10	52	3	13	4	24
	2004-05	15	7	89	0	0	5	46
	Change, %			70 [−51, 134]		−100 [−, −]		90 [−74, 169]
	Rate ratio (ref=survey population)1			5.0 [2.2, 11.4]		0.0 [−, −]		5.3 [1.9, 15.0]

¹Ratio between the given subgroup’s mortality rate in 2004-05 and that of the population in the survey frame.

²Age-standardized mortality rate per 100 000 inhabitants

The population outside the survey frame made only 1.6% of the whole population among men and 1.3% among women. As a whole, this part of the population had a much higher rate of alcohol-related deaths, particularly for the directly alcohol-attributable causes and the third, heterogeneous alcohol-related cause-of-death category. Among men, the rate ratio (RR), as compared to the male population in the survey frame, was 3.7 for the former causes and 4.0 for the latter (Table 1), i.e, men outside the survey frame had about a 4-fold alcohol-related mortality rate compared to men in the survey frame. Among women, the corresponding RR’s were 4.6 and 5.3 (Table 2).

Of the three subgroups of the non-survey population, the homeless and the institutionalized had the highest level of alcohol-related mortality among men. Among women, the homeless -- who were a much smaller group among women than among men -- far exceeded any other groups in the level of alcohol-related mortality. It should be noted that in some categories, particularly among women, the number of cases was so small that the results should be read with caution.

Even if the non-survey population had a much higher than average rate of alcohol-related problems, their exclusion only accounted for a very small proportion (1-4% depending on sex and cause-of-death category) of alcohol-related mortality. This can be seen in Tables 1-2 by comparing the level of alcohol-related mortality in the whole population and in the part of the population included in the survey frame.

A similar observation can be made with regard to the change in alcohol-related mortality: the percentage change is hardly affected at all when the population outside the survey frame is excluded. This is partly due to their small relative size, partly due to the fact that typically the relative increase in alcohol-related mortality was not larger in the population outside the survey frame than inside it. In some cases, like with homeless men, the age standardization actually caused the change estimate to be somewhat misleading, as there was a decrease in alcohol-related mortality among the elderly homeless people. Homelessness is rare among the elderly in Finland, but age standardization meant that the small number of elderly homeless people were now given a many times bigger weight than without age standardization. However, this did not substantially affect the main findings.

DISCUSSION

According to the results, the level of alcohol-related mortality is a lot higher in the part of the population typically excluded from the survey sampling frames in Finland, particularly because of a high rate among the homeless and, among men, also among the institutionalized. However, the results also showed that in the Finnish context the size of this population is so small that its effect on the population-level estimates of mortality was marginal. This conclusion applies to all three different types of alcohol-related mortality examined here.

The usefulness of the study design used here rests on the reliability of the data registers in Finland. In Finland, even the homeless are found in registers. Due to the free maternity services, also the children of down-and-out mothers are within the registration system, and even if adult vagrants can be “missing from authorities” for years or decades, they will be registered at death again: there is only approximately one death every two years in Finland for which the identity of the deceased remains unknown to the authorities (source of information: the cause-of-death archive, Statistics Finland). At least so far, immigrants have not posed a great problem for the cause-of-death registration, either, partly due to their relatively low numbers and partly due to the effectiveness of the system. The diagnosing of alcohol-related deaths in Finland is much more reliable than in many other countries [13, 14]. The possible bias in these diagnoses with respect to the social status of the deceased has not been studied, but if there was a bias to over-emphasize the role of alcohol among the homeless – or under-estimate its role among socially integrated individuals (in the sampling frame) – the estimates of the impact of the sampling frame obtained in the current study would rather be over-than underestimates.

However, the limits to the generalizability of these results to other settings should be acknowledged. In particular, survey sampling frames vary from one survey to the next. In Finland, census-based population lists are typically used as the source for sampling in scientific studies, and only about 1.5% of the population is excluded from the sampling frame at the outset. In many other places, the proportion of the population excluded from the sampling frame can be substantially larger, especially when telephone surveys are used, and then the impact of exclusions can be greater. We know of only one other study our results can be compared to [16]. The estimates of the prevalence of problem drinking in a Northern California county changed from 11.3% on the basis of a population sample, to 11.7% on the basis of all information, including estimates on non-household population. Hence, also their result suggests that the effect of the non-household population on population level estimates of alcohol-related problems is small.

Gmel and Rehm [7] assumed that the non-survey population has an important role in why surveys under-estimate alcohol consumption. This assumption concerned alcohol consumption, not alcohol-related mortality. Likewise, survey researchers would like an estimate of the proportion of the missing alcohol in surveys that can be accounted for by the non-survey population. However, it is far from straight-forward to transform estimates of excess mortality to estimates on alcohol consumption. If the risk function of the outcome is linear, each liter of alcohol consumed produces the same amount of harm. In this case, the proportion of consumption accounted for by the non-survey population is equal to their proportion of alcohol-related mortality. We used this assumption with the directly alcohol attributable deaths in Finland among men and women combined in 2003 and arrived at the following tentative calculation. In the current situation, survey-based consumption estimates can be taken to assume that the non-survey population (1.4% of the whole population) consumes the same 5.2 litres of alcohol the survey population reports drinking per person, and hence cover approximately 0.7% of all litres of alcohol sold in Finland. If we instead assumed that they accounted for 4.3% of consumption (the same as for mortality), their share of all sold alcohol would increase by a factor of 6.4, and our coverage rate would improve by nearly 4%, from 45% to 49%. This improvement makes the unrealistic assumption that, had we been able to survey the non-survey population, we could have captured every alcohol litre consumed by them. If they only revealed half of their consumption like the rest of the population, their share of sold alcohol would increase from 0.7% to 2.1%, and being able to interview them would improve the coverage rate by nearly 2%, from 45% to 47%. If instead the risk function for alcohol attributable mortality was more convex, like is the case for liver cirrhosis [17], the calculation would have an infinite number of solutions, as the same mortality outcome could be produced with very different consumption distributions – the distribution in the non-survey population may well be e.g. bimodal [6]. However, even in this case the data indicates that the estimate of 2-4% improvement in coverage rates is a maximum value for Finland: as the relative risk among heavy drinkers is so much higher in the case of the convex risk function, any consumption distribution in the non-survey population that would produce the same (or higher) consumption estimate as was obtained with the linear risk function (producing the 2-4% estimate) would also produce a higher mortality rate than the observed one, i.e. would not be supported by the data. A concave risk function, on the other hand, is not very likely.

The results of this paper reproduced the finding of Herttua et al. [10] who reported considerable increases in alcohol-related mortality after the alcohol tax cuts in 2004. The current paper sought to find out if the non-survey sample could be the reason for the puzzle of Mustonen et al. [11] and Mäkelä et al. [12] who were unable to reproduce with survey data the increase found for per capita consumption and for alcohol-related mortality. Using similar calculations as above (i.e. assuming that the proportion of alcohol consumption accounted for by the group is the same as the proportion of directly alcohol attributable mortality, and assuming that one-half of consumption could be captured if the non-survey population had been surveyed) it could be estimated that under one-tenth of the gap in the change in consumption implied by the 10% increase in per capita consumption and the 6% decrease observed in the survey could be accounted for by the non-survey population. Therefore, the failure of the surveys to reproduce the results obtained with other types of data cannot be accounted for by the fact that the homeless and the institutionalized were excluded from the survey’s sampling frame. Instead, the problem lies either in the large group of non-respondents, which even increased from the first survey year to the following years, or alternatively there was an increase in the extent to which respondents underestimated their consumption.

In conclusion, the question of the reasons for under-coverage of surveys remains. The role of the non-survey population, when it is small like in the Finnish surveys, seems marginal. Hence, the effects of selective non-response and underestimation of consumption among respondents are the remaining explanations. As is evident in the review by Gmel and Rehm [7], studies so far have suggested a minor role for selective non-response. However, these results may be severely flawed, as they are typically based on data obtained when additional effort has been made to get a response from non-respondents. Those who do not respond despite the greater efforts have been shown to suffer from more alcohol-related harm than who could be persuaded to respond after all [5].

The history of surveys In Finland shed further light on the effect of non-response. The first drinking habit survey in Finland in 1968 had a response rate of 97% but yet a coverage rate of 39%. The response rate has steadily decreased in the series of surveys carried out every eight years, with a response rate of 74% in 2008, but there is no systematic decrease in the coverage rate. This suggests that selective non-response can only be a partial explanation at most, even if it has to be acknowledged that the role of e.g. concealing consumption may well have been different in 1968 than in 2008. The need for further illumination of the reasons for under-coverage of alcohol consumption in surveys remains a challenge for future research.