Skip to main content
NCBI home page
International Journal of Methods in Psychiatric Research logoLink to International Journal of Methods in Psychiatric Research
. 2012 Feb 16;21(1):17–28. doi: 10.1002/mpr.1345

Difficulties with telephone‐based surveys on alcohol consumption in high‐income countries: the Canadian example

Kevin D Shield 1,2,✉, Jürgen Rehm 1,2,3
PMCID: PMC3561771  NIHMSID: NIHMS352887  PMID: 22337654

Abstract

Accurate information concerning alcohol consumption level and patterns is vital to formulating public health policy. The objective of this paper is to critically assess the extent to which survey design, response rate and alcohol consumption coverage obtained in random digit dialling, telephone‐based surveys impact on conclusions about alcohol consumption and its patterns in the general population. Our analysis will be based on the Canadian Alcohol and Drug Use Monitoring Survey (CADUMS) 2008, a national survey intended to be representative of the general population. The conclusions of this paper are as follows: (1) ignoring people who are homeless, institutionalized and/or do not have a home phone may lead to an underestimation of the prevalence of alcohol consumption and related problems; (2) weighting of observations to population demographics may lead to a increase in the design effect, does not necessarily address the underlying selection bias, and may lead to overly influential observations; and (3) the accurate characterization of alcohol consumption patterns obtained by triangulating the data with the adult per capita consumption estimate is essential for comparative analyses and intervention planning especially when the alcohol coverage rate is low like in the CADUMS with 34%. Copyright © 2012 John Wiley & Sons, Ltd.

Keywords: alcohol, average volume of consumption, patterns of drinking, adult per capita consumption, survey, random digit dialling, bias

Introduction

Alcohol consumption accounts for a high level of burden of disease, mortality and social harm in both developing and developed countries (WHO, 2009; Rehm et al., 2009). Alcohol consumption has been previously shown to be associated with 230 International Classification of Diseases, 10th Revision (ICD‐10) codes for disease and injury (English et al., 1995; Rehm et al., 2003b; Rehm et al., 2010a), resulting in 4.5% of the disability‐adjusted life years lost globally in the year 2004 (Rehm et al., 2009). This impact of alcohol consumption on the global burden of disease and injury is only exceeded by the two risk factors “childhood underweight” and “unsafe sex” (WHO, 2009), and seems to have increased in recent years [for 2000 the estimate was 4.0% (WHO, 2002)]. The estimates of alcohol‐attributable burden are net numbers, already taking into consideration the beneficial effect of moderate consumption on ischemic diseases, especially ischemic heart disease, and diabetes (Rehm et al., 2003c; Baliunas et al., 2009).

Beyond light to moderate consumption, the risk of mortality or morbidity from alcohol‐related diseases increases as consumption increases (Rehm et al., 2010a). Furthermore, both the average volume of alcohol intake and the drinking pattern are associated with the risk of mortality and morbidity from alcohol‐attributable disease and injury (Rehm et al., 2010a). Thus, it is important to accurately measure alcohol intake and drinking patterns for accurate estimates of alcohol‐attributable hospitalization or mortality, both important for public health and health care planning.

Usually, alcohol intake as a basis for planning is accessed via surveys. Many countries, especially high income countries, or regions such as the European Union conduct regular alcohol surveys (Rehm et al., 2003a; http://ec.europa.eu/health/eurobarometers/index_en.htm), usually by telephone. Telephone‐based survey designs are susceptible to multiple sources of bias, the first of which is that they typically do not capture those individuals without a land line phone in the household, which would include, but is not limited to, the homeless, people who are institutionalized, and those who have only a mobile phone (Health Canada, 2009). These individuals who are not captured by a survey may have a different drinking pattern than those who are included in the survey. Additionally, the homeless are more likely to drink surrogate alcohols, a fact that is not captured by nation‐wide surveys or per capita consumption estimates (Canadian Institute for Health Information, 2007). Surrogate alcohols may also contain ingredients other than alcohol associated with poisonings and other toxic effects (Lachenmeier et al., 2007). The described inclusion bias in the results of the survey is attributable to the survey design, and causes the extrapolation of the survey results that do not accurately reflect total alcohol consumption levels and patterns in Canada.

Additionally, there is the potential for participation bias depending on the number of people who answer the survey. In general, response rates vary by socio‐economic status, marital status, gender, and age (Boeing et al., 1999; Boström et al., 1993; Bergstrand et al., 1983; Rodes et al., 1990; Etter and Perneger, 1997; Eastwood et al., 1996). In addition, participants in surveys often report better health‐related behaviours than do non‐respondents (Boeing et al., 1999; Boström et al., 1993; Jacobsen and Thelle, 1998; Janzon et al., 1986; Macera et al., 1990; Hill et al., 1997; O'Neill et al., 1995; Criqui, 1979). If response bias is present for key demographics in a survey, observations may be weighted in order that the observed demographics resemble the demographics of the population; this can be achieved by triangulating survey information with census information (Kish, 1992). Even when observations are weighted there still may be differences in key demographics between groups (Health Canada, 2009). Weighting of observations may also result in a particular observation having a much greater influence than other observations thereby leading to chance findings (Kish, 1992). In fact, weighted results, especially if based on very few observations, may actually be more biased than the unweighted results. Consider the following example: in the age group of young men of lower socio‐economic status, the response rates are usually low, and each respondent is weighted multiple times. In a situation where the behaviour of a respondent in this group is an exception to the behaviour of the group as a whole and is associated with the respondent's probability of participation in the survey, the behaviour which is not representative of the group will be magnified leading to biased population and group estimates.

Overall low response rates may also affect the generalizability of a survey, if those who do not respond differ greatly from responders with respect to key demographics (Churchill, 1995; Lin and Schaeffer, 1995). Non‐response may be a particular problem for alcohol population surveys, as they may impact on so‐called coverage rates, i.e. the proportion of true alcohol consumption which is covered by the answers in surveys. True consumption here is defined by sales/taxation and/or by production and export/import (Rehm et al., 2003a). Most surveys do not cover true consumption, and coverage rates of 40 to 70% are the rule rather than the exception (Midanik, 1982; Rehm et al., 2007). Undercoverage when uncorrected may lead to incorrect population estimates and incorrect association between alcohol consumption and important outcomes if weights are used in regression analysis (Caetano, 2001; Midanik, 1982).

In summary, there are three major potential sources of bias that occur with the use of random digit dialling telephone‐based surveys as a tool for measuring alcohol consumption:

  1. the methodology of the survey design and resulting biases,

  2. non‐response to the survey and response rates to specific questions in the survey,

  3. undercoverage of alcohol consumption.

The objective of this paper is to quantify the extent of these potential problems in a telephone‐based survey. This will be done by analysing the data from our telephone survey under different assumptions and comparing these data with other information on the Canadian population from sources outside the sampling frame of the example used here. What is more, we will examine population weights using different assumptions and will model the distribution of alcohol consumption on the Canadian Alcohol and Drug Use Monitoring Survey (CADUMS) 2008, a national survey intended to be representative of the general population, based on adult per capita data (Health Canada, 2008).

Methods

Data sources

Alcohol telephone survey

The data source used in this analysis is the CADUMS 2008. It was designed to be representative for Canada and used a provincially stratified two‐stage (telephone household, respondent) probability sampling performed in eight waves between April 2008 and December 2008. The survey was conducted using random digit dialling methods and computer‐assisted telephone interviewing. The sampling frame was defined by an electronic inventory of all active telephone area codes and exchanges in Canada. Within each household, respondents who were 15 years of age or older were eligible to complete the survey in either English or French. The eligible individual in the household with the closest next birthday to the interview date was asked to complete the survey. Households were contacted up to 12 times in order to obtain a participant.

Response rates were calculated according to L'Association de l'Industrie de la Recherche Marketing et Sociale by dividing the number of households that agreed to participate by the total number of households that were randomly selected and eligible for the survey (AIMS, 2010). Not included in those eligible to participate in the survey were individuals with language difficulties, those who were ill, people who were not of an eligible age, and those situations where the random digit dialling produced duplicate phone numbers.

Although the survey was stratified by province (and excluded Canadian territories), each province was further broken down for information purposes into two regions: (1) selected census metropolitan areas, and (2) areas outside of these census metropolitan areas. A posteriori weighting of observations was performed on the basis of age, sex, and region using population estimates obtained from the 2006 census in order to correct for selection bias. In total, 16,687 people were interviewed, of whom 47 were not included for analysis because information on sex and age was not available. Furthermore, 27 people did not provide a specific age inside one of the weighting age groups and were eliminated from our analysis.

Data for enumerating the homeless and institutionalized

The total Canadian population and the number of Canadians living in institutions were obtained for 2006 from Statistics Canada (2008, 2009). Specific data on the number of mental health separations, and average length of stay by hospital and reasons for admittances were obtained for 2005 from the Hospital Mental Health Database (Canadian Institute for Health Information, 2008). Data on the number of separations and average length of stay in acute care hospitals were obtained from the Canadian Institute for Health Information (2002).

It is necessary to determine housing situations and the frequency and duration of homeless episodes in order to quantify the effect of excluding the homeless population in the CADUMS 2008. Information regarding the demographics of the homeless in Canada is sparse; however, we roughly calculated the number of homeless in Canada by reviewing the demographics of the homeless who use shelters in the United States using data from Burt et al. (1999), and then by applying these data to the cross‐sectional estimate of people who use homeless shelters in Canada. In addition to this method, the effects of excluding homeless individuals on consumption estimates were calculated using the upper estimate of the number of homeless people according to media reports (Laird, 2007).

Main statistical analyses

Calculating the distribution of alcohol consumption in the CADUMS

Average (daily) drinking

Multiple measures were used to quantify alcohol consumption. The main measure of alcohol intake (in grams of pure ethanol per day) was calculated from questions on alcohol consumption asked separately for each of the past seven days. Specifically, participants were asked for the number of drinks they had consumed in each of the seven days before the survey, and this number of drinks was then converted into average grams of alcohol consumed by multiplying the number of standard drinks per day by the size of a Canadian standard drink (13.6 grams) (Canadian Centre on Substance Abuse, 2004). Alcohol consumption was alternatively measured as a categorical variable using quantity‐frequency measures as well as a frequency measure of five or more drinks per occasion (5+ measure). A missing data analysis was performed for those individuals who refused to respond to the question of how much they drank in the previous seven days; a χ 2 test compared the 5+ question responses for those with missing data with the responses from those individuals who answered the seven‐day consumption question.

Drinking status was described by three categories: (1) current drinkers (had at least one drink within the last past year), (2) former drinkers (had at least one drink within their lifetime but not within the past year), and (3) lifetime abstainers.

Heavy drinking

We assessed the coverage of heavy drinking in the CADUMS 2008 and compared the prevalence of two “heavy drinking” groups: (1) those who were drinking more than 60 grams of alcohol per day and (2) those drinking 40 or more grams of alcohol per day.

Modelling alcohol consumption through a gamma distribution

The unadjusted alcohol intake distribution was modelled for current drinkers using a gamma distribution. Given that it is unlikely that a person can consume more than 150 grams of alcohol per day over an extended period of time we capped the gamma distribution at 150 grams of alcohol per day. To account for this capping we normalized the gamma distribution so that, if integrated, the value of the integral between zero and 150 would equal one. If we were to calculate the shape (κ) and the scale (θ) of the gamma distribution using maximum likelihood methods, values of zero would lead to unsolvable equations. To solve for this problem, current drinkers who indicated that they did not drink in the week before the survey were set as having consumed 0.1 grams of alcohol per day (by definition current drinkers consumed alcohol in the past year so their weekly average for 2008 would have been greater than zero).

The coverage rate (after outliers were removed) for total recorded and unrecorded adult per capita consumption accounted for unrecorded alcohol consumption as well, estimated to be 19.5% of recorded consumption. Thus, we multiplied recorded adult per capita consumption by 1.195 (Macdonald et al., 1999). The mean alcohol intake was then shifted by multiplying the unadjusted mean by the reciprocal of the 90% of the coverage rate (calculated for both recorded, and recorded and unrecorded adult per capita consumption), allowing 10% for abstention (Rehm et al., 2010b).

Adjusting the gamma distribution for undercoverage

To model the upshifted distribution of alcohol consumption we used the methods outlined by Rehm et al. (2010b). To calculate the prevalence of heavy drinking, we used two upshifted distributions: (1) alcohol intake adjusted for recorded adult per capita consumption, and (2) alcohol intake adjusted for recorded and unrecorded adult per capita consumption. The methods proposed by Rehm and colleagues are based on an analysis of 1001 alcohol consumption distributions by sex and age from over 60 comparable surveys conducted around the world and empirically determined that the distribution of alcohol consumption was best modelled using a gamma distribution and that the standard deviation (SD) of this distribution could be estimated using the mean. Accordingly, we calculated the SD of the upshifted distribution by multiplying the upshifted mean by 1.258 for men and 1.171 for women (Rehm et al., 2010b). The shape and scale parameters used to describe the gamma distribution were calculated using the mean and the SD (σ), θ = σ 2/μ and κ = μ 2/σ 2. This methodology allowed for triangulation of these survey data with per capita consumption to be modelled by sex and age. When calculating the prevalence of heavy drinkers by integration, the upshifted distribution was capped at 150 grams per day and normalized accordingly. This method of up‐shifting the mean intake per day is dependent on the assumption that the estimated population prevalence of lifetime abstainers, former drinkers, and current drinkers are correct. Furthermore, it assumes a homogeneity in the undercoverage of all age and sex groups.

Examining the effects of weighting observations

To show the effects of weighting observations and influential observations on estimating the prevalence of heavy drinking the prevalence of heavy drinkers was calculated for an unweighted population estimate, a weighted population estimate, and a weighted population estimate with outliers removed. The percentage of heavy drinkers was calculated from modelling daily alcohol intake using a gamma distribution before and after shifting these alcohol coverage data upward based on the adult per capita consumption estimate. The percentages of heavy drinkers identified by each of the two definitions for both the adjusted (shifted) and unadjusted (not yet shifted) drinking estimates were also calculated by integrating the respective gamma distributions.

Sensitivity analyses were performed by removing outliers (those who reported having drunk 200 grams or more of alcohol per day). These analyses were only done using the weighted estimates of the prevalence of people who drank 40+ and 60+ grams, respectively.

Design effects (DEFF) were calculated according to Kish (1992). This method accounts for the effects of clustering, stratification, and weighting at the stratum level.

Estimating the effect of excluding Canadian sub‐populations

To estimate the adult alcohol consumption coverage of the CADUMS 2008 as if the homeless, institutionalized, and those without a home phone were included in the sampling frame, we enumerated these groups and estimated their expected daily alcohol intake based on the prevalence of alcohol use disorders and, where appropriate, conservative assumptions for each group.

For those with alcohol use disorders we assumed that men consumed on average 11.1 drinks per day and women consumed two thirds of this amount (Kremer, 2001). For those in acute care hospitals we assumed the prevalence of alcohol use disorders was 9.9% for men and 4.8% for women, which was calculated based on the prevalence of alcohol use disorders in acute care hospitals in Germany, correcting that estimate for the population prevalence of alcohol use disorders in Canada (Kremer, 2001; Rehm et al., 2005; Rush et al., 2008). Data on the prevalence of alcohol use disorders in prison populations were obtained for 2006 from Statistics Canada (2009). With respect to people in psychiatric hospitals we assumed that two thirds of those individuals who had a primary or secondary diagnosis of substance abuse had an alcohol use disorder. Estimates of alcohol use disorders in the homeless population were obtained from Kuhn and Culhane (1998).

To estimate daily consumption of those people without alcohol use disorders, we used the estimate of average daily alcohol consumption from the CADUMS 2008, correcting this latter estimate for the estimated prevalence of alcohol use disorders in Canada. Estimates of daily consumption were adjusted (1) for the homeless and institutionalized by increasing the estimates of those in the general population without alcohol use disorders by 25%, and (2) for those people without a home phone by increasing the estimates for the general population by 10%, since we assumed that the homeless, institutionalized, and those people without a home phone would have a higher prevalence of current drinkers, and for the current drinkers a higher daily consumption. For those without a home phone we also assumed that 60% of university and college students who live on campus would be included in this group.

To estimate the alcohol intake of students we used data on binge drinking obtained from the Canadian Campus Survey 2004 (Adlaf et al., 2005). This survey indicated that 26% of males and 11% of females drank eight drinks or more on at least one occasion every two weeks. We assumed that on an average binge day drinkers drank 10 drinks, and estimated that this occurred on average once per week. For students we assumed that on a non‐binge drinking day they drank 10% more than the general population did on a non‐binge drinking day.

All statistics were performed using R version 2.10.0.

Results

Selection bias and weighting

The overall response rate for the CADUMS 2008 survey was 43.5%. This response rate varied depending on the wave and the province of the survey. Of the numbers eligible for the CADUMS 2008 11.0% did not answer the 12 calls or were contacted fewer than 12 times without answering at the time the survey ended. No information about response rate was provided by sex, age, or region. The response rate varied between 32.6% for British Columbia in April 2008 and 57.3% for Quebec in December 2008. After a posteriori weighting for age, sex, and region was applied the CADUMS 2008 overestimated (1) the number of people who were married, and (2) those with a university degree, and underestimated (1) the number of people who were never married, and (2) those individuals with less than a high school education, when compared to 2006 census data.

Although the CADUMS 2008 was stratified by province, selection and response bias were present; more participants were men than were women, and participation was disproportionate according to age and region. To correct for these biases data from the CADUMS 2008 were weighted according to population estimates derived from the 2006 Canadian census. Population frequency weights in the CADUMS 2008 ranged from 103 to 28,370. This weighting of observations led to an increase in the DEFFs for the estimates of drinking status and quantitative frequency. The DEFFs pertaining to alcohol showed a particularly high value ranging from 1.69 to 11.47. These high DEFFs indicate heterogeneity in the weights of drinkers within categories. This can be explained in part by the weights. The minimum and maximum weights and the DEFFs for alcohol intake and current drinking status are found in Table 1. The weights showed a skewed right distribution with the mean being much lower than maximum weight.

Table 1.

Weights and DEFFs by age and sex

Gender Age Average weight Maximum weight Design effect (DEFF)
Alcohol intake Current drinker
Women 15–19 2610.0 21979.7 8.024 3.646
20–24 2826.0 21979.7 1.692 4.327
25–34 1482.0 12442.1 4.363 4.230
35–64 1194.0 12492.8 3.627 3.870
65+ 1105.0 7486.3 3.893 3.576
Total 1330.0 21979.7 2.708 4.219
Men 15–19 2767.0 22553.1 3.643 2.664
20–24 3636.0 22553.1 2.084 3.072
25–34 2264.0 28365.7 2.221 5.330
35–64 1757.0 12500.4 2.400 3.297
65+ 1462.0 11410.4 4.279 4.173
Total 1911.0 28365.7 11.472 3.722
Total 1560.0 28365.7 5.207 4.026
Open in a new tab

Alcohol coverage rates

The coverage of alcohol as measured by the CADUMS 2008 was 34% for recorded (27% for recorded and unrecorded together); however, the variable of consumption during the previous seven days had 4.49% missing observations. The individuals who had missing observations for quantitative frequency were more likely to state that they had four or more drinks per occasion per week (p < 0.05). Correcting for heavier drinking in the excluded groups would result in a coverage rate of 37% (30% for recorded) based on coverage rates without outliers removed.

The various estimates for proportions of people who drank more than 60 and 40 grams of alcohol per day are presented in Tables 2 and 3, respectively. The estimates vary considerably based on the assumptions. For Canada, we would estimate 13.93% of men and 3.94% of women drink more than 60 grams of alcohol per day if based on the triangulation of survey and per capita data, but only 2.17% of men and 0.24% of women, if based on the raw estimates of surveys. The differences are considerable. All statistics showed the largest percentage of heavy drinkers in men aged 20–24, with the second largest percentage being in the age group 15–19 while in women age groups 15–19 had the highest prevalence of heavy drinkers with the second largest being 20–24 except for the weighted analysis with outliers removed.

Table 2.

Proportions of heavy drinkers (60+)

Sex Age Calculated using unweighted observations Calculated using weighted observations Calculated using weighted observations with outliers removed Calculated using the gamma distribution that was:
Calculated based on unadjusted data Upshifted for recorded adult per capita consumption Upshifted for recorded and unrecorded adult per capita consumption
Women 15–19 0.82% 2.54% 0.26% 1.90% 11.27% 13.44%
20–24 1.40% 0.80% 0.80% 0.51% 3.95% 6.25%
25–34 0.37% 0.78% 0.78% 0.31% 2.47% 4.30%
35–64 0.14% 0.17% 0.17% 0.04% 0.96% 1.92%
65+ 0.10% 0.05% 0.05% 0.05% 0.78% 1.56%
Total 0.24% 0.47% 0.30% 0.16% 2.24% 3.94%
Men 15–19 2.75% 2.44% 2.44% 2.88% 9.85% 13.10%
20–24 6.45% 4.15% 4.15% 3.87% 14.00% 17.54%
25–34 3.91% 3.02% 2.42% 3.01% 13.43% 16.78%
35–64 1.82% 1.36% 1.09% 1.77% 10.06% 13.48%
65+ 0.87% 1.25% 1.25% 1.47% 7.84% 10.73%
Total 2.17% 1.91% 1.68% 2.18% 10.58% 13.93%
Total 0.99% 1.15% 0.95% 0.93% — —
Open in a new tab

Table 3.

Proportions of heavy drinkers (40+)

Sex Age Calculated using unweighted observations Calculated using weighted observations Calculated using weighted observations with outliers removed Calculated using the gamma distribution that was:
Calculated based on unadjusted data Upshifted for recorded adult per capita consumption Upshifted for recorded and unrecorded adult per capita consumption
Women 15–19 1.10% 2.74% 0.46% 4.24% 18.67% 21.07%
20–24 1.96% 0.94% 0.94% 1.81% 9.29% 12.84%
25–34 1.19% 1.77% 1.77% 1.25% 6.68% 9.83%
35–64 0.36% 0.36% 0.36% 0.29% 3.21% 5.19%
65+ 0.24% 0.11% 0.11% 0.30% 2.63% 4.27%
Total 0.54% 0.76% 0.59% 0.79% 6.14% 9.11%
Men 15–19 5.50% 5.06% 5.06% 5.85% 18.36% 22.37%
20–24 9.27% 9.07% 9.07% 7.93% 24.37% 28.44%
25–34 6.96% 6.37% 5.79% 6.74% 23.30% 27.14%
35–64 4.07% 4.04% 3.79% 4.63% 18.93% 23.19%
65+ 2.88% 3.70% 3.70% 3.86% 15.14% 18.84%
Total 4.54% 4.82% 4.60% 5.26% 19.49% 23.59%
Total 2.09% 2.69% 2.49% 2.78% — —
Open in a new tab

Enumerating and describing the homeless and institutionalized

By design, the CADUMS 2008 left out those people who were homeless, institutionalized, and those individuals without a home phone. Overall, an estimated 8.00% of Canadians had only a mobile phone in 2008, 1.53% of the Canadian population was institutionalized, and 0.45–0.90% of Canadians were homeless (Kuhn and Culhane, 1998; Kunic and Grant, 2006; Canadian Institute for Health Information, 2008). Estimates of the number of people in Canada who were homeless, institutionalized, or did not have a home phone, together with their estimated alcohol consumption are provided in Table 4.

Table 4.

Percentage of Canadians by category

Population group Number of Canadiansa Percentage of Canadians Prevalence of alcohol use disorders
Homeless Estimate 1: back calculated from shelter bed usea 155,286 0.49% 25.91%
Transitional 115,904 0.37% 20.8%
Episodic 16,731 0.05% 33.7%
Chronic (sheltered) 14,194 0.04% 46.3%
Chronic (unsheltered) 8,457 0.03% 46.3%
Estimate 2: upper limit as reported in press 300,000 0.95% 25.91%
Institutionalized Total 579,496 1.83% —
Health care and related institutionsb 377,435 1.19% —
Acute care hospitals 34,801 0.11% 6.7%
Psychiatric hospitals 5,329 0.02% 12.3%
Correctional and penal institutionsc 37,181 0.12% 43.7%
Federal 12,935 0.04% —
Youth 2,006 0.01% —
Provincial 22,240 0.07% —
Group homes for children and youthb 3,710 0.01% —
Service collective dwellingsb 53,945 0.17% —
Religious establishmentsb 15,840 0.05% —
Military bases and otherb 14,075 0.04% —
No home phone 2,529,032 8.00% —
Students living on campus with no home phone 70,534 0.22% —
Total 3,351,687–3,496,401 10.60–11.06% —
Open in a new tab
a

Back calculated based on shelter bed use in 2006 (Statistics Canada, 2008) using shelter demographics as reported by Kuhn and Culhane (1998).

b

As reported in the 2006 Census (Statistics Canada, 2008).

c

Number of people in correctional institutions in 2006 (Statistics Canada, 2009).

The prevalence of alcohol use disorders in the homeless and institutionalized populations is relatively high compared to the prevalence of such disorders in Canadians who responded to the CADUMS 2008. Of the Canadian homeless population, we estimated a prevalence of 25.9% with alcohol use disorders (20.8% transitional, 33.7% episodic and 46.3% chronically sheltered and unsheltered). Of those individuals who were admitted into psychiatric hospitals due to mental illness‐related reasons, 34.56% had a primary or secondary diagnosis of a substance abuse disorder. In a study performed on a similar population in south London, England, it was observed that 31.6% (20.0% women and 38.7% men) of the individuals who had contact with mental health services had an alcohol abuse problem as measured by the Alcohol Screening Questionnaire which was a variant of the Michigan Alcohol Screening Test (Menzes et al., 1996). Of those individuals in correctional institutions in 2006, 43.7% were found to have alcohol abuse problems defined as a score above 13 on the Alcohol Dependence Scale (Skinner and Horn, 1984). It is important to note that although the homeless and institutionalized estimates are categorized separately, there is overlap between the categories. In 2005 52% of the 3596 hospitalizations of the homeless were due to mental illness (Canadian Institute for Health Information, 2008).

Discussion

Telephone‐based surveys have multiple problems that stem from their design, sampling, and response rates as indicated by the impact of the assumptions surveys make about applying population weights, the use of sampling frames that exclude part of the population, and assumptions about the accuracy of the information obtained. Furthermore, telephone‐based surveys that ask questions about alcohol drinking habits are particularly affected by the assumptions made about population coverage, response bias and non‐response bias. The effect of these assumptions is reflected in the under‐coverage of alcohol consumption when compared to the per capita estimates. To correct for these assumptions which lead to lack of representivity of the survey, a posteriori weighting may be used but may result in influential observations if the variables chosen to weight by and the survey sampling design are not both chosen carefully.

The response rate for the CADUMS 2008 survey was relatively low with an overall response rate of 43.5%. Although this is a relatively low response rate, it has been previously demonstrated that higher response rates do not necessarily change the resulting distribution of alcohol consumption when the response rate varies between 30% and 60% and similar sampling methods are used (Gmel and Rehm, 2004). Previous research has also shown that those who respond later to surveys and are hard to reach by telephone or mail survey methods have a higher daily alcohol intake than those who are easy to reach (Zhao et al., 2009; Van Loon et al., 2003). Additionally, when the response rate is high (i.e. over 90%) we can assume that those who are hard to reach are included in this sample and that the distribution will not be as greatly affected by non‐response bias. To deal with differential response rates, the observations were weighted by information on the population obtained from the 2006 census. However, no adjustment was made for the differential response rate by education or marital status. Although these differential response rates are common to telephone‐based surveys, there was a difference in alcohol consumption by marital status and/or education leading to an uncorrected selection bias as observed with the CADUMS 2008 (Trewin and Lee, 1988; Health Canada, 2009). Therefore, weighting by age and sex may not correct entirely for the underlying non‐response bias and selection in alcohol consumption surveys.

Adult per capita consumption estimates of recorded alcohol do not include the consumption of surrogate alcohols, which are defined as alcohols not intended for human consumption, illegally produced, or homemade (Lachenmeier et al., 2007; Macdonald et al., 1999). Data concerning surrogate and unrecorded alcohol use are currently scarce, but such use is estimated to account for 12% to 19.5% of the total per capita alcohol consumption in Canada and is as high as 74.1% in India (Lachenmeier et al., 2007). Ignoring the consumption of surrogate alcohols and the individuals who drink them introduces error into the consumption estimates.

As a result of selection bias, non‐response bias, response bias and excluding the homeless, the institutionalized and those without a home phone, the adult consumption estimate derived from the CADUMS 2008 underestimates the amount of alcohol consumed when compared to the adult per capita consumption estimate. This was observed in our analysis where it was determined that the alcohol intake measure had a coverage rate of 33.95%. This coverage rate is not unusual for Canadian alcohol surveys; the previous Canadian alcohol survey had a coverage rate of 30% to 40% (Canadian Centre on Substance Abuse, 2004). Survey information available by sex, age, and sometimes race generally has an alcohol coverage rate (alcohol consumption) of between 40% and 70%, with outliers below 25% and above 100% as demonstrated by recent nationally representative surveys (Rehm et al., 2007). The magnitude of the coverage rate is dependent on the measure(s) used in the survey (Rehm et al., 2006). In addition, the number of heavy drinkers according to the CADUMS 2008 was underestimated when not adjusted by the adult per capita consumption estimate. More research is necessary to determine the sources of undercoverage in surveys such as the CADUMS. Currently, surveys on alcohol consumption are published with relatively little regard to concerns of the earlier‐mentioned systematic biases (Rehm, 2010).

The estimation of coverage rates for both alcohol consumption and alcohol use disorders leads to two problems: (1) the exposure information about alcohol consumption and use disorders is incorrectly estimated, and (2) inter‐ and intra‐population comparisons for alcohol use and consumption are not possible as the degree of underestimation or overestimation will vary depending on the survey, the population, and the time of administration of the survey (Rehm et al., 2007). By up‐shifting the distribution of alcohol consumption, the number of people who drink more than a certain amount increases in an exponential fashion. This was illustrated in our results by the number of people who drank more than 60 grams of alcohol per day proportionately increasing more than the mean intake per day when corrected for undercoverage. Most chronic and infectious diseases associated with alcohol show an exponential risk relationship with alcohol when on an unlogarithmized scale. Accordingly, for a small increase in heavy alcohol consumption, there is a rapid increase in the risk of developing a disease. Given the previously noted risk relationship, if surveys do not take into account undercoverage, estimates of population‐attributable fractions of chronic and infectious diseases, and of injuries, will be greatly underestimated. Information on the patterns of consumption is important for estimating morbidity and mortality associated with alcohol consumption, as well as for formulating public health policies and observing their effects over time (Rehm et al., 2006; Taylor et al., 2008). Thus, national survey information on drinking patterns, which identifies age and sex, should always be triangulated with data concerning adult per capita consumption to correct for undercoverage.

Disproportionate a posteriori weights were used in the CADUMS 2008 to correct for its unrepresentative sample caused by selection bias, non‐response bias, and exclusion of population groups by design. In our study, we also found a large degree of heterogeneity in the weights, as reflected in our DEFF estimates. This heterogeneity in the weights is a concern. The application of weights also created highly weighted outliers within the groups of women aged 15–19 and males aged 24–34. If not eliminated from the data analysis, these individuals would have led to an overestimation of the quantitative frequency of alcohol intake when measured by a gamma distribution and corrected for the adult per capita consumption estimate. If data without outliers eliminated are used to calculate a population estimate of morbidity and mortality attributable to alcohol use in these age groups, the results will be exaggerated.

Although the weights for age and region in our study restore the representative nature of the sample, they do not correct for the selection bias that led to the need to weight these observations in the first place since after weighting there was still a biased representation of people who were married and who had achieved a higher education. Both higher education and marriage are associated with positive health behaviours and, thus, we would expect our sample to systematically under‐represent alcohol consumption in Canada (Bergstrand et al., 1983; Reijneveld and Stonks, 1999).

Apart from the negative aspects of survey weights, survey weights are necessary when surveys oversample from marginalized populations. If data on missing populations are available, survey weights can be used to restore representivity of an excluded group by weighting a survey based on propensity score methods. Additionally, when information is available on the selection of participants it can be used to create design weights which can be further triangulated with population expansion weights. For example, if it had been available, information regarding the number of persons in each household surveyed by the CADUMS 2008 could have been used to create a better weighting system since the probability of selection changes depending on household size.

Given the need for weights when surveys do not sample proportionate to size, and the weights' potential to represent populations excluded from the sampling frame, more thought should be directed towards survey design and the variables used for weighting to avoid the creation of overly influential observations.

Telephone‐based surveys using random digit dialling are problematic in terms of population coverage since those individuals without a home phone, including residents of institutions, the homeless, and people with only mobile phones will not be captured. People who live in institutions, excluding people who live in senior homes, service collective dwellings, religious institutions, live on military bases and other institutions, are on average more likely to consume higher levels of alcohol when compared to the general population as these people are more likely to have alcohol abuse problems (Rossi, 1989; Gmel and Rehm, 2004). In our research we found that the per capita consumption coverage was not greatly affected by the exclusion of these groups; however, the exclusion of these groups leads to an underestimation of heavy drinkers as these populations have a much greater prevalence of alcohol use disorders. Thus, we hypothesized that the majority of undercoverage of alcohol consumption is due to non‐response and response bias.

For our research, we assumed that Canadians without home phones have a larger proportion of current drinkers, and that current drinkers consume more alcohol on average, leading us to hypothesize a 10% increase in their average alcohol consumption compared to the general population. This assumption is based on theory alone, and additional research is required to determine if the 8.00% of Canadians who do not have a home phone differ in alcohol consumption patterns from those individuals who do have a home phone as this first described group represents a relatively large percentage of the population.

Overall, national telephone‐based surveys have major design difficulties and problems with determining the coverage of alcohol consumption and alcohol abuse. Triangulating consumption information using the adult per capita consumption estimates and weights to restore the representative nature can rectify some of these problems when obtaining population estimates. However, weighting leads to an increase in the DEFFs, may lead to further problems with overly influential observations, and in some cases does not correct completely for underlying selection bias. Furthermore, survey data on alcohol consumption and abuse do not address the issue of excluding the homeless or institutionalized individuals, nor do such data address the use of surrogate alcohols. To account for the exclusion of populations, new means of analysis are needed, such as weighting by propensity score methods. Failure to take into account the methodological problems of common telephone‐based surveys often leads to an underestimation of the societal impact of alcohol and subsequently to the recommendation of insufficient public health policies.

Declaration of interest statement

The authors have no competing interests.

Acknowledgements

Financial support for this study was provided to the last author by the National Institute for Alcohol Abuse and Alcoholism (NIAAA) with contract # HHSN267200700041C to conduct the study titled “Alcohol‐ and Drug‐attributable Burden of Disease and Injury in the US”. In addition, the last author received a salary and infrastructure support from the Ontario Ministry of Health and Long‐Term Care. The authors would like to thank Benjamin Taylor for proof reading the article.

References

  1. Adlaf E.M., Demers A., Gilksman L. (2005) Canadian Campus Survey 2004, Centre for Addiction and Mental Health: Toronto, ON. [Google Scholar]
  2. AIMS . (2010) L'Association de l'Industrie de la Recherche Marketing et Sociale. Quantitative Norms. http://www.airms.net/en/quantitative_norms.php [11 April 2011]
  3. Baliunas D., Taylor B., Irving H., Roerecke M., Patra J., Mohapatra S., Rehm J. (2009) Alcohol as a risk factor for type 2 diabetes – a systematic review and meta‐analysis. Diabetes Care, 32(11), 2123–2132, DOI: 10.2337/dc09-0227 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Bergstrand R., Vedin A., Wiehelmsson C., Wilhelmsen L. (1983) Bias due to non‐participation and heterogenous sub‐groups in population surveys. Journal of Chronic Diseases, 36(10), 725–728. [DOI] [PubMed] [Google Scholar]
  5. Boeing H., Korfmann A., Bergmann M.M. (1999) Recruitment procedures of EPIC‐Germany. Annals of Nutrition & Metabolism, 43(4), 205–215. [DOI] [PubMed] [Google Scholar]
  6. Boström G., Hallqvist J., Haglund B.J.A., Romelsjö A., Svanström L., Diderichsen F. (1993) Socioeconomic differences in smoking in an urban Swedish population. Scandinavian Journal of Social Medicine, 21(2), 77–81. [DOI] [PubMed] [Google Scholar]
  7. Burt M., Aron L., Douglas T., Valente J., Lee E., Iwen B. (1999) Homelessness: Programs and the people they serve: Findings of the National Survey of Homeless Assistance Providers and Clients. http://www.urban.org/publications/310291.html [11 April 2011]
  8. Caetano R. (2001) Non‐response in alcohol and drug surveys: A research topic in need of further attention. Addiction, 96(11), 1541–1545, DOI: 10.1080/09652140120080679 [DOI] [PubMed] [Google Scholar]
  9. Canadian Centre on Substance Abuse . (2004) Canadian Addiction Survey 2004: Microdata eGuide, Ottawa, ON, Canadian Centre on Substance Abuse. [Google Scholar]
  10. Canadian Institute for Health Information . (2002) Tabular Reports. Hospital Mortality Data Base 2000/2001, Toronto, ON, Canadian Institute for Health Information. [Google Scholar]
  11. Canadian Institute for Health Information . (2007) Improving the Health of Canadians 2007–2008: Mental Health and the Homelessness, Ottawa, ON, Canadian Institute for Health Information. [Google Scholar]
  12. Canadian Institute for Health Information . (2008) Hospital Mental Health Services in Canada, 2005–2006, Ottawa, ON, Canadian Institute for Health Information. [Google Scholar]
  13. Churchill G.A. (1995) Marketing Research: Methodological Foundations, 6th edn, Forth Worth, TX, Dryden Press. [Google Scholar]
  14. Criqui M.H. (1979) Response bias and risk ratios in epidemiologic studies. American Journal of Epidemiology, 109(4), 394–399. [DOI] [PubMed] [Google Scholar]
  15. Eastwood B.J., Gregor R.D., Maclean D.R., Wolf H.K. (1996) Effects of recruitment strategy on response rates and risk factor profile in two cardiovascular surveys. International Journal of Epidemiology, 25(4), 763–769. [DOI] [PubMed] [Google Scholar]
  16. English D., Holman C., Milne E., Winter M., Hulse G., Codde G., Bower G., Corti B., de Klerk N., Knuiman M., Kurinczuk J., Lewin G., Ryan G. (1995) The Quantification of Drug Caused Morbidity and Mortality in Australia 1995, Canberra, Commonwealth Department of Human Services and Health. [Google Scholar]
  17. Etter J.F., Perneger T.V. (1997) Analysis of non‐response bias in a mailed health survey. Journal of Clinical Epidemiology, 50(10), 1123–1128. [DOI] [PubMed] [Google Scholar]
  18. Gmel G., Rehm J. (2004) Measuring alcohol consumption. Contemporary Drug Problems, 31(3), 467–540, DOI: 10.1111/j.1360-0443.2005.01224.x [DOI] [Google Scholar]
  19. Health Canada . (2008) Canadian Alcohol and Drug Use Monitoring Survey. http://www.hc-sc.gc.ca/hc-ps/drugs-drogues/cadums-esccad-eng.php [11 April 2010].
  20. Canada Health. (2009) Canadian Alcohol and Drug Use Monitoring Survey 2008: Microdata User Guide, Ottawa, ON, Health Canada. [Google Scholar]
  21. Hill A., Roberts J., Ewings P., Gunnell D. (1997) Non‐response bias in a life‐ style survey. Journal of Public Health Medicine, 19(2), 203–207. [DOI] [PubMed] [Google Scholar]
  22. Jacobsen B.K., Thelle D.S. (1998) The Tromsø Heart Study: Responders and non‐responders to a health questionnaire. Do they differ? Scandinavian Journal of Social Medicine, 16(2), 101–104. [DOI] [PubMed] [Google Scholar]
  23. Janzon L., Hanson B.S., Isacsson S.O., Lindell S.E., Steen B. (1986) Factors influencing participation in health surveys. Results from prospective population study ‘men born in 1914’ in Malmö. Sweden. Journal of Epidemiology and Community Health, 40(2), 174–177. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Kish L. (1992) Weighting for unequal pi. Journal of Official Statistics, 8(2), 183–200. [Google Scholar]
  25. Kremer G. (2001) Alkoholprobleme im Allgemeinkrankenhaus. Früherkennung und Kurzintervention bei Patientinnen und Patienten mit Alkoholproblemen in der somatischen Medizin, PhD Thesis, University of Bielefeld.
  26. Kuhn R., Culhane D.P. (1998) Applying cluster analysis to test of a typology of homelessness by pattern of shelter utilization: Results from the analysis of administrative data. American Journal of Community Psychology, 26(2), 207–232. [DOI] [PubMed] [Google Scholar]
  27. Kunic D., Grant B.A. (2006) The Computerized Assessment of Substance Abuse (CASA): Results from the Demonstration Project. Addictions Research Centre Research Branch Correctional Service of Canada. http://www.csc-scc.gc.ca/text/rsrch/reports/r173/r173-eng.shtml [11 April 2011]
  28. Lachenmeier D.W., Rehm J., Gmel G. (2007) Surrogate alcohol: What do we know and where do we go? Alcoholism, Clinical and Experimental Research, 31(10), 1613–1624, DOI: 10.1111/j.1530-0277.2007.00474.x [DOI] [PubMed] [Google Scholar]
  29. Laird G. (2007) Shelter – Homelessness in a Growth Economy: Canada's 21st Century Paradox, Calgary, AB, Sheldon Chumir Foundation for Ethics in Leadership. [Google Scholar]
  30. Lin I.F., Schaeffer N.C. (1995) Using survey participants to estimate the impact of nonparticipation. Public Opinion Quarterly, 59(2), 236–258. [Google Scholar]
  31. Macdonald S., Wells S., Giesbrecht N. (1999) Unrecorded alcohol consumption in Ontario, Canada: Estimation procedures and research implications. Alcohol and Drug Review, 18(1), 21–29, DOI: 10.1080/09595239996725 [DOI] [Google Scholar]
  32. Macera C.A., Jackson K.L., Davis D.R., Kronfeld J.J., Blair S.N. (1990) Patterns of non‐response to a mail survey. Journal of Clinical Epidemiology, 43(12), 1427–1430. [DOI] [PubMed] [Google Scholar]
  33. Menzes P.R., Johnson S., Thornicroft G., Marshall J., Porsser D., Bebbington P., Kuipers E. (1996) Drug and alcohol problems among individuals with severe mental illness in south London. The British Journal of Psychiatry, 168(5), 612–619. [DOI] [PubMed] [Google Scholar]
  34. Midanik L.T. (1982) The validity of self‐reported alcohol consumption and alcohol problems: A literature review. British Journal of Addiction, 77(4), 357–382. [DOI] [PubMed] [Google Scholar]
  35. O'Neill T.W., Marsden D., Silman A.J. (1995) Differences in the characteristics of responders and non‐responders in a prevalence survey of vertebral osteoporosis. Osteoporosis International, 5(5), 327–334, DOI: 10.1007/BF01622254 [DOI] [PubMed] [Google Scholar]
  36. Rehm J. (2010) Commentary on Rey et al. (2010): How to improve estimates on alcohol‐attributable burden? Addiction, 105(6), 1030–1031, DOI: 10.1111/j.1360-0443.2010.02939.x [DOI] [PubMed] [Google Scholar]
  37. Rehm J., Baliunas D., Borges G.L.G., Graham K., Irving H.M., Kehoe T., Parry C.D., Patra J., Popova L., Poznyak V., Roerecke M., Room R., Samokhvalov A.V., Taylor B. (2010a) The relation between different dimensions of alcohol consumption and burden of disease – an overview. Addiction, 105(5), 817–843, DOI: 10.1111/j.1360-0443.2010.02899.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Rehm J., Kehoe T., Gmel G., Stinson F., Grant B., Gmel G. (2010b) Statistical modelling of volume of alcohol exposure for epidemiological studies of population health: The example of the US. Population Health Metrics, 8, 3, DOI: 10.1186/1478-7954-8-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Rehm J., Klotsche J., Patra J. (2007) Comparative quantification of alcohol exposure as risk factor for global burden of disease. International Journal of Methods in Psychiatric Research, 16(2), 66–76, DOI: 10.1002/mpr.204 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Rehm J., Mathers C., Popova S., Thavorncharoensap M., Teerawattananon Y., Patra J. (2009) Global burden of disease and injury and economic cost attributable to alcohol use and alcohol‐use disorders. Lancet, 373(9682), 2223–2233, DOI: 10.1016/S0140-6736(09)60746-7 [DOI] [PubMed] [Google Scholar]
  41. Rehm J., Patra J., Popova S. (2006) Alcohol‐attributable mortality and potential years of life lost in Canada 2001: Implications for prevention and policy. Addiction, 101(3), 373–384, DOI: 10.1111/j.1360-0443.2005.01338.x [DOI] [PubMed] [Google Scholar]
  42. Rehm J., Rehn N., Room R., Monteiro M., Gmel G., Jernigan D., Frick U. (2003a) The global distribution of average volume of alcohol consumption and patterns of drinking. European Addiction Research, 9(4), 147–156, DOI: 10.1159/000072221 [DOI] [PubMed] [Google Scholar]
  43. Rehm J., Room R., Graham K., Monteiro M., Gmel G., Sempos C. (2003b) The relationship of average volume of alcohol consumption and patterns of drinking to burden of disease – an overview. Addiction, 98(9), 1209–1228. [DOI] [PubMed] [Google Scholar]
  44. Rehm J., Room R., van den Brink W., Jacobi F. (2005) Alcohol use disorders in EU countries and Norway: An overview of the epidemiology. European Neuropsychopharmacology, 15(4), 377–388, DOI: 10.1016/j.euroneuro.2005.04.005 [DOI] [PubMed] [Google Scholar]
  45. Rehm J., Sempos C., Trevisan M. (2003c) Average volume of alcohol consumption, patterns of drinking and risk of coronary heart disease – a review. Journal of Cardiovascular Risk, 10(1), 15–20, DOI: 10.1097/01.hjr.0000051961.68260.30 [DOI] [PubMed] [Google Scholar]
  46. Reijneveld S.A., Stonks K. (1999) The impact of response bias on estimates of health care utilization in a metropolitan area: The use of administrative data. International Journal of Epidemiology, 25(6), 763–769. [DOI] [PubMed] [Google Scholar]
  47. Rodes A., Sans S., Balana L.L., Paluzie G., Aguilera R., Balaguer‐Vintro I. (1990) Recruitment methods and differences in early, late and non‐respondents in the first MONICA‐Catalonia population survey. Revue D'epidemiologie et de Sante Publique, 38(5‐6), 447–453. [PubMed] [Google Scholar]
  48. Rossi P.H. (1989) Down and Out in America: The Origins of Homelessness, Chicago, IL, University of Chicago Press. [Google Scholar]
  49. Rush B., Urbanoski K., Bassani D., Castel S., Wild T.C., Strike C., Kimberley D., Somers J. (2008) Prevalence of co‐occurring substance use and other mental disorders in the Canadian population. Canadian Journal of Psychiatry, 53(12), 800–809. [DOI] [PubMed] [Google Scholar]
  50. Skinner H.A., Horn J.L. (1984) Alcohol Dependence Scale (ADS): User's Guide, Toronto, ON, Addiction Research Foundation. [Google Scholar]
  51. Statistics Canada . (2008) Population in Collective Dwellings, by Province and Territory (2006). http://www40.statcan.ca/l01/cst01/famil62a-eng.htm [11 April 2011]
  52. Statistics Canada . (2009) Corrections Key Indicator Report for Adults and Young Offenders (KIR). http://www.statcan.gc.ca/cgi-bin/imdb/p2SV.pl?Function=getSurvey&SDDS=3313&lang=en&db=imdb&adm=8&dis=2 [11 April 2011]
  53. Taylor B., Rehm J., Room R., Patra J., Bondy S. (2008) Determination of lifetime injury mortality risk in Canada in 2002 by drinking amount per occasion and number of occasions. American Journal of Epidemiology, 168(10), 1119–1125, DOI: 10.1093/aje/kwn215 [DOI] [PubMed] [Google Scholar]
  54. Trewin D., Lee G. (1988) International comparisons of telephone coverage In Groves R., Biemer P., Lyberg L., Massey J.T., Nicholls W., Waksberg J. (eds) Telephone Survey Methodology, pp. 9–24, New York, John Wiley & Sons. [Google Scholar]
  55. Van Loon A.J., Tijhuis M., Picavet H.S., Surtees P.G., Ormel J. (2003) Survey non‐response in the Netherlands: Effects on prevalence estimates and associations. Annals of Epidemiology, 13(2), 105–110, DOI: 10.1016/S1047-2797(02)00257-0 [DOI] [PubMed] [Google Scholar]
  56. World Health Organization (WHO) . (2002) The World Health Report 2002: Reducing Risks, Promoting Healthy Life, Geneva, WHO. [Google Scholar]
  57. World Health Organization (WHO) . (2009) Global Health Risks. Mortality and Burden of Disease Attributable to Selected Major Risks, Geneva, WHO. [Google Scholar]
  58. Zhao J., Stockwell T., Macdonald S. (2009) Non‐response bias in alcohol and drug population surveys. Drug and Alcohol Review, 28(6), 648–657, DOI: 10.1111/j.1465-3362.2009.00077.x [DOI] [PubMed] [Google Scholar]

Articles from International Journal of Methods in Psychiatric Research are provided here courtesy of Wiley

RESOURCES