The impact of non-response weighting in health surveys for estimates on primary health care utilization

Abstract

Background

Low response rates in health surveys may affect the representativeness and generalizability of results if non-response is systematically related to the indicator of interest. To account for such potential bias, weighting procedures are widely used with an overall aim to obtain less biased estimates. The aim of this study was to assess the impact of applying calibrated weights on prevalence estimates of primary health care utilization among respondents compared to the entire sample of a representative Danish survey of adults aged ≥16 years.

Methods

Registry-based 1-year prevalence data on health care utilization of chiropractor/physiotherapist, dentist and psychologist in 2016 were linked to the entire sample (n = 312 349), including respondents (n = 183 372), from the Danish National Health Survey in 2017. Calibrated weights, which applied information on e.g. sex, age, ethnic background, education and overall health service use were used to assess their impact on prevalence estimates among respondents.

Results

Across all included types of health care, weighting for non-response decreased prevalence estimates among respondents, which resulted in less biased estimates. For example, the overall 1-year prevalence of chiropractor/physiotherapist, dentist and psychologist utilization decreased from 19.1% to 16.9%, 68.4% to 62.5% and 1.9% to 1.8%, respectively. The corresponding prevalence in the entire sample was 16.5%, 59.4% and 1.7%.

Conclusions

Applying calibrated weights to survey data to account for non-response reduces bias in primary health care utilization estimates. Future studies are needed to explore the possible impact of weighting on other health estimates.

Introduction

In recent decades, a tendency towards declining response rates in health surveys and other surveys has been observed in several countries,^1–3 and almost all surveys suffer from non-response.⁴ If non-response is missing completely at random, it may not limit the validity of survey data.^2,5 But if, on the other hand, the non-respondents deviate markedly from respondents, i.e. if non-response is systematically related to the indicator of interest, the representativeness and generalizability of the survey data are compromised, resulting in biased population estimates.^4,6–9

This may indeed be the case in health surveys, as it is well-known that differences in sociodemographic characteristics exist between respondents and non-respondents. Thus, the majority of previous research suggests that non-respondents are more likely to be e.g. young men, unmarried and less educated compared to respondents.^1,10,11 Moreover, non-respondents tend to exhibit a less healthy lifestyle.¹² This finding is also reflected by a higher cause-specific (i.e. smoking-, alcohol- and drug-related morbidity)¹³ and overall morbidity and mortality among non- respondents compared to respondents.¹⁴ In all, evidence exists to support the concern that non- response is not missing at random in health surveys, and that non-respondents may differ substantially from respondents in terms of both sociodemographic characteristics, lifestyle, morbidity and mortality.

To reduce the potential challenges related to differential non-response patterns in health surveys, weighting procedures are widely used with an overall aim to obtain less biased estimates. Calibration adjustment for non-response is a commonly used procedure where sampling weights are adjusted to reproduce known population totals on the basis of sociodemographic and socioeconomic variables that are under- or overrepresented among the respondents.¹⁵

The degree to which weighting procedures may reduce non-response bias can be assessed by applying non-response weights to data available for the entire population among both respondents and non-respondents of a survey. In Denmark, this is possible for registry data on e.g. health care utilization as these data are nationwide and, thus, cover all residents. By linking such information with data from the entire sample (i.e. both respondents and non-respondents) of a nationally representative health survey, an assessment of applying calibrated weights on registry-based prevalence estimates becomes possible.

Previous research have demonstrated differences between non-respondents and respondents in terms of health care utilization.^16–20 For example, Gundgaard et al.¹⁶ found that health care utilization was generally higher among non-respondents than among respondents, except for dentist use (which is not free of cost in Denmark as opposed to most other types of health care). This finding indicates that illness may negatively affect survey participation.^21,22 Although rather old, one study supports this tendency of a higher health care utilization among non-respondents,²⁰ whereas others find a lower utilization.^17–19 In conclusion, there is a risk that non-response may result in biased population estimates and conclusions.

To our knowledge, no previous study has yet investigated the impact of applying calibrated weights to survey data on estimates of primary health care utilization using data from a national administrative registry. Thus, the aim of this study was to examine the impact of applying calibrated weights to health survey data on the bias of estimates. This was examined by comparing weighted 1-year registry-based prevalence estimates of primary health care utilization (i.e. chiropractor/physiotherapist, dentist and psychologist) among respondents in the large Danish National Health Survey 2017 with the ‘true’ estimates obtained from the entire sample of the survey (i.e. all invited individuals), both overall and according to sociodemographic variables. The included types of primary health care were chosen as they were not included in the weighting procedure, which on the other hand was the case for e.g. visits to general practitioner, hospital admissions and outpatient hospital visits.

Methods

Survey data

Survey data for this study were derived from the Danish National Health Survey 2017.¹¹ The overall aim of the survey is to describe the status and trends in health and morbidity in the adult Danish population aged 16 years or older. This also includes monitoring of e.g. health behaviour, mental health and environmental health risks. The Danish National Health Survey is based on six mutually exclusive random subsamples, one from each of the five Danish regions and one national sample, and has been carried out in 2010, 2013, 2017, and 2021. In 2017, a total of 312 349 individuals aged 16 years or older were randomly selected using the Danish Civil Registration System and invited to participate in the survey (every individual with a permanent residence in Denmark has a unique personal identification number, i.e. a CPR number).²³ Each regional sample was based on random samples of individuals from each municipality in the region, and the minimum sample size of each municipality was 2000 individuals or half of the adult population in municipalities with <4000 citizens eligible for study participation,¹¹ however with exceptions for the two largest municipalities in Denmark, i.e. Copenhagen and Aarhus.

Data were collected using a concurrent mixed-mode approach of self-administration, allowing for the invited individuals to complete either web questionnaire or an identical paper-and-pencil questionnaire. Introduction letters and questionnaires were distributed by both postal mail (around 10% of the sample) and the secure electronical mail service, Digital Post (around 90% of the sample). A total of 183 372 individuals completed the questionnaire, yielding a response rate of 58.7%. The study design and data collection procedure in the Danish National Health Survey are described in detail elsewhere.¹¹

Registry-based data

Data on primary health care utilization for all invited individuals were derived from The Danish National Health Service Register.²⁴ According to Danish law,²⁵ all citizens residing in Denmark are entitled to free medical care (general practitioner, specialist physician) as well as subsidies to selected health services offered by specified health care providers, including chiropractor, physiotherapist, dentist and psychologist. From the Danish National Health Service Register, data on primary health care utilization of chiropractor, physiotherapist, dentist and psychologist, respectively, in 2016 were used. In this study, the 1-year prevalence of primary health care utilization was defined as the use of at least one health service registered in The Danish National Health Service Register in 2016 among the described primary health care providers.

CPR numbers of the entire sample, including respondents, from the Danish National Health Survey 2017 were linked on an individual level to data on primary health care utilization from the Danish National Health Service Register.²⁴ Information on various sociodemographic variables, including sex, age, marital status and ethnic background were derived from the Danish Civil Registration System.²³ Information on the highest completed educational level were derived from the Danish Education Registers²⁶ and was categorized as: ‘Basic school’, ‘Upper secondary or vocational education’, ‘Higher education’ or ‘No information’. Ethnic background was classified according to information on the individual’s citizenship, country of birth and parental country of birth and divided into three groups. Statistics Denmark’s definition was used to define which countries were considered Western or non-Western. According to this definition, the following countries were considered Western countries: The European Union Member States, UK, Andorra, Iceland, Liechtenstein, Monaco, Norway, San Marino, Switzerland, Vatican City, Canada, USA, Australia and New Zealand. All other countries are defined as non-Western countries. The following classification was applied: if both parents’ birthplaces were known, the country of origin was based on the birthplace of the mother, except if the father was born in Denmark, the mother was born abroad, and the respondent was a Danish citizen. If so, the respondent was categorized as having a Danish background. If birthplace information were available for only one parent, that parent’s birthplace was used. When information for both parents was missing, the country of origin was defined from the respondent’s birthplace. If the respondent’s birthplace also was unknown, the country of origin was determined from the respondent’s citizenship.

Weighting

Design weights were constructed based on the number of eligible individuals in each of the 98 municipalities to account for unequal sampling probabilities. In Denmark, the number of citizens ranges from 1793 in the smallest municipality to 602 481 in the largest in the first quarter of 2017. As suggested by Pfeffermann,²⁷ the design weights were calculated as the reciprocals of the sample inclusion probabilities and were then scaled such that their sum equals the net sample size and the mean equals one. The use of design weights was used to correct for the unequal sampling probabilities of the respondents and the entire sample, respectively. Accordingly, an individual with small probability of being selected is viewed as representing more individuals than an individual with a large selection probability.²⁷

As reported by Christensen et al.,¹¹ calibrated weights for the Danish National Health Survey 2017 were constructed by means of the generalized regression estimator. In this study, this method relies on auxiliary information from Statistics Denmark’s registers and takes account both the study design, i.e. unequal sampling probabilities as described above, and differential non-response across the sample.^28,29 The information used to compute the weights included sex, age, ethnic background, municipality of residence, highest completed level of education, socioeconomic status, household status, income, hospital admissions and visits to general practitioner. Weighting for non-response implies that a weighting value is calculated for each respondent in a survey, and that this value indicates how much the respondent’s response will count in a statistical procedure (39). Accordingly, individuals more likely to respond were given a lower weight. Weighting values are often expressed as a fraction with an overall mean value of 1.0, which is always positive, and the sum of the weighting values usually equals the sample size. To illustrate, a respondent with a weighting value of 2.0 means that his or her response is counted two times, whereas a response from a respondent with a weighting value of 0.5 is half a count.³⁰

Statistical analysis

Descriptive statistics (percentages) were used to describe the sociodemographic characteristics of respondents and the entire sample from the Danish National Health Survey 2017, respectively. To assess the possible impact of applying calibrated weights on the 1-year registry-based prevalence estimates of primary health care utilization in 2016 in this study, we stratified the analyses into different weighting schemes and by various sociodemographic variables. Accordingly, percentages [weighted and with 95% confidence intervals (95% CIs)] among respondents were weighted by the two following schemes: (i) weighted for design (i.e. unequal sampling probabilities), and (ii) weighted for design and non-response (i.e. unequal sampling probabilities and non-response). Percentages for the entire sample were weighted for design to account to unequal sampling probabilities.

Primary health care utilization was described for the use of chiropractor, physiotherapist (collapsed into one category due to similarities in symptoms and treatment, i.e. chiropractor/physiotherapy) and psychologist among individuals aged 16 years or older in the year prior to data collection (i.e. 2016), whereas individuals aged 18 years or older were included for dentist utilization as dental services are free of charge in Denmark among children and adolescents below this age. The included primary health care providers were chosen, as they were not included in the weighting procedure, which else would have interfered with the interpretation of the results. Analyses were performed in SAS, version 9.4.

Results

Table 1 shows the sociodemographic characteristics of the respondents, according to weighting schemes, and the entire sample of the Danish National Health Survey 2017, respectively. All estimates on sociodemographic characteristics among respondents weighted for design and non- response are very similar to those among the entire sample weighted for design. This finding was both expected and intended, as the weighting procedure specifically aimed at generating a ‘sociodemographic profile’ of the respondents as similar as possible to that of the entire sample.

Table 1.

Characteristics of the respondents according to weighting scheme, and the entire sample, respectively, in the Danish National Health Survey 2017 (number of individuals and percentage)

	Respondents (n = 183 372)				Entire sample (n = 312 349)
	n	%a	%b	%c	N	%a	%b
Sex
Men	84 607	46.1	46.1	49.4	154 468	49.5	49.4
Women	98 765	53.9	53.9	50.6	157 881	50.6	50.6
Age
16–24 years	18 913	10.3	11.1	14.3	41 804	13.4	14.1
25–44 years	44 121	24.1	25.4	30.4	89 641	28.7	29.9
45–64 years	67 221	36.7	35.9	32.1	103 357	33.1	32.3
≥65 years	53 117	29.0	27.6	23.3	77 547	24.8	23.7
Educational leveld
Basic school	34 311	20.9	20.2	23.0	63 789	23.6	22.9
Primary/vocational education	69 098	42.0	41.2	39.2	109 769	40.6	39.9
Higher education	56 561	34.4	35.9	32.2	82 342	30.4	31.8
No information	4489	2.7	2.7	5.6	14 645	5.4	5.4
Marital scheme
Married	100 168	54.6	53.1	46.0	148 087	47.4	45.9
Divorced	20 655	11.3	11.1	11.4	35 012	11.2	11.1
Widowed	12 067	6.6	6.2	6.5	20 553	6.6	6.3
Unmarried	50 482	27.5	29.5	36.0	108 697	34.8	36.7
Ethnic background
Danish	168 932	92.1	91.9	86.7	272 290	87.2	86.9
Western	6125	3.3	3.4	5.5	15 072	4.8	5.0
Non-Western	8315	4.5	4.7	7.8	24 987	8.0	8.1

^aUnweighted data.

^bData weighted for design.

^cData weighted for design and non-response.

^dAnalyses restricted to individuals aged ≥25 years.

Estimates of primary health care utilization in 2016 according to weighting scheme among respondents and the entire sample, respectively, are displayed in figures 1–3, according to various sociodemographic variables.

Figure 1.

Use of ‘chiropractor/physiotherapist’ in 2016 among respondents and the entire sample aged 16 years or older, respectively, according to weighting scheme. Percentage and 95% CIs

The tendency across all three figures was that weighting decreased registry-based prevalence estimates of primary health care utilization among respondents, which resulted in less biased estimates compared to those of the entire sample. For example, the overall estimate of the use of chiropractor/physiotherapist during the past year among respondents decreased from 19.1% (95% CI: 18.9; 19.3) to 16.9% (95% CI: 16.7; 17.1) when weighted for non-response. Among the entire sample, the prevalence was 16.5%.

For the purpose of result description in the following section, we consider estimates weighted for non-response among respondents as ‘acceptably non-biased’ in those cases, in which the prevalence in the entire sample lies within the 95% CI for the corresponding prevalence estimate among respondents.

For ‘chiropractor/physiotherapist’ utilization (figure 1), acceptably non-biased estimates were obtained among male respondents overall, including three of four age groups, excluding the oldest age group (≥65 years). Among female respondents, only estimates in two of four age groups (16–24 years and 45–64 years) were acceptably non-biased. Moreover, acceptably non-biased estimates were obtained for respondents with a higher educational level and those with a non-Western background. For ‘dentist’ utilization (figure 2), acceptably non-biased estimates were observed only for respondents with a higher education. For ‘psychologist’ utilization (figure 3), all estimates were acceptably non-biased except for those for respondents with basic school.

Figure 2.

Use of ‘dentist’ in 2016 among respondents and the entire sample aged 18 years or older, respectively, according to weighting scheme. Percentage and 95% CIs

Figure 3.

Use of ‘psychologist’ in 2016 among respondents and the entire sample aged 16 years or older, respectively, according to weighting scheme. Percentage and 95% CIs

Discussion

In this study, we explored the impact of applying calibrated weights to the 1-year prevalence estimates of three types of primary health care utilization among the entire sample, including respondents, of a large nationwide health survey in Denmark. The weighted registry-based prevalence estimates among respondents were compared with the corresponding 1-year registry-based prevalence estimates for the entire sample. In this way, it was possible to assess the degree to which the weighting procedure resulted in less biased estimates among respondents compared to the ‘true values’ in the entire sample. To our knowledge, no previous study has yet examined this issue in relation to primary health care utilization.

A key finding from our study was that, across all included types of primary health care and sociodemographic variables, applying calibrated weights to registry-based prevalence estimates from respondents generated less biased estimates than if left unweighted. However, weighting did not result in estimates equal to the ‘true values’ in the entire sample and, thus, did not eliminate non- response bias. The tendency was that the weighted registry-based prevalence estimates among respondents were higher than the ‘true values’, i.e. an overestimation, however, with effect sizes varying between types of primary health care. Overall, the most biased estimates were found for dentist use. Differences in both referral procedures, reduced payments and self-payments across the types of health care included in this study may both partly compromise comparability and explain the differential effects on weighting on registry-based prevalence estimates. For example, individuals receiving social benefits can apply for reduced payment for dental care and may receive a subsidy to cover up to 100% of their expenses.³¹ Also, a large Danish health insurance association automatically provides subsidies for healthcare services, including dental care and visits to chiropractor, physiotherapist and psychologist.³² However, the size of subsidies and types of healthcare services differ by insurance group, with individuals holding more expensive memberships receiving larger subsidies for more services. Moreover, reduced payments visits to a psychologist can be provided in certain specified cases and only by referral from a general practitioner.³³ Information how these different factors are distributed across both types of health care and subgroups is not accessible from registries and, thus, not included in the weighting procedure in this study and may potentially explain some of the varying effects.

When stratified by sociodemographic characteristics, results across the three types of primary health care showed a tendency towards more biased weighting effects among respondents holding an education with ‘No information’ and with a ‘Western’ background. Thus, it seems as if the weighting procedure was not able to reduce non-response bias in these groups to a degree comparable to other sociodemographic groups. A possible explanation for this finding may be that these groups are both rather small and more heterogenous than other groups, which may decrease the weighting effect on estimates. Thus, these groups may cover individuals with different resources, which enable them to adapt to the Danish society, including the healthcare system, heterogeneously. As a result of such heterogeneity, non-response weighting may not correct estimates as well in these subgroups as in more homogeneous subgroups.

As mentioned earlier, no previous study has investigated the impact of applying calibrated weights to registry-based prevalence estimates of primary health care utilization among respondents from a health survey. Other studies have, however, explored differences in primary health care utilization among respondents and non-respondents, but with conflicting results, especially across various types of health care. Thus, some studies find a higher primary health care utilization among non-respondents/the entire sample than respondents,^16,20 others find a lower use,^16–19 while one study finds no difference.¹⁶

The Danish study by Gundgaard et al.¹⁶ suggests that, overall, respondents from a health interview survey and a telephone interview survey, respectively, use less health care than both non-respondents and the entire sample. However, the opposite pattern is seen for dentist use, which was higher among respondents. This finding is partly in contrast to our results, as we found higher registry-based prevalence estimates of primary health care utilization during the past year among respondents than among the entire sample across all included types of health care. Our results are, however, not directly comparable to those by Gundgaard et al.,¹⁶ as we looked at the prevalence of at least one visit to each of the different types of primary health care, whereas Gundgaard et al.¹⁶ explored health care costs in 1 year. Two other studies also find a higher dentist utilization among respondents.^17,18 Again, these studies are not directly comparable to those in this study, e.g. as only individuals covered by a specified health insurance were included. Accordingly, the entire sample was not a random sample of the general population as in our study.

In the study by Gundgaard et al.,¹⁶ data weighted for only design were used. This also means that their findings may, at least to some degree, be suffering from non-response bias. In this study, we used calibrated weights accounting for both design and non-response, which may have provided us an additional opportunity to reduce the bias that non-response may cause in primary health care utilization estimates. The calibrated weights reduced the impact of non-response bias on estimates, but it did not eliminate it.

Other studies have aimed at assessing the impact of applying calibrated weights to survey data, although comparisons of the results from these studies are difficult because of e.g. different outcome variables, different weighting procedures, different sampling frames and data collection modes, cultural differences.^30,34,35 However, a striking observation is that several studies using survey data do not report whether or not data have been weighted for design and/or non-response, which indeed compromises result transparency.

The main strength of this study is that we used representative data from a large sample of the general population and not from a specific patient or insurance group. Moreover, due to the unique possibility of obtaining and extracting registry data on primary health care utilization from the entire sample by linkage through CPR numbers, the included data are considered of very high quality. A limitation that needs to be mentioned is the possibility that not all relevant sociodemographic variables were included in the weighting due to registry shortcomings, or that factors affecting primary health care utilization remain to be identified and, thus, taken into account. Also, there is a maximum of variables that can be included in the weighting procedure in order not to obtain too wide confidence intervals or degenerate weights. Another limitation of this study is that no international standard has been defined to assess when weighted estimates could be considered ‘acceptably non-biased’. We therefore acknowledge the potential shortcomings related to the approach applied in this study.

In conclusion, the results from this study showed that applying calibrated weights to the 1-year registry-based prevalence estimates on three types of primary health care utilization (chiropractor/physiotherapist, dentist and psychologist) among respondents from a nationally representative health survey reduced the impact of non-response bias on estimates but did not eliminate it. Thus, even calibrated weights based on a large number of variables cannot eliminate all the differences between respondents and non-respondents and provide estimates equal to the ‘true values’ in the background population. Future studies should take these findings into account when analyzing and interpreting their results. Also, future studies should aim at exploring whether survey data also suffer from ‘in-sufficient weighting’, and how to optimize weighting procedures to reduce non-response bias. Optimally, the primary aim in survey research design would be to identify and implement key factors to increase response rates in surveys. In this way weighting procedures should ‘correct less’ in estimates. Future studies should focus on validating survey questions against comparable registry data in order to be able to transfer findings from this study on weighting effects on non-response bias in health surveys.

Funding

No funding source.

Conflicts of interest: None declared.

• Response rates in surveys are declining, and calibrated weighting is a widely used method to ensure generalizable and representative results by reducing the impact of non-response bias.

• No previous study has investigated the impact of applying calibrated weights on registry-based prevalence estimates of primary health care utilization among a representative sample of the general adult population.

• We found weighting for non-response to decrease the 1-year prevalence estimates for chiropractor/physiotherapist, dentist and psychologist utilization among respondents, which resulted in less biased estimates when compared to the entire sample.

• When stratified by sociodemographic characteristics, results across the three types of primary health care showed a tendency towards more biased weighting effects among respondents holding an education with ‘No information’ and with ‘Western’ background.