Skip to main content
NCBI home page
European Journal of Ageing logoLink to European Journal of Ageing
. 2024 Oct 5;21(1):29. doi: 10.1007/s10433-024-00826-w

Do middle-aged and older people underreport loneliness? experimental evidence from the Netherlands

Thijs van den Broek 1,✉, Jack Lam 2, Cecilia Potente 1
PMCID: PMC11456021  PMID: 39367963

Abstract

Despite the growing acknowledgment of the importance of loneliness among older individuals, questionnaire length constraints may hinder the inclusion of common multi-item loneliness scales in surveys. Direct, single-item loneliness measures are a practical alternative, but scholars have expressed concerns that such measures may lead to underreporting. Our aim was to test whether such reservations are justified. We conducted a preregistered list experiment among 2,553 people aged 50 + who participated in the Dutch Longitudinal Internet studies for the Social Sciences (LISS) panel. The list experiment method has been developed to unobtrusively gather sensitive information. We compared the list experiment estimate of the prevalence of frequent loneliness with the corresponding direct question estimate to assess downward bias in the latter. Next to pooled models, we estimated models stratified by gender to assess whether loneliness underreporting differed between women and men. Relying on the direct question, we estimated that 5.9% of respondents frequently felt lonely. Our list experiment indicated that the prevalence of frequent loneliness was 13.1%. Although substantial in magnitude, the difference between both estimates was only marginally significant (Δb: 0.072, 95% CI: − 0.003;0.148, p = .06). No evidence of gender differences was found. Although we cannot be conclusive that loneliness estimates are biased downward when a direct question is used, our results call for caution with direct, single-item measures of loneliness if researchers want to avoid underreporting. Replications are needed to gain more precise insights into the extent to which direct, single-item loneliness measures are prone to downward reporting bias.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10433-024-00826-w.

Keywords: Loneliness, Reporting bias, Survey experiment, Social desirability, Stigma

Introduction

A significant majority of middle-aged and older people consider the absence of loneliness, defined as the distressing feeling that one’s social connections are insufficient in terms of quantity or quality (De Jong Gierveld et al. 2006; Peplau & Perlman 1982), to be a key component of a fulfilling later life (Fernández-Ballesteros et al. 2008; Phelan et al. 2004). In addition, the association of loneliness with heightened risks of illness and mortality has been well-established (Courtin & Knapp 2017; Schutter et al. 2022), leading to an increasing recognition of loneliness as a significant public health concern (Cacioppo & Cacioppo 2018).

The growing acknowledgment of the importance of loneliness among middle-aged and older individuals is prompting researchers in gerontology, public health, and related fields to integrate measures of loneliness into their surveys. While the inclusion of a single direct question, such as “How often in the past week have you felt lonely?”, is the most straightforward approach, scholars have expressed concerns about potential underreporting due to the stigma associated with loneliness (De Jong Gierveld 1998; De Jong Gierveld et al. 2006). Thus far, there has been little empirical work that assessed whether direct questions indeed lead to underreporting in a large sample of older adults.

To address this gap in the literature, the current study presents findings from a preregistered list experiment conducted among 2,553 middle-aged and older individuals in the Netherlands. The list experiment method is well-suited for our research goal, as it has been designed to measure sensitive topics while mitigating reporting bias related to social desirability. If the prevalence of loneliness as estimated using the list experiment method consistently surpasses the prevalence obtained with a direct single-item measure, it can be interpreted as evidence that the direct single-item measure of loneliness is susceptible to underreporting. This study contributes valuable insights to the ongoing discourse on loneliness assessment among middle-aged and older populations.

Background

Stigma and potential underreporting of loneliness

Various multi-item scales of loneliness have been developed (Cramer & Barry 1999; Valtorta et al. 2016; Victor et al. 2005), with the most widely used scales being the University of California—Los Angeles (UCLA) Loneliness Scale (Russell et al. 1978) and the De Jong Gierveld Loneliness Scale (De Jong-Gierveld & Kamphuis 1985). Although the psychometric properties of both scales are generally evaluated satisfactorily (Alsubheen et al. 2023a, 2023b; Penning et al. 2014), their multi-item structure may pose challenges for scholars who wish to capture loneliness in midlife and old age within their surveys. Bowling (2005) named several reasons for why researchers may be hesitant to use lengthy health or wellbeing scales: core questionnaires may already be long, the target group of interest may be frail or ill, or researchers may wish to minimize the burden on respondents and the research team. Considerations such as these may make scholars lean towards including a direct, single-item question of loneliness in their surveys rather than a multi-item scale (cf. Ratikainen et al. 2010).

The body of empirical work using a single direct question of loneliness, either as an explanatory variable or as the outcome measure, is vast (e.g., Akhter-Khan et al. 2021; Fokkema et al. 2012; Lam 2022; Lam & Vuolo 2023; Rapolienė & Aartsen 2022; Salinas et al. 2022; Sundström et al. 2009; Vozikaki et al. 2018). Drawing on three experiments, Mund et al. (2022) concluded that loneliness is a highly schematized concept that can validly and reliably be measured with a single item. In line with this conclusion, Newmyer et al. (2021) established that single-item measures of loneliness contribute meaningful information in diverse settings. Nevertheless, the use of single, direct questions to measure loneliness has been criticized (e.g., De Jong Gierveld 1998; De Jong Gierveld et al. 2006). An important reason why some scholars caution against using such measures is related to the social stigma associated with loneliness, that is, “a constellation of beliefs that derogate and devalue those who feel lonely” (Barreto et al. 2022, p. 2659).

In a lab experiment with 96 college students in the United States, Lau and Gruen (1992) highlighted loneliness as a stigmatized state. Their study showed that hypothetical lonely individuals were perceived as less psychologically adjusted, intellectually and socially competent, less liked, and rated as weaker and less attractive than their counterparts who were not described as lonely. However, a recent replication of Lau and Gruen’s work by Kerr and Stanley (2021) suggested that loneliness was not as highly stigmatized as previously reported. The authors argued that stigma seemed to be exaggerated by the confounding link between sociability and loneliness in the original study, revealing that only college students, and not a larger, more diverse sample, tended to stigmatize lonely people.

Regardless of the extent to which lonely individuals are truly stigmatized by others, lonely people may be less likely to report loneliness when directly asked due to self-stigma. Self-stigma is the perceived belief that a person will be stigmatized for being lonely, irrespective of the actual prevalence of stigmatization of lonely individuals in society (Barreto et al. 2022). Barreto et al. (2022) highlighted that individuals feeling lonely are more likely to perceive a stigma of loneliness in their community, engaging in shaming and concealing behaviors. Hanna et al. (2022), in a qualitative study involving allied health professionals working with terminally ill individuals, emphasized loneliness as a taboo feeling. They noted a tendency for people to ‘suffer in silence’, rather than articulate it to their social network, citing a lack of desire to associate with the label. Tsai and Reis (2009) found that lonely people felt less positively regarded by others and more negatively regarded themselves. Lau and Kong (1999) also explored the self-concept of loneliness, indicating that lonely people had more negative self-perceptions and that they rated their relationships and appearance lower than did non-lonely individuals. These studies underscore a link between internal self-doubts and their projection onto perceived, unfavorable attitudes of others.

Scholars skeptical about employing direct single-item measures of loneliness often argue that the negative connotations attached to loneliness may lead respondents to refrain from disclosing such feelings when asked directly (De Jong Gierveld 1998; De Jong Gierveld et al. 2006; Russell 1982). Stegen et al. (2024) recently reviewed the literature on loneliness prevalence among community-dwelling older adults. They found that the estimated loneliness prevalence was lower in studies relying on direct single-item measures than in studies where loneliness was captured with indirectly phrased multi-item measures, and mentioned underreporting due to social desirability bias as a potential explanation for this finding. The authors also mentioned other potential explanations for this pattern, however, including differences in the interpretation of and the meaning assigned to the different measures (cf. Shiovitz-Ezra & Ayalon 2012). Hence, more research is needed to assess whether common reservations about direct questions due to a presumed susceptibility to underreporting are justified. In the present study, we extend the literature by assessing empirically whether middle-aged and older adults in the Netherlands indeed underreport loneliness when asked directly.

We also recognize the importance of considering gender when assessing potential underreporting of loneliness among middle-aged and older adults. Drivers and levels of loneliness are known to differ between women and men (Dykstra & Fokkema 2007; Van den Broek 2017), and willingness to disclose feelings of loneliness in a survey may also be gendered. Building on the finding of Borys and Perlman (1985) that fictitious lonely men were evaluated even more negatively than fictitious lonely women by participants in their study (cf. Lau & Gruen 1992), De Jong Gierveld et al. (2006) argued that disclosing loneliness was particularly embarrassing for men. This compelled us to explore whether there are indeed gender differences in the extent to which loneliness is underreported.

Using list experiments to detect underreporting

The list experiment method, also known as the item-count technique, is well-fit for the pursuit of the research aims stated above. It is a survey technique to gather sensitive information about particular attributes of individuals – such as feelings, attitudes, beliefs, or behaviors – in an unobtrusive manner (Mutz 2011). The method proves particularly useful when respondents might be hesitant to openly admit that a sensitive attribute of interest applies to them due to fear of judgment. The list experiment approach has been employed in recent studies to shed light on a wide range of sensitive issues, including, for instance, physical partner violence (Lépine et al. 2020), bribes to public officials (Agerberg 2022) and anti-immigrant sentiments (Creighton et al. 2019).

In list experiments, respondents are randomly assigned into two groups. Those in the control group are given a list of j nonsensitive items, while those in the treatment group receive a list of j + 1 items, with the additional item capturing the sensitive attribute of interest. Following the presentation of the lists, respondents in both groups are asked to indicate how many of the items on the list apply to them. Importantly, they are instructed to provide only the total number of items, without specifying which items apply to them personally. The premise of the list experiment is that this can reduce respondents’ fear of being judged, which, in turn, makes them less hesitant to disclose whether the sensitive item applies to them.

The list experiment is underpinned by three key assumptions. First, the random assignment of respondents into the treatment and control groups is crucial to ensure that the two groups do not differ systematically with regard to relevant background characteristics. Second, it is assumed that there are no design effects, meaning that respondents do not count the j non-sensitive items in the lists differently based on whether they are assigned to the treatment group or the control group. Third, it is assumed that treatment group respondents respond truthfully to the sensitive item included in the list that is presented to them. This is called the “no liar” assumption, and the unobtrusive manner in which the information on the sensitive item is collected in list experiments is presumed to make this assumption more plausible compared to direct questioning. Under these three assumptions, the differences between control group and the treatment group in the number of agreed statements can be interpreted as an unbiased estimate of the prevalence of the sensitive attribute (Tsai 2019).

Lastly, the estimate of the prevalence of the sensitive attribute obtained through the list experiment can be compared with the share of people who confirm that the attribute of interest applies to them when asked directly. If the former estimate exceeds the latter, then this is indicative that the direct question is prone to downward social desirability bias. In the current study, this approach will be applied to assess potential underreporting of loneliness when relying on a single direct question.

Data and methods

The current study has been preregistered on the Open Science Framework (registration: 10.17605/OSF.IO/58X7E).

Participants

Our list experiment was fielded in the autumn of 2023 as part of the Longitudinal Internet studies for the Social Sciences (LISS) panel (Scherpenzeel 2011). The panel was established in 2007 and is based on a true probability sample of households in the Netherlands, drawn from the population register by Statistics Netherlands. The panel is maintained by Centerdata at Tilburg University. It consists of 5,000 households, comprising approximately 7,500 individuals aged 16 + . For households that would otherwise be unable to take part, a basic computer and internet access are provided. Members of the LISS panel are asked to complete online questionnaires each month and they receive a financial reward for each questionnaire they finish.

For the current study, Centerdata drew a random sample of 2,848 members of the LISS panel who were aged 50 years or older and invited them to participate in the module that included our list experiment. Of these respondents, 2,553 completed the experiment (participation rate: 89.6%).

List experiment

In the current study, 1,251 respondents were randomly assigned to the control group. They were presented with a list containing four nonsensitive statements pertaining to the week prior to the interview (in a random order): “In the past week I watched TV on four or more evenings”, “In the past week I have exercised at least twice”, “In the past week I ate my favorite food”, and “In the past week I read a book”. Respondents were asked to indicate how many of the statements applied to them (0–4). An explanatory note emphasized that they were only requested to provide a number and not to specify which statements applied to them. Next, the respondents in the control group were asked a direct question about feelings of loneliness: “In the past week, did you feel lonely on three days or more?”, with response categories being “Yes” and “No”. This question was based on the loneliness item included in the 20-item Center for Epidemiologic Studies Depression (CES-D) scale (Radloff 1977). In the CES-D, respondents have a wider range of response categories on the question about the frequency of feelings of loneliness in the previous week (“Rarely or never”, “1–2 days”, “3–4 days”, “5–7 days”), but in many prior studies using this item to measure loneliness scores are subsequently dichotomized with having felt lonely on 3 + days as the threshold for being coded as frequently lonely (e.g., Menec et al. 2019; Peltzer & Pengpid 2019; Salinas et al. 2022).

The 1,302 respondents randomly assigned to the treatment group were presented with a list that contained five statements instead of four. The fifth statement that was included in the list in addition to the four non-sensitive statements presented to the control group was: “In the past week I felt lonely on three or more days”. Note that this additional statement corresponds exactly with the direct question about loneliness asked to respondents in the control group. As it was the case for the control group, the statements in the list for the treatment group were presented in a random order, and respondents were asked to indicate how many statements applied to them (0–5).

Analytical strategy

The list experiment data are analyzed in multiple steps. First, we present basic descriptive statistics for the whole sample as well as separately for the control and treatment groups. Importantly, the share of the respondents randomly assigned to the control group who answered affirmatively to the single question about frequent loneliness will be presented as the direct question estimate of the prevalence of frequent loneliness. We also perform basic Chi-square tests and F-tests to assess whether there are systematic differences between the two groups in key background characteristics associated with loneliness (Fokkema et al. 2012; Van den Broek et al. 2019; Vozikaki et al. 2018), despite of the random assignment into groups. Background characteristics taken into account are gender (female; male; other), age in years, educational attainment (lower secondary or less; upper secondary or lower tertiary; higher tertiary; unknown) and marital status (married; separated or divorced; widowed; never married). These background characteristics were provided by Centerdata based on information already available in the LISS panel. Testing for group differences on relevant background characteristics is the recommended procedure to assess whether the assumption of treatment randomization that underlies the list experiment approach is violated (Tsai 2019).

Second, we regress the total number of statements agreed with on whether respondents were in the treatment group using ordinary least squares regression. Remember that the list presented to the treatment group contained all four nonsensitive statement presented to the control group, as well as an additional fifth statement about frequent loneliness. The magnitude to which the number of statements agreed with in the treatment group exceeds the number of statements agreed with in the control group can be interpreted as the list experiment estimate of the prevalence of frequent loneliness. We will also assess whether the assumption of no design effects, i.e., the assumption that respondents count the nonsensitive statements in the same way regardless of whether they are in the treatment or control groups, is violated using the test proposed by Blair and Imai (2012; cf. Tsai 2019). This test assesses the joint distribution of the probability of the numbers of sensitive and non-sensitive statements agreed with. It tests whether there systematically are one or more combinations of responses to the j non-sensitive items and the sensitive item with a predicted probability below zero. If the test indicates that this is the case, then the “no design effects”-assumption is violated.

Third, we compare the list experiment estimate and the direct question estimate of the prevalence of frequent loneliness. For this purpose, we adjust the number of statements agreed with for the control group. For control group respondents who answered affirmatively to the single question about frequent loneliness we increase the reported the number of statements agreed with by 1. For treatment group respondents and for control group respondents who answered negatively to the single loneliness question we leave the number of statements agreed with unchanged. Note that the separate question about loneliness asked to control group respondents corresponds exactly with the fifth statement in the list presented to treatment group respondents, albeit in the form of a question. In the absence of group differences in underreporting of frequent loneliness, one would, therefore, expect no systematic differences between the treatment group and the control group in the mean scores on the adjusted number of statements agreed with. We perform ordinary least squares regression analyses to assess whether the adjusted number of statements agreed with is systematically larger in the treatment group than in the control group. Support for this hypothesis would imply that middle-aged and older people in the Netherlands underreport frequent loneliness when asked directly.

In the final step, we will assess gender differences in the extent to which middle-aged and older people underreport loneliness. For this purpose, we will stratify the model in which the adjusted number of statements agreed with is regressed on whether respondents were in the treatment group by gender. Formal tests will be performed to assess whether coefficient estimates significantly differ between women and men.

All analyses are performed in Stata 18.0. A replication package containing instructions on how to access the data as well as annotated Stata code for the data preparation and all analyses is available on the Open Science Framework: https://osf.io/93xrw.

Results

Descriptive statistics for the whole sample as well as for the treatment group and the control group separately are presented in Table 1. Among the 1,251 control group respondents who were directly asked whether or not they had felt lonely on three or more days in the past week, the proportion who answered affirmatively to this question was 5.9%. This is our direct question estimate of the prevalence of frequent loneliness. Table 1 showed no evidence for systematic group differences concerning gender, age group, educational attainment, and marital status. We thus have no indication of violation of the assumption of random assignment into the treatment and control groups. The Blair and Imai (2012) test showed that all potential combinations of responses to the j non-sensitive items and the sensitive item had predicted probabilities exceeding zero. Hence, the test provided no evidence that the assumption of no design effects was violated (See Appendix A in the supplemental material for full results).

Table 1.

Descriptive statistics

Pooled sample Control group Treatment group Treatment group versus control group
Mean / % (SD) Mean / % (SD) Mean / % (SD) Test of difference
Lonely (direct question estimate) – 5.9% –
Gender χ2(2, n = 2,553) = 1.0, p = 0 .594
 Male 49.8% 49.8% 49.8%
 Female 50.1% 50.1% 50.2%
 Other 0.0% 0.1% 0.0%
Age 66.7 (9.5) 66.6 (9.5) 66.8 (9.6) F(1, 2,551) = 0.2, p =  0.648
Educational attainment χ2(3, n = 2,553) = 5.5, p = 0 .137
 Lower secondary or less 29.8% 31.8% 28.0%
 Higher secondary or lower tertiary 32.7% 32.2% 33.3%
 At least higher tertiary 37.3% 35.9% 38.6%
 Unknown 0.2% 0.1% 0.2%
Marital status χ2(3, n = 2,553) = 3.3, p = 0 .353
 Married 62.9% 62.6% 63.1%
 Separated or divorced 14.8% 16.0% 13.7%
 Widowed 10.0% 9.5% 10.4%
 Never married 12.3% 11.9% 12.7%
 Number of respondents 2,553 1,251 1,302
Open in a new tab

Data are from LISS panel; SD: standard deviation

Results of the regression analyses performed on the full sample are presented in Table 2. In Model 1, the non-adjusted number of statements agreed with was regressed on whether or not respondents had been assigned to the treatment group. In the control group, the mean number of statements agreed with was 2.409. In the treatment group, the mean number of statements agreed with was 0.131 (95% CI: 0.057;0.206, p < 0.001) higher. This implies that the estimated prevalence of frequent loneliness when using the list experiment method was 13.1%.

Table 2.

Results of linear regression analyses

Number of statements agreed with Adjusted number of statements agreed with
B (95%CI) B (95%CI)
Treatment group 0.131*** [0.057,0.206] 0.072† [−0.003, 0.148]
Intercept 2.409*** [2.336,2.463] 2.468*** [2.415, 2.522]
Number of respondents 2,553 2,553
Open in a new tab

Data are from LISS Panel; B: Coefficient estimate; CI: Confidence interval;

† p < 0.1; * p < .05; ** p < .01; *** p < .001

Figure 1 shows the estimated prevalence of frequent feelings of loneliness by estimation strategy. The estimated prevalence of frequent loneliness was 7.2 percentage points higher when the list experiment method was used than when we relied on the direct question. In Model 2 of Table 1 we tested whether the difference in these estimates was statistically significant by regressing the adjusted number of statements agreed with on assignment to the treatment group. Despite the large magnitude of the difference in the estimated prevalence of frequent loneliness by estimation strategy and the substantial sample size of the current study, the analyses indicated that the difference between the direct question estimate and the list experiment estimate of frequent loneliness only approximated statistical significance at the conventional alpha level of α = 0.05 (Δb: 0.072, 95% CI: −0.003; 0.148, p = 0.06). The evidence presented here is therefore insufficient to conclude that estimates of the prevalence of frequent loneliness are systematically biased downwards when a direct question is used.

Fig. 1.

Fig. 1

Open in a new tab

Estimated prevalence of frequent loneliness with 95% confidence intervals among people aged 50 + in the Netherlands by estimation strategy

In Table 3, we re-estimated the regression models stratified by gender. The estimated effect of being assigned to the treatment group on both the non-adjusted and the adjusted number of statements agreed with were very similar for women and men. Not surprisingly, therefore, formal tests did not provide any evidence of systematic gender differences in the list experiment estimates of the prevalence of frequent loneliness (Δb: −0.004, 95% CI: −0.153;0.145, p = 0.959) or the extent to which the list experiment estimate exceeded the direct question estimate (Δb: -0.006, 95% CI: −0.156;0.143, p = 0.930).

Table 3.

Results of linear regression analyses stratified by gender

Number of statements agreed with Adjusted number of statements agreed with
B (95%CI) B (95%CI)
Men only models
Treatment group 0.131* [0.026,0.237] 0.074 [−0.032, 0.180]
Intercept 2.295*** [2.220,2.371] 2.353*** [2.277, 2.429]
Number of respondents 1,272 1,272
Women only models
Treatment group 0.128* [0.023,0.233] 0.067 [−0.038,0.172]
Intercept 2.526*** [2.451,2.601] 2.587*** [2.512,2.662]
Number of respondents 1,280 1,280
Open in a new tab

Data are from LISS Panel; B: Coefficient estimate; CI: Confidence interval; † p < 0.1; * p < .05; ** p < .01; *** p < .001

Discussion

The rising scholarly attention on loneliness underscores the value of incorporating measures of loneliness in surveys within gerontology, public health and related fields. The need to restrict the length of questionnaires may, however, pose a significant obstacle to including common multi-item loneliness scales. While direct, single-item measures may form an attractive alternative, scholars have expressed concerns that direct, single-item measures may lead to underreporting due to respondents’ fear of judgment. These concerns tend to be based on assumptions, however, and whether they are warranted has hitherto not been tested empirically. The current, preregistered study is a first contribution to filling this gap in the literature.

Drawing on data from 2,553 respondents aged 50 + from the Dutch LISS-panel, we assessed whether the estimated prevalence of frequent loneliness (i.e., at least three days in the week prior to the interview) differed notably based on whether we relied on a direct question, or a list experiment approach designed to mitigate reporting bias resulting from social desirability. Relying on the direct question, we estimated that 5.9% of respondents experienced frequent feelings of loneliness. When using the list experiment approach, the estimated prevalence of frequent loneliness was considerably higher: 13.1%. It should be considered, however, that the difference between both estimates, although substantial in magnitude, was only marginally significant. Stratification of our models by gender did not provide any evidence that the patterns reported here differed between men and women.

Although the list experiment technique used here is known to be effective in mitigating social desirability bias, it is also well-established that this quality comes at the expense of efficiency. Our study underscores this drawback, as the estimates presented here had relatively wide confidence intervals despite our substantial sample size. Nevertheless, the results of the power analyses presented by Blair et al. (2020) suggest that our sample size suffices to get a lower variance estimate in terms of root mean square error of the true prevalence rate than a direct question, even when the sensitivity bias is as low as 5 percentage points. Yet, our sample size only suffices to detect sensitivity bias at the conventional power threshold of 80 percent when the rate of underreporting is above 10 percentage points. Thus, our study may be expected to generate an estimate of the prevalence of frequent loneliness that is closer to the true prevalence rate than a direct question would, but it is slightly underpowered to detect sensitivity bias that is low in magnitude. Replications of our study, preferably in combination with a meta-analysis (cf. Blair et al. 2020), are therefore called for to gain more precise insights into the extent to which direct, single-item loneliness measures are prone to downward reporting bias. Replications in surveys that are collected via face-to-face interviews would be particularly welcome. Our list experiment was fielded as part of an online panel survey, and, given the impersonal nature of such surveys, respondents’ tendency to underreport sensitive items may be smaller in online surveys than in in face-to-face interviews (cf. Kays et al. 2012). This would imply that direct question estimates of frequent loneliness may be even lower in face-to-face interviews. Future research is needed to assess whether this indeed is the case.

Scholars aiming to conduct future studies on loneliness underreporting could, if their questionnaire length permits, consider adopting a double list experiment design as opposed to the current study’s single list experiment design. In a double list experiment design, respondents participate in two list experiments with different lists of non-sensitive statements, but the same sensitive statement. Each respondent is randomly assigned to the treatment group in either the first or the second list experiment, and, by design, to the control group in the other list experiment. The benefit of this design as compared to the current study’s single list experiment design is that the combined estimate of the prevalence of the sensitive attribute of interest is considerably more precise (Blair et al. 2020; cf. Krumpal et al. 2018).

As explained in the introduction, the list experiment approach adopted in the current study rests on three assumptions. As reported, we performed a series of tests to assess violations of the assumptions of random treatment assignment and of no design effects. The results did not provide any reason to suspect that these two assumptions were violated. The “no liars”-assumption – the third assumption underlying the list experiment approach – cannot be tested, however (Tsai 2019). The discrete manner in which information on sensitive items is collected in list experiments may be expected to reduce respondents’ fear of judgment and, consequently, their tendency to underreport the sensitive attribute of interest. Blair et al. (2020) reviewed list experiment studies on various topics of major interest to political scientists and found that for some sensitive issues list experiment estimates tended to differ systematically from direct question estimates. For instance, estimates of vote buying were higher and estimated support for authoritarian regimes was lower when list experiments were used. This suggests that a list experiment design can indeed reduce respondents’ tendency to lie about sensitive topics. Nevertheless, we cannot be certain that respondents’ hesitancy to disclose adherence to a sensitive statement is taken away entirely in a list experiment. Applied to the current study, this means that the prevalence of frequent loneliness may still be underestimated somewhat when using the list experiment approach. Consequently, the extent to which direct question estimates of loneliness are downward biased may have been underestimated as well. This implies that the current study’s evaluation of the extent to which direct single-item loneliness measures lead to underreporting is arguably conservative.

Relatedly, ceiling effects may have contributed to the underestimation of downward bias. Ceiling effects in list experiments occur when all statements in the list apply to treatment group respondents. In the control group, 12.5 percent of respondents reported agreeing with all 4 non-sensitive statements in the list that was presented to them. Given the random assignment of respondents into the treatment and control groups, we may assume that approximately one out of every eight respondents in the treatment group also agreed with all four non-sensitive statements in the lists presented to them. For these respondents, responding affirmatively to the fifth item about loneliness would result in a score of 5 out of 5 statements agreed with, from which it can directly be inferred that the sensitive statement applied to the respondent. Compared with a direct question, the list experiment approach therefore plausibly does not reduce the incentive to underreport frequent loneliness for this subset of respondents. The problem caused by ceiling effects may be reduced in future studies by optimizing the list of non-sensitive items. The current study’s list was arguably suboptimal, because it did not include at least one statement that very few people agreed with. Inclusion of a low prevalence statement in the list would reduce the share of respondents that agree with all non-sensitive statements and consequently mitigate the risk of ceiling effects (Blair & Imai 2012).

Loneliness can be captured with a range of instruments, each with its own strengths and drawbacks. Direct, single-item measures of loneliness are commonly used and prior work suggested that they may be an adequate option when scholars are faced with questionnaire length constraints (Mund et al. 2022; Newmyer et al. 2021). Nevertheless, researchers may have good reasons to prefer other instruments. For instance, loneliness is sometimes conceptualized as a multidimensional concept rather than as unidimensional (Mund et al. 2022). When scholars want to distinguish multiple dimensions of loneliness, e.g., social loneliness and emotional loneliness, then a single-item measure is not useful and multi-item scales such as the De Jong Gierveld loneliness scale are called for (De Jong Gierveld et al. 2006). The current study’s results suggest that concerns about underreporting may also be a reason for researchers to opt for loneliness measures other than a direct single-item measure. Although the findings presented here do not allow us to be conclusive that estimates of the prevalence of frequent loneliness are systematically biased downwards when a direct question is used, the large difference in the estimates in combination with the minimal overlap of the confidence intervals of the list experiment estimate and the direct question estimate of frequent loneliness suggests, in our view, that researchers should be cautious about using direct, single-item measures of loneliness if they want to avoid underreporting. The point made earlier that our estimate of the extent to which direct question estimates of frequent loneliness are downward biased is plausibly conservative further strengthens this issue.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 95 KB) (95.4KB, pdf)

Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 895537 (SAMBa). Additional funding from the Erasmus School of Health Policy & Management via the Frans Rutten Research Prize scheme is gratefully acknowledged.

Author contributions

TvdB designed the study, performed the statistical analyses and drafted the main manuscript text. JL contributed to the literature review and the conceptualisation. CP contributed to the statistical analyses and the interpretation of results. All authors reviewed the manuscript.

Funding

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 895537 (SAMBa). This research was partially supported by an Australian Research Council Discovery Early Career Researcher Award (project number DE210100582). Additional funding from the Erasmus School of Health Policy & Management via the Frans Rutten Research Prize scheme is gratefully acknowledged.

Data availability

The data used in the current study were collected as part of the LISS panel, and deposited in the LISS Data Archive (https://www.dataarchive.lissdata.nl/). The two LISS panel data modules used in the current study are available via: https://doi.org/10.57990/mtve-qc91https://doi.org/10.57990/qn3k-as78.

Declarations

Ethics approval and consent to participate

This study was assessed and approved by the Research Ethics Review Committee of the Erasmus School of Health Policy & Management at Erasmus University Rotterdam (reference: ETH2324-0130).

Competing interests

The authors declare no competing interests.

Footnotes

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

  1. Agerberg M (2022) Corrupted estimates? response bias in citizen surveys on corruption. Polit Behav 44(2):653–678. 10.1007/s11109-020-09630-5 [Google Scholar]
  2. Akhter-Khan SC, Tao Q, Ang TFA, Itchapurapu IS, Alosco ML, Mez J, Qiu WQ (2021) Associations of loneliness with risk of Alzheimer’s disease dementia in the Framingham heart study. Alzheimer’s & Dementia 17(10):1619–1627. 10.1002/alz.12327 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Alsubheen SA, Oliveira A, Habash R, Goldstein R, Brooks D (2023a) Measurement properties and cross-cultural adaptation of the De Jong gierveld loneliness scale in adults. Eur J Psychol Assess. 10.1027/1015-5759/a000784 [Google Scholar]
  4. Alsubheen SA, Oliveira A, Habash R, Goldstein R, Brooks D (2023b) Systematic review of psychometric properties and cross-cultural adaptation of the University of California and los angeles loneliness scale in adults. Curr Psychol 42(14):11819–11833. 10.1007/s12144-021-02494-w [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Barreto M, van Breen J, Victor C, Hammond C, Eccles A, Richins MT, Qualter P (2022) Exploring the nature and variation of the stigma associated with loneliness. J Soc Pers Relat 39(9):2658–2679. 10.1177/02654075221087190 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Blair G, Imai K (2012) Statistical analysis of list experiments. Polit Anal 20(1):47–77. 10.1093/pan/mpr048 [Google Scholar]
  7. Blair G, Coppock A, Moor M (2020) When to worry about sensitivity bias: a social reference theory and evidence from 30 years of list experiments. Am Polit Sci Rev 114(4):1297–1315. 10.1017/S0003055420000374 [Google Scholar]
  8. Borys S, Perlman D (1985) Gender differences in loneliness. Pers Soc Psychol Bull 11(1):63–74. 10.1177/0146167285111006 [Google Scholar]
  9. Bowling A (2005) Just one question: if one question works, why ask several? J Epidemiol Community Health 59(5):342–345. 10.1136/jech.2004.021204 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Cacioppo JT, Cacioppo S (2018) The growing problem of loneliness. The Lancet 391(10119):426. 10.1016/S0140-6736(18)30142-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Courtin E, Knapp M (2017) Social isolation, loneliness and health in old age: a scoping review. Health Soc Care Community 25(3):799–812. 10.1111/hsc.12311 [DOI] [PubMed] [Google Scholar]
  12. Cramer KM, Barry JE (1999) Conceptualizations and measures of loneliness: a comparison of subscales. Personality Individ Differ 27(3):491–502. 10.1016/S0191-8869(98)00257-8 [Google Scholar]
  13. Creighton MJ, Schmidt P, Zavala-Rojas D (2019) Race, wealth and the masking of opposition to immigrants in the Netherlands. Int Migr 57(1):245–263. 10.1111/imig.12519 [Google Scholar]
  14. De Jong Gierveld J (1998) A review of loneliness: concept and definitions, determinants and consequences. Rev Clin Gerontol 8(1):73–80. 10.1017/S0959259898008090 [Google Scholar]
  15. De Jong-Gierveld J, Kamphuis F (1985) The development of a Rasch-type loneliness scale. Appl Psychol Meas 9(3):289–299. 10.1177/014662168500900307 [Google Scholar]
  16. De Jong Gierveld J, Van Tilburg T, Dykstra PA (2006) Loneliness and social isolation. In: Vangelisti AL, Perlman D (eds) Cambridge handbook of personal relationships. Cambridge University Press, Cambridge, UK, pp 485–500 [Google Scholar]
  17. Dykstra PA, Fokkema T (2007) Social and emotional loneliness among divorced and married men and women: comparing the deficit and cognitive perspectives. Basic Appl Soc Psychol 29(1):1–12. 10.1080/01973530701330843 [Google Scholar]
  18. Fernández-Ballesteros R, García LF, Abarca D, Blanc L, Efklides A, Kornfeld R, Patricia S (2008) Lay concept of aging well: cross-cultural comparisons. J Am Geriatr Soc 56(5):950–952. 10.1111/j.1532-5415.2008.01654.x [DOI] [PubMed] [Google Scholar]
  19. Fokkema T, De Jong Gierveld J, Dykstra PA (2012) Cross-national differences in older adult loneliness. J Psychol 146(1–2):201–228. 10.1080/00223980.2011.631612 [DOI] [PubMed] [Google Scholar]
  20. Hanna JR, McConnell T, Harrison C, Patynowska KA, Finucane AM, Hudson B, Reid J (2022) ‘There’s something about admitting that you are lonely’ – prevalence, impact and solutions to loneliness in terminal illness: an explanatory sequential multi-methods study. Palliat Med 36(10):1483–1492. 10.1177/02692163221122269 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Kays K, Gathercoal K, Buhrow W (2012) Does survey format influence self-disclosure on sensitive question items? Comput Hum Behav 28(1):251–256. 10.1016/j.chb.2011.09.007 [Google Scholar]
  22. Kerr NA, Stanley TB (2021) Revisiting the social stigma of loneliness. Personality Individ Differ 171:110482. 10.1016/j.paid.2020.110482 [Google Scholar]
  23. Krumpal I, Jann B, Korndörfer M, Schmukle S (2018) Item sum double-list technique: an enhanced design for asking quantitative sensitive questions. Surv Res Methods 12(2):91–102. 10.18148/srm/2018.v12i2.7247 [Google Scholar]
  24. Lam J (2022) Neighborhood characteristics, neighborhood satisfaction, and loneliness differences across ethnic–migrant groups in Australia. J Gerontol: Series B 77(11):2113–2125. 10.1093/geronb/gbab219 [DOI] [PubMed] [Google Scholar]
  25. Lam J, Vuolo M (2023) Longitudinal associations between loneliness and prescription medication use. J Gerontol: Series B 78(4):730–735. 10.1093/geronb/gbad010 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Lau S, Gruen GE (1992) The social stigma of loneliness: effect of target person’s and perceiver’s sex. Pers Soc Psychol Bull 18(2):182–189. 10.1177/0146167292182009 [Google Scholar]
  27. Lau S, Kong C-K (1999) The acceptance of lonely others: effects of loneliness and gender of the target person and loneliness of the perceiver. J Soc Psychol 139(2):229–241. 10.1080/00224549909598377 [DOI] [PubMed] [Google Scholar]
  28. Lépine A, Treibich C, D’Exelle B (2020) Nothing but the truth: consistency and efficiency of the list experiment method for the measurement of sensitive health behaviours. Soc Sci Med 266:113326. 10.1016/j.socscimed.2020.113326 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Menec VH, Newall NE, Mackenzie CS, Shooshtari S, Nowicki S (2019) Examining individual and geographic factors associated with social isolation and loneliness using Canadian longitudinal study on aging (CLSA) data. PLoS ONE 14(2):e0211143. 10.1371/journal.pone.0211143 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Mund M, Maes M, Drewke PM, Gutzeit A, Jaki I, Qualter P (2022) Would the real loneliness please stand up? the validity of loneliness scores and the reliability of single-item scores. Assessment 30(4):1226–1248. 10.1177/10731911221077227 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Mutz DC (2011) Population-based survey experiments. Princeton University Press, Princeton, NJ [Google Scholar]
  32. Newmyer L, Verdery AM, Margolis R, Pessin L (2021) Measuring older adult loneliness across countries. J Gerontol: Series B 76(7):1408–1414. 10.1093/geronb/gbaa109 [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Peltzer K, Pengpid S (2019) Loneliness correlates and associations with health variables in the general population in Indonesia. Int J Ment Heal Syst 13(1):24. 10.1186/s13033-019-0281-z [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Penning MJ, Liu G, Chou PHB (2014) Measuring loneliness among middle-aged and older adults: The UCLA and De Jong Gierveld loneliness scales. Soc Indic Res 118(3):1147–1166. 10.1007/s11205-013-0461-1 [Google Scholar]
  35. Peplau LA, Perlman D (1982) Perspectives on loneliness. In: Peplau LA, Perlman D (eds) Loneliness: A sourcebook of current theory, research and therapy. Wiley, New York, pp 1–18 [Google Scholar]
  36. Phelan EA, Anderson LA, Lacroix AZ, Larson EB (2004) Older adults’ views of “successful aging” how do they compare with researchers’ definitions? J Am Geriatr Soc 52(2):211–216. 10.1111/j.1532-5415.2004.52056.x [DOI] [PubMed] [Google Scholar]
  37. Radloff LS (1977) The CES-D scale: a self-report depression scale for research in the general population. Appl Psychol Meas 1(3):385–401. 10.1177/014662167700100306 [Google Scholar]
  38. Rapolienė G, Aartsen M (2022) Lonely societies: low trust societies? further explanations for national variations in loneliness among older Europeans. Eur J Ageing 19(3):485–494. 10.1007/s10433-021-00649-z [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Ratikainen J, Hansen T, Slagsvold B (2010) Report on evaluation of existing and feasibility for development of new psychological instruments in the GGS. NOVA Norwegian Social Research, Oslo [Google Scholar]
  40. Russell DW (1982) The measurement of loneliness. In: Peplau LA, Perlman D (eds) Loneliness: A sourcebook of current theory, research and therapy and therapy. Wiley, New York, pp 81–104 [Google Scholar]
  41. Russell DW, Peplau LA, Ferguson ML (1978) Developing a measure of loneliness. J Pers Assess 42(3):290–294. 10.1207/s15327752jpa4203_11 [DOI] [PubMed] [Google Scholar]
  42. Salinas J, Beiser AS, Samra JK, O’Donnell A, DeCarli CS, Gonzales MM, Seshadri S (2022) Association of loneliness with 10-year dementia risk and early markers of vulnerability for neurocognitive decline. Neurology 98(13):e1337–e1348. 10.1212/WNL.0000000000200039 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Scherpenzeel A (2011) Data collection in a probability-based internet panel: How the LISS panel was built and how it can be used. Bull Sociol Methodol/bull De Méthodol Sociol 109(1):56–61. 10.1177/0759106310387713 [Google Scholar]
  44. Schutter N, Holwerda TJ, Comijs HC, Stek ML, Peen J, Dekker JJM (2022) Loneliness, social network size and mortality in older adults: a meta-analysis. Eur J Ageing 19(4):1057–1076. 10.1007/s10433-022-00740-z [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Shiovitz-Ezra S, Ayalon L (2012) Use of direct versus indirect approaches to measure loneliness in later life. Res Aging 34(5):572–591. 10.1177/0164027511423258 [Google Scholar]
  46. Stegen H, Duppen D, Savieri P, Stas L, Pan H, Aartsen M, De Donder L (2024) Loneliness prevalence of community-dwelling older adults and the impact of the mode of measurement, data collection, and country: a systematic review and meta-analysis. Int Psychogeriatr. 10.1017/S1041610224000425 [DOI] [PubMed] [Google Scholar]
  47. Sundström G, Fransson E, Malmberg B, Davey A (2009) Loneliness among older Europeans. Eur J Ageing 6(4):267–275. 10.1007/s10433-009-0134-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Tsai C (2019) Statistical analysis of the item-count technique using Stata. Stand Genomic Sci 19(2):390–434. 10.1177/1536867X19854018 [Google Scholar]
  49. Tsai F-F, Reis HT (2009) Perceptions by and of lonely people in social networks. Pers Relat 16(2):221–238. 10.1111/j.1475-6811.2009.01220.x [Google Scholar]
  50. Valtorta NK, Kanaan M, Gilbody S, Hanratty B (2016) Loneliness, social isolation and social relationships: what are we measuring? A novel framework for classifying and comparing tools. BMJ Open 6(4):e010799. 10.1136/bmjopen-2015-010799 [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Van den Broek T (2017) Gender differences in the correlates of loneliness among Japanese persons aged 50–70. Australas J Ageing 36(3):234–237. 10.1111/ajag.12448 [DOI] [PubMed] [Google Scholar]
  52. Van den Broek T, Tosi M, Grundy E (2019) Offspring and later-life loneliness in Eastern and Western Europe. J Fam Res 31(2):199–215. 10.3224/zff.v31i2.05 [Google Scholar]
  53. Victor C, Grenade L, Boldy D (2005) Measuring loneliness in later life: a comparison of differing measures. Rev Clin Gerontol 15(1):63–70. 10.1017/S0959259805001723 [Google Scholar]
  54. Vozikaki M, Papadaki A, Linardakis M, Philalithis A (2018) Loneliness among older European adults: results from the survey of health, aging and retirement in Europe. J Public Health 26(6):613–624. 10.1007/s10389-018-0916-6 [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary file1 (PDF 95 KB) (95.4KB, pdf)

Data Availability Statement

The data used in the current study were collected as part of the LISS panel, and deposited in the LISS Data Archive (https://www.dataarchive.lissdata.nl/). The two LISS panel data modules used in the current study are available via: https://doi.org/10.57990/mtve-qc91https://doi.org/10.57990/qn3k-as78.

Articles from European Journal of Ageing are provided here courtesy of Springer

RESOURCES