Abstract
As the COVID-19 pandemic continues, interest in mental health impacts is shifting from short-term to long-term outcomes. As part of a longitudinal online survey study examining mental health impacts of the pandemic, we assessed the risk of attrition bias related to a history of depression—a condition research shows can increase challenges of recruitment and retention. Among 5023 participants who completed the baseline survey, significantly more reporting a history of depression were lost to follow-up: baseline to 3 months: 497/760 (65.4%) vs 2228/4263 (52.3%), P < 0.001; 3 to 6 months: 179/263 (68.1%) vs 1183/2035 (58.1%), P = 0.002. Participants reporting a history of depression also had greater adjusted odds of a Patient Health Questionnaire-8 score ≥10 (odds ratio [OR] = 3.97, 95% confidence interval [CI] 3.27, 4.84), Generalized Anxiety Disorder-7 score ≥10 (OR = 3.77, 95% CI 3.07, 4.62), and Posttraumatic Diagnostic Scale for DSM V score ≥ 28 (OR = 7.17, 95% CI 4.67, 11.00) at baseline, indicating a need to account for attrition bias when examining these outcomes. Similar considerations likely apply to other longitudinal survey studies and are important to address to ensure accurate evidence is available to support policy decisions regarding resource allocation and funding.
Keywords: Attrition bias, COVID-19, longitudinal study, mental health
The adverse mental health impacts of the early months of the COVID-19 pandemic have been well documented.1 Attention is turning to longitudinal impacts, which have important implications for ongoing mental health resource and funding needs. Most studies assessing mental health impacts of the COVID-19 pandemic have employed online surveys.1 While this was a practical approach in pandemic conditions, researchers and policymakers have had to keep in mind the selection bias that online recruitment and participation introduces.2 In considering longitudinal data, attrition bias must also be considered. The threat that differential dropout over the course of a study between two groups being compared poses to the internal validity of that study has long been recognized,3 together with the need to consider its impact when assessing health care interventions.4 Depending on the relationships among the exposure of interest, the outcome of interest, and loss to follow-up, it can meaningfully impact results.4,5 Where the exposure of interest increases likelihood of both loss to follow-up and the longitudinal outcome of interest, attrition may cause underestimation of the outcome.5 Depression is known to exacerbate the challenges of research recruitment and retention6,7 and is highly recurrent, raising the possibility of attrition bias in studies investigating the longitudinal mental health impact of the pandemic.
METHODS
In June 2020, we began a longitudinal study investigating the mental health impact of the COVID-19 pandemic in the United States.8 The study was approved by the institutional review board at Baylor Scott and White Research Institute (#020-139) with a waiver of the requirement for written informed consent. Participants completed three online surveys via the Qualtrics™ survey platform (Qualtrics, Inc.; Seattle, WA): at enrollment and then at 3- and 6-month follow-up. Completion targets were 45% and 40%, respectively. Once each of these targets was reached, data collection for that time point ceased. Data collected included demographic characteristics, clinical and mental health history, experience of the COVID-19 pandemic, and symptoms of depression (using the Patient Health Questionnaire-8 [PHQ-8]), generalized anxiety (Generalized Anxiety Disorder-7 [GAD-7]), and post-traumatic stress (Posttraumatic Diagnostic Scale [PDS-5]). We examined the proportions lost to follow-up among those who did vs did not report a history of depression and compared the demographic characteristics of those reporting a history of depression who were vs were not lost to follow-up.
Significant differences in means and counts/percentages were assessed using t tests and chi-square tests, respectively. Multivariable logistic regression was performed to determine whether history of depression was significantly associated with mental health measures (PHQ-8, GAD-7, and PDS-5), adjusting for a propensity score for a reported history of depression calculated using all the demographic variables listed in Table 1. Cutoffs of ≥10 for the PHQ-8 (which shows 88% sensitivity and 88% specificity in discriminating ‘probable’ depression9) and the GAD-7 (which shows 89% sensitivity and 82% specificity in discriminating ‘probable’ anxiety10), and ≥28 for the PDS-5 (which shows 79% sensitivity and 78% specificity compared to a diagnosis based on the PTSD Symptom Scale – Interview Version for DSM-511) were applied to dichotomize these measures.
Table 1.
Demographic characteristics of survey participants by reported history of depression at baseline, and follow-up status
| Characteristicb | History of depression |
No reported history of depression |
||||
|---|---|---|---|---|---|---|
| Not lost to follow-up (n = 263) |
Lost to follow-up (n = 497) |
P valuea |
Not lost to follow-up (n = 2035) |
Lost to follow-up (n = 2228) |
P valuea |
|
| Age | 49.9 ± 13.4 | 43.9 ± 14.5 | <0.0001 | 52.8 ± 12.9 | 48.7 ± 14.9 | <0.0001 |
| Body mass index | 30.2 ± 8.6 | 30.5 ± 2.8 | 0.736 | 27.6 ± 7.7 | 28.4 ± 7.1 | 0.002 |
| Sex: Female | 177 (67.3%) | 337 (67.8%) | 0.986 | 1133 (55.7%) | 1313 (58.9%) | 0.044 |
| Race | 0.392 | 0.020 | ||||
| White | 217 (82.5%) | 407 (81.9%) | 1512 (74.3%) | 1588 (71.3%) | ||
| Black | 23 (8.7%) | 30 (6.0%) | 139 (6.8%) | 200 (9.0%) | ||
| Hispanic | 11 (4.2%) | 32 (6.4%) | 142 (7.0%) | 191 (8.6%) | ||
| Asian | 6 (2.3%) | 11 (2.2%) | 171 (8.4%) | 171 (7.7%) | ||
| Other | 6 (2.3%) | 17 (3.4%) | 71 (3.5%) | 78 (3.5%) | ||
| Marital status | 0.017 | <0.0001 | ||||
| Single | 86 (32.7%) | 212 (42.7%) | 515 (25.3%) | 734 (32.9%) | ||
| Married/common law | 134 (50.9%) | 202 (40.6%) | 1278 (62.8%) | 1192 (53.5%) | ||
| Divorced/separated | 39 (14.8%) | 80 (16.1%) | 231 (11.3%) | 280 (12.6%) | ||
| Unknown/prefer not to answer | 4 (1.5%) | 3 (0.6%) | 11(0.5%) | 22 (1.0%) | ||
| Occupation segment | 0.160 | 0.090 | ||||
| Essential workers | 18 (6.8%) | 55 (11.1%) | 185 (9.1%) | 243 (10.9%) | ||
| General population | 159 (60.5%) | 293 (58.5%) | 1265 (62.2%) | 1386 (62.2%) | ||
| Healthcare providers | 86 (32.7%) | 149 (29.8%) | 585 (28.7%) | 599 (26.9%) | ||
| Current living situation | 0.012 | <0.0001 | ||||
| Owns home/apartment | 154 (58.6%) | 262 (52.7%) | 1509 (74.1%) | 1414 (63.5%) | ||
| Rents home/apartment | 89 (33.8%) | 160 (32.2%) | 405 (19.9%) | 593 (26.6%) | ||
| Lives with family | 15 (5.7%) | 59 (11.9%) | 106 (5.2%) | 183 (8.2%) | ||
| Lives in community housing/homeless | 0 (0.0%) | 9 (1.8%) | 2 (0.1%) | 7 (0.3%) | ||
| Rehabilitation facility | 1 (0.4%) | 0 (0.0%) | 1 (0.1%) | 3 (0.1%) | ||
| Other/unknown | 4 (1.5%) | 7 (1.4%) | 12 (0.6%) | 28 (1.3%) | ||
| Current smoker | 34 (12.9%) | 95 (19.1%) | 0.031 | 119 (5.8%) | 168 (7.5%) | 0.028 |
aSignificant differences in means and counts/percentages were assessed using t tests and chi-square tests, respectively.
bOther characteristics compared included highest education level, current work status, number of people supported by total household income, employment status before COVID-19, and any chronic condition. All had P values >0.2 and are not shown here for space reasons.
An additional multivariable regression with repeated measures was performed to assess the association of depression and time with loss to follow-up and to assess whether the interaction between depression and time was associated with loss to follow-up. All analyses were performed using SAS version 9.4 (SAS Inc., Cary, NC).
RESULTS
Of the 5023 participants who completed the first survey, responses were obtained from 2298 and 936 at the 3- and 6-month follow-up points, meeting the preset completion targets. Significantly more participants who reported a history of depression were lost to follow-up than those not reporting such a history: baseline to 3 months: 497/760 (65.4%) vs 2228/4263 (52.3%), P < 0.001; 3 months to 6 months: 179/263 (68.1%) vs 1183/2035 (58.1%), P = 0.002. Overall, participants with a history of depression were significantly more likely to be lost to follow-up (odds ratio [OR] = 1.38, 95% confidence interval [CI] 1.15, 1.65). Participants (with or without a history of depression) were more likely to be lost to follow-up at 6 months than 3 months (OR = 3.82, 95% CI 3.55, 4.11), but the increase in loss to follow-up between these two time points did not differ significantly between those who did vs did not report a history of depression.
Participants reporting a history of depression had greater adjusted odds of a PHQ-8 score ≥10 (OR = 3.97, 95% CI 3.27, 4.84), GAD-7 score ≥10 (OR = 3.77, 95% CI 3.07, 4.62), and PDS-5 score ≥28 (OR = 7.17, 95% CI 4.67, 11.00) at baseline compared to those in the group without depression. We also found significant differences in demographic characteristics according to follow-up status at 3 months among participants, both with and without a history of depression at baseline (Table 1).
DISCUSSION
Our results show that a history of depression was associated with higher scores on the PHQ-8, GAD-7, and PDS-5 at baseline and with increased loss to follow-up over the course of the study. The association between a history of depression and higher scores on the mental health measures was expected. Depression is a highly recurrent disorder, with at least 50% of people who recover experiencing a return of symptoms and 80% of those likely to experience additional recurrences,12 and depression and anxiety are frequently comorbid.13 The association between a history of depression and loss to follow-up is likewise unsurprising, given previous research reporting depression’s exacerbation of the challenges inherent in research recruitment and retention.6,7 The practical decision made in our study design to cut off follow-up when the 45% (at 3 months) and 40% (at 6 months) completion targets were reached may have inadvertently exacerbated attrition bias related to a history of depression if participants with such a history were slower to respond, and further research is needed on this question to inform future survey study designs. In the meantime, our findings indicate the need to account for attrition bias in longitudinal analyses of the psychological outcome measures.
Some, but not all, of the COVID-19 studies starting to report longitudinal mental health outcomes have considered attrition bias related to mental health: one found no relationship between a history of mental illness and continued participation,14 another found that participants who did not complete all the follow-up surveys tended to have poorer mental health at baseline but merely noted this in the limitations,15 while others have either not reported how participants who did not complete the necessary follow-up differed from those who did16,17 or did not include mental health history among the factors considered.18 Research indicates the magnitude of attrition bias can increase over length of follow-up,19 suggesting this issue may be of growing importance as the follow-up time examined extends from weeks or months to years. In our results, although a greater proportion of participants reporting a history of depression were lost to follow-up at both 3 and 6 months, the increase in loss between these timepoints was not significantly greater than the increase seen among participants not reporting a history of depression.
Statistical methods, including stratification-based techniques, multiple imputation, and inverse probability-of-censoring weighted estimation, are available to mitigate the effects of attrition bias.5 These are only useful, however, if researchers assess attrition bias and determine the appropriate approach for mitigating any identified. Failing to account for mental health–related attrition bias in studies examining the mental health impact of COVID-19 risks introducing underestimates to the evidence base informing policy decisions related to funding for and access to mental health resources. A similar issue—attrition bias related to socioeconomic status—has contributed to the perceived “murkiness” of effectiveness of the Head Start early childhood intervention program,19 which in turn results in periodic calls for defunding this program.20 Given the misinformation plaguing public understanding of and reaction to COVID-19, rigorous scrutiny of data, methods, and interpretations is essential to ensure an accurate evidence base for understanding the impact of the pandemic and to inform decisions about resource allocation and directions of further research.
Disclosure statement/Funding
This work was supported by the Baylor Scott & White Dallas Foundation and the W. W. Caruth, Jr. Fund at Communities Foundation of Texas. The authors report no conflicts of interest. Deidentified data are available upon reasonable request to the corresponding author.
References
- 1.Vindegaard N, Benros ME.. COVID-19 pandemic and mental health consequences: systematic review of the current evidence. Brain Behav Immun. 2020;89:531–542. doi: 10.1016/j.bbi.2020.05.048. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Ball HL. Conducting online surveys. J Hum Lact. 2019;35(3):413–417. doi: 10.1177/0890334419848734. [DOI] [PubMed] [Google Scholar]
- 3.Campbell DT, Stanley JC.. Experimental and Quasi-Experimental Designs for Research. Boston, MA: Houghton Mifflin Company; 1966. [Google Scholar]
- 4.Slack MK, Draugalis JR.. Establishing the internal and external validity of experimental studies. Am J Health Syst Pharm. 2001;58(22):2173–2181. [PubMed] [Google Scholar]
- 5.Howe CJ, Cole SR, Lau B, Napravnik S, Eron JJ. Jr.. Selection bias due to loss to follow up in cohort studies. Epidemiology. 2016;27(1):91–97. doi: 10.1097/EDE.0000000000000409. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Edwards L, Salisbury C, Horspool K, Foster A, Garner K, Montgomery AA.. Increasing follow-up questionnaire response rates in a randomized controlled trial of telehealth for depression: three embedded controlled studies. Trials. 2016;17(1):107. doi: 10.1186/s13063-016-1234-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Brown JSL, Murphy C, Kelly J, Goldsmith K.. How can we successfully recruit depressed people? Lessons learned in recruiting depressed participants to a multi-site trial of a brief depression intervention (the ‘CLASSIC’ trial). Trials. 2019;20(1):131. doi: 10.1186/s13063-018-3033-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Warren AM, Zolfaghari K, Fresnedo M, et al. Anxiety sensitivity, COVID-19 fear, and mental health: results from a United States population sample. Cogn Behav Ther. 2021;50(3):204–216. doi: 10.1080/16506073.2021.1874505. [DOI] [PubMed] [Google Scholar]
- 9.Kroenke K, Spitzer RL.. The PHQ-9: a new depression diagnostic and severity measure. Psychiatr Ann. 2002;32(9):509–515. doi: 10.3928/0048-5713-20020901-06. [DOI] [Google Scholar]
- 10.Spitzer RL, Kroenke K, Williams JB, Lowe B.. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006;166(10):1092–1097. doi: 10.1001/archinte.166.10.1092. [DOI] [PubMed] [Google Scholar]
- 11.Foa EB, McLean CP, Zang Y, et al. Psychometric properties of the Posttraumatic Stress Disorder Symptom Scale Interview for DSM-5 (PSSI-5). Psychol Assess. 2016;28(10):1159–1165. doi: 10.1037/pas0000259. [DOI] [PubMed] [Google Scholar]
- 12.American Psychiatric Association . Diagnostic and Statistical Manual of Mental Disorders: DSM-5™. 5th ed. Washington, DC: American Psychiatric Association Publishing; 2013. [Google Scholar]
- 13.Kalin NH. The critical relationship between anxiety and depression. Am J Psychiatry. 2020;177(5):365–367. doi: 10.1176/appi.ajp.2020.20030305. [DOI] [PubMed] [Google Scholar]
- 14.Daly M, Sutin AR, Robinson E.. Longitudinal changes in mental health and the COVID-19 pandemic: evidence from the UK Household Longitudinal Study. Psychol Med. 2020;1–10. doi: 10.1017/S0033291720004432. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.O'Connor RC, Wetherall K, Cleare S, et al. Mental health and well-being during the COVID-19 pandemic: longitudinal analyses of adults in the UK COVID-19 Mental Health & Wellbeing study. Br J Psychiatry. 2021;218(6):326–333. doi: 10.1192/bjp.2020.212. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Fancourt D, Steptoe A, Bu F.. Trajectories of anxiety and depressive symptoms during enforced isolation due to COVID-19 in England: a longitudinal observational study. Lancet Psychiatry. 2021;8(2):141–149. doi: 10.1016/S2215-0366(20)30482-X. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.McGinty EE, Presskreischer R, Han H, Barry CL.. Trends in psychological distress among US adults during different phases of the COVID-19 pandemic. JAMA Netw Open. 2022;5(1):e2144776. doi: 10.1001/jamanetworkopen.2021.44776. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Riehm KE, Holingue C, Smail EJ, et al. Trajectories of mental distress among U.S. adults during the COVID-19 pandemic. Ann Behav Med. 2021;55(2):93–102. doi: 10.1093/abm/kaaa126. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Weuve J. Magnitude matters: beyond detection in the presence of selection in research on socioeconomic inequalities in health. Epidemiology. 2013;24(1):10–13. doi: 10.1097/EDE.0b013e3182788390. [DOI] [PubMed] [Google Scholar]
- 20.Burke L, Muhlhausen D. Head Start impact evaluation report finally released. https://www.heritage.org/education/report/head-start-impact-evaluation-report-finally-released. Accessed November 15, 2021. [Google Scholar]