Abstract
In response to the rapidly evolving coronavirus disease 2019 (COVID-19) pandemic, the All of Us Research Program longitudinal cohort study developed the COVID-19 Participant Experience (COPE) survey to better understand the pandemic experiences and health impacts of COVID-19 on diverse populations within the United States. Six survey versions were deployed between May 2020 and March 2021, covering mental health, loneliness, activity, substance use, and discrimination, as well as COVID-19 symptoms, testing, treatment, and vaccination. A total of 104,910 All of Us Research Program participants, of whom over 73% were from communities traditionally underrepresented in biomedical research, completed 275,201 surveys; 9,693 completed all 6 surveys. Response rates varied widely among demographic groups and were lower among participants from certain racial and ethnic minority populations, participants with low income or educational attainment, and participants with a Spanish language preference. Survey modifications improved participant response rates between the first and last surveys (13.9% to 16.1%, P < 0.001). This paper describes a data set with longitudinal COVID-19 survey data in a large, diverse population that will enable researchers to address important questions related to the pandemic, a data set that is of additional scientific value when combined with the program’s other data sources.
Keywords: COVID-19, diversity, mental health, public health, social determinants of health, social medicine, survey
Abbreviations
- COPE
COVID-19 Participant Experience
- COVID-19
coronavirus disease 2019
- EHR
electronic health record
- PHQ-9
Patient Health Questionnaire-9
- SMS
short message service
In December 2019 the global medical community was alerted about a novel virus, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), which causes coronavirus disease 2019 (COVID-19) (1). Subsequently, the COVID-19 outbreak spread globally, transforming daily lives. Individuals quarantined in their homes or restricted their activities and social interactions over extended periods, and businesses changed their operations virtually overnight. The pandemic resulted in mental, social, and physical health impacts that devastated many individuals, families, communities, and economies (2).
In addition to posing significant risks to physical health, the COVID-19 pandemic exposed social and mental health challenges across the United States. Data collected about these challenges over the course of the unfolding pandemic could provide insight into experiences of and health impacts on diverse populations within the country. Previous surveys and studies have lacked the sampling scale needed to enable well-powered analyses, the demographic diversity necessary to understand impacts across different populations (3–5), or a longitudinal design that enables researchers to follow the full scope and impact of the pandemic over time (6).
The All of Us Research Program is a longitudinal cohort study that aims to accelerate health research and advance precision medicine by collecting and enabling the study of participant data including electronic health record (EHR) data, surveys, whole genome sequences, and more from one million or more people living in the United States (7). This program is well-positioned to respond to the research challenges posed by the COVID-19 pandemic, having enrolled more than 400,000 participants reflecting the broad diversity of the United States. Eighty-three percent of current participants belong to communities traditionally underrepresented in biomedical research, such as people of certain races, sexual and/or gender minorities, older adults, or people of lower income or education levels; 50% of current participants are from self-identified racial and ethnic minority groups (5, 8). Participants enroll into the program using a Web or mobile application, called the All of Us participant portal. There they may review videos about the program, provide their consent to participate, and agree to share their EHR data. They are then invited to complete a series of online surveys, which include information about their basic demographic characteristics, health, family health history, access to care, and other topics. Many participants also provide biological samples (blood, urine, and/or saliva) as well as physical measurements (height, weight, blood pressure, heart rate, and/or waist/hip circumference). These data are collected, deidentified, encrypted, and made available for research studies through the All of Us Research Hub.
The COVID-19 Participant Experience (COPE) survey was one such participant activity, designed to understand how experiences during the pandemic were affecting people’s lives and health, and their communities’ health, and how these experiences changed over time. This survey was designed to be responsive to participant feedback, contribute to pressing research questions related to the COVID-19 pandemic, and include assessments that are not commonly included in EHR data. As part of the All of Us Research Program data set, the COPE survey responses can be linked to ongoing EHR data, genomic data, physical measurements, other demographic and health surveys, and data collected from mobile devices. Combined, these resources enable contextual analyses of responses and further the All of Us Research Program data’s potential to accelerate health research and medical breakthroughs pertinent to the pandemic.
METHODS
Measures
In addition to COVID-19–specific questions from the NIH Common Data Elements Repository and C-19 app (https://covid.joinzoe.com/), the first version of the COPE survey included the following validated instruments: Patient Health Questionnaire (PHQ)-9 (9), Generalized Anxiety Disorder Assessment (GAD)-7 (10), portions of the UK Biobank’s Mental Health and Well-being Questionnaire (11), UCLA Loneliness Scale (12), RAND MOS Social Support Survey Instrument (13), and International Physical Activity Questionnaires (Table 1) (14). In response to feedback from community partners, operational data from the survey rollout, and the evolution of the COVID-19 pandemic, the survey was modified over time, balancing programmatic desire for increased participation rates with relevance and usability of the data for researchers (details of content and operational changes are outlined in Web Table 1, available at https://doi.org/10.1093/aje/kwad035). The strategies incorporated feedback from participants about the length of the initial COPE survey and included simplifying the survey by removing questions that showed little month-to-month variability and removing question sets to enhance survey focus. These actions resulted in reduced burden on survey participants while maintaining scientific integrity.
Table 1.
Instruments Included in COVID-19 Participant Experience Surveys, United States, 2020–2021
| Full Instruments Deployed | Scaled Instruments Deployed | New Questions Deployed |
|---|---|---|
| Henry Ford Social Distancing Survey | CDC/NIH Common Data Element Bank | All of Us Research Program “The Basics” survey |
| Impact of Event Scale–6, based on IES-Revised | Optimism: Life Orientation Test–Revised | All of Us Research Program “Overall Health” survey |
| RAND Medical Outcomes Study Social Support Survey Instrument | Coronavirus Pandemic Epidemiology Consortium Tool | All of Us Research Program “Lifestyle” survey |
| Generalized Anxiety Disorder–7 | UK Biobank Mental Health and Well-being Questionnaire | |
| Patient Health Questionnaire–9 | Columbia COVID-19 Questionnaire | |
| Cohen’s Perceived Stress Scale | International Physical Activity Questionnaires | |
| Brief Resilient Coping Scale | UCLA Loneliness Scale | |
| Everyday Discrimination Scale | Alcohol Use Disorders Identification Test–Concise | |
| Texas Christian University Drug Screen 5 |
Abbreviations: CDC, Centers for Disease Control; COVID-19, coronavirus disease 2019; IES, Impact of Event Scale; NIH, National Institutes of Health; UCLA, University of California, Los Angeles.
Six survey versions were launched with corresponding communications reminders on an approximately monthly schedule and remained active in participant portals for an average of 35.3 days (Table 2). The survey initially consisted of 105 stem questions with a total of 158 items available through branching logic. This version of the survey was used for the first 3 administrations. A major survey redesign was deployed 6 months after initial launch, in November 2020, and subsequent versions reduced the number of stem questions to 27 with a total of 75 available through branching (Table 2, Web Table 1).
Table 2.
COVID-19 Participant Experience Survey Version Specifications, United States, 2020–2021
| COPE Version | Survey Start Date | Survey End Date | No. of Primary Questions | Number of Questions (With Branching Logic) | No. of Days Survey Was Available | Median Completion Time (Minutes: Seconds) |
|---|---|---|---|---|---|---|
| May 2020 | 5/7/20 | 5/29/20 | 105 | 158 | 21 days | 20:43 |
| June 2020 | 6/2/20 | 6/26/20 | 105 | 129 | 23 days | 19:53 |
| July 2020 | 7/7/20 | 09/25/20 | 102 | 168 | 80 days | 19:00 |
| November 2020 | 10/27/20 | 12/3/20 | 27 | 72 | 37 days | 8:58 |
| December 2020 | 12/8/20 | 1/4/21 | 27 | 72 | 27 days | 8:38 |
| February 2021 | 2/9/21 | 3/5/21 | 27 | 75 | 24 days | 8:58 |
Abbreviations: COVID-19, coronavirus disease 2019; COPE, COVID-19 Participant Experience
Resources related to the survey content were embedded in the survey. In addition, participants who selected a nonzero response option on item 9 of the PHQ-9 assessment (denoting any suicidality) were presented a pop-up displaying resources (Web Figure 1) relevant to this risk. These were made available to participants both within the survey itself and within the participant portal.
Study population
The entire All of Us Research Program cohort was invited to complete every administration of the COPE survey provided they had completed the consent process and “The Basics” survey, a baseline questionnaire that collects general profile and demographic information (15). The All of Us Research Program cohort is composed of a voluntary, nonrepresentative sample of adults living across the United States; focus is placed on recruiting individuals from demographically diverse backgrounds. No financial incentives were provided for completion of the COPE surveys.
At the time of analysis, the All of Us Research Program defined “underrepresented in biomedical research” as individuals “with inadequate access to medical care; under the age of 18 or over 65; with an annual household income at or below 200% of the federal poverty level; have less than a high-school education or equivalent; are intersex; identify as a sexual or gender minority; or live in rural or non-metropolitan areas” (5).
The program has not begun to enroll participants under the age of 18 years. Additionally, the All of Us Research Program is currently developing metrics to calculate the number of participants with physical or mental disabilities and participants experiencing barriers to accessing care. Individuals are considered “represented in biomedical research” if they are not part of an underrepresented population as defined above.
Completion and incompletion definitions
Completion rates were calculated as the fraction of eligible participants in each demographic category who submitted a survey, regardless of how many individual questions were answered or skipped. Participants with null values or who skipped or selected “prefer not to answer” on baseline demographic questions were excluded from the analysis for the associated category.
Surveys were considered complete if submitted via the final survey page, regardless of the quantity of survey questions answered. Incomplete COPE surveys were defined as surveys in which a participant had not clicked a “submit” button on the final page. Survey design across all 6 versions was such that after completing all survey questions, participants were prompted through 1–2 additional screens (depending on COPE version) prior to reaching a final screen, which included a “submit” button. COPE surveys did not include an explicit call to action asking participants to click the “submit” button. The additional screens provided survey respondents with a “thank you” message, mental health and COVID-related resources, and COVID-19 health insights.
Communications strategy
The survey communications strategy consisted of automated messages at the time of survey launch, including direct-to-participant emails, short message service (SMS) text messages; in-portal notifications (short alerts that participants can see when they log into their All of Us Research Program account), and push notifications (alerts sent to participants who have downloaded the All of Us Research Program app to their mobile devices). Subsequent emails, SMS, and push notifications were delivered 2 additional times throughout each survey deployment period to participants who had not already completed the relevant survey. Each reminder message was spaced between 6 and 13 days from the most recent reminder message. Throughout the campaign, communications were iterated to include embedded images, targeted textual content, and participant testimonials, attempts to increase the survey completion rates.
The February COPE survey integrated 2 significant changes from previous COPE surveys. First, the survey was accessible through a link in the notifications that allowed most participants to complete the survey without having to recall login information (i.e., direct link and no login-required feature). Second, in addition to being able to complete the survey online on their own, participants were able to work with trained program staff using computer-assisted telephone interviewing, which enabled them to complete the survey over the phone instead of being dependent on digital access to the survey.
Survey and data cataloging
COPE survey concepts were cataloged according to the Observational Medical Outcomes Partnership (OMOP) data model and made publicly searchable via the online Athena repository (https://athena.ohdsi.org/). Formatted REDCap data dictionary versions of the survey instruments are also available for download through the REDCap Consortium’s Shared Library (16, 17), which is freely accessible to researchers from affiliated institutions.
Statistical comparisons of response rates were made with 2-sample proportion z-tests assuming a 2-tailed distribution and carried out with Microsoft Excel, version 16 (Redmond, Washington).
RESULTS
Survey completion rates according to participant demographics
A total of 104,910 out of 342,204 eligible All of Us Research Program participants completed at least 1 COPE survey for an overall response rate of 30.7% (Table 3, Table 4). Participants from communities underrepresented in biomedical research were less likely to complete at least 1 survey (73,787 of 275,077 participants or 26.8%) than participants from communities represented in biomedical research (31,123 of 67,127 or 46.4%, P < 0.001). The survey was completed a total of 275,201 times by 104,910 unique participants. All 6 surveys were completed by 2,879 (4.5%) participants from communities represented in biomedical research and 6,814 (2.6%) participants from communities underrepresented in biomedical research (P < 0.001) (Web Figure 2). A mean of 45,867 responses were received per survey version. Overall, the proportion of participants from communities underrepresented in biomedical research was the same for those that completed at least 1 COPE survey and those that completed all 6 (70.3%).
Table 3.
COVID-19 Participant Experience Survey Completion Rates According to Participant Demographic Characteristics, United States, May 2020 to July 2020
| All of Us Full Cohort | May 2020 | June 2020 | July 2020 | Any Summer Survey (May, June, or July) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Representation | No. Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % |
| All research program participants | 342,204 | 100 | 44,917 | 323,753 | 13.9 | 34,393 | 325,559 | 10.6 | 41,792 | 327,702 | 12.8 | 71,553 | 327,702 | 21.8 |
| Representation | ||||||||||||||
| UBR overall | 275,077 | 80.4 | 30,341 | 261,313 | 11.6 | 23,893 | 262,711 | 9.1 | 28,923 | 264,321 | 10.9 | 49,076 | 264,321 | 18.6 |
| RBR overall | 67,127 | 19.6 | 14,576 | 62,440 | 23.3 | 10,500 | 62,848 | 16.7 | 12,869 | 63,381 | 20.3 | 22,477 | 63,381 | 35.5 |
| UBR sexual orientation | 35,824 | 10.5 | 4,540 | 33,422 | 13.6 | 3,427 | 33,635 | 10.2 | 4,312 | 33,868 | 12.7 | 7,316 | 33,868 | 21.6 |
| UBR gender identity | 15,020 | 4.4 | 1,761 | 14,261 | 12.3 | 1,329 | 14,367 | 9.3 | 1,554 | 14,466 | 10.7 | 2,744 | 14,466 | 19.0 |
| UBR race/ethnicity | 163,752 | 47.9 | 8,280 | 159,107 | 5.2 | 5,934 | 159,554 | 3.7 | 7,695 | 159,972 | 4.8 | 14,491 | 159,972 | 9.1 |
| UBR geography | 22,920 | 6.7 | 3,884 | 20,817 | 18.7 | 3,207 | 21,097 | 15.2 | 3,824 | 21,490 | 17.8 | 6,380 | 21,490 | 29.7 |
| UBR education | 34,260 | 10.0 | 344 | 33,917 | 1.0 | 247 | 33,948 | 0.7 | 387 | 33,976 | 1.1 | 697 | 33,976 | 2.1 |
| UBR income | 96,442 | 28.2 | 4,615 | 93,853 | 4.9 | 3,540 | 94,122 | 3.8 | 4,568 | 94,372 | 4.8 | 8,070 | 94,372 | 8.6 |
| UBR age at consent | 81,367 | 23.8 | 16,126 | 75,522 | 21.4 | 13,638 | 76,152 | 17.9 | 15,529 | 76,909 | 20.2 | 25,062 | 76,909 | 32.6 |
| Age at enrollment | ||||||||||||||
| 18–25 | 26,980 | 7.9 | 1,705 | 25,859 | 6.6 | 1,039 | 25,943 | 4.0 | 1,496 | 26,047 | 5.7 | 2,915 | 26,047 | 11.2 |
| 26–35 | 52,397 | 15.3 | 4,902 | 49,839 | 9.8 | 3,259 | 50,033 | 6.5 | 4,352 | 50,291 | 8.7 | 8,165 | 50,291 | 16.2 |
| 36–45 | 50,565 | 14.8 | 5,203 | 47,953 | 10.9 | 3,511 | 48,164 | 7.3 | 4,757 | 48,421 | 9.8 | 8,577 | 48,421 | 17.7 |
| 46–55 | 61,576 | 18.0 | 6,933 | 58,726 | 11.8 | 5,007 | 59,010 | 8.5 | 6,328 | 59,323 | 10.7 | 11,087 | 59,323 | 18.7 |
| 56–65 | 73,207 | 21.4 | 10,828 | 69,517 | 15.6 | 8,582 | 69,949 | 12.3 | 10,075 | 70,425 | 14.3 | 16,943 | 70,425 | 24.1 |
| 66–75a | 55,637 | 16.3 | 11,729 | 51,508 | 22.8 | 9,912 | 51,971 | 19.1 | 11,445 | 52,512 | 21.8 | 18,161 | 52,512 | 34.6 |
| 76–85a | 18,941 | 5.5 | 3,345 | 17,620 | 19.0 | 2,843 | 17,741 | 16.0 | 3,058 | 17,916 | 17.1 | 5,223 | 17,916 | 29.2 |
| ≥86a | 2,893 | 0.8 | 271 | 2,724 | 9.9 | 239 | 2,741 | 8.7 | 281 | 2,760 | 10.2 | 480 | 2,760 | 17.4 |
| Race | ||||||||||||||
| Asiana | 10,694 | 3.1 | 1,211 | 10,082 | 12.0 | 821 | 10,144 | 8.1 | 1,074 | 10,190 | 10.5 | 2,016 | 10,190 | 19.8 |
| Black or African Americana | 67,817 | 19.8 | 2,240 | 66,474 | 3.4 | 1,646 | 66,641 | 2.5 | 2,183 | 66,778 | 3.3 | 4,057 | 66,778 | 6.1 |
| HLS onlya | 53,143 | 15.5 | 1,890 | 52,103 | 3.6 | 1,315 | 52,198 | 2.5 | 2,087 | 52,290 | 4.0 | 3,657 | 52,290 | 7.0 |
| HLS and Whitea | 4,836 | 1.4 | 611 | 4,500 | 13.6 | 423 | 4,533 | 9.3 | 672 | 4,561 | 14.7 | 1,024 | 4,561 | 22.5 |
| More than 1 racea | 12,750 | 3.7 | 1,374 | 12,029 | 11.4 | 1,057 | 12,080 | 8.8 | 926 | 12,143 | 7.6 | 2,166 | 12,143 | 17.8 |
| HLS and non-White racea | 17,627 | 5.2 | 1,721 | 16,702 | 32.0 | 1,292 | 16,765 | 22.5 | 1,156 | 16,842 | 22.0 | 2,737 | 16,842 | 52.1 |
| Other racea | 9,547 | 2.8 | 597 | 9,158 | 25.8 | 432 | 9,185 | 18.3 | 518 | 9,223 | 21.3 | 987 | 9,223 | 41.8 |
Table 4.
COVID-19 Participant Experience Survey Completion Rates According to Participant Demographic Characteristics, United States, November 2020 to February 2021
| November 2020 | December 2020 | February 2021 | Any Survey Version | All Survey Versions | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Representation | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % |
| All research program participants | 48,314 | 335,019 | 14.4 | 50,841 | 337,325 | 15.1 | 54,944 | 342,204 | 16.1 | 104,910 | 342,204 | 30.7 | 9,693 | 327,702 | 3.0 |
| Representation | |||||||||||||||
| UBR overall | 33,708 | 269,727 | 12.5 | 35,615 | 271,467 | 13.1 | 40,270 | 275,077 | 14.6 | 73,787 | 275,077 | 26.8 | 6,814 | 264,321 | 2.6 |
| RBR overall | 14,606 | 65,292 | 22.4 | 15,226 | 65,858 | 23.1 | 14,674 | 67,127 | 21.9 | 31,123 | 67,127 | 46.4 | 2,879 | 63,381 | 4.5 |
| UBR sexual orientation | 4,816 | 34,871 | 13.8 | 5,199 | 35,190 | 14.8 | 5,439 | 35,824 | 15.2 | 10,877 | 35,824 | 30.4 | 976 | 33,868 | 2.9 |
| UBR gender identity | 1,655 | 14,744 | 11.2 | 1,840 | 14,844 | 12.4 | 1,924 | 15,020 | 12.8 | 3,962 | 15,020 | 26.4 | 352 | 14,466 | 2.4 |
| UBR race/ethnicity | 8,476 | 161,806 | 5.2 | 9,114 | 162,450 | 5.6 | 11,216 | 163,752 | 6.8 | 23,829 | 163,752 | 14.6 | 1,182 | 159,972 | 0.7 |
| UBR geography | 4,303 | 22,203 | 19.4 | 4,533 | 22,463 | 20.2 | 4,948 | 22,920 | 21.6 | 9,221 | 22,920 | 40.2 | 873 | 21,490 | 4.1 |
| UBR education | 447 | 34,087 | 1.3 | 426 | 34,156 | 1.2 | 791 | 34,260 | 2.3 | 1,535 | 34,260 | 4.5 | 39 | 33,976 | 0.1 |
| UBR income | 5,082 | 95,413 | 5.3 | 5,532 | 95,756 | 5.8 | 6,609 | 96,442 | 6.9 | 13,388 | 96,442 | 13.9 | 810 | 94,372 | 0.9 |
| UBR age at consent | 19,360 | 79,119 | 24.5 | 20,142 | 79,827 | 25.2 | 23,290 | 81,367 | 28.6 | 36,724 | 81,367 | 45.1 | 4,475 | 76,909 | 5.8 |
| Age at enrollmenta | |||||||||||||||
| 18–25 | 1,413 | 26,495 | 5.3 | 1,540 | 26,672 | 5.8 | 1,523 | 26,980 | 5.6 | 4,332 | 26,980 | 16.1 | 184 | 26,047 | 0.7 |
| 26–35 | 4,315 | 51,381 | 8.4 | 4,573 | 51,702 | 8.8 | 4,573 | 52,397 | 8.7 | 11,775 | 52,397 | 22.5 | 716 | 50,291 | 1.4 |
| 36–45 | 4,965 | 49,525 | 10.0 | 5,305 | 49,861 | 10.6 | 5,350 | 50,565 | 10.6 | 12,709 | 50,565 | 25.1 | 755 | 48,421 | 1.6 |
| 46–55 | 7,112 | 60,469 | 11.8 | 7,498 | 60,819 | 12.3 | 7,785 | 61,576 | 12.6 | 16,383 | 61,576 | 26.6 | 1,234 | 59,323 | 2.1 |
| 56–65 | 12,038 | 71,829 | 16.8 | 12,723 | 72,274 | 17.6 | 13,462 | 73,207 | 18.4 | 24,677 | 73,207 | 33.7 | 2,557 | 70,425 | 3.6 |
| 66–75b | 14,096 | 54,060 | 26.1 | 14,627 | 54,567 | 26.8 | 16,415 | 55,637 | 29.5 | 26,186 | 55,637 | 47.1 | 3,341 | 52,512 | 6.4 |
| 76–85b | 4,005 | 18,421 | 21.7 | 4,208 | 18,574 | 22.7 | 5,319 | 18,941 | 28.1 | 8,042 | 18,941 | 42.5 | 853 | 17,916 | 4.8 |
| ≥86b | 369 | 2,832 | 13.0 | 366 | 2,849 | 12.8 | 516 | 2,893 | 17.8 | 804 | 2,893 | 27.8 | 53 | 2,760 | 1.9 |
| Racec | |||||||||||||||
| Asianb | 1,091 | 10,464 | 10.4 | 1,130 | 10,533 | 10.7 | 1,245 | 10,694 | 11.6 | 2,902 | 10,694 | 27.1 | 211 | 10,190 | 2.1 |
| Black or African Americanb | 2,516 | 67,320 | 3.7 | 2,699 | 67,490 | 4.0 | 3,594 | 67,817 | 5.3 | 7,297 | 67,817 | 10.8 | 328 | 66,778 | 0.5 |
| HLS onlyb | 1,945 | 52,677 | 3.7 | 2,064 | 52,821 | 3.9 | 2,816 | 53,143 | 5.3 | 6,214 | 53,143 | 11.7 | 221 | 52,290 | 0.4 |
| HLS and Whiteb | 622 | 4,681 | 13.3 | 645 | 4,729 | 13.6 | 697 | 4,836 | 14.4 | 1,471 | 4,836 | 30.4 | 124 | 4,561 | 2.7 |
| More than 1 raceb | 1,403 | 12,433 | 11.3 | 1,557 | 12,550 | 12.4 | 1,670 | 12,750 | 13.1 | 3,418 | 12,750 | 26.8 | 183 | 12,143 | 1.5 |
Table 3.
Continued
| All of Us Full Cohort | May 2020 | June 2020 | July 2020 | Any Summer Survey (May, June, or July) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Representation | No. Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % |
| Unset, skip, or prefer not to answer | 6,175 | 1.8 | 499 | 5,664 | 8.8 | 382 | 5,705 | 6.7 | 503 | 5,751 | 8.7 | 862 | 5,751 | 15.0 |
| White | 172,365 | 50.4 | 36,148 | 159,070 | 22.7 | 28,082 | 160,388 | 17.5 | 33,599 | 162,067 | 20.7 | 56,213 | 162,067 | 34.7 |
| Sex assigned at birth | ||||||||||||||
| Female | 207,317 | 60.6 | 29,366 | 195,836 | 15.0 | 22,224 | 196,995 | 11.3 | 27,491 | 198,325 | 13.9 | 47,062 | 198,325 | 23.7 |
| Male | 130,432 | 38.1 | 15,274 | 123,642 | 12.4 | 11,946 | 124,269 | 9.6 | 14,003 | 125,059 | 11.2 | 24,000 | 125,059 | 19.2 |
| Unset, intersexa, none of these describe mea, skip, or prefer not to answer | 4,455 | 1.3 | 277 | 4,275 | 18.5 | 223 | 4,295 | 13.5 | 298 | 4,318 | 22.3 | 491 | 4,318 | 32.9 |
| Gender identity | ||||||||||||||
| Man | 129,882 | 38.0 | 15,174 | 123,151 | 12.3 | 11,869 | 123,776 | 9.6 | 13,912 | 124,566 | 11.2 | 23,858 | 124,566 | 19.2 |
| Nonbinarya | 1,119 | 0.3 | 211 | 994 | 21.2 | 149 | 1,005 | 14.8 | 202 | 1,015 | 19.9 | 330 | 1,015 | 32.5 |
| Transgendera | 1,031 | 0.3 | 190 | 924 | 20.6 | 144 | 927 | 15.5 | 191 | 937 | 20.4 | 301 | 937 | 32.1 |
| Unset, skip, none of these describe mea, or prefer not to answer | 4,158 | 1.2 | 264 | 3,978 | 20.0 | 220 | 3,997 | 15.3 | 296 | 4,010 | 22.9 | 486 | 4,010 | 35.9 |
| Woman | 206,014 | 60.2 | 29,078 | 194,706 | 14.9 | 22,011 | 195,854 | 11.2 | 27,191 | 197,174 | 13.8 | 46,578 | 197,174 | 23.6 |
| Sexual orientation | ||||||||||||||
| Bisexuala | 12,221 | 3.6 | 1,618 | 11,208 | 14.4 | 1,211 | 11,301 | 10.7 | 1,584 | 11,384 | 13.9 | 2,672 | 11,384 | 23.5 |
| Gaya | 7,853 | 2.3 | 1,293 | 7,302 | 17.7 | 987 | 7,347 | 13.4 | 1,184 | 7,410 | 16.0 | 1,994 | 7,410 | 26.9 |
| Lesbiana | 4,275 | 1.2 | 701 | 3,979 | 17.6 | 512 | 4,004 | 12.8 | 641 | 4,038 | 15.9 | 1,096 | 4,038 | 27.1 |
| None of these describe me, and I’d like to see additional optionsa | 7,102 | 2.1 | 721 | 6,705 | 10.8 | 532 | 6,736 | 7.9 | 672 | 6,771 | 9.9 | 1,164 | 6,771 | 17.2 |
| Straight, i.e., not gay or lesbian | 301,298 | 88.0 | 40,144 | 285,402 | 14.1 | 30,784 | 286,980 | 10.7 | 37,245 | 288,876 | 12.9 | 63,805 | 288,876 | 22.1 |
| Unset, skip, or prefer not to answer | 9,455 | 2.8 | 440 | 9,157 | 4.8 | 367 | 9,191 | 4.0 | 466 | 9,223 | 5.1 | 822 | 9,223 | 8.9 |
Table 3.
Continued
| All of Us Full Cohort | May 2020 | June 2020 | July 2020 | Any Summer Survey (May, June, or July) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Representation | No. Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % |
| Income, $ | ||||||||||||||
| <10,000a | 53,928 | 15.8 | 1,438 | 52,957 | 2.7 | 1,058 | 53,060 | 2.0 | 1,469 | 53,141 | 2.8 | 2,697 | 53,141 | 5.1 |
| 10,000–24,999a | 42,514 | 12.4 | 3,177 | 40,896 | 7.8 | 2,482 | 41,062 | 6.0 | 3,099 | 41,231 | 7.5 | 5,373 | 41,231 | 13.0 |
| 25,000–34,999 | 24,801 | 7.2 | 2,749 | 23,559 | 11.7 | 2,142 | 23,675 | 9.0 | 2,714 | 23,793 | 11.4 | 4,543 | 23,793 | 19.1 |
| 35,000–49,999 | 26,599 | 7.8 | 4,033 | 24,871 | 16.2 | 3,138 | 25,030 | 12.5 | 3,859 | 25,238 | 15.3 | 6,556 | 25,238 | 26.0 |
| 50,000–74,999 | 34,618 | 10.1 | 6,838 | 31,870 | 21.5 | 5,270 | 32,148 | 16.4 | 6,438 | 32,523 | 19.8 | 10,805 | 32,523 | 33.2 |
| 75,000–99,999 | 26,320 | 7.7 | 6,019 | 24,009 | 25.1 | 4,657 | 24,225 | 19.2 | 5,593 | 24,526 | 22.8 | 9,431 | 24,526 | 38.5 |
| 100,000–149,999 | 32,116 | 9.4 | 8,076 | 29,173 | 27.7 | 6,392 | 29,475 | 21.7 | 7,364 | 29,842 | 24.7 | 12,456 | 29,842 | 41.7 |
| 150,000–199,999 | 14,688 | 4.3 | 3,900 | 13,304 | 29.3 | 2,832 | 13,445 | 21.1 | 3,499 | 13,603 | 25.7 | 5,936 | 13,603 | 43.6 |
| ≥200,000 | 20,035 | 5.9 | 5,171 | 18,270 | 28.3 | 3,768 | 18,431 | 20.4 | 4,464 | 18,622 | 24.0 | 7,873 | 18,622 | 42.3 |
| Unset, skip, or prefer not to answer | 66,585 | 19.5 | 3,516 | 64,844 | 5.4 | 2,654 | 65,008 | 4.1 | 3,293 | 65,183 | 5.1 | 5,883 | 65,183 | 9.0 |
| Education | ||||||||||||||
| Never attended school or only attended kindergartena | 523 | 0.2 | <20 | 519 | <20 | 519 | <20 | 519 | <20 | 519 | ||||
| Grades 1–4 (primary)a | 2,974 | 0.9 | <20 | 2,958 | <20 | 2,960 | <20 | 2,960 | <20 | 2,960 | ||||
| Grades 5–8 (middle school)a | 8,010 | 2.3 | 73 | 7,941 | 0.9 | 52 | 7,948 | 0.7 | 79 | 7,952 | 1.0 | 142 | 7,952 | 1.8 |
| Grades 9–11 (Some high school)a | 22,753 | 6.6 | 260 | 22,499 | 1.2 | 191 | 22,521 | 0.8 | 295 | 22,545 | 1.3 | 531 | 22,545 | 2.4 |
| Grade 12 or GED (high-school graduate) | 68,818 | 20.1 | 2,931 | 67,120 | 4.4 | 2,282 | 67,293 | 3.4 | 2,821 | 67,473 | 4.2 | 5,072 | 67,473 | 7.5 |
| Some college, associate’s degree or technical school | 87,928 | 25.7 | 9,791 | 83,220 | 11.8 | 7,431 | 83,701 | 8.9 | 9,521 | 84,223 | 11.3 | 16,155 | 84,223 | 19.2 |
| College 4 years or more (college graduate) | 73,879 | 21.6 | 14,237 | 68,426 | 20.8 | 10,892 | 68,958 | 15.8 | 12,979 | 69,582 | 18.7 | 22,417 | 69,582 | 32.2 |
| Advanced degree (master’s, doctorate, etc.) | 69,332 | 20.3 | 17,368 | 63,321 | 27.4 | 13,371 | 63,889 | 20.9 | 15,838 | 64,657 | 24.5 | 26,786 | 64,657 | 41.4 |
| Unset, skip, or prefer not to answer | 7,987 | 2.3 | 246 | 7,749 | 3.2 | 170 | 7,770 | 2.2 | 246 | 7,791 | 3.2 | 426 | 7,791 | 5.5 |
| Primary language | ||||||||||||||
| English | 321,952 | 94.1 | 44,532 | 303,665 | 14.7 | 34,114 | 305,443 | 11.2 | 41,312 | 307,574 | 13.4 | 70,736 | 307,574 | 23.0 |
| Spanish | 20,251 | 5.9 | 385 | 20,088 | 1.9 | 279 | 20,116 | 1.4 | 479 | 20,128 | 2.4 | 816 | 20,128 | 4.1 |
Abbreviations: COVID-19, coronavirus disease 2019; GED, General Educational Development; HLS, Hispanic, Latino, or Spanish; RBR, represented in biomedical research; UBR, underrepresented in biomedical research.
a Underrepresented in biomedical research groups.
Table 4.
Continued
| November 2020 | December 2020 | February 2021 | Any Survey Version | All Survey Versions | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Representation | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % |
| HLS and non-White raceb | 1,690 | 17,224 | 28.2 | 1,905 | 17,367 | 32.7 | 2,091 | 17,627 | 37.3 | 4,329 | 17,627 | 24.6 | 209 | 16,842 | 3.1 |
| Other raceb | 604 | 9,352 | 25.6 | 666 | 9,422 | 27.1 | 764 | 9,547 | 30.2 | 1,600 | 9,547 | 64.3 | 88 | 9,223 | 3.8 |
| Unset, skip, or prefer not to answer | 602 | 5,951 | 10.1 | 654 | 6,027 | 10.9 | 804 | 6,175 | 13.0 | 1,452 | 6,175 | 23.5 | 89 | 5,751 | 1.5 |
| White | 39,244 | 167,350 | 23.5 | 41,078 | 168,936 | 24.3 | 42,933 | 172,365 | 24.9 | 79,645 | 172,365 | 46.2 | 8,423 | 162,067 | 5.2 |
| Sex assigned at birth | |||||||||||||||
| Female | 31,216 | 202,930 | 15.4 | 32,854 | 204,354 | 16.1 | 35,379 | 207,317 | 17.1 | 68,788 | 207,317 | 33.2 | 5,988 | 198,325 | 3.0 |
| Male | 16,759 | 127,693 | 13.1 | 17,595 | 128,549 | 13.7 | 19,137 | 130,432 | 14.7 | 35,310 | 130,432 | 27.1 | 3,650 | 125,059 | 2.9 |
| Unset, intersexb, none of these describe mea, skip, or prefer not to answer | 339 | 4,396 | 20.9 | 392 | 4,422 | 29.5 | 428 | 4,455 | 26.8 | 812 | 4,455 | 51.0 | 55 | 4,318 | 5.1 |
| Gender identity | |||||||||||||||
| Man | 16,661 | 127,172 | 13.1 | 17,498 | 128,022 | 13.7 | 19,035 | 129,882 | 14.7 | 35,093 | 129,882 | 27.0 | 3,632 | 124,566 | 2.9 |
| Nonbinaryb | 202 | 1,070 | 18.9 | 219 | 1,086 | 20.2 | 238 | 1,119 | 21.3 | 495 | 1,119 | 44.2 | 47 | 1,015 | 4.6 |
| Transgenderb | 178 | 996 | 17.9 | 189 | 1,008 | 18.8 | 204 | 1,031 | 19.8 | 410 | 1,031 | 39.8 | 50 | 937 | 5.3 |
| Unset, skip, none of these describe meb, or prefer not to answer | 326 | 4,092 | 23.9 | 371 | 4,115 | 27.1 | 413 | 4,158 | 24.6 | 794 | 4,158 | 51.3 | 53 | 4,010 | 4.8 |
| Woman | 30,947 | 201,689 | 15.3 | 32,564 | 203,094 | 16.0 | 35,054 | 206,014 | 17.0 | 68,118 | 206,014 | 33.1 | 5,911 | 197,174 | 3.0 |
| Sexual orientation | |||||||||||||||
| Bisexualb | 1,654 | 11,818 | 14.0 | 1,802 | 11,952 | 15.1 | 1,872 | 12,221 | 15.3 | 3,942 | 12,221 | 32.3 | 321 | 11,384 | 2.8 |
| Gaya | 1,421 | 7,647 | 18.6 | 1,526 | 7,713 | 19.8 | 1,521 | 7,853 | 19.4 | 2,929 | 7,853 | 37.3 | 321 | 7,410 | 4.3 |
| Lesbianb | 753 | 4,157 | 18.1 | 778 | 4,191 | 18.6 | 803 | 4,275 | 18.8 | 1,565 | 4,275 | 36.6 | 146 | 4,038 | 3.6 |
| None of these describe mea | 704 | 6,927 | 10.2 | 791 | 6,991 | 11.3 | 871 | 7,102 | 12.3 | 1,765 | 7,102 | 24.9 | 145 | 6,771 | 2.1 |
| Straight, i.e., not gay or lesbian | 43,243 | 295,128 | 14.7 | 45,363 | 297,087 | 15.3 | 49,172 | 301,298 | 16.3 | 93,342 | 301,298 | 31.0 | 8,685 | 288,876 | 3.0 |
| Unset, skip, or prefer not to answer | 539 | 9,342 | 5.8 | 581 | 9,391 | 6.2 | 705 | 9,455 | 7.5 | 1,367 | 9,455 | 14.5 | 75 | 9,223 | 0.8 |
Table 4.
Continued
| November 2020 | December 2020 | February 2021 | Any Survey Version | All Survey Versions | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Representation | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % |
| Income, $ | |||||||||||||||
| <10,000a | 1,578 | 53,498 | 2.9 | 1,732 | 53,640 | 3.2 | 2,137 | 53,928 | 4.0 | 4,695 | 53,928 | 8.7 | 200 | 53,141 | 0.4 |
| 10,000–24,999a | 3,504 | 41,915 | 8.4 | 3,800 | 42,116 | 9.0 | 4,472 | 42,514 | 10.5 | 8,693 | 42,514 | 20.4 | 610 | 41,231 | 1.5 |
| 25,000–34,999 | 3,013 | 24,288 | 12.4 | 3,185 | 24,462 | 13.0 | 3,609 | 24,801 | 14.6 | 6,959 | 24,801 | 28.1 | 597 | 23,793 | 2.5 |
| 35,000–49,999 | 4,384 | 25,912 | 16.9 | 4,570 | 26,128 | 17.5 | 5,217 | 26,599 | 19.6 | 9,673 | 26,599 | 36.4 | 868 | 25,238 | 3.4 |
| 50,000–74,999 | 7,417 | 33,607 | 22.1 | 7,790 | 33,931 | 23.0 | 8,390 | 34,618 | 24.2 | 15,565 | 34,618 | 45.0 | 1,579 | 32,523 | 4.9 |
| 75,000–99,999 | 6,485 | 25,473 | 25.5 | 6,822 | 25,751 | 26.5 | 7,185 | 26,320 | 27.3 | 13,372 | 26,320 | 50.8 | 1,380 | 24,526 | 5.6 |
| 100,000–149,999 | 8,667 | 30,985 | 28.0 | 9,134 | 31,333 | 29.2 | 9,191 | 32,116 | 28.6 | 17,363 | 32,116 | 54.1 | 1,867 | 29,842 | 6.3 |
| 150,000–199,999 | 4,064 | 14,135 | 28.8 | 4,238 | 14,279 | 29.7 | 4,207 | 14,688 | 28.6 | 8,146 | 14,688 | 55.5 | 863 | 13,603 | 6.3 |
| ≥200,000 | 5,322 | 19,366 | 27.5 | 5,461 | 19,584 | 27.9 | 5,360 | 20,035 | 26.8 | 10,799 | 20,035 | 53.9 | 1,086 | 18,622 | 5.8 |
| Unset, skip, or prefer not to answer | 3,880 | 65,840 | 5.9 | 4,109 | 66,101 | 6.2 | 5,176 | 66,585 | 7.8 | 9,645 | 66,585 | 14.5 | 643 | 65,183 | 1.0 |
| Education | |||||||||||||||
| Never attended school or only attended kindergartenb | <20 | 521 | <20 | 521 | <20 | 523 | <20 | 523 | <20 | 519 | |||||
| Grades 1–4 (primary)a | <20 | 2,967 | <20 | 2,970 | 51 | 2,974 | 1.7 | 80 | 2,974 | 2.7 | <20 | 2,960 | |||
| Grades 5–8 (middle school)b | 100 | 7,972 | 1.3 | 84 | 7,989 | 1.1 | 210 | 8,010 | 2.6 | 368 | 8,010 | 4.6 | <20 | 7,952 | |
| Grades 9–11 (some high school)b | 332 | 22,627 | 1.5 | 332 | 22,676 | 1.5 | 526 | 22,753 | 2.3 | 1,080 | 22,753 | 4.7 | 31 | 22,545 | 0.1 |
| Grade 12 or GED (high-school graduate) | 3,335 | 68,097 | 4.9 | 3,518 | 68,359 | 5.1 | 4,392 | 68,818 | 6.4 | 8,562 | 68,818 | 12.4 | 536 | 67,473 | 0.8 |
Table 4.
Continued
| November 2020 | December 2020 | February 2021 | Any Survey Version | All Survey Versions | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Representation | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % | No. | Eligible | % |
| Some college, associate’s degree or technical school | 10,685 | 85,992 | 12.4 | 11,466 | 86,605 | 13.2 | 12,906 | 87,928 | 14.7 | 24,861 | 87,928 | 28.3 | 1,966 | 84,223 | 2.3 |
| College 4 years or more (college graduate) | 14,849 | 71,818 | 20.7 | 15,799 | 72,473 | 21.8 | 16,489 | 73,879 | 22.3 | 31,871 | 73,879 | 43.1 | 3,056 | 69,582 | 4.4 |
| Advanced degree (master’s, doctorate, etc.) | 18,671 | 67,134 | 27.8 | 19,320 | 67,801 | 28.5 | 19,972 | 69,332 | 28.8 | 37,323 | 69,332 | 53.8 | 4,051 | 64,657 | 6.3 |
| Unset, skip, or prefer not to answer | 327 | 7,891 | 4.1 | 312 | 7,931 | 3.9 | 394 | 7,987 | 4.9 | 758 | 7,987 | 9.5 | 45 | 7,791 | 0.6 |
| Primary languaged | |||||||||||||||
| English | 47,912 | 314,825 | 15.2 | 50,429 | 317,114 | 15.9 | 54,049 | 321,952 | 16.8 | 103,244 | 321,952 | 32.1 | 9,656 | 307,574 | 3.1 |
| Spanish | 402 | 20,193 | 2.0 | 412 | 20,210 | 2.0 | 895 | 20,251 | 4.4 | 1,665 | 20,251 | 8.2 | 37 | 20,128 | 0.2 |
Abbreviations: COVID-19, coronavirus disease 2019; GED, General Educational Development; HLS, Hispanic, Latino, or Spanish; RBR, represented in biomedical research; UBR, underrepresented in biomedical research.
a Total number of participants in the age category does not sum to the total participant number because it was pulled from the data resource at a later date after which 8 participants had withdrawn from the program.
b Underrepresented in biomedical research groups.
c Participants are counted in more than 1 category resulting in a sum greater than the total number of participants.
d Total number of participants in the language category does not sum to the total participant number because it was pulled from the data resource at a later date after which a single participant had withdrawn from the program.
Survey completion rates of any COPE survey were significantly lower among eligible self-identified Black (10.8%) and Latino (11.7%) participants compared with White (46.2%) participants (both P < 0.001); among participants with annual incomes below 200% of the individual federal poverty level (13.9%) compared with those with annual incomes above $200,000 (53.9%, P < 0.001); among participants who had less than high-school educations (4.5%) compared with college graduates (48.3%, P < 0.001); and among eligible participants preferring the Spanish language versions (8.2%), compared with eligible participants preferring the English language versions (32.1%, P < 0.001) of the COPE surveys (Tables 3 and 4). Completion rates across geographic lines varied, with total response rates of eligible participants ranging from 10% in Mississippi to 64% in Maine (Figure 1).
Figure 1.
COVID-19 Participant Experience (COPE) survey completion according to state, 2020–2022. COPE survey completion rates varied widely, ranging from 9.6% (Mississippi) to 64.2% (Maine) and showing large regional differences. COVID-19, coronavirus disease 2019.
Survey completion rates according to survey version
The longer survey versions (May, June, and July 2020) garnered an average response rate of 12.4%, while the streamlined version in November and December 2020 and February 2021 had a 15.2% average response rate from participants. The impact of streamlining and simplification of the survey was notable: The highest response month after streamlining (February, 16.1%) had a 2.2% higher response rate than the highest response rate before streamlining (May, 13.9%, P < 0.001). The June survey had the lowest response rate (10.6%). For the first 3 surveys (May, June, and July 2020), median completion time ranged from 19 to 21 minutes, while for November, December, and February surveys, median completion time ranged from 8 to 9 minutes (Table 2).
A total of 113 COPE February surveys were completed using computer-assisted telephone interviewing, 102 (or 90.27%) of which were for participants from communities underrepresented in biomedical research.
Resource provision and addressing questions regarding suicidality
The pop-up displaying resources to participants with any level of suicide risk was displayed 15,571 times across all survey versions, meaning that an average of 5.5% of respondents were shown the pop-up in any survey month (Web Figure 1). Participants from sexual and/or gender minority groups as well as individuals with lower incomes had the highest suicide pop-up display rate, at above 12% for all surveys.
Incomplete survey responses
Overall, across the 6 survey versions, there were a total of 22,166 incomplete surveys. Incompletion rates were not meaningfully different between represented and underrepresented survey respondents (Web Table 2). The February survey, which incorporated the direct link and no-login-required feature, had higher numbers of both survey completions and incompletions compared with earlier versions of the survey. The number of complete surveys increased by 7.7% (from 50,993 to 54,930) between the December and February COPE surveys, and incomplete surveys increased by 380% (from 2,590 to 9,860) (Web Table 2).
DISCUSSION
The COPE survey represents the All of Us Research Program’s first longitudinal survey data collection effort and first data on COVID-19, mental health, and social determinants of health (15). The COPE survey and accompanying analysis represent important steps in understanding the impact of the COVID-19 pandemic in general, and specifically among individuals from communities underrepresented in biomedical research. The COPE survey adds elements to the All of Us Research Program’s data set that could significantly affect health outcomes but are not typically captured in EHR data. These data, along with accompanying program data, are currently available to the research community through the All of Us Researcher Workbench (https://www.researchallofus.org/).
The COPE survey efforts provided insight into the effect of survey modifications on survey completions at scale and across diverse populations. While the general trend was increased survey completions for each iteration of the COPE survey after the second survey, disparities in response rates among demographic groups remained regardless of survey content or implementation changes. It is important to note that a systematic scientific approach to increasing survey completions among disparate populations was not the goal of the COPE survey. Instead, small iterative changes were made to improve the participant experience over time while maintaining the scientific integrity of the overall COPE assessment survey. Some of the changes (e.g., shortening the survey, enhancing email communications, implementing direct links) appeared to increase overall survey completion rates, whereas other changes (location of resources, explicitness of “submit” button) generated unanticipated consequences, such as increased numbers of incomplete survey responses. Computer-assisted telephone interviewing interactions were a pilot method for the program; the relatively small number of completions (113/59,944, or 0.2% of February responders) cannot be credited for the significant increase in COPE February response rates. However, because some participants prefer the telephonic method, the All of Us Research Program has expanded the use of this method for other program surveys.
Given the urgency and emergent nature of the COVID-19 pandemic, the COPE survey was conceived, designed, and administered rapidly. Programmatic prioritization of COPE survey development in response to the surging pandemic enabled streamlining of the regulatory processes and empowered the COPE survey development team in its rapid action. As a result, the timeline from development to survey rollout was markedly shorter than it was for past All of Us Research Program surveys, spanning just over 1 month from concept approval to first survey deployment.
Due to the program’s desire to swiftly respond to the evolving pandemic, some risks were accepted during the development of the COPE survey. Primarily, cognitive and user testing were done only on subsections of the survey, not on the survey as a whole. The program held multiple listening sessions with participants, community partners, and frontline staff, although due to the shortened development timeline for the survey not all recommendations from these listening sessions could be implemented. Finally, while the COPE survey incorporated previously validated scales when possible and was consistent with the NIH Common Data Elements repository (https://cde.nlm.nih.gov/home), not all survey items were validated prior to the survey launch.
One consistent element across all COPE survey deployments was the inclusion of a set of resources available to participants. Given the challenging times in which the COPE survey was deployed and the inclusion of suicide assessment questions, the team deemed it necessary to provide support for those in need. Resources were presented based on a conditional response during the PHQ-9 assessment for select participants, and again at the end of the survey for all participants (Web Table 3). This represents a well-balanced approach and a comprehensive method for supporting participants in a digital manner when asking questions related to suicide.
The burden of the COVID-19 pandemic has disproportionately fallen on persons belonging to racial and ethnic minority groups (18), on individuals with low levels of income and education (19), and on sexual and gender minority communities (20), all of whom are often underrepresented in biomedical research. This survey successfully collected data from these demographic groups, which can be combined with EHR data, genomics, and data collected from mobile devices within the All of Us Researcher Workbench. Compared with other large longitudinal studies and surveys, COPE survey respondents represent a significantly more diverse population. For example, the Framingham Heart Study consists of predominantly White participants, leading to known racial and ethnic disparities in predictive capabilities (4). Similarly, the Nurses’ Health Study includes women, with self-identified racial and ethnic minorities comprising 14% of respondents (21), and the UK Biobank is less than 5% non-White (22).
COPE survey respondents were not demographically representative of the All of Us Research Program cohort overall, being more frequently White and older, with higher levels of income and education. Similar patterns in survey completion rates by demographic category have been seen in other surveys completed by All of Us Research Program participants. Notably, survey completion patterns for other postenrollment surveys mirror those of the COPE survey (56.2% response rates in secondary surveys from participants belonging to communities well represented in biomedical research vs. 33.3% response rates in secondary surveys from participants belonging to communities traditionally underrepresented in biomedical research) (15). However, these other surveys were largely completed before the pandemic, with different timing and communications strategies. While not directly comparable, lower response rates from certain underrepresented communities are commonly observed in patient surveys (23, 24) and might be affected by a variety of factors for different individuals and groups, including justified distrust of the medical establishment, divided attention due to competing priorities and concerns, lack of access to stable internet, or lower internet literacy. Programmatic outreach and retention strategies must address the challenges faced by underrepresented communities in order to lower barriers to completion and improve equitable access. Additionally, the COVID-19 pandemic may have created unique disproportionate barriers to survey completion, such as lack of time or interest due to life stressors such as loss of employment, essential-worker status, lack of childcare, lack of stable housing, and COVID-19-related illness, among others (25). The All of Us Research Program data set does not currently include detailed data on occupation, which is likely correlated with COVID-19–related risks and behaviors and may be considered a strong unmeasured confounder. The All of Us Research Program intends to collect occupational history in the future, enabling future data releases to include participant occupation history.
Highly differential survey response rates among demographic groups indicate that COPE survey results may not be generalizable to the full All of Us Research Program cohort or US population. Response rate disparities are compounded among cross-tabulations of multiple demographic categories (data not shown but available in the All of Us Researcher Workbench). The application of appropriate weighting would reduce but not remove the impact of this nonresponse bias due to the inability to correct for unmeasured factors. Researchers using COPE data in the All of Us Researcher Workbench will need to apply appropriate statistical techniques to manage missingness and bias. When conducted with appropriate caution, descriptive analysis leading to the development of research questions for future studies may be the most obvious use case for COPE survey data. Causal analyses require particular attention to understanding the limitations of the COPE data set, although the All of Us Research Program is well positioned to support researchers in this process. Research support is offered in the All of Us Researcher Workbench through educational resources (written documentation, videos, tutorials, interactive forums); sample, tutorial, and example notebooks; virtual office hours offering 1:1 support from data scientists; and a review board of experienced scientists and statisticians available to review workbooks for potential bias or stigmatizing research. As of this publication, a demonstration project led by a team of experienced researchers using COPE survey data was completed, and a tutorial notebook guiding users through analysis is being made available to researchers. Demonstration projects are intended to provide an example of minimally biased, high-integrity research and methods for less experienced researchers. Additionally, external resources are available to help researchers understand and apply techniques to appropriately handle selection bias and to conduct bias analysis or other techniques (26–28). The longitudinal complexity of the data and strong temporal trends in COVID-19 risk and associated health behaviors admittedly require rigorous analytical treatment and acknowledgement of limitations by researchers using the data set.
Survey completion strategies we found to be successful in the COPE survey and plan to retain for future surveys include sending direct, no-login-required links from email and SMS messages; reducing survey length; incorporating testimonials and personal stories as part of the messaging platform; and working across the All of Us Research Program consortium to build national-to-local outreach in ways that are integrated into the larger communications strategy. Many of these strategies were suggested by community partners and participants during listening sessions specific to the COPE survey.
Regarding incomplete surveys, we hypothesize that aspects of the user experience design, such as the positioning of resource pages prior to the “submit” screen to accommodate the direct link functionality, affected and perhaps increased “functionally complete” incomplete survey rates (incomplete surveys where participants had responded to all survey questions but had not clicked the final “submit” button). An improved survey design would be to present resources and “thank you” pages after completion of the last question.
Insights into what factors—biological, environmental, and social, among others—might make individuals more vulnerable or more resilient in periods of increased and prolonged stress such as during a pandemic are limited due to the relative infrequency of pandemics as well as the logistical and technical challenges associated with studying these emergent situations. Responding to the historic crisis posed by the COVID-19 pandemic, the All of Us Research Program swiftly developed and deployed COPE surveys to provide participants with an opportunity to share their experiences. Data regarding these experiences can also be combined with additional programmatic data such as genetics, EHR data, and other survey responses.
The COPE survey represents a successful survey implementation and iteration to collect longitudinal data and to improve response rates across a large and diverse cohort, offering lessons to other groups proposing similar surveys. A total of 65,339 participants filled out the COPE survey at least twice (9,693 filling out all 6), providing a substantial longitudinal data set spanning 10 months following the initial emergence of SARS-CoV-2 in the United States. The deployment timeline and midstream pause enabled assessment of longitudinal effects within the survey cohort, developed and enacted strategies to increase the number of responses, and assessed the effectiveness of survey and communications changes for increasing the number and diversity of participant responses. Efforts to reduce completion bias in future All of Us Research Program surveys include focused communications, outreach, and accessibility improvements.
In addition to being the first longitudinal survey deployed by the All of Us Research Program, the COPE surveys represent the first significant contribution of participant data on COVID-19 pandemic experiences, mental health, and social determinants of health. As the program evolves, its aim is to enhance and increase the prevalence of data on mental health and social determinants of health that participants may share. These contributions will build an increasingly robust data set, one generated by a diverse cohort of participant partners that is available to researchers as a foundation for future medical breakthroughs.
Supplementary Material
ACKNOWLEDGMENTS
Author affiliations: All of Us Research Program, National Institutes of Health, Bethesda, Maryland, United States (Claire E. Schulkey, Tamara R. Litwin, Genevieve Ellsworth, Heather Sansbury, Geeta Bhat, Rubin Baskir, Joshua Denny, Holly A. Garriock); Center for Health Policy and Health Services Research, Henry Ford Health System, Detroit, Michigan, United States (Brian K. Ahmedani); Behavioral Health Services, Henry Ford Health System, Detroit, Michigan, United States (Brian K. Ahmedani); Center for Precision Psychiatry, Department of Psychiatry, Massachusetts General Hospital, Boston, Massachusetts, United States (Karmel W. Choi, Jordan W. Smoller); Psychiatric and Neurodevelopmental Genetics Unit, Center for Genomic Medicine, Massachusetts General Hospital, Boston, Massachusetts, United States (Karmel W. Choi, Jordan W. Smoller); Department of Internal Medicine, The Ohio State University, Columbus, Ohio, United States (Robert M. Cronin); NIH BRAIN Initiative, National Institutes of Health, Bethesda, Maryland, United States (Yasmin Kloth); Vibrent Health, Fairfax, Virginia, United States (Alan W. Ashbeck, Scott Sutherland, Mark Begale); Vanderbilt Institute for Clinical and Translational Research, Vanderbilt University Medical Center, Nashville, Tennessee, United States (Brandy M. Mapes, Kayla Marginean, Keri Ann Wolfe, Aymone Kouame, Francis Ratsimbazafy); The Scripps Research Institute, La Jolla, California, United States (Paula King); Vagelos Life Sciences & Management, University of Pennsylvania, School of Arts and Sciences and Wharton, Philadelphia, Pennsylvania, United States (Carmina Raquel); CareEvolution Healthcare Technology, Ann Arbor, Michigan, United States (Zach Bornemeier, Kyle Neumeier); and Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland, United States (Kelly A. Gebo).
C.E.S., T.R.L., G.E., and H.S. contributed equally to this work. K.A.G. performed this work in her role as the Chief Medical and Scientific Officer of the All of Us Research Program (2018–2020).
This work was supported by the National Institutes of Health (grant 5U2COD023196-04 to the All of Us Data and Research Center at Vanderbilt University Medical Center).
COPE survey materials and data are available to researchers through the All of Us Research Hub, which contains publicly accessible English language versions of the survey questions, source instrumentation, and aggregate response data for each of the survey versions (https://databrowser.researchallofus.org/survey/covid-19-participant-experience). Row-level response data was made available to registered researchers through the Researcher Workbench on the All of Us Research Hub beginning in November 2020, accompanied by reference materials and virtual quality-assurance sessions. Data refreshes will add additional response data to the Researcher Workbench over time. As of August 2021, the survey codebook has been downloaded 191 times in English and 24 times in Spanish across all versions.
We thank the entire All of Us Research Program consortium, including participant ambassadors, for their incalculable contributions to this study.
Conflict of interest: none declared.
REFERENCES
- 1. Huang C, Wang Y, Li X, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet. 2020;395(10223):497–506. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Usher K, Durkin J, Bhullar N. The COVID-19 pandemic and mental health impacts. Int J Ment Health Nurs. 2020;29(3):315–318. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Fry A, Littlejohns TJ, Sudlow C, et al. Comparison of sociodemographic and health-related characteristics of UK Biobank participants with those of the general population. Am J Epidemiol. 2017;186(9):1026–1034. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Tsao CW, Vasan RS. Cohort profile: the Framingham Heart Study (FHS): overview of milestones in cardiovascular epidemiology. Int J Epidemiol. 2015;44(6):1800–1813. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Mapes BM, Foster CS, Kusnoor SV, et al. Diversity and inclusion for the All of Us Research Program: a scoping review. PloS One. 2020;15(7):e0234962. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Thombs BD, Bonardi O, Rice DB, et al. Curating evidence on mental health during COVID-19: a living systematic review. J Psychosom Res. 2020;133:110113. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. All of Us Research Program Investigators . The “All of Us” Research Program. N Engl J Med. 2019;381(7):668–676. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Ramirez AH, Gebo KA, Harris PA. Progress with the All of Us Research Program: opening access for researchers. JAMA. 2021;325(24):2441–2442. [DOI] [PubMed] [Google Scholar]
- 9. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606–613. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Spitzer RL, Kroenke K, Williams JB, et al. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006;166(10):1092–1097. [DOI] [PubMed] [Google Scholar]
- 11. Davis KAS, Coleman JRI, Adams M, et al. Mental health in UK Biobank—development, implementation and results from an online questionnaire completed by 157 366 participants: a reanalysis. BJPsych Open. 2020;6(2):e18. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Russell D, Peplau LA, Ferguson ML. Developing a measure of loneliness. J Pers Assess. 1978;42(3):290–294. [DOI] [PubMed] [Google Scholar]
- 13. Hays RD, Sherbourne CD, Mazel R. User's Manual for the Medical Outcomes Study (MOS) Core Measures of Health-Related Quality of Life. Santa Monica, CA: RAND Corporation; 1995. [Google Scholar]
- 14. Lee PH, Macfarlane DJ, Lam TH, et al. Validity of the international physical activity questionnaire short form (IPAQ-SF): a systematic review. Int J Behav Nutr Phys Act. 2011;8(1):115. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Cronin RM, Jerome RN, Mapes B, et al. Development of the initial surveys for the All of Us Research Program. Epidemiology. 2019;30(4):597–608. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Vanderbilt University . REDCap shared library, https://projectredcap.org/resources/library/. Accessed November 2, 2022.
- 17. Harris PA, Taylor R, Thielke R, et al. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–381. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Webb Hooper M, Nápoles AM, Pérez-Stable EJ. COVID-19 and racial/ethnic disparities. JAMA. 2020;323(24):2466–2467. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Wiemers EE, Abrahams S, AlFakhri M, et al. Disparities in vulnerability to complications from COVID-19 arising from disparities in preexisting conditions in the United States. Res Soc Stratif Mobil. 2020;69:100553. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Ruprecht MM, Wang X, Johnson AK, et al. Evidence of social and structural COVID-19 disparities by sexual orientation, gender identity, and race/ethnicity in an urban environment. J Urban Health. 2021;98(1):27–40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Bao Y, Bertoia ML, Lenart EB, et al. Origin, methods, and evolution of the three Nurses' Health Studies. Am J Public Health. 2016;106(9):1573–1581. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. UK Biobank . Data-Field 21000, https://biobank.ctsu.ox.ac.uk/crystal/field.cgi?id=21000. Updated July 19, 2022. Accessed November 2, 2022.
- 23. Sheldon H, Graham C, Pothecary N, et al. Increasing Response Rates Amongst Black and Minority Ethnic and Seldom Heard Groups: A Review of Literature Relevant to the National Acute Patients' Survey. Oxford, UK: Picker Institute; 2007. [Google Scholar]
- 24. United States Census Bureau . 2020 Census Barriers, Attitudes, and Motivators Study (CBAMS) Survey and Focus Groups Report Findings Presentation. Hillcrest Heights, MD: US Department of Commerce; 2019. [Google Scholar]
- 25. Tai DBG, Shah A, Doubeni CA, et al. The disproportionate impact of COVID-19 on racial and ethnic minorities in the United States. Clin Infect Dis. 2021;72(4):703–706. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology. 2004;15(5):615–625. [DOI] [PubMed] [Google Scholar]
- 27. Lash TL, Vanderweele TJ, Haneuse S, et al. Modern Epidemiology. Philadelphia, PA: Wolters Kluwer; 2021. [Google Scholar]
- 28. VanderWeele TJ. Explanation in Causal Inference: Methods for Mediation and Interaction. Cary, NC: Oxford University Press; 2015. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.