Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Jul 1.
Published in final edited form as: Epidemiology. 2019 Jul;30(4):597–608. doi: 10.1097/EDE.0000000000001028

Development of the Initial Surveys for the All of Us Research Program

Robert M Cronin 1,2, Rebecca N Jerome 3, Brandy Mapes 3, Regina Andrade 3, Rebecca Johnston 3, Jennifer Ayala 3, David Schlundt 4, Kemberlee Bonnet 4, Sunil Kripalani 5,6,7, Kathryn Goggins 5,6,7, Kenneth A Wallston 8, Mick P Couper 9,10, Michael R Ellitt 11,12, Paul Harris 3, Mark Begale 13, Fatima Munoz 14, Maria Lopez-Class 15, David Cella 16, David Condon 16, Mona AuYoung 17, Kathleen M Mazor 18, Steve Mikita 19, Michael Manganiello 20, Nicholas Borselli 20, Stephanie Fowler 15, Joni L Rutter 15, Joshua C Denny 1, Elizabeth W Karlson 21, Brian K Ahmedani 22, Chris O’Donnell 23,24; the Vanderbilt University Medical Center Pilot Team, and the Participant Provided Information Committee
PMCID: PMC6548672  NIHMSID: NIHMS1526300  PMID: 31045611

Abstract

Background

The All of Us Research Program is building a national longitudinal cohort and collecting data from multiple information sources (e.g., biospecimens, electronic health records (EHRs), and mobile/wearable technologies) to advance precision medicine. Participant-provided information, collected via surveys, will complement and augment these information sources. We report the process used to develop and refine the initial three surveys for this program.

Methods

The All of Us survey development process included: (1) prioritization of domains for scientific needs, (2) examination of existing validated instruments, (3) content creation, (4) evaluation and refinement via cognitive interviews and online testing, (5) content review by key stakeholders, and (6) launch in the All of Us electronic participant portal. All content was translated into Spanish.

Results

We conducted cognitive interviews in English and Spanish with 169 participants, and 573 individuals completed online testing. Feedback led to over 40 item content changes. Lessons learned included: (1) validated survey instruments performed well in diverse populations reflective of All of Us; (2) parallel evaluation of multiple languages can ensure optimal survey deployment; (3) recruitment challenges in diverse populations required multiple strategies; and (4) key stakeholders improved integration of surveys into larger Program context.

Conclusions

This efficient, iterative process led to successful testing, refinement, and launch of three All of Us surveys. Reuse of All of Us surveys, available at http://researchallofus.org, may facilitate large consortia targeting diverse populations in English and Spanish to capture participant-provided information to supplement other data, such as genetic, physical measurements, or data from EHRs.

Keywords: Health surveys, Precision medicine, Questionnaires, Cohort studies, Cognitive interviews, Online surveys

Introduction

Precision medicine is an approach to identifying risk factors, etiology, treatment, and prevention of disease emphasizing variability in an individual’s genes, environment, and lifestyle. Precision medicine research has yielded numerous discoveries regarding genomic influences on diseases and drug responses1–4, which has resulted in improved patient outcomes5,6. These studies have historically leveraged phenotypic information from physiologic measurements, health surveys, and bioassays. Surveys have been an important part of observational research for decades. Prominent studies such as the National Health and Nutrition Examination Survey7, the National Health Interview Survey8, and UK Biobank9,10, have added substantially to biomedical knowledge in part through survey-based capture of exposures and outcomes. Research has also demonstrated the power of routine clinical information extracted from the electronic health record (EHR); others are also exploring the utility of newer modalities such as wearable biosensors and environmental data11–23. Collectively, these strategies hold promise for more precise ways to identify patterns in large datasets and, ultimately, understanding of factors that contribute to health and disease.

The All of Us Research Program (All of Us) has the goal of enrolling a longitudinal cohort of at least one million participants reflecting the rich diversity of the U.S. population24–26, prioritizing groups historically underrepresented in biomedical research27–29. A broad range of data will be collected to describe each participant, including surveys, EHRs, biospecimens, physical exams, wearable technologies, and geospatial and environmental sources. These data will enable creation of a robust research resource to facilitate deep exploration of biologic, clinical, social, environmental, and behavioral determinants of health and disease.

All of Us will use surveys to collect data directly from a diverse cohort of participants across the socio-economic spectrum with different racial/ethnic backgrounds and for whom English may not be their primary language. These surveys are primarily designed to complement information collected from other sources such as EHRs. EHRs do not routinely collect data on many social, environmental, or behavioral determinants of health, or these variables may be difficult to find within the text of EHR clinical notes30, thus the additional data from surveys have the potential to enable researchers to test hypotheses at a greater scope and level of precision than before.

Surveys also offer an opportunity to engage participants, explicitly recognizing the importance of the information that individuals can share over time about their own socio-demographic characteristics, health, and other factors. In All of Us, participants answer survey questions via an online participant portal, available on both computers and mobile devices, which guide participants through surveys to be completed, while keeping track of their progress and engaging them throughout the life of the Program. This paper describes the rigorous process for rapid development, testing, and refinement of the initial three surveys included in the launch of All of Us in 2018.

Methods

The All of Us survey development process comprised a multidisciplinary collaboration and included: 1) preparatory work such as choosing the scientific domains of focus and examining existing validated instruments to create content; 2) content testing and refinement; and 3) stakeholder agreement on final versions for program deployment (Figures 1 and 2).

Figure 1:

Figure 1:

Open in a new tab

Full process of creation, testing, and approval of the English survey materials.

Figure 2:

Figure 2:

Open in a new tab

Full process of creation, testing, and approval of the Spanish survey materials.

Drafting initial surveys

This process began in October 2015 with an NIH All of Us Protocol Working Group convened by the National Institutes of Health (NIH) Office of the Director, consisting of approximately 25 NIH staff and non-NIH scientists from a variety of disciplines. The working group drafted a list of desired survey domains to be included in All of Us. Priorities of the domains were determined based on a set of criteria (eTable 1). This group identified and reviewed existing validated survey instruments for use in the All of Us Research Program from October 2015 to March 2016, including 19 instruments from large research endeavors (Table 1). Instrument questions were chosen based on validation evidence in diverse populations, usage in other national studies, and access to their use (lack of copyright).

Table 1:

Original question sources for the All of Us surveys at launch

All of Us Survey Module: Number of Questions Included
Question Source Overall Health The Basics Lifestyle
ATS-DLD-78 6a
Audit-C 3
BHLS 3
BRFSS 8 1a
BS 4
2020 US Census 8
GENIUSS 4
MVP 7a
NESARC 1
NHANES 4
NHCHC 1
NHIS 2
NM-Assist 2
PATH 8
PLCO 2
PROMIS Measures 10
TUS-CPS 5a
UK Bio 7 1
VA Homeless Screening 1
New questions developed by All of Us 0 2 0
Total number of questions for All of Us Surveya 24 31 22
Open in a new tab

Key: Audit-C: Alcohol Use Disorders Identification Test; BHLS - Brief Health Literacy Screen; BHLS: Brief Health Literacy Scale; BRFSS - Behavioral Risk Factor Surveillance System; BS: Blood bank screening; GENIUSS: Gender Identity in U.S. Surveillance; HINTS - Health Information National Trends Survey; MVP: Million Veteran Program; NESARC - National Epidemiologic Survey on Alcohol and Related Conditions; NHANES - National Health and Nutrition Examination Survey; NHCHC: National Health Care for the Homeless Council; NHIS - National Health Interview Survey; NM-Assist: National Institute on Drug Abuse-Modified Alcohol, Smoking, and Substance Involvement Screening Test; PATH - Population Assessment of Tobacco and Health Study; PLCO - Prostate, Lung, Colorectal, and Ovarian Cancer Screening Trial; PROMIS Patient-Reported Outcomes Measurement Information System; TUS-CPS - Tobacco Use Supplement - Current Population Survey; UK Bio - UK Biobank; VICS - Vanderbilt Inpatient Cohort Study

a

Questions were sourced from more than one original instrument. The final question was a composite of question(s) from multiple instruments modified to best fit All of Us.

The All of Us Pilot team was formed in early 2016 to continue survey development. The Pilot team employed qualitative methods to test and refine survey content to ensure optimal implementation, described below. This interdisciplinary team consisted of approximately 20 experts representing a range of fields: (1) cognitive sciences and electronic/non-electronic survey design and related methodology; (2) health literacy and effective health communication; (3) engagement and inclusion of underrepresented minority populations, such as African Americans and Hispanics/Latinos, Lesbian, Gay, Bisexual, Transgender, Queer or Questioning, and Intersex (LGBTQI), and rural residents; (4) information science, for evidence review and synthesis; (5) racially and ethnically diverse bilingual research staff proficient in English and Spanish; and (6) biomedical informatics, for guidance around data collection methods, data harmonization, and quality and quantity of information.

The Pilot team built upon the materials originally drafted by the Working Group. The three surveys prioritized by the Program for initial testing and included as part of the national launch of All of Us recruitment, guided by the information most typically collected at time of enrollment in large cohort studies included: The Basics (sociodemographic questions), Overall Health (general health overview), and Lifestyle (tobacco, alcohol, and drug questions).

The team created a standardized process, described below, to test each survey’s readability, response format and options, participants’ confidence in their ability to answer the questions accurately, and other key issues, such as redundancy and sensitivity.

As part of this phase of work, the Pilot team also collated all source instruments from which content was drawn to create a metadata database. The metadata database was used to ensure that principles of survey development were adhered to, including: (1) maintenance of scientific validity of the original survey instruments, (2) conversion of interviewer-led surveys to an online participant-completed format, and (3) usage of validated items widely used in the same diverse populations recruited to the Program. The Pilot team reviewed the metadata for gaps related to these guiding principles.

As the Program grew, survey creation became the responsibility of the Participant-Provided Information Committee. This committee is a group of approximately 10–15 experts tasked with overseeing the entire survey development cycle, comprised of the types of domain experts mentioned above, as well as representatives from participant recruitment sites. This committee has been instrumental in refining content as well as determining new areas of focus. Table 2 describes prioritized domains and current development status.

Table 2:

Survey topics currently undergoing development and/or publicly released in All of Us

Released to Participants at Enrollment To be Released after Enrollment In Development Future survey topics not yet in development
1. Basics
2. Lifestyle
3. Overall Health 		4. Family Health History
5. Healthcare Access and Utilization
6. Personal Medical History		 7. Diet
8. Disability
9. Environmental Exposure and Occupational Health
10. Mental Health and Substance Use
11. Physical Activity
12. Social Determinants of Health
13. Medications		 14. Oral Health
15. Pain
16. Reproductive Health
17. Sleep
Open in a new tab

Testing sample

The Pilot team developed an approach for qualitative testing, including cognitive interviewing and web-based testing. Participants were recruited from a robust and diverse pool of ~5,000 individuals who had previously expressed interest in helping to develop All of Us content., as well as through additional methods such as in-person events with community partners. These recruitment approaches enabled the Pilot team to reach minority populations and those with limited access to the internet, as well as rural and urban areas.

Qualitative and quantitative testing

Cognitive interviewing employed a “think aloud” approach, exploring participants’ understanding about each question and response option. Probes, including interviewer prompts and follow-up questions, were used as needed to elicit detailed discussion of each item31–33. Cognitive interviews were conducted in person or by Skype31,33,34 based on participant preference. Transcripts of interview audio recordings were analyzed qualitatively. A hierarchical coding system was developed to thematically assess the qualitative data. Quotes supporting the different codes were recorded as part of the analysis. Interviews continued until saturation was achieved.

Online qualitative and quantitative testing, done in parallel with cognitive interviews, intended to augment the data from cognitive interviews by engaging a larger number of diverse participants. Online testing employed the Research Electronic Data Capture (REDCap)35 system hosted in a secure online environment, which required that each participant login using unique account credentials to help ensure only the proper participant completed the survey. The online qualitative testing allowed participants to first answer the question, then share feedback about the clarity, understandability, and sensitivity of each question and its response options (Figure 3); in quantitative testing, used to estimate the time to complete each survey, participants only answered questions without a feedback option. Where appropriate, analysis of these results included exploratory factor analysis using principal components extraction and varimax rotation, frequency statistics, and assessment of internal consistency using Cronbach’s alpha36.

Figure 3: Online testing through Pilot Expression of Interest website.

Figure 3:

Open in a new tab

REDCap was used as the engine for questionnaire testing. For this project, we developed the ability to answer a feedback questionnaire for any question.

Translation and testing in Spanish

The surveys were developed and tested in both English and Spanish (Figures 1 & 2). Prior to testing, the English surveys were translated into Spanish. If the English language questions had an equivalent Spanish version from the original source instrument, then that Spanish equivalent was used as the primary translated version. However, if the English version did not have an existing Spanish translation, those questions went through a multi-step translation process developed by the Pilot team. This included primary translation by a third-party company (MotionPoint), during which a certified professional translator translated the content into Spanish, followed by secondary review of the translation by community reviewers to ensure the content was easily understandable to a wide range of Spanish speakers with diverse educational attainment and country of origin. This secondary review also identified opportunities for improving readability.

For the first Spanish language iteration of the initial surveys (The Basics, Overall Health and Lifestyle), secondary translation review was conducted by a small group of Program stakeholders representing diverse geographical native Spanish speakers ranging from Mexico, Spain, Venezuela, and El Salvador to ensure that the Spanish language review captured wide ethno-geographic and cultural differences. This group also reviewed the final recommendations for the translated Spanish materials used in the Program.

Testing of the Spanish-language materials primarily included recruiting diverse Spanish-speaking participants across the U.S. at in-person events and connecting with community organizations embedded within Latino communities. The approach for identifying any critical changes in the Spanish-language materials followed the same process outlined for the English versions.

Review by key Program stakeholders

The Pilot team reviewed and proposed further refinements to the materials based on review of the original validated instruments and analyses from the cognitive interviews and online testing. The survey materials and their accompanying proposed list of changes were then reviewed by key Program leadership, including the Participant-Provided Information and Steering Committees. Feedback from these groups was used to inform a refined iteration of each survey, leading to a second round of testing (Figures 1 & 2, eTable 2). After this second round of testing, the Pilot team provided recommendations that led to a final review and approval by key Program leadership.

Implementation in participant portal

In the final step of development, the three surveys were incorporated into the All of Us participant portal, the mechanism for survey self-administration in the program. These surveys are available to participants once they sign into the portal, allowing for completion in their own time.

Results

Process overview

The testing and refinement process successfully implemented for the first three modules in the program included two rounds of cognitive interviews and online testing, complemented by expert review and input after each round of testing. The results of each of the development phases is described in further detail below.

First round of testing and revisions

The first round of testing focused on both qualitative and quantitative testing of the English survey materials. Cognitive interviews were conducted with 74 participants. A total of 337 individuals provided online feedback on the English surveys, of which 225 provided qualitative feedback, and 112 provided quantitative feedback (Table 3). Qualitative testing revealed major themes that demonstrated issues with clarity, understandability, and sensitivity (eTable 3). A summary of the estimated completion time data calculated based on this round of testing is included in Table 4.

Table 3:

Demographic characteristics of testing cohort for launch surveys before and after content revisions. The percentage for each category is in parentheses.

Preliminary testing before content revisions (Summer 2016) Final testing after content revisions (Summer 2017)
Cognitive Interview Online Testing Cognitive Interview Online Testing
Total n=74 (%) n=337 (%) n=95 (%) n=236 (%)
Gender
Male 20 (27) 150 (45) 26 (27) 102 (43)
Female 54 (73) 185 (55) 63 (66) 130 (55)
Missing 0 (0) 2 (0) 6 (6) 4 (2)
Race/Ethnicity
White 38 (51) 312 (93) 32 (34) 128 (54)
African American 19 (26) 4 (1) 12 (13) 67 (28)
Asian or Pacific Islander 4 (5) 3 (8) 3 (3) 18 (8)
Hispanic/Latino 5 (7) 8 (2) 53 (56) 21 (9)
aOther 5 (7) 10 (3) 4 (4) 34 (14)
Prefer not to answer 1 (1)  0 (0) 0 (0) 3 (1)
Age
18–29 7 (9) 46 (14) 24 (25) 45 (19)
30–49 32 (43) 90 (27) 24 (25) 82 (35)
50–64 22 (30) 113 (34) 21 (22) 64 (27)
65+ 13 (18) 86 (26) 21 (22) 41 (17)
Missing 0 (0) 2 (0) 5 (5) 4 (2)
Education
Bachelor’s degree or higher 54 (75) 210 (62) 33 (35) 112 (47)
Some college, but not a Bachelor’s degree 18 (25) 117 (35) 26 (27) 78 (33)
High school/GED or less 2 (0) 10 (3) 33 (35) 40 (17)
Missing 0 (0) 0 (0) 3 (3) 6 (3)
Residential Size
City(Large urban area-more than 50,000 people) 58 (81) 187 (55) 49 (52) 154 (65)
Town(Small urban area-between 2,500–50,000 people) 10 (14) 125 (37) 36 (38) 63 (27)
Rural Community(less than 2,500 people) 4 (6) 42 (12) 5 (5) 16 (7)
Missing 2 (0) 17 (5) 5 (5) 3 (1)
Other Demographic Characteristics
Sexual and/or Gender Minorities 5 (7) 23 (7) 14 (15) 40 (17)
Individuals with Low Health Literacy 38 (51) 148 (44) 27 (28) 12 (5)
Individuals with Physical Disabilities 13 (18) 32 (9) 8 (8) 16 (7)
Individuals with 3 or More Chronic Conditions 13 (18) 99 (29) 11 (12) 45 (19)
Individuals with Low Tech Literacy 6 (8) 33 (10) 19 (20) 7 (3)
Individuals with no internet access 1 (1) 2 (1) 0 (0) 2 (1)
Individuals with Children under 18 24 (33) 61 (18) 28 (29) 51 (22)
Individuals whose preferred language is Spanish 0 (0) 0 (0) 47 (49) 0 (0)
Open in a new tab
a

Other race/ethnicity counts include those individuals who self-identify as more than one race/ethnicity.

Table 4:

Completion time data for surveys included at All of Us Research Program launch: prior to content revisions and afterwards. Participants may not be required to answer certain questions if they are not applicable (e.g., a non-smoker will skip the questions asking about smoking history).

Number of questions (min – max) Preliminary data before content revisions (Summer 2016) Final data after content revisions (Summer 2017)
The All of Us Research Program Survey Median (min:sec) Range (min:sec) Median (min:sec) Range (min:sec)
The Basics 26 – 29 10:29 2:44 – 27:51 6:30 2:00–17:51
Lifestyle 22 – 57 2:47 0:33 – 8:00 1:23 0:17–7:05
Overall Health 14 – 24 2:48 1:20 – 7:36 2:00 0:14–6.06
Open in a new tab

Second round of testing and revisions

The second round of cognitive interviewing focused on expanding participant diversity and testing both English (n = 48) and Spanish (n = 47) versions (total n = 95). Online testing in English was completed with 236 participants (qualitative testing n=159, quantitative testing n=77) (Table 3). There were no new substantial changes in items that emerged from online qualitative testing in the second round. See Table 4 for estimated completions times. Review of the data led to very minor recommendations involving changing wording and adding clarifying language to some of the questions.

The yield of this development process: final revisions to survey content

A summary of the over 40 recommended changes based on English language testing is included in Table 5. Minor changes consisted of small edits in phrasing of a question or response options (e.g., converting questions that were originally administered by interviewers in the parent survey to a format appropriate for self-administration). Major changes included more substantive modifications to a question and/or response options when noteworthy concerns were expressed by participants during qualitative testing. For example, the questions about race, ethnicity, and gender that were included in the first round of qualitative testing were modified and retested in the second round of testing based on input from participants and key Program stakeholders. These modifications led to refinement of our approach to race and ethnicity. We ultimately leveraged extensive testing completed by the US Census Bureau, which found that a combined question for capturing participant-reported race and ethnicity was the strongest approach for gathering complete and accurate data37. The recommendations for changes in the Spanish language version included minor changes that implemented more simplified and common terminology. In addition, testing Spanish-language materials resulted in feedback and recommendations that went beyond translation, and included conceptual changes to questions to make them more understandable and accessible to their specific communities.

Table 5:

Question modification summaries of Round 1 and Round 2

Question modification summary from round 1 (Summer 2016)
Nature of change as compared with original survey instrument
All of Us Survey Domain Total questions (n) No change or N/A n (%) Minor change n (%) Major change n (%)
The Basics 36 14 (39) 20 (56) 2 (5)
Overall Health 14 14 (100) 0 0
Lifestyle 44 21 (48) 22 (50) 1 (2)
Question modification summary from round 2 (Summer 2017)
Nature of change as compared with original survey instrument
All of Us Survey Domain Total questions (n) No change or N/A Minor change n (%) Major change n (%)
The Basics 31 0 5 (16) 0
Overall Health 24 0 2 (8) 0
Lifestyle 22 0 1 (2) 0
Open in a new tab

We explored concerns from members of the Program, such as about asking potentially sensitive questions regarding topics like gender identity and sexual orientation. Our testing did not confirm anticipated concerns but instead found appreciation of being asked these questions among participants, including those self-identifying as sexual and gender minorities.

All questions were finalized by the committees and are summarized with their original instruments in eDocuments 1–6. The summary of iterative changes for both English and Spanish are summarized in eTable 4.

Exploratory factor analysis

While much of the data in the three surveys were not amenable to further statistical analysis, an exploratory factor analysis was appropriate for two subcomponents (i.e. PROMIS, Brief Health Literacy Screen) of the English version of Overall Health. This factor analysis showed three distinct factors with coefficients alpha being 0.92 for General Physical Health, 0.81 for Emotional Health, and 0.55 for Health Literacy (eTable 5).

Discussion

We created an iterative process leveraging diverse experts to develop and refine materials for collection of participant-provided information for All of Us that is applicable to a diverse audience, leverages existing validated surveys, and supports English and Spanish. We initially launched three surveys and are following this model for ongoing development of future surveys in All of Us.

Lessons learned

Other large consortia such as the Million Veterans Program38 and the U.K. Biobank9,10 have included survey materials as a core data component; however, approaches for combining survey items from multiple sources in the context of a large research program are sparse. We learned several lessons useful for future Program development, as well as others undertaking similar work. First, we discovered generally minor issues with clarity and sensitivity for some module questions drawn from previously validated survey instruments. While questions from certain validated instruments, such as the PROMIS Global Health Scale39 and the Brief Health Literacy Screen40, performed well in our testing (eTable 3), we implemented minor modifications for other items (eTable 4). Modifications included adding examples to clarify a question (e.g., providing the number of cigarettes in a pack); new response options (e.g., e-cigarettes); and explanatory text before some items. As many of the module questions are derived from existing national surveys, we wanted to avoid substantial wording revisions, as this would compromise established scientific validity. Instead, we targeted all newly created explanatory text to the 5th grade level and focused on identifying areas within the question text where minor revisions could lead to major improvement in readability. Second, we explored potential issues of concern to All of Us. In fact, similar to other findings41, participants expressed appreciation for the Program’s recognition of the importance of asking about gender identity and sexual orientation, which initially concerned members of the Program. Third, we successfully leveraged prior extensive participant testing within national programs, such as the US Census37, as our findings echoed the Census Bureau’s observations that a combined approach for querying race and ethnicity is more aligned with the way participants identify themselves and, thus, allows gathering more granular data. Fourth, we found that many validated questions from studies that may have been developed for a specific population, such as the California Teachers Study42, only required minor wording changes, but otherwise performed well in a more diverse population. Finally, collaborating with consortium experts led to substantial improvements in integrating survey materials into the larger context of the Program. This collaboration led to a volume of survey items reasonable to ask participants to complete at enrollment, within the larger scope of enrollment activities such as consent and physical exam.

Our experience emphasizes the value of systematically vetting multiple languages to ensure optimum survey deployment, as well as the importance of testing with various dialects of a specific language. Creating and assessing surveys in both English and Spanish led to improved clarity, while maintaining concordance between translations. Testing in both languages ensured alignment in understability and accessibility for the English and Spanish versions. This experience continues to inform the consortium’s work regarding the complexities of testing surveys in different languages. The translation process included review by members from multiple Spanish speaking regions to develop surveys understood by Spanish speakers from different regions. Instituting multiple versions of Spanish surveys is worth consideration, however, we did not find evidence within our interviews with participants that indicated this need.

We learned multiple lessons about cognitive interviewing strategies in this population. First, this testing process yielded valuable lessons learned regarding strategies for overcoming challenges in recruiting under-represented or hard to reach populations. Over time, the Pilot team expanded recruitment to include methods such as in-person recruitment within the community, facilitating increased enrollment of harder-to-reach populations such as those without internet access, those with lower educational attainment, racial/ethnic minority populations, and Spanish-speaking participants. This face-to-face community interaction was an effective method to recruit certain populations that were more likely to engage in research opportunities through a trusted and familiar entity such as an established community organization. As echoed by others these methods require more time, but are critical to ensure inclusion of diverse populations43–45. Second, cognitive interview probes authored by content experts allowed a deeper understanding of what needed to be explored to ensure accurate answers. Third, using a web application to perform interviews over the internet provided a cost-effective way to interview people who were not located locally. Fourth, our sample size was large, and we achieved saturation prior to reaching our full sample size. Because enrollment of underrepresented populations is a major aim of the All of Us Research Program, we wanted to be thorough in our attempts to include key populations in this component of the program planning. Future efforts of this type will likely include smaller enrollment targets. However, based on our experiences, we feel that the size and demographic characteristics of the sample are important and should be tailored to best fit the scope and goals of the project. For example, researchers interested in exploring readability and comprehension alone may reach saturation with a small sample. In instances where sensitivity and individual perspectives are also being sought, researchers may want to consider a larger and more diverse sample size.

Limitations

We acknowledge several limitations: (1) Our testing did not include all populations underrepresented in biomedical research. While we included very important populations, including sexual and gender minorities, those of low socioeconomic status, and Spanish-only speakers, gaps may require additional testing with other populations.(2) The speed at which initial survey development work proceeded limited our ability initially to include individuals from certain key groups, such as lower educational attainment. However, this was addressed during online testing and out second round of cognitive interviews. (3) The validity of combining questions from existing instruments was not thoroughly tested. To mitigate this, we intentionally minimized changes from existing instruments and executed entire scales or sets of questions about a topic area from a single instrument. (4) Email communication was the main method of contact for study recruitment, potentially limiting our reach of those less comfortable with technology. (5) Small sample size and potential lack of representativeness limits generalization of our exploratory factor analysis. (6) Online surveying poses an inherent risk, however small, of receiving a response from someone other than the intended participant. Finally, (7) we tested primarily on a computer platform and not on different technology platforms such as smartphones. Our plan will be to test future surveys on different platforms.

Deployment and Future Directions

To achieve the ambitious scope and scale of All of Us, surveys will need to be modular to not overwhelm participants, accessible on a digital platform so surveys can be completed in a variety of settings, available to be completed at a participant’s own pace, and engaging so participants continue to contribute after initial enrollment in the Program.

Building upon this successful process for development of these initial materials, All of Us is developing other surveys (Table 2). Future areas of interest for the program will include roll out of these surveys, engaging participants to complete these additional surveys, and repeat administration of some surveys to reflect participant changes over time. As the Program evolves, survey development will continue to be a core activity. As described above, the marriage of scientific value, engagement, and participant experience will remain an important consideration for future work in this area. Future testing and integration with other sources of information, such as EHR data, mobile sensors, and a range of technology platforms, genetics, and physical measurements, will help enhance the value and completeness of All of Us data for future hypothesis exploration. Further validation of our findings related to the surveys in the launch of the Program will be needed to ensure what we found in our initial testing holds true for the larger cohort. Translation into other languages and the testing of these translations will also be an important area of activity as the Program grows.

Participant-provided information is a critically important part of the data that will power All of Us. The survey questions and response options must be carefully documented and communicated to researchers in a way that makes these data accessible and easy to integrate into external aspects of clinical research. An All of Us survey codebook is under development and is being designed to map All of Us survey items to standard vocabularies that can help align these data with EHR data where possible (e.g., for diagnoses). Finally, All of Us surveys reflect a mixture of questions from validated instruments in the public domain, accompanied by supplementary questions that are not currently publicly available. All of Us will publish these surveys and their metadata, as they become available, at http://researchallofus.org.

Conclusions

This flexible process combined multidisciplinary expertise with Program leadership input and proven methods to create and refine surveys that are appropriate for use in the diverse participant population of All of Us. The process, which was generalizable across multiple survey domains, formed a firm roadmap for the development and testing of future materials. Other large consortia that target a diverse population in multiple languages could employ this process to create surveys that supplement other data sources, such as genetics and EHR data.

Prioritization and generation of All of Us surveys will continue in close collaboration with representatives across the NIH Institutes and Centers to ensure item integrity and scientific validity and help ensure that questions reflect the mission of the NIH to improve health outcomes broadly. Other surveys may be driven by participant interest. Participant engagement and experience will also be important components of future development. Gathering and incorporating participant-provided information in a systematic way will enhance the scientific validity and breadth of information obtained from All of Us, leading to exciting new advances in the era of precision medicine.

Supplementary Material

Supplemental Digital Content

Acknowledgements:

We wish to thank our participants who have joined All of Us and contributed to participant-provided information; helped refine early materials; engaged in the development and evaluation of the surveys, and provided other ongoing feedback. We thank the countless other co-investigators and staff across all awardees and partners without which All of Us would not have achieved our current goals. All of Us is supported by grants through the National Institutes of Health Office of the Director: Regional Medical Centers: 1 OT2 OD026549; 1 OT2 OD026554; 1 OT2 OD026557; 1 OT2 OD026556; 1 OT2 OD026550; 1 OT2 OD 026552; 1 OT2 OD026553; 1 OT2 OD026548; 1 OT2 OD026551; 1 OT2 OD026555; IAA #: AOD 16037; Federally Qualified Health Centers: HHSN 263201600085U; Data and Research Center: 5 U2C OD023196; Biobank: 1 U24 OD023121; The Participant Center: U24 OD023176; Participant Technology Systems Center: 1 U24 OD023163; Communications and Engagement: 3 OT2 OD023205; 3 OT2 OD023206; Community Partners: 1 OT2 OD025277; 3 OT2 OD025315; 1 OT2 OD025337; 1 OT2 OD025276. Pilot Team: OT2 OD023132, OT2 OD023132–02S1, K23HL141447.

Original NIH Protocol Working Group Members: Teri Manolio, Rebekah Rasooly, Josephine Briggs, Rory Collins, Caroline Dilworth, Montserrat Garcia-Closas, Germaine Buck Lewis, Daniel Masys, Jean Olson, Tim Peakman, Bill Riley, Joni Rutter, Paul Sorlie, Elizabeth Wagner, Debbie Winn, Dana Wolff, Kathleen Meister, Luenda Charles, Michael Gaziano, Emily Harris, Carolyn Hutter, Sue Krebs-Smith, Sara Luckhaupt, Steven Moore, Kathleen Merikangas, Dale Sandler, Amy Subar, Jennifer Thornton, Gordon Willis, Ellen O’Donnell

Pilot Team Members: Charles Mouton, Katherine Donato, Taneya Koonce, Sheila Kusnoor, Marcia Epelbaum, Mallory Blasingame, Robert Cronin, Rebecca Jerome, Brandy Mapes, Regina Andrade, Rebecca Johnston, David Schlundt, Kemberlee Bonnet, Sunil Kripalani, Kathryn Goggins, Ken Wallston, Mick Couper, Michael Elliott

All of Us Participant Provided Committee Members: Paul Harris, Christopher/Chris O’Donnell, Stephanie Fowler, James McClain, Brian Ahmedani, Regina Andrade, Maria Argos, Mona AuYoung, Mark Begale, Bartali Benedetta, Pedro Rey Biel, Louise Bier, Marnie Bloom, Nicolas Borselli, Clinton Brawner, Beth Calhoun, David Cella, Carmen Chinea, David Condon, Rob Cronin, Julien Dedier, Olivier Elemento, Chris Foster, David Garcia, Holly Garriock, John Jackicic, Rebecca Jerome, Daozhong Jin, Christine Johnson, Christine Joseph, Elizabeth Karlson, Mike Kellen, Michelle Kienholz, Andrea LaCroix, Elizabeth Lee, Maria Lopez-Class, Michael Manganiello, Brandy Mapes, Heather Marino, Fernando Martin-Sanchez, Kathy Mazor, Wanda Montalvo, Fatima Munoz, Jyotishman Pathak, Susan Redline, Carolina Rodriguez-Cook, Heather Sansbury, David Schlundt, August Slater, Vicki Smith, Carolina Stamoulous, Susan Tirhi, Rhonda Trousdale, Febe Wallace, Joyce Winkler, Jennifer Worthington, Jennifer Yttriz, Alvaro Alonso

Spanish Translation Committee: James McClain, Michael Stokes, Regina Andrade, Oscar Beita, Dianne Beltran, Amaris Castellanos, Carmen Chinea, Rene Covarrubias, Alfredo Ramirez Clark, Alvaro Donayre, Angelica Espinoza, Iliana Faries, Sergio Fernandez-Gonzales, Pablo Gejman, Viridiana Johnson, Maria Lopez-Class, Elizabeth Lugo, Fernando Martin-Sanchez, Rima Matsumoto, Daniel Mompoint, Jorge Navarrete, Vijay Rayanker, Carolina Rodriguez-Cook, Carolina Stamoulous, Gregory Talavera, Solomon Torres, Sujel Valentin, Emma Viera, Carmen Zaldivar, Alejandra Zapien-Hidalgo, Flor McKinley, Ilse Salinas, Janence Ortiz, Janisse Mercado, Jose Guadalupe Martinez Lopez, Marcela Gaitán, Marcia Lobos

We also wish to thank All of Us Research Program Direct Eric Dishman as well as our partners as Verily, Vibrent, Scripps, and Leidos.

“Precision Medicine Initiative, PMI, All of Us, the All of Us logo, and The Future of Health Begins with You are service marks of the U.S. Department of Health and Human Services.”

Sources of financial support: This work was funded by the National Institutes of Health (https://allofus.nih.gov/) OT2OD023132 (RMC, RNJ, BMM, RA, RJ, JA, DGS, KRB, SK, KG, KAW, MPC, MRE, PAH, and JCD), U2COD023196 (JCD), R01AG0532641U24OD023163–01 (MJB), HHSM500201200008I/HHSN263201800208U (FAM), U2CCA186878 (DC), U24 OD023176 (MA), 3OTOD 024610–01S1 (KMM), OT2OD023206 (MM, NB), 1OT2OD026553 (EWK), and OT2OD026550 (BKA). Additional support included the National Heart, Blood, and Lung Institute (https://www.nhlbi.nih.gov/) K23HL141447 (RMC), United States Veteran’s Administration Healthcare System (CJO), the National Institute on Aging (https://www.nia.nih.gov/) U2C AG060426–01 (DMC). Program staff of the funding organization, National Institutes of Health, co-contributed to this work through conceptualization, methodology, validation, project administration and review and editing of this manuscript.

Footnotes

Conflicts of interest: The authors have no conflicts of interest

Data availability: All relevant data are within the manuscript and its Supporting Information files or are available from http://researchallofus.org.

Research Ethics and Informed Consent:

The research assessments described in this manuscript were approved by the Institutional Review Board (IRB) of the All of Us Research Program. Those participants that completed cognitive interview gave oral consent to do so.

References

  • 1.MacArthur J, Bowler E, Cerezo M, Gil L, Hall P, Hastings E, Junkins H, McMahon A, Milano A, Morales J, Pendlington ZM, Welter D, Burdett T, Hindorff L, Flicek P, Cunningham F, Parkinson H. The new NHGRI-EBI Catalog of published genome-wide association studies (GWAS Catalog). Nucleic Acids Res 2017;45(D1):D896–D901. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Delaney JT, Ramirez AH, Bowton E, Pulley JM, Basford MA, Schildcrout JS, Shi Y, Zink R, Oetjens M, Xu H, Cleator JH, Jahangir E, Ritchie MD, Masys DR, Roden DM, Crawford DC, Denny JC. Predicting clopidogrel response using DNA samples linked to an electronic health record. Clin Pharmacol Ther 2012;91(2):257–63. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Schildcrout JS, Denny JC, Bowton E, Gregg W, Pulley JM, Basford MA, Cowan JD, Xu H, Ramirez AH, Crawford DC, Ritchie MD, Peterson JF, Masys DR, Wilke RA, Roden DM. Optimizing drug outcomes through pharmacogenetics: a case for preemptive genotyping. Clin Pharmacol Ther 2012;92(2):235–42. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Wilke RA, Ramsey LB, Johnson SG, Maxwell WD, McLeod HL, Voora D, Krauss RM, Roden DM, Feng Q, Cooper-Dehoff RM, Gong L, Klein TE, Wadelius M, Niemi M, Clinical Pharmacogenomics Implementation C. The clinical pharmacogenomics implementation consortium: CPIC guideline for SLCO1B1 and simvastatin-induced myopathy. Clin Pharmacol Ther 2012;92(1):112–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Mallal S, Phillips E, Carosi G, Molina JM, Workman C, Tomazic J, Jagel-Guedes E, Rugina S, Kozyrev O, Cid JF, Hay P, Nolan D, Hughes S, Hughes A, Ryan S, Fitch N, Thorborn D, Benbow A, Team P-S. HLA-B*5701 screening for hypersensitivity to abacavir. N Engl J Med 2008;358(6):568–79. [DOI] [PubMed] [Google Scholar]
  • 6.Cavallari LH, Lee CR, Beitelshees AL, Cooper-DeHoff RM, Duarte JD, Voora D, Kimmel SE, McDonough CW, Gong Y, Dave CV, Pratt VM, Alestock TD, Anderson RD, Alsip J, Ardati AK, Brott BC, Brown L, Chumnumwat S, Clare-Salzler MJ, Coons JC, Denny JC, Dillon C, Elsey AR, Hamadeh IS, Harada S, Hillegass WB, Hines L, Horenstein RB, Howell LA, Jeng LJB, Kelemen MD, Lee YM, Magvanjav O, Montasser M, Nelson DR, Nutescu EA, Nwaba DC, Pakyz RE, Palmer K, Peterson JF, Pollin TI, Quinn AH, Robinson SW, Schub J, Skaar TC, Smith DM, Sriramoju VB, Starostik P, Stys TP, Stevenson JM, Varunok N, Vesely MR, Wake DT, Weck KE, Weitzel KW, Wilke RA, Willig J, Zhao RY, Kreutz RP, Stouffer GA, Empey PE, Limdi NA, Shuldiner AR, Winterstein AG, Johnson JA, Network I. Multisite Investigation of Outcomes With Implementation of CYP2C19 Genotype-Guided Antiplatelet Therapy After Percutaneous Coronary Intervention. JACC Cardiovasc Interv 2018;11(2):181–191. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Massey JT, Moore TF, Tadros W, Parsons V. Design and estimation for the National Health Interview Survey 1985–94. VITAL AND HEALTH STATISTICS. SERIES 2: DATA EVALUATION AND METHODS RESEARCH 1989(110):1–33. [PubMed] [Google Scholar]
  • 8.Parsons VL, Moriarity CL, Jonas K, Moore TF, Davis KE, Tompkins L. Design and estimation for the national health interview survey, 2006–2015. 2014. [PubMed] [Google Scholar]
  • 9.Sudlow C, Gallacher J, Allen N, Beral V, Burton P, Danesh J, Downey P, Elliott P, Green J, Landray M. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS medicine 2015;12(3):e1001779. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Sudlow C, Gallacher J, Allen N, Beral V, Burton P, Danesh J, Downey P, Elliott P, Green J, Landray M, Liu B, Matthews P, Ong G, Pell J, Silman A, Young A, Sprosen T, Peakman T, Collins R. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med 2015;12(3):e1001779. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Jensen PB, Jensen LJ, Brunak S. Mining electronic health records: towards better research applications and clinical care. Nat Rev Genet 2012;13(6):395–405. [DOI] [PubMed] [Google Scholar]
  • 12.Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood) 2014;33(7):1123–31. [DOI] [PubMed] [Google Scholar]
  • 13.Hripcsak G, Albers DJ. Next-generation phenotyping of electronic health records. J Am Med Inform Assoc 2013;20(1):117–21. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Martin-Sanchez F, Verspoor K. Big data in medicine is driving big changes. Yearb Med Inform 2014;9:14–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Baig MM, GholamHosseini H, Moqeem AA, Mirza F, Linden M. A Systematic Review of Wearable Patient Monitoring Systems - Current Challenges and Opportunities for Clinical Adoption. J Med Syst 2017;41(7):115. [DOI] [PubMed] [Google Scholar]
  • 16.Uddin MZ, Khaksar W, Torresen J. Ambient Sensors for Elderly Care and Independent Living: A Survey. Sensors (Basel) 2018;18(7). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Rucco R, Sorriso A, Liparoti M, Ferraioli G, Sorrentino P, Ambrosanio M, Baselice F. Type and Location of Wearable Sensors for Monitoring Falls during Static and Dynamic Tasks in Healthy Elderly: A Review. Sensors (Basel) 2018;18(5). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Camomilla V, Bergamini E, Fantozzi S, Vannozzi G. Trends Supporting the In-Field Use of Wearable Inertial Sensors for Sport Performance Evaluation: A Systematic Review. Sensors (Basel) 2018;18(3). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Johansson D, Malmgren K, Alt Murphy M. Wearable sensors for clinical applications in epilepsy, Parkinson’s disease, and stroke: a mixed-methods systematic review. J Neurol 2018;265(8):1740–1752. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Papi E, Koh WS, McGregor AH. Wearable technology for spine movement assessment: A systematic review. J Biomech 2017;64:186–197. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Rovini E, Maremmani C, Cavallo F. How Wearable Sensors Can Support Parkinson’s Disease Diagnosis and Treatment: A Systematic Review. Front Neurosci 2017;11:555. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Kozitsina AN, Svalova TS, Malysheva NN, Okhokhonin AV, Vidrevich MB, Brainina KZ. Sensors Based on Bio and Biomimetic Receptors in Medical Diagnostic, Environment, and Food Analysis. Biosensors (Basel) 2018;8(2). [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Mukhopadhyay SC. Wearable sensors for human activity monitoring: A review. IEEE sensors journal 2015;15(3):1321–1330. [Google Scholar]
  • 24.Hudson K, Lifton R, Patrick-Lake B. The precision medicine initiative cohort program-building a research foundation for 21st century medicine. Precision Medicine Initiative (PMI) Working Group Report to the Advisory Committee to the Director, ed 2015. [Google Scholar]
  • 25.Health NIo. Program Overview - All of Us | National Institutes of Health. https://www.ncbi.nlm.nih.gov/pubmed/.
  • 26.Investigators AoURP. The All of Us Research Program – Building a Foundation for 21st Century Precision Health. In: Personal Communication Cronin R, ed, 2018. [Google Scholar]
  • 27.Ford JG, Howerton MW, Lai GY, Gary TL, Bolen S, Gibbons MC, Tilburt J, Baffi C, Tanpitukpongse TP, Wilson RF, Powe NR, Bass EB. Barriers to recruiting underrepresented populations to cancer clinical trials: a systematic review. Cancer 2008;112(2):228–42. [DOI] [PubMed] [Google Scholar]
  • 28.Heller C, Balls-Berry JE, Nery JD, Erwin PJ, Littleton D, Kim M, Kuo WP. Strategies addressing barriers to clinical trial enrollment of underrepresented populations: a systematic review. Contemp Clin Trials 2014;39(2):169–82. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.National Academy of Sciences NAoE, Institute of Medicine Committee on Underrepresented G, the Expansion of the S, Engineering Workforce P. The National Academies Collection: Reports funded by National Institutes of Health Expanding Underrepresented Minority Participation. Washington (DC): National Academies Press (US) National Academy of Sciences, 2011. [Google Scholar]
  • 30.Hripcsak G, Forrest CB, Brennan PF, Stead WW. Informatics to support the IOM social and behavioral domains and measures. Journal of the American Medical Informatics Association : JAMIA 2015;22(4):921–924. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Beatty PC, Willis GB. Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly 2007;71(2):287–311. [Google Scholar]
  • 32.DeMuro CJ, Lewis SA, DiBenedetti DB, Price MA, Fehnel SE. Successful implementation of cognitive interviews in special populations. Expert review of pharmacoeconomics & outcomes research 2012;12(2):181–187. [DOI] [PubMed] [Google Scholar]
  • 33.Willis GB. Cognitive interviewing: A tool for improving questionnaire design Sage Publications, 2004. [Google Scholar]
  • 34.Dean E, Head B, Swicegood J. Virtual cognitive interviewing using Skype and Second Life. Social Media, Sociality, and Survey Research 2013:107–132. [Google Scholar]
  • 35.Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42(2):377–81. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Cronbach L Coefficient alpha and the internal structure of tests. psychometrika 1951;16(3):297–334. [Google Scholar]
  • 37.Jones NA. Update on the US Census Bureau’s race and ethnic research for the 2020 Census. 2017.
  • 38.Gaziano JM, Concato J, Brophy M, Fiore L, Pyarajan S, Breeling J, Whitbourne S, Deen J, Shannon C, Humphries D. Million Veteran Program: a mega-biobank to study genetic influences on health and disease. Journal of clinical epidemiology 2016;70:214–223. [DOI] [PubMed] [Google Scholar]
  • 39.Cella D, Riley W, Stone A, Rothrock N, Reeve B, Yount S, Amtmann D, Bode R, Buysse D, Choi S. The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of clinical epidemiology 2010;63(11):1179–1194. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Wallston KA, Cawthon C, McNaughton CD, Rothman RL, Osborn CY, Kripalani S. Psychometric properties of the brief health literacy screen in clinical practice. Journal of general internal medicine 2014;29(1):119–126. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Haider AH, Schneider EB, Kodadek LM, Adler RR, Ranjit A, Torain M, Shields RY, Snyder C, Schuur JD, Vail L, German D, Peterson S, Lau BD. Emergency Department Query for Patient-Centered Approaches to Sexual Orientation and Gender Identity : The EQUALITY Study. JAMA Intern Med 2017;177(6):819–828. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Parikh-Patel A, Allen M, Wright WE, California Teachers Study Steering C. Validation of self-reported cancers in the California Teachers Study. Am J Epidemiol 2003;157(6):539–45. [DOI] [PubMed] [Google Scholar]
  • 43.Alvarez RA, Vasquez E, Mayorga CC, Feaster DJ, Mitrani VB. Increasing minority research participation through community organization outreach. West J Nurs Res 2006;28(5):541–60; discussion 561–3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Williams MM, Meisel MM, Williams J, Morris JC. An interdisciplinary outreach model of African American recruitment for Alzheimer’s disease research. Gerontologist 2011;51 Suppl 1:S134–41. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Gauthier MA, Clarke WP. Gaining and sustaining minority participation in longitudinal research projects. Alzheimer Dis Assoc Disord 1999;13 Suppl 1:S29–33. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Digital Content
NIHMS1526300-supplement-Supplemental_Digital_Content.docx (96.9KB, docx)

RESOURCES