Abstract
Background
Traditionally, clinical studies rely on brick-and-mortar sites to recruit participants. Newer technology-based studies have utilized non-traditional virtual methods that can potentially recruit more diverse populations and shorten recruitment timelines. This manuscript aims to quantify how sample metrics across three virtual studies compare to traditionally recruited samples, as a first step in building an empirical evidence base for the experience of participant recruitment in virtual studies.
Methods
We conducted a systematic search of the literature using PubMed to identify relevant studies conducted in the United States in cognitive health, diabetes, and hypertension (which we called comparator studies) to compare to three virtual studies. For each included study, we extracted participant demographic characteristics and information on recruitment methods and timing. Two investigators independently extracted this data, compared results for consistency, and contacted comparator study authors for clarifications. Characteristics for measurement included age, sex, race/ethnicity, states represented, recruitment time, and recruitment rate.
Results
We identified 19 comparator studies. Virtually recruited samples were slightly younger, had more female participants, and were split on enrollment of racial minorities as compared to comparator studies. Virtually recruited samples were more diverse geographically and recruited faster.
Conclusions
Virtual recruitment may enhance efficiency and enable more individuals to participate in clinical research. To our knowledge, this is the first rigorous and replicable study comparing participant demographic characteristics and recruitment metrics between virtual and traditional recruitment methodologies. Future research should compare a wider range of studies on other metrics such as overall cost of recruitment and quality of participants.
Keywords: Sample representativeness, Recruitment, Virtual studies, Remote patient-centered trials
1. Introduction
It is impractical to enroll all members of a target population into a clinical research study [1]. Instead, the goal is to recruit a representative sample of the target population to enhance the external validity or generalizability of study findings to the population of interest [2,3]. A thoughtfully selected sample is important; this need, combined with other recruitment and enrollment hurdles, impacts research timelines. Participant recruitment and enrollment is often the most time-consuming aspect of the clinical research process [4]. An analysis of clinical studies found that recruitment can take up to 30% of the research timeline, and is the leading cause of missed clinical trial deadlines [5]. Another estimate suggests that up to 80% of clinical trials do not meet participant enrollment timelines [6,7].
Today, participant recruitment into clinical research occurs at a limited set of physical sites, and relies on in-person screening and consent. Studies struggle to enroll the target sample within the specified timeframe [8,9]. A recent study found that nearly 40% of sites for clinical studies under-enroll, and 11% fail to enroll one participant [10]. On-site recruitment requires time to identify enough eligible participants, and additional time to consent and enroll participants; this time yields higher costs and delays answering pressing research questions. A 2014 study estimated that patient recruitment and retention for clinical trials cost over $2.3 billion each year [11].
In recent years, however, virtual studies have begun to transform feasibility for participant recruitment into clinical research [12]. Virtual studies recruit, consent, and enroll participants online, and often administer interventions and track responses remotely via mobile or other online devices [[13], [14], [15], [16]]. The clearest potential advantage of virtual studies is that they maximize the number of individuals able to participate in each study [13]. By removing barriers to participation, virtual studies allow researchers to identify a larger pool of eligible participants more quickly, and enable individuals previously unable to participate (due to disability, geography, access to transportation, etc.) to now participate. Further, by reducing recruitment time and removing physical presence of staff at study sites, these studies can potentially reduce recruitment costs. Many virtual study platforms are also increasingly linking participation to online communities of individuals with shared health conditions, which may promote participant engagement and retention [13].
Yet, not all virtual studies have been successful with recruitment. Notably, the first virtual clinical trial conducted by Pfizer in 2011 failed to reach target enrollment numbers [4]. Virtual trials may face setbacks in participant recruitment; many individuals may enroll in studies because their physician recommends participation, they appreciate meaningful relationships that are built by visiting a site in-person, or they value the support system that is developed with study staff over time [9,12]. Further, many participants may have concerns about data privacy and protection issues unique to virtual studies [17].
The benefits of virtual recruitment seem substantial, yet concerns about hurdles are valid. With this paper, we begin building an empirical evidence base for the experience of participant recruitment. We compare the sample metrics of three virtually recruited samples to relevant traditionally recruited comparison samples.
2. Materials and methods
2.1. Study identification
We conducted a systematic search of the literature to identify relevant comparator studies for three virtual studies (conducted by Evidation Health, Inc., San Mateo, CA) in cognitive health, diabetes, and hypertension in the United States, and to extract selected information on participant and recruitment features. The included virtual studies were selected because they were the first three conducted by Evidation Health that relied exclusively on virtual recruitment across a broad eligible population (e.g., online recruitment through social media, relevant mobile apps, and targeted advertisements), and to have concluded recruitment at the time this review was initiated, thus providing a full study sample for comparison purposes. The virtual study on cognitive health was a 12-month long, prospective, intent-to-treat, single arm study that aimed to evaluate the impact of a virtual cognitive health coaching program (via telephone and email/text messaging) on cognitive function and mental health for older individuals who showed signs of subjective cognitive decline [16]. The virtual study on diabetes was a 12-week long, prospective, intent-to-treat, single-arm study that aimed to evaluate the impact of a mobile diabetes app and experts coaching program on HbA1c1 levels for individuals with Type 2 diabetes and HbA1c ≥ 7.5% [14]. The virtual study on hypertension was a 12-week long, prospective, intent-to-treat, 2-arm randomized clinical trial that aimed to evaluate the impact of a smartphone application on blood pressure control and self-reported medication adherence for patients with poorly controlled blood pressure [15]. All three studies were IRB-approved and conducted via an online study platform, where potential participants answered a set of screener questions to assess eligibility and signed an electronic informed consent form [[14], [15], [16]].
Comparator studies were eligible for inclusion if they were published in English, presented original research from the United States, involved a minimal risk intervention, and were conducted among a non-restricted population of individuals (i.e., not targeted to a specific racial/ethnic group) with Type 2 diabetes, hypertension, or cognitive decline recruited through non-virtual means. The PubMed database was searched using specific search terms for all relevant studies published in English in the three years preceding the search date (between 2014 and 2017) separately for each of three disease areas (cognitive decline, diabetes, and hypertension). Search terms included (1) ((cognitive decline[Title/Abstract]) AND intervention[Title/Abstract]) AND (“2014/07/13"[Date - Publication]: “2017/07/13"[Date - Publication])); (2) (((type a diabetes mellitus[MeSH Terms]) AND diabetes[Title/Abstract]) AND intervention[Title/Abstract]) AND (“2014/04/01"[Date - Publication]: “2017/04/27"[Date - Publication]); and [18] (((hypertension[MeSH Terms]) AND hypertension[Title/Abstract]) AND intervention[Title/Abstract]) AND (“2014/06/01"[Date - Publication]: “2017/06/22"[Date - Publication]). The date of the search, the number of results, and reasons for exclusion were tracked for each study population. The search was conducted on July 13, 2017.
2.2. Study selection
One investigator reviewed all titles returned by the PubMed searches for relevance and eligibility with regard to the study inclusion and exclusion criteria. Studies not excluded by title review progressed to abstract review. One investigator read all remaining abstracts in each of the three study areas. In this second stage of screening, we excluded articles if the abstract indicated misalignment with the inclusion criteria. Following abstract review, two investigators reviewed the full text of remaining studies. As in prior steps, we excluded full text articles if the research was conducted in another country, within a narrowly targeted or overly broad population, focused on an unrelated outcome, included an invasive intervention, or presented results from a systematic review or meta-analysis (as opposed to an original research article).
2.3. Data extraction
For each included study, we extracted published information on participant demographic characteristics, as well as information on recruitment methods and timing. Demographic characteristics extracted included study population age (mean and standard deviation), sex (female/male/other), race (white versus another racial identity), and distribution of states of residence within the United States. Recruitment data included the number of participants recruited, the number of months over which that recruitment took place, and where and how participants were recruited. Two investigators independently extracted this data from final included studies and compared results for consistency. Where data on the above variables were missing in a manuscript, one investigator contacted the corresponding author via email to request the data. Of eight authors contacted, six replied with the requested information.
2.4. Analysis
We utilized a fixed-effects mini meta-analysis described by Goh et al. [19] to compare mean age weighted by sample size, and separately compared the proportion of the study population that identified as female, proportion of the study population that identified as white, number of states in which study participants resided, number of months to complete participant recruitment, and recruitment rate (average number of participants enrolled per month) across studies.
For continuous outcomes (age, number of states in which participants resided, number of participants recruited, recruitment rate), for each therapeutic area, we pooled data from the comparator studies to generate a group mean, weighted by sample size, and standard deviation. We then compared the pooled mean and standard deviation for each outcome to the individual mean and standard deviation from the virtually recruited study in that therapeutic area (cognitive health, diabetes, or hypertension) via unpaired two-sample t-tests that did not assume equal variance. Despite the robustness of the t-test to violations of the normality assumption in samples of these sizes, we also tested continuous outcomes for a difference of medians using the nonparametric unpaired Wilcoxon-Mann-Whitney test (all excluding age, which was reported as a mean). We assessed equality of proportions of white participants pair-wise between each included study and the comparator virtually recruited study with a two-sample test of proportions. Similarly, we assessed the equality of distributions of participant sex pair-wise between each included study and the comparator virtually recruited study using Fisher's exact test. As the data for age, race, and sex came from the same group of participants, separate comparisons of those metrics were not fully independent; however, this does not violate the basic rule of independence within a meta-analysis [19]. All statistical analyses were conducted in Stata version 15.
3. Results
3.1. Selected studies
We identified 19 comparator studies across the three disease areas (cognitive health, diabetes, and hypertension) that fit our eligibility criteria (Fig. 1a, Fig. 1b, Fig. 1ca–c, Table 1) (one study was relevant for both diabetes and hypertension comparisons). The four comparator cognitive health studies assessed the association between cognitive function in older adults and water-based exercise [20], yoga and meditation [21], meditation and music [22], and yoga [23]. The nine comparator diabetes studies assessed the association between a low-level behavioral weight loss intervention [24,25], telehealth intervention [26,27], cloud-based diabetes management program [28], physical activity and weight loss intervention [29], treatment adherence intervention [30], mobile coaching system [31], and diet and exercise counseling [32] and change in HbA1c levels (among other outcomes). Finally, the seven comparator hypertension studies assessed the association between a personalized physician learning intervention [33], provider communication skills intervention [34], telehealth intervention [26,35], electronic health game [36], dietary approaches [37], and a home based digital medicine program [38] and reduction in blood pressure.
Fig. 1a.
Flow diagram for the cognitive health study selection process for the systematic search.
Fig. 1b.
Flow diagram for the diabetes study selection process for the systematic search.
Fig. 1c.
Flow diagram for the hypertension study selection process for the systematic search.
Table 1.
Studies included in systematic search.
| Study ID | Population | Total, N | Study Design | Review Outcomes reported |
|---|---|---|---|---|
| Courcoulas et al., 2015 | Diabetes | 61 | RCT | Age, sex, geography, recruitment method, recruitment time |
| de vries McClintock et al., 2015 | Diabetes | 180 | RCT | Age, sex, race, geography, recruitment method, recruitment time |
| Delahanty et al., 2015 | Diabetes | 57 | RCT | Age, sex, race, geography, recruitment method, recruitment time |
| Edelman et al., 2015 | Diabetes & Hypertension | 377 | RCT | Age, sex, race, geography, recruitment method, recruitment time |
| Eyre et al., 2016 | Cognitive Decline | 25 | RCT | Age, sex, race, geography, recruitment method, recruitment time |
| Eyre et al., 2017 | Cognitive Decline | 79 | RCT | Age, sex, race, geography, recruitment method, recruitment time |
| Fedor et al., 2015 | Cognitive Decline | 60 | Observational | Age, sex, race, geography, recruitment method, recruitment time |
| Greenwood et al., 2015 | Diabetes | 90 | RCT | Age, sex, race, geography, recruitment method, recruitment time |
| Hickman et al., 2015 | Hypertension | 144 | RCT | Age, sex, race, geography, recruitment method, recruitment time |
| Hsu et al., 2016 | Diabetes | 40 | RCT | Geography, recruitment method |
| Innes et al., 2016 | Cognitive Decline | 60 | RCT | Age, sex, race, geography, recruitment method, recruitment time |
| Liss et al., 2016 | Diabetes | 331 | RCT | Age, sex, race, geography, recruitment method, recruitment time |
| Manze et al., 2014 | Hypertension | 203 | RCT | Sex, race, geography, recruitment method, recruitment time |
| Margolis et al., 2015 | Hypertension | 403 | RCT | Age, sex, race, geography, recruitment method, recruitment time |
| Milani et al., 2017 | Hypertension | 556 | Observational | Age, sex, race, geography, recruitment method, recruitment time |
| O'Connor et al., 2014 | Hypertension | 4568 | RCT | Age, sex, race, geography, recruitment method |
| Quinn et al., 2016 | Diabetes | 118 | RCT | Age, sex, race, geography |
| Rock et al., 2014 | Diabetes | 227 | RCT | Age, sex, race, geography, recruitment method, recruitment time |
| Sayer et al., 2015 | Hypertension | 19 | Random crossover | Age, sex, geography, recruitment time |
3.2. Syntheses of results
Table 2a, Table 2b, Table 2ca–c present the data points extracted from each of the 19 included studies. Regarding sample demographic characteristics, 17 of the 19 included studies reported a mean and standard deviation for age, 18 of 19 reported sex, 16 of 19 reported race (two of which were obtained by emailing study authors), and all 19 reported the distribution of state of residence for the study population. Regarding recruitment, 17 of 19 included studies reported the method of recruitment used, and 15 of 19 included studies provided data on the duration of recruitment in months (seven of which were obtained via email to study authors).
Table 2a.
Extracted data points from cognitive health studies.a
| Study Name | Virtual study on cognitive health16 | Eyre et al., J of Alzheimer's Disease 2016 |
Eyre et al., Int Psychogeriatr 2017 |
Fedor et al., Arch Clin Neuropsych 2015 |
Innes et al., Comp Ther Med 2016 |
|||
|---|---|---|---|---|---|---|---|---|
| Study Arm | Intervention | Control | Intervention | Control | Intervention | Control | ||
| Sample Size | n = 82 | n = 14 | n = 11 | n = 38 | n = 41 | n = 27 | n = 33 | n = 60 |
| Age, Mean±SD | 64± 4 | 67±10 | 68±10 | 68±9 | 68±8 | 63±8 | 66±7 | 61±1 |
| Sex, n (%) | ||||||||
| Female | 61 (74) | 6 (43)b | 6 (55)b | 25 (66) | 27 (66) | 21 (78) | 25 (76) | 51 (85) |
| Male | 20 (24) | 8 (57) | 5 (45) | 13 (34) | 14 (34) | 6 (22) | 8 (24) | 9 (15) |
| Not listed | 1 (1) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) |
| Race/Ethnicity, n (%) | ||||||||
| White | 72 (88) | 12 (86) | 8 (73) | 24 (63)c | 30 (73)c | 26 (96)d | 33 (100)d | 56 (93) |
| Another race | 10 (12) | 2 (14) | 3 (27) | 14 (37) | 11 (27) | 1 (4)d | 0 (0)d | 4 (7) |
| Geographic distribution | ||||||||
| Number of states represented | 29 | 1 | 1 | 1 | 1 | |||
| Participant Recruitment | ||||||||
| Where recruited | Online study platform (Achievement) | Outpatient clinics, longevity center, community | Outpatient clinics, longevity center, community | Community recreation and wellness centers | Community, health care, and workplace settings | |||
| Recruitment method | Online | Advertisements | Advertisements | Flyers | Flyers and emails | |||
| Recruitment time | 4mo | 24mod | 24mo | 12mod | 5mod | |||
| Avg number of participants recruited per month | 21 | 1 | 3 | 5 | 12 | |||
Because of rounding, n's may not be exact.
p < 0.05 as compared to virtual study.
p < 0.01 as compared to virtual study.
Not specified in publication, but confirmed via email with corresponding author.
Table 2b.
Extracted data points from diabetes studies.a
| Study Name |
Virtual study on diabetes management14 |
de vries McClintock et al., J Behav Med 2015 |
Courcoulas et al., JAMA 2015 |
Delahanty et al., Obesity 2015 |
Edelman et al., J Gen Int Med 2015 |
||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Study Arm | Control | Intervention | Control | Intervention | Control | Intervention | |||||||
| Sample Size | n = 146 | n = 88 | n = 92 | n = 61 | n = 29 | n = 28 | n = 184 | n = 193 | |||||
| Age, Mean±SD | 52±9 | 57±10 | 58±9 | 47±7 | 61±11 | 62±10 | 60±11 | 58±11 | |||||
| Sex, n (%) | |||||||||||||
| Female | 103 (71) | 58 (66) | 64 (70) | 50 (82) | 12 (41)b | 11 (39)b | ~101 (55)b | ~104 (54)b | |||||
| Male | 43 (30) | 30 (34) | 28 (30) | 11 (18) | 17 (59) | 17 (61) | ~83 (45) | ~89 (46) | |||||
| Not listed | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | |||||
| Race/Ethnicity, n (%) | |||||||||||||
| White | 107 (73) | 36 (41)b | 29 (32)b | 48 (79) | 18 (64) | 24 (86) | ~93 (50)b | ~95 (49)b | |||||
| Another race | 39 (27) | 52 (59) | 63 (68) | 13 (21) | 11 (36) | 4 (14) | ~91 (50) | ~98 (51) | |||||
| Geographic distribution | |||||||||||||
| Number of states represented | 45 | 1 | 1 | 1 | 1 | ||||||||
| Participant recruitment | |||||||||||||
| Where recruited | Online study platform (Achievement) | Primary care practices | Academic medical center | Primary care practices | Primary care practices | ||||||||
| Recruitment method | Online | Medical records | Advertisements (TV, print, internet) | Medical records | Medical records | ||||||||
| Recruitment time | 3mo | 12mo | 18moc | 10mo | 25mo | ||||||||
| Avg number of participants recruited per month |
49 |
15 |
2 |
6 |
15 |
||||||||
| Table 2b(continued). Extracted data points from diabetes studiesa | |||||||||||||
| Study Name | Greenwood et al., J of Med Int Res 2015 |
Hsu et al., Diabetes Tech & Ther 2016 |
Liss et al., Contemp Clin Trials 2016 |
Quinn et al., J of App Ger 2016d |
Rock et al., Diabetes Care 2014 |
||||||||
| Study Arm |
Control |
Intervention |
Control |
Intervention |
Age <55 years |
Age ≥55 years |
Low-Fat Diet |
Low-Carb Diet |
Control Diet |
||||
| Sample Size | n = 45 | n = 45 | n = 20 | n = 20 | n = 331 | n = 37 | n = 25 | n = 74 | n = 77 | n = 76 | |||
| Age, Mean±SD | 58±11 | 54±10 | 54 | 53 | 57±11 | 47±7 | 59±3 | 56±9 | 57±9 | 57±9 | |||
| Sex, n (%) | |||||||||||||
| Female | 19 (42)b | 23 (51)b | – | – | 166 (50)b | 23 (62)e | 8 (32)e | 35 (47)b | 37 (48)b | 44 (58)b | |||
| Male | 26 (58) | 22 (49) | – | – | 165 (50) | 14 (38) | 17 (68) | 39 (53) | 40 (52) | 32 (42) | |||
| Not listed | 0 (0) | 0 (0) | – | – | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | |||
| Race/Ethnicity, n (%) | |||||||||||||
| White | 27 (60) | 29 (64) | – | – | 114 (34)b | 20 (54) | 19 (76) | 59 (80) | 63 (82) | 59 (78) | |||
| Another race | 16 (36)f | 16 (36)f | – | – | 217 (66) | 17 (46) | 6 (24) | 15 (20) | 14 (18) | 17 (22) | |||
| Geographic distribution | |||||||||||||
| Number of states represented | 1 | 1 | 1 | 1 | 2 | ||||||||
| Participant recruitment | |||||||||||||
| Where recruited | Large health care system | Tertiary diabetes care center | Primary care practices | Community physician practice groups | Universities | ||||||||
| Recruitment method | Medical records | Health care providers | Primary care providers and medical records | – | Word of mouth, direct marketing letters mailed to large cohorts, radio ads, local e-mail subscription services, ClinicalTrials.gov, social media, and flyers | ||||||||
| Recruitment time | – | – | 19mo | – | 6mo | ||||||||
| Avg number of participants recruited per month | – | – | 17 | – | 38 | ||||||||
Because of rounding, n's may not be exact
p ≤ 0.001 as compared to virtual study
Not specified in publication, but confirmed via email with corresponding author
Reporting on intervention group only due to reporting limitations in original publication
p ≤ 0.01 as compared to virtual stud
Missing data for some participants.
Table 2c.
Extracted data points from hypertension studies.a
| Study Name |
Virtual study on hypertension15 |
Edelman et al., J Gen Int Med 2015 |
Hickman et al., 2015 |
Manze et al., 2014 |
|||||
|---|---|---|---|---|---|---|---|---|---|
| Study Arm | Control | Intervention | Intervention | Control | |||||
| Sample Size | n = 411 | n = 184 | n = 193 | n = 97 | n = 47 | n = 203 | |||
| Age, Mean±SD | 52±10 | 60±11 | 58±11 | 49±13 | 47±14 | 61 (range: 29–88) | |||
| Sex, n (%) | |||||||||
| Female | 247 (60) | ~101 (55) | ~104 (54) | 57 (59) | 31 (66) | 147 (72)b | |||
| Male | 162 (39) | ~83 (45) | ~89 (46) | 40 (41) | 16 (34) | 56 (28) | |||
| Not listed | 2 (1) | 0 (0) | 0 (0) | – | – | – | |||
| Race/Ethnicity, n (%) | |||||||||
| White | 268 (65)c | ~93 (50)c | ~95 (49)c | 34 (24)c | 68 (34)c | ||||
| Another race | 143 (35) | ~91 (50) | ~98 (51) | 110 (76) | 135 (66) | ||||
| Geographic distribution | |||||||||
| Number of states represented | 46 | 1 | 1 | 1 | |||||
| Participant Recruitment | |||||||||
| Where recruited | Online study platform (Achievement) | Primary care practices | Community | Primary care clinics | |||||
| Recruitment method | Online | Medical records | Newspaper, business and local bus advertisements | Medical records | |||||
| Recruitment time | 5mo | 25mo | 10 mo | 19mo | |||||
| Avg number of participants recruited per month |
82 |
15 |
16 |
11 |
|||||
| Table 2c (continued). Extracted data points from hypertension studiesa | |||||||||
| Study Name | Margolis et al., 2015 | Milani et al., 2017 | O'Connor et al., 2014 | Sayer et al., 2015 | |||||
| Study Arm |
Intervention |
Control |
Intervention |
Control |
PPL-EMR |
PPL-Assess |
Control |
||
| Sample Size | n = 206 | n = 197 | n = 156 | n = 400 | n = 1594 | n = 1501 | n = 1473 | 19 | |
| Age, Mean±SD | 62±11 | 60±12 | 68 ± 10 | 68 ± 10 | 61±15 | 60±15 | 60±16 | 61±2 | |
| Sex, n (%) | |||||||||
| Female | 88 (43)c | 86 (44)c | 84 (54)d | 216 (54)d | 886 (56)c | 758 (51)c | 795 (54)c | 13 (68) | |
| Male | 118 (57) | 111 (56) | 72 (46) | 184 (46) | 708 (44) | 743 (49) | 678 (46) | 6 (32) | |
| Not listed | – | – | – | – | – | – | – | 0 (0) | |
| Race/Ethnicity, n (%) | |||||||||
| White | 335 (83)c | 120 (77)c,e | 308 (76)c,e | 1321 (83)c | 1202 (80)c | 1209 (82)c | – | ||
| Another race | 68 (17) | 36 (23)e | 92 (24)e | 273 (17) | 299 (20) | 264 (18) | – | ||
| Geographic distribution | |||||||||
| Number of states represented | 1 | 1 | 2 | 1 | |||||
| Participant Recruitment | |||||||||
| Where recruited | Primary care clinics | Community health system | Primary care clinics | Community | |||||
| Recruitment method | Medical records | Medical records | Medical records | – | |||||
| Recruitment time | 25moe | 13moe | – | 16moe | |||||
| Avg number of participants recruited per month | 16 | 43 | – | 1 | |||||
Because of rounding, n's may not be exact.
p ≤ 0.01 as compared to virtual study.
p ≤ 0.001 as compared to virtual study.
p < 0.05 as compared to virtual study.
Not specified in publication, but confirmed via email with corresponding author
3.3. Comparisons based on demographic characteristics
The mean age of study participants in the virtually recruited cognitive health study did not differ from the pooled mean age across the four comparator studies (64 versus 65, p = 0.12). As compared to the eight studies related to diabetes that reported mean and standard deviation for age, the virtually recruited diabetes study was five years younger, on average (52 versus 57 years, p < 0.001). Similarly, as compared to the six hypertension studies with the requisite data on age, the virtually recruited hypertension sample was nine years younger on average (52 versus 61 years, p < 0.001).
In over half (n = 11) of the 18 studies that reported sex, the proportion of participants reported as female statistically significantly differed between the traditionally versus the virtually recruited samples. In 10 of the comparator studies, the percentage of female participants was lower compared to that in the virtually recruited studies. In one hypertension study, the virtually recruited sample had a lower percentage of female participants.
Of the 17 comparator studies that reported data on the racial identity (white versus another race) of study participants, the virtually recruited samples statistically significantly differed from the comparator samples in 10 instances. In six cases, the virtually recruited sample was less diverse (fewer participants of another racial identity) than the traditionally recruited sample, and in four cases, the virtually recruited sample was more diverse.
Study samples were also compared based on the distribution of state of residence of included participants. As compared to the 19 comparator studies that enrolled participants from an average of 1.1 states (range: 1–2), the three virtual studies included participants from an average of 40 states (range: 29–46) (p-value for difference = 0.02). A non-parametric Wilcoxon-Mann-Whitney (or rank sum) test of distributions confirmed the difference in number of included states (p < 0.01).
3.4. Comparisons based on speed of recruitment
The three virtual studies took an average of 4.0 months (SD 1) to recruit and enroll the target sample size, in contrast to an average of 15.9 months (SD 7) for the comparator studies (p-value for difference<0.01). A test of median recruitment time between virtual and comparator studies confirms a similar difference (virtual median = 4 months, traditional median = 16 months; p-value for difference<0.01).
Similarly, virtual studies enrolled a higher mean and median number of participants per month as compared to comparator studies. The virtual studies enrolled an average of 51 participants per month (SD 31), as compared to 13 participants per month (SD 12) among the 15 comparator studies (p-value for difference<0.01) (Fig. 2). The virtual studies enrolled a median of 49 participants per month as compared to a median of 12 participants per month for the comparator studies (p-value for difference = 0.02).
Fig. 2.
Number of participants enrolled per month by study type.
4. Discussion
In this systematic search of the literature, we identified 19 studies in cognitive health, diabetes, and hypertension that relied on traditional recruitment methods and compared the resulting study samples to three virtually recruited samples on several demographic and timing metrics. Our goal was to report observational differences between traditionally and virtually recruited samples. We found that where traditionally recruited samples differed from virtually recruited samples, the virtually recruited samples were slightly younger, enrolled a higher percentage of female participants, and were split on enrollment of minorities (the virtual samples were less diverse in 32% of comparisons, but more diverse in 21% of comparisons). The virtually recruited samples outperformed the traditionally recruited samples in terms of geographic spread of participants, overall recruitment time, and average number of participants recruited per month. While some of the observed differences could be explained by confounding factors, our comparison approach nevertheless provides a view into correlational differences. The findings regarding geographic spread and recruitment volume per month, in particular, support the hypothesis that virtual recruitment methodologies may help facilitate opportunities for more individuals to participate in research studies.
A main goal of most studies is to recruit a representative sample so that the results can be applied to the population that the study is intended to help. Given the increasing costs of clinical research in recent years coupled with difficulty in achieving the required number of research participants, attention has focused on ways to increase research participation and improve participant recruitment [9]. Investigators have called for the development of novel and diverse strategies to expand the pool of eligible research participants to address this issue. Findings presented in this study provide encouraging evidence that virtual recruitment may be one such strategy.
Historically, clinical research has suffered from underrepresentation of women in study samples [39]. While gender identity is not always consistent with sex, the higher proportion of female participants in the virtually recruited studies assessed here could represent an important opportunity to address the gaps in our knowledge about women and other people assigned female sex at birth's unique response to treatments and disease trajectories. Further, as these results suggest that virtual studies may in some instances be less diverse in terms of the racial composition of samples, investigators can use this knowledge to carefully consider strategies for increasing outreach and recruitment of individuals from underrepresented populations that could address this potential limitation. Both researchers and designers of healthcare interventions alike should consider the advantages that virtual recruitment may offer for increasing the efficiency and impact of health research with regard to sample recruitment.
As with all research, this study has limitations. The three virtual studies assessed were all designed and run by one digital health company (Evidation Health, Inc., San Mateo, CA). Recruitment metrics from these study samples may therefore not be representative of other virtually recruited studies conducted by other entities. Further, the literature review was limited to peer-reviewed publications included in the PubMed database. While PubMed is considered the largest database for peer-reviewed articles in health research and is updated more frequently and with a broader scope than comparator databases such as MEDLINE [40], it is still possible that we may have missed relevant comparator studies by relying solely on PubMed. This review also only focused on studies run in the United States; there may be additional efforts taking place in other countries that could serve as beneficial comparison points. Some of the measures compared across studies could be influenced on a study-by-study basis by specific inclusion and exclusion criteria or recruitment targeting strategies (e.g., age restrictions). Along the same lines, characteristics of individuals who enroll into research studies vary widely by the therapeutic area and population of interest, and there can be differences in results among individuals with varying demographic characteristics. Therefore, findings from these three therapeutic areas may not be generalizable to other therapeutic areas. Lastly, the three virtual studies presented here not only utilized virtual recruitment methods, but were completely virtual in nature (i.e., allowed participants to participate in the study remotely, had no in-person or site-based study procedures). Part of the improved efficiency in recruitment and geographic diversity in the three studies that utilized virtual recruitment methods may be attributed to participants' increased willingness to participate in a study remotely. These study design characteristics would influence the values that then were compared across studies.
5. Conclusions
This is the first study, to our knowledge, to compare participant recruitment metrics between virtual and traditional recruitment methodologies. We adopted a clear and replicable study design so that similar studies can be conducted to compare metrics for other virtual studies as they are published, adding to the evidence base for the strengths and limitations of virtual recruitment for health research.
Although this study is by no means an exhaustive analysis of all virtual samples, it serves as a useful first contribution to improving our understanding of the opportunities that virtual studies may offer for enhancing efficiency and empowering a wider number of individuals to participate in health research. We hope that the findings presented here provide a useful quantification of the comparison of virtual and traditional recruitment, and will inspire future research to compare a wider range of studies on these metrics, as well as additional metrics that we were not able to assess, such as the overall cost of recruitment. While much work remains to be done, we are optimistic that virtual recruitment may help to accelerate the adoption of urgently needed health solutions for a wider range of people.
Declaration of competing interest
The authors declare that there is no conflict of interest.
Acknowledgments
The authors would like to thank Mai Ka Vang for assistance with preparing the manuscript.
Footnotes
Hemoglobin A1c (HbA1c).
Supplementary data to this article can be found online at https://doi.org/10.1016/j.conctc.2020.100590.
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Appendix A. Supplementary data
The following is the Supplementary data to this article:
References
- 1.Kukull W., Ganguli M. Generalizability: the trees, the forest, and the low-hanging fruit. Neurology. 2012;78(23):1886–1891. doi: 10.1212/WNL.0b013e318258f812. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Martinez-Mesa J., Gonzalez-Chica D., Duquia R., Bonamigo R., Bastos J. Sampling: how to select participants in my research study? An. Bras. Dermatol. 2016;91(3):326–330. doi: 10.1590/abd1806-4841.20165254. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Tyrer S., Heyman B. Sampling in epidemiological research: issues, hazards and pitfalls. BJPsych Bull. 2016;40(2):57–60. doi: 10.1192/pb.bp.114.050203. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Orri M., Lipset C., Jacobs B., Costello A., Cummings S. Web-based trial to evaluate the efficacy and safety of tolterodine ER 4mg in participants with overactive bladder: REMOTE trial. Contemp. Clin. Trials. 2014;38(2):190–197. doi: 10.1016/j.cct.2014.04.009. [DOI] [PubMed] [Google Scholar]
- 5.Bachenheimer J., Brescia B. Reinventing patient recruitment: revolutionary ideas for clinical trial success. 2007. London. [DOI]
- 6.Clinical trial delays: America's patient recruitment dilemma. 2012. http://www.drugdevelopment-technology.com/features/featureclinical-trial-patient-recruitment
- 7.Sanofi bets on virtual clinical trials. 2017. http://www.centerwatch.com/news-online/2017/03/13/sanofi-bets-virtual-clinical-trials/ accessed 30 August 2017.
- 8.McDonald A., Knight R., Campbell M., Entwistle V., Grant A., Cook J., Elbourne D., Francis D., Garcia J., Roberts I., Swnodon C. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7(9) doi: 10.1186/1745-6215-7-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Newington L., Metcalfe A. Factors influencing recruitment to research: qualitative study of the experiences and perceptions of research teams. BMC Med. Res. Methodol. 2014;14(10) doi: 10.1186/1471-2288-14-10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Tufts Center for the Study of Drug Development New research from Tufts Center for the Study of Drug Development characterizes effectiveness and variability of patient recruitment and retention practices. 2013. https://www.biospace.com/article/releases/new-research-from-tufts-center-for-the-study-of-drug-development-characterizes-effectiveness-and-variability-of-patient-recruitment-and-retention-prac
- 11.U.S. Department of Health and Human Services Examination of clinical trial costs and barriers for druge development: analysis of costs. 2014. https://aspe.hhs.gov/report/examination-clinical-trial-costs-and-barriers-drug-development/3-analysis-costs
- 12.Clinical Research Trends How new virtual clinical trials have improved patient recruitment. 2016. https://www.clinicalresearchtrends.com/index.php/2016/11/21/virtual-clinical-trials-improved-recruitment
- 13.Jadhav S. Virtual clinical trials: the future of patient engagement, Applied Clinical Trials. 2016. http://www.appliedclinicaltrialsonline.com/virtual-clinical-trials-future-patient-engagement
- 14.Kumar S., Moseson H., Uppal J., Juusola J.L. A diabetes mobile app with in-app coaching from a certified diabetes educator reduces A1C for individuals with Type 2 Diabetes. Diabetes Educat. 2018;44(3):226–236. doi: 10.1177/0145721718765650. [DOI] [PubMed] [Google Scholar]
- 15.Morawski K., Ghazinouri R., Krumme A., Lauffenburger J.C., Lu Z., Durfee E., Oley L., Lee J., Mohta N., Haff N., Juusola J.L., Choudhry N.K. Association of a smartphone application with medication adherence and blood pressure control: the MedISAFE-BP randomized clinical trial. JAMA Intern Med. 2018;178(6):802–809. doi: 10.1001/jamainternmed.2018.0447. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Bott N., Kumar S., Krebs C., Glenn J.M., Madero E.N., Juusola J.L. A remote intervention to prevent or delay cognitive impairment in older adults: design, recruitment, and baseline characteristics of the Virtual Cognitive Health (VC Health) study. JMIR Res Protoc. 2018;7(8) doi: 10.2196/11368. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Covington D. The remote patient-centered approach in clinical research, Applied Clinical Trials. 2014. http://www.appliedclinicaltrialsonline.com/remote-patient-centered-study-approach-clinical-research
- 18.De Laet C., Kanis J.A., Oden A., Johanson H., Johnell O., Delmas P., Eisman J.A., Kroger H., Fujiwara S., Garnero P., McCloskey E.V., Mellstrom D., Melton L.J., 3rd, Meunier P.J., Pols H.A., Reeve J., Silman A., Tenenhouse A. Body mass index as a predictor of fracture risk: a meta-analysis. Osteoporos. Int. 2005;16(11):1330–1338. doi: 10.1007/s00198-005-1863-y. [DOI] [PubMed] [Google Scholar]
- 19.Goh J., Hall J., Rosenthal R. Mini meta-analysis of your own studies: some arguments on why and a primer on how. Soc Personal Psychol Compass. 2016;10(10):535–549. doi: 10.1111/spc3.12267. [DOI] [Google Scholar]
- 20.Fedor A., Garcia S., Gunstad J. The effects of a brief, water-based exercise intervention on cognitive function in older adults. Arch. Clin. Neuropsychol. 2015;30:139–147. doi: 10.1093/arclin/acv001. [DOI] [PubMed] [Google Scholar]
- 21.Eyre H., Acevedo B., Yang H., Siddarth P., Van Dyk K., Ercoli L., Leaver A., Cyr N. St, Narr K., Baune B., Khalsa D., Lavretsky H. Changes in neural connectivity and memory following a yoga intervention for older adults: a pilot study. J Alzheimers Dis. 2016;52:673–684. doi: 10.3233/JAD-150653. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Innes K., Selfe T., Khalsa D., Kandati S. A randomized controlled trial of two simple mind-body programs, Kirtan Kriya mediation and music listening, for adults with subjective cognitive decline: feasibility and acceptability. Compl. Ther. Med. 2016;26:98–107. doi: 10.1016/j.ctim.2016.03.002. [DOI] [PubMed] [Google Scholar]
- 23.Eyre H., Siddarth P., Acevedo B., Van Dyk K., Pattharee P., Ercoli L., Cyr N. St, Yang H., Khalsa D., Lavretsky H. A randomized controlled trial of Kundalini yoga in mild cognitive impairment. Int. Psychogeriatr. 2017;29(4):557–567. doi: 10.1017/S1041610216002155. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Courcoulas A., Belle S., Neiberg R., Pierson S., Eagleton J., Kalarchian M., DeLany J., Lang W., Jakicic J. Three-year outcomes of bariatric surgery vs lifestyle intervention on Type 2 Diabetes Mellitus treatment: a randomized trial. JAMA Surg. 2015;150(10):931–940. doi: 10.1001/jamasurg.2015.1534. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Delahanty L., Dalton K., Porneala B., Chang Y., Goldman V., Levy D., Nathan D., Wexler D. Improving diabetes outcomes through lifestyle change - a randomized controlled trial. Obesity. 2015;23:1792–1799. doi: 10.1002/oby.21172. [DOI] [PubMed] [Google Scholar]
- 26.Edelman D., Dolor R., Coffman C., Pereira K., Granger B., Lindquist J., Neary A., Harris A., Bosworth H. Nurse-led behavioral management of diabetes and hypertension in community practices: a randomized trial. J. Gen. Intern. Med. 2015;30(5):626–633. doi: 10.1007/s11606-014-3154-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Greenwood D., Blozis S., Young H., Nesbitt T., Quinn C. Overcoming clinical intertia: a randomized clinical trial of a telehealth remote monitoring intervention using paired glucose testing in adults with Type 2 Diabetes. J. Med. Internet Res. 2015;17(7):e178. doi: 10.2196/jmir.4112. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Hsu W., Lau K., Huang R., Ghiloni S., Le H., Gilroy S., Abrahamson M., Moore J. Utilization of a cloud-based diabetes management program for insulin initiation and titration enables collaborative decision making between healthcare providers and patients. Diabetes Technol. Therapeut. 2016;18(2):59–67. doi: 10.1089/dia.2015.0160. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Liss D., Finch E., Gregory D., Cooper A., Ackermann R. Design and participant characteristics for a randomized effectiveness trial of an intensive lifestyle intervention to reduce cardiovascular risk in adults with Type 2 Diabetes: the I-D-HEALTH study. Contemp. Clin. Trials. 2016;46:114–121. doi: 10.1016/j.cct.2015.11.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.De Vries McClintock H., Morales K., Small D., Bogner H. A brief adherence intervention that improved glycemic control: mediation by patterns of adherence. J. Behav. Med. 2015;38(1):39–47. doi: 10.1007/s10865-014-9576-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Quinn C., Shardell M., Terrin M., Barr E., Park D., Shaikh F., Guralnik J., Gruber-Baldini A. Mobile diabetes intervention for glycemic control in 45- to 64-year-old persons with Type 2 Diabetes. J. Appl. Gerontol. 2016;35(2):227–243. doi: 10.1177/0733464814542611. [DOI] [PubMed] [Google Scholar]
- 32.Rock C., Flatt S., Pakiz B., Taylor K., Leone A., Brelje K., Heath D., Quintana E., Sherwood N. Weight loss, glycemic control, and cardiovascular disease risk factors in response to differential diet composition in a weight loss program in Type 2 Diabetes: a randomized controlled trial. Diabetes Care. 2014;37:1573–1580. doi: 10.2337/dc13-2900. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.O'Connor P., Magid D., Sperl-Hillen J., Price D., Asche S., Rush W., Ekstrom H., Brand D., Tavel H., Godlevsky O., Johnson P., Margolis K. Personalised physician learning intervention to improve hypertension and lipid control: randomized trial comparing two methods of physician profiling. BMJ Qual. Saf. 2014;23(12):1014–1022. doi: 10.1136/bmjqs-2014-002807. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Manze M., Orner M., Glickman M., Pbert L., Berlowitz D., Kressin N. Brief provider communication skills training fails to impact patient hypertension outcomes. Patient Educ. Counsel. 2015;98(2):191–198. doi: 10.1016/j.pec.2014.10.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Margolis K., Asche S., Bergdall A., Dehmer S., Maciosek M., Nyboer R., O'Connor P., Pawloski P., Sperl-Hillen J., Trower N., Tucker A., Green B. A successful multifaceted trial to improve hypertension control in primary care: why did it work? J. Gen. Intern. Med. 2015;30(11):1665–1672. doi: 10.1007/s11606-015-3355-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Hickman R., Jr., Clochesy J., Pinto M., Burant C., Pignatiello G. Impact of a serious game for health on chronic disease self-management: preliminary efficacy among community dwelling adults with hypertension. J. Health Hum. Serv. Adm. 2015;38(2):253–275. [PubMed] [Google Scholar]
- 37.Sayer R., Wright A., Chen N., Campbell W. Dietary approaches to stop hypertension diet retains effectiveness to reduce blood pressure when lean pork is substituted for chicken and fish as the predominant source of protein. Am. J. Clin. Nutr. 2015;102:302–308. doi: 10.3945/ajcn.115.111757. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Mliani R., Lavie C., Bober R., Milani A., Ventura H. Improving hypertension control and patient engagement using digital tools. Am. J. Med. 2016;130(1):14–20. doi: 10.1016/j.amjmed.2016.07.029. [DOI] [PubMed] [Google Scholar]
- 39.Liu K., Dipietro Mager N. Women's involvement in clinical trials: historical perspective and future implications. Pharm. Pract. 2016;14(1):708. doi: 10.18549/PharmPract.2016.01.708. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Kelly L. So many databases, such little clarity: searching the literature for the topic aboriginal. Can. Fam. Physician. 2008;54(11):1572–1573. e5. [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.