Abstract
This analysis of digital health studies in ClinicalTrials.gov examines the clinical evidence underlying digital health interventions.
Digital health is the application of software or hardware, often using mobile smartphone or sensor technologies to improve patient or population health and health care delivery.1 In contrast to drugs and traditional medical devices, which have strict regulatory guidelines on safety and efficacy, the clinical evidence generation for digital health tools may be motivated by other factors, including adoption, utilization, and value, that may influence study design and quality. The landscape of clinical evidence underlying digital health interventions has not been well characterized.2,3 We sought to evaluate the characteristics of digital health studies registered in ClinicalTrials.gov.
Methods
We performed a cross-sectional analysis of digital health studies in ClinicalTrials.gov.4,5 To identify studies evaluating mobile-, web-, and electronic-based tools as well as digital medical devices, we searched ClinicalTrials.gov on January 22, 2017, using the Medical Subject Heading concepts (mobile health, mHealth, ehealth, telehealth, and telemedicine) and commonly used lay terms (digital health, consumer health, mobile application, and wireless technology). Variables were exported as structured fields when downloaded from ClinicalTrials.gov.6 A single reviewer (C.E.C.) verified studies for inclusion, removed duplicates, and assigned each study to 1 of 13 clinical areas determined by iterative qualitative clustering against commonly accepted medicine domains. Descriptive statistics were calculated for key study characteristics, with additional stratification by study type (interventional vs observational, randomization status). We used the χ2 test to compare proportions, and P < .05 was considered to be statistically significant.
Results
We identified 1783 studies that met our inclusion criteria (from the top-level search of 3833 and after deduplication); 1570 studies (88.1%) were interventional, and 1257 (70.8%) were randomized. Among interventional studies, 107 (6.9%) were double-blinded and 417 (26.7%) single-blinded. Among observational studies, the most common study designs were case-control (26 [14.4%]), case-only (39 [21.7%]), and cohort (100 [55.6%]) (Table 1).
Table 1. Digital Health Studies Registered in ClinicalTrials.gov.
| Study Type | No. (%) of Studies |
|---|---|
| All (N = 1783) | |
| Interventional | 1570 (88.1) |
| Observational | 213 (11.9) |
| Study allocation (n = 1776) | |
| Randomized | 1257 (70.8) |
| Nonrandomized | 519 (29.2) |
| Recruitment statusa | |
| Not yet recruiting | 218 (12.2) |
| Recruiting or enrolling | 535 (30.0) |
| Active, not recruiting | 176 (9.9) |
| Completed | 692 (38.9) |
| Withdrawn or terminated | 56 (3.1) |
| Unknown | 103 (5.8) |
| Interventional (n = 1570) | |
| Intervention model (n = 1563) | |
| Parallel assignment | 1147 (73.4) |
| Crossover assignment | 88 (5.6) |
| Factorial assignment | 43 (2.8) |
| Single group assignment | 282 (18.0) |
| Masking (n = 1561) | |
| Double-blind | 107 (6.9) |
| Single-blinda | 417 (26.7) |
| Open-label or no masking | 1031 (66.0) |
| Observational (n = 213) | |
| Observational model (n = 180) | |
| Case-control | 26 (14.4) |
| Case-only | 39 (21.7) |
| Cohort | 100 (55.6) |
| Other | 15 (8.3) |
| Time perspective (n = 199) | |
| Prospective | 157 (78.9) |
| Retrospective | 19 (9.5) |
| Cross-sectional | 21 (10.5) |
| Completed, Suspended, Withdrawn, Terminated (n = 751) | |
| Study results | |
| Available | 85 (11.3) |
| Not available | 666 (88.7) |
Randomization status unknown for 7 studies.
Patient, principal investigator, or assessor.
Most studies consisted of adults or elderly individuals; 374 (20.1%) enrolled children. The most common clinical areas were cardiometabolic (382 [21.4%]), mental health (216 [12.1%]), and wellness (183 [10.2%]). Funding sources included federal and National Institutes of Health (369 [20.7%]) and industry (214 [12.0%]). Median enrollment was 120 (interquartile range, 50-300), although study sample size varied from fewer than 100 individuals (829 [46.5%]) to more than 1000 (142 [8.0%]) (Table 2). A higher proportion of publicly funded studies were interventional (338 [21.5%]) or randomized (294 [23.4%]), whereas a higher proportion of industry-funded studies were observational (48 [22.5%]) or nonrandomized (88 [17.0%]) (Table 2). Overall, 692 of 1783 studies (38.9%) had completed recruitment, and 85 completed or terminated studies (11.3%) had reported results. After multivariate adjustment, federally funded trials were more likely to have reported results (odds ratio, 4.9; 95% CI, 3.1-7.8; P < .001).
Table 2. Study Characteristics Stratified by Study Design Characteristics.
| Study Characteristic | No. (%) of Studies | χ2 P Value | No. (%) of Studies | χ2 P Value | |||
|---|---|---|---|---|---|---|---|
| All (N = 1783) | Observational (n = 213) | Interventional (n = 1570) | Nonrandomized (n = 519) | Randomized (n = 1257) | |||
| Start year | |||||||
| Before 2011 | 244 (13.7) | 27 (12.7) | 215 (13.7) | .26 | 53 (8.6) | 185 (13) | .26 |
| 2011 | 97 (5.4) | 14 (6.6) | 83 (5.3) | 31 (5.6) | 66 (5.1) | ||
| 2012 | 119 (6.7) | 13 (6.1) | 106 (6.8) | 37 (6.2) | 82 (6.3) | ||
| 2013 | 200 (11.2) | 27 (12.7) | 173 (11) | 53(10) | 147 (12) | ||
| 2014 | 259 (14.5) | 35 (16.4) | 224 (14.3) | 81 (16) | 178 (15) | ||
| 2015 | 334 (18.7) | 41 (19.2) | 293 (16.7) | 100 (20) | 234 (20) | ||
| 2016 | 415 (23.3) | 47 (22) | 368 (23.4) | 132 (28) | 282 (23) | ||
| 2017 | 108 (6.1) | 9 (8.3) | 101 (6.4) | 30 (5.4) | 79 (6.5) | ||
| Not stated | 7 (0.4) | NA | 7 (0.4) | 2 | 4 | ||
| Target agea | |||||||
| Child (<18 y) | 374 (20.1) | 45 (21.1) | 329 (21) | .95 | 117 (22.5) | 253 (20.1) | .25 |
| Adult (18-65 y) | 1680 (94.2) | 204 (95.8) | 1476 (94) | .30 | 487 (93.8) | 1186 (94.3) | .67 |
| Senior (≥66 y) | 1302 (73) | 174 (81.7) | 1128 (71.8) | .002 | 387 (74.5) | 908 (72.2) | .31 |
| Target sex | |||||||
| Men only | 34 (1.9) | 2 (0.9) | 32 (2.0) | .005 | 7 (1.4) | 27 (2.1) | .04 |
| Women only | 139 (9.0) | 10 (4.7) | 129 (8.22) | 33 (6.4) | 106 (8.4) | ||
| Both | 1607 (89) | 199 (93.4) | 1408 (89.7) | 477 (91.9) | 1124 (89.4) | ||
| Not stated | 3 (0.1) | 2 (0.9) | 1 (0.06) | NA | NA | ||
| Clinical area | |||||||
| Autoimmune | 84 (4.7) | 5 (2.3) | 79 (5.0) | <.001 | 20 (3.9) | 64 (5.1) | <.001 |
| Cardiometabolic | 382 (21.4) | 44 (20.7) | 338 (21.5) | 94 (18.1) | 287 (2.9) | ||
| Hematology-oncology | 107 (6) | 10 (4.7) | 97 (6.2) | 38 (7.3) | 69 (5.5) | ||
| Infectious disease | 77 (4.3) | 5 (2.4) | 72 (4.6) | 13 (2.5) | 64 (5.1) | ||
| Mental health | 216 (12.1) | 15 (7.0) | 201 (12.8) | 51 (9.8) | 163 (13) | ||
| Musculoskeletal or pain | 54 (3) | 6 (2.8) | 48 (3.1) | 12 (2.3) | 41 (3.3) | ||
| Neurology | 114 (6.4) | 29 (13.7) | 85 (5.4) | 58 (11.2) | 56 (4.5) | ||
| Obstetrics-gynecology | 63 (3.5) | 10 (4.7) | 53 (3.4) | 20 (3.9) | 43 (3.4) | ||
| Pulmonary | 113 (6.3) | 15 (7.0) | 98 (6.2) | 37 (7.1) | 76 (6.1) | ||
| Renal | 24 (1.4) | 3 (1.4) | 21 (1.3) | 8 (1.5) | 16 (1.3) | ||
| Substance abuse | 112 (6.3) | 9 (4.2) | 103 (6.6) | 21 (4.1) | 91 (7.2) | ||
| Surgery | 32 (1.8) | 7 (3.3) | 25 (1.6) | 14 (2.7) | 17 (1.4) | ||
| Wellness | 183 (10.2) | 4 (2.2) | 179 (11.4) | 35 (6.7) | 146 (11.6) | ||
| Other | 222 (12.4) | 51 (23.9) | 171 (10.9) | 98 (18.9) | 124 (9.9) | ||
| Fundingb | |||||||
| NIH or federal | 369 (20.7) | 31 (14.6) | 338 (21.5) | .02 | 72 (13.9) | 294 (23.4) | <.001 |
| Commercial or industry | 214 (12.0) | 48 (22.5) | 166 (0.6) | <.001 | 88 (17.0) | 125 (9.9) | <.001 |
| Other | 1602 (91) | 184 (86.4) | 1418 (90.3) | .07 | 467 (89.9) | 1131 (70.8) | >.99 |
| Trial size | |||||||
| 0-100 | 829 (46.5) | 102 (47.9) | 727 (46.3) | .002 | 317 (61.1) | 508 (40.4) | <.001 |
| 101-1000 | 812 (45) | 82 (38.5) | 730 (46.5) | 154 (29.7) | 657 (52.3) | ||
| >1000 | 142 (8.0) | 29 (13.7) | 113 (7.2) | 48 (9.3) | 92 (7.3) | ||
Abbreviations: NA, not applicable; NIH, National Institutes of Health.
Some studies listed multiple funding sources and target audiences.
Discussion
We characterized the digital health clinical research landscape. Although the number of registered studies increased by a mean of 29% per year from 2011 to 2017, many were small. Federally funded studies were more likely to use interventional designs and randomization. However, few studies have reported findings to date, even among studies marked completed or terminated.
Our use of the ClinicalTrials.gov database has some limitations, most notably that submission of digital health trials, unlike that for drugs and devices, remains voluntary.5 Although most stakeholders and sponsors generally require that prospective interventional trials be reported to ClinicalTrials.gov, these standards do not always apply to observational studies. Selection bias could lead to overestimation of the proportion of studies that are randomized across the full landscape and mean trial size, particularly if small pilot and nonregulated validation studies are underascertained. Because these data were extracted in 2017, ongoing assessment of the state of digital health studies is warranted.
Whether results will drive substantial clinical adoption is unknown because small studies, even if randomized, are unlikely to be significantly powered to demonstrate meaningful treatment effects. Although the pipeline of digital health studies appears to be promising, these factors could limit their ability to yield a high level of evidence, demonstrate value, or motivate stakeholder adoption.
References
- 1.Turakhia MP, Desai SA, Harrington RA. The outlook of digital health for cardiovascular medicine: challenges but also extraordinary opportunities. JAMA Cardiol. 2016;1(7):743-744. doi: 10.1001/jamacardio.2016.2661 [DOI] [PubMed] [Google Scholar]
- 2.Ahern DK. Challenges and opportunities of ehealth research. Am J Prev Med. 2007;32(5)(suppl):S75-S82. doi: 10.1016/j.amepre.2007.01.016 [DOI] [PubMed] [Google Scholar]
- 3.O’Neil A, Cocker F, Rarau P, et al. . Using digital interventions to improve the cardiometabolic health of populations: a meta-review of reporting quality. J Am Med Inform Assoc. 2017;24(4):867-879. doi: 10.1093/jamia/ocw166 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.National Institutes of Health ClinicalTrials.gov. https://clinicaltrials.gov/. Accessed June 19, 2017.
- 5.National Institutes of Health ClinicalTrials.gov FDAAA 801 requirements. https://clinicaltrials.gov/ct2/search/advanced?cond=&term=&state1=&cntry1=. Accessed June 19, 2017.
- 6.National Institutes of Health ClinicalTrials.gov advanced search. https://clinicaltrials.gov/ct2/search/advanced?cond=&term=&state1=&cntry1=. Accessed June 19, 2017.