Abstract
This database study evaluates the labeling, justifications, and limitations of pragmatic randomized clinical trials published in 2016.
Pragmatic randomized clinical trials (pRCTs) are considered a highly valuable design, providing evidence for clinical decisions in real-world settings.1 The conduct and setting of pRCTs mimic the usual practice of care while trying to maintain the internal validity of randomization,2 and they are becoming increasingly common.3 Their perceived and documented strengths and weaknesses need to be empirically studied beyond the theoretical expectations. In this study, we assessed the main characteristics among numerous recently published RCTs tagged as pragmatic, how the authors supported their claims for pragmatism, and whether major limitations stemming from the pragmatic design were noted in the articles.
Methods
We searched PubMed for articles published between January 1, 2016, and December 31, 2016. Title search terms included pragmatic or naturalistic and trial. Thus, we focused on trials whose authors were so confident about their trial’s pragmatism that they mentioned it in the title. Only pRCTs that reported primary results were eligible. We recorded any justifications and limitations regarding pragmatism noted in the articles. This study did not include individual patient–level data and used data only at the trial level. These data have been published and are widely available; thus, no institutional review board approval or informed consent was required.
Results
We identified 156 articles, including 73 eligible pRCTs (72 were labeled as pragmatic and 1 as naturalistic). Protocols (n = 38), nonprimary results (n = 25), non-RCTs (n = 14), duplicates (n = 2), and inaccessible articles (n = 4) were excluded. Characteristics of the pRCTs included in the study appear in Table 1. Of the 73 pRCTs included, 58 (79%) were registered on public databases and 33 (45%) reported having a protocol (available as a separate paper for 25 trials [76%]). Only 16 pRCTs (22%) had a median follow-up of more than 1 year. However, the median (interquartile range) time from the beginning of recruitment to publication was 6 (4-8) years and from the end of recruitment to publication was 3 (3-4) years (timing data were available for 65 trials [89%]).
Table 1. Design Features of the Pragmatic Randomized Clinical Trials.
| Features | Trials, No. (%) (N = 73) |
|---|---|
| No. of sites | |
| Single center | 24 (33) |
| 2-5 | 9 (12) |
| 6-10 | 11 (15) |
| 11-20 | 7 (10) |
| 21-50 | 13 (18) |
| >50 | 8 (11) |
| Missing data | 1 (1) |
| Country | |
| United Kingdom | 29 (40) |
| United States | 16 (22) |
| The Netherlands | 4 (5) |
| 18 Other countries | 22 (30) |
| >1 Country | 2 (3) |
| Design | |
| Participant-level RCT | 63 (86) |
| Cluster RCT | 10 (14) |
| No. of participants enrolled | |
| <200 | 22 (30) |
| 200-500 | 21 (29) |
| >500 | 30 (41) |
| Length of follow-up, mo | |
| <6 | 25 (34) |
| 6-12 | 31 (43) |
| 13-18 | 6 (8) |
| >18 | 11 (15) |
| No. of arms | |
| 2 | 64 (88) |
| ≥3 | 9 (12) |
| Blinding | |
| Assessor | 31 (43) |
| Double | 5 (7) |
| Participants only | 1 (1) |
| None | 36 (49) |
| Recruitment embedded in health care | |
| Yes | 33 (45) |
| No: Identification through records and invitations | 16 (22) |
| No: Use of advertisements | 7 (10) |
| Unclear | 17 (23) |
| Use of EHR | |
| To recruit and collect data | 7 (10) |
| To recruit | 7 (10) |
| To collect data | 4 (5) |
| Mention of medical records but unclear if EHR | 13 (18) |
| Unclear | 33 (45) |
| Clearly nonea | 9 (12) |
| Type of primary end points used | |
| Clinical | 23 (32) |
| Clinical but self-reported | 12 (16) |
| Diagnostic questionnaires | 11 (15) |
| Quality-of-life scales/questionnaires | 10 (14) |
| Composite | 2 (3) |
| Surrogate | 6 (8) |
| Multiple pimary end points | 5 (7) |
| Others | 4 (5) |
| Therapeutic area | |
| Psychiatry/addiction | 18 (25) |
| Cardiovascular | 8 (11) |
| Gynecology/obstetrics | 5 (7) |
| Infectious diseases | 5 (7) |
| Others | 37 (51) |
| Interventions | |
| Care managementb | 23 (32) |
| Medicines (drugs, biologics) | 11 (15) |
| Psychotherapy | 11 (15) |
| Lifestylec | 4 (5) |
| Professional training | 4 (5) |
| Other | 20 (27) |
| Control | |
| Usual care | 32 (44) |
| Different mode of administration | 26 (36) |
| Different nature of intervention | 6 (8) |
| No intervention or wait-list | 6 (8) |
| Placebo | 1 (1) |
| Different administration or technique plus placebo | 2 (3) |
| Funding source | |
| Public | 58 (79) |
| Public, but supplies provided by industry | 6 (8) |
| Private foundation | 4 (6) |
| Industry | 4 (6) |
| Public and private foundations | 1 (1) |
Abbreviations: EHR, electronic health record; RCT, randomized clinical trial.
Three trials were performed involving health care professionals with no health-related outcomes and did not require the use of EHRs.
Five trials were performed using only the internet or the phone as an interface.
Two trials were performed using only the internet.
In 33 articles (45%), the authors did not justify the pragmatic label. In only 1 article did the authors formally assess pragmatism in their trial using the PRECIS (Pragmatic-Explanatory Continuum Indicator Summary) tool.4 The justifications for the claimed pragmatism were diverse (Table 2). Twenty-eight articles (38%) reported limitations resulting from the pragmatic nature of the trial. The two most common limitations were the high rates of missing data with an increased risk of attrition bias (cited in 14 of 28 articles [50%]) and the heterogeneity of the delivery and uptake of the intervention (cited in 13 of 28 articles [46%]). These limitations were often used as a potential explanation for the lack of substantial results. Lack of blinding was mentioned as a limitation in 4 articles (14%).
Table 2. Justifications for the Degree of Pragmatism Claimed in Randomized Clinical Trials.
| Justificationa | Trials, No. (%) (n = 40) |
|---|---|
| Settings and conduct of the trial mimic current practice, with the standard of care remaining unchanged | 25 (63) |
| Flexibility in the delivery of the intervention | 16 (40) |
| Broad eligibility criteria and representative sample | 12 (30) |
| Flexibility in the intervention itself | 6 (15) |
| Flexibility in the follow-up of participants | 5 (13) |
| Use of existing resources | 3 (8) |
| Absence of blinding of participants and investigators | 3 (8) |
| Primary outcome clinically relevant to clinicians and participants | 2 (5) |
| Intention-to-treat analysis | 2 (5) |
| Informed consent waived completely or until data were collected | 2 (5) |
| Flexibility in the recruitment process | 1 (3) |
| Change in the protocol reflecting a change in policies during trial | 1 (3) |
| Use of the PRECIS tool | 1 (3) |
Abbreviation: PRECIS, Pragmatic-Explanatory Continuum Indicator Summary.
Only 40 articles reported justifications for the pragmatic nature of the trials. All except 10 reported more than 1 justification.
Discussion
Recent RCTs labeled as pragmatic are mainly multicenter trials randomized at the participant level. They typically compare an intervention of interest with usual care or with the same kind of intervention that uses different modes of administration, and they are conducted in a single country. Notably, 58 pRCTs in this study (79%) were supported by only public funds. Few trials conducted more than 12 months of follow-up, but most trials took several years to get published, suggesting delays in both recruitment and publication after completion, which are shown also in studies of other sets of RCTs.5 The efficiency of the pRCT agenda can probably be substantially improved. Whether many more pRCTs exist but never get published is unknown. In our study, a minority of published trials were unregistered and most lacked publicly available protocols.
Many pRCTs have some clear explanatory features,3 such as single center, double-blind, placebo, and incentives for participants. The authors of 45% of the articles did not justify or discuss the claimed pragmatism of their trials, and many other justifications were rudimentary. The term pragmatic in the title may be attractive to readers and policymakers, but the inclusion of the term may not guarantee the application of pragmatism. Use of tools such as PRECIS-21 to assess the degree of pragmatism should be encouraged and would help the scientific community adequately label trial designs in a transparent and standardized way.3 In addition, investigators should be encouraged to discuss routinely the limitations and potential consequences of the pragmatic study design.
The US Food and Drug Administration is increasingly considering the use of real-world data for regulatory decision making,6 and pRCTs are becoming more prominent for nonregulated interventions. For these reasons, establishing and adopting standards for high-quality pRCTs have become important.
References
- 1.Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147. doi: 10.1136/bmj.h2147 [DOI] [PubMed] [Google Scholar]
- 2.Chalkidou K, Tunis S, Whicher D, Fowler R, Zwarenstein M. The role for pragmatic randomized controlled trials (pRCTs) in comparative effectiveness research. Clin Trials. 2012;9(4):436-446. doi: 10.1177/1740774512450097 [DOI] [PubMed] [Google Scholar]
- 3.Dal-Ré R, Janiaud P, Ioannidis JPA. Real-world evidence: how pragmatic are randomized controlled trials labeled as pragmatic? BMC Med. 2018;16(1):49. doi: 10.1186/s12916-018-1038-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Thorpe KE, Zwarenstein M, Oxman AD, et al. . A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62(5):464-475. doi: 10.1016/j.jclinepi.2008.12.011 [DOI] [PubMed] [Google Scholar]
- 5.Chen R, Desai NR, Ross JS, et al. . Publication and reporting of clinical trial results: cross sectional analysis across academic medical centers. BMJ. 2016;352:i637. doi: 10.1136/bmj.i637 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Sherman RE, Anderson SA, Dal Pan GJ, et al. . Real-world evidence—what is it and what can it tell us? N Engl J Med. 2016;375(23):2293-2297. doi: 10.1056/NEJMsb1609216 [DOI] [PubMed] [Google Scholar]