A review of pragmatic trials found a high degree of diversity in design and scope, deficiencies in reporting and trial registry data, and poor indexing

Abstract

Objective:

We established a large database of trials to serve as a resource for future methodological and ethical analyses. Here, we use meta-data to describe the broad landscape of pragmatic trials including research areas, identification as pragmatic, quality of trial registry data and enrolment.

Study Design and Setting:

Trials were identified by a validated search filter and included if a primary report of a health-related randomized trial published January 2014-April 2019. Data were collated from MEDLINE, Web of Science, ClinicalTrials.gov, and full text.

Results:

4337 eligible trials were identified from 13,065 records, of which 1988 were registered in ClinicalTrials.gov. Research areas were diverse, with the most common being general and internal medicine; public, environmental and occupational health; and health care sciences and services. The term “pragmatic” was seldom used in titles or abstracts. Several domains in ClinicalTrials.gov had questionable data quality. We estimated that one-fifth of trials under-accrued by at least 15%.

Conclusion:

There is a need to improve reporting of pragmatic trials and quality of trial registry data. Under accrual remains a challenge in pragmatic RCTs despite calls for more streamlined recruitment approaches. The diversity of pragmatic trials should be reflected in future ethical analyses.

1. Introduction

Randomized controlled trials (RCTs) vary widely in purpose and design. Some focus on efficacy of an intervention under highly controlled conditions, while others study effectiveness of an intervention under ‘real-world’ circumstances and seek to provide information pertinent to a specific health care decision. The former are often described as having an explanatory or mechanistic intent [1, 2] and the latter as “practical”, “naturalistic” [2–4] or pragmatic in their intent [1, 5, 6]. As interest in more pragmatic RCT designs has grown in recent years [7, 8], tools such as the Pragmatic-Explanatory Continuum Indicatory Summary (PRECIS-2) [9] have sought to assist investigators in matching specific design features to the intended study question/clinical decision, while an extension of the Consolidated Standards for Reporting of Trials also exists to support authors reporting the results of pragmatic RCTs [10].

While a robust body of methodological and ethical literature exists to inform RCTs in general, this has largely been developed in the context of explanatory RCTs testing investigational drugs. However, more pragmatic RCT designs potentially raise distinct challenges [11, 12]. For example, the use of cluster randomization, inclusion of broader and more heterogenous populations, streamlined recruitment processes, and greater flexibility in the delivery of interventions have implications for design and sample size calculation procedures [13]. These design features also raise questions regarding consideration of benefits and harms, the appropriate inclusion and protection of vulnerable or neglected populations, and determining when it may be appropriate to adopt alternative consent approaches or to apply a waiver of consent [11].

To better understand and address these challenges, reviews and analyses of published pragmatic trials are required. Unfortunately, existing reviews of RCTs determined to be pragmatic have been small, have focused on specific subsets of trials, used arbitrary criteria to select studies or have been restricted to a finite number of journals [14–17]. Furthermore, of the few studies that have attempted to review the literature on pragmatic RCTs, most rely on the use of the term “pragmatic” in the title or abstract as a major inclusion criterion [4, 14] (see Supplementary Table S1). Although this approach is reasonable on its face, it will fail to capture trials that may be consistent with a pragmatic intent or design, but which do not use the term “pragmatic”.

As part of a larger project to develop guidance for the ethical design and conduct of more pragmatic RCTs [18], we created a large database of trials deemed to be more likely pragmatic and which would serve as a resource for future methodological and ethical analyses. Here, we describe the population of trials with respect to registration and reporting, as well as key features such as interventions under study, trial size, and accrual success given recent interest in pragmatic design features which may facilitate recruitment [19–22]. Our main objectives were to:

1. Describe this sample of RCTs published between 2014–2019 with respect to characteristics such as clinical areas, countries(s) of investigators, funding sources, purpose, interventions, and participant demographics;

2. Determine completeness of registration and its reporting;

3. Determine the extent to which the sample of trials are identified as pragmatic in titles or abstracts and indexed by the National Library of Medicine (NLM) as pragmatic trials;

4. Describe trial size and accrual success using information available in the ClinicalTrials.gov registry.

2. Methods

2.1. Search strategy

Trials with a pragmatic intent have no single unique design feature, making them challenging to identify in the literature. Similarly, no reporting guidelines require trials to self-label as “pragmatic” [10], meaning that reliance on the term pragmatic would likely miss many relevant trials. Given the lack of an existing validated search string, we developed and validated a highly specific electronic search filter to identify pragmatic RCTs in MEDLINE based on terms such as “pragmatic”, “real world”, “registry based”, “comparative effectiveness”, “evidence based”, “patient oriented”, or “usual care” [23]. In the present analysis we applied the previously published sensitivity-maximizing variant of the filter. The filter consists of a design domain which includes terms relating to the trial design, while the attribute domain is organized into descriptors pertaining to the setting, data collection or data sources, intervention and comparator, type of analysis, and outcomes. In a validation cohort, the sensitivity-maximizing variant had a false positive rate of 1.9%, indicating that few trials retrieved by the filter would be deemed not to be pragmatic. The search filter was applied in MEDLINE on 3 April 2019 and covered the period 1 January 2014 to that date. We chose 2014 as the start date as this was the year that the National Library of Medicine began indexing articles as “pragmatic clinical trial as publication type” as well as topic.

2.2. Inclusion and exclusion criteria

Inclusion and exclusion criteria, with examples and definitions, are available in the Supplementary Material S2. Studies were included if they were the primary report of an RCT of a health or health care intervention with a target accrual of at least 100 individuals. A health RCT was defined as an RCT which evaluates interventions aimed at changing subjective or objective measures of individual or group health status, or of processes which lead to changes in health status. A health care RCT was defined as an RCT aimed at evaluating changes to the processes of delivering care. Studies were excluded if they were not a primary report of a completed RCT (e.g., studies presenting only secondary, interim or subgroup analyses) or obviously were not of pragmatic orientation. While tools such as PRECIS-2 have been developed to assist investigators during the design of their trial [9, 24], studies that have attempted to retrospectively assess the degree of pragmatism within a trial using PRECIS-2 have faced multiple problems with respect to the quality of trial reporting and application of the tool [25–28]. In addition, studies vary in the thresholds applied to demarcate more pragmatic from less pragmatic trials [29, 30] and there is a lack of conceptual clarity regarding the appropriate weighting or relevance of the individual domains [31–33]. Thus, given the high specificity of our published search filter and the large sample size, we did not review each trial individually to score its degree of pragmatism; instead, we relied on our search filter to retrieve trials more likely to be pragmatic [23]. However, as the search filter did not have perfect specificity, we anticipated having to eliminate a small number of trials that were obviously not of a pragmatic orientation. To do this we chose to apply a subjective evaluation based on the overall gestalt of the trial, informed by the design features, but also the stated goals and the interpretation of results. For example, trials that focused on isolating a biological impact of an intervention without a clear clinical implication, or that did not assess clinical outcomes, were deemed more likely to indicate an explanatory trial and were thus excluded. Finally, we set a target sample size threshold of 100 as we wanted to exclude pilot trials where the intention is not to inform practice or policy.

2.3. Study selection

The titles and abstracts of all articles retrieved were imported into Covidence [34] to facilitate screening. A sample of 468 records was reviewed in duplicate by three reviewers (KC, SN, and MT) as part of a training and calibration process. Following review all reviewers met to discuss inclusion and exclusion decisions until consensus was achieved. The remaining records were distributed among the three reviewers with a single reviewer screening each record. Abstracts were only excluded if they could clearly be assigned to one of the exclusion criteria. If the reviewer was uncertain, the trial was included and progressed to full text screening.

Full text publications of studies potentially meeting the eligibility criteria were examined by three reviewers (KC, SN, JZ) to extract information pertaining to trial registration. Trial registration information was used to facilitate identification of the primary trial report from among multiple publications from the same trial. Reports were also scrutinized for any explicit statement referring to primary results being published previously or that the present article was presenting a secondary analysis. Following the exclusion of non-primary reports, and after further calibration, the full texts of articles were distributed among five reviewers (MT, SN, KC, JZ, HN). Each article was reviewed against all inclusion and exclusion criteria by one individual. When there was uncertainty regarding the assessment of a trial, the trial was discussed by all reviewers as well as clinical members of the team where necessary. In all cases, articles were discussed until a group consensus was reached.

2.4. Data elements and extraction

To maximize the sample that ould feasibly be included within the analysis, data extraction was streamlined by utilizing existing data relating to included trials and trial reports. Data pertaining to the included studies were drawn from four sources: Ovid MEDLINE, Web of Science [35], ClinicalTrials.gov (where available), and the full text manuscript. ClinicalTrials.gov is the oldest and largest trial registry [36, 37], and has been used for hundreds of analyses mapping the scope of different clinical specialties and trial reporting [e.g. 36, 38–41]. All data elements were automatically extracted from the XML files of each record, using the Beautiful Soup Python library [42]. The full text of included articles was manually searched for any use of the word “pragmatic” to describe the trial. Data elements were combined within a single database using Airtable [43]. Table 1 outlines the data elements extracted from each data source.

Table 1.

Data sources and data elements used in review of trials retrieved by our search.

Data source	Data elements
MEDLINE	Title, Author list, Abstract, Year of publication, Keywords, Medical Subject Heading (MeSH) terms, Trial registration number contained within the Secondary Source ID field, Use of the term “pragmatic” in title or abstract, Use of “pragmatic” as MeSH topic or publication type *
Web of Science (WoS)	Designated research area, Country/region of all listed authors (derived by WoS from the listed institutional affiliations), Funding source(s)
ClinicalTrials.gov	Trial phase, Primary purpose of the trial, Type(s) of intervention, Trial masking, Sex of participants, Minimum and maximum eligible ages for trial participation, Listed facilities (study sites)$, Target accrual (obtained from the first registered entry in the “original estimated enrolment” field, or if not available, the “estimated enrolment” field), Actual enrolment, Primary outcome$
Full text article	Registry information not in abstract or Secondary Source ID, Use of the word “pragmatic” to describe the trial within the main text

^*As database indexing lags behind publication (records are made available without MeSH terms assigned but are later updated), the data regarding indexing of the trial was updated after the search was conducted (last update December 2, 2020).

^$We counted the number of separate entries to derive a numerical value for the number of primary outcomes and number of study sites.

2.5. Analysis

We described categorical variables using frequencies and percentages, and continuous variables using mean and standard deviation, and median and interquartile range (Q1–Q3). To assess success of accrual, we calculated the ratio of actual accrual versus target enrolment. As per Carlisle et al. [44], we used different thresholds to categorize trials in terms of their accrual success, namely; less than 50% accrual of target sample size, 50% < 85% of target sample size, 85% < 115% of target sample size, 115% < 150% of target sample size and 150% or more of the target sample size. Given a lack of consensus regarding successful or unsuccessful accrual, the thresholds used were arbitrary but allowed for comparison with previous studies.

3. Results

Fig. 1 shows the flow of articles through the screening process. Of 13,065 articles retrieved, 8728 (67%) were excluded following screening of titles and abstracts, and full text. The main reasons for exclusion were non-primary trial report (32%), target accrual of less than 100 participants (20%), and not an RCT (19%). Across both stages of screening, 159 (1.2%) studies were excluded on the basis that they were deemed not to be pragmatic, consistent with the low false positive rate of 1.9% reported in the external validation of the search filter. Our final sample included 4337 full text articles, of which 4241 (98%) could be linked to records in Web of Science.

Fig. 1.

Study flow diagram.

3.1. Trial demographics

Descriptive information for the included trials is presented in Table 2. Of the 4241 records linked to Web of Science, 141 different countries were associated with affiliations for listed authors. The most frequent countries were the USA, UK, Australia, the Netherlands and Canada (see Supplementary Figure). A total of 3793 trials had funding information listed from 4932 different funders. The major funders were the United States Department of Health & Human Services (949; 22%), US National Institutes of Health (871; 21%), UK National Institute for Health Research (336; 8%), UK Medical Research Council (191, 5%) the US National Cancer Institute (173; 4%), and the National Health and Medical Research Council of Australia (166; 4%). The most common research areas were general & internal medicine (1012, 24%); public, environmental & occupational health (577; 14%); health care sciences & services (365; 9%); psychiatry (318; 8%), and psychology (244; 6%).

Table 2.

Descriptive overview of the sample of trials included [Data sources: MEDLINE (N = 4337) or Web of Science (N = 4241)].

Item	N (%)
Year of publication (N = 4337)
2014	694 (16%)
2015	816 (18.8%)
2016	778 (17.9%)
2017	872 (20.1%)
2018	866 (20%)
2019 *	311 (7.2%)
Top 10 Research Areas according to Web of Science ** (N = 4241)
General & Internal Medicine	1012 (23.9%)
Public, Environmental & Occupational Health	577 (13.6%)
Health Care Sciences & Services	365 (8.6%)
Psychiatry	318 (7.5%)
Psychology	244 (5.8%)
Pediatrics	195 (4.6%)
Science Technology Other topics	173 (4.1%)
Cardiovascular systems & Cardiology	172 (4.1%)
Oncology	170 (4%)
Infectious Diseases	165 (3.9%)
Top 10 Countries of Author Affiliations *** (N = 4241)
USA	1779 (41.9%)
UK	1086 (25.6%)
Australia	452 (10.7%)
Netherlands	400 (9.4%)
Canada	295 (7%)
China	266 (6.2%)
Germany	220 (5.2%)
France	156 (3.7%)
Denmark	148 (3.5%)
Spain	145 (3.4%)
Top 10 Funders (N = 4241)Ψ
United States Department of Health & Human Services	949 (22.4%)
National Institutes of Health (USA)	871 (20.5%)
National Institutes for Health Research (UK)	336 (7.9%)
Medical Research Council (UK)	191 (4.5%)
NIH National Cancer Institute (NCI)	173 (4.1%)
National Health and Medical Research Council of Australia	166 (3.9%)
NIH National Center for Advancing Translational Sciences (NCATS)	155 (3.7%)
NIH National Heart Lung Blood Institute (NHLBI)	133 (3.1%)
NIH National Institute of Mental Health (NIMH)	130 (3.1%)
Netherlands Organization for Health Research and Development	114 (2.7%)

^*Search conducted April 3, 2019.

^**Articles could be assigned multiple Research Areas, so does not sum to 4241.

^***Derived from author affiliation information for all listed authors. Authors could have multiple affiliations, hence does not sum to 4241.

^ΨFunding agency as recorded in the field funding within Web of Science: https://images.webofknowledge.com/images/help/WOS/hs_funding_agency.html.

3.2. Self-identification as pragmatic trial

Overall, 964 (22%) of the 4337 studies used the word pragmatic to describe the trial anywhere in the title, abstract, or full text (Table 3); of these, 534 (55%) used the word in the title or abstract with the remaining 430 (45%) using it only in the full text.

Table 3.

Prevalence of reporting as pragmatic trial and trial registration (Data source: MEDLINE N = 4337).

Item	N (%)
Identification as pragmatic trial
Self-identification as “pragmatic”:
Anywhere (title, abstract or main text)	964 (22.2%)
In title or abstract	534 (12.3%)
In main text only (not in title or abstract))	430 (9.9%)
Identification as pragmatic by NLM
Pragmatic Clinical Trial as Publication type *	268 (6.2%)
Pragmatic Clinical Trials as Topic *	22 (0.5%)
Trial registration
Registration indicated anywhere in text	3386 (78.1%)
In abstract (n = 3386)	1991 (58.8%)
In full text only (n = 3386)	1395 (41.2%)
Registration not reported	952 (21.9%)
Trial Registry, if registered ** (n = 3386)
ClinicalTrials.gov	1988 (58.7%)
ISRCTN	660 (19.5%)
ANZCTR	286 (8.4%)
Other registry only	466 (13.8%)

^*As per December 2, 2020.

^**Trials may be registered in multiple registries, so does not sum to 3386.

A total of 268 articles (6%) were indexed in MEDLINE as Pragmatic Clinical Trial as Publication Type (See Supplementary Material S3) and 22 (0.5%) as Pragmatic Clinical Trials as a MeSH Topic. Of the 268 indexed as Pragmatic Clinical Trial as Publication Type, 231 (86%) self-identified as pragmatic in the title or abstract. When considering all articles that self-identified as pragmatic in either the title, abstract, or full text, 702 (73%) were not indexed as Pragmatic Clinical trial as Publication Type. Further, of the 534 articles that identified as pragmatic in the title or abstract, 303 (57%) were not indexed as Pragmatic Clinical trial as Publication Type.

3.3. Registration

Trial registration data are summarized in Table 3. Of the 4337 included reports, 3386 (78%) explicitly mentioned within the manuscript that the trial was registered in a clinical trials registry. Of those trials that were registered, 1991 (59%) had registration numbers included in the abstract in MEDLINE. Overall, studies were registered across 44 different repositories. The most commonly used registries were ClinicalTrials.gov (1988, 59%), the UK International Standard Randomized Controlled Trial Number registry (660, 20%), and Australian New Zealand Clinical Trials Registry (286, 8%).

3.4. ClinicalTrials.gov subset

Descriptive summaries of trial characteristics from ClinicalTrials.gov data (N = 1988) are presented in Table 4 (for additional comparisons with the full sample see Supplementary Material S4). The most commonly registered primary purposes were treatment (746, 38%), prevention (500, 25%), and health services research (342, 17%). The most common types of interventions were indicated as behavioral (903, 45%), other (635, 32%), or drug (324, 16%). Half of the registrations indicated no masking or blinding (1005, 51%), while 30% (588) indicated single blinding and 9% (172) double blinding.

Table 4.

Trial participant, intervention, and other descriptors in registry (Data source: ClinicalTrials.gov, N = 1988).

Item	N (%)
Participant Sex
Female only	225 (11.3%)
Male only	29 (1.5%)
Both	1730 (87%)
Missing	4 (0.2%)
Participant Age *
Children (< 18) only	268 (13.5%)
Senior (65+) only	72 (3.6%)
Mixed ages	1617 (81.3%)
Missing	31 (1.6%)
Phase (categories as in ClinicalTrials.gov)
Early Phase 1	12 (0.6%)
Phase 1	9 (0.5%)
Phase 1/Phase 2	15 (0.8%)
Phase 2	63 (3.2%)
Phase 2/Phase 3	38 (1.9%)
Phase 3	208 (10.5%)
Phase 4	244 (12.3%)
Phase N/A	1368 (68.8%)
Missing	31 (1.6%)
Type of intervention (categories as in ClinicalTrials.gov)Ψ
Behavioral	903 (45.4%)
Biological	41 (2.1%)
Device	155 (7.8%)
Dietary	53 (2.7%)
Drug	324 (16.3%)
Procedure	128 (6.4%)
Radiation	10 (0.5%)
Other	635 (31.9%)
Missing	14 (0.7%)
Primary purpose (categories as in ClinicalTrials.gov)
Treatment	746 (37.5%)
Prevention	500 (25.2%)
Health Services Research	342 (17.2%)
Supportive Care	158 (7.9%)
Screening	53 (2.7%)
Basic Science	8 (0.4%)
Diagnostic	66 (3.3%)
Education/Counseling	1 (0.1%)
Other	38 (1.9%)
Missing	76 (3.8%)
Number of primary outcomes
Min, Max	1, 20
Median (Q1–Q3)	1 (1–1)
Mean (SD)	1.46 (1.21)
1	1516 (76.2%)
2	279 (14%)
3 or more	186 (9.4%)
Missing	7 (0.4%)
Masking
None	1005 (50.6%)
Single	588 (29.5%)
Double	172 (8.6%)
Triple	72 (3.6%)
Quadruple	106 (5.3%)
Missing	45 (2.3%)
Number of facilities
Min, Max	1, 916
Median (Q1–Q3)	1 (1–2)
Mean (SD)	8.6 (45.4)
1	1203 (61%)
> 1	619 (31.1%)
Missing	166 (8.4%)

^*Using the minimum and maximum ages a potential participant must be to be eligible for the study, we established whether a trial solely involved children or infants (applying an upper threshold of 18 years using the maximum age category, and applying no minimum threshold), an older population (applying a lower threshold of 65 years using the minimum listed age for participants and no maximum age limit), or included a mix of ages.

^ΨDoes not sum to 100% as trials may have multiple interventions.

Over two-thirds of the registrations indicated trial phase as ‘Not Applicable’ (1368, 69%) and almost one in four were registered as either Phase 3 or Phase 4 (452, 23%). However, 12 trials were listed as ‘Early Phase 1’ while 31 (2%) had no phase listed. Scrutiny of the trials listed as “Early phase 1” indicated that they were often large health services intervention trials including some cluster randomized trials, as opposed to smaller single-center studies.

The number of primary outcomes ranged from 1 to 20. The majority of RCTs (1516, 76%) had a single primary outcome listed, 279 (14%) had two, while 186 (9%) had three or more. Almost a third were multisite studies (619, 31%). The number of facilities (study sites) per study ranged from 1 to 916.

3.5. Accrual rates

Target enrolment and actual accrual data are summarized in Table 5. Of the 1988 trials in ClinicalTrials.gov, a small proportion had no target accrual (42, 2%) or actual accrual (231, 12%) data. We set a further 86 registry values to missing as these were in the extreme tails of the distribution, and after comparison to their full text reports, were clearly erroneous. Thus, we had available target accrual data for 1897 trials, actual accrual data for 1720, and both target and actual accrual data for 1682 trials. Examples of obvious erroneous trial registry data included accrual data based on the number of clusters, mean cluster sizes, and total population sizes rather than the number of participants. Target enrolment in the registry entry ranged from 30 to 800,000 with a median (Q1–Q3) of 440 (224–1200), while actual accrual ranged from 60 to 933,789 with a median (Q1–Q3) of 414 (216 – 1147).

Table 5.

Planned and actual accrual data in registry (Data Source: ClinicalTrials.gov, N = 1988).

Accrual data	Descriptive summary
Target enrolment/anticipated enrolment *
Min, Max	30, 800000
Median (Q1–Q3)	440 (224–1200)
Frequency Missing	91
Actual enrolment *
Min, Max	60, 933789
Median (Q1–Q3)	413.5 (216.25–1146.50)
Frequency Missing	268
Accrual Ratio (actual/target enrolment) *
Min, Max	0.1, 4.8
Median (Q1–Q3)	1.0 (0.9–1.0)
Frequency Missing	306
Ratio thresholds (N = 1682)	N (%)
<0.5 (i.e., less than 50% of target enrolment achieved)	83 (4.9%)
0.5 to <0.85	274 (16.3%)
0.85 to <1.15	1082 (64.3%)
1.15 to <1.5	141 (8.4%)
≥1.5 (i.e., more than 150% of target enrolment achieved)	102 (6.1%)

^*Extreme outliers that were clearly erroneous have been set to missing.

According to the trial registry data, most trials (1082/1682, 64%) accrued between 85% and 115% of their target. One-fifth of trials (357, 21%) under-accrued 15% or more, while a smaller number severely under-accrued, reaching less than 50% of their registered target (83, 5%). In contrast, a smaller proportion (243, 14%) over-accrued by at least 15%. Almost 100 trials (102, 6%) substantially over-accrued, achieving greater than 150% of their target.

4. Discussion

4.1. Summary of key findings

Our results indicate that RCTs that are more likely to be pragmatic are diverse in scope and interventions. Among the sample of registered trials reviewed, the most common types of interventions were behavioral, as opposed to drugs or devices, while many were registered as ‘other’. While treatment was the most commonly registered purpose, prevention and health services research were also frequently registered.

Our sample of more pragmatic RCTs appeared to be predominantly conducted by investigators with affiliations in North America and Europe and were largely funded by non-industry sources. Target enrolment was hundreds to thousands of participants, although many studies were single center, and most studies met or exceeded their planned enrolment. However, around one in five reported achieving less than 85% of their registered target. Almost one in four trials did not report trial registration; of those that did, just over half did so in the abstract. The quality of trial registry data was poor in several areas: over 10% of the trials in ClinicalTrials.gov did not include accrual data despite a published manuscript, while an additional 4% of trials were deemed to have obvious errors in this same field. For example, accrual data for cluster randomized trials were notably erroneous for several studies within this sample. Similarly, several studies were registered as early phase trials, despite being large comparative effectiveness studies. Failure to clearly identify a pre-specified primary trial outcome at the time of registration increases the likelihood of outcome selection; yet, nearly 10% of registry entries listed multiple “primary” outcomes. Finally, only a small proportion of trials self-identified as pragmatic in titles or abstracts; almost half of all articles that identified as pragmatic did so only in the main text, and thus will not be retrieved by text searches of the title or abstract. Relying on indexing as a pragmatic clinical trial by the NLM will retrieve only a small fraction of trials likely to be pragmatic.

4.2. Comparison with other studies

The primacy of authorship by investigators in the UK, USA, Australia, and Europe aligns with previous analyses of trial authorship of RCTs more generally [14–16, 45]. While the USA was the country with most trials registered in ClinicalTrials.gov, the UK had the most trials registered in ClinicalTrials.gov where the trial report self-identified as a pragmatic trial. This distribution is consistent with the findings of Patsopoulos who noted a similar trend in trials registered in ClinicalTrials.gov up to 2011 [8], and may point to linguistic differences across jurisdictions. When reviewing the trials registered in ClinicalTrials.gov and those that were not, there were no obvious differences with respect to the main research areas (Supplementary Material – Table S4).

Our results also point to potential differences between more pragmatic RCTs and the larger body of published RCTs. First, a higher proportion of RCTs within our sample were funded by non-industry sources compared to published studies of RCTs more generally [40, 46–50], where industry funding has been reported to be as high as 66% [47]. Indeed, our findings indicate a greater similarity in funding to health policy and systems research, where up to 77% of studies may have governmental funding, and only 3% receiving solely private-for-profit funding [51]. This may also be associated with the finding that trial accrual was suboptimal for approximately 20% of the analyzed trials. Several studies of accrual have found that non-industry funded trials tend to have poorer accrual than industry funded trials [44, 52, 53].

Further, a substantial number of trials in our sample had a listed purpose of prevention (25%) or health services research (17%), both of which are higher than previous studies of RCTs more generally [40]. Our finding that 51% of trials registered in ClinicalTrials.gov were classified as open-label is consistent with that of Janiaud et al., who reported that 36/73 (49%) of self-declared pragmatic trials reported no blinding [14].

4.3. Limitations

Our results must be interpreted in light of the following limitations: First, while using the sensitivity-maximizing version, our search filter was developed with an overall focus on specificity, aiming to improve the efficiency of pragmatic trial retrieval over general RCT search filters such as the Cochrane highly sensitive search; thus, our results should not be regarded as describing the complete landscape of all pragmatic trials published since 2014. Second, no consensus exists in the literature for the retrospective categorization of trials as pragmatic. Given the infeasibility of conducting a PRECIS-2 analysis for all retrieved articles [54], and lack of agreed-upon thresholds upon which to classify a trial as pragmatic [32, 33], no formal assessment was made with respect to scoring each trial on the PRECIS-2 framework. While the degree of pragmatism of each trial in our database was not assessed, our full text analysis illustrates the limitations of relying on the self-identification of trials using the word “pragmatic” in the title and abstract alone. Despite these limitations, our cohort of trials remains the largest sample of more pragmatic RCTs published to date.

4.4. Key implications

Our study has several important implications for future studies. First, similar to previous studies which have noted issues with data quality [55, 56] and discrepancies between trial registry data and published trial results [39, 50, 57, 58], our analysis points to several trial registry data elements that may be incomplete or unreliable for RCTs that have more pragmatic designs. For example, almost 10% of trials had no study sites listed in their registration record. A small proportion listed no interventions, and some studies were miscategorized as “observational” despite clearly being reported as an RCT. Study phase was also often listed as “N/A”. This may point to difficulties in categorizing RCTs with more pragmatic designs, particularly when trials do not involve drug interventions with a clearly defined developmental pathway. While this may be appropriate, insofar as it reflects a system of categorization designed for investigational drugs, it also creates a category of trials that are highly heterogenous and which lack key information for policy decision-makers or other knowledge users who may wish to use the trial results to inform practice. Furthermore, we noted several trials with substantial discrepancies between sample size estimates registered at the first entry and upon completion (actual accrual), as well as several examples where data were clearly erroneous. Notably, for studies with trial registry data indicating a target accrual of less than 100 participants, common errors were registration of the number of clusters, mean cluster sizes, total cluster sizes, or registration of participants in the intervention arm only. Validation studies of trial registry data are thus essential prerequisites to future analyses of trial registry data, particularly in relation to analyses of accrual success.

Second, our results have important implications with respect to trial reporting and the planning of future reviews of pragmatic trials: a substantial proportion of trial reports only identified themselves as a pragmatic RCT in the main text of the article. This indicates that a considerable number of trials would not be identified through text word searches of titles and abstracts and would thus have been systematically overlooked by the methods used in some prior reviews [4, 14, 59]. However, virtually all trials that were tagged by NLM as pragmatic self-identified as such, perhaps indicating that NLM’s classification scheme is reliant on self-identification of a trial as pragmatic by authors, particularly within the title or abstract of the article. While no reporting guidelines currently require RCTs to self-identify as being pragmatic [10, 26], we recommend that authors who view their trial as having a pragmatic intent clearly signal this through explicit use of the term pragmatic in the title or abstract of their trial report. This will not only facilitate the identification of pragmatic RCTs through title and abstract searching but may also facilitate NLM indexing of these trials as Pragmatic Clinical Trial with respect to publication type or topic.

Third, the International Committee of Medical Journal Editors (ICMJE) recommends registration of clinical trials in a public trial registry prior to first patient enrolment, and publication of the trial registration number at the end of the abstract in published trial reports. In the present study, we found variation in the reporting of trial registration information, with 78% of trial reports including registration information, but just over half of those (59%) providing this in the abstract or meta-data of the publication record. There is a need for further work to better understand the barriers and facilitators to reporting trial registration, with a view to developing interventions that can improve practices beyond the levels seen here.

Fourth, our results have implications for future studies about ethical issues in trials with more pragmatic designs. To date, such studies have often focused on a limited subset of trials, such as comparative effectiveness RCTs of existing drugs, or RCTs conducted within clinical settings and with a limited range of interventions [60–65]. Our sample of likely pragmatic RCTs suggests a broader range of more complex interventions—being studied across a broader range of contexts—than is currently reflected in the pragmatic RCT ethics literature. This raises important questions as to whether current ethics guidance is sufficient for the broader range of RCTs identified here. For example, the broad scope and diversity of pragmatic RCTs may point to differences regarding trial responsibilities. In more traditional clinical research trials, there are typically clear lines of accountability and responsibility for a limited number of stakeholders, such as clinician-investigators and hospital administrators. In more pragmatic RCT designs of health policy interventions or in public health, by contrast, these lines of responsibility may be less well established or less clearly defined [12, 66]. There may also be differences in the calculus of benefits and harms within these broader range of trials. For example, in health systems and policy trials, there may be different benefits or harms depending on whether one considers individuals or the population as a whole. This may differ from clinical RCTs of individually administered drugs and where both benefits and harms are considered at the individual level. How benefits and harms are traded and balanced within risk considerations, or in research ethics review processes within the broader set of pragmatic RCTs, requires further discussion.

Finally, our results suggest that despite proposals for the wider application of waivers of consent with low-risk pragmatic RCTs, [67] under-accrual remains a substantial challenge for many pragmatic RCTs. One possibility is that pragmatism within different PRECIS-2 domains may both positively and negatively affect trial accrual. For example, whereas streamlined recruitment (such as the use of waivers of consent or ‘integrated consent’ [68]) may serve to facilitate accrual by removing burdensome consent procedures, more pragmatic approaches limiting research support and infrastructure may negatively affect trial accrual [69]. We suggest that future work explore in more detail the factors associated with under- or over-accrual within pragmatic RCTs, but that this be done in tandem with close examination of the ethical considerations regarding consent.

5. Conclusion

Our results point to the need for guidance regarding the reporting of trial registration, criteria for indexing of pragmatic RCTs, and the reporting of RCTs to indicate their pragmatic intent. There is a need for ongoing review and verification of trial registry data, particularly in relation to sample sizes for pragmatic cluster RCTs. The diversity of RCTs included in our sample of likely pragmatic trials should be reflected in future ethical analyses and guidance. Future work is required to inform ethically acceptable approaches to maximizing accrual.

What is new?

Key findings

• We report on key features of pragmatic randomized controlled trials (RCTs), noting their diversity in scope and the interventions under study as well as limitations in existing registry data, trial indexing, and reporting.

What this adds to what is known?

• Many trials with a pragmatic intent do not use the term pragmatic in the title or abstract, complicating their retrieval in search strategies.

• Data quality of several domains in ClinicalTrials.gov could be improved, specifically data regarding study phase, intervention type, and target and actual enrolment.

What is the implication and what should change now?

• Ethical guidance for the design and conduct of pragmatic RCTs should be critically assessed with respect to the extent to which it is relevant to the diversity of designs, areas of application, and purposes identified in this review.

• Work is needed to better understand the challenges faced by investigators completing trial registry information and, where necessary, new guidance should be developed..