Skip to main content
BMC Medical Research Methodology logoLink to BMC Medical Research Methodology
. 2023 Jun 24;23:148. doi: 10.1186/s12874-023-01975-9

Analysis of the characteristics and the degree of pragmatism exhibited by pragmatic-labelled trials of antineoplastic treatments

Robbe Saesen 1,2,, Kevin Depreytere 1, Karyna Krupianskaya 1, Joël Langeweg 1, Julie Verheecke 1, Denis Lacombe 2,#, Isabelle Huys 1,#
PMCID: PMC10290324  PMID: 37355603

Abstract

Background

Pragmatic clinical trials (PCTs) are designed to reflect how an investigational treatment would be applied in clinical practice. As such, unlike their explanatory counterparts, they measure therapeutic effectiveness and are capable of generating high-quality real-world evidence. However, the conduct of PCTs remains extremely rare. The scarcity of such studies has contributed to the emergence of the efficacy-effectiveness gap and has led to calls for launching more of them, including in the field of oncology. This analysis aimed to identify self-labelled pragmatic trials of antineoplastic interventions and to evaluate whether their use of this label was justified.

Methods

We searched PubMed® and Embase® for publications corresponding with studies that investigated antitumor therapies and that were tagged as pragmatic in their titles, abstracts and/or index terms. Subsequently, we consulted all available source documents for the included trials and extracted relevant information from them. The data collected were then used to appraise the degree of pragmatism displayed by the PCTs with the help of the validated PRECIS-2 tool.

Results

The literature search returned 803 unique records, of which 46 were retained upon conclusion of the screening process. This ultimately resulted in the identification of 42 distinct trials that carried the ‘pragmatic’ label. These studies examined eight different categories of neoplasms and were mostly randomized, open-label, multicentric, single-country trials sponsored by non-commercial parties. On a scale of one (very explanatory) to five (very pragmatic), the median PCT had a PRECIS-2 score per domain of 3.13 (interquartile range: 2.57–3.53). The most and least pragmatic studies in the sample had a score of 4.44 and 1.57, respectively. Only a minority of trials were described in sufficient detail to allow them to be graded across all domains of the PRECIS-2 instrument. Many of the studies examined also had features that arguably precluded them from being pragmatic altogether, such as being monocentric or placebo-controlled in nature.

Conclusion

PCTs of antineoplastic treatments are generally no more pragmatic than they are explanatory.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-023-01975-9.

Keywords: Pragmatic trials, Explanatory trials, Randomized controlled trials, Oncology, Cancer, Neoplasms, Treatments, Efficacy-effectiveness gap, Real-world evidence, PRECIS-2

Background

Randomized controlled trials (RCTs) are considered the bedrock of evidence-based medicine [1]. Their reliance on randomization as a core methodological principle renders them invaluable for assessing the benefits and risks of experimental therapeutic approaches, it has been argued [2]. Both individually and collectively, these studies generate data that shape clinical guidelines in many fields, including oncology. Although the first cancer RCTs were initiated by academic institutions, the growing complexity of the legal and ethical framework which governs the conduct of clinical trials has made it challenging to set up such studies with limited resources, resulting in the gradual expansion of the pharmaceutical industry’s role in developing antitumor medicines [3]. Demonstrating this shift towards the for-profit development of anticancer agents, a recent analysis showed that approximately 90% of RCTs which investigated systemic treatments against common cancers and whose results were published in major scientific journals in the past decade had been either fully or partially funded by companies [4].

The increasingly commercial nature of clinical research in oncology has been accompanied by a rise in the use of surrogate endpoints [4], which for the most part lack validation as predictors of overall survival or quality of life [57]. Moreover, trials undertaken by manufacturers to support the regulatory approval and market entry of antineoplastic drugs often feature comparator arms deviating from the standard of care [810] and recruit patients on the basis of stringent eligibility criteria that reduce the external validity of their findings [1113]. Despite their inherent limitations, studies that incorporate elements like these into their design offer important insights into the efficacy of investigational cancer therapies, i.e. their effects measured under ideal circumstances, when they are applied in highly controlled and artificial environments [1416]. Such trials, which are known as explanatory RCTs [17], address the question of whether an intervention can work, regardless of any external factors that could influence the treatment outcome [14, 18]. They are of particular interest to industry sponsors since they can present a product in the best possible light, and because regulators demand them as part of the requirements that need to be satisfied in order to obtain a marketing authorization [19].

Conversely, so-called pragmatic RCTs [17] investigate whether an intervention will actually work, taking into account additional aspects that could have an impact on the therapeutic response [14, 18]. As such, they provide clarity on the effectiveness of anticancer medicines, i.e. their effects assessed under real-life conditions, when they are employed in the clinic [1416]. These studies are characterized by their patient-centered outcome measures, their clinically relevant comparator arms, their real-world settings, their routine follow-up schedules, and their inclusive recruitment of participants, bringing about heterogeneous samples [18, 20, 21]. Pragmatic RCTs produce valuable insights for patients and clinicians by filling existing evidence gaps and contributing to the realization of a learning health system [22]. Moreover, such trials are of great value to payers and healthcare policymakers as they can directly inform health technology assessment (HTA) and reimbursement-related decision-making [23, 24]. Nevertheless, studies of this nature are infrequently performed, judging by the number of self-labelled pragmatic trials reported in the literature [2527]. According to one estimate [26], they represented fewer than 1% of all RCTs carried out between 1990 and 2010. Pharmaceutical companies are reluctant to conduct them due to their associated business risks, their omission from the list of studies that are standardly imposed on manufacturers, and the paucity of regulatory guidance on how they should be run, among other reasons [19, 20, 28]. Consequently, pragmatic RCTs have so far mainly been undertaken independently from commercial interests [28], using public funding when available [23, 29, 30].

The dominance of explanatory RCTs has contributed to the emergence of what has come to be known as the efficacy-effectiveness gap, i.e. the phenomenon where significant disparities may be seen between a treatment’s efficacy as recorded in pre-approval trials and its effectiveness as observed in clinical practice [31]. In the field of oncology, where the discrepancy between the survival rates of study participants and real-life patients receiving the same therapies can be large, this gap manifests itself particularly clearly [3236]. Regulators are cognizant of its existence, but they have also dismissed calls for requiring marketing authorization applicants to submit data from pragmatic trials as part of their product dossiers, fearing that this would introduce more uncertainty into the evaluation of the application and lead to the rejection of potentially useful agents [37]. Nonetheless, such studies, being considered sources of high-quality real-world evidence [38, 39], are accepted by regulatory authorities if they allow sponsors to deliver on their post-approval commitments [39, 40]. Bridging the efficacy-effectiveness gap may necessitate an overhaul of the current medicines development paradigm and the creation of new models of partnership between the various stakeholders involved in the process of getting a drug from the bench to the bedside [4144].

The distinction between pragmatic and explanatory RCTs is not always apparent: many trials exhibit characteristics of both [18]. In reality, clinical studies exist on a spectrum, ranging from fully pragmatic to fully explanatory, with most trials positioning themselves somewhere in between the two ends [18]. To determine an RCT’s place on this continuum, an instrument was developed that enables researchers to evaluate the degree of pragmatism displayed by the study, namely the Pragmatic Explanatory Continuum Indicator Summary (PRECIS) tool [18]. This tool, which was revised after its original publication and is referred to as PRECIS-2 in its current version [21], makes its users appraise the trial in question across nine distinct domains (Table 1), each of which is rated on a scale from 1 (i.e. very explanatory) to 5 (i.e. very pragmatic). The cumulative score awarded then reflects the extent to which the study integrates facets of pragmatism into its design. PRECIS-2, which unlike its predecessor has been validated [45], is in principle intended to be applied prospectively by investigators, when they are writing a trial protocol, so that they are aware of the impact of the study’s setup on the generalizability of the results they will eventually acquire [18, 21]. However, the instrument can also be employed retrospectively, to grade pragmatic RCTs which have already been completed, in the context of a systematic review [46].

Table 1.

Overview of the nine domains of the PRECIS-2 instrument, their descriptions, and the corresponding questions that users of the tool need to ask themselves when evaluating the degree of pragmatism exhibited by a particular trial [21, 47]

PRECIS-2 domain Description Question
Eligibility The extent to which the trial is open for recruitment to patients with varying demographic characteristics and medical histories To what extent do the participants of the trial resemble the patients that would receive the intervention in clinical practice?
Recruitment The amount of effort that is expended to enroll new participants onto the trial How much effort is made to recruit participants for the study, going beyond what can be considered a standard level of interaction between patients and their doctor?
Setting The environment in which the trial takes place To what extent does the environment in which the study is conducted mirror the usual care setting?
Organization The resources and the expertise that are required to deliver the intervention To what extent are the resources and the expertise needed to deliver the intervention different from those that would be available in the clinic?
Flexibility—delivery The degree of flexibility that is permitted with respect to the delivery of the intervention by investigators How much discretion is granted to the investigators in terms of their administration of the intervention, as compared with the usual care setting in which doctors may or may not choose to prescribe the standard treatment schedule?
Flexibility—adherence The level of flexibility that is allowed with regard to the compliance of trial participants with the investigational treatment schedule How much discretion is granted to the participants in terms of their adherence to the intervention, as compared with the usual care setting in which patients may or may not follow the treatment schedule that their doctor prescribes them?
Follow-up The intensity with which trial participants are monitored and followed up, and the amount of data that are collected during the follow-up period How intensively are the participants of the study followed up, going beyond what can be considered a standard monitoring and data collection schedule?
Primary outcome The nature of the endpoint that the trial was statistically designed to measure To what extent is the primary outcome measure relevant for patients?
Primary analysis The nature of the method by which the primary outcome data are analyzed statistically To what extent are all the primary outcome data that were collected for the study included in the analysis?

It has been noted that many self-described pragmatic trials of pharmacological interventions are placebo-controlled and/or double-blinded studies, sometimes performed in a single institution or hospital, examining substances that have not yet been licensed [25]. Given that these features inherently limit the real-life applicability of a trial’s findings, this observation suggests that the ‘pragmatic’ label is often misused, likely because claims of pragmatism can attract attention from readers [25, 48, 49]. Moreover, RCTs that call themselves pragmatic do not commonly explain why they deserve to be designated as such, and the use of the PRECIS-2 tool for this purpose remains extremely rare [50]. Some authors have advocated the conduct of pragmatic cancer trials to address the efficacy-effectiveness gap in oncology [41, 42, 44, 5154], but it remains unclear what the current landscape of these studies looks like. In this article, we identify clinical trials from the literature that investigated antineoplastic treatments and that were tagged as pragmatic at least once. Subsequently, we assess whether they were justifiably labelled as such by applying the PRECIS-2 instrument.

Methods

Literature screening and study selection

We searched PubMed® and Embase® for publications which outlined the design and/or the results of clinical trials of antineoplastic interventions and whose titles, abstracts or index terms made reference to the pragmatic nature of the studies in question. To this end, a search strategy was developed which contained synonyms and variations of the terms ‘pragmatic clinical trial’ (hereinafter abbreviated as PCT) and ‘neoplasm’, including the corresponding MeSH® and Emtree® descriptors (Additional File 1). The list of PubMed® and Embase® entries that were retrieved on 3 February 2021 using this strategy was imported into EndNote® for removal of duplicates. After all duplicates were eliminated, the shortened list was uploaded to Rayyan®, a website which was designed to facilitate the process of reviewing scientific literature in a systematic manner. The titles and abstracts of the items catalogued in Rayyan® were screened by a team of five researchers (RS, KD, KK, JL, JV), with each individual item being examined by three different persons (half of the items by RS, KD and JV, and the other half by RS, KK and JL). Based on various inclusion and exclusion criteria that were defined a priori (Table 2), the researchers selected the publications that they thought should be scrutinized further, without being able to see each other’s selections. Once they had gone through the items in Rayyan® that were assigned to them, the researchers discussed their choices amongst each other (in two groups of three) and any disagreements were resolved through consensus. This procedure was repeated for the full text screening, ultimately leading to the identification of a number of distinct trials carrying the ‘pragmatic’ label.

Table 2.

Overview of the inclusion and exclusion criteria applied during the literature screening and study selection process

Inclusion criteria Exclusion criteria

• Publications (full articles or conference abstracts) discussing the design and/or the results of clinical trials

• Publications discussing clinical trials that investigate antineoplastic treatments

• Publications discussing clinical trials that are described as pragmatic in the title, abstract or index terms

• Publications discussing other types of studies than clinical trials (e.g. cohort studies)

• Publications discussing clinical trials conducted in other fields than oncology

• Publications written in other languages than English

• Publications that are not accessible

• Publications discussing clinical trials that investigate non-therapeutic interventions (e.g. screening programs)

• Publications discussing clinical trials that investigate complementary therapies (e.g. acupuncture)

• Publications discussing clinical trials that investigate symptomatic therapies (e.g. analgesics)

Data extraction and PRECIS-2 scoring

Next, an in-depth analysis of the identified PCTs was undertaken by consulting publicly accessible sources of information about these studies, such as journal articles, conference posters and abstracts, records from ClinicalTrials.gov and other trial registries, study protocols, and dedicated trial websites, if available. These source materials were again reviewed by five researchers (RS, KD, KK, JL, JV), following the same method that was employed during the literature screening phases. The researchers collected specific data on the studies (Table 3) by filling in a standardized data extraction form. Simultaneously, they applied the PRECIS-2 tool according to the instructions provided by its creators [21, 46, 47] and graded the trials across the nine domains that this instrument encompasses, documenting the motivation behind each score given by quoting relevant excerpts from the source materials. If a domain could not be appraised due to a lack of information found, it was left blank and the trial-level score per domain was determined by adjusting the denominator to reflect the number of assessable domains (e.g. if two domains could not be evaluated and the remaining seven together received a score of 21, the trial-level score per domain would be 21/7 = 3). The researchers were blinded to each other’s data extraction forms and PRECIS-2 scores until they had all completed their review of the source materials. Eventually, they met to compare their forms and scores and to settle any conflicting interpretations by mutual agreement.

Table 3.

Overview of the characteristics that were recorded for each trial included in the analysis, grouped by category

Category Characteristic
General aspects Type(s) of neoplasm(s)
Year in which the trial was started
Year in which the trial was completed
Study duration
Year in which the main results were published*
Journal in which the main results were published*
Impact factor of journal in which the main results were published
Digital object identifier of consulted publication(s)
Trial registry identifier(s)
Availability of protocol
Design elements Clinical phase
Use of randomization
Use of blinding
Number of study arms
Intervention(s)

Type of intervention(s)

(e.g. pharmacotherapy, surgery, radiotherapy, etc.)

Comparator(s)

Type of comparator(s)

(active or placebo; standard of care or not)

Primary endpoint(s)/outcome measure(s)
Secondary endpoint(s)/outcome measure(s)
Organizational facets Number of participants

Site-level setting

(monocentric or multicentric)

Country-level setting

(national or international)

Countries in which the trial took place
Legal sponsorship

Funding source

(commercial or non-commercial/academic)±

Information on when the studies were initiated and finalized was obtained from the trial registry records. If such data were missing from these files, they were extracted from other sources (e.g. journal articles)

(∟) Calculated by subtracting the year in which the trial was started from the year in which it was completed

*If the main results of the study had not (yet) been published, these characteristics were instead documented for publications discussing the interim results of the trial or, absent any such articles or abstracts, its setup

°Impact factors were taken from Web of Science® journal metrics. If the 2021 impact factor was unavailable, the 2020 one was used as a substitute

Note that the clinical phase concept is not applicable to studies primarily investigating any non-pharmacological interventions other than gene therapies

A trial was considered single-blinded if either the patients participating in the study or the healthcare professionals administering the intervention were unaware of the treatment allocation, and double-blinded if both parties were. Masking on the part of outcome assessors and data analysts was left out of consideration

The assessment of whether or not a specific comparator treatment constituted standard of care was based on how it was described in the source materials that were scrutinized for each trial

This characteristic reflects the number of participants that were randomized or allocated to receive a study treatment. For ongoing trials that are still recruiting patients, it represents the planned sample size

This characteristic denotes whether the study was conducted in a single center or across multiple centers

This characteristic indicates whether the trial was rolled out in a single country or across multiple countries

Information on the legal sponsorship of the studies was collected from the trial registry records

±A trial was deemed to have a commercial funding source if it received material or financial support from companies

Data analysis and reporting

Once the separate data extraction forms had been merged into one and the PRECIS-2 scores had been agreed upon by the researchers, the data gathered were analyzed descriptively in Excel® and inferentially using IBM® SPSS® Statistics 28.0. For nominal variables (e.g. type of neoplasm, primary endpoint, comparator treatment, etc.), proportions were calculated. Ordinal (e.g. sample-level PRECIS-2 scores per domain), discrete (e.g. number of participants, study duration, number of assessable PRECIS-2 domains, etc.) and continuous (e.g. journal impact factor, trial-level PRECIS-2 scores per domain) variables were characterized by their median values, with the associated interquartile ranges (IQRs) being represented by their lower and upper boundaries. Mann–Whitney U tests were performed to verify whether trials with particular features displayed a PRECIS-2 score per domain that was significantly different from that of the studies that did not have those features. To probe the strength of the correlation between the sample size of the trials and their PRECIS-2 score per domain, Kendall’s tau-b coefficient was computed. The significance threshold was set at 0.05.

Results

Literature screening and study selection

The search strategy returned 1,075 results in total, of which 473 were obtained from PubMed® and 602 from Embase® (Fig. 1). After 272 duplicates were filtered out in EndNote®, the titles, abstracts and index terms of the remaining 803 publications were screened in Rayyan® for adherence to the prespecified eligibility criteria. 64 items were withheld for the full text screening, which eventually led to the identification of 42 distinct trials that had been tagged as pragmatic at least once, either by the authors themselves or by the PubMed® and Embase® reviewers that manually assign relevant MeSH® and Emtree® descriptors to new entries in these two repositories.

Fig. 1.

Fig. 1

Flowchart of the literature screening and study selection process

Characteristics of included trials

The characteristics of the included trials are summarized in Table 4. The completed data extraction form outlining the information collected for each individual trial is provided in Additional File 2.

Table 4.

Overview of the characteristics of the 42 trials included in the analysis

Characteristic Total: n = 42
n %
Type of neoplasm Gastrointestinal 9 21.4
Breast 9 21.4
Hematological 6 14.3
Gynecological 4 9.5
Lung 3 7.1
Genitourinary 3 7.1
Skin 3 7.1
Head and neck 2 4.8
Non-specific 3 7.1
Decade of initiation 1980s 2 4.8
1990s 5 11.9
2000s 12 28.6
2010s 23 54.8
Study status Completed 35 83.3
Ongoing 7 16.7
Protocol availability Accessible 23 54.8
Inaccessible 19 45.2
Clinical phase I 2 4.8
I/II 1 2.4
II 9 21.4
II/III 1 2.4
III 7 16.7
IV 1 2.4
N/A 21 50.0
Randomization Randomized 31 73.8
Non-randomized 5 11.9
Single-arm 6 14.3
Blinding Blinded 2 4.8
Open-label 40 95.2
Number of study arms 1 6 14.3
2 30 71.4
3 or more 6 14.3
Type of intervention Pharmacotherapy 24 57.1
Surgery 13 31.0
Radiotherapy 16 38.1
Other 7 16.7
Combination 15 35.7
Type of comparator Active 34 81.0
Standard of care or equivalent 28 66.7
Placebo 2 4.8
None 7 16.7
Primary endpoint(s) Overall survival 6 14.3
Quality of life 1 2.4
Surrogate 26 61.9
Secondary endpoint(s) Overall survival 26 61.9
Quality of life 23 54.8
Surrogate 36 85.7
Site-level setting Monocentric 11 26.2
Multicentric 31 73.8
Country-level setting National 29 69.0
International 13 31.0
Legal sponsorship By industry 3 7.1
By other party 39 92.9
Funding source Commercial 15 35.7
Non-commercial or academic 27 64.3
Median IQR
Study duration (in years) 5.5 3–8
Impact factor journal of publication 9.162 4.853–41.316
Number of participants Phase II trials 68 51–121
Phase III trials 501 231.5–748
Overall 161 87.5–538.5

General aspects

The 42 identified PCTs encompassed a wide variety of different neoplasms, which could be categorized into eight distinct groups (Fig. 2A): gastrointestinal neoplasms (9/42, 21.4%), breast neoplasms (9/42, 21.4%), hematological neoplasms (6/42, 14.3%), gynecological neoplasms (4/42, 9.5%), genitourinary neoplasms (3/42, 7.1%), lung neoplasms (3/42, 7.1%), skin neoplasms (3/42, 7.1%), and neoplasms of the head and neck (2/42, 4.8%). Of the three studies that did not focus on a specific tumor type, two dealt with spinal metastases of various primary origins and one was a basket trial. The bulk (23/42, 54.8%) of the PCTs in our sample had been launched in the past decade (Fig. 2B). The oldest study was started in 1982, whereas the most recent one began recruiting participants in 2018. Seven trials were ongoing at the time of writing, and completed PCTs had run for a median of 5.5 years (IQR: 3–8). When results were available for a study (39/42, 92.9%), they had generally been published in high-impact journals (median impact factor of 9.162, with IQR of 4.853–41.316). For most of the PCTs (23/42, 54.8%), the study protocol was publicly accessible, usually as an appendix to a publication detailing the trial’s findings or as a separate article.

Fig. 2.

Fig. 2

Breakdown of the included studies by (A) the types of neoplasms they focused on (with the area of each rectangle/tumor type being proportional to its share of the sample), B their decades of initiation, (C) the types of interventions they investigated, and (D) the extent to which the industry was involved in their conduct. The percentages shown may not add up to 100% due to rounding (A), or because the categories displayed are not mutually exclusive (C)

Design elements

Half (21/42, 50.0%) of the trials included in the analysis covered particular phases of the clinical development paradigm, with phase II (9/21, 42.9%) and phase III (7/21, 33.3%) studies being more prevalent than those of any other phases (I, I/II, II/III, or IV; 5/21, 23.8%). While most (31/42, 73.8%) of the examined PCTs employed randomization and could therefore be considered RCTs, non-randomized and single-arm trials together accounted for more than a quarter (11/42, 26.2%) of the studies scrutinized. The use of masking was exceptional: only two trials (4.8%) had implemented blinding schemes into their design, and in both cases neither the participants nor the investigators were at any point in the data collection process aware of which intervention(s) they had been allocated to receive or administer. All phase III PCTs (7/7, 100%) were randomized, open-label studies.

The sample mainly (30/42, 71.4%) consisted of trials that evaluated two treatment strategies against each other. For a majority (24/42, 57.1%) of the analyzed PCTs, at least one of the study interventions could be classified as pharmacological in nature (e.g. chemotherapy, targeted therapy, hormonal therapy; Fig. 2C). Many trials also investigated therapeutic approaches that comprised surgical (13/42, 31.0%) or radiotherapeutic (16/42, 38.1%) procedures. Over a third (15/42, 35.7%) of the studies explored combinations of different treatment modalities (e.g. intraoperative radiotherapy). If comparator arms were present in the PCTs (35/42, 83.3%), they typically reflected what constituted the standard of care or the most commonly used choice of therapy at the time the trials were initiated (28/35, 80.0%). Placebo controls were exclusively featured in the two blinded studies.

With regard to the outcome measures recorded within the trials, a majority (26/42, 61.9%) of the PCTs had been statistically designed to generate data on one or more surrogate endpoints (e.g. progression-free survival, disease-free survival, response rate, etc.), including all (9/9, 100%) of the phase II and a significant proportion (5/7, 71.4%) of the phase III studies. Overall survival was infrequently (6/42, 14.3%) adopted as a primary endpoint, but often (26/42, 61.9%) served as a secondary one instead. In most (23/42, 54.8%) trials, an assessment of patients’ quality of life was carried out, albeit rarely (1/23, 4.3%) with the intent of delivering on the study’s main objective. Some PCTs (5/42, 11.9%) had been set up to document other types of clinically meaningful outcomes, such as changes in the mobility status or pain response of their participants.

Organizational facets

The studies’ sample sizes, whether actual (for completed trials) or planned (for ongoing trials), varied significantly, ranging from 4 to 3,581. The median phase II PCT had 68 participants (IQR: 51–121), whereas the median phase III one had 501 (IQR: 231.5–748). Collectively, the trials that were reviewed in this analysis had a median of 161 patients (IQR: 87.5–538.5) taking part in them, ignoring two prematurely terminated PCTs from which no meaningful conclusions could be drawn, and assuming studies that were still running reached their accrual targets. Approximately three-quarters (31/42, 73.8%) of the trials had been conducted across multiple hospitals or institutions and could thus be described as multicentric. However, only a minority (13/42, 31.0%) of the PCTs had been rolled out in more than a single country. The three countries which saw the most studies being undertaken within their borders were the United Kingdom (16/42, 38.1%), the United States (8/42, 19.0%) and the Netherlands (7/42, 16.7%).

The legal sponsorship of the trials was almost always (39/42, 92.9%) assumed by an entity that did not stand to gain monetarily from their outcomes, such as a university hospital, a research consortium, a learned society, or a not-for-profit organization (Fig. 2D). University College London alone was the sponsor of six (14.3%) of the studies in our sample. The few (3/42, 7.1%) PCTs that were sponsored by companies examined interventions of a commercial nature, namely a type of gene therapy, a topical cream, and a chemotherapeutic agent. Nevertheless, many (12/39, 30.8%) of the trials that were not performed at the industry’s behest still received some degree of financial or material support from the manufacturers of the products they investigated (e.g. educational grants, free drug supplies, etc.).

PRECIS-2 scores

The completed data extraction form listing the PRECIS-2 scores given to each individual trial is provided in Additional File 2.

For the median PCT, 8 domains of the PRECIS-2 tool could be assessed (IQR: 7–9), resulting in a trial-level score per domain of 3.13 after rounding (IQR: 2.57–3.53). The scores awarded did not depend on whether or not the studies employed randomization (P = 0.059), took place in multiple countries (P = 0.276), or received support from the industry (P = 0.066). Moreover, PCTs for which a protocol was available were not graded significantly higher than those for which no such document could be retrieved (P = 0.294). Multicentric trials on the other hand garnered greater scores than their monocentric counterparts (P = 0.010).

A positive correlation of moderate strength was observed between the studies’ sample sizes (planned or actual) and their PRECIS-2 scores (τb = 0.234, P = 0.031). PCTs that were labelled as pragmatic in the index terms of their corresponding publications exclusively (n = 9) were rated as more explanatory overall (P = 0.023). The most pragmatic study included in the analysis was the UKHAN1 trial [55] (PRECIS-2 score per domain of 4.44, Fig. 3A), while the most explanatory one was the RASHEC RCT [56] (PRECIS-2 score per domain of 1.57, Fig. 3B).

Fig. 3.

Fig. 3

PRECIS-2 wheels of (A) the highest-scoring, (B) the lowest-scoring, and (C) the median trial. The concentric circles represent the different possible scores that can be awarded to each domain of the PRECIS-2 tool, with the innermost one being equivalent to 1 and the outermost one corresponding with 5. The area of the polygons enclosed by the green lines connecting the ‘score dots’ is proportional to the degree of pragmatism exhibited by the studies in question. Note that for (B), the ‘recruitment’ and ‘flexibility (adherence)’ domains could not be graded, and the matching dots are therefore missing

At the sample level, the PRECIS-2 domains that accumulated the highest scores were ‘primary analysis’ (median of 4.5, with IQR of 4–5), ‘recruitment’ (median of 4, with IQR of 3–5) and ‘setting’ (median of 4, with IQR of 3–5), whereas the ones that accrued the lowest were ‘organization’ (median of 2, with IQR of 1–3) and ‘flexibility (adherence)’ (median of 2, with IQR of 2–3). A detailed breakdown of the median PRECIS-2 scores by domain is presented in Table 5 and displayed in Fig. 3C. In most cases (24/42, 57.1%), one or more domains of the PRECIS-2 instrument could not be appraised for a given PCT due to a lack of information provided in the source documents that were consulted for each study. Even when a protocol was accessible (23/42, 54.8%), the appraisal of certain domains was oftentimes (7/23, 30.4%) impossible for this reason. ‘Flexibility (adherence)’ (20/42, 47.6%) and ‘recruitment’ (21/42, 50.0%) were the least frequently assessable domains altogether, with the former sometimes being inapplicable (8/42, 19.0%). Just four domains were scorable across all trials, namely ‘eligibility’, ‘flexibility (delivery)’, ‘primary outcome’ and ‘setting’.

Table 5.

Overview of the sample-level PRECIS-2 scores per domain. The last column of the table displays the number of times the individual domains could not be graded, either because they were not applicable to the studies in question (e.g. flexibility in terms of adherence cannot be assessed for trials of surgical techniques, as these are single-application interventions), or due to a lack of relevant information provided in the source documents that were consulted for each PCT

PRECIS-2 domain PRECIS-2 score Number of times no score could be given
Median IQR
Eligibility 3 2–4 0
Flexibility (adherence) 2 2–3 22a
Flexibility (delivery) 3.5 2–4 0
Follow-up 3 2–4 4
Organization 2 1–3 4
Primary analysis 4.5 4–5 4b
Primary outcome 3 2–4 0
Recruitment 4 3–5 21
Setting 4 3–5 0

aMore specifically, 14 times on account of insufficiently detailed reporting, and 8 times owing to the inapplicability of the domain to the trials at issue

bOn 3 occasions, this domain was not scorable as a result of it not being pertinent to the studies under evaluation

Discussion

To our knowledge, this is the first study that has analyzed the characteristics and the degree of pragmatism exhibited by pragmatic-labelled trials of antineoplastic interventions. We found that such trials were heterogeneous in terms of the methodological and organizational features they displayed, and that they were overall only slightly more pragmatic than explanatory in their setup, as evidenced by their moderate PRECIS-2 scores. Additionally, we observed that in most cases, the PCTs could not be graded across all domains of the PRECIS-2 tool because their design and procedures had been inadequately reported in their corresponding publications.

Our analysis shows that the number of pragmatic-labelled studies initiated in the field of oncology has roughly doubled every decade since the 1980s. This rising trend suggests that the use of PCTs to investigate the effectiveness of antitumor therapies has become more popular over time. However, it is debatable whether this actually means that sponsors of trials examining antineoplastic interventions are increasingly embracing the concept of pragmatism, as it could also signify that the ‘pragmatic’ descriptor is being misapplied more frequently. In fact, the PRECIS-2 scores that were obtained seem to support the latter interpretation, and prior literature points in that direction as well [25, 50]. Nevertheless, the expanding role of real-world evidence in the development of anticancer treatments [57, 58] will likely spur the future conduct of PCTs, since these studies are considered to be some of the most robust sources of such evidence [38, 39].

Some authors [5972] have employed the PRECIS-2 instrument as a critical appraisal tool to investigate the extent to which the trials that they included in their systematic or scoping literature reviews could be classified as PCTs. Others [7386] have used it in a similar manner to us, with the aim of verifying whether studies that were tagged as pragmatic or that were running as part of a publicly funded PCT program deserved to carry the ‘pragmatic’ label. However, few publications [69, 71, 72] in which PRECIS-2 assessments were undertaken looked at trials of antineoplastic interventions, and only one reported the outcomes of an analysis that was comparable to ours in terms of its sample size and methodological depth. Sorigue and Kuittinen [72] evaluated the position of studies underpinning the therapeutic recommendations listed in the 2020 European Society for Medical Oncology clinical guidelines for follicular lymphoma on the pragmatic-explanatory continuum. In total, they incorporated 28 distinct trials into their review, more than half of which could only be scrutinized based on articles outlining their results due to the inaccessibility of their protocols. The median trial received a PRECIS-2 score per domain of 3.5, which is slightly higher than in our analysis. This is remarkable, given that the studies we examined had been explicitly labelled as pragmatic, whereas the ones appraised by Sorigue and Kuittinen [72] had not. Moreover, the sample-level PRECIS-2 scores per domain differed significantly across the two analyses: while ‘organization’ was for instance the joint-most explanatory domain for our trials, it was among the most pragmatic for those reviewed by Sorigue and Kuittinen [72]. These observations further illustrate how the ‘pragmatic’ label is often used in a misleading manner, offering no assurance that the study in question is more pragmatic than any given trial not carrying this tag. Nevertheless, it should be stressed that a direct comparison between our findings and those of Sorigue and Kuittinen [72] may not be fully appropriate: unlike us, Sorigue and Kuittinen [72] did not consider the ‘flexibility (adherence)’ and ‘recruitment’ domains of the PRECIS-2 instrument at all in their analysis, since they could not find any meaningful information on the basis of which these domains could be graded. Consequently, differences in scoring may be the result of divergence in the way the PRECIS-2 instrument was applied.

Concerning the methodological design of the PCTs in our analysis, two studies employed the so-called trials-within-cohorts (TwiCs) approach [87], namely VERTICAL [88] and RECTAL-BOOST [89]. These studies are therefore examples of cohort multiple randomized controlled trials (cmRCTs) [87, 90, 91]. In such trials, patients are followed prospectively as part of a large observational cohort and a randomly selected subset among them is given the choice to undergo the investigational intervention [87, 90, 91]. Participants who are randomized to remain within the cohort receive usual care and make up the control group of the study. Since these patients are not informed of the fact that a trial is being conducted and that treatment allocation has taken place, the recruitment process can be significantly more efficient and less contrived for cmRCTs than for regular RCTs [9092]. Moreover, because their cohorts are intrinsically embedded into clinical practice, cmRCTs are standardly performed in a real-world setting [87, 90, 91]. As a result, cmRCTs tend to be very pragmatic with respect to the ‘recruitment’ and ‘setting’ domains of the PRECIS-2 tool. However, as TwiCs schemes are relatively new, investigators may not yet be very familiar with them. Consequently, additional training could be required, lowering the score that can be given for the ‘organization’ domain. Besides cmRCTs, our sample also contained a basket trial [93], namely TAPUR [94]. Here as well, the innovative nature and relative complexity of biomarker-driven studies [93] limit the degree to which they can be pragmatic in terms of their organization. Furthermore, the stringent criteria that such trials apply to determine which drugs should be administered to which patients [93] inherently restrict the level of flexibility that is allowed with regard to the delivery of the targeted agents in question. Nevertheless, basket trials can still be justifiably labelled as pragmatic, as long as they implement enough elements of pragmatism into their design.

Notably, over a quarter of the PCTs that were examined in this analysis could not be regarded as RCTs, either because they were single-arm trials or because their participants were not randomly assigned to their study arms. In principle, a lack of randomization does not necessarily exclude a trial from being pragmatic: in their seminal paper introducing the concept of pragmatism [17], Schwartz and Lellouch never specifically refer to randomized trials, only to trials in general. However, the authors of the PRECIS-2 tool explicitly state that their instrument is intended to be applied to RCTs [21], even though none of its domains directly pertain to the method used for treatment allocation. In fact, it can be argued that randomization is a highly artificial procedure, since in a real-life environment, patients and clinicians can actively choose which interventions they will undergo or administer. Nevertheless, RCT designs are methodologically indispensable for producing robust and actionable evidence which is free from confounding-related biases [2, 22]. Hence, it may be reasonable to consider randomization separately from the PRECIS-2 assessment, for example as part of a critical appraisal exercise based on the Cochrane risk-of-bias tool RoB 2 [95]. By randomizing at the site level rather than the individual participant level, RCTs can preserve their statistical strengths and at the same time emulate real-world practice more closely. This technique, which is referred to as cluster randomization [96], is commonly employed in PCTs [97] but was not featured in any of the studies in our sample.

Despite reviewing all of the available source documents for the included studies, we were unable to appraise most of the PCTs in our sample across every domain of the PRECIS-2 instrument. Other authors [63, 65, 68, 70, 72, 76, 7981] have likewise struggled to conduct exhaustive PRECIS-2 evaluations within the context of a systematic or scoping literature review. The fields corresponding with the ‘flexibility (adherence)’ and ‘recruitment’ domains on the tool’s accompanying appraisal sheet seem to be especially difficult to complete, even when considering that the former domain is not universally applicable. The inability to fully grade a given trial due to missing information is a known limitation of retrospective PRECIS-2 assessments [98]. Examining the protocol in addition to publications describing the results of the study does not always solve the problem: in our analysis, almost one-third of the trials for which this document could be accessed had at least one outstanding PRECIS-2 domain. Echoing the proposals made by Dal-Ré et al. [25], we believe that PCT sponsors and investigators should (a) systematically and prospectively evaluate the degree of pragmatism exhibited by their studies using the PRECIS-2 tool, and (b) publish the scores given for each domain along with the underlying motivations so that journal editors, reviewers and readers can independently validate them.

None of the studies in our sample that were started after 2009 (i.e. the year in which the first version of the PRECIS instrument was released [18]) substantiated their claim of being pragmatic by reporting their PRECIS or PRECIS-2 scores. Whenever justification for the use of the ‘pragmatic’ label was provided, it mostly addressed just one or two domains of these tools, usually the ones relating to the nature of the eligibility criteria that were applied to select participants for the trial, the setting in which the study took place, or the level of flexibility that was permitted regarding the administration of the investigational intervention. Janiaud et al. [50] have made similar observations: in their review of 73 pragmatic-labelled RCTs whose results were published in 2016, they found that 45% offered no explanation for why they carried that label, and that only one trial publication included a PRECIS assessment. The studies that tried to justify their ‘pragmatic’ tag mainly did so by citing the same type of arguments that we came across in our analysis. There seems to be a general lack of awareness among PCT sponsors and investigators of the PRECIS(-2) instrument’s existence and of the fact that pragmatism is a multifaceted concept, encompassing all aspects of a trial’s design and organization. Additional efforts may be needed to educate these stakeholders on the hallmarks of PCTs. In light of the important role such studies can play in the decision-making of HTA agencies, payers, clinicians and patients, it is imperative that the ‘pragmatic’ descriptor is employed correctly in the literature.

Some authors [25, 98] have argued that certain design elements can preclude a trial from being pragmatic altogether. More specifically, it has been asserted that placebo-controlled, blinded, single-center and/or pre-approval studies can never be considered PCTs, since such characteristics would inherently render them explanatory. In our analysis, we encountered many trials that displayed at least one of these features. However, instead of immediately excluding the possibility that they could be pragmatic, we still subjected them to a full PRECIS-2 evaluation. That way, if a study scored low for one or two domains, it could nonetheless accrue a relatively high total score by exhibiting a greater degree of pragmatism across the other domains of the PRECIS-2 instrument. This approach is consistent with the recommendations formulated by the tool’s developers [46]. Nevertheless, if the core aim of a PCT is indeed to emulate clinical practice, we would agree that having a monocentric setup and making use of masking would prevent that aim from being achievable. Regardless of how the concept of pragmatism is interpreted, it is remarkable that a large number of pragmatic-labelled trials of antitumor treatments show such overt signs of actually being considerably explanatory.

Strengths of the present study include its wide scope, encompassing all types of neoplasms and interventions, its comprehensive review of virtually every document available on the included trials, and its methodological rigor stemming from its blinded and comparative approaches with respect to data extraction and PRECIS-2 scoring. However, this analysis also suffers from a number of limitations. Firstly, our literature search only allowed us to find studies that had been tagged as pragmatic in the titles, abstracts or index terms of their corresponding publications. This implies that we were unable to identify PCTs for which no results or design characteristics had been reported yet or which had not explicitly been labelled as pragmatic. Nevertheless, the former would likely not have been assessable using the PRECIS-2 instrument given the scarcity of accessible information on such trials, and the latter were not considered for inclusion as they would not have contributed to answering the main research question. To compose a detailed overview of the PCT landscape in oncology, a different search strategy would need to be used, such as the one described by Taljaard et al. [99]. Secondly, several domains of the PRECIS-2 tool were challenging to appraise without in-depth knowledge on the way in which usual care is organized for patients with specific tumors. Since none of the researchers involved in the PRECIS-2 assessment had any experience working as a clinician, some facets of clinical practice may have been unknown to them. To familiarize themselves with such aspects, they consulted additional source materials (e.g. journal articles, chapters from medical textbooks, websites targeted towards patients) where necessary. Lastly, our sample size was relatively small, which could have negatively affected the power of the statistical tests that were performed. As a result, we may have mistakenly failed to reject their null hypotheses on multiple occasions.

Conclusions

Pragmatic-labelled trials of antineoplastic interventions with published results and/or designs varied in their characteristics, but were mainly randomized, open-label, multicentric, single-country studies that were sponsored by non-commercial parties and that compared pharmacological therapies against standard-of-care treatments in terms of their potential to improve outcomes serving as surrogate markers for the survival and/or quality of life of patients with specific tumors. Despite having been tagged as pragmatic, the trials generally did not exhibit a high degree of pragmatism, judging by the relatively low PRECIS-2 scores they accrued. Some of the studies also displayed features that were arguably irreconcilable with their claims of being pragmatic, like having a monocentric setup or a placebo comparator arm. However, most of the trials were not described in enough detail in their corresponding publications to enable them to be appraised across all domains of the PRECIS-2 tool. Sponsors of future PCTs undertaken in the field of oncology and beyond should systematically justify their use of the ‘pragmatic’ label by prospectively evaluating the position of their studies on the pragmatic-explanatory continuum with the help of the PRECIS-2 instrument or its forthcoming successor, and such an evaluation should be independently reviewed and validated.

Supplementary Information

Additional file 1. (88.7KB, pdf)
Additional file 2. (378.1KB, xlsx)

Acknowledgements

Not applicable.

Abbreviations

cmRCT

Cohort multiple randomized controlled trial

HTA

Health technology assessment

IQR

Interquartile range

PCT

Pragmatic clinical trial

PRECIS

PRagmatic Explanatory Continuum Indicator Summary

RCT

Randomized controlled trial

TwiCs

Trials-within-cohorts

Authors’ contributions

RS, DL and IH were responsible for the conception and the design of the study. RS, KD, KK, JL and JV performed the literature review, extracted information from the source documents identified and carried out the PRECIS-2 assessment. RS analyzed the data and prepared the manuscript. All authors reviewed the manuscript and provided comments. RS revised the manuscript based on those comments. All authors approved the submitted version of the manuscript.

Funding

Robbe Saesen's work as a Fellow at EORTC Headquarters was supported by a grant from the EORTC Cancer Research Fund (ECRF). This publication was supported by a donation from Kom op tegen Kanker (Stand up to Cancer), the Flemish cancer society from Belgium. The funding sources had no role in the design of the study, the analysis and interpretation of the data, the writing of the manuscript or the decision to submit it for publication.

Availability of data and materials

All data generated or analyzed during this study are included in this published article and its supplementary information files.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Footnotes

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Denis Lacombe and Isabelle Huys are shared last authors.

References

  • 1.Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg. 2011;128:305–310. doi: 10.1097/prs.0b013e318219c171. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Collins R, Bowman L, Landray M, Peto R. The magic of randomization versus the myth of real-world evidence. N Engl J Med. 2020;382:674–678. doi: 10.1056/nejmsb1901642. [DOI] [PubMed] [Google Scholar]
  • 3.Broes S, Saesen R, Lacombe D, Huys I. Past, current, and future cancer clinical research collaborations: the case of the European Organisation for Research and Treatment of Cancer. Clin Transl Sci. 2021;14:47–53. doi: 10.1111/cts.12863. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Del Paggio JC, Berry JS, Hopman WM, Eisenhauer EA, Prasad V, Gyawali B, et al. Evolution of the randomized clinical trial in the era of precision oncology. JAMA Oncol. 2021;7:728–734. doi: 10.1001/jamaoncol.2021.0379. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Kim C, Prasad V. Strength of Validation for Surrogate End Points Used in the US Food and Drug Administration’s Approval of Oncology Drugs. Mayo Clin Proc. 2016;91:713–725. doi: 10.1016/j.mayocp.2016.02.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Prasad V, Kim C, Burotto M, Vandross A. The strength of association between surrogate end points and survival in oncology. JAMA Intern Med. 2015;175:1389–1398. doi: 10.1001/jamainternmed.2015.2829. [DOI] [PubMed] [Google Scholar]
  • 7.Gyawali B, Hey SP, Kesselheim AS. Evaluating the evidence behind the surrogate measures included in the FDA’s table of surrogate endpoints as supporting approval of cancer drugs. EClinicalMedicine. 2020;21:100332. doi: 10.1016/j.eclinm.2020.100332. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Hilal T, Sonbol MB, Prasad V. Analysis of control arm quality in randomized clinical trials leading to anticancer drug approval by the US Food and Drug Administration. JAMA Oncol. 2019;5:887–892. doi: 10.1001/jamaoncol.2019.0167. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Hilal T, Gonzalez-Velez M, Prasad V. Limitations in clinical trials leading to anticancer drug approvals by the US food and drug administration. JAMA Intern Med. 2020;180:1108–1115. doi: 10.1001/jamainternmed.2020.2250. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Naci H, Davis C, Savović J, Higgins JPT, Sterne JAC, Gyawali B, et al. Design characteristics, risk of bias, and reporting of randomised controlled trials supporting approvals of cancer drugs by European Medicines Agency, 2014–16: cross sectional analysis. BMJ. 2019;366:l5221. doi: 10.1136/bmj.l5221. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J. A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. Trials. 2015;16:495. doi: 10.1186/s13063-015-1023-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Jin S, Pazdur R, Sridhara R. Re-evaluating eligibility criteria for oncology clinical trials: analysis of investigational new drug applications in 2015. J Clin Oncol. 2017;35:3745–3752. doi: 10.1200/jco.2017.73.4186. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Unger JM, Hershman DL, Osarogiagbon RU, Gothwal A, Anand S, Dasari A, et al. Representativeness of Black Patients in Cancer Clinical Trials Sponsored by the National Cancer Institute Compared With Pharmaceutical Companies. JNCI Cancer Spectr. 2020;4:pkaa034. doi: 10.1093/jncics/pkaa034. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci. 2011;13:217–24. doi: 10.31887/DCNS.2011.13.2/npatsopoulos. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Sedgwick P. Explanatory trials versus pragmatic trials. BMJ. 2014;349:g6694. doi: 10.1136/bmj.g6694. [DOI] [PubMed] [Google Scholar]
  • 16.Roland M, Torgerson DJ. Understanding controlled trials: What are pragmatic trials? BMJ. 1998;316:285. doi: 10.1136/bmj.316.7127.285. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Schwartz D, Lellouch J. Explanatory and pragmatic attitudes in therapeutical trials. J Chronic Dis. 1967;20:637–648. doi: 10.1016/0021-9681(67)90041-0. [DOI] [PubMed] [Google Scholar]
  • 18.Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, et al. A pragmatic–explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62:464–475. doi: 10.1016/j.jclinepi.2008.12.011. [DOI] [PubMed] [Google Scholar]
  • 19.Zwarenstein M, Treweek S. What kind of randomized trials do we need? J Clin Epidemiol. 2009;62:461–463. doi: 10.1016/j.jclinepi.2009.01.011. [DOI] [PubMed] [Google Scholar]
  • 20.Tunis SR, Stryer DB, Clancy CM. Practical Clinical Trials. JAMA. 2003;290:1624–1632. doi: 10.1001/jama.290.12.1624. [DOI] [PubMed] [Google Scholar]
  • 21.Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147. doi: 10.1136/bmj.h2147. [DOI] [PubMed] [Google Scholar]
  • 22.Simon GE, Platt R, Hernandez AF. Evidence from Pragmatic Trials during Routine Care — Slouching toward a Learning Health System. N Engl J Med. 2020;382:1488–1491. doi: 10.1056/nejmp1915448. [DOI] [PubMed] [Google Scholar]
  • 23.Neyt M, Christiaens T, Demotes J, Walley T, Hulstaert F. Publicly funded practice-oriented clinical trials: of importance for healthcare payers. J Comp Eff Res. 2016;5:551–560. doi: 10.2217/cer-2016-0018. [DOI] [PubMed] [Google Scholar]
  • 24.Ratner J, Mullins D, Buesching DP, Cantrell RA. Pragmatic Clinical Trials: US Payers’ Views on Their Value. Am J Manag Care. 2013;19:e158–65. [PubMed] [Google Scholar]
  • 25.Dal-Ré R, Janiaud P, Ioannidis JPA. Real-world evidence: How pragmatic are randomized controlled trials labeled as pragmatic? BMC Med. 2018;16:49. doi: 10.1186/s12916-018-1038-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Chalkidou K, Tunis S, Whicher D, Fowler R, Zwarenstein M. The role for pragmatic randomized controlled trials (pRCTs) in comparative effectiveness research. Clin Trials. 2012;9:436–446. doi: 10.1177/1740774512450097. [DOI] [PubMed] [Google Scholar]
  • 27.Vallvé C. A critical review of the pragmatic clinical trial. Med Clin (Barc) 2003;121:384–388. doi: 10.1016/s0025-7753(03)73957-8. [DOI] [PubMed] [Google Scholar]
  • 28.Buesching DP, Luce BR, Berger ML. The role of private industry in pragmatic comparative effectiveness trials. J Comp Eff Res. 2012;1:147–156. doi: 10.2217/cer.12.9. [DOI] [PubMed] [Google Scholar]
  • 29.Weinfurt KP, Hernandez AF, Coronado GD, DeBar LL, Dember LM, Green BB, et al. Pragmatic clinical trials embedded in healthcare systems: generalizable lessons from the NIH Collaboratory. BMC Med Res Methodol. 2017;17:144. doi: 10.1186/s12874-017-0420-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Johnson KE, Tachibana C, Coronado GD, Dember LM, Glasgow RE, Huang SS, et al. A guide to research partnerships for pragmatic clinical trials. BMJ. 2014;349:g6826. doi: 10.1136/bmj.g6826. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Nordon C, Karcher H, Groenwold RHH, Ankarfeldt MZ, Pichler F, Chevrou-Severac H, et al. The “Efficacy-Effectiveness Gap”: Historical Background and Current Conceptualization. Value Health. 2016;19:75–81. doi: 10.1016/j.jval.2015.09.2938. [DOI] [PubMed] [Google Scholar]
  • 32.Phillips CM, Parmar A, Guo H, Schwartz D, Isaranuwatchai W, Beca J, et al. Assessing the efficacy-effectiveness gap for cancer therapies: A comparison of overall survival and toxicity between clinical trial and population-based, real-world data for contemporary parenteral cancer therapeutics. Cancer. 2020;126:1717–1726. doi: 10.1002/cncr.32697. [DOI] [PubMed] [Google Scholar]
  • 33.Cramer-Van Der Welle CM, Peters BJM, Schramel FMNH, Klungel OH, Groen HJM, Van De Garde EMW. Systematic evaluation of the efficacy–effectiveness gap of systemic treatments in metastatic nonsmall cell lung cancer. Eur Respir J. 2018;52:1801100. doi: 10.1183/13993003.01100-2018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Boyle JM, Hegarty G, Frampton C, Harvey-Jones E, Dodkins J, Beyer K, et al. Real-world outcomes associated with new cancer medicines approved by the Food and Drug Administration and European Medicines Agency: A retrospective cohort study. Eur J Cancer. 2021;155:136–144. doi: 10.1016/j.ejca.2021.07.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Green AK, Curry M, Trivedi N, Bach PB, Mailankody S. Assessment of Outcomes Associated With the Use of Newly Approved Oncology Drugs in Medicare Beneficiaries. JAMA Netw Open. 2021;4:e210030. doi: 10.1001/jamanetworkopen.2021.0030. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Soni PD, Hartman HE, Dess RT, Abugharib A, Allen SG, Feng FY, et al. Comparison of population-based observational studies with randomized trials in oncology. J Clin Oncol. 2019;37:1209–1216. doi: 10.1200/jco.18.01074. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Eichler HG, Abadie E, Breckenridge A, Flamion B, Gustafsson LL, Leufkens H, et al. Bridging the efficacy–effectiveness gap: a regulator’s perspective on addressing variability of drug response. Nat Rev Drug Discov. 2011;10:495–506. doi: 10.1038/nrd3501. [DOI] [PubMed] [Google Scholar]
  • 38.Zuidgeest MGP, Goetz I, Groenwold RHH, Irving E, van Thiel GJMW, Grobbee DE. Pragmatic trials and real world evidence: Paper 1. Introduction. J Clin Epidemiol. 2017;88:7–13. doi: 10.1016/j.jclinepi.2016.12.023. [DOI] [PubMed] [Google Scholar]
  • 39.Food and Drug Administration (FDA) Framework for FDA’s Real-World Evidence Program. 2018. [Google Scholar]
  • 40.European Medicines Agency (EMA) Scientific guidance on post-authorisation efficacy studies. 2016. [Google Scholar]
  • 41.Kempf E, Bogaerts J, Lacombe D, Liu L. “Mind the gap” between the development of therapeutic innovations and the clinical practice in oncology: A proposal of the European Organisation for Research and Treatment of Cancer (EORTC) to optimise cancer clinical research. Eur J Cancer. 2017;86:143–149. doi: 10.1016/j.ejca.2017.08.028. [DOI] [PubMed] [Google Scholar]
  • 42.Lacombe D, Bogaerts J, Tombal B, Maignen F, Osipienko L, Sullivan R, et al. Late translational research: putting forward a new model for developing new anti-cancer treatments that addresses the needs of patients and society. Mol Oncol. 2019;13:558–566. doi: 10.1002/1878-0261.12431. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Lacombe D, O’Morain C, Casadei B, Hill K, Mateus E, Lories R, et al. Moving forward from drug-centred to patient-centred research: A white paper initiated by EORTC and developed together with the BioMed Alliance members. Eur Respir J. 2019;53:1801870. doi: 10.1183/13993003.01870-2018. [DOI] [PubMed] [Google Scholar]
  • 44.Lacombe D, Quaglio G, Lejeune S, Saesen R, Rübig P. Establishing treatment optimisation as part of personalised medicine development. Eur J Cancer. 2019;113:96–97. doi: 10.1016/j.ejca.2019.03.014. [DOI] [PubMed] [Google Scholar]
  • 45.Loudon K, Zwarenstein M, Sullivan FM, Donnan PT, Gágyor I, Hobbelen HJSM, et al. The PRECIS-2 tool has good interrater reliability and modest discriminant validity. J Clin Epidemiol. 2017;88:113–121. doi: 10.1016/j.jclinepi.2017.06.001. [DOI] [PubMed] [Google Scholar]
  • 46.Zwarenstein M, Thorpe K, Treweek S, Loudon K. PRECIS-2 for retrospective assessment of RCTs in systematic reviews. J Clin Epidemiol. 2020;126:202–206. doi: 10.1016/j.jclinepi.2020.06.023. [DOI] [PubMed] [Google Scholar]
  • 47.Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe K, Zwarenstein M. PRECIS-2 Toolkit. 2015. [DOI] [PubMed] [Google Scholar]
  • 48.Dal-Ré R, Mentz RJ, Rosendaal FR. Thoughtful selection and use of scientific terms in clinical research: The case of 'pragmatic' trials. J Investig Med. 2021;69:1056–1058. doi: 10.1136/jim-2021-001789. [DOI] [PubMed] [Google Scholar]
  • 49.Dal-Ré R. The misleading use of the term pragmatic in pre-licensing medicine trials. Eur J Clin Pharmacol. 2019;75:1033–1035. doi: 10.1007/s00228-019-02672-7. [DOI] [PubMed] [Google Scholar]
  • 50.Janiaud P, Dal-Ré R, Ioannidis JP. Assessment of Pragmatism in Recently Published Randomized Clinical Trials. JAMA Intern Med. 2018;178:1278–1280. doi: 10.1001/jamainternmed.2018.3321. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 51.Mullins CD, Montgomery R, Tunis S. Uncertainty in Assessing Value of Oncology Treatments. Oncologist. 2010;15:58–64. doi: 10.1634/theoncologist.2010-s1-58. [DOI] [PubMed] [Google Scholar]
  • 52.Banerjee R, Prasad V. Pragmatic trials with prespecified subgroups: what oncologists can learn from COVID-19. Nat Rev Clin Oncol. 2020;18:7–8. doi: 10.1038/s41571-020-00448-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53.Koehler M, Donnelly ET, Kalanovic D, Dagher R, Rothenberg ML. Pragmatic randomized clinical trials: a proposal to enhance evaluation of new cancer therapies with early signs of exceptional activity. Ann Oncol. 2016;27:1342–1348. doi: 10.1093/annonc/mdw143. [DOI] [PubMed] [Google Scholar]
  • 54.Nipp RD, Yao NA, Lowenstein LM, Buckner JC, Parker IR, Gajra A, et al. Pragmatic study designs for older adults with cancer: Report from the U13 conference. J Geriatr Oncol. 2016;7:234–241. doi: 10.1016/j.jgo.2016.02.005. [DOI] [PubMed] [Google Scholar]
  • 55.Tobias JS, Monson K, Gupta N, MacDougall H, Glaholm J, Hutchison I, et al. Chemoradiotherapy for locally advanced head and neck cancer: 10-year follow-up of the UK Head and Neck (UKHAN1) trial. Lancet Oncol. 2010;11:66–74. doi: 10.1016/s1470-2045(09)70306-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56.Salehi S, Åvall-Lundqvist E, Legerstam B, Carlson JW, Falconer H. Robot-assisted laparoscopy versus laparotomy for infrarenal paraaortic lymphadenectomy in women with high-risk endometrial cancer: A randomised controlled trial. Eur J Cancer. 2017;79:81–89. doi: 10.1016/j.ejca.2017.03.038. [DOI] [PubMed] [Google Scholar]
  • 57.Skovlund E, Leufkens HGM, Smyth JF. The use of real-world data in cancer drug development. Eur J Cancer. 2018;101:69–76. doi: 10.1016/j.ejca.2018.06.036. [DOI] [PubMed] [Google Scholar]
  • 58.Greshock J, Lewi M, Hartog B, Tendler C. Harnessing Real-World Evidence for the Development of Novel Cancer Therapies. Trends Cancer. 2020;6:907–909. doi: 10.1016/j.trecan.2020.08.006. [DOI] [PubMed] [Google Scholar]
  • 59.Luoma KA, Leavitt IM, Marrs JC, Nederveld AL, Regensteiner JG, Dunn AL, et al. How can clinical practices pragmatically increase physical activity for patients with type 2 diabetes? A systematic review. Transl Behav Med. 2017;7:751–772. doi: 10.1007/s13142-017-0502-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60.Lu L, Zhou L, Dong J, Xiang Y, Wen Z, Lu L, et al. The application of PRECIS-2 ratings in randomized controlled trials of Chinese herbal medicine. Oncotarget. 2017;8:107002–10. doi: 10.18632/oncotarget.22204. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61.Le Tourneau N, Germann A, Thompson R, Ford N, Schwartz S, Beres L, et al. Evaluation of HIV treatment outcomes with reduced frequency of clinical encounters and antiretroviral treatment refills: A systematic review and meta-analysis. PLoS Med. 2022;19:e1003959. doi: 10.1371/journal.pmed.1003959. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 62.Titz M, Schwameis M, Kienbacher C, Buchtele N, Roth D, Duma A, et al. Generalisability of randomised trials evaluating perioperative β-blocker therapy in noncardiac surgery. Br J Anaesth. 2020;125:926–934. doi: 10.1016/j.bja.2020.08.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 63.Fitzpatrick C, Gillies C, Seidu S, Kar D, Ioannidou E, Davies MJ, et al. Effect of pragmatic versus explanatory interventions on medication adherence in people with cardiometabolic conditions: a systematic review and meta-analysis. BMJ Open. 2020;10:e036575. doi: 10.1136/bmjopen-2019-036575. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 64.Robinson NB, Fremes S, Hameed I, Rahouma M, Weidenmann V, Demetres M, et al. Characteristics of Randomized Clinical Trials in Surgery From 2008 to 2020: A Systematic Review. JAMA Netw Open. 2021;4:e2114494. doi: 10.1001/jamanetworkopen.2021.14494. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 65.Gastaldon C, Mosler F, Toner S, Tedeschi F, Bird VJ, Barbui C, et al. Are trials of psychological and psychosocial interventions for schizophrenia and psychosis included in the NICE guidelines pragmatic? A systematic review. PLoS One. 2019;14:e0222891. doi: 10.1371/journal.pone.0222891. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66.Devos F, Foissac F, Bouazza N, Ancel PY, Tréluyer JM, Chappuy H. Study characteristics impacted the pragmatism of randomized controlled trial published in nursing: a meta-epidemiological study. J Clin Epidemiol. 2019;116:18–25. doi: 10.1016/j.jclinepi.2019.07.017. [DOI] [PubMed] [Google Scholar]
  • 67.Sepehrvand N, Alemayehu W, Das D, Gupta AK, Gouda P, Ghimire A, et al. Trends in the Explanatory or Pragmatic Nature of Cardiovascular Clinical Trials Over 2 Decades. JAMA Cardiol. 2019;4:1122–1128. doi: 10.1001/jamacardio.2019.3604. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 68.Choi MY, Barnabe C, Barber CE, Bykerk V, Pope JE, Hazlewood GS. Pragmaticism of Randomized Controlled Trials of Biologic Treatment With Methotrexate in Rheumatoid Arthritis: A Systematic Review. Arthritis Care Res. 2019;71:620–628. doi: 10.1002/acr.23620. [DOI] [PubMed] [Google Scholar]
  • 69.Xu C, Zhang Y, Chen L, Li WF, Tang SQ, Tang LL, et al. Association between outcome disparities and pragmatic features related to clinical trial and real-world settings in nasopharyngeal carcinoma: A population-based retrospective cohort study, 2006–2016. Radiother Oncol. 2020;151:306–313. doi: 10.1016/j.radonc.2020.08.019. [DOI] [PubMed] [Google Scholar]
  • 70.Ettori-Ajasse I, Tatin E, Forbes G, Eldridge S, Dibao-Dina C. How pragmatic are the randomised trials used in recommendations for control of glycosylated haemoglobin levels in type 2 diabetic patients in general practice: An application of the PRECIS II tool. Trials. 2020;21:281. doi: 10.1186/s13063-020-4215-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 71.Mittal N, George A, Leanos L, Kent P. Children’s Oncology Group (COG) clinical trials as exemplars of pragmatism in clinical research. J Clin Oncol. 2017;35:e18188. doi: 10.1200/jco.2017.35.15_suppl.e18188. [DOI] [Google Scholar]
  • 72.Sorigue M, Kuittinen O. Robustness and pragmatism of the evidence supporting the European Society for Medical Oncology guidelines for the diagnosis, treatment, and follow-up of follicular lymphoma. Expert Rev Hematol. 2021;14:655–668. doi: 10.1080/17474086.2021.1943351. [DOI] [PubMed] [Google Scholar]
  • 73.Johnson KE, Neta G, Dember LM, Coronado GD, Suls J, Chambers DA, et al. Use of PRECIS ratings in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory. Trials. 2016;17:23. doi: 10.1186/s13063-016-1158-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 74.Varas-Doval R, Saéz-Benito L, Gastelurrutia MA, Benrimoj SI, Garcia-Cardenas V, Martinez-Martínez F. Systematic review of pragmatic randomised control trials assessing the effectiveness of professional pharmacy services in community pharmacies. BMC Health Serv Res. 2021;21:156. doi: 10.1186/s12913-021-06150-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 75.Lipman PD, Loudon K, Dluzak L, Moloney R, Messner D, Stoney CM. Framing the conversation: Use of PRECIS-2 ratings to advance understanding of pragmatic trial design domains. Trials. 2017;18:532. doi: 10.1186/s13063-017-2267-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 76.Dal-Ré R. Articles provided insufficient information to conduct an appropriate retrospective assessment of the pragmatic/explanatory features of medicine trials with the PRECIS-2 tool. Eur J Clin Pharmacol. 2020;76:1093–1102. doi: 10.1007/s00228-020-02901-4. [DOI] [PubMed] [Google Scholar]
  • 77.Forbes G, Loudon K, Treweek S, Taylor SJC, Eldridge S. Understanding the applicability of results from primary care trials: lessons learned from applying PRECIS-2. J Clin Epidemiol. 2017;90:119–126. doi: 10.1016/j.jclinepi.2017.06.007. [DOI] [PubMed] [Google Scholar]
  • 78.Dal-Ré R. Could phase 3 medicine trials be tagged as pragmatic? A case study: The Salford COPD trial. J Eval Clin Pract. 2018;24:258–261. doi: 10.1111/jep.12796. [DOI] [PubMed] [Google Scholar]
  • 79.Chen G, Ma Q. Development of A Precis-2 Scoring Grid: How Pragmatic are Pragmatic Trials of Surgical and Pharmaceutical Interventions? Value Health. 2015;18:A729–A730. doi: 10.1016/j.jval.2015.09.2781. [DOI] [Google Scholar]
  • 80.Hohenschurz-Schmidt D, Kleykamp BA, Draper-Rodi J, Vollert J, Chan J, Ferguson M, et al. Pragmatic trials of pain therapies: A systematic review of methods. Pain. 2022;163:21–46. doi: 10.1097/j.pain.0000000000002317. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 81.Chan KW, Lee PW, Leung CP sha, Law YK, Gao L, Chan GC wang, et al. PRAgmatic Clinical Trial Design of Integrative MediCinE (PRACTICE): A Focus Group Series and Systematic Review on Trials of Diabetes and Kidney Disease. Front Med. 2021;8:668913. 10.3389/fmed.2021.668913. [DOI] [PMC free article] [PubMed]
  • 82.Palmer JA, Mor V, Volandes AE, McCreedy E, Loomer L, Carter P, et al. A dynamic application of PRECIS-2 to evaluate implementation in a pragmatic, cluster randomized clinical trial in two nursing home systems. Trials. 2018;19:453. doi: 10.1186/s13063-018-2817-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 83.Brunette CA, Miller SJ, Majahalme N, Hau C, MacMullen L, Advani S, et al. Pragmatic Trials in Genomic Medicine: The Integrating Pharmacogenetics In Clinical Care (I-PICC) Study. Clin Transl Sci. 2020;13:381–390. doi: 10.1111/cts.12723. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 84.Carlson LE, Oberoi DV, Qureshi M, Subnis U. Integrative Oncology Trials in the Real World: Assessing the Pragmatism of an Ongoing Integrative Oncology Trial of Mindfulness and T’ai Chi/Qigong. J Altern Complement Med. 2018;24:926–932. doi: 10.1089/acm.2018.0208. [DOI] [PubMed] [Google Scholar]
  • 85.Darker C, Loudon K, O’Connell N, Castello S, Burke E, Vance J, et al. An application of PRECIS-2 to evaluate trial design in a pilot cluster randomised controlled trial of a community-based smoking cessation intervention for women living in disadvantaged areas of Ireland. Pilot Feasibility Stud. 2022;8:19. doi: 10.1186/s40814-022-00969-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 86.Jamali M, Frey MA. Are Findings From a Pragmatic Dry Needling Trial Always Applicable in the Real World? J Orthop Sport Phys Ther. 2021;51:470–471. doi: 10.2519/jospt.2021.0202. [DOI] [PubMed] [Google Scholar]
  • 87.TwiCs network. Trials within Cohorts – Facilitating Multiple Randomised Controlled Trials. https://www.twics.global. Accessed 7 Jun 2022.
  • 88.Pielkenrood BJ, van der Velden JM, van der Linden YM, Bartels MMT, Kasperts N, Verhoeff JJC, et al. Pain Response After Stereotactic Body Radiation Therapy Versus Conventional Radiation Therapy in Patients With Bone Metastases-A Phase 2 Randomized Controlled Trial Within a Prospective Cohort. Int J Radiat Oncol Biol Phys. 2021;110:358–367. doi: 10.1016/j.ijrobp.2020.11.060. [DOI] [PubMed] [Google Scholar]
  • 89.Couwenberg AM, Burbach JPM, Berbee M, Lacle MM, Arensman R, Raicu MG, et al. Efficacy of Dose-Escalated Chemoradiation on Complete Tumor Response in Patients with Locally Advanced Rectal Cancer (RECTAL-BOOST): A Phase 2 Randomized Controlled Trial. Int J Radiat Oncol Biol Phys. 2020;108:1008–1018. doi: 10.1016/j.ijrobp.2020.06.013. [DOI] [PubMed] [Google Scholar]
  • 90.Relton C, Torgerson D, O’Cathain A, Nicholl J. Rethinking pragmatic randomised controlled trials: introducing the “cohort multiple randomised controlled trial” design. BMJ. 2010;340:963–967. doi: 10.1136/bmj.c1066. [DOI] [PubMed] [Google Scholar]
  • 91.van der Velden JM, Verkooijen HM, Ayoung-Afat D, Burbach JPM, van Vulpen M, Relton C, et al. The cohort multiple randomized controlled trial design: a valid and efficient alternative to pragmatic trials? Int J Epidemiol. 2017;46:96–102. doi: 10.1093/ije/dyw050. [DOI] [PubMed] [Google Scholar]
  • 92.Couwenberg AM, Burbach JPM, May AM, Berbee M, Intven MPW, Verkooijen HM. The trials within cohorts design facilitated efficient patient enrollment and generalizability in oncology setting. J Clin Epidemiol. 2020;120:33–39. doi: 10.1016/j.jclinepi.2019.12.015. [DOI] [PubMed] [Google Scholar]
  • 93.Tao JJ, Schram AM, Hyman DM. Basket Studies: Redefining Clinical Trials in the Era of Genome-Driven Oncology. Annu Rev Med. 2018;69:319–331. doi: 10.1146/annurev-med-062016-050343. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 94.Mangat PK, Halabi S, Bruinooge SS, Garrett-Mayer E, Alva A, Janeway KA, et al. Rationale and Design of the Targeted Agent and Profiling Utilization Registry Study. JCO Precis Oncol. 2018;2:1–14. doi: 10.1200/po.18.00122. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 95.Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898. doi: 10.1136/bmj.l4898. [DOI] [PubMed] [Google Scholar]
  • 96.Lorenz E, Köpke S, Pfaff H, Blettner M. Cluster-randomized studies Dtsch Arztebl Int. 2018;115:163–8. 10.3238/arztebl.2018.0163. [DOI] [PMC free article] [PubMed]
  • 97.Ford I, Norrie J. Pragmatic Trials. N Engl J Med. 2016;375:454–463. doi: 10.1056/nejmra1510059. [DOI] [PubMed] [Google Scholar]
  • 98.Dal-Ré R, de Boer A, James SK. The design can limit PRECIS-2 retrospective assessment of the clinical trial explanatory/pragmatic features. J Clin Epidemiol. 2020;126:193–201. doi: 10.1016/j.jclinepi.2020.03.027. [DOI] [PubMed] [Google Scholar]
  • 99.Taljaard M, McDonald S, Nicholls SG, Carroll K, Hey SP, Grimshaw JM, et al. A search filter to identify pragmatic trials in MEDLINE was highly specific but lacked sensitivity. J Clin Epidemiol. 2020;124:75–84. doi: 10.1016/j.jclinepi.2020.05.003. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Additional file 1. (88.7KB, pdf)
Additional file 2. (378.1KB, xlsx)

Data Availability Statement

All data generated or analyzed during this study are included in this published article and its supplementary information files.


Articles from BMC Medical Research Methodology are provided here courtesy of BMC

RESOURCES