Systematic Review of the “Pragmatism” of Pragmatic Critical Care Trials

OBJECTIVES:

To assess the pragmatism of published critical care randomized controlled trials self-described as pragmatic using a validated tool.

DATA SOURCES:

Medical Literature Analysis and Retrieval Online database and PubMed interface from inception to November 1, 2021.

STUDY SELECTION:

We performed a systematic search of randomized controlled trials evaluating interventions for critically ill adults that self-identified as pragmatic in title or abstract.

DATA EXTRACTION:

Reviewers independently performed study selection and data extraction in duplicate; discrepancies were resolved by consensus. Pragmatism was assessed independently in duplicate by trained reviewers using the Pragmatic-Explanatory Continuum Indicator Summary 2 (PRECIS-2), a validated tool designed to represent how explanatory/pragmatic a trial is on the pragmatic to explanatory continuum. Trials were scored in nine domains on a 5-point continuum (from 1 = very explanatory to 5 = very pragmatic). Discrepancies of greater than 2 points were adjudicated by consensus discussion.

DATA SYNTHESIS:

The search resulted in 284 studies; 56 met eligibility criteria. Forty-one of the trials had a discrepancy in at least one domain that required consensus discussion, most commonly in domains of eligibility and follow-up. Twelve studies (21.4%) were scored as “overall pragmatic,” defined as score of greater than 4 in five domains provided the scores in the remaining domains were three. The overall PRECIS-2 score of self-identified pragmatic studies increased from 1995 to 2021 suggesting increasing pragmatism over time. Pragmatic trials were more likely to have a waiver of informed consent (p = 0.05).

CONCLUSIONS:

The number and pragmatism of self-identified pragmatic trials have increased, particularly in the past decade. However, less than one-quarter of these trials that use the term pragmatic in title or abstract were retrospectively rated as pragmatic. Our results support the concept that trials are designed on a spectrum of pragmatic to explanatory. Advances in the design and reporting of critical care trials are needed to ensure their real-world applicability.

Critical care medicine is increasingly complex, with intensivists making over 100 decisions per day (1). These decisions are guided by the results of clinical trials when available—however, the unique practice context of critical care invokes challenges for trials such as problems of patient selection and recruitment as well as heterogeneity in treatment and care delivery (2). For example, participants may be required to meet strict enrollment criteria and intervention protocols may be impractical to implement in the general critical care community. Pragmatic trials use a research design that may overcome some of these barriers and increase the applicability of trial results when applied outside the typical research setting. Experts have called for increased adoption of pragmatic trial methodology in the field of critical care (3, 4).

Schwartz and Lellouch (5) first coined the terms “pragmatic” and “explanatory” as terms to describe the focus of clinical trials in 1967. Explanatory trials are optimized to determine the efficacy of an intervention, confirming a clinical or physiologic hypothesis. Explanatory trials seek to answer the question, “Does this intervention work in ideal conditions?” Pragmatic clinical trials are designed to provide evidence of the real-world effectiveness of an intervention in a broad patient group and therefore inform a clinical or policy decision (6). Pragmatic trials focus on the question, “Does this intervention work under usual conditions?” In practice, most trials exist across a continuum of explanatory to pragmatic (7). The original Pragmatic-Explanatory Continuum Indicator Summary (PRECIS) tool, published in 2009, attempted to clarify the concept of pragmatism and provided a scoring system across various trial design features for use by researchers at the design phase of a clinical trial (7). This tool was subsequently adapted into PRECIS-2, a validated tool that focuses on trial design choices, which determine the applicability of a trial (8).

Pragmatic trials have their own complexities and, in many cases, may not meet the criteria to constitute a true pragmatic trial (9). We conducted a systemic review of randomized controlled trials (RCTs) that evaluated interventions for critically ill adults and used the term pragmatic in title or abstract. Our study had the following objectives: 1) Quantify the number of critical care trials self-identified as pragmatic and assess the change in prevalence over time of self-described pragmatic critical care intervention trials and 2) Assess the degree of pragmatism for these across different domains of trial design using the PRECIS-2 tool (10).

MATERIALS AND METHODS

We conducted a systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines as detailed in Appendix A (http://links.lww.com/CCX/B41) (11). The protocol was registered in the International Prospective Register for Systematic Reviews (registration CRD42021282329). The study did not meet criteria for human subjects research and was exempt from review by the Institutional Review Board at our institution.

Search Strategy

We searched the literature from inception up to November 1, 2021, using the Medical Literature Analysis and Retrieval Online database and the PubMed interface. The search strategy was determined a priori. We identified RCTs evaluating interventions for critically ill adults that included the word pragmatic in title or abstract. Our complete search strategy is listed in Appendix B (http://links.lww.com/CCX/B41).

Study Selection and Data Extraction

Two reviewers (J.A.P., S.P.T.) independently screened titles and abstracts for prespecified eligibility criteria. Articles were included for full-text review if the following eligibility criteria were met 1) RCT study design, 2) enrolled critically ill adults, and 3) used the term pragmatic in title or abstract. Discrepancies on eligibility criteria were resolved by consensus. Four trained physician reviewers assessed pragmatism using the PRECIS-2.

Outcome and Scoring Process

We scored all articles selected for review using the PRECIS-2 tool (10). The PRECIS-2 rating system has been recommended as a tool to plan pragmatic trials and has also been used to categorize published trial designs (8, 12, 13). PRECIS-2 is represented as a nine-spoked “wheel” with the following individual domains: 1) eligibility criteria; 2) recruitment; 3) setting; 4) organization; 5) flexibility delivery; 6) flexibility adherence; 7) follow-up; 8) primary outcome; and 9) primary analysis (8). Each domain can be scored using a 5-point Likert scale in which 1 means very explanatory, 2 rather explanatory, 3 equally pragmatic and explanatory, 4 rather pragmatic, and 5 very pragmatic. All four physician reviewers involved in this study underwent training on use of the PRECIS-2 trial prior to study initiation. Training included viewing the National Institutes of Health Health Care Systems Research Collaboratory webinar (available at www.nihcollaboratory.org/Pages/Grand-Rounds-01-22-16.aspx), rating two RCTs excluded from the initial search using the PRECIS-2 criteria, and participating in a consensus discussion on the rating.

All articles selected for review were rated independently across all nine domains of the PRECIS-2 tool by two physician reviewers. Any domain with a discrepancy of greater than 2 in scoring on the Likert scale was discussed with all four physician reviewers until a consensus was reached. For example, if one reviewer scored a study across a particular domain as 2, rather explanatory, and a second reviewer scored the domain as 4, rather pragmatic, the study and domain were discussed among all four reviewers until a consensus was reached on the score. For domains with only 1 point difference, we used the average of the two scores as the domain score included in our analyses. Although there is no standardized cutoff score for when a trial is considered pragmatic enough to be labeled as pragmatic, for this review, we defined a PRECIS-2 summary score as pragmatic after consensus review if scores were 4 or greater in five domains provided the scores in the remaining domains are three as previously described (14).

We also collected data on reporting practices in the selected trials. We recorded whether studies included their own PRECIS-2 wheel, as has been recommended. Because pragmatic trials often involve complex interventions, we also recorded whether authors included a Template for Intervention Description and Replication (TIDieR) checklist, which provides key features of an intervention such as duration, dose or intensity, mode of delivery, essential processes, and monitoring in sufficient detail that the intervention can be understood and replicated (15). We assessed risk of bias using Version 2 of the Cochrane risk-of-bias tool for randomized controlled trials (RoB 2) (16).

Statistical Analyses

Study characteristics are reported as number (%) for categorical data and median (interquartile range [IQR]) for continuous data. PRECIS-2 scores across individual domains were summarized with descriptive statistics. We used parametric tests (mean) and nonparametric tests (median) given lack of consensus about most appropriate measure of central tendency when reporting results of Likert scale and prior published results using PRECIS-2 (17–19). Differences between PRECIS-2 summary score across key study characteristics were evaluated using Fisher exact tests given small number of pragmatic score trials. Linear regression was used to evaluate for change in mean pragmatic scores over time. All p value reported are for two-sided alpha of less than 0.05. All statistical analyses were conducted using Stata statistical software Version 17.0 (StataCorp, College Station, TX).

RESULTS

Characteristics of Included Studies

After excluding duplicates, our search retrieved 284 articles of which 56 met criteria for full-text review (Fig. 1). The citations for the articles included in this review are included in Appendix C (http://links.lww.com/CCX/B41). Characteristics of included studies are shown in Table 1. The pragmatic trials were published in 28 distinct journals, the median impact factor of which (in March 2022) was 5.70 (IQR, 3.17–17.66). The majority of studies (n = 45, 80.3%) were conducted in North America or Europe and were multicenter (n = 39, 69.6%). Most (n = 51, 91.1%) had an intention-to-treat primary analysis and 18 of the trials (32%) reported the studied intervention improved the primary outcome (positive trial). Two studies (4%) reported a PRECIS wheel; one of these also reported a TIDieR checklist. One additional study reported a TIDieR checklist. Thirty-five studies (63%) were determined to be at low risk of bias. We had some concerns of bias in 18 studies (32%) and 3 (5%) were deemed to be at high risk using the RoB 2 tool.

Figure 1.

Article screening and selection. Adapted from Page et al (11). MEDLINE = Medical Literature Analysis and Retrieval Online.

Table 1.

Characteristics of Included Studies (n = 56)

Characteristic	n (%)
Publication period
Prior to 2010	3 (5.4)
2011–2015	17 (30.4)
2016–2020	30 (53.6)
2021 to present	6 (10.7)
Geographic location
North America	22 (39.3)
Europe	23 (41.1)
Asia and Middle East	2 (3.6)
Australia	3 (5.4)
Other	6 (10.7)
Study type
Individual level randomization	39 (69.6)
Cluster RCT, parallel design	4 (7.1)
Cluster RCT, crossover (includes stepped-wedge)	13 (23.2)
Multicenter study (vs single-center)	39 (69.6)
Unit of analysis
Individual patient	47 (83.9)
ICU or ward	6 (10.7)
Other	3 (5.4)
Type of intervention studied
Drug or medication	14 (25.4)
Device	10 (18.2)
Patient-level care intervention	25 (44.6)
Ward-level intervention	4 (7.3)
Postdischarge intervention	3 (5.5)
Informed consent
Consent prior to randomization	24 (42.9)
Delayed consent	5 (8.9)
No individual level informed consent	27 (48.2)
Funding source
Federal	31 (55.4)
Industry	12 (21.4)
Departmental or internal	11 (19.6)
Other	2 (3.6)
Primary analysis
Intention to treat	51 (91.1)
Per protocol	5 (8.9)
Primary result, intervention improved outcome	18 (32)

RCT = randomized controlled trial.

Scoring Across PRECIS-2 Domains

After independent review by two reviewers, 41 of the 56 trials reviewed had a discrepancy of greater than 2 in at least one domain and required consensus discussion. The most common discordance between reviewers was on the domains of “eligibility” and “follow-up”; 14 of the 56 trials required a consensus discussion in these domains. Reviewers most frequently agreed on scoring the domain of “primary analysis” where only four trials required a consensus discussion. After discussion, the mean and median scores for each domain are listed in Table 2. Trials were most pragmatic in “primary analysis” (mean score, 4.25; sd, 0.83) and least pragmatic in “primary outcome” (mean score, 3.49; sd, 1.33). The mean PRECIS-2 score across all nine domains in the 56 trials was 3.81 (sd, 0.63) and median 3.86 (IQR, 3.4–4.3). A total of 12 studies (21.4%) scored 3 or greater across all domains. Using our predetermined cutoff (scores ≥ 4 in five domains provided the scores in the remaining domains are three as previously described), these 12 self-identified pragmatic trials were labeled as pragmatic after consensus review.

Table 2.

Pragmatic-Explanatory Continuum Indicator Summary 2 Domain Scores and Consensus Review

Pragmatic-Explanatory Continuum Indicator Summary 2 Domain	Median Score (IQR)	Mean (sd)	Consensus Review, n (%)a
Eligibility	4 (4–4.5)	3.85 (1.03)	14 (25.0)
Recruitment	4 (2.4–5)	3.81 (1.22)	9 (16.07)
Setting	4 (2.25–4.5)	3.64 (1.20)	6 (10.71)
Organization	4 (3.5–4.5)	3.88 (0.96)	10 (17.86)
Flexibility—delivery	4 (3.25–4.5)	3.79 (1.07)	12 (21.43)
Flexibility—adherence	4 (3.5–4.75)	3.95 (1.00)	8 (14.29)
Follow-up	4 (2–4.75)	3.49 (1.33)	14 (25.0)
Primary outcome	4 (2.5–4.5)	3.61 (1.25)	6 (10.71)
Primary analysis	4.5 (4–5)	4.24 (0.83)	4 (7.14)

IQR = interquartile range.

^aNumber of studies that required consensus discussion for ≥ 2 point difference in Pragmatic-Explanatory Continuum Indicator Summary 2 score on individual review.

Association of Trial Characteristics With Pragmatism

There has been an increase in the mean PRECIS-2 scores over time (Fig. 2) with a weak correlation between year (R² = 0.15; p = 0.002). All of the 12 studies scored as pragmatic after consensus review were published after 2011; nine of the 12 were published between 2018 and 2021. The majority of the pragmatic studies were funded by governmental funds and included individual level randomization; 6 (50%) were cluster randomized trials (Table 2). Nine of the pragmatic trials (75.0%) required individual level informed consent versus 18 (40.9%) of nonpragmatic scored trials (p = 0.05). Compared with trials with a nonpragmatic summary score, trials with a pragmatic summary score after consensus review were more likely to have a waiver of individual consent. There were no differences in primary outcome results (positive or negative trial), funding source, unit of analysis, or cluster versus individual level randomization (Table 3). Six of the trials (50%) scored as pragmatic after consensus review were deemed low risk of bias and six (50%) were deemed to have some concerns for bias. Trials with a pragmatic summary score after review were not more likely to have bias concerns compared with trials with a nonpragmatic summary score (p = 0.32).

Figure 2.

Mean Pragmatic-Explanatory Continuum Indicator Summary 2 (PRECIS-2) scores of included studies over time. Linear regression line represents best fit; 95% CIs are represented by gray shading.

Table 3.

Differences in Study Characteristics Between Pragmatic and Nonpragmatic Overall Pragmatic-Explanatory Continuum Indicator Summary 2 Scores

Study Characteristic	Pragmatic PRECIS-2 Summary Scorea (n = 12)	Nonpragmatic PRECIS-2 Summary Score (n = 44)	p
Intervention improved primary outcome (yes)	7 (58.3%)	31 (70.5%)	0.43
Federal funding	10 (83.3%)	37 (84.1%)	0.63
Unit of analysis at individual level	10 (83.0%)	37 (84.1%)	0.63
Waiver of individual level of consent	9 (75%)	18 (50.9%)	0.05
Cluster randomized study design	6 (50%)	11 (25%)	0.15

PRECIS-2 = Pragmatic-Explanatory Continuum Indicator Summary 2.

^aPragmatic PRECIS-2 summary score is defined as a consensus score of ≥ 4 in five domains provided the scores in the remaining domains are three or greater.

DISCUSSION

Our study confirms an increase in the number of self-described pragmatic trials of critical care interventions, particularly in the past decade. Although studies’ average PRECIS-2 score indicated slightly increasing overall pragmatism, 79% of self-identified pragmatic trials did not meet our proposed definition of pragmatic after consensus review.

Pragmatism in trial design arose from concerns that result of trials optimized to demonstrate efficacy may not apply to real-world settings (6). In reality, it may not be feasible or appropriate to be pragmatic in all domains and most trials exist along a continuum of pragmatic to explanatory (10, 14). We found self-identified pragmatic trials also existed along this continuum. Trials were most pragmatic in the domain of “primary analysis” (“to what extent are all data included?”) (10). For this domain, a score of 5 (very pragmatic) reflects an intention-to-treat analysis with all available data, whereas a score of 1 (very explanatory) may be given for a primary analysis that included only those participants that followed treatment protocol. We found the majority of self-identified pragmatic trials (> 90%) used an intention-to-treat primary analysis with all available data, now widely considered to be the gold standard for assessing superiority of an intervention (20, 21). In general, study authors were clear in reporting the primary analysis performed, and there was high agreement among reviewers in scoring this domain (only four required consensus discussion).

We found self-identified pragmatic trials were least pragmatic in the PRECIS-2 domains of “setting” (“how different is the setting of the trial than usual care?”) and “primary outcome” (“to what extent is the outcome relevant to participants?”). Consistent with prior reports, we found the “setting” domain difficult to rate relative to usual care as centers able to conduct complex critical care interventions (even if quality improvement focused) are often relatively well-resourced in terms of organization and resources (19). Furthermore, 30% of self-identified pragmatic critical care trials were single-center studies, a more explanatory and less generalizable approach to trial design. Although this may be necessary in some cases, the implications of single-center designs on pragmatism should be clearly acknowledged by the study authors. The domain “primary outcome,” scored by the extent to which the outcome selected was relevant to participants, also required discussion in our review, partly because much is still unknown about how patients value particular outcomes (22). We categorized 32% of study primary outcomes (from 18 studies) as explanatory (score of < 3), consistent with other reports that remarkably few critical care RCTs select patient-important outcomes as primary outcomes (23). To improve the real-world applicability of critical care research, continued efforts are needed to identify key patient-important outcomes along with efficient methods for their measurement and analysis (22, 24, 25).

Retrospectively scored pragmatic trials were more likely to be conducted with a waiver of individual consent, likely related to the higher frequency of cluster randomization (26). The role of individual level informed consent in cluster RCTs is debated, with arguments for adhering strictly to ethical principles countered by the limitations individual consent pose to the scientific validity of the study (27, 28). Reporting around individual consent for cluster RCTs is poor (26, 29), and we urge trialists to explicitly report their decision to seek or waive informed consent along with ethics committee approval.

Although we were able to reach consensus scores for all studies, the disagreement in independent reviewer scores (41/56 studies required discussion) underscores both the subjectivity in rating domains as well as lack of adequate reporting of study elements related to pragmatism. Only two of the trials in this review included a PRECIS-2 wheel in their publication (30, 31). In those two trials, our retrospective ratings were very similar to the authors’ prospective ratings (within 1 point for all domains). Whether factors related to pragmatism were considered prior to study start (as recommended) is unknown (14). Although the PRECIS-2 tool has been used retrospectively to assess the degree of pragmatism, the findings may not be reliable unless clear information is available in each of the nine domains (10, 19). We recommend that journals encourage authors of pragmatic RCTs to include their preregistered PRECIS-2 tool assessment, allowing for reviewers and readers to appraise the degree of pragmatism of the RCT (14). Additionally, despite increasing focus on studying complex health interventions, only two studies included an explicit intervention description such as the TIDieR checklist (15, 30, 33). The lack of clear intervention reporting made it challenging to retrospectively discern whether certain trial processes were organizational elements of the trial or part of the intervention itself, leading to discrepant scoring for the organization domain.

Our study has several strengths. We conducted this systematic review of pragmatic RCTs using strategies to minimize bias, with a comprehensive search for trials and independent duplicate data abstraction. All reviewers were trained in the use of the PRECIS-2 study wheel, and our design allowed for discussion and consensus to be reached on all disagreements. Our study also has several limitations. We only reviewed RCTs self-identified as pragmatic in title or abstract, and there may be pragmatic trials that were not captured by this search. As we describe, the use of the PRECIS-2 tool to determine pragmatism based on reviewing the publication retrospectively was challenging in many cases. We acknowledge that another group of experienced reviewers or those involved in the design of the individual studies may score pragmatism differently. Finally, we chose a cutoff for pragmatism as described in prior work; we acknowledge that other definitions of pragmatism may be appropriate and provide a different perspective to the relationship of pragmatism with clinical trial characteristics. Most trials exist on a continuum of explanatory to pragmatic across individual domains; however, we feel it is important for study authors to explicitly discuss in what aspects a trial labeled as pragmatic may or may not be pragmatic.

CONCLUSIONS

The number of critical care trials that included the word pragmatic in title or abstract has increased over time; however, the majority of these trials had one or more features of an explanatory trial. Advances in the design, conduct, and reporting of pragmatic critical care trials are needed to ensure real-world applicability.