Contemporary strategies to improve clinical trial design for critical care research: insights from the First Critical Care Clinical Trialists Workshop

Abstract

Background

Conducting research in critically-ill patient populations is challenging, and most randomized trials of critically-ill patients have not achieved pre-specified statistical thresholds to conclude that the intervention being investigated was beneficial.

Methods

In 2019, a diverse group of patient representatives, regulators from the USA and European Union, federal grant managers, industry representatives, clinical trialists, epidemiologists, and clinicians convened the First Critical Care Clinical Trialists (3CT) Workshop to discuss challenges and opportunities in conducting and assessing critical care trials. Herein, we present the advantages and disadvantages of available methodologies for clinical trial design, conduct, and analysis, and a series of recommendations to potentially improve future trials in critical care.

Conclusion

The 3CT Workshop participants identified opportunities to improve critical care trials using strategies to optimize sample size calculations, account for patient and disease heterogeneity, increase the efficiency of trial conduct, maximize the use of trial data, and to refine and standardize the collection of patient-centered and patient-informed outcome measures beyond mortality.

Electronic supplementary material

The online version of this article (10.1007/s00134-020-05934-6) contains supplementary material, which is available to authorized users.

Randomized clinical trials (RCTs) are undertaken to identify interventions to improve clinically important outcomes using pre-specified criteria (e.g., p value < 0.05). Though several landmark critical care trials have achieved this goal, most have not [1–3]. In February 2019, a diverse group of patient representatives, regulators from the USA and European Union, federal grant managers, industry representatives, clinical trialists, epidemiologists, and clinicians convened the First Critical Care Clinical Trialists (3CT) Workshop. The 2-day meeting included five pre-planned sessions focused on specific critical illnesses, as well as overarching discussions focused on specific challenges in conducting and assessing critical care trials more broadly. Additional details are provided in the online supplement. The goal of the 3CT Workshop was to share experiences, enumerate potentially modifiable trial challenges, identify shared priorities across stakeholders, and consolidate these into consensus recommendations and research priorities (Table 1).

Table 1.

Recommendations to improve clinical trial design for critical care research from the First Critical Care Clinical Trialists (3CT) Workshop

Domain	Recommendation	Description and comment
Study design	Pre-specify plans for sample size re-estimation during trial design	Allows for the adjustment of the targeted sample size if outcome event rates observed in the trial differ from the initial power calculation
	Use predictive enrichment strategies for interventions in which there is a mechanistic rationale (physiologic, biologic, or genetic) to suggest why some patients may respond while others do not	Uses data from prior trials or observational data to identify patients who are likely to experience the most benefit from a given intervention, with the goal of developing enrollment criteria to selectively enroll these patients
	Use pragmatic trials to evaluate supportive therapies that might benefit a wide range of conditions or patients (e.g., early mobilization, ventilator weaning strategies, types of fluid resuscitation)	Uses broad enrollment criteria to enroll a diverse group of patients that are representative of those who would receive the intervention in usual care
	Use response-adaptive randomization for early phase trials and trials evaluating conditions with many available treatments	Incorporates information learned during the trial to (i) optimize allocation to study arms yielding the best results, which minimizes risks to patients; or (ii) optimize enrollment criteria enriching for better performing subgroups
	Evaluate opportunities to incorporate multiple trial interventions into platform trials	Simultaneously randomizes multiple, independent interventions or intervenes at multiple points in the same disease process (e.g., a trial evaluating initial therapy for a condition that feeds directly into a second trial of rescue therapies)
Study design and analysis	Incorporate a pre-specified Bayesian analysis plan with a range of priors	Analyzes trial results in the context of previously observed or presumed treatment effect distributions, producing results in terms of a likelihood of an effect on a probabilistic scale (i.e., the probability of an effect being present on a scale of 0–100)
Study conduct	Improve collaboration between critical care and pre-ICU providers (emergency medicine, pre-hospital)	Allows intervention earlier in the course of critical illness and significantly improves enrollment for interventions with narrow therapeutic windows
Outcome measures	Attempt to standardize common outcome measures across trials	Allows for meaningful across-trial comparisons
	Integrate diverse stakeholders (such as patients and families) into trial design and continue research on the development, measurement, and timing of patient-reported outcome measures	Promotes patient-centered critical care, while addressing the key challenges of patient-reported outcome measures, including the ideal timing of collection, how to account for the competing risk of mortality, and the possibility of biases introduced by incomplete long-term follow-up
Data Sharing	Encourage data sharing of de-identified patient data	Sharing data with robust data dictionaries to investigators who have pre-specified secondary analyses provides opportunities to maximize the knowledge gained from clinical trials and maximally leverages the investments made by patients, funding organizations, and researchers

The challenge

It is possible that most critical care trials have failed to demonstrate clinical benefit because the studied interventions are truly ineffective. However, there are unique challenges with critical care trials that make definitive outcome assessment difficult. For example, patients are often unable to provide consent, and surrogate decision-makers may not be available within the narrow therapeutic window of the studied intervention. Further, improved care for chronic conditions such as cancer, rheumatologic disease, cardiovascular disease, and HIV has created an increasingly complex, elderly, and immunocompromised population of critically ill patients. These pre-existing comorbidities are significant contributors to patient outcomes, but are unlikely to be modified by treatments targeting acute critical illness [4]. Finally, poorly characterized heterogeneity in patient populations and disease states may result in trials enrolling patients with opposing responses to treatment [5].

Demanding realistic trial design

Given the time and cost required to conduct a clinical trial, there is an incentive to enroll the smallest allowable number of patients. Consequently, sample size and timeline calculations have often been criticized as overly optimistic (or even unrealistic) and, therefore, predestined to produce results less informative than desired [6–9]. More worrisome, underpowered trials risk wasting resources and delivering false or uncertain conclusions about the effectiveness of tested interventions. The latter can result in the potential promotion of an ineffective therapy as well as delaying or permanently thwarting the identification of a promising therapy.

Several assumptions are required to select a target sample size for a trial. Investigators must estimate the expected control group outcome, the variation in the outcome, and the intervention-associated change. Pilot trials can be used to provide estimates of the frequency and distribution of a planned primary end point and can aid in choosing which therapies warrant further investigation by demonstrating safety, feasibility, and the therapy’s effect on a surrogate in the causal pathway. However, the small size of pilot trials may cause imprecise and potentially biased estimates of treatment effects [10]. Using historical cohorts also presents challenges, as they may differ from future trial settings in patient composition and aspects of care other than the trial intervention(s). Target trial emulation, the application of design principles from randomized trials to the analysis of observational data, may provide more accurate estimates of baseline event rates and expected treatment effects [11]. However, at their core, these approaches are forms of educated guessing and may fail to reflect patient characteristics in a future trial, particularly when trials may be designed years before they enroll patients. The 3CT Workshop participants suggested building flexibility into sample size calculations when possible to allow adjustment of sample size targets mid-trial if lower than expected event rates are observed in the control group. This approach has been successfully used in several critical care trials (e.g., PROWESS-SHOCK [12] and SMART [13] trials) but requires funding agencies to allow flexible budgets and timelines so the most promising ongoing trials can be completed.

Leveraging the benefits of new trial design methodology

There is a growing interest in the application of more flexible trial designs in critical care. In this section, we discuss considerations for the trial design innovations that the 3CT Workshop participants considered most promising. A visual summary is presented in Fig. 1, and the key advantages and disadvantages of these trial design methodologies are listed in Table 2.

Fig. 1.

Visual depiction of key design elements and differences in randomized trial design

Table 2.

Advantages and disadvantages of new trial design methodologies

Design type	Advantages	Disadvantages	Examples from critical care
Prognostic enrichment	Increases trial efficiency by enrolling a population with a higher likelihood of an event, allowing adequate statistical power with fewer patients	Requires models that can reliably predict patient outcomes. Assumes a therapy provides a consistent treatment effect across the range of risks for the primary outcome; an assumption that may fail for patients at an advanced stage of illness, or for therapies that provide the largest benefit to less severely ill patients	PROWESS-SHOCK [12]
Predictive enrichment	May increase trial efficiency, reduce the impact of heterogeneity, and enhance the likelihood of identifying personalized therapies	Requires bedside methods to identify biomarkers or differentiate proposed subphenotypes prior to trial enrollment. Results may not be generalizable into the clinical setting where rapid biomarker or subphenotype identification is often difficult	AdrenOSS-2 [92] COMBAT-SHINE (NCT04123444) VIOLET [93] EUPHRATES [94]
Pragmatic trial	Maximizes generalizability and facilitates accurate effect estimates for all patients likely to receive a given intervention	Must enroll enough patients to detect heterogeneity across diverse levels of illness severity and patient subgroups	SMART [13] CRASH [95]
Sample size re-estimation (adaptive trial)	Reduces the likelihood of promising trials ending for futility and being underpowered	Requires flexible budgets and timelines. If performed blinded to treatment effect, may lead to increased expenditure of resources on ineffective treatments. If performed using treatment effects, may introduce operational biases and increase the risk of type 1 errors	PROWESS-SHOCK [12] SMART [13]
Bayesian sequential design	Increases trial efficiency by allowing early stopping for efficacy, safety, or futility. Lessens the risk of underpowered trials	Increases complexity of trial planning and execution. Difficult to predict trial duration or cost. Requires significant central effort to perform frequent analyses. May not be possible for trials with longer-term outcomes	SEPSIS-ACT [38]
Response-adaptive randomization	May increase the likelihood of identifying beneficial treatments by prospectively identifying and targeting enrollment of subgroups receiving the largest benefit or increasing treatment allocation to study arms yielding the best results, increasing power, and protecting patients	Increases the complexity of trial planning and execution. Introduces potential operational biases, as the path of trial adaptations provides insight into the outcomes of enrolled participants	PROSpect (NCT03896763)
Platform trials	Allows for more efficient conduct of clinical trials and provides the opportunity to answer multiple scientific questions with a relatively small iterative addition of effort	Dramatic increase in complexity, particularly for designs that include adaptive features. May be challenging for institutional review boards and regulatory bodies to appropriately review and oversee. Raises ethical issues regarding the ability of patients to understand full trial protocols and provide informed consent	REMAP-CAP (NCT02735707)
Bayesian trial analysis	Promotes interpretation of trial results in the context of prior research and may provide more information than dichotomous trial interpretations using a fixed p value	Trial interpretations may be driven by the selected prior which can be incorrect or manipulated, and for which there is not a community standard. Each prior used will result in a different trial interpretation, which can complicate decision-making and overall trial interpretations. Conducting both Bayesian and frequentist analyses increases the risk of selective reporting	EOLIA Re-analysis [67] ANDROMEDA-SHOCK Re-analysis [68]

Prognostic enrichment

Clinical trialists have long used inclusion and exclusion criteria to create trial populations likely to benefit and unlikely to experience harm from the studied intervention, a process known as “enrichment” of the trial population. In recent years, there has been a great deal of interest in finding more sophisticated forms of trial enrichment. Prognostic enrichment designs seek to enroll patients with characteristics that suggest a high likelihood of experiencing the primary outcome. Assuming a therapy provides a consistent treatment effect to all patients, patients at the highest risk of the outcome will receive the largest absolute treatment effect. Enrolling patients with a high expected event rate facilitates greater statistical power with fewer patients. Furthermore, there may be cases where a trial intervention carries risks that might only be justified in patients with a high likelihood of a disease-related outcome [14]. Recent studies have attempted to improve upon existing risk scores and identify factors associated with disease-specific mortality that could be used for prognostic enrichment of future critical care trials [15–17]. Though attractive in principle, there are considerable limitations to this approach [18]. Patients at the highest risk of an outcome may not benefit from therapy due to the advanced stage of illness, comorbidities, or concomitant conditions. Furthermore, there are several therapies where an effect size appeared larger in the less severely ill patients (heterogeneity of treatment effect) [19–21]. Thus, at present, there are limitations to the broad uptake of prognostic trial designs in critical care.

Predictive enrichment

Predictive designs seek to enroll patients with an increased likelihood of a benefit from a therapy, independent of the absolute risk of the outcome. Predictive enrichment is part of a broader push to provide “personalized” therapies that are chosen based on complex patient characteristics. Predictive enrichment efforts have focused on interventions in which there is a mechanistic rationale (physiologic, biologic, or genetic) suggesting why some patients may respond while others do not. Recent work has identified potential patient phenotypes of acute respiratory distress syndrome (ARDS) [22–25] and sepsis [26] which appear to experience differential responses to interventions, and several ongoing trials are using novel biomarkers to identify study populations (e.g. COMBAT-SHINE NCT04123444). A principal challenge to the use of predictive enrichment is the need for rapid, bedside identification of patient phenotypes, including the need to quantify biomarkers at the time of trial enrollment. For many interventions, it may not be practical to wait for the results of the highly specific tests needed to identify these unique phenotypes.

Pragmatic trials

Limited generalizability is an additional limitation of prognostic and predictive enrichment strategies. For supportive therapies benefitting a wide range of patients (e.g., early mobilization, ventilator weaning strategies, types of fluid resuscitation), it may be preferable to enroll a broad population of patients with few inclusion/exclusion criteria. This approach, frequently labeled as “pragmatic,” attempts to enroll patients similar to those who would receive the intervention in routine care, thereby providing real-world estimation of treatment effects rather than extrapolating results from highly selective explanatory trials [27]. Because heterogeneity may exist in the way patients respond to trial interventions, pragmatic trials must enroll enough patients to detect heterogeneity across diverse levels of illness severity and patient subgroups. Innovations such as the ability to easily identify patient cohorts and capture patient characteristics using electronic health records (EHRs) have facilitated the design of large pragmatic trials. Further, several recent trials have been conducted by embedding all study procedures (screening, enrollment, intervention delivery, and outcome collection) into automated tools in the EHR [13, 28, 29]. This dramatically increases study efficiency and facilitates the detection of small but clinically important treatment effects. Countries and health systems with unified EHRs have unique opportunities for multi-center, EHR-based trials [30]. Most research centers, however, are limited by the technical challenges of simultaneously accessing information from multiple, distinct EHR systems. One potential solution is the use of registry-based trials. These leverage pre-existing, prospective patient registries to identify patients and/or collect outcomes while retaining otherwise traditional trial designs [31]. Such registries could also form the foundations for platform trials (discussed below). While there is support for adopting pragmatic methodology, the delivery of complex interventions and the collection of outcomes will be a challenge for many critical care conditions.

Adaptive trials

Adaptive trial designs include plans to modify study design based on the results from trial participants (i.e., interim results). These adaptations can include changes to almost any aspect of trial design including: (i) enrollment criteria (increasing enrollment of subgroups experiencing the most benefit), (ii) sample size (preventing trials from being underpowered and stopping trials early for efficacy, safety, or futility), (iii) randomization (increasing allocation of patients to the most promising trial arm), (iv) interventions (adjusting drug doses or dropping a trial arm), and (v) analysis (converting from a non-inferiority trial to a superiority trial) (Fig. 1) [32].

Adaptive trial designs have the potential to increase efficiency, reduce costs, and increase the likelihood of identifying beneficial therapies [33]. However, there are clear challenges to adaptive trials, in addition to the increased complexity of design, conduct, and analysis. To maintain the validity of trial results, all trial adaptations must be pre-specified with a clear design rationale and an analysis plan that controls for the potential increases to type 1 error [34]. Further, adaptive trials introduce possible operational biases as investigators may learn something about the results of enrolled patients from trial adaptions, even if the analyses are conducted in a blinded fashion. This is a particular problem in critical care research where blinding is not possible for many study interventions (physical therapy, fluid management strategy, paralysis, prone positioning, ECMO), increasing the risk of bias, where operators will observe changes in allocation ratios that modify the way they care for subsequent trial participants. To date, adaptive trial designs have primarily been used for drug development, particularly in the field of oncology. Regulatory bodies are supportive of adaptation because of the potential benefits of minimizing the number of patients exposed to less promising treatments and shortening the time to regulatory approval [35, 36]. Many of these adaptive methodologies are applicable to clinical trials in critical care. Recent examples of adaptive design features in critical care include the RACE trial [37], an adaptive, dose-finding, phase 2 clinical trial of levocarnitine for septic shock, the SEPSIS-ACT trial of selepressin for septic shock [38], and the phase 3 PROSpect trial (NCT03896763). The PROSpect trial plans to randomize pediatric patients with ARDS to supine or prone positioning and low tidal volume or high-frequency ventilation, with adjustments in the enrollment ratios occurring every 100 patients.

Platform trials

Most clinical trials are designed to answer one specific question, but for any given patient population or disease process, there are typically many important clinical questions. Rather than committing resources to build separate trial infrastructures to screen, consent, and follow up separate patient populations for each research question, platform trials attempt to answer multiple study questions simultaneously. This can be achieved by comparing multiple interventions against a common control arm, randomizing multiple interventions for the same patient in unrelated domains, or randomizing the same patient at several different phases of their illness. Recent platform trials have begun to incorporate adaptive features (“adaptive platform trials”), allowing the addition or termination of study arms, and transition between interventions at various stages of disease. A recent example in critical care is the Randomized, Embedded, Multifactorial Adaptive Platform Trial for Community-Acquired Pneumonia (REMAP-CAP). This study randomizes patients with pneumonia to one of five antibiotic strategies to treat pneumonia, with parallel randomizations controlling the decision to give corticosteroids and the decision to give immunomodulatory macrolides (NCT02735707). While there is considerable enthusiasm regarding the potential of adaptive platform trials, these trials share the challenges of all adaptive trials and add the complexity of conducting multiple parallel interim analyses within each domain, while accounting for potential interactions among the treatment domains. Furthermore, the complex designs pose challenges for regulatory bodies and institutional review boards, and raise ethical concerns regarding whether patients can truly understand and provide informed consent for these complicated trial designs [39].

Maximizing information from clinical trials

Clinical trials demand significant investment on the part of patients, funding organizations, and researchers. There is an ethical mandate that the information gathered during clinical trials is used to provide the maximal benefit for patient care. Methods to garner additional information from clinical trials include improved pathways for data sharing, standardization of electronically abstracted data from different institutions, greater use of standardized trial definitions to support comparisons across trials, and the incorporation of pre-specified Bayesian analysis plans to augment the interpretation of trial results.

Increased data access and sharing

There has been considerable debate in recent years regarding publicly sharing de-identified data from individual clinical trial participants. The ethical and scientific arguments for data sharing are sensible and logical. Clinical trial participants have put themselves at risk, and investigators should maximize the knowledge gained from these trials. Further, clinical trial participants and funding agencies overwhelmingly favor the sharing of de-identified data to support additional research [40–42]. Therefore, in 2016, the International Committee of Medical Journal Editors (ICMJE) published a proposal requiring authors submitting clinical trial results to share de-identified individual patient data underlying trial results within 6 months of publication of the primary results [43]. Following the announcement, some researchers expressed concerns that clinical trials would be disincentivized if investigators perceived a reduced opportunity to publish important secondary analyses [44]. Others argued data sharing could produce misleading results, as subsequent researchers would be unfamiliar with the details of how data were collected [44, 45]. Finally, some noted there were significant costs and effort associated with creating de-identified data sets, and some existing data sets resulted in little additional research [46]. In response, the ICMJE retracted their proposal, instead requiring researchers to include a data sharing statement describing the opportunities for data sharing while still advocating for a long-term goal of uniform data sharing. This requirement for a data sharing plan has also been adopted by many funding bodies including the National Institutes of Health. While issues clearly remain regarding the appropriate mechanisms to protect patient privacy, acknowledge the work of clinical trialists, and protect against invalid secondary analyses, 3CT Workshop participants broadly agreed with the importance of data sharing to advance scientific knowledge and improve patient care.

Use of standardized trial definitions

Comparing results across trials requires researchers in different settings use similar definitions. However, many common critical care outcomes lack standard definitions [47–50]. To address this problem, researchers have recently proposed ‘core outcome sets’ [51]. These outcome sets seek a community-wide agreement on the ideal outcome for a given disease state, its definition, and how it should be measured. To date, core outcome sets have been developed for studies evaluating interventions such as invasive mechanical ventilation [52], physical rehabilitation following critical illness [53], and extracorporeal membrane oxygenation [54]. Other core outcomes sets have focused on specific critically ill patient populations, such as the survivors of acute respiratory failure [55]. In addition to standardizing trial outcomes, maximizing the validity of across trial comparisons requires the alignment of enrollment criteria and trial interventions. A successful implementation of this approach was demonstrated by the Protocolized Resuscitation in Sepsis Meta-Analysis (PRISM) study [56], which combined data from three, recent, large clinical trials assessing early, goal-directed therapy for septic shock [57–59]. The authors prevented the biases often seen with meta-analyses by harmonizing enrollment criteria and outcome definitions and publishing a pre-specified analysis plan before pooling and unblinding trial data. Using patient-level data, they were subsequently able to demonstrate that early goal-directed therapy did not improve outcomes across a broad range of patient and care delivery factors.

Integrating Bayesian inference

Traditionally, critical care trials have been designed using a “frequentist” framework with a null and alternative hypothesis. Most trials are designed with a null hypothesis of no difference between groups, which is rejected if a difference in the outcomes between groups is sufficiently different. This design often leads to binary “positive” or “negative” trial interpretations, resulting from the dichotomous interpretation of p value thresholds to determine the existence of “statistically significant” trial results. However, a dichotomous interpretation of a p value is not inherently required (e.g., p value function [60]), and 2019 saw trends across the global scientific community to promote evaluating effect sizes and differences rather than just the use of p value thresholds. This included, for example, a compendium of articles by leading statisticians promoting new methods and views in The American Statistician [61], and a major statement co-signed by several hundred researchers published in Nature [62].

An increasingly popular alternative approach discussed by the 3CT Workshop participants, is the assessment of trial results using Bayesian inference. A key benefit of Bayesian inference is the empirical ability to formally incorporate prior knowledge when evaluating new trials. Specifically, Bayesian trial analysis involves the combining of previously observed or presumed treatment effects (called “prior” probability distributions or functions) with new trial results (called the “likelihood” function) to create a Bayesian effect estimate distribution (i.e., “posterior” probability distribution) [63, 64] (Fig. 2). Interested readers are directed to a much more comprehensive tutorial on the technical aspects and complexities of conducting a Bayesian trial analysis [64].

Fig. 2.

Depiction of a hypothetical Bayesian trial analysis using four different prior probability distributions. To conduct a Bayesian trial analysis, researchers must first select (ideally a priori) “priors,” (shown in red in the figure). Priors are meant to reflect the range of potential effect distributions that are expected before starting a trial. These priors are combined with the observed treatment effect in the trial (referred to as the likelihood function; indicated by the light brown shading). A Bayesian trial analysis typically uses a range of prior distributions, but the likelihood function is constant as it reflects the actual data observed in the trial. Each prior distribution is statistically combined with the likelihood function to create the Bayesian treatment effect estimate (i.e., posterior probability) distribution (outlined in black). As shown, the Bayesian treatment effect estimates may vary considerably based on the selected prior. There are several possible methods for choosing priors including (i) using results of observational studies, trials, or meta-analyses, (ii) eliciting expert opinion, or (iii) using a range of hypothetical distributions that assume very skeptical to enthusiastic effect distributions. The selection of priors is a critically important step in Bayesian trial analysis, and interested readers are directed to a recent technical tutorial [64], and the Bayesian re-analyses of the EOLIA [67] and ANDROMEDA-SHOCK trials [68] for guidance on prior selection and execution of a Bayesian analysis

The differences between frequentist and Bayesian trial interpretations have been highlighted by two recent high-profile studies, the EOLIA [65] and ANDROMEDA-SHOCK trials [66]. Both trials demonstrated large, clinically significant mortality differences but failed to reach p value  < 0.05. Separate post hoc Bayesian re-analyses of these trials, where the potential existence of an effect is quantified on a probabilistic scale from 0 to 100, demonstrated that the trial interventions were likely effective across a broad range of prior distributions [67, 68]. For example, in the re-analysis of the EOLIA trial even the use of a strongly skeptical prior (one which assumed an equal probability of benefit or harm) was associated with an 88% posterior probability that ECMO improved mortality [67]. Bayesian analyses cannot “fix” problems with trial design. In these two examples, highly optimistic absolute treatment-related mortality reductions > 10% were assumed along with higher than observed event rates in the control arm. However, Bayesian analysis can help ensure that trial results are more informative than a coarse “positive” or “negative” delineation, particularly in trials appearing to have clinically significant treatment differences.

There are limitations to Bayesian analyses, just as there are limitations to the popular frequentist framework. Perhaps most challenging is that each ‘prior’ probability is associated with a different Bayesian effect estimate (posterior) distribution, as shown in Fig. 2. Thus, a single interpretation of a trial may not be straightforward. Second, Bayesian effect estimates are heavily determined by the selected prior, which can be manipulated or incorrect, and there is not currently a standard set of agreed-upon priors for researchers to use. However, the impact of a selected prior on the posterior probability diminishes as the sample size of the newer trial increases. Further, there are strong views in the community for and against each framework. In general, the 3CT Workshop participants agreed that Bayesian and frequentist analyses provide different but valuable information and perspectives and investigators should consider planning and reporting both frequentist and Bayesian analyses in future trials. While this promotes a less rigid, and more nuanced interpretation of trial results, this also raises the concern of selective reporting. Accordingly, the 3CT Workshop participants emphasized the importance of pre-specifying whatever analyses trialists will use to ensure the objectivity of trial interpretation.

A need for clinically meaningful, patient-centered non-mortality outcome measures

The primary outcome measure for an RCT is a critical decision as it dominates the interpretation of the trial by regulatory bodies, practicing physicians, and the broader public. The 3CT Workshop participants noted the clear relevance and importance of mortality as a safety outcome measure. However, there was a shared desire expressed, particularly from regulators and patient representatives, to incorporate patient-centered outcomes other than mortality, reflecting patients’ quality of life (i.e., the challenge of surviving critical illness) in future trials.

Mortality as an outcome measure

Mortality is a logical trial outcome and many would argue survival is the most patient-centered outcome. The ideal time horizon for mortality in critical care interventions, however, remains unclear. Short-term mortality (e.g., intensive care unit [ICU], hospital, or 28-day mortality) has long been prioritized, but death is common in the period immediately following critical illness, introducing the potential for discharge bias particularly with open-label interventions [69]. Conversely, longer-term mortality (60-day, 90-day, and 180-day) is susceptible to statistical noise from deaths unrelated to the critical illness, which may increase the number of patients needed to demonstrate a treatment effect. However, there are interventions where differences in treatment effect only become apparent with longer-term follow-up [70]. Even with short-term mortality, a significant portion can be attributed to pre-existing comorbidities and may not be modifiable [4, 71]. Further, decisions to withdraw or withhold life-sustaining treatments are common in the ICU, regardless of the effectiveness of available interventions [72–74]. One potential solution is to use disease-specific modifiable mortality in sample size estimations, but there are no reliable methods to estimate disease-specific modifiable mortality, and these calculations are likely to result in infeasibly large sample size targets [4]. These considerations have, in part, led to increased enthusiasm for non-mortality outcomes.

Non-mortality outcome measures

In recent years, there has been an increased focus on post-ICU quality-of-life, morbidity, and survivorship (e.g., post-intensive care syndrome) [75, 76]. Reductions in the number of days that patients require organ support, are in the ICU, or are in the hospital are attractive to most stakeholders and are associated with reduced long-term morbidity and mortality. A significant hurdle in greater use of these and other non-mortality outcome measures as a primary outcome is the competing risk of mortality [50, 77, 78]. When a trial participant dies, their outcomes after death are either missing (e.g., quality of life, daily organ failure, or biomarker measures) or truncated (e.g., length of stay) in a clinically relevant manner that cannot be ignored when assessing the impact of an intervention. There are effectively two general solutions available to researchers. These are (i) the use of a composite end point that combines mortality and the non-mortality outcome measure, or (ii) the analysis of the non-mortality outcome measure using an advanced statistical modeling approach to account for death [50, 78–83]. Historically, composite outcomes, such as event-free day composite measures (e.g., ventilator-free days, organ failure-free days), have been preferred by the critical care community [84, 85]. However, composite outcomes can be difficult to interpret clinically, or in other terms relevant to patients and policy makers [86–88]. There are several promising ongoing activities to develop new composite outcome measures (e.g., hierarchical end points [89]) and methods of analysis that improve upon these event-free day metrics [50, 77, 78, 90, 91]. As short-term priorities, the 3CT Workshop participants agreed that non-mortality outcomes are essential for critical care intervention assessment, but acknowledged these outcomes are more difficult to collect and analyze (particularly in the longer term). The 3CT Workshop participants also noted that establishing standardized non-mortality outcomes in a way that facilitates across-trial comparisons is a clear priority of future critical care research.

Conclusions

Significant progress has been made in the understanding of and care for patients with critical illness, but the epidemiology of critical illness is changing. Continuing advancements in critical care will require new therapies and corresponding advances in critical care trial methodology, with an emphasis on improved sample size calculations, strategic leveraging of novel trial designs, and standardization of patient-centered and patient-informed outcomes beyond mortality.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Author contributions

All authors contributed to the discussions in the meeting, selection of the content for the manuscript, and the writing of the manuscript. The first draft of the manuscript was written by MOH and JDC, and all authors reviewed, edited, and commented on several previous versions of the manuscript. All authors read and approved the final manuscript.

Funding

MOH was supported by the United States National Institutes of Health (NIH)/National Heart, Lung, and Blood Institute (NHLBI) grant K99 HL141678 and R00 HL141678. JDC was supported in part by the NIH/NHLBI (T32HL087738-12, K12HL133117). MAM was supported in part by NHLBI U01 HL 123004, HL140026, HL134828. MNG was supported by NHLBI UH2 HL125119, U01 HL122998, AHRQ R18 HS026188. TWR was supported in part by NHLBI U01 HL123009, NCRR U54 RR 032646, NCATS U24 TR 001608. WHS was supported in part by NHLBI U01HL123009 and NIGMS K23GM110469. JCM was supported in part by the Canadian Institutes of Health Research MYG158584.

Compliance with ethical standards

Conflicts of interest

Dr. Jaber reports receiving consulting fees from Drager, Fisher & Paykel, Medtronic, Baxter, and Fresenius-Xenios. Dr. Laterre reports personal fees from Adrenomed, Ferring, and Inotrem. Dr. Mebazaa reports personal fees from Novartis, Orion, Roche, Servier, Sanofi, Otsuka, Philips, grants and personal fees from Adrenomed, Abbott, and grants from 4TEEN4. On behalf of all authors, the corresponding author states that there are no additional potential conflicts of interest.