Powering bias and clinically important treatment effects in randomized trials of critical illness

Abstract

Objective:

Recurring issues in clinical trial design may bias results toward the null, yielding findings inconclusive for treatment effects. This study evaluated for powering bias among high-impact critical care trials and the associated risk of masking clinically important treatment effects.

Design, Settings, and Patients:

Secondary analysis of multicenter randomized trials of critically ill adults in which mortality was the main endpoint. Trials were eligible for inclusion if published between 2008–2018 in leading journals. Analyses evaluated for accuracy of estimated control group mortality, adaptive sample size strategy, plausibility of predicted treatment effect, and results relative to the minimal clinically important difference. The main outcome was the mortality risk difference at the study-specific follow-up interval.

Interventions:

None.

Measurements and Main Results:

Of 101 included trials, 12 met statistical significance for their main endpoint, five for increased intervention-associated mortality. Most trials (77.3%) overestimated control group mortality in power calculations (observed minus predicted difference: −6.7% ± 9.8%; p<0.01). Due to this misestimation of control group mortality, in 14 trials the intervention would have had to prevent at least half of all deaths to achieve the hypothesized treatment effect. Seven trials prespecified adaptive sample size strategies that might have mitigated this issue. The observed risk difference for mortality fell within 5% of predicted in 20 trials, of which 16 did not reach statistical significance. Half (47.0%) of trials were powered for an absolute risk reduction ≥10%, but this effect size was observed in only three trials with a statistically significant treatment benefit. Most trials (67.3%) could not exclude clinically important treatment benefit or harm.

Conclusions:

The design of most high-impact critical care trials biases results toward the null by overestimating control group mortality and powering for unrealistic treatment effects. Clinically important treatment effects often cannot be excluded.

INTRODUCTION

Lack of statistical significance of clinical trial results may not exclude clinically important findings (1–3). “Negative” randomized controlled trials (RCTs)—those that do not demonstrate a statistically significant treatment benefit—may indeed reflect an ineffective treatment on average for the study population in some instances. Yet, “negative” RCTs also may be the product of design choices that mask important treatment effects within the population of study.

RCTs examining treatments for critically ill patients afford an ideal case study for design challenges. Phenotypic heterogeneity (4–7), practice variation (8, 9), and secular trends in mortality from critical illness (10) may contribute to inaccurate prediction of control group mortality and treatment effect size, which in turn threaten statistical power. The complexity of critical illnesses and ubiquity of competing risks often preclude adoption of endpoints other than all-cause mortality. Trials may power for overly optimistic treatment effects that do not reflect smaller but clinically important differences or expend substantial resources to include many participating centers and accrue large sample sizes within a reasonable time. Neither approach is ideal.

This study evaluated multicenter critical care RCTs published in leading medical journals to determine recurring power and sample size estimation issues that contribute to “negative” trial results, and to assess whether such trials reliably exclude clinically important differences in survival. The main hypothesis was that two recurring issues—control group mortality misestimation and overly optimistic predicted treatment effects—bias trial results toward the null. This study also evaluated how often trial results fail to exclude potentially clinically important treatment effects.

METHODS

Eligibility Criteria for Trials

Studies eligible for inclusion were multicenter RCTs of critically ill adults in which mortality was the main endpoint. For inclusion, the publication must have appeared between January 2008 and December 2018 in one of seven high-impact general medical or specialty journals: New England Journal of Medicine, JAMA, Lancet, American Journal of Respiratory and Critical Care Medicine, Lancet Respiratory Medicine, Intensive Care Medicine, or Critical Care Medicine. Journals were selected by considering impact factor and content relevance in effort to identify trials most influential to future trial design and clinical practice. Trials were excluded if not designed as superiority trials or if patient-level randomization was not employed.

Data Extraction

Tables of contents for each journal issue were screened independently by two study physicians and a PubMed search conducted to identify articles for inclusion (Supplement). Results from all search strategies were combined with discordance resolved by an independent third reviewer to create the final study list.

Data were extracted in duplicate, blinded fashion. Discordant entries were resolved by an independent third reviewer. Data sources included the main trial publication, accompanying online data and protocol supplements, published trial protocols, and online trial registration sites.

Main Outcome

The main outcome for this study was the unadjusted risk difference in mortality (treatment group minus control group) at each study-specific interval for primary follow-up.

Determination of Minimal Clinically Important Difference

The minimal clinically important difference (MCID) was defined from the clinician’s perspective as the smallest treatment effect required to change routine clinical practice for a comparable patient (11). A 5% absolute risk difference and 20% relative risk difference in mortality were selected a priori as the MCID based on existing literature (12–17) and concern that smaller differences might be viewed as unlikely to change clinical practice. Other MCID thresholds were considered in secondary analyses (Supplement).

Statistical Analysis

Predicted control group mortality and predicted risk difference were obtained directly from each trial’s reported sample size calculation. Observed risk difference was calculated using observed enrollment and mortality extracted from each trial; corresponding 95% confidence intervals (CI) were computed from extracted data.

A paired t-test was used to determine whether the average difference in observed versus predicted control group mortality among trials was significantly different from zero, with observed and expected mortality handled as paired observations for each trial.

To evaluate impact of control group mortality misestimation on statistical power, the predicted absolute risk reduction was assessed as a fraction of the true observed control group risk of death. The association between control group mortality and statistical power was evaluated graphically for representative sample sizes and relative risk.

Conclusiveness of trial results for important treatment effects was evaluated according to the approach of Kaul and Diamond (18). Trials were evaluated for whether the risk difference 95% CI included the MCID. Trials also were evaluated for whether the risk difference 95% CI included the trial’s own predicted treatment effect. Bayesian models were developed via the Markov Chain Monte Carlo method with noninformative priors, 1000 burn-in iterations, 50,000 iterations and thinning rate of five. Model results were used to calculate the posterior probability of observing the proposed MCID on the absolute and relative risk scales. Posterior probabilities for treatment effect of each individual trial were reported. Other treatment effect thresholds for clinical importance were considered in secondary analyses (Supplement).

Frequentist hypothesis testing applied a two-sided alpha threshold of 0.05 without adjustment for multiple comparisons. Analyses were conducted using SAS 9.4 and PASS 14.

RESULTS

Characteristics of Included Trials

Of 657 unique publications identified on initial screen, 101 multicenter superiority trials with patient-level randomization and mortality as the main endpoint were included (Table 1, Figure S1). A complete list of included trials is provided in the online supplement. Twelve trials (11.9%) met the statistical significance threshold for the primary endpoint, five of which demonstrated increased mortality with the intervention.

Table 1.

Characteristics of included trials

Characteristic	Value (n = 101)
Journal, no. (%)
New England Journal of Medicine	48 (47.5%)
JAMA	18 (17.8%)
Critical Care Medicine	13 (12.9%)
American Journal of Respiratory and Critical Care Medicine	6 (5.9%)
Intensive Care Medicine	6 (5.9%)
Lancet	6 (5.9%)
Lancet Respiratory Medicine	4 (4.0%)
Main source of funding, no. (%)1
Government outside US	54 (53.5%)
Industry	23 (22.8%)
Non-profit organization	11 (10.9%)
US government	8 (7.9%)
Local institution	5 (5.0%)
Geographic region(s) in which trial conducted, no. (%)
Europe	75 (74.3%)
US	27 (26.7%)
Canada	23 (22.8%)
Australia and/or New Zealand	19 (18.8%)
Asia	22 (21.8%)
Central or South America	10 (9.9%)
Africa	6 (5.9%)
Topic of trial, no. (%)
Sepsis or infection	34 (33.7%)
Respiratory	21 (20.8%)
Cardiovascular	19 (18.8%)
General critical care	14 (13.9%)
Renal	5 (5.0%)
Trauma	4 (4.0%)
Other2	4 (4.0%)
Trial design, no. (%)
Two-arm parallel group	84 (83.2%)
Three-arm parallel group3	8 (7.9%)
Factorial 2×24	9 (8.9%)
Trial stopped early, no. (%)5	35 (34.7%)
Sample size in final analysis, median (IQR)	843 (411–1588)
Main endpoint, no. (%)
≤ 7-day mortality	3 (3.0%)
28-day or 30-day mortality	63 (62.4%)
60-day or 90-day mortality	23 (22.8%)
≥ 180-day mortality	3 (3.0%)
In-hospital mortality without specified interval	9 (8.9%)
P-value ≤ 0·05 for primary endpoint, no. (%)	12 (11.9%)
Significant benefit with intervention	7 (6.9%)
Significant harm with intervention	5 (5.0%)

^1.For studies with multiple funding sources, the primary funding source was considered governmental whenever the government was a trial sponsor regardless of other funding sources reported.

^2.Other trial topics included hepatic (3) and toxic ingestion (1).

^3.For all analyses, the five three-arm parallel group trials were handled as two-arm trials. One trial merged two groups in the main analysis. Two trials stopped enrollment in one arm early and reported as the main result a two-group comparison. Two trials explicitly stated the a priori main analysis consisted of two-group comparison without consideration for the third study arm.

^4.Eight trials of factorial 2×2 design, reported in four publications, specified pairwise comparison of both factors in the main analysis. One trial of factorial 2×2 design specified comparing only one factor in the main analysis.

^5.Reasons for trials stopping early included futility (n = 17), harm (n = 10), poor enrollment (n = 6), sponsor termination (n = 1), and drug/device withdrawal from market (n = 1).

Sample Size

The median (IQR) sample size for analysis of the main endpoint was 843 (411–1588) patients. The smallest sample size was 62 patients, and the largest was 20,127 patients. Seven trial protocols (6.9%) incorporated an adaptive sample size design that permitted increasing enrollment targets if prespecified criteria were met (19–24).

Control Group Mortality

Reported power calculations contained sufficient data to ascertain predicted control group mortality in 97 of 101 trials. Median (IQR) predicted control group mortality was 40% (30–45%).

Power calculations significantly overestimated control group mortality (observed: 33.9 ± 18.0%; predicted: 40.6 ± 17.5%; mean difference −6.7 ± 9.8%; p < 0.01). Seventy-five trials (77.3%) overestimated control group mortality, whereas only 22 trials (22.7%) underestimated control group mortality.

The distribution of control group mortality misestimation is displayed graphically in Figure 1.

Figure 1. Waterfall plot of control group mortality misestimation.

Absolute difference in control group mortality, observed minus expected. Each bar represents an individual trial.

Predicted Risk Difference

On the absolute risk scale, trials were powered to detect a median (IQR) mortality risk difference of 9.0% (6.0–12.0%), corresponding to an NNT of 11 (8–17) patients. Only nineteen trials (19.0%) were powered to detect a risk difference of 5% or less (NNT = 20) (Figure 2). Large predicted effect sizes were common across all included journals and disease areas (Figures S2–S3).

Figure 2. Butterfly plot of predicted and observed mortality risk difference.

Data are sorted by sample size within each study type. One trial (sample size 1,439 patients) contained insufficient information in its reported power and sample size determination to identify predicted risk difference.

On the relative risk scale, trials were powered to detect a median (IQR) relative risk reduction of 23% (20–30%). The hypothesized relative risk reduction was at least one-third of all deaths in nineteen trials and at least half of all deaths in five trials.

Influence of Control Mortality Misestimation on Predicted Treatment Effect

The hypothesized absolute risk reduction in mortality, as a percent of the true observed baseline (control group) risk in the trial, was median (IQR) 28.7% (20.1–42.5%). To achieve the hypothesized absolute risk reduction, in 14 trials the intervention would have had to prevent at least half of all deaths given the observed control group mortality, an exceedingly unlikely treatment effect. In two trials, the hypothesized absolute risk reduction could not have been achieved even if the intervention prevented all deaths.

The relationship between control group mortality, risk ratio, sample size, and statistical power is shown in Figure 3.

Figure 3. Impact of control group mortality misestimation, hypothesized risk ratio, and sample size on statistical power.

For a given sample size and risk ratio (also known as relative risk), power decreases with lower control group mortality. Overestimation of control group mortality in sample size calculations decreases power (1 – ß) and therefore increases probability of a “false negative” trial result (ß). The impact of lower control group mortality on diminishing power is exacerbated by smaller sample size.

Observed Risk Difference

The median (IQR) observed risk difference (treatment minus control) in mortality was −0.2% (−1.7% to 2.0%), and ranged from −16.7% to 11.7% (Figure 2).

Among trials observing a statistically significant treatment benefit, the median (IQR) observed risk difference was −9.1% (−13.1% to −1.5%), corresponding to an NNT of 11 (8 to 68) patients to prevent one death.

In only five trials (5.0% of included trials) was an absolute risk reduction of 10% or greater observed (NNT ≤ 10) (25–29), and two of these trials did not reach statistical significance (26, 27). The sample size for four of the five trials with this large observed treatment benefit fell within the smallest third of included trials, with corresponding wide confidence intervals, raising doubts about the large magnitude of treatment effect (Figure 4).

Figure 4. Trial results according to clinically important difference in mortality on absolute and relative scales.

Several trials that did not achieve statistical significance in conventional frequentist analysis nevertheless failed to exclude clinically important benefit or harm for the intervention studied. The prespecified threshold used for MCID was a 5% absolute risk difference (number needed to treat = 20) or 20% relative risk difference (risk ratio ≤ 0.8 or ≥ 1.2) for either benefit or harm with treatment. Thresholds are indicated by the shaded areas: blue for benefit and red for harm. Forest plots indicate the effect estimate and 95% confidence interval for absolute risk difference (left) and risk ratio (right) for each trial. The size of each square corresponds to the trial sample size relative to other included trials. The probability of a clinically important treatment effect (benefit or harm), given the trial results, was calculated for both the absolute risk difference and risk ratio using Bayesian statistics. MCID = minimal clinically important difference. * denotes statistical significance according to the trial’s main analysis.

Clinically Important Difference in Treatment Effect

The observed risk difference for mortality fell within ± 5% of the predicted treatment effect in 20 of 100 trials with requisite data. Yet, 16 of these 20 trials did not achieve statistical significance for the primary endpoint (Figure 2). Larger sample size was strongly correlated with a smaller difference in observed versus hypothesized treatment effect (Pearson r = −0.42, 95% CI −0.57 to −0.24; p < 0.01).

“No benefit” trials:

In 21 of the 94 trials (22.3%) without a statistically significant benefit, the 95% CI for risk difference included the effect size the trial was designed to detect. In 44 of these trials (46.8%), the 95% CI included a 5% absolute risk reduction in death (Figure 4), the a priori MCID selected for this analysis.

“No harm” trials:

In 49 of the 96 trials (51.0%) without a statistically significant worsening of mortality with the intervention, the 95% CI included a 5% absolute risk increase in death, the MCID (Figure 4). Overlap with other risk difference thresholds for benefit and harm are presented in the Supplement (Tables S6–S8).

Inconclusive frequentist results for important treatment effects:

Most trials (68 of 101 included trials; 67.3%) failed to exclude a clinically important treatment benefit or harm, assessed via the treatment effect estimate 95% CI crossing the 5% absolute risk difference threshold for either benefit or harm without achieving statistical significance.

Inconclusive Bayesian results for important treatment effects:

The intervention of study was more probable than not (posterior probability > 0.5) to decrease risk of death by at least the 5% MCID (treatment benefit) in eleven trials, six of which did not reach statistical significance (Figure 4). The intervention of study was more probable than not to increase risk of death by at least the 5% MCID (treatment harm) in eleven trials, seven of which did not reach statistical significance. Results grouped by journal and disease area are presented in Figures S4–S5. Results for other treatment effect thresholds and the posterior probability calculated for each trial, as both absolute risk difference and risk ratio, are presented in Figure 4 and Figures S6–S9.

DISCUSSION

The main findings of this study are that multicenter RCTs of critical illness routinely overestimate control group mortality, power for overly optimistic treatment effects, and fail to provide conclusive results for clinically important differences in survival. These issues collectively may be referred to as powering bias.

Relevance for Clinicians

These findings have important implications for clinicians seeking to practice evidence-based medicine. Particularly in critical care, widespread adoption of a given treatment often hinges on results of one or two trials (30–33). When one large-scale trial does not find a statistically significant difference for main endpoints, the common interpretation is often conclusive, that the therapy is ineffective, thereby dissuading use in clinical practice. If trial results confidently exclude the MCID and other sources of bias are not identified, this interpretation may be reasonable. However, the present study indicates most multicenter critical care trials power for improbably large effect sizes, an issue compounded by misestimation of control group mortality, biasing results toward the null regardless of true treatment effect. In such trials, the correct interpretation may be that data are inconclusive due to powering bias.

Learning to assess for powering bias therefore is an essential skill for the astute clinician weighing evidence to guide management decisions. Classic teaching on how to assess trials for bias (34, 35) does not routinely include these potential sources, related to power and sample size calculations, which occur often and may bias results toward the null.

The present study brings to light two simple questions to ask when reading a primary trial report to ascertain the threat of powering bias. One, did the control group mortality predicted in the sample size calculation overestimate the actual control group mortality observed in the trial? Two, did the treatment effect size for which the trial was powered fail to include what the reader deems a minimal clinically important difference that would change his/her practice? If the answer to either of these questions is yes, then powering bias may be present. Particularly when the treatment effect 95% confidence interval includes the MCID, a “negative” trial might better be described as inconclusive. Regardless, interpretation and implications for clinical practice always should be weighed in context of the totality of trial results (beneficial and harmful effects), existing literature, and current understanding of pathophysiology.

Relevance for Trialists

The present study also highlights several issues to consider in trial planning. Phenotypic (4, 6, 36) and practice heterogeneity (37) inherent in critical care are not easily mitigated even with rigorous trial eligibility criteria and protocol standardization of care. Coupled with secular trends in mortality (10), such heterogeneity likely accounts for the systematic overestimation of control group mortality found here.

Powering for overly optimistic treatment effects is common in critical care trials. Nearly half (47%) of trials in this study powered for an absolute mortality reduction of 10% or greater; yet, this treatment effect was seen in only five trials, four of which had modest sample sizes (114–466 patients) with wide 95% CIs, suggesting the true effect might be smaller. Thus, trials designed to detect a mortality risk difference exceeding 10% may bias toward a “negative” result and should require compelling justification to go forward.

Several factors may contribute to improbably large predicted treatment effects, including: (i) need to balance sample size against budget constraints and trial feasibility; (ii) publication bias in existing literature; and (iii) large effect sizes in early-phase trials, when observed, may tend to overestimate treatment effects but also are likeliest to garner support for advancing to large-scale trials (38–43). Because not all factors reflect overly optimistic views of treatment effect per se (“optimism bias” (38)), the term “delta inflation” has been proposed to refer more broadly to improbably large predicted effect estimates regardless of rationale (13).

Targeting large sample sizes to detect risk differences approaching the MCID often requires adding trial sites or extending the enrollment period, which may introduce additional phenotypic and practice heterogeneity to the trial. With large sample size, costs may become prohibitive or reduce the number of studies that can be supported by a given funding body.

All-cause mortality remains the most widely accepted critical care endpoint for phase III trials in part because validated disease-specific or intermediate surrogate endpoints are lacking for most critical illnesses (44). Yet, mortality may not always be the appropriate main endpoint (45), particularly when lower baseline risk of death is anticipated, an issue best evidenced by considering relative risk reduction. For example, a trial powered to detect a 5% absolute risk reduction with control group mortality of 15% would require that the treatment prevent one-third of all deaths to achieve statistical significance, an improbable treatment effect observed in only three trials in this study. Powering for an absolute risk reduction less than 5% to maintain the predicted relative risk reduction within a plausible range may be appropriate in select instances. Simply enrolling patients with higher risk of death may not improve trial performance if doing so dilutes disease- and target-specific attributable risk, i.e. the proportion of deaths due to the particular disease of interest and mechanism being targeted (46–48).

Addressing Powering Bias in Trial Design and Reporting

Two underutilized strategies for mitigating powering bias are adaptive trial design and MCID-based reporting.

Adaptive sample size strategies can overcome misestimation of control group mortality (49) but were employed by only 6.9% of multicenter trials in this analysis. Adaptive trial designs of any kind require flexibility from funding and regulatory agencies to account for and respond to increases in sample size, and due care must be taken to avoid introducing bias (50, 51).

Trial analysis plans should include assessment of clinical importance relative to a prespecified MCID to facilitate interpretation for clinical practice. We favor a two-pronged strategy (Figure 4) (18). First, report where the effect estimate 95% CI falls relative to the MCID (52, 53). If the 95% CI includes the MCID, clinical importance cannot be excluded, and the trial may be considered inconclusive (54). Second, calculate the Bayesian posterior probability of treatment effect greater than or equal to the MCID (55). By assuming a noninformative prior, the computed posterior probability is effectively determined by trial data alone, an approach more familiar to readers than specifying various informative Bayesian priors. The resultant statistic is intuitive for clinical practice: the probability of a clinically important treatment effect based on trial results. The MCID should be prespecified in the trial protocol, and both benefit and harm should be evaluated (56). Incorporating MCID into trial design may help establish benchmarks to persuade funding agencies to promote smarter and more clinically meaningful trials (57).

Limitations

This study is not the first to identify issues around powering bias (12, 13, 58). However, it does confirm its persistence in recent critical care clinical trials. This study builds on existing literature in part by evaluating the problem according to an MCID and by using Bayesian statistics to quantify the posterior probability of achieving the MCID. Importantly, it also reframes the issue of powering bias as an essential consideration not just for trialists but also for clinicians to weigh in interpreting the evidence.

This study included only multicenter RCTs published in seven high-impact journals, criteria intended to select trials most influential to scientific direction, future trial design, and clinical practice. Whether powering bias is similarly widespread in smaller trials published in lower-impact journals could be important because such trials often have outsized influence in meta-analyses. To facilitate comparison between trials, only trials with mortality as the main endpoint were included. This study did not restrict eligible trials to intensive care unit-based interventions only but instead included the more broadly defined population of critically ill adults, potentially increasing heterogeneity among included trials.

Other strategies for improving trial design, including prognostic and predictive enrichment, were not explored in this analysis.

This analysis prespecified a 5% absolute risk difference and 20% relative risk difference as the MCID for all studies, but MCID may depend on the risk profile of the associated intervention and extent of long-term morbidity among survivors. MCID is a difficult concept to apply to mortality when characterized from the individual patient’s perspective. Instead, it was defined for this analysis using a clinical perspective, as an effect size sufficient to change clinical practice, an approach similar to that proposed by professional societies for use in oncology trials (57). To our knowledge, no validated mortality MCID exists for the critical illnesses studied.

The Bayesian posterior probability of achieving the MCID was modeled using a noninformative prior distribution so that the posterior probability would be determined entirely by study data as in a frequentist analysis; however, in practice a trial reporting Bayesian results might incorporate informative prior distributions based on preliminary data.

CONCLUSION

Most multicenter RCTs of critical illness bias toward the null by overestimating control group mortality and powering for overly optimistic treatment effects. As a result of this powering bias, trials rarely provide conclusive results on smaller but clinically important treatment effects. Including assessment of trial results relative to the MCID would greatly improve clinical interpretation. Incorporating adaptive designs in future trials may mitigate the impact of misestimation in power and sample size calculations.