Challenging orthodoxy in critical care trial design: physiological responsiveness

Research involving critically ill adults poses unique challenges in addition to the usual difficulties involved in conducting quality, replicable scientific research. The critical care research community has responded admirably to these challenges by rigorously conducting numerous large and often multicenter randomized controlled trials (RCTs) of putative therapies for critical illnesses. Yet in spite of two decades of work, few incontrovertibly efficacious therapies have resulted from this herculean effort (1). The reasons for this are unsettled, but several categories of problems have emerged. The first problem is that of “positive” trials that cannot be replicated (2-15). Since the clinical trial is in essence a diagnostic test of a hypothesis (16), these non-replicable studies represent “false positives.” False positive trials are due to type I errors which are increased by selection of a conventional and lax statistical significance threshold (e.g., α=0.05) (17,18), bias in the study at any stage of design, conduct, analysis and reporting (19), and fraud (20). Recently, the center for open science collaboration demonstrated that the majority of 100 “positive” psychological research studies could not be replicated, suggesting a false positive rate of 63% in that field (21). The generalizability of this result to medicine is uncertain, but the problems of non-replicability and false positives are not.

The second problem plaguing critical care research is a spate of negative trials of what were thought to be promising therapies. One possible explanation is that these negative trials represent “true negatives” and the trialed therapies do not work, for myriad reasons: unknown or redundant causal pathways to the outcome of interest (22), multiplicity of effects of the active treatment (pleiotropic and “off-target” effects), time dependency of causal pathways (23), etc. A second possibility is that some of the therapies are efficacious but for an outcome that was not assigned as the primary outcome (24) or was not measured at all. A third possibility is that the negative trials represent “false negatives” (16). False negative trials can result from inadequate assigned study power (25), from subversion of power calculations by delta inflation (use of an overly optimistic effect size in sample size calculations) (26), from inadequate dosing of active treatment causing failure of separation (27), or from dilution of effective sample size by patients unlikely to benefit because of severity of illness (too high or too low) (28,29) or because of heterogeneity introduced by non-specific disease definitions (30-33).

In a recent article (34), Goligher et al. propose one possible solution to the specific problem of false negative trials due to dilution of effective sample sizes. They reason that, prior to enrollment in a trial testing a physiological intervention for ARDS such as different doses of PEEP, a “test dose” of PEEP could be applied to prospective enrollees to determine PEEP responsiveness which would be an inclusion criterion for enrolment in the trial. By excluding patients who do not respond to PEEP with an increase in P/F ratio by a pre-specified margin, this strategy may exclude patients such as those with milder lung injury who cannot benefit from PEEP or those with severe disease and little recruitable lung who may be harmed by it (35). If this reasoning is correct, many fewer patients would need to be screened and enrolled to satisfy sample size requirements for such a trial.

The authors of this article are to be commended for scrutinizing the orthodoxy of contemporary trial design in light of its frequent failings and for proposing a possible solution. As the authors point out, for their strategy to work, the effect of PEEP on the outcome measure for the test dose (P/F ratio increase) must be an accurate predictor of the effect of PEEP on the primary outcome of the trial. That is to say, this strategy involves multiple bets (as do all trials in the assignment of enrollment criteria). If the predictive validity of the screening outcome for the primary outcome is imperfect, we will have excluded otherwise eligible patients with this procedure. Unanswered questions include whether informed consent will be required prior to test dose administration, and what other diseases besides ARDS will lend themselves to screening with physiological responsiveness. For example, in a trial of vasopressors for shock, would we be content to administer a test dose of levophed and exclude from enrolment patients who did not have an increase in mean arterial pressure of a given amount? Is it possible that physiological responsiveness is dynamic and that failure to respond at one time does not predict failure to respond over the course of the illness?

While screening with physiological responsiveness is seductive for its potential to reduce sample size requirements, it ignores and potentially perpetuates larger problems in our current paradigms for evidence generation in critical care medicine. The exclusion of patients unlikely to benefit provides justification for using a larger delta value in power calculations. As there is no precedent in modern critical care for a reduction in short term proportional absolute mortality of 8–11% as the authors propose in Table 1, these values are likely to be overly optimistic and thus to represent delta inflation (26). Indeed, an emerging trend in critical care trials is to enroll not fewer but more patients, increasing the statistical precision of the results which better allows clinicians to exclude clinically meaningful effects outside the resulting 95% confidence intervals (36,37). The use of mortality as a universal primary endpoint for two decades without any consistent success in RCTs should lead us to reevaluate the suitability of this metric for the achievement of our goals. Other suggested measures include QALYs and composite outcomes (24,38) that include chronic encumbrances of critical illness such as artificial nutrition, supplemental oxygen, renal replacement therapy, and need for devices to assist with walking as components. Measurement of patient-centered outcomes such as these may inform not only our choices of outcomes in future trials if we see “signal” in an individual outcome, but will also help us refocus our attention on survivors of critical illness.

We have learned much in the modern era of critical care research. Our negative results should not discourage us, especially in light of data that outcomes in practice are improving in spite of them (39). Progress and future successes will depend not only on persistence and perseverance, but also on our willingness to challenge existing paradigms and dogma. Goligher et al. have taken us one step farther in the direction of progress.