The limitations of evidence from randomized trials

Endotracheal intubation is often necessary for the management of severe respiratory distress caused by heart failure or acute respiratory failure (ARF). Before intubation, physicians may be tempted to use noninvasive positive-pressure ventilation (NIPPV). Since 1995, no fewer than 5 randomized clinical trials (RCTs) have shown that, compared with the simple administration of oxygen, NIPPV results in improvement in blood gas values, less likelihood that patients will subsequently need to undergo intubation and improved survival.^1,2,3,4,5 But can we simply step from an RCT onto a typical hospital ward in the real world and expect to achieve similar results? The study by Tasnim Sinuff and colleagues⁶ reported in this issue (page 969) suggests that we need to be careful.

Sinuff and colleagues reviewed the records of consecutive patients treated with NIPPV over 15 months in a single teaching hospital. They found that the death rate both for their patients with chronic obstructive pulmonary disease (COPD) and for those with congestive heart failure was higher than the pooled rate from a meta-analysis of randomized trials. Moreover, the death rates in their study were higher than the pooled rates for the control groups in the randomized trials.

Why did this happen? As the authors point out, one reason is that, in contrast to the patients in the RCTs, the patients who received NIPPV in a typical clinical setting were less intensively monitored, chest radiographs and blood gas levels were not always obtained, the experience of the physicians and other health care workers was varied (and was usually considerably inferior to that of the personnel in the RCTs), and NIPPV was administered mainly in locations other than the intensive care unit.

It may seem at first that these differences in the application of NIPPV might account for the differences in outcome. Indeed, the authors plan to develop, implement and evaluate a practice guideline for the optimal use of NIPPV in ARF. This is a worthwhile objective, but to my mind, as the authors also discuss, the most important reason for the difference between their results and those of the randomized trials is the patient population.

For the assessment of a new technique such as NIPPV, one must first know whether it has any immediate effect. In the case of NIPPV, this means determining whether it lowers the arterial partial pressure of carbon dioxide, raises the pH (since a low pH value is associated with a poor outcome), increases the arterial partial pressure of oxygen and decreases the respiratory rate. The data are clear for NIPPV: it does have an important positive effect on gas exchange and respiratory rate but, like any other therapy, not in all patients. It has been shown to be particularly effective for patients with heart failure,² but it appears to be less effective for those with ARF.

It is also important to know the proportion of patients in whom the technique can be used. In the study by Brochard and coworkers,⁴ 69% of patients admitted with respiratory failure were excluded from the trial. In the study by Antonelli and colleagues,¹ 13 of 77 patients chose not to participate, those with COPD were not included, and an additional 295 patients could not be included because they had already undergone intubation. Wysocki and coworkers⁵ excluded 67% of patients, including those with COPD. In contrast, Bott and colleagues³ studied only patients with COPD.

A further important clinical question concerns whether NIPPV makes intubation unnecessary. NIPPV may make this unnecessary by supporting ventilation and decreasing the respiratory rate, which leads to a less dysfunctional and inefficient pattern of breathing. The randomized trials were set up largely to address this question. In patients with heart failure the answer seems to be clearly yes,² most probably because heart failure is a readily reversible process and NIPPV buys time for the other therapies to work, and they can work quite fast. The evidence also supports the role of NIPPV in respiratory failure, although it is not clear that all intubation procedures in patients in the control groups were necessary. For example, in the study by Brochard and associates,⁴ only 31% of patients in the control (usual care) group met the study criteria for intubation, whereas 73% of those in the NIPPV group met these criteria. The exclusion of large numbers of patients from the trials and the possibility that there are important physician-related factors in these unblinded studies that influence outcomes are both arguments for caution in anticipating that the success seen in RCTs can be reproduced in usual hospital practice.

In order to come up with a clear answer, especially on end points such as survival, randomized trials exclude patients with poor outcomes and also must exclude patients who do not want to undergo intubation, since this is an important end point. Sinuff and colleagues found that these patients constituted a large part of their population. Therefore, the only information from the previous trials that is useful for determining the benefit of NIPPV in such patients is whether it has any immediate effect. The major question presented in the randomized trials, namely, whether NIPPV makes intubation unnecessary, is not really that useful or important in the usual clinical setting, because intubation is affected by so many variables and patient needs. It has been suggested that intubation is associated with a higher complication rate than NIPPV,¹ however, this result is heavily biased by patient selection and other management issues. For example, over half of Antonelli and coworkers‚ patients had nasal intubation,¹ which is associated with a high rate of sinus infection. What we can learn from the trials is that there seems to be no harm in trying noninvasive ventilation first in many patients.

The specific question as to whether the therapy is working or not can easily be analyzed in individual patients by establishing clear clinical goals and recording the outcome. However, this requires monitoring and establishing end points, which was not always done in the population studied by Sinuff and colleagues. This should be the basis of future studies, and it would be worthwhile to study this in a prospective cohort analysis, with careful records of patients in whom the treatment could not be used or failed to produce the desired results. This type of study would potentially provide far more useful clinical data than are currently available in the randomized trials.

Finally, Sinuff and colleagues‚ results tell us that we can become complacent about the use of new technology. There needs to be a constant program of maintenance of competence among physicians, nurses and respiratory therapists. This can be made easier by having this type of therapy applied in specialized areas with supervising teams. Guidelines concerning its use are helpful, but they should probably be aimed more at determining the effectiveness of the treatment, so that treatment can be stopped if it does not produce the desired therapeutic goals. For these purposes, randomized trials are only a start and are not the only evidence that should determine our practice patterns and guidelines.