Caution! Overestimation of treatment effects of corticosteroid therapy for community-acquired pneumonia in a meta-analysis of randomized controlled trials

Abstract

It is well known that a meta-analysis of randomized controlled trials aims to increase the power and precision of the estimated intervention effects. However, when a meta-analysis includes a limited number of patients and a small number of events, overestimation of intervention effect estimates may occur and could cause spurious results. Although many biases can cause the overestimation, random error may be the most common cause. Trial sequential analysis (TSA) can explore the independent effect of random error on intervention effect estimates in meta-analyses and protect meta-analyses against overestimation due to random error.

Meta-analyses aim to increase the power and precision of the estimated intervention effects. Meta-analyses of high-quality randomized clinical trials (RCTs) are generally considered the highest level of evidence for intervention effects (1). However, when the relevant evidence is limited, meta-analyses are often underpowerd to establish realistic intervention effect estimates. Of particular note is that when meta-analyses include a limited number of patients and a small number of events, overestimation of intervention effect estimates may occur and could cause spurious results (2). Random error often is the more frequent cause for the overestimation. To overcome the issue, trial sequential analysis (TSA) is introduced to project the required information size (RIS) for meta-analyses (3), which can explore the independent effect of random error on intervention effect estimates in meta-analyses and protect meta-analyses against overestimation due to random error.

Siemieniuk and colleagues investigated the effect of corticosteroid therapy on mortality and morbidity in adults with community-acquired pneumonia (CAP) and concluded that corticosteroid therapy may reduce all-cause mortality by approximately 3%, need for mechanical ventilation by approximately 5%, and duration of hospitalization by approximately 1 day (4). The authors should be commend for their excellent and important work. However, I believe that the conclusion requires further comments. In this study, there are few events and limited trials for many outcomes, as acknowledged by the authors. Thus, overestimation of treatment effects of corticosteroid therapy for CAP is inevitable, and potentially spurious evidence of effects may exist. Here, illustrating with example of one of many outcomes (i.e., mortality), I apply TSA to determine whether the evidence in this meta-analysis is reliable and conclusive. I calculated the RIS to yield “moderate” meta-analytic evidence based on an α=0.05 (two sided), β=0.20 (power 80%), an anticipated relative risk reduction of 20%, and an event proportion of 7.9% in the control arm. TSA on mortality showed the RIS (9,251 patients) is not reached, with the absence of reliable and conclusive evidence, as shown in Figure 1. Similarly, using TSA on other outcomes, the corresponding RIS also is not reached (not shown here).

Figure 1.

Trial sequential analysis (TSA) of 12 trials investigating the effect of corticosteroid therapy on mortality in adults with community-acquired pneumonia (CAP) (Equal Trial Distance). TSA of 12 trials (black square fill icons) illustrating that the cumulative Z-curve crossed neither the conventional boundary for benefit nor the trial sequential monitoring boundary for benefit, establishing insufficient and inconclusive evidence and suggesting further trials are required. A diversity adjusted required information size (RIS) of 9,251 patients was calculated using α=0.05 (two sided), β=0.20 (power 80%), an anticipated relative risk reduction of 20%, and an event proportion of 7.9% in the control arm. X-axis, the number of patients randomized; Y-axis, the cumulative Z-score; horizontal green dotted lines, conventional boundaries (upper for benefit, Z-score =1.96, lower for harm, Z-score =−1.96, two sided P=0.05); sloping red full lines with black circle fill icons, trial sequential monitoring boundaries calculated accordingly; blue full line with black square fill icons, Z-curve; vertical red full line, RIS calculated accordingly.

In summary, the treatment effects of corticosteroid therapy for CAP may be inflated, which limits the strength of the inferences that can be drawn. The current evidence on corticosteroid therapy for CAP is insufficient and inconclusive, and further trials are desirable to obtain firm and reliable evidence.