Methodological issues in meta-analyses of observational studies: the need for attention to the details

Editor—We read with interest the meta-analysis of awake prone position for spontaneously breathing patients by Fazzini and colleagues.¹ Systematic review and meta-analysis has been an important tool to bring insight into the care of patients with COVID-19. In this context, we appreciate all efforts to deal with the pandemic and minimise its burden. However, the main goal of systematic reviews cannot be forgotten: to find effect sizes as precisely as possible without bias, accounting for the risk of bias in study design (i.e. excluding studies that may bias the results) and the use of appropriate analytical tools. This paper caught our attention for several reasons.

First, the primary outcome was change in the $P{\text{a}}_{{\text{o}}_2}$ :FiO₂ ratio, which is a physiological outcome of questionable patient-centredness that should not be a guide to clinical management per se. Moreover, the authors used a paediatric linear equation to estimate $P{\text{a}}_{{\text{o}}_2}$:FiO₂ ratio mean differences from the SpO₂:FiO₂ ratio. This issue has been studied and the current recommendation is to use non-linear imputation of $P{\text{a}}_{{\text{o}}_2}$:FiO₂ ratios from SpO₂:FiO₂ ratios.² This very important issue makes study results questionable and probably invalidates their primary outcome assessment.

More importantly, the data extracted to pool odds ratios for the secondary outcome of tracheal intubation were not the adjusted results. For non-randomised studies of interventions (NRSI), the Cochrane Handbook recommends extracting adjusted data and subsequently pooling the results.³ Miguel Hernan furthers this concept stating that, although sample size may not be the most important issue, adequately designed and analysed observational studies (i.e. studies addressing confounding and other methodological issues such as immortal time bias) are quite important for evidence synthesis.⁴ For example, our study,⁵ which represents one of the highest weights in the meta-analysis of this outcome among the NRSI, presented both unadjusted and adjusted results. Although in the unadjusted analysis the hazard ratio was 1.21 (95% confidence interval [CI], 0.78–1.88), in our adjusted analysis for relevant characteristics the hazard ratio was 0.90 (95% CI, 0.55–1.49), with modifying the direction of the point estimate as a result of confounding by indication accounted for at least partially.

Using raw data without statistical adjustment introduces bias in the meta-analysis as confounding will not be properly dealt with. The estimates will be inherently biased, thus making it questionable to meta-analyse the data of clinical trials along with observational studies. To tackle this issue and as an example, we have reanalysed the secondary outcome of tracheal intubation rate, including only studies that dealt with confounding, at least to some extent.5, 6, 7, 8 We extracted the adjusted hazard ratios with their respective CIs from non-randomised studies and extracted the same data from the clinical trial.⁹ We performed a random effects meta-analysis using the Hartung–Knapp–Sidik–Jonkman method to avoid overly optimistic results. Our results are presented in Fig. 1 . The first finding is that all studies had point estimates either favouring awake prone positioning or neutral. The corresponding heterogeneity was much lower (I ²=21%) than that presented in Fazzini and colleagues'¹ Figure 3 (I ²=75%). This is expected, because the large observed heterogeneity can be explained by the inclusion of unadjusted (i.e. confounded) analyses in the results.

Fig 1.

Forest plot of the hazard ratio for tracheal intubation among observational studies presenting adjusted estimates and the clinical trial. Estimates obtained from a random effects model using the Hartung–Knapp–Sidik–Jonkman method. APP, awake prone positioning; CI, confidence interval; HR, hazard ratio.

The final finding of this reanalysis is that awake prone positioning is associated with a reduced tracheal intubation rate (hazard ratio 0.79, 95% CI, 0.63–0.98), without worrisome inconsistency and with some impreciseness as a result of the moderately large CI. These results suggest that awake prone positioning, when it comes to the outcome of intubation, should be at least suggested in clinical practice per the GRADE approach.¹⁰ This recommendation cannot be strong (recommend statement) because of some impreciseness and because of indirectness (gathering data from observational studies in the absence of more randomised clinical trials). The exercise we did with intubation hazard can also be done with the meta-analysis presented for mortality (Fig. 4 of Fazzini and colleagues¹), in which unadjusted (and therefore biased) analyses were done, coming to biased and potentially wrong conclusions.

Finally, we observed that subgroup analyses shown in the supplement grouped patients proned for >4 h or for <4 h. Our study results are included in that specific analysis considering that all patients were proned for >4 h. However, according to our results, only 29 (58%) patients tolerated prone positioning for >4 h.⁵ Therefore, a correction is necessary: of the 57 proned patients included in the subgroup analysis,¹ only 29 should have been included.

Although observational studies have been increasingly recognised as important to be included in systematic reviews and meta-analyses, especially where data from randomised trials are not of low risk of bias, this is not straightforward. Adequate selection of studies, thorough risk of bias assessment, adequate extraction of data, and finally proper analysis are of utmost importance to draw unbiased inferences as precisely as possible. With proper methodology, our conclusions are different from the authors' conclusion. Pending the publication of further trials (the COVI-PRONE trial), which will increase sample size and therefore provide more precise effect estimates, awake prone positioning could at least be suggested (weak recommendation) as a strategy to avoid tracheal intubation for adult patients with COVID-19-related respiratory failure who are not in imminent need of invasive mechanical ventilation.

Declarations of interest

The authors declare that they have no conflicts of interest.