Apples and oranges: international comparisons of COVID-19 observational studies in ICUs

Multiple observational cohorts describing the outcome of patients with COVID-19 from across the world have been published.1, 2, 3 Typically, these studies have reported regional or national cohorts and no two countries have had the same experience. The reasons for these differences are complex and difficult to quantify. Nonetheless, to be able to draw meaningful inferences from these data we must tackle the issues associated with international comparison.

Initial reports of outcomes in COVID-19, which emerged from China early in the pandemic, reported a range of mortality rates from intensive care units (ICUs) (0–78%).³ Case series from North America and Europe have been equally variable (with ICU mortality ranging 0–85%).³ A major issue has been the large number of patients in these series who had incomplete outcomes at the time of reporting, a factor that has commonly resulted in mortality being overestimated or underestimated. For example, in UK Intensive Care National Audit and Research Centre (ICNARC) data, early reports from March, 2020, estimated ICU mortality for COVID-19 to be 79 (48%) of 165 patients admitted, when 610 (79%) of 775 patients had an incomplete outcome (ie, were still in the ICU). In the latest report, from July 31, 2020, ICU mortality had decreased to 40% in 10 341 patients with complete outcomes.⁴ In the appendix (pp 1–2), we have summarised European data on COVID-19 mortality, as of Aug 8, 2020, highlighting the range of outcome measures reported. Another key difference is the status of the health systems in which these patients have been managed, in particular the degree of so-called stress that those systems were under.⁵ This factor is more difficult to adjust for. Variations in clinical decision making between health-care systems, reflected in the characteristics of patients admitted to ICUs and in the methods of ventilation used, also confound direct comparison. This confounding is potentially evident when comparing ICU admissions between the UK and Germany, where the median age of patients receiving invasive mechanical ventilation was 72 years in a large German series² versus 60 years in the latest ICNARC report.⁴ However, ICU mortality was similar, emphasising the role of admission criteria. Regardless, the wide variation observed suggests the possibility that some factors are modifiable. Therefore, making comparisons between countries and systems is important.

Beyond careful epidemiological analysis, we could improve comparisons in several ways. The most obvious way to improve comparisons is via a multinational collaboration. Indeed, it is difficult to see how we can mount an effective response to a global pandemic without such collaboration. The fight against COVID-19 has already produced some commendable examples, including the work of the Coalition for Epidemic Preparedness Innovations, the Global Alliance for Vaccines and Immunizations, and the International Severe Acute Respiratory and Emerging Infection Consortium. However, global comparative data on the outcomes from COVID-19 are lacking because a single observational study of global data, with consistent outcomes and definitions used in all sites, has not yet been reported. An issue with the current reports is the inconsistency of the hospital, ICU, and individual patient-level data definitions. Similarly, fundamental differences are often present in the design of these studies. Together, these limitations make linkage or comparison difficult, which restricts our ability to generate sufficiently robust data to form conclusions. Although a single global study might be logistically and politically improbable, simpler measures could be taken that might ease the task of generating international data with fewer limitations.

First, the development of harmonised case report forms and data dictionaries would permit straightforward comparison of studies. This approach would also allow investigators to create tiers of data collection, with increasing levels of complexity. This method of data collection is of real importance in resource-limited settings, in which the advanced monitoring, diagnostics, and research infrastructure required to perform extensive study protocols are not available. The use of core outcomes in ICU observational studies should also be addressed—eg, the WHO Working Group on the Clinical Characterisation and Management of COVID-19 Infection outcome set.⁶ Second, the inclusion of measures of system stress and resource availability have been largely missing from published studies to date. These measures are essential to understand the observational data collected in the middle of a pandemic. For example, evidence of national variation in ICU outcomes has been described between Feb 8 and May 22, 2020, during the COVID-19 pandemic in England by Qian and colleagues.⁷ In this study, the authors hypothesised that national variation might have occurred as a result of local strain or resource constraints in the face of a surge in admissions to hospital. Likewise, a recent study from Brazil has highlighted disparities in the outcomes of patients admitted to hospital between the south of the country and the economically poorer north.⁸ In practice, this disparity might mean collecting hospital-level and unit-level information on resource availability and staffing over time, in addition to patient-level data. Third, integrating ICU observational studies with those that capture patient-level data before and after admission to critical care, and linkage to clinical trial data, would be advantageous. Such integration would better characterise patients and reduce the replication of data collection. Another benefit would be the ability to assess the influence of variations in clinical decision making on ICU outcomes, which hinder the comparison of existing studies. Furthermore, the effective linkage of large observational studies and clinical trials would allow investigators to better explore subgroup effects and identify heterogeneity of treatment effect, such as was identified in the RECOVERY dexamethasone study.⁹

Current reports of ICU outcomes for patients with COVID-19 vary between countries. The reasons for these variations are unknown but are unlikely to be attributable to differences in the virus or host response alone. Understanding the reasons that are modifiable would be a major step forward in improving care.

© 2020 Science Photo Library