Re: ‘Methodological evaluation of bias in observational COVID-19 studies on drug effectiveness’ by Wolkewitz et al.

Dear Editor,

We read with interest the paper by Martinuka et al. published in Clinical Microbiology and Infection [1]. Although we agree with the general issue that “making valid causal inferences from real-world observational data is a demanding task that requires high-quality data and adequate statistical methods as well as clinical knowledge and statistical expertise”, a few points regarding specific criticisms to our TESEO study need to be pointed out [2]. Indeed, the authors seemed to have misread both the design and statistical methods used in our study.

First, the study population was people with COVID-19 pneumonia admitted to a tertiary hospital, not people entering the intensive care unit ICU as incorrectly reported in Table 1.

Immortal bias seems to be a non-issue in the setting of people hospitalized with COVID-19 pneumonia. Indeed, the probability of dying before starting any treatment in such a target population is close to zero so immortal bias is unlikely to occur.

The second common misconception regards the presence of competing risks and how to control for these. Although we agree that people who are discharged before day 28 are no longer at risk of undergoing mechanical ventilation or dying and this was a competing risk in our analysis, our aim was to give an estimate of the average treatment effect equivalent to what could be estimated in the emulated randomized trial [3]. Thus, the aim was to quantify the survival time distribution for the situation without the competing risk. Specifically, for unbiased estimation of the effect of the intervention, we had to assume that participants whose follow-up was censored due to the competing risk could be represented by the ones who remained in follow-up. This was achieved in the secondary analysis which correctly adjusted for informative censoring using inverse probability of censoring weights (not reported in Table 3 by Martinuka et al). A competing risk analysis would have been appropriate if the aim was to quantify the risks after taking into account that participants could also experience an early discharge, not causal inference using a marginal model. The two paradigms are often confused [4].

We also agree that to treat the intervention as time-fixed and to control only for time-fixed confounding factors was a simplification. Nevertheless, again the amount of potential bias introduced by this simplification depends on specific settings. In our setting, treatment was initiated almost immediately after hospital admission (typically within 48 hr) and although some time-varying variables could change very rapidly (e.g. the PaO₂/FiO₂ ratio) the introduction of large bias by using a time-fixed approach is likely to be negligible. In addition, to report that we ignored time-varying confounding is simply inaccurate (Table 2). Indeed, in our secondary analysis we did control for post-baseline varying confounding of starting other pharmaceutical interventions such as steroids.

Moreover, as an example, we report the results of another recent analysis of ours aiming to emulate the RECOVERY trial (comparing the risk of death in people who were randomized to remain on steroids alone or to add tocilizumab to steroids) [5]. We performed this analysis using a time-fixed intervention variable with time fixed confounding or, alternatively as recommended by Martinuka et al., using all time-varying factors. As shown in the Table 1 , because events occurred very quickly after admission to hospital, all the approaches led to very similar results (a maximum difference of 10% in the estimated effect size of the intervention on risk of death, with no difference in the overall conclusions). Of note, using standard regression techniques to control for time-varying intervention in the presence of time-varying confounders affected by prior intervention led to the same amount of bias introduced by the time-fixed simplification [6]. Thus, at least in this specific analysis, to appropriately control for confounding appeared to be as crucial as the choice between a time-fixed vs. a time-varying intervention design.

Table 1.

Effect size of tocilizumab intensification in people treated with steroids in our observational cohort

	Hazard ratios of death (95% CI)	p
Unadjusted (time-varying intervention)
Never started tocilizumab	1
Intensified with tocilizumab	0.56 (0.36, 0.87)	0.010
Adjusteda(time-fixed intervention)
Never started tocilizumab	1
Intensified with tocilizumab	0.48 (0.26, 0.87)	0.016
Adjusted for time-fixed covariatesb(time-varying intervention)
Never started tocilizumab	1
Intensified with tocilizumab	0.53 (0.33, 0.86)	0.010
Adjusted for time-varying covariatesc(time-varying intervention)
Never started tocilizumab	1
Intensified with tocilizumab	0.50 (0.31, 0.83)	0.007
Weightedd(time-varying intervention)
Never started tocilizumab	1
Intensified with tocilizumab	0.66 (0.41, 1.05)	0.081

CRP, C-reactive protein; CCI, Charlson Comorbidity Index; IPW, Inverse probability weights.

^aWeighted model adjusted for age, ethnicity, baseline CCI, baseline CRP and censoring using IPW.

^bStandard Cox model adjusted forage, ethnicity, CCI, baseline CRP and PaO₂/FiO₂ ratio.

^cStandard Cox model adjusted forage, ethnicity, CCI, baseline and time-varying PaO₂/FiO₂ ratio and CRP.

^dWeighted Cox model controlled forage, ethnicity, CCI, baseline and time-varying PaO₂/FiO₂ ratio and CRP using IPW.

Finally, an important way to evaluate the validity of the results of an observational study, not mentioned in the article by Martinuka et al., is to compare its results with those of the reference randomized trial [[5], [7], [8]]. In our case, the results of the TESEO study for the effect of tocilizumab vs. standard of care in people enrolled during the first wave (HR 0.61; 95% CI 0.40–0.92) were remarkably consistent with those of the reference REMAP-CAP trial conducted on a similar study population (HR 0.57; 95% CI 0.47–0.80) [3]. Other RCTs showed conflicting results but were conducted in different target populations and effect measure modification is a key issue when evaluating the efficacy of tocilizumab [9].

Transparency declaration

Alessandro Cozzi-Lepri has no conflicts of interest. No external funding was received for this work.

Author contributions

Alessandro Cozzi-Lepri: letter conceptualization, formal statistical analysis, data interpretation, writing and revising for intellectual content. Cristina Mussini: letter conceptualization and revising for intellectual content. Marianna Meschiari: data curation and revising for intellectual content. Giovanni Guaraldi: data curation and revising for intellectual content.

Editor: L. Leibovici