Did an effect of kidney transplantation on COVID‐19 mortality go unnoticed due to selection bias?

I have read the letter by Hugo et al. [1] and the commentary with interest. Hugo et al. [1] conclude that they could not show adverse effects of prior solid organ transplantation on COVID‐19 mortality, based on a case‐control analysis. The commenters claim that this conclusion is incorrect, due to selection bias in the case‐control matching. I acknowledge the good scientific intentions of all authors involved; in what is written below, I will share my insights, form a statistical point of view, without expert knowledge of transplantation science and the related literature.

The research question in the letter by Hugo et al. [1] is clear: ‘Is solid organ transplantation history associated with mortality in COVID‐19 patients?’. The study adopts an observational, retrospective design. To the best of my understanding, the data in the sample were collected in a non‐probabilistic way, that is there was no process to choose participants before the data were collected in function of the research question at hand. Instead, the researchers use a database that contains a subset of the COVID‐19 patient population. Probabilistic matching is then used to correct for baseline differences between cases, that is COVID‐19 patients with a transplantation history, and controls, that is COVID‐19 patients without a transplantation history. The matching is important, since these differences might confound the association under investigation, namely the effect of transplantation history on COVID‐19 mortality. The researchers match for age, gender and comorbidities. In other words, if matched correctly, they draw conclusions on differences in mortality between COVID‐19 patients with and without a transplantation history who have similar characteristics in terms of gender, age and comorbidity distributions.

One condition of the case‐control approach is that cases and controls are sampled from the same population. I assume that this is the case, since the authors state: ‘a matched‐pair analysis (1:30) of 46 transplant recipients with 1380 controls without transplantation within one registry (LEOSS) was performed’. I also deduct this from the header in Table 1 [1]. However, the commenters, if I understand them correctly, hint towards a scenario where only controls were obtained from the LEOSS registry, while cases originated from the general population. If that is true, I agree with the commenters that this will very likely result in selection bias, since the controls originate from a subset of the general population of COVID‐19 patients that likely has an overrepresentation of severe cases [2], while this is not true for the cases.

If, however, both cases and controls originate from the LEOSS registry, which I assume in the remainder of this text, I believe that the comparison by Hugo et al. [1] could be valid in theory, if we can make at least three assumptions: (i) gender, age and comorbidities are the only, or at least the most important confounders. Note that the matched sample of controls will then likely have a higher mortality than the general public, since patients with a transplantation history have an increased probability to be old and to have comorbidities [3]. Comparing their mortalities without matching would not allow the researchers to disentangle the effects of the confounders and transplantation history. The mortality rates in the control group are very high though, and it is counterintuitive that the findings, after matching, are not roughly in line with findings documented in other larger‐scale studies [4]. (ii) Inclusion in LEOSS is alike for transplantation and non‐transplantation patients; if this is not the case, selection bias can occur. I do not have enough information on the LEOSS study to draw a conclusion on this. (iii) The effects of the severity of COVID‐19 and transplantation history on mortality do not interact. If they would interact, a conclusion on the comparison between COVID‐19 patients with or without a transplantation history would likely differ between a study using cases and controls from the LEOSS data, where severe COVID‐19 patients are likely overrepresented, and a study where cases and controls were obtained from the general population including those with mild pathologies. This would render the study by Hugo et al. [1] only comparable with studies that investigate patients with similar COVID‐19 severity.

The previous paragraph mentions theoretical considerations that underlie the analysis of Hugo et al. [1] Although I appreciate that not all assumptions can be perfectly validated in a complicated real‐life setting, I have some concerns about the execution of the analysis: (i) I agree with the commenters that the age matching is poor, which potentially convolutes the association under investigation, especially since age was not taken up as a covariate. (ii) The sample size of the cases is small and the case‐to‐control ratio of (1:30) is extreme, while I am not convinced of its beneficial effects on the analysis’ power. The small sample limitation has been acknowledged by the authors though. (iii) I do not understand the rationale behind the sensitivity analysis: (1:10), (1:20) and (1:30) are in my opinion all extreme ratios, possibly resulting in similar power. (iv) Model selection in the multivariable logistic regression could have finetuned the model, and it could have resulted in clearer insights on the effects of borderline significant effects, such as that of mechanical ventilation (P = 0.040).

I, therefore, conclude that (i) there should be more information on the data selection process, namely the potential bias originating from opportunistic participation in the LEOSS registry and the selection of cases, (ii) the performance of the age matching is poor and (iii) a number of modelling choices are unclear and/or suboptimal, while they are not thoroughly discussed. On the other hand, I appreciate that a formal test via matching has been undertaken, which was not the case in the two referenced papers [3, 5]. But, note additionally that Pereira et al. [3] have a control group (from literature) that consists of hospitalized patients, so somewhat severe cases as well. Their mortalities are much lower than those in the study of Hugo et al., so it remains unclear to me whether they are so large in Hugo et al. [1] due to the matching or due to another underlying mechanism that may cause selection bias.