Statistical models to predict clinical outcomes with anakinra vs. tocilizumab treatments for severe pneumonia in COVID19 patients

Dear Editor,

The most common complication of coronavirus disease 2019 (COVID-19) related pneumonia is the acute respiratory distress syndrome (ARDS), characterized by dramatic inflammatory features like cytokine release syndrome and macrophage activation syndrome (MAS) [1].

Interleukin-6 blockade with tocilizumab and recombinant interleukin-1 receptor antagonist anakinra may control this observed cytokine storm [2]. About tocilizumab treatment there are not concordant results regarding clinical efficacy in term of mortality and adverse events except for a better clinical response in patients with demonstrated hyper-inflammation syndrome [3,4]. Results from meta-analysis of non-randomized clinical trials (RCT) demonstrated that anakinra seemed to be associated with reduced mortality and lower needing for mechanical ventilation in severe and hyper-inflammed COVID19 patients and probably it is superior compared to tocilizumab in terms of COVID19 death prevention [5,6]. Moreover, early treatment with anakinra was associated with a better clinical outcome compared with placebo in 594 patients at risk of progressing respiratory failure identified by plasma soluble urokinase plasminogen activator receptor (suPAR) > 6 ng/mL [7]. Considering the results derived from RCTs, observational studies and meta-analysis, and the uncertain clinical advantage of previous elaborated prognostic scores [8], the challenge is how to select the appropriate immunomodulating treatment (anakinra vs tocilizumab) based on clinical conditions (Charlson Comorbidity Index – CCI) and biochemical parameters to reduce adverse events (i.e. secondary infections, 30-day mortality) and obtain better clinical response. Our purpose was to develop a novel predictive model which uses biochemical markers and anamnestic data (CCI) to finding similar patterns in behavior and forecasting patient's responses or actions to occurring events (secondary infection risk and/or survival).

We conducted a retrospective analysis including all patients admitted to our Institution with a diagnosis of COVID-19 pneumonia during the first wave of the pandemic (from March 1 and May 15, 2020), who required different oxygen supports, and who received off-label treatment with anakinra or tocilizumab.

Nonparametric tests (Mann–Whitney and Kruskal–Wallis) and Pearson χ2 test (or Fisher's exact test, when applicable) were used to compare the continuous and categorical variables of patients, respectively, with the different treatment strategies. Statistical analysis was performed using R (version 4.0.0) and R Studio (version 1.2.5042) software.

Predictive models based on Artificial Intelligence (AI) were used to predict the outcome of a COVID-19 patient treated with anakinra or tocilizumab using CCI and biochemical/inflammatory parameters at baseline (day of start immunomodulant treatment). Specifically, our AI predictive models adopt Logistic Regression [8,9] using the Scikit-learn package in Python. Logistic Regression identifies the relationship between a continuous dependent variable (30-day mortality and secondary infections) and one or more independent variables (CCI and blood inflammatory markers).

To provide a quantitative evaluation of the performance of the models and their effectiveness in making predictions on the corresponding test sets, we computed: (1) the R-squared (R²), which explains to what extent the variance of one variable explains the variance of the second variable (e.g., if the R² of a model is 0.50, then approximately half of the observed variation can be explained by the model's inputs); and (2) the mean squared error (MSE), which measures the amount of error in statistical models. The K-fold cross-validation technique was also exploited to detect overfitting in a model (i.e., the model is not effectively generalizing patterns and similarities in the newly inputted data). K refers to the number of groups the data sample is split into.

71 and 39 patients were treated with anakinra and tocilizumab, respectively. 71.8% of patients were male, and the median age was 65 years (interquartile range [IQR] 59–71 years). No differences were observed in CCI and inflammatory biomarkers at baseline between the anakinra and tocilizumab groups.

Overall, 30-day mortality was 23% (20/71) in patients receiving anakinra versus 35% (12/39) in those receiving tocilizumab, without a statistically significant difference between the groups.

Same results were found about major secondary infection events (19.7% vs 28.2%, anakinra vs tocilizumab; p = 0.44). No significant difference in mortality was observed between patients experiencing secondary infections compared with those without this adverse event in the anakinra and tocilizumab groups (35.7% vs. 27.2%, p = 0.40 and 27.3% vs. 32.1%, p = 0.19, respectively).

A comparative analysis of anakinra and tocilizumab treatments to discover the relationship between CCI (independent variable) with secondary infection risk and probability of survival (dependent variables) was performed. The results show a positive correlation between CCI values and secondary infection (R²=0.88) with anakinra, whereas inverse correlation (R² = 0.70) with tocilizumab treatment.

The relationship between CCI and survival decreases as CCI values increase in both groups of patients (anakinra R²=0.80 and tocilizumab R²=0.64).

Table 1 summarizes the quantitative evaluation of the performance of AI logistic regression models and their effectiveness in predicting the probability of secondary infection and/or survival based on the relationship between CCI and baseline biomakers. To this end, forecasting performance and model accuracy have been evaluated using the MSE error measure and R², respectively.

Table 1.

Comparative analysis among several treatments, evaluating the logistic regression models.

$\mathit{f}\left(\mathit{z}\right)=\frac{1}{1+{\mathit{e}}^{-\left(\mathit{model}\right)}}$
		Model	MSE	5-fold score (R2)	Mean 5-fold score (R2)
Anakinra group	Secondary infection risk	0.838*CCI	0.17	0.83; 0.82; 0.91; 0.91; 0.91	0.88
		0.574*CCI – 0.009*LDH	0.125	0.87; 0.87; 0.87; 0.875; 1	0.9
		6.14e-01*CCI – 1.8e-05*XDP	0.1	0.9; 0.9; 0.88; 0.88; 0.88	0.89
		0.068*CCI – 0.262*CREA	0.18	0.9; 0.81; 0.9; 0.9; 0.9	0.88
		0.502*CCI – 0.494*CRP	0	0.9; 1; 0.9; 0.8; 0.88	0.9
		0.335*CCI – 0.078*WBC	0.18	0.9; 0.9; 0.72; 0.9; 0.9	0.86
	Probability of survival	0.793*CCI – 1.003*CREA	0.18	0.91; 0.81; 0.9; 0.9; 0.8	0.86
		0.839*CCI	0.25	0.83; 0.72; 0.81; 0.91; 0.72	0.80
		0.015*CCI – 0.989*CRP	0.2	0.8; 0.7; 0.8; 0.9; 0.77	0.79
		8.18e-01*CCI – 4.49e-05*XDP	0.2	0.8; 0.8; 0.77; 0.77; 0.77	0.78
		1.354*CCI – 0.007*LDH	0.125	0.625; 0.75; 0.75; 0.875; 0.85	0.77
		0.721*CCI – 0.247*WBC	0.36	0.54; 0.72; 0.72; 0.6; 0.8	0.68
Tocilizumab group	Secondary infection risk	1.31e-07*CCI	0.28	0.85; 0.71; 0.71; 0.57; 0.66	0.70
		1.27e-01*CCI – 3.62e-05*XDP	0.28	0.71; 0.57; 0.66; 0.66; 0.66	0.65
		0.134*CCI – 0.002*LDH	0.33	0.5; 0.66; 0.66; 0.66; 0.66	0.63
		0.494*CCI – 0.723*CREA	0.43	0.71; 0.57; 0.5; 0.66; 0.66	0.62
		0.008*CCI – 0.786*CRP	0.71	0.71; 0.5; 0.5; 0.66; 0.66	0.61
		0.539*CCI – 0.021*WBC	0.57	0.428; 0.14; 0.66; 0.5; 0.33	0.41
	Probability of survival	0.209*CCI – 0.420*CRP	0	1; 0.66; 1; 0.83; 0.5	0.80
		8.22e-01*CCI – 9.32e-05*XDP	0.428	0.57; 0.57; 0.66; 1; 0.5	0.66
		0.579*CCI – 0.610*CREA	0.285	0.57; 0.57; 0.66; 0.83; 0.66	0.66
		0.684*CCI	0.428	0.71; 0.57; 0.57; 0.71; 0.66	0.64
		0.386*CCI – 0.006*LDH	0	0.5; 0.5; 0.5; 1; 0.66	0.63
		0.768*CCI – 0.127*WBC	0.428	0.71; 0.57; 0.66; 0.66; 0.5	0.62

Abbreviation: CCI: Charlson Comorbidity Index; CRP: C-reactive protein; LDH: lactate dehydrogenase; XDP: d-dimer; CREA: creatinine; WBC: white blood cells.

Anakinra treatment seems to be safer with respect to 30 days mortality and secondary infection-related events, which were more frequent in patients receiving tocilizumab without a statistically significant difference between the groups as reported from other authors [10]. This may be possibly related to the different half-life of tocilizumab and anakinra treatments. It is probably confirmed by R²-squared analysis that showed a direct correlation between CCI and secondary infection events in anakinra treated patients. This relationship is not found in the tocilizumab group, where secondary infections are independent to baseline comorbidity conditions. On the contrary, the correlation between 30 days survival and CCI is not influenced by immunomodulant treatment in our population.

The advantages of using models like ours should be: (i) improved diagnostics; (ii) high cost-effectiveness; (iii) increased operational efficiency; (iv) reducing readmission rates and finally (v) personalized medical care with predictive models that improve patient-centered care based on personal health records and contributes to the creation of the most effective and personalized treatment plans for each patient. To this end, as showed in Table 1 considering CCI and CREA, we found that, when properly correlated with each other with the coefficients identified by the mathematical model, they are able to predict with good accuracy (MSE=0.18 and R2=0.88) our dependent variable (i.e., the probability of risk of secondary infection) for anakinra, while discretely for tocilizumab (MSE=0.43 and R²=0.62). Thus, if we observe for Patient A at day 0, hypothetical values of CCI=2 and creatinine (CREA) =3, a patient treated with anakinra has a 34% probability of secondary infection, while 23% with tocilizumab.

A limitation of the present study is the small sample size, reducing the power to detect a significant association between specific drug usage and outcomes. Moreover, we acknowledge the lack of assessment of immune systems activation markers such as interleukin-6 or other inflammatory biomarkers (suPAR) at baseline. Nevertheless, their utility was not clear in the first phase of the pandemic, when the clinical picture of hyper-inflammation should be defined by CRP and ferritin levels, combined with the available respiratory performance status.

In this context, we believe that the correct place in therapy of these immunomodulant drugs should be carefully evaluated; they could be used in the context of proven hyper-activation of the immune system, but also when the onset of cytokine storm in COVID-19 patients is suspected [4]. Thus, we believe that anakinra and tocilizumab have represented valid rescue therapy when used in conjunction with corticosteroids given from the beginning of COVID-19 respiratory symptoms in agreement with the earlier evidence and guidelines. However, the recent early use of COVID-19 antiviral therapies and the large participation in the SARS CoV-2 vaccination programs reduced their needing, especially because there are not evidences yet about their clinical efficacy in completely vaccinated patients with severe COVID19 infection. This work highlighted with a data-driven approach that during the emergency of a new pandemic, predictive modeling should be important to support clinicians in clinical decision-making and the healthcare system in strategic decision-making, planning, and formulation of health policies that contribute to the fight against a unknown disease. More studies are needed to validate the usefulness and safety of these models in the “real life” management of COVID-19 infection and other inflammatory clinical conditions.

Declarations

Funding

None to declare.

Availability of data and materials

All data for this study will be made available upon reasonable request to the corresponding author.

Declaration of Competing Interests

The authors have no financial or personal conflicts of interest.