Development and validation of a simple web-based tool for early prediction of COVID-19-associated death in kidney transplant recipients

Abstract

This analysis, using data from the Brazilian kidney transplant (KT) COVID-19 study, seeks to develop a prediction score to assist in COVID-19 risk stratification in KT recipients. In this study, 1379 patients (35 sites) were enrolled, and a machine learning approach was used to fit models in a derivation cohort. A reduced Elastic Net model was selected, and the accuracy to predict the 28-day fatality after the COVID-19 diagnosis, assessed by the area under the ROC curve (AUC-ROC), was confirmed in a validation cohort. The better calibration values were used to build the applicable ImAgeS score. The 28-day fatality rate was 17% (n = 235), which was associated with increasing age, hypertension and cardiovascular disease, higher body mass index, dyspnea, and use of mycophenolate acid or azathioprine. Higher kidney graft function, longer time of symptoms until COVID-19 diagnosis, presence of anosmia or coryza, and use of mTOR inhibitor were associated with reduced risk of death. The coefficients of the best model were used to build the predictive score, which achieved an AUC-ROC of 0.767 (95% CI 0.698–0.834) in the validation cohort. In conclusion, the easily applicable predictive model could assist health care practitioners in identifying non-hospitalized kidney transplant patients that may require more intensive monitoring.

Trial registration: ClinicalTrials.gov NCT04494776.

1. INTRODUCTION

The coronavirus disease 2019 (COVID-19) pandemic has caused an unprecedented global health crisis,¹ strongly affecting health care systems in most countries worldwide. The number of deaths due to COVID-19 skyrocketed throughout 2020, as did the number of death due to other etiologies, owing to the collapse of several health care systems as the pandemic unfolded,² and confirmed by the high excess mortality observed in several countries.³ Many hospitals and health care services have become overloaded and the number of medical procedures unrelated to the management of COVID-19 has fallen dramatically.⁴ One medical area that has been most affected is organ transplantation, especially kidney transplantation, which has experienced a significant reduction in the number of transplants performed worldwide.⁵

In addition to the observed decrease in transplant activity, solid organ transplant recipients have been considered a high-risk group.⁶ As the full spectrum of COVID-19, from asymptomatic to severe acute respiratory syndrome,⁷ has already been reported, the major challenge is to identify, as early as possible, the most accurate prognostic factors that can predict the need for hospitalization, intensive care unit, and, ultimately, death. In the general population, advanced age and the presence of comorbidities, such as hypertension, diabetes, chronic cardiovascular or pulmonary diseases, and chronic kidney disease has been associated with worse outcomes.⁸ Consequently, by accumulating comorbidities, the recipients of solid organs would be susceptible to worse outcomes. The effect of chronic use of immunosuppressive drugs, however, is uncertain, as some evidence suggests that COVID-19 in kidney transplant (KT) recipients have similar outcomes to the general population when the comorbidities are closely matched.^{9 , 10}

In this scenario, predictive models using readily available data could be particularly useful to support decision-making clinical management, including remote assessment performed by primary health care professionals using telehealth medicine.¹¹ Hypothetically, the health care services could benefit from this burden-reduction strategy. Actually, predictive scores have been developed to assist risk stratification in the general population, although such score to assess the risk for KT recipients is not yet available.12, 13, 14

Brazil has the largest public transplant program and is one of the countries most affected by the pandemic. Therefore, the present study aimed to develop a prognostic model for KT recipients that could assist in risk stratification on an outpatient basis, using data extracted from the COVID-19-KT Brazil study group carried out throughout 2020.

2. MATERIALS AND METHODS

2.1. Population and setting

A multicenter retrospective cohort study has been carried out in transplant centers in Brazil, the COVID-19-KT Brazil. All 81 active KT centers in Brazil were invited, 78 have agreed to participate, 37 have effectively completed the regulatory process, and 35 have included patients. These centers represent 57% of the national transplantation activity. The study was approved by the National Ethics Research Committee (identification number CAEE 30631820.0.1001.8098 and approval number 4.033.525) and by the local ethics committee of all participating centers, and it was registered in the Clinical.Trails.gov (NCT04494776). Informed consent or its exemption followed specific national legislation, local Institutional Review Board recommendations, and the guidelines of the Declaration of Helsinki. We followed the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement.¹⁵ Data were anonymized, de-identified, and stored in the REDCap platform.¹⁶

2.2. Inclusion criteria and definitions

The eligible participants for this analysis were KT recipients who underwent transplantation performed at any time, of any age, diagnosed with COVID-19 through reverse-transcription polymerase chain reaction (RT-PCR) assay between March 3 and October 31, 2020. The final follow-up date was November 30, 2020. Aimed to have an extra validation, a second cohort composed of patients diagnosed in 2021 was fitted. Thus, those diagnosed between January 1 and April 30, 2021, were enrolled in the second validation cohort. For this second group of patients, the final follow-up date was May 30, 2021. For all patients, the diagnosis was considered only in patients who presented at least one COVID-19-attributable symptom associated with a positive RT-PCR of sample collected from the nasopharyngeal or oropharyngeal swab. The attributable symptoms were defined by the local investigator. According to their practices, the local investigators defined the allocation to home care or hospital for clinical management.

2.3. Variables of interest: predictor variables

The variables of interest were grouped into four categories: demographic data, comorbidities, immunosuppression, and symptoms of COVID-19. Demographic data included age, sex, ethnicity, etiology of chronic kidney disease, type of donor (deceased or living donor), body mass index (BMI), and the baseline glomerular filtration rate (eGFR), estimated by the Chronic Kidney Disease Epidemiology Collaboration Equation (CKD-EPI),¹⁷ and the interval between transplantation and the infection diagnosis (in years). For the graft function estimative, the baseline creatinine value was assessed from the mean value of the three last available serum creatinine measurements before the COVID-19 diagnosis. The comorbidities evaluated were diabetes, hypertension, neoplasia, smoking, and cardiovascular, lung, liver, autoimmune or neurological diseases. The use of angiotensin-converting enzyme (ACE) inhibitors or angiotensin II receptor blockers (ARB) was included in the comorbidities group.¹⁸ The immunosuppression included the drugs of maintenance regime at the COVID-19 diagnosis. Last, the time from symptoms onset (in days) and the most frequently reported symptoms or signs of COVID-19 were included in the analysis. Dyspnea was defined as any degree of shortness of breath or difficulty in breathing subjectively reported by the patient.

2.4. Outcome

The main outcome was death by any cause within 28 days from the COVID-19 diagnosis.

2.5. Statistics

2.5.1. Exploratory data analysis

All variables were compared between patients who survived with those who died up to 28 days after the diagnosis. This comparison was performed by the X ² test for categorical variables and by the Mann-Whitney test for continuous variables.

2.5.2. Predictive model

For the predictive model, the categorical variables were transformed into dummy variables, and the missing values were imputed by the most frequency class as there was an exceptionally low rate of missing values. The continuous variables were normalized by dividing their values by means (center) and standard deviation (scale), and the median value was imputed for missing data. Variables with zero or near-zero variance were removed from the model. Natural splines were used in the variables age and eGFR with four degrees of freedom owing to a linear relationship with outcome was not found to be a good approximation. For missing we imputed the median value. The total missing value was below 1%, and most of the variables had a missing value below 5%, except eGFR with 10.9%.

2.5.3. Model training

A derivation (training) and a validation (test) data set were created using a random split stratified by the target into training (75%, n = 1035) and test (25%, n = 344). An algorithm created a single binary split of the data into training and testing sets at random, and a seed approach was used to ensure the productiveness of the analysis. Details about derivation and validation are shown in Table S1. In the training data, 10-fold-cross validation was used to select the hyperparameters of the models and to reduce the bias and variability of the performance estimates. To adjust to the class imbalance, the synthetic minority over-sampling (smote) method was used to create synthetic classes in the training set (Balancing).¹⁹ A full model was fitted in the derivation cohort using all candidate predictors. Additionally, a feature selection by a least absolute shrinkage and selection operator (LASSO) model was performed, and the predictors with non-zero coefficients were selected to fit a reduced model. Gradient boosting decision trees (XGBoost) and an Elastic Net were fitted to develop the candidate equations. The hyperparameters tuned in XGBoost and Elastic Net are described in the supplementary material (Tables S2 and S3). Finally, the best hyperparameters were selected using machine learning approaches by 10-fold-cross validation in a train set aiming to maximize the area under the receiver operating characteristic (AUC-ROC) curve, detailed in the supplementary material (Table S4).

2.5.4. Assessment of accuracy and calibration

The accuracy of the derivation cohort models was tested on the validation cohort using the AUC-ROC curve by 28-day fatality. The 95% confidence interval of AUC-ROC curves were estimated by bootstrap resampling (2000 samples) to reduce overfit bias. To evaluate the goodness of fit of models, the predicted versus observed target values were plotted in a confusion matrix of the first validation cohort. The best model was selected to minimize the number of false negatives. The calibration of models was evaluated throughout the Brier Score²⁰ and Slope values in the test set using the RMS R package.

2.5.5. Score fit and model visualization

The model with a higher AUC-ROC curve in the validation cohort and better calibration values was used to build the ImAgeS score. Shapley Additive Explanations (SHAP) were chosen to visualize and explain the importance of the predictors. SHAP plots are used to reduce the difficulties in interpreting machine learning models.²¹

2.5.6. Accuracy metrics for previous published COVID-19 models

The final model was compared with three available models that have been externally validated in the general population: the CHA2DS2-VASc, the clinical predictive model proposed by Wang et al. and the COVID SEIMC score.22, 23, 24 Details about these scores are described in the supplementary material. The comparisons were performed throughout the assessment of sensitivity, specificity, and AUC-ROC.

2.5.7. Sensitivity analysis

For sensitivity analysis, the first validation cohort was split into four factors: allocation for treatment (in-hospital or domiciliary), center according to the volume of enrolled patients (high or low volume), the time between transplantation and COVID-19 diagnose (more than 1 year or less) and type of donor (living and deceases). Center was considered as high enrollment volume if the number of patients was higher than 100, and low if the number was lower than 50. The analysis was performed by the AUC-ROC.

The software R version 4.0.2 and the packages tidymodels and DALEx were used to create and visualize the models. The R packages “glmnet” and “xgboot” statistical software (R Foundation) were used to perform the Elastic Net regression and XGBoost models.

3. RESULTS

3.1. Demographic data, the COVID-19 presentation, and comparison between survivors and non-survivors

Between March and October 2020, data from 1635 KT recipients with COVID-19 were reported by 35 centers. We excluded 256 because the diagnosis was performed by serology, and therefore data from 1379 patients were included in the present analysis ( Figure 1). The baseline characteristics and immunosuppressive drug regimens are detailed in Table 1. The median age was 52 (42, 60) years and most were male (61%). The main etiology of chronic kidney disease was unknown (28%), and 16% had diabetes. The kidney transplant was performed with a deceased donor in 68%, and the time interval between the transplantation and the COVID-19 diagnosis was 6.0 (2.1, 10.7) years. Baseline eGFR was 47 (31, 64) ml/min/1.73 m². The median time between the symptoms or signs onset and the diagnosis of COVID-19 was 5.0 (3.0, 9.0) days. The clinical presentation is detailed in Table 2. The most frequent respiratory symptoms/signs were fever or chills (62%), cough (54%), dyspnea (40%), and myalgia (40%). Diarrhea was reported in 32%, anosmia in 23%, and hypoxemia in 14% of the patients.

FIGURE 1.

Participant flow diagram and proportion of patients enrolled in the derivation and validation cohorts. Using a random split, 1,035 patients were grouped in the training cohort (training data set), which represents 75% of the entire cohort, whereas 344 patients were grouped in the validation cohort (test data set). A more detailed flow diagram of the population can be consulted in the supplementary material (Figure S1)

TABLE 1.

Baseline characteristics: demographic, comorbidities, and immunosuppression

Variables	Non-missing values	Overall N = 1379	Survivors N = 1144	Non-survivors N = 235	p-value
Age (years)	1379	52 (42, 60)	51 (41, 58)	59 (51, 67)	<.001
Male sex – n (%)	1379	839 (61%)	701 (61%)	138 (59%)	.5
African-Brazilian ethnicity – n (%)	1379	166 (12%)	139 (12%)	27 (11%)	.9
Etiology of CKD – n (%)
Hypertension	1379	194 (14%)	163 (14%)	31 (13%)	<.001
Diabetes	1379	222 (16%)	166 (15%)	56 (24%)
Glomerulonephritis	1379	254 (18%)	221 (19%)	33 (14%)
ADPKD	1379	106 (7.7%)	80 (7.0%)	26 (11%)
Urologic	1379	24 (1.7%)	20 (1.7%)	4 (1.7%)
Others	1379	187 (14%)	164 (14%)	23 (9.8%)
Unknown	1379	392 (28%)	330 (29%)	62 (26%)
BMI (kg/m2)	1307	26.4 (23.5, 29.8)	26.4 (23.5, 29.7)	26.9 (23.7, 30.5)	.3
Deceased donor – n (%)	1379	942 (68%)	762 (67%)	180 (77%)	.003
Comorbidities – n (%)
Hypertension	1379	1057 (77%)	857 (75%)	200 (85%)	.001
Diabetes		477 (35%)	367 (32%)	110 (47%)	<.001
Cardiovascular disease		178 (13%)	118 (10%)	60 (26%)	<.001
Cancer		71 (5.1%)	54 (4.7%)	17 (7.2%)	.2
Liver disease		53 (3.8%)	45 (3.9%)	8 (3.4%)	.8
Pulmonary disease		46 (3.3%)	38 (3.3%)	8 (3.4%)	>.9
Autoimmune disease		39 (2.8%)	34 (3.0%)	5 (2.1%)	.6
Neurology disease		16 (1.2%)	13 (1.1%)	3 (1.3%)	.7
Without comorbidities		147 (11%)	137 (12%)	10 (4.3%)	<.001
Smoking – n (%)
Never	1379	900 (65%)	765 (67%)	135 (57%)	.021
Previous		243 (18%)	191 (17%)	52 (22%)
Currently		236 (17%)	188 (16%)	48 (20%)
ACE or ARB use – n (%)	1359	928 (67%)	779 (68%)	149 (63%)	.3
Immunosuppression – n (%)
CNI	1370	1096 (80%)	910 (80%)	186 (80%)	>.9
MPAA or AZA	1370	1043 (76%)	862 (76%)	181 (78%)	.5
mTORi	1350	204 (15%)	184 (16%)	20 (8.7%)	.004
Steroids	1379	1292 (94%)	1072 (94%)	220 (94%)	>.9
eGFR baseline (mL/min/1.73 m2)	1229	47 (31, 64)	50 (33, 66)	39 (24, 53)	<.001

Abbreviations: ACE, angiotensin-converting enzyme inhibitors; ADPKD: autosomal dominant polycystic kidney disease; ARB, angiotensin II receptor blockers; AZA, azathioprine; BMI, body mass index; CKD, chronic kidney disease; CNI, calcineurin inhibitors; eGFR, glomerular filtration rate estimated by CKD-EPI; MPAA, mycophenolate acid analogs; mTORi, mammalian target of rapamycin inhibitors.

TABLE 2.

Clinical presentation of COVID-19: symptoms and signs

Symptoms or signs – n (%)	Overall N = 1379	Survivors N = 1144	Non-survivors N = 235	p-value
Fever and/or chills	848 (62%)	720 (63%)	128 (54%)	.017
Fever	830 (60%)	704 (62%)	126 (54%)	.027
Chills	424 (31%)	358 (31%)	66 (28%)	.4
Cough	741 (54%)	614 (54%)	127 (54%)	>.9
Dyspnea	546 (40%)	393 (34%)	153 (65%)	<.001
Myalgia	556 (40%)	490 (43%)	66 (28%)	<.001
Headache	320 (23%)	292 (26%)	28 (12%)	<.001
Hypoxemia	195 (14%)	126 (11%)	69 (29%)	<.001
Nasal congestion	154 (11%)	139 (12%)	15 (6.4%)	.014
Sore throat	114 (8.3%)	108 (9.5%)	6 (2.6%)	<.001
Expectoration	47 (3.4%)	37 (3.2%)	10 (4.3%)	.6
Coryza	232 (17%)	210 (18%)	22 (9.4%)	.001
Chest pain	62 (4.5%)	52 (4.6%)	10 (4.3%)	>.9
Anosmia	323 (23%)	295 (26%)	28 (12%)	<.001
Ageusia	110 (8.0%)	98 (8.6%)	12 (5.1%)	.10
Fatigue, and/or adynamia, and/or asthenia	256 (19%)	225 (20%)	31 (13%)	.025
Diarrhea	441 (32%)	370 (32%)	71 (30%)	.6
Nausea and/or vomiting	120 (8.7%)	105 (9.2%)	15 (6.4%)	.2
Arthralgia	25 (1.8%)	24 (2.1%)	1 (0.4%)	.10
Conjunctivitis	3 (0.2%)	3 (0.3%)	0 (0%)	>.9
Rash	3 (0.2%)	3 (0.3%)	0 (0%)	>.9

Note: Missing values for the whole population and each symptom or sign: 2.

Hospitalization for clinical management was required for 73% of patients, 40% of them in an ICU. The rates of invasive mechanical ventilation and KRT were 29% and 27%, respectively. Two hundred and thirty-five (17%) patients died up to 28 after the diagnosis. Several demographic differences were observed when patients who died were compared with survivors (Table 1). Non-survivors were older (p < .001), most frequently had chronic kidney disease due to diabetes (p < .001), and had received a graft from a deceased donor (p = .003). Among them, the frequency of hypertension (p = .001), diabetes (p < .001), and previous cardiovascular events (p < .001) were more frequent, and smoking (p = .02). The use of mTOR inhibitor was more frequent in patients who survived (p = .004), whereas the baseline graft function was higher (p < .001).

Similarly, some differences were observed in the clinical presentation of COVID-19 (Table 2). The following symptoms or signs were most frequent among the survivors: fever and/or chills (p = .017), myalgia (p < .001), coryza (p = .001), sore throat (p < .001), anosmia (p < .001), and headache (p < .001). On the other hand, dyspnea (p < .001) and hypoxemia (p < .001) were significantly most frequent among patients who died.

3.2. Development of model prediction risk for COVID-19 associated mortality

The patients were grouped randomly in two cohorts: the derivation cohort or train set (n = 1035, 75%) and the internal validation cohort or test set (n = 344, 25%). A more detailed diagram flow depicting the cohort split is presented in Figure S1. Among all variables of interest, the number of recipients with chronic kidney disease due to diabetes (p = .026) and the presence of diabetes as comorbidity (p = .005) were higher in the internal validation cohort (Table S1).

All candidate predictors were fitted in a predictive model named here as the full model (n = 36 predictors, Table S2). A reduced model using feature selection aimed to retain only the most important predictors was analyzed, named here as the reduced model (n = 15 predictors, Table S3).

In a first step, several candidate models were fitted with 10-fold cross-validation and the performance of these full and reduced models were analyzed throughout the AUC-ROC curves in the derivation cohort. In the full model, the AUC were 0.753 and 0.783 for XGBoost and Elastic Net, respectively, whereas, in the reduced model, the AUC were 0.788 and 0.776, respectively. In a second step, the performance of these models was tested in the internal validation cohort. In the full models, the AUC were 0.766 and 0.750 for XGBoost and Elastic Net, respectively, whereas for reduced models they were 0.764 and 0.767, respectively ( Table 3). In the calibration, full and reduced XGBoost models achieved a Brier score of 0.358 and 0.319, respectively, whereas, for full and reduced Elastic Net, it was 0.128 and 0.119, respectively (Table 3). The calibrated model, optimism corrected model using logistic calibration, and nonparametric calibration are depicted in Figure 2, and detailed calibration information is presented in Table S5.

TABLE 3.

Performance metrics and calibration of COVID-19 mortality models in derivation and in the first validation cohorts

Model	AUC-ROC		Calibration Brier score Internal validation cohort (n = 344)
	Derivation cohort (n = 1035)	Internal validation cohort (n = 344)
XGBoost full	0.753 (0.724–0.798)	0.766 (0.704–0.835)	0.358
XGBoost reduced	0.788 (0.745–0.801)	0.764 (0.706–0.823)	0.319
Elastic net full	0.783 (0.751–0.827)	0.750 (0.672–0.827)	0.128
Elastic net reduced	0.776 (0.745–0.804)	0.767 (0.698–0.834)	0.119

Note: 95% Confidence intervals (in parentheses) are based on 2000 bootstrap resamples.

FIGURE 2.

Calibration plot of COVID-19 mortality models in the validation cohort: (A) XGBoost full model, (B) XGBoost reduced model, (C) Elastic Net full model, (D) Elastic Net reduced model. Gray line represents perfectly calibrated model, solid black line represents optimism corrected model using logistic calibration, and doted black line represents optimism corrected model using nonparametric calibration

To choose the most useful model, AUC-ROC values of XGBoost and Elastic Net were additionally plot, as shown in Figure 3, and a confusion matrix of 28-day fatality in the derivation cohort, shown in Figure 4. As it is depicted in the red line of Figure 3, the reduced Elastic Net showed a good discrimination ability for COVID-19 mortality with an AUC of 0.767 (95% CI 0.698–0.834).

FIGURE 3.

AUC-ROC in the derivation cohort of COVID-19-associated death. The red line represents the ROC curve of the reduced Elastic Net, which achieved the best performance to predict 28-day mortality in the derivation cohort: 0.767 (95% CI 0.698–0.834) [Color figure can be viewed at wileyonlinelibrary.com]

FIGURE 4.

Confusion matrix of 28-day COVID-19-associated death in the derivation cohort. The lower number of patients for whom the model did not predict the outcome but it occurred in the real life was achieved by the reduced Elastic net (n = 15) [Color figure can be viewed at wileyonlinelibrary.com]

For sensitivity analysis, the first validation cohort was split considering four scenarios: type of donor, the time between transplantation and COVID-19 diagnosis, patients’ allocation for treatment, and type of center, according to the volume of enrolled patients. As shown in Table 4, the accuracy of the model assessed by AUC-ROC ranged from 0.706 to 0.788 for different scenarios.

TABLE 4.

Sensitivity analysis of COVID-19 mortality models in the first validation cohort

Groups	AUC-ROC First validation cohort
All cohort (n = 344)	0.767 (0.698–0.834)
Type of donor
Living (n = 98)	0.706 (0.558–0.853)
Deceased (n = 246)	0.788 (0.711–0.865)
Time between transplant and COVID−19 diagnose
More than 1 year (n = 291)	0.775 (0.700– 0.849)
Less than 1 year (n = 53)	0.753 (0.554–0.952)
Allocation for treatment
In-hospital (n = 265)	0.784 (0.617–0.952)
Domiciliary (n = 79)	0.762 (0.683–0.842)
Type of center (number of patients enrolled)
High volume (n = 152)	0.762 (0.663–0.862)
Low volume (n = 137)	0.763 (0.627–0.897)
Time between transplant and COVID-19 diagnosis
More than 1 year (n = 291)	0.775 (0.700– 0.849)
Less than 1 year (n = 53)	0.753 (0.554–0.952)

Note: Center was considered as high volume if the number of patients enrolled was higher than 100, and low if the number was lower than 50. For this analysis, centers with mild volume (between 50 and 100) were not included (55 patients). 95% Confidence intervals (in parentheses) are based on 2000 bootstrap resamples.

3.3. Making a score-based prediction

The results of reduced Elastic Net showed that age, hypertension, previous cardiovascular disease, higher BMI, use of mycophenolate acid analogs or azathioprine, and presence of dyspnea were related to a worse outcome. The higher baseline eGFR, use of mTOR inhibitor, longer time of COVID-19 symptoms onset, presence of anosmia, and coryza were related to a better outcome. These results are depicted in Figure 5 and detailed in Table S6. The coefficients of the Elastic Net model were used to build the ImAgeS score.

FIGURE 5.

Coefficients of Elastic Net of COVID-19-associated death model. The plot represents the variable importance. The red bars represent the variables related to the probability of death, whereas the blue bars were related to the probability of surviving. The model was fitted with 15 predictors and natural splines in the variables age and eGFR were derived. The natural splines computed a different risk for each stratum aiming to capture the non-linear association between these predictors and outcome. AZA, azathioprine; BMI, body mass index; ESKD, end stage kidney disease; DM, diabetes mellitus; eGFR, estimated glomerular filtration rate; MPAA, mycophenolate acid analogs; mTOR, mammalian target of rapamycin [Color figure can be viewed at wileyonlinelibrary.com]

3.4. Results from the second validation cohort

A second validation cohort was composed of 374 patients who had the COVID-19 diagnosed in 2021 (from January to April). The baseline data and symptoms/signal of COVID-19 are shown in Table S7. For these patients, the hospitalization rate, ICU, and mechanical ventilation requirement were 65%, 34%, and 30%, respectively. The 30-day fatality rate was 22%. The reduced Elastic Net achieved an AUC-ROC of 0.787 (0.731–0.843), which was not different from the derivation and the first internal validation cohorts (Table S8).

3.5. Performances of models derived from the general population in transplanted patients

The performances of three derived from the general population models were evaluated in our population. The results are shown in Table 5. The sensitivity, specificity and AUC-ROC were, respectively: 0.84, 0.25, and 0.62 for CHA2DS2-VASc score; 0.93, 0.21, and 0.68 for model derived from Wuhan’s cohort; and 0.86, 0.37, and 0.69 for COVID SEIMC score. Therefore, all of them resulted in low specificity and lower AUC values for KT recipients, underperforming the ImAgeS score. Details are summarized in Table S9.

TABLE 5.

Performances of models derived from the general population in transplanted patients

Scores	Sensitivity	Specificity	PPV	NPV	AUC-ROC (95% CI)
CHA2DS2-VASC	0.84	0.25	0.88	0.18	0.62 (0.598–0.654)
Wuhan model	0.93	0.21	0.87	0.34	0.68 (0.651–0.711)
COVID SEIMC	0.86	0.37	0.88	0.37	0.69 (0.654–0.728)
Images score	0.72	0.63	0.90	0.31	0.76 (0.698–0.834)

Note: The ImAgeS score metrics were performed in the first validation cohort.

Abbreviations: AUC-ROC, area under curve of receiving operator curve; CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value.

3.6. Practical application

The ImAgeS score could be used to predict the probability of death for each KT recipient using predictors easily available at the time of COVID-19 diagnosis. Examples of predictions for four different hypothetical patients are showing in Table 6. Patients 1 and 2 are the same age (40 years old), however, patient 2 has a higher BMI and lower baseline eGFR. The immunosuppressive regimen is different, as well as the first COVID-19 symptoms and the onset time. In these scenarios, the first patient has a low probability of death, 3.5% (RR = 0.04), and must be followed at home by remote call appointments. On the other hand, the second one has a 67.8% probability of death (RR = 2.11) and must have an in-person clinical evaluation and should be considered for hospitalization. For patients 20 years older (patients 3 and 4), the probability of death increased to more than 70%, and the relative risk of death was higher than 3 and 4, respectively. They must have a presential clinical evaluation. For better demonstration, the contribution and importance of each predictor are visualized in a SHAP plot, shown in Figure 6. Finally, a web app to estimate the individual probability for a point of care decision was developed, and it is available at: https://covidmodels.shinyapps.io/COVID_score_app/

TABLE 6.

COVID-19 mortality prediction (ImAgeS score) in four hypothetical kidney transplant recipients

	Patient 1	Patient 2	Patient 3	Patient 4
Demography
Age (years)	40	40	60	60
Diabetes as CKD etiology	No	No	No	Yes
Hypertension as comorbidity	Yes	Yes	Yes	Yes
Previous cardiovascular disease	No	No	No	Yes
Smoking	No	No	No	No
BMI (kg/m2)	24	35	25	30
eGFR (ml/min/1,73m2)	60	20	50	40
Immunosuppression
Steroid	Yes	Yes	Yes	Yes
MPA or AZA	No	Yes	Yes	Yes
mTORI	Yes	No	No	No
Symptoms
Time of COVID−19 symptoms (days)	5	2	5	6
Dyspnea	No	Yes	Yes	Yes
Anosmia	Yes	No	No	No
Headache	No	No	No	No
Diarrhea	No	No	No	No
Predictions
Probability 28 days death	3.5%	67.8%	78.0%	82.0%

Abbreviations: AZA, azathioprine; BMI, body mass index; CKD, chronic kidney disease; COVID-19, coronavirus disease 2019; eGFR, glomerular filtration rate estimated by CKD-EPI; MPA, mycophenolate; mTORI, mammalian target of rapamycin inhibitors.

FIGURE 6.

Shapley Additive Explanations (SHAP plot) showing the contribution of each predictor in COVID-19-associated death score in simulated transplant patients. The red bars represent variables with a positive coefficient that means a positive association between the predictor and the outcome, while the blue bars represent variables with a negative coefficient that means an inverse association between the predictor and the outcome. eGFR, estimated glomerular filtration rate; mTOR, mammalian target of rapamycin [Color figure can be viewed at wileyonlinelibrary.com]

4. DISCUSSION

In this study, we presented a model to predict 28-day COVID-19-associated fatality among KT recipients based on easily available information. Considering the current burden of health care services, this tool might help to screen through phone call the patients who need more intensive monitoring.

The predictors of death after COVID-19 have already been established for non-transplanted population, such as advanced age, high BMI, presence of diabetes, hypertension, and cardiovascular disease.^{8 , 25} Risk factors for death were also previously explored for KT recipients,^{26 , 27} but no study focused on the baseline and initial clinical presentation, enabling to stratify the patient into risk groups.28, 29, 30 In our analyses, two variables should be pointed out owing to the particularities of this group of patients: the important impact of baseline graft function and the association between maintenance immunosuppressive regimen and death.

First, reduced baseline kidney function has been associated with poor outcome in the course of COVID-19 in the general population. For instance, in a national cohort study carried out in England that included more than 17 million patients, the risk of death was increased by 33% when the baseline eGFR (estimated by CKD-EPI) between 30 and 60 was compared to eGFR >60 ml/min/1.73m², whereas this risk more than doubled when eGFR was lower than 30.³¹ Although the association between baseline eGFR and unfavorable outcomes has been frequently described in several scenarios,^{32 , 33} it has not been consistently demonstrated in the COVID-19 infection. Second, it is still unclear whether immunosuppressive drugs impact on COVID-19-related signs and symptoms and outcomes.³⁴ Similarly, despite the well-known beneficial effects of corticosteroids on the management of the severe forms of COVID-19,³⁵ its effect on patients who are chronically under corticosteroids has not been established.³⁶ In our analyses, the use of mycophenolate acid analogs or azathioprine was associated with higher fatality risk while the use of mTOR inhibitors was protective. Some hypothesis to explain the negative impact of antiproliferative drugs on outcomes were the commonly associated lymphopenia, a known risk factor for COVID-related death,^{37 , 38} and the potential impairment in the development of neutralizing antiviral antibodies. In contrast, in vitro studies have suggested that SARS-CoV-2 replication depends on the Akt/mTOR/HIF-1 pathway, potentially explaining the protective effect of chronic use of mTOR inhibitors.39, 40, 41

Four initial symptoms were included in the prediction model: anosmia, headache, and coryza were associated with better outcomes, while dyspnea was associated with the risk of death. The typical COVID-19 symptoms, such as fever, dry cough, myalgia, fatigue, and anorexia⁴² were not discriminant. Anosmia, which could be present in half of the infected patients,⁴³ has been previously associated with a better outcome resulting in lower COVID-19 mortality in the general population.⁴⁴ The reason why upper respiratory symptoms are associated with favorable outcome is not clear. However it is possible that these typical flu or flu-like symptoms drive the perception of the disease, while asymptomatic hypoxia is associated with poor outcome, and infected patients who are feeling well or only slightly ill can suddenly progress to severe respiratory impartment within a few hours.⁴⁵ Additionally, the shorter time from COVID-19 symptoms associated with death suggests that the longer time between first symptoms and the requirement for in-person medical evaluation is a predictor of less aggressive disease.⁴⁶

Previous published prediction models were developed in the general population. Most of them included physical examination findings, laboratory, and chest radiological exams.47, 48, 49 Distinctly, our purpose was to construct a model including only information easily available before the presential medical evaluation. This tool can be useful in the decision-making process regarding timely presential appointments, hospital admissions, and clinical management, minimizing unnecessary medical visits, and enabling stratifying patients to closer remote monitoring. Importantly, the ImAgeS score achieved the optimal discriminative capacity to detect patients with a high probability of death within 28 days.

Our study has important strengths that should be emphasized. The data of the large number of patients were extracted from the COVID-19 KT Brazilian study. Brazil has the largest public transplant program in the world,⁵⁰ and the country has been dramatically affected by the pandemic since March 2020. Furthermore, the use of machine learning principles to fit different models, the internal validation in a cohort independent from those that were used to fit the model, validation in a second cohort, and the calibration⁵¹ contributed to improving the robustness and quality of the ImAgeS Score. The final model was developed through the generalized linear model Elastic Net, and the regularization path was computed for the LASSO penalty at a grid of values for the regularization parameter lambda.⁵²

Despite the start of immunization, COVID-19 is still a concern. First, the vaccination rollout is limited in low- and mild-income owing to the shortage of vaccines. Second, even in countries where vaccines are wildly available, the rate of vaccine refusal is relevant.⁵³ Last, some initial evidence has suggested that the humoral response to vaccines in KT recipients is lower than non-transplanted; consequently, the effectiveness of vaccination for this population can be disappointed.^{54 , 55} Therefore, a tool for early identification of cases with potential for unfavorable outcomes explicitly fitted and validated for kidney transplanted patients is valuable.

Although, to date, it is the largest cohort of KT recipients diagnosed with COVID-19 to date, some limitations should be pointed out. Being a multicenter and historical study, some regional variations in the clinical management are expected. Owing to its retrospective nature, some information was missing, although this amount was extremely low considering the total number of patients included. The present analysis focused on predictors of death in an acute scenario of infection, the COVID19-associated severe acute respiratory acute syndrome. Despite the well-known association between donor parameters, anti-HLA donor-specific antibody, proteinuria, and acute rejection with long-term clinical outcomes, we believe that baseline graft function is a suitable proxy in our analysis, confirmed by the robust association between baseline graft function and the outcome. Additionally, we acknowledge that assuming the strategy of using only rapidly accessible parameters, which was thought to be used in the remote assistance, without the need for biochemical or scale-based predictors, our study ultimately lacks some basic determinants of death. For instance, biological and physiological predictors strongly associated with COVID-19-associated mortality in the general population, such as Glasgow coma scale, C-reactive protein, D-dimer, and neutrophil/lymphocyte ratio, were not included in the analysis. Yet, validated scores for the general population that included these parameters did not outperform the ImAgeS Score. Finally, the predictive models had a primary aim in prediction with lower explanatory capacity compared to classic statistical analysis, which could reduce the inferential conclusions. Thus, additional studies are required to determine the impact of specific immunosuppressive agents on the outcome of COVID-19.

In conclusion, the factors associated with higher fatality in KT recipients were similar to the general population. Some clinical symptoms at baseline such as anosmia and coryza had a better prognosis. Baseline immunosuppression could predict the outcome. The use of machine learning techniques allowed the development of a predictive model with good accuracy, easily applicable using demographics and symptoms. Its application in triage can indicate patients that require observation or more intensive monitoring.

Funding information Novartis Pharma Brazil; Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – Brasil (CAPES), Grant/Award Number: 88881.507066/2020-01

APPENDIX

Beyond the authors, the COVID-19-KT Brazil Study Group includes the following participants: Elizete Keitel, MD, PhD (Santa Casa de Misericórdia de Porto Alegre, Porto Alegre, RS, Brazil); Claudia Maria Costa de Oliveira, MD, PhD (Hospital Universitário Walter Cantídio, Fortaleza, CE, Brazil); Beatriz de Oliveira Neri, MD (Hospital Geral de Fortaleza, Fortaleza, CE, Brazil); Ida Maria Maximina Fernandes Charpiot, MD, PhD (Hospital de Base de SJRP, SJRP, SP, Brazil); Teresa Cristina Alves Ferreira, MD, PhD (Hospital Universitário da UFMA, São Luis, MA, Brazil); Alessandra Rosa Vicari, RN, MsC (Hospital de Clínicas de Porto Alegre, Federal Univertisy of Rio Grande do Sul, Porto Alegre, RS, Brazil); Tomás Pereira Júnior, MD (Hospital Universitário Onofre Lopes, Natal, RN, Brazil); Maria Eduarda Heinzen de Almeida Coelho, MD (Hospital Santa Isabel, Blumenau, SC, Brazil); Marilda Mazzali, MD, PhD (University of Campinas – UNICAP, Scholl of Medical Sciences, Division of Nephrology, Renal Transplant Unit, Renal Transplant Research Laboratoy, Campinas, SP, Brazil); Gustavo Fernandes Ferreira, MD, PhD (Santa Casa de Misericórdia de Juiz de Fora, Juiz de Fora, MG, Brazil); Juliana Bastos Campos, MD (Santa Casa de Misericórdia de Juiz de Fora, Juiz de Fora, MG, Brazil); Nicole Gomes Campos Rocha, MD (Hospital de Base de Brasília, Brasilia, DF, Brazil); Anita Leme da Rocha Saldanha, MD (Hospital Beneficência Portuguesa, São Paulo, SP, Brazil); Tania Leme da Rocha Martinez, MD, PhD (Hospital Beneficência Portuguesa, São Paulo, SP, Brazil); João Egídio Romão Junior, MD, PhD (Hospital Beneficência Portuguesa, São Paulo, SP, Brazil); Maria Regina Teixeira Araújo, MD, PhD (Hospital Beneficência Portuguesa, São Paulo, SP, Brazil); Sibele Lessa Braga, MD (Hospital Beneficência Portuguesa, São Paulo, SP, Brazil); Luciane Mônica Deboni, MD, MsC (Hospital Municipal São José (HMSJ), Joinville, SC, Brazil); Franco Silveira da Mota Krüger, MD (Hospital Municipal São José de Joinville, Fundação Pró-Rim, Joinville, SC, Brazil); Miguel Moysés Neto, MD, PhD (Division of Nephrology, School of Medicine of Ribeirão Preto, University of Sao Paulo, Ribeirão Preto, SP, Brazil); Auro Buffani Claudino, MD, MSc (Hospital Santa Marcelina, São Paulo, SP, Brazil); Lívia Cláudio de Oliveira, MD (Hospital Universitário de Brasília, University of Brasilia – UnB, DF, Brazil); Tereza Azevedo Matuck, MD (Hospital São Francisco na Providência de Deus, Rio de Janeiro, RJ, Brazil); Alexandre Tortoza Bignelli, MD, MSc (Hospital Universitário Cajuru, Curitiba, PR, Brazil); Silvia Regina Hokazono, MD (Hospital Universitário Cajuru, Curitiba, PR, Brazil); José Hermógenes Rocco Suassuna, MD, PhD (Hospital Universitário Pedro Ernesto, Rio de Janeiro, RJ, Brazil); Suzimar da Silveira Rioja, MD, PhD (Hospital Universitário Pedro Ernesto, Rio de Janeiro, RJ, Brazil); Rafael Lage Madeira, MD (Hospital Felicio Rocho, Belo Horizonte, MG, Brazil); Sandra Simone Vilaça, MD (Hospital Felicio Rocho, Belo Horizonte, MG, Brazil); Carlos Alberto Chalabi Calazans, MD, PhD (Hospital Marcio Cunha, Ipatinga, MG, Brazil); Daniel Costa Chalabi Calazans, MD (Hospital Marcio Cunha, Ipatinga, MG, Brazil); Patrícia Malafronte, MD, PhD (Santa Casa de Misericórdia de São Paulo, São Paulo, SP, Brazil); Antonio Miorin, MD, PhD (Santa Casa de Misericórdia de São Paulo, São Paulo, SP, Brazil); Filipe Carrilho de Aguiar, MD, MSc (Universidade Federal de Pernambuco, Recife, PE, Brazil); Larissa Guedes da Fonte Andrade, MD, MSc (Universidade Federal de Pernambuco, Recife, PE, Brazil); Fabiana Loss de Carvalho, MD (Hospital do Rocio, Curitiba, PR, Brazil); Karoline Sesiuk Martins, MD (Hospital do Rocio, Curitiba, PR, Brazil); Hélady Sanders Pinheiro, MD, PhD (Hospital Universitário da UFJF, Juiz de Fora, MG, Brazil); Emiliana Spadarotto Sertório, MD (Hospital Universitário da UFJF, Juiz de Fora, MG, Brazil); André Barreto Pereira, MD, PhD (Hospital Marieta Konder Bornhausen, Itajai, SC, Brazil); David José Barros Machado, MD, PhD (Hospital Alemão Osvaldo Cruz, São Paulo, SP, Brazil); Carolina Maria Pozzi, MD (Hospital Universitário Evangélico de Curitiba, Curitiba, PR, Brazil); Leonardo Viliano Kroth, MD, PhD (Hospital São Lucas, Porto Alegre, RS, Brazil); Lauro Monteiro Vasconcellos Filho, MD, PhD (Hospital Meridional, Cariacica, ES, Brazil); Rafael Fabio Maciel, MD, MSc (Hospital Nossa Senhora das Neves, João Pessoa, PB, Brazil); Amanda Maíra Damasceno Silva, MD (Hospital Antônio Targino, Campina Grande, PB, Brazil); Ana Paula Maia Baptista, MD, MsC (Hospital São Rafael, Salvador, BA, Brazil); Pedro Augusto Macedo de Souza, MD (Santa Casa de Belo Horizonte, Belo Horizonte, MG, Brazil); Marcus Faria Lasmar, MD, PhD (Hospital Universitário Ciências Médicas, Belo horizonte, MG, Brazil); Luciana Tanajura Santamaria Saber, MD, PhD (Santa Casa de Ribeirão Preto, Ribeirão Preto, SP, Brazil); Lilian Monteiro Pereira Palma, MD, PhD (Centro Médico de Campinas, Campinas, SP, Brazil); Ricardo Augusto Monteiro de Barros Almeida, MD, PhD (Department of Internal Medicine, Universidade Estadual Paulista-UNESP, Botucatu, Brazil).

SUPPORTING INFORMATION

Additional supporting information may be found online in the Supporting Information section.

Supplementary Material