Unraveling the impact of abdominal arterial calcifications on kidney transplant waitlist mortality through ensemble machine learning

Abstract

The scarcity of organ donors relative to the number of patients with End Stage Kidney Disease (ESKD) has led to prolonged waiting times for kidney transplants, contributing to elevated cardiovascular mortality risk. Transplant professionals are tasked with the complex allocation of limited organs to a vulnerable patient group facing heightened morbidity and mortality risk. The need for continuous re-evaluation of waitlisted patients is evident due to the significant number who perish while awaiting transplantation. Among individuals with ESKD, vascular calcification, particularly Abdominal Arterial Calcifications (AAC), holds predictive value for cardiovascular events and mortality. However, a standardized method to quantify AAC’s prognostic potential remains lacking, especially for kidney transplant evaluations. This study presents an ensemble machine learning (ML) approach to study the relationship between AAC score and mortality in patients on the waitlist and triage patients needing transplantation. Using the AAC score, the proposed ML model can predict kidney transplant waitlist morality with an accuracy of 78% while its accuracy is 68% without using this score. This study leverages explainable ML to explore the relationship between predictors and mortality in waitlisted patients, aiming to improve patient triage accuracy.

Introduction

Fewer organ donors are available than the patients with End Stage Kidney Disease (ESKD)¹. As of January 2023, more than 88,000 patients were on the waiting list for kidney transplantation in the USA¹. This reality contributes to prolonged waiting periods before obtaining a kidney transplant and an increased risk of cardiovascular mortality linked to ESKD^2,3. Transplant professionals are confronted with the challenging responsibility of allocating scarce organs to a susceptible patient population facing increased risk of morbidity and mortality⁴. Moreover, continuous re-evaluation of these waitlisted patients is essential to ensure their ongoing suitability, especially considering the substantial number who die while waiting, never having received an organ^5,6.

Vascular calcification is common among individuals with ESKD and is independently predictive for cardiovascular incidents and mortality^7,8. Specifically, the Abdominal Arterial Calcifications (AAC) is a prognostic factor for survival among patients with ESKD^9–11. However, no standardized methods to quantify the calcification score value of the large abdominal vessels have been widely applied and used for risk stratification when ESKD patients are being evaluated for kidney transplants. To create such a score, we modified the coronary calcium score software to quantify the AAC and generate a calcification score value of the abdominal large vessels. Previous works have identified scoring of coronary calcium using software to be accurate and reliable^12,13. Further, these scores have been used to assess cardiovascular disease progression risk and related outcomes such as mortality¹⁴.

We hypothesized that applying a standardized AAC scoring for abdominal vasculature using available pre-transplant CT scans would serve as a novel radiographic biomarker to assist transplant centers in the evaluation and recurring triage of kidney transplant candidates and waitlist patients. The first aim of this study is to utilize explainable machine learning (ML) as a function to investigate the relationship between the AAC score and mortality in kidney waitlist patients by training and validating ML models with and without the AAC score as a predictor. The second aim is to show that an ensemble of ML models can reduce bias and increase accuracy in identifying kidney transplant waitlist patients at risk of mortality.

Method

Clinical setting and study design

Our Institutional Review Board approved (ID 21-005015) a retrospective study focusing on patients from 2005 to 2015 at Mayo Clinic Florida. The average wait time for a kidney transplant during this period was approximately 5 years. We chose to study a decade’s worth of data to ensure we captured sufficient time for all patients in the study to reach the primary endpoints of either survival or death while on the waiting list. Data were collected using the electronic medical record system. Exclusion criteria included patients actively listed for combined organ transplants and/or those listed for a second kidney transplant. Inclusion criteria consisted of the following: (1) all adults (greater than 18 years old) approved for a kidney transplant and placed on the transplant waiting list, (2) had a non-contrast abdominal and pelvic CT done within 2 years of listing. Exclusion criteria consisted of patients who were actively listed for combined organ transplants and/or listed for a second kidney transplant. Patients were divided into 2 groups: (i) Group 1: died on the waiting list (DWL), (ii) Group 2: survived on the waiting list (SWL).

The outcome of interest is prediction of overall patient survival based on the AAC score and other predictor variables. These variables are AAC score, age, sex, body mass index (BMI), hypertension (HTN), coronary artery disease (Myocardial infarction (MI), Angioplasty, coronary artery bypass graft (CABG)), diabetes mellitus (treated with oral medication, or insulin), cholesterol levels (low-density lipoprotein (LDL), or high-density lipoprotein (HDL)), triglyceride, left ventricular ejection fraction (LVEF) percent, history of neurologic event (cerebral vascular accident (CVA), transient ischemic attack (TIA)), albumin, dialysis status, tobacco history, troponin, and hemoglobin.

A total of 488 patients on the kidney transplant waitlist from 2005 to 2015 met the study criteria in this single-center study. Table 1 shows 151 patients died on the waiting list (DWL group), and 337 patients survived on the waiting list (SWL group) and received kidney transplantation during the follow-up period.

Table 1.

This table provides an overview of the patient characteristics used in the study.

Predictors	Type	Categories	Died on waitlist (DWL) (N = 151)	Received transplant (SWL) (N = 337)
Age at listing (years)	Continuous	–	(27.00, 59.02, 79.00)	(19.00,53.37,78.00)
Sex	Discrete	Male	102	200
		Female	49	137
BMI	Continuous	-	(18.83,28.72,47.42)	(16.00,29.72,44.00)
HTN	Discrete	Positive	134	298
		Negative	17	39
Diabetes Oral	Discrete	Positive	20	9
		Negative	131	328
Diabetes Insulin	Discrete	Positive	56	70
		Negative	95	267
Cholesterol LDL	Continuous	–	(26.00,91.49,241.00)	(23.00,95.27,230.00)
Cholesterol HDL	Continuous	–	(24.00,51.70,155.00)	(18.00,50.50,133.00)
Triglyceride	Continuous	–	(36.00,162.14,1054.00)	(33.00,158.92,1179.00)
Albumin	Continuous	–	(2.30,4.01,5.10)	(2.50,4.11,5.10)
Troponin	Continuous	–	(0.01,0.07,0.48)	(0.01,0.05,0.86)
Hemoglobin	Continuous	–	(7.20,11.28,15.70)	(5.30,11.27,19.40)
LVEF %	Continuous	–	(18.00,61.89,77.00)	(25.00,62.2,82.00)
Current or Former smoker/Tobacco user at time of listing	Discrete	Positive	64	129
		Negative	87	208
MI	Discrete	Positive	20	23
		Negative	131	314
Stent	Discrete	Positive	19	23
		Negative	132	314
CABG	Discrete	Positive	19	8
		Negative	132	329
CVA	Discrete	Positive	17	12
		Negative	134	325
TIA	Discrete	Positive	11	7
		Negative	140	330
Dialysis at time of listing	Discrete	Positive	119	201
		Negative	32	136
AAC Score	Continuous	-	(0.00,8328.03,38061.70)	(0.00,3296.13,36421.30)

The patients were divided into two groups: those who died on the waiting list (DWL) and those who survived on the waiting list (SWL) and received kidney transplantation during the follow-up period. The table presents various predictor variables and their impact on predicting patient outcomes.

Abdominal arterial calcification score system

At our transplant center, the kidney transplant evaluation protocol includes a non-contrast abdominal and pelvic CT for every patient with either diabetes (regardless of age) or those older than 50 years of age. The most recent non-contrast abdominopelvic CT performed within 2 years before the listing date was selected for analysis. Images were obtained from Visage’s picture archiving and communication system (version 7.1.14, Visage Imaging, Inc., San Diego, CA). A modified Agatston method to measure coronary calcifications was applied to generate a scoring system based on the abdominal arteries (aorta, common iliac, external iliac, and internal iliac)^11,15. This was completed by marking arterial calcification with a semiautomated software package (syngo.via Client 5.1, Siemens Healthcare, Erlangen, Germany or TeraRecon Aquarius iNtuition ver 4.4.13.P6, Durham, North Carolina) designed for quantifying coronary artery calcium by the Agatston method. In brief, the software automatically identifies calcium using a density threshold of 130 Hounsfield units (HU) or higher and measures the area of each calcification. The software calculated the score by multiplying each area of calcification by an established weighted value determined from the maximum HU in that calcification. Since the software identifies the calcium but cannot determine its anatomic location, a study researcher assigns the scored calcification to the appropriate anatomic labeling, with two study radiologists’ subsequent review of labeling. The total calcium score was calculated as the sum of the calcium scores measured in the abdominal aorta (from beneath the celiac artery origin to the iliac bifurcation), in the bilateral common iliac arteries, the bilateral internal iliac arteries (from iliac bifurcation to the first branch), and the external iliac arteries (from the iliac bifurcation to the inferior epigastric artery).

Figure 1 demonstrates a CT scan image of the abdomen, which shows our implementation of the VISAGE application for precise calcium (Ca) scoring. The scoring procedure involves the assessment of calcium deposits in the abdominal arterial system.

Fig. 1.

Coronal CT scan of the abdomen for precise determination of the AAC score. The scoring procedure involves the assessment of calcium deposits in two critical vascular regions, which are identified with a green marker (abdominal aorta) and yellow marker (common Iliac arteries).

Machine learning models

An ensemble ML model is developed to predict SWL and DWL using the variables in Table 1. This model is an ensemble of Random Forest (RF), XGBoost, and Extra-Trees classifiers trained in a supervised fashion. The continuous variables are normalized in [0,1], and the categorical variables are one-hot encoded. Each model is trained independently using the concatenation of represented variables. Outputs of the trained models are aggregated and passed to a voting mechanism to predict the outcome. The voting mechanism predicts the outcome based on the arguments of the maxima of the normalized sums of the predicted probabilities (Fig. 2).

Fig. 2.

This figure represents the ensemble machine learning model used in the study. The model combines Random Forest (RF), XGBoost, and Extra-Trees classifiers in a supervised fashion. It demonstrates how the continuous and categorical predictor variables are processed and how the models’ outputs are aggregated to predict patient outcomes.

A grid-search mechanism was used to tune the hyperparameters. The XGBoost model was trained with a step size shrinkage of 0·3, maximum depth of a tree 10, subsample ratio of the training instances 0·5, number of parallel trees 10, and number of estimators 300. The Extra-Trees classifier was trained with 100 estimators using the Gini criterion and a minimum sample split of 2 and a minimum sample leaf of 1. The RF model was trained with 100 estimators, Gini criterion, a minimum sample split of 2, and a minimum sample leaf of 1.

Evaluation and cross-validation

A 10-fold cross-validation was applied, and the average and standard deviation of performance metrics were collected. In each independent run, balanced splits of the dataset were used for training and testing. Since the dataset was naturally imbalanced, the occurrences from the not adverse outcome data class were randomly down-sampled to the number of occurrences in the adverse outcome data class. The test dataset was sampled from the balanced dataset with a 20% contribution from each data class, and the rest of the balanced dataset was used as the training dataset.

Statistical analysis

Classification performance of each model was assessed using the area under the receiver operating curve or AUC values. Bootstrap sampling with replacement with 10-fold cross-validation was used to obtain 95% CIs for all performance measures. The models were developed and evaluated using Python programming language, version 3·9 (Python Software Foundation). The threshold for statistical significance was 2-tailed with P = 0·05.

Results

Performance analysis of the models

Each ML model was trained and validated independently and compared with the ensemble mechanism, with and without using the AAC score predictor (Table 1). The results show that the AAC score significantly improves the prediction of the DWL and SWL outcomes.

The RF model has an accuracy of 0.68 ± 0.03 (95% CI [0.58,0.78]) and 0.75 ± 0.03 (95% CI [0.66,0.84]) without and with using the AAC score, respectively. The ExtraTrees model has an accuracy of 0.65 ± 0.04 (95% CI [0.55,0.75]) and 0.75 ± 0.03 (95% CI [0.66,0.84]) without and with using the AAC score, respectively. The XGBoost model has an accuracy of 0.67 ± 0.04 (95% CI [0.57,0.77]) and 0.77 ± 0.03 (95% CI [0.68,0.86]) without and with using the AAC score, respectively. The ensemble model has an accuracy of 0.68 ± 0.04 (95% CI [0.58,0.78]) and 0.78 ± 0.03 (95% CI [0.69,0.87]) without and with using the AAC score, respectively (Table 2). The ROC-AUC plots in Fig. 3 show that including the AAC score improves the AUC of RF 0.1, ExtraTree 0.04, XGBoost 0.05, and the ensemble model 0.14.

Table 2.

This table provides a comparative analysis of the performance metrics of different machine learning models used in the study.

Model	AAC Score	Accuracy	Specificity	Sensitivity	F1-Score	AUC	95% CI Acc
RF	No	0.68 ± 0.03	0.73 ± 0.03	0.63 ± 0.03	0.68 ± 0.04	0.70 ± 0.03	0.58,0.78
	Yes	0.75 ± 0.03	0.67 ± 0.05	0.83 ± 0.04	0.75 ± 0.05	0.80 ± 0.03	0.66,0.84
ExtraTrees	No	0.65 ± 0.04	0.67 ± 0.04	0.63 ± 0.05	0.65 ± 0.04	0.68 ± 0.03	0.55,0.75
	Yes	0.75 ± 0.03	0.60 ± 0.03	0.90 ± 0.02	0.74 ± 0.04	0.72 ± 0.02	0.66,0.84
XGB	No	0.67 ± 0.04	0.70 ± 0.03	0.63 ± 0.04	0.67 ± 0.04	0.68 ± 0.03	0.57,0.77
	Yes	0.77 ± 0.03	0.87 ± 0.03	0.67 ± 0.05	0.76 ± 0.04	0.83 ± 0.03	0.68,0.86
Ensemble	No	0.68 ± 0.04	0.67 ± 0.04	0.70 ± 0.04	0.68 ± 0.03	0.73 ± 0.04	0.58,0.78
	Yes	0.78 ± 0.03	0.77 ± 0.04	0.80 ± 0.03	0.78 ± 0.04	0.87 ± 0.03	0.69,0.87

The models are evaluated both with and without the inclusion of the AAC score as a predictor. The table presents key performance indicators, including accuracy, area under the receiver operating curve (ROC-AUC), specificity, and sensitivity, to assess the impact of AAC score on the prediction accuracy and balance of the models.

Fig. 3.

This figure displays the Receiver Operating Characteristic (ROC) curves for different machine learning models, both with and without the AAC score predictor. The curves show the trade-off between specificity and sensitivity in predicting patient outcomes, highlighting the improved performance when the AAC score is included.

Interpretation of the ML model

A game theoretic approach was used to interpret the output of the models (Fig. 4). Each model was trained independently with and without the AAC score. The magnitude of each predictor variable shows the impact of that variable on the model output. The AAC score has the most significant impact on all models in predicting the outcomes.

Fig. 4.

This figure provides insight into the impact of predictor variables on the machine learning models’ outcomes. It employs a game theoretic approach to visualize the magnitude of each variable’s impact, with a focus on the significance of the AAC score as a predictor.

Dialysis, hemoglobin, triglyceride, HDL, BMI, age at the time of listing, and diabetes (treated with insulin) are the top five most impactful predictors in the RF model when trained without the AAC score. When including the AAC score in training the model as an input predictor, the most impactful predictors are reordered as AAC score, diabetes (treated with insulin), dialysis, troponin, and BMI, which leads to a 7% improvement in predicting survival. The ExtraTrees model found dialysis, diabetes (treated with Oral medications), diabetes (treated with insulin), sex, and age at the time of listing as the five most important predictors. The same model trained with the AAC Score included, identifies AAC Score as the most important predictor followed by dialysis, diabetes (treated with oral medication), diabetes (treated with insulin), and age at the time of listing. Prediction accuracy and F1-Score of the ExtraTrees model were improved by 10% and 9%, respectively, by utilizing the AAC Score predictor. The top five predictors in the XGBoost model are AAC score, dialysis, BMI, troponin, and triglyceride, where the AAC score has the highest impact on the model outcome, with an accuracy of 77% and F1-score of 76%. Excluding the AAC score from input variables drops the accuracy and F1-score by about 10% and 9%, respectively. The top five predictors are dialysis, triglyceride, HDL, hemoglobin, and diabetes (treated with insulin).

Impact of ensemble learning

Analysis of the predictor variables on the predictive models showed an overlap among the ML models in identifying impactful variables. Ensemble learning incorporates multiple ML models and makes predictions based on aggregation of the models. This approach reduces bias in the decision-making process and improves generalization performance of the predictive model. Specificity and sensitivity analysis of the ML models in Fig. 5 shows that each standalone ML model (i.e., RF, ExtraTrees, and XGBoost) is biased toward a high sensitivity or specificity. For example, the ExtraTrees model has a specificity of 0.60 and a sensitivity of 0.90, while the XGBoost model has a specificity of 0.87 and a sensitivity of 0.67. The ensemble model has a more stable and unbiased predictive capability with a specificity of 0.77 and a sensitivity of 0.80. Aggregating the independent classifiers incorporated as an ensemble model is more robust with lower false positive and false negative rates.

Fig. 5.

This figure conducts a specificity and sensitivity analysis of the machine learning models, including the ensemble model. It demonstrates the bias in standalone models and how ensemble learning helps achieve a more balanced and unbiased predictive capability with lower false positive and false negative rates.

Discussion

We developed ML models to validate the importance of AAC score as a predictor variable in identifying patients at risk of mortality waiting for kidney transplantation. The RF, ExtraTrees, and XGBoost models were trained and validated independently and in an ensemble scheme on retrospective data collected from our transplant center. The proposed ensemble ML can be used as a triage system for prioritizing patients needing kidney transplantation.

Previously, medical experts have established a clear link between vascular calcification and atherosclerotic disease^16,17. Nonetheless, previous studies utilized varied methods for evaluating arterial calcification, lacking a standardized scoring system associated with ESKD patient survival^18,19. We refined the Agatston method to rectify this, resulting in a consistent imaging biomarker. This imaging biomarker can potentially assess the survival risk for kidney transplant candidates.

In light of the study’s findings, a significant question emerges concerning the practical implementation of the AAC score-based risk stratification approach in the context of kidney transplant allocation. One potential application of the AAC score-based risk stratification is the possibility of reshaping the allocation of kidney transplants. Patients with higher AAC scores, indicative of increased mortality risks, could be prioritized for transplantation, potentially improving their chances of survival post-transplant. This approach aligns with the principle of maximizing the benefit gained from each available organ and aligning it with patients who stand to gain the most^20,21. Furthermore, by expediting the transplant process for those at heightened risk, the transplant community could take proactive measures to mitigate these patients’ cardiovascular mortality risks during the waiting period^22,23.

Conversely, an alternative perspective underscores the importance of patient safety and long-term outcomes. Patients with elevated AAC scores, indicative of a greater mortality risk, might warrant a more cautious approach toward transplantation^24,25. Given that transplantation is not without its own set of risks and complications, it could be argued that for individuals with higher mortality risks, the benefits of transplantation might be outweighed by the potential complications. In such cases, alternative treatment strategies or interventions that target cardiovascular health could be explored to reduce mortality risks in these high-risk patients before transplantation^26–28. Ultimately, deciding whether to modify the transplant order or be cautious about transplantation in high-risk patients should be made carefully considering several factors. These factors include the severity of AAC scores, the overall health status of the patients, the availability of organs, ethical considerations, and the potential impact on waitlisted patients without elevated risks.

While the AAC score-based risk stratification offers a promising avenue for enhancing the effectiveness of kidney transplant allocation, the decision of how to incorporate this stratification into clinical practice is multifaceted. Individualized patient care remains paramount, and a multidisciplinary approach involving transplant professionals, clinicians, ethicists, and patients is necessary to make informed and ethical decisions^29,30. As the medical community navigates this nuanced terrain, ongoing research, and collaborative efforts will be essential in shaping a well-considered and ethically sound approach to kidney transplant prioritization based on risk stratification.

When considering the outcomes of this study, we also wish to highlight its limitations. First, as with all retrospective studies, the dependence on the completeness and accuracy of secondary data is of concern. While we have no reason to believe there are systemic inaccuracies in the data, issues such as missing data or data entry errors could potentially bias results. One of the advantages of ensemble models is less sensitivity to such issues by incorporating multiple independent models. However, ML models are prone to bias, which can lead to high specificity and low sensitivity or vice versa, which practically reduces the usability of such models in practice. Ensemble ML reduces this bias by aggregating outputs of various independently trained ML models. The proposed ensemble ML model balances the specificity and sensitivity criteria in prioritizing patients needing kidney transplantation. Further, one of the major limitations of ML models, particularly deep learning models, is the lack of explainability. To improve understanding of the models, input features visualization and importance analysis were conducted. Also, we utilized a game theoretic approach to analyze the importance of input variables in the decision-making process of the ML models. Finally, this approach to quantifying abdominal calcifications is relatively new and has not been widely adopted in clinical practice. As such, we have focused on quantifying arterial calcification in a high-risk population. Training ML models on larger datasets can potentially increase their prediction accuracy as well as generalizability. Future work is required as the models need to be evaluated on larger and multiple external sites for further generalization performance evaluation.

Conclusion

This study introduces a new strategy to tackle the complex challenges surrounding patient risk assessment while waiting for a kidney transplant. By integrating AAC scores into mortality prediction models by applying advanced ML techniques, we present a tool to enhance patient outcomes and optimize the utilization of valuable donor organs. The standardized AAC scoring system emerges as a radiographic biomarker capable of effectively assessing mortality risks in individuals afflicted by End Stage Kidney Disease (ESKD). By utilizing explainable machine learning models, including an ensemble of RF, XGBoost, and Extra-Trees classifiers, we amplify prediction accuracy and establish a framework to mitigate potential biases in decision-making processes. In summary, the AAC score is a major factor in identifying survival risk in patients needing kidney transplantation, and the ML model can identify patients at risk of death with potential prioritization of patients in urgent need of kidney transplantation.

Nevertheless, it is vital to acknowledge the multifaceted challenges associated with the real-world implementation of AAC-based risk stratification within clinical practice. Ethical considerations concerning patient prioritization and allocation strategies necessitate thorough contemplation, highlighting the indispensable role of interdisciplinary collaboration among medical practitioners, ethicists, and stakeholders. While the models showcased in this study yield promising results, the existing limitations pertaining to model interpretability and potential biases should drive further research for refinement and advancement.

Author contributions

H.S., A.C.S., and T.J. participated in designing the study and reviewing the literature. T.J. supervised the study. H.S. contributed to code writing, model training, and data analysis. H.S. prepared the code repository. H.S., A.C.S., T.H., and T.J. contributed to expert review and data interpretation. H.S. designed the figures. H.S. drafted the manuscript. H.S., A.C.S., T.H., and T.J. revised the manuscript. All authors had access to the result data presented in the final manuscript, and all authors read and approved the final manuscript. Verification and access to raw data were available to all authors. All authors agreed to submit.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Data availability

The material and dataset are not publicly available due to restrictions by privacy laws. Model output and ground truth labels are available upon reasonable request by contacting the corresponding author. The source code used in this project can be made available on reasonable request by contacting the first author.

Code availability

The code used in this study is available upon reasonable request. Due to the sensitive nature of the data and intellectual property considerations, access to the code will be granted only after the requester signs a Non-Disclosure Agreement (NDA). Requests for access to the code can be directed to the corresponding author.

Declarations

Competing interests

The authors declare no competing interests.

Statement of ethics

I confirm that the research described in this manuscript has been conducted in accordance with ethical principles outlined in the Declaration of Helsinki and has received approval from the institutional review board at our institution.

Study approval statement

The study was approved my Mayo Clinic IRB ID 21-005015.

Consent to participate

Patient consent were not required as this study was based on publicly available data. The need for informed consent was waived by the institutional review board at our institution.