Machine learning in liver surgery: Benefits and pitfalls

Abstract

The application of machine learning (ML) algorithms in various fields of hepatology is an issue of interest. However, we must be cautious with the results. In this letter, based on a published ML prediction model for acute kidney injury after liver surgery, we discuss some limitations of ML models and how they may be addressed in the future. Although the future faces significant challenges, it also holds a great potential.

Core Tip: Artificial intelligence is trending topic in healthcare research. Machine learning classifiers have been explored in the field of liver surgery and liver transplantation. However, despite of promising results, a real applicability is limited by several factors.

TO THE EDITOR

We read with interest the retrospective study by Dong et al[1] that developed a machine learning (ML) prediction model for acute kidney injury (AKI) following liver resection (LR). We thank the authors for their work and contribution in this field. LR is the first-line treatment of various liver lesions. However, the reported incidence of AKI after LR ranges from 10% to 15%[2], significantly impacting patient morbidity and mortality. Hence, identifying factors that may lead to the development of AKI is relevant. Dong et al[1] explored the potential contribution of ML classifiers to this issue.

The authors analyzed a retrospective cohort of 2450 patients and trained and validated four ML classifiers (logistic regression, random forest, support vector machine, extreme gradient boosting, and decision tree). The training methodology (10-fold cross-validation) and validation (a holdout technique with 30% patterns) were adequate. Random forest exhibited the highest performance [area under the curve (AUC) = 0.92] among the classifiers. Although the results were satisfactory, certain considerations must be addressed.

First, the rate of missing values should be reported because it can affect model training, subsequently affecting model performance and generalizability. Hence, random forest classifiers are the best algorithms for a significant rate of missing values[3]. Conversely, if this rate is low, artificial neural networks (ANNs) could offer promising dataset results. Second, several factors reported in the literature are associated with AKI after LR, such as major hepatectomy, surgery duration, hepatojejunostomy, increased Model for End-Stage Liver Disease score (MELD), and blood transfusion[2,4-8]. Among these factors, only surgery duration was included in the baseline characteristics. The inclusion of these variables may have increased the robustness of the model. Finally, performing external validation is challenging. Differences between the training and external validation cohorts may impact model accuracy. Therefore, a prospective validation may be an alternative.

Some recent studies in ML applications ranges from protein structure prediction or COVID-19 diagnosis from X-ray images to optimizing donor-recipient matching to reduce waitlist mortality or improve post-transplant outcomes[9-11]. Our experience in the field of ML in liver surgery started from liver transplantation and efforts primarily focused on improving donor-recipient matching. Using graft survival as the endpoint, we developed an ANN model that achieved an AUC of approximately 0.8212[12]. This method was validated in an external cohort and improved AUC by 15%[13]. This ANN was integrated into a rule system with the MELD score to prioritize graft allocation. Although this method was explored in the United Network of Organ Sharing database, limited results were obtained because of a significant proportion of missing values were found[14]. Dong et al[1] found that the model performance was better than the current scores for AKI prediction. Similarly, we reported the difference of ML models that outperformed traditional scores, such as MELD, Survival Outcomes Following Liver Transplantation score, Donor Risk Index score, and Balance of Risk score (Figure 1). In medicine, certain variables do not necessarily have to assume a linear relationship. Hence, ML models are superior to statistical methods (linear regression), from which most of these scores are derived[15]. However, these findings may be attributed to model overtraining; therefore, validation is required.

Figure 1.

External validation of artificial neural network models[13]. The performance obtained by these models is compared to other published score in terms of area under curve. A receiver operating characteristic curve depicts these metrics. Artificial neural network models based on the concept of minimum sensitivity and correct classification rate are represented such as sig minimum sensitivity and sig correct classification rate respectively. These models outperformed other traditional scores such as Model for End-Stage Liver Disease, Model for End-Stage Liver Disease score excluding exception points and donor age, Survival Outcomes Following Liver Transplantation, Preallocation Survival Outcomes Following Liver Transplantation, Donor Risk Index or Balance of Risk. CCR: Correct classification rate; MS: Minimum sensitivity; MELD: Model for End-Stage Liver Disease score; DMELD: Model for End-Stage Liver Disease score excluding exception points and donor age; SOFT: Survival Outcomes Following Liver Transplantation score; P-SOFT: Preallocation Survival Outcomes Following Liver Transplantation score; DRI: Donor Risk Index score; BAR: Balance of Risk score; AUC: Area under curve; KCH: Kings College Hospital; ROC: Receiver operating characteristic. Citation: Ayllón MD, Ciria R, Cruz-Ramírez M, Pérez-Ortiz M, Gómez I, Valente R, O'Grady J, de la Mata M, Hervás-Martínez C, Heaton ND, Briceño J. Validation of artificial neural networks as a methodology for donor-recipient matching for liver transplantation. Liver Transpl 2018; 24: 192-203. Copyright© The Authors 2018. Published by Wolters Kluwer Health, Inc.

The most significant lesson learned from using these models is their high dependency on the datasets on which they were trained. This issue affects the practical applicability. Retrospective data, external validation, the “black box issues” in ANN, and data-protection policies are considered significant contributing factors. To overcome these barriers, better data-handling policies are needed. Applicability relies on the clinicians’ confidence in using these models. Therefore, if external validation is impossible (region-specific rather than universal models), prospective validation should be considered. Moreover, the databases must be updated regularly to reinforce the learning of these models. Clinical scenarios are dynamic, and models must change accordingly.

Recently, interest in artificial intelligence and ML has increased. They can handle large amounts of data quickly and yield accurate results. However, we must note the limitations of these models and address them to achieve a real integration.