A machine learning model for colorectal liver metastasis post-hepatectomy prognostications: several strategies for the model evaluation

With great interest, we read the article by Lam et al. (1) entitled “A machine learning model for colorectal liver metastasis post-hepatectomy prognostications”. In this study, the authors included colorectal liver metastasis (CRLM) patients from four hospitals in Hong Kong who underwent hepatic resection, and developed a survival prediction model based on the patients’ demographic, oncologic, clinicopathologic, and therapeutic characteristics using machine learning. Through Cox proportional hazards and least absolute shrinkage and selection operator (LASSO) regression analyses, the authors successfully developed a predictive model consisting of eight predictors that could accurately predict overall survival (OS) and recurrence-free survival (RFS) after hepatectomy in patients with CRLM. This is an intriguing study with significant clinical value, and the authors deserve to be commended for their efforts. However, there are still several issues that need to be addressed in this study.

First, the authors employed only Cox proportional hazards and LASSO regression to construct the predictive models. We recommend that the authors develop a nomogram to predict OS and RFS based on the results of multivariate Cox proportional hazards analysis to visualize the predictive probabilities of the predictors for survival. The nomogram predicts the specific probability of an event occurring, rather than the level of risk. It integrates the predictors in the regression model by plotting the scaled line segments on the same plane at a certain scale, which displays the interrelationships between the variables in the prediction model and the predictive probabilities of the variables for the outcome events. By assigning scores to the regression coefficients of predictors in the regression model and the functional transformation relationship between total scores and outcome events, the nomogram transforms complex regression equations into visual graphs, making the results of prediction models more readable and easier to use, and has been widely applied in the construction of various prediction models (2,3). Second, the authors assessed the predictive ability of the prediction model using Harrell’s Concordance index (C-index), but they did not provide a calibration curve. The C-index measures the discrimination of a predictive model, while the calibration curve measures the agreement between predicted and actual probabilities, i.e., calibration (4,5). Both discrimination and calibration are the most common and crucial metrics for evaluating the performance of a predictive model. Therefore, when evaluating the predictive performance of a model, the authors should consider both discrimination and calibration. Furthermore, in addition to the C-index, we recommend that authors supplement receiver operating characteristic (ROC) curve analysis, which not only evaluates the discriminative ability of the predictive model, but also calculates the sensitivity and specificity of the predictive model. Third, although the C-index, ROC curves, and calibration curves can be used to assess the discrimination and calibration of a predictive model, they do not adequately reflect the clinical utility. In clinical practice, a predictive model is not always fully applicable, and false positives and false negatives are unavoidable. Sometimes, patients benefit more from avoiding false positives, while at other times, it is preferable to avoid false negatives. To find a method to maximize the net benefit, the decision curve analysis (DCA) was subsequently developed (6). The advantage of DCA is that it integrates patient or decision maker preferences into the analysis, which meets the practical requirements of clinical decision-making (6-8). Given that the predictive model developed by the authors ultimately needs to be implemented in clinical practice, we recommend that the authors utilize DCA to ascertain the net benefit of the predictive model, thereby assessing the clinical utility of the model. Finally, the patients recruited for this study were from Hong Kong, China, with regional limitations. We recommend that the authors attempt to recruit CRLM populations from healthcare organizations in various international regions as an external validation cohort for future studies. This will help validate the predictive effect of the model on patient survival from different regions. By broadening the geographic range of the validation cohort, it will be beneficial for the model to be generalized to diverse populations, independent of the clinical setting.

In summary, we would like to thank the authors for their significant contribution to exploring the superior survival prediction model for patients with CRLM in this study. Regarding the evaluation of the survival prediction model, we propose several strategies that could be beneficial in enhancing this study and future similar studies.

Supplementary

The article’s supplementary files as

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.