Machine Learning-Based Algorithm for the Early Prediction of Postoperative Hypocalcemia Risk After Thyroidectomy

Abstract

Objective:

We used machine learning to develop and validate a multivariable algorithm allowing the accurate and early prediction of postoperative hypocalcemia risk.

Background:

Postoperative hypocalcemia is frequent after total thyroidectomy. An early and accurate individualized prediction of the risk of hypocalcemia could guide the selective prescription of calcium supplementation only to patients most likely to present with hypocalcemia after total thyroidectomy.

Methods:

This retrospective study enrolled all patients undergoing total thyroidectomy in a single referral center between November 2019 and March 2022 (derivation cohort) and April 2022 and September 2022 (validation cohort). The primary study outcome was postoperative hypocalcemia (serum calcium under 80 mg/L). Exposures were multiple clinical and biological variables prospectively collected and analyzed with various machine learning methods to develop and validate a multivariable prediction algorithm.

Results:

Among 610/118 participants in the derivation/validation cohorts, 100 (16.4%)/26 (22%) presented postoperative hypocalcemia. The most accurate prediction algorithm was obtained with random forest and combined intraoperative parathyroid hormone measurements with 3 clinical variables (age, sex, and body mass index) to calculate a postoperative hypocalcemia risk for each patient. After multiple cross-validation, the area under the receiver operative characteristic curve was 0.902 (0.829–0.970) in the derivation cohort, and 0.928 (95% CI: 0.86; 0.97) in the validation cohort. Postoperative hypocalcemia risk values of 7% (low threshold) and 20% (high threshold) had, respectively, a sensitivity of 92%, a negative likelihood ratio of 0.11, a specificity of 90%, and a positive of 7.6 for the prediction of postoperative hypocalcemia.

Conclusions:

Using machine learning, we developed and validated a simple multivariable model that allowed the accurate prediction of postoperative hypocalcemia. The resulting algorithm could be used at the point of care to guide clinical management after total thyroidectomy.

Postoperative hypocalcemia is frequent after total thyroidectomy (TT).¹ In most cases, hypocalcemia results from hypoparathyroidism secondary to the peroperative devascularization of the parathyroid glands. Its occurrence can be associated with severe clinical symptoms, requiring the need for calcium supplementation,² and prolonged hospitalization. Routine postoperative calcium supplementation after total thyroidectomy has been proposed by some authors to reduce the burden of hypocalcemia.^3,4 This practice is, however, not currently recommended in international guidelines, as it may mask the onset of severe, long-lasting hypocalcemia, and delay the diagnosis of definitive hypoparathyroidism.⁵

An early and precise prediction tool could help to guide the prescription of calcium supplementation to patients most likely to present with postoperative hypocalcemia. This is particularly useful in the context of outpatient thyroid surgery,⁶ which requires minimization of potential complications.

There is currently no consensus on the best predictor of postoperative hypocalcemia. Several preoperative and peroperative clinical and biological variables have been associated with the risk of postoperative hypocalcemia. Among them, immediate postoperative serum intact parathyroid hormone (PTH) level has recently gained a lot of attention.^7,8 PTH is biochemically directly associated to blood calcium level and its short half-life, provides an early insight into the patient’s parathyroid function following thyroidectomy.

Numerous studies have analyzed PTH serum levels at different time points following thyroidectomy to predict the onset of postoperative hypocalcemia, from skin closure in the operating room to the first postoperative day.^9,10 There appears to be no clear difference in the prediction values of these different time points.^11–13 The percentage decay between preoperative and postoperative levels of serum PTH (%PTH) has also been proposed,^8,14,15 and reported to correlate linearly with postoperative calcium levels.¹⁶ Furthermore, the use of immediate postoperative serum PTH would have the advantage of easy implementation for the early initiation of calcium and vitamin supplementation therapy.

However, even if PTH seems to be the most accurate predictive factor, the potential of the combination of PTH with other predictive factors to reinforce the accuracy of prediction is unclear. Recent methods of artificial intelligence such as machine learning, could be useful to identify such favorable combinations.

In this study, we leveraged machine learning to identify the most accurate predictive factors for postoperative hypocalcemia after total thyroidectomy and define an early surrogate marker of the risk of developing postoperative hypocalcemia after total thyroidectomy.

METHODS

Population and Preoperative Data Collection

In this retrospective study, we enrolled every patient who underwent total thyroidectomy at Lille University Hospital, first between November 1, 2019, and March 31, 2022 (derivation cohort) and second between April 1, 2022, and September 13, 2022 (validation cohort), in the absence of associated hyperparathyroidism and/or planned parathyroidectomy.

Preoperative data consisted of demographic data (age, sex, height, weight, and body mass index (BMI)], surgical indication, history of surgical neck dissection or hyperthyroidism, and biological data (serum protein or albumin level, serum creatinine level and serum vitamin D level before supplementation). A history of vitamin D supplementation was also collected.

Surgery, PTH Dosage Protocol, and Intraoperative Data

Thyroidectomy was performed or supervised by 5 experimented senior surgeons, following codified dissection, which aimed at formal identification and in situ preservation of superior parathyroid glands, as well as at preservation of the vascularization of all 4 parathyroid glands. Recurrent nerve function was monitored via endotracheal neuromonitoring, and confirmed, when necessary, by postoperative laryngoscopy. Every patient had a serum PTH dosage at the time of anesthetic induction (T0), and at the time of skin closure once total thyroidectomy was performed (T1). PTH serum levels were considered null when not detectable (inferior detection threshold 8 pg/mL).

Intraoperative data consisted of operating time, performance of a central neck dissection, number of parathyroid glands identified, number of parathyroid glands reimplanted, and the weight of the thyroid specimen.

Postoperative Management, Postoperative Hypocalcemia Definition, and Exclusion Criteria

Postoperative management consisted of a serum-level dosage of calcium, phosphorus, and PTH on postoperative day (POD) 1 of surgery. Discharge was allowed at POD 1 when serum calcium was ≥80 mg/dL, and POD 1 PTH was above 8 pg/mL in the absence of clinical symptoms of hypocalcemia (paresthesia, myoclonia) or other surgical complication (dysphony, dysphagia, or dyspnea).

When these criteria were not met, patients remained hospitalized, and serum phosphorus, calcium, and PTH levels were evaluated daily, or in case of symptoms consistent with hypocalcemia. Postoperative hypocalcemia was defined by a serum calcium level inferior to 80 mg/L, measured at any postoperative time. Patients were excluded from the analysis if they received calcium and/or vitamin D in the absence of biological evidence for hypocalcemia, regardless of PTH serum levels or symptom occurrence.

All patients experiencing postoperative hypocalcemia were seen 1 month after thyroidectomy for serum calcium and PTH levels evaluation, 5 days after the interruption of supplementation. Supplementation treatment was suspended when calcium levels were normal (>80 mg/L) in absence of hypocalcemia symptoms.

Univariate Statistical Analysis and Postoperative Hypocalcemia Prediction

Qualitative variables were expressed as numbers and percentages of the study population, and quantitative variables were expressed as mean and SD. Statistical comparison was performed between patients with or without postoperative hypocalcemia, using a Mann-Whitney test for quantitative variables and a χ² test for qualitative variables. The predictive performance of significantly different variables was evaluated by the area under the receiver operator characteristic curve (AUROC curve), sensitivity, specificity, and positive and negative likelihood ratios for predicting the onset of postoperative hypocalcemia.

Identification of Predictive Variables of Interest and Multivariate Hypocalcemia Prediction Via Machine Learning

We first conducted a multivariable analysis in the derivation cohort for identifying predictive variables of interest among the following candidate variables: sex, age, BMI, surgical indication, preoperative serum vitamin D level, preoperative serum creatinine, history of hyperthyroidism, operating time, number of parathyroids seen, parathyroid reimplantation, thyroid weight, central neck dissection, serum PTH at T0, PTH T1, and %PTH. Squared terms, as well as interaction terms, were added for the a priori most important variables: %PTH, neck dissection, surgical indication, sex, and age.

We used a repeated stratified k-fold cross-validation (k=10, n=100) on our center data to find the best model, according to the AUROC metric, among 6 different mathematical models. Notably, cross-validation is a method to evaluate a model by partitioning the total sample into a training set to train it, and a validation set to evaluate it. The 6 candidate models were linear models with %PTH only, logistic regression with or without l1-regularization, decision tree, random forest, and neural network. We specifically choose a heterogeneous set of candidate models: logistic regressions exploiting the linearity assumption, tree-based models learning simple decision rules to distinguish meaningful subgroups of patients, and neural networks using the multilayer perceptron to differentiate data that is not linearly separable.

All models included a first step of feature selection using Least Absolute Shrinkage and Selection Operator, to extract the most statistically relevant variables concerning outcome prediction. All models hyperparameters were tuned using a nested cross-validation grid search approach. Finally, the selected model corresponded to the best model from the previous benchmark and retrained on the whole data set to increase the precision of predictions. It was used to find clinically relevant levels of sensitivity and specificity.¹⁷ The feature importance was calculated as the mean accumulation of the impurity decreased.

The two chosen thresholds were probability thresholds of postoperative hypocalcemia risk, expressed as percentages, estimated from clinically relevant levels of sensitivity and specificity. Analysis was performed using Python 3.10 and the Scikit-learn library.¹⁸

To ensure the reproducibility of our findings, the predictive performances of the selected model were finally measured in the validation cohort.

Ethics Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board and the register of data analysis of the University Hospital of Lille.

RESULTS

The derivation cohort gathered 646 patients who underwent total thyroidectomy in our center during the study period. Among them, 36 patients (5.6%) were excluded from the analysis, either because they were supplemented for symptoms without biological hypocalcemia (16 patients), for low serum levels of PTH without biological hypocalcemia (17 patients), or abusive supplementation (3 patients). Among the 610 patients enrolled in the present study, 100 (16,4%) presented with hypocalcemia. Overall, 9 patients (1.5% of the overall population and 9% of patients with postoperative hypocalcemia) presented with persistent hypocalcemia, ie, still requiring calcium and/or vit D treatment 6 months after surgery.

Table 1 presents the demographic, preoperative, intraoperative, and postoperative characteristics of patients with or without postoperative hypocalcemia. Hypocalcemia patients were significantly younger (median age 44.5 vs 52 y, P=0.0006), more likely to be female (85% vs. 73.3%, P=0.015), and had a significantly lower BMI (24.2 vs 27.6, P=0.001). Both groups were comparable in terms of preoperative history, surgical indications, preoperative biological data, and intraoperative data. Total thyroidectomy was indicated in 219 patients for a suspicious node, for the presence of a Bethesda score of 5 or 6 at cytoponction, and/or a size above 4 cm. Malignancy was confirmed at pathology in 181 cases (82.6%).

TABLE 1.

Comparison of Preoperative, Intraoperative, and Postoperative Characteristics in Patients With or Without Postoperative Hypocalcemia

	Hypocalcemia	Missing data, n (%)	Control	Missing data, n (%)	P
N	100	—	510	—	—
Median age, y (IQR)	44.5 (35.2–55)	0	52 (39–63)	0	0.0006
Female sex, n (%)	85 (85)	0	374 (73.3)	0	0.015
Median BMI, kg/m2 (IQR)	24.2 (22.3–31.2)	18 (18)	27.6 (23.9–32.5)	62 (12)	0.001
Preoperative history, n (%)
Hyperthyroidism	32 (32.0)	0	181 (35.5)	0	0.566
Preoperative vitamin D deficiency supplementation	16 (16.0)	1 (1)	78 (15.29)	8 (1.6)	0.879
Surgical indication, n (%)	—	0	—	0	—
Grave disease	16 (16.0)	—	95 (18.6)	—	—
Suspicious node	36 (36.0)	—	183 (35.9)	—	—
Toxic node	4 (3.0)	—	7 (1.37)	—	—
Suspicious multinodular goiter	11 (10.0)	—	45 (8.8)	—	0.530
Toxic multinodular goiter	11 (11.0)	—	64 (12.5)	—	—
Symptomatic goiter	22 (22.0)	—	113 (22.2)	—	—
Other*	0	—	3 (0.59)	—	—
Preoperative biological data, median (IQR)
Serum albumin (g/dL)	43 (39.5–46)	23 (23)	43 (41–46)	141 (27,6)	0.148
Serum creatinine (mg/L)	7 (6–8)	7 (7)	7 (6–8)	57 (11.2)	0.208
Serum Vitamin D (ng/mL)	25 (17–29)	18 (18)	25 (18–32)	102 (20)	0.462
Serum PTH at anesthetic induction (pg/mL)	59 (38.2–81.7)	0 (0)	61 (44.7–80.2)	0 (0)	0.441
Intraoperative data
Median operating time, minutes (IQR)	166 (131–229)	0 (0)	159 (132–197)	2 (0.4)	0.166
No. parathyroids seen, median (IQR)	4 (3–4)	1 (1)	4 (3–4)	26 (5.1)	0.7743
Neck dissection, n (%)	40 (40.0)	0	167 (32.7)	0	0.167
Parathyroid reimplantation, n (%)	13 (13.0)	0	46 (9.0)	0	0.265
Abnormal postoperative laryngoscopy n (%)	8 (8.0)	2 (2)	27 (5.29)	12 (2.4)	0.343
Median thyroid weight, gram (IQR)	44 (24–90)	1 (1)	43 (22–86)	11 (2.2)	0.705
Postoperative biological data
Median serum PTH at skin closure, pg/mL (IQR)	0 (0-10)	0	26 (16–43)	0	<0.0001
Median serum PTH at POD 1, pg/mL (IQR)	5 (0–12)	0	26 (18–36)	1 (0.2)	<0.0001
Median serum calcium at POD 1, mg/L (IQR)	80 (77–83)	0	88 (86–91)	1 (0.2)	<0.0001
Median %PTH (%) (IQR)	100 (83.2–100)	0	51.4 (30–71.6)	0 (0.2)	<0.0001
Median hospital stay, days (IQR)	3 (2–4)	0	1 (1–2)	0	<0.0001

*Other indications consisted in 1 myasthenia and 2 drug-induced hyperthyroidism; %PTH : Percentage decay between PTH at anesthetic induction and PTH at skin closure; IQR: interquartile range

N indicates number.

Postoperative data analysis showed that median values of serum PTH at skin closure, %PTH, and PTH at POD 1 were significantly lower in hypocalcemia group (0 vs 26 pg/mL, 100% vs 51.4%, and 5 vs. 26 pg/mL respectively, P<0.0001). Serum calcium level at POD 1 was also significantly lower in hypocalcemia group (80 vs 88 mg/L, P<0.0001). Hospital stay was significantly higher in hypocalcemia group (3 days vs 1 day, P<0.0001).

The validation cohort gathered 118 patients operated during the second study period, including 26 (22.0%) individuals who presented postoperative hypocalcemia. Patient characteristics at baseline were similar to the derivation cohort (Supplemental Table 1, Supplemental Digital Content 1, http://links.lww.com/SLA/F251).

We then evaluated the performances of %PTH alone to predict the risk of postoperative hypocalcemia according to 3 threshold values (Table 2). A %PTH superior to 60% had a sensitivity of 92% (CI: 85%–96%) and a specificity of 61% (CI: 57%–65.3%) for hypocalcemia prediction, with a positive likelihood ratio (LR+) of 2.5 and a negative likelihood ratio (LR−) of 0.14. A %PTH superior to 75% had a sensitivity of 82% (CI: 73.3%–88.3%) and a specificity of 81.6% (CI: 78.0%–84.7%) for hypocalcemia prediction, with an LR+ of 4.45 and a LR− of 0.22. A %PTH superior to 85% had a sensitivity of 74% (CI: 64.6%–81.6%) and a specificity of 90% (CI: 87.1%–92.3%) for hypocalcemia prediction, with a LR+ of 7.4 and a LR− of 0.29.

TABLE 2.

Performances of %PTH Alone for Predicting Postoperative Hypocalcemia According to Various Thresholds

	%PTH
Threshold (%)	>60	> 75	> 85
Sensitivity (%)	92	82	74
Specificity (%)	61	81.6	90
LR+	2.5	4.45	7.4
LR−	0.14	0.22	0.29

LR+ indicates positive likelihood ratio; LR−, negative likelihood ratio.

Table 3 shows the performances of the 6 models tested for multivariable analysis. The best model according to AUROC curves, evaluated on repeated cross-validation, was the model with feature selection followed by random forest. This model achieves an AUROC of 0.902 (CI: 0.829–0.970), versus 0.889 (CI: 0.806–0.966) for the %PTH alone model. The AUROC curves of both models are displayed in Figure 1, using the chosen model retrained on the whole data set.

TABLE 3.

AUROC Curves Values for Postoperative Hypocalcemia Prediction Evaluated by Repeated Cross-validation for %PTH Only Model, and for the Different Machine Learning Models

	Mean AUC	95% CI
%PTH only	0.889	0.806–0.966
Logistic regression	0.898	0.810–0.958
Logistic regression with regularization	0.896	0.820–0.961
Decision tree	0.865	0.740–0.963
Random forest	0.902	0.829–0.970
Gradient boosting	0.901	0.866–0.964
Neural network	0.884	0.818–0.954

FIGURE 1.

Relation between postoperative PTH decrease (%PTH) and predicted risk of PHR, according to the random forest model, for four virtual patients with distinct characteristics. F indicates Female; M, Male; PHR, postoperative hypocalcemia; PTH₀: Parathormone at anesthetic induction; y, years.

Among all variables included in the model, only 5 selected variables were identified as predictive factors for hypocalcemia: %PTH squared, PTH at anesthetic induction, the interaction between %PTH and sex, BMI, and age squared. These variables are reported in Table 4, by order of feature importance. Noteworthy, feature importance is only relevant for the model itself and its value cannot be used to tell if a given feature has a negative or positive impact on postoperative hypocalcemia occurrence. This 5 variables model was used to determine, for each patient of the data set, a value of postoperative hypocalcemia risk (PHR), expressed in percent. We then identified 2 thresholds of PHR, which were associated with clinically useful levels (≥90%) of sensitivity or specificity, respectively (Table 5). A PHR of 7% (low threshold) was associated with a sensitivity of 92% resulting in a LR− of 0.11 for the prediction of postoperative hypocalcemia. A PHR of 20% (high threshold) was associated with a specificity of 90, resulting in a LR+: 7.6 for postoperative hypocalcemia prediction. Figure 1illustrates the relation between postoperative hypocalcemia risk according to the random forest model (PHR) and %PTH for 4 virtual individual patients with distinct characteristics.

TABLE 4.

Feature Importance of the Random Forest Model

	Feature importance
%PTH squared	0.46
PTH T0	0.34
Interaction between %PTH and sex	0.19
BMI	0.01
Age squared	0.005

TABLE 5.

Performances of the Random Forest Model for Predicting Postoperative Hypocalcemia According to Low and High Values of PHR

	Low threshold: PHR 7%	High threshold: PHR 20%
Sensitivity (%)	92	79
Specificity (%)	73	90
LR+	3.4	7.6
LR−	0.11	0.23

LR+ indicates positive likelihood ratio; LR−, negative likelihood ratio; PHR, postoperative hypocalcemia risk, a pretest risk for postoperative hypocalcmia calculated by the model for a given patient.

Finally, the predictive performance of the selected random forest model was tested in the validation cohort, resulting in an AUROC curve value of 0.928 (95% CI: 0.86; 0.97), well within the confidence interval of the AUROC curve value estimated through repeated cross-validation in the initial data set.

Eventually, the model’s mathematical algorithm was implemented in a shinyapp, available online (Fig. 2) and allowing the automated calculation of the PHR of any given patient, at the point of care, according to the 5 variables of interest (https://lille-model.shinyapps.io/Hypocal/)

FIGURE 2.

adapted from the screen capture of the shinyapp allowing the automated calculation of postoperative hypocalcemia risk of an individual patient, according to the 5 variables of interest (https://lille-model.shinyapps.io/Hypocal/). The color code indicates the risk of experiencing postoperative hypocalcemia as low (green ≤7%), intermediate (orange), and high (red, ≥20%).

DISCUSSION

Predicting the risk of postoperative hypocalcemia after total thyroidectomy has been an important research question for several decades.¹⁹ In the present study, we used machine learning to develop and validate a multivariable model allowing the early and accurate prediction of postoperative hypocalcemia, based on age, sex, BMI, preoperative PTH, and %PTH.

Serum PTH has a 4-minute half-life and directly reflects parathyroid function. The early measurement of this direct biomarker of parathyroid function has been increasingly proposed to predict hypocalcemia after total thyroidectomy.^20,21 In 2018, a literature review by Marthur et al⁸ concluded that the best PTH-based predictor of postoperative hypocalcemia was PTH decay between preoperative and postoperative levels. Furthermore, Liu et al¹⁶ described a linear correlation between the percentage of PTH decay and postoperative calcium levels.

In addition, other variables have been proposed to assess the risk of hypocalcemia,^16,22,23 and improve prediction. This suggests that even if, at first, PTH appears as a near-perfect biomarker for hypocalcemia prediction, there might be other independent and clinically relevant predictive factors. Here, we initially evaluated the performance of intraoperative PTH measurement alone to predict the onset of hypocalcemia after total thyroidectomy. We found that both preoperative PTH and %PTH measured immediately after surgery were independent predictors of postoperative hypocalcemia. In addition, our multivariable analysis also identified 3 other preoperative variables (age, sex, and body mass index) that were independently associated with the risk of postoperative hypocalcemia in our population. We then apply random forest, a machine learning method used for classification and regression models, operating by constructing a multitude of decision trees at training time. Using this method, we found that even if %PTH was a major predictive factor, the prediction could be further improved when other variables of interest were added to the model. Following multiple cross-validation, the algorithm resulted in a PHR score with an AUROC curve of 0.902 (95% CI: 0.829; 0.970). We also identified 2 low and high thresholds of the PHR score with very good positive (>5) and negative (<0.2) likelihood ratios to predict hypocalcemia. The 5 simple variables selected are all available immediately after surgery; the resulting model has been incorporated in an online shinyapp that can be used at the point of care to guide early postoperative clinical management.

Several studies have also shown the impact of sex^24–26 and age²⁶ as a predictive factor for postoperative hypocalcemia. In line with our results, two studies^27,28 have previously identified higher body mass index as a potential protective factor for transient hypocalcemia after thyroidectomy. Furthermore, serum PTH level has been shown to be linked with obesity in the general population,²⁹ providing a potential explanation of this apparent “obesity paradox” for postoperative hypocalcemia after total thyroidectomy. Of note, other variables which have been previously associated with the risk of hypocalcemia, like lymphadenectomy, hyperthyroidism, and parathyroid reimplantation,^24–26 were not found to be independent predictor in the present study. This suggests that postoperative PTH, and hence %PTH, already captures most of the information needed to estimate the risk of hypocalcemia. On the other hand, our results suggest that female sex, younger age, and low body mass index may be associated with hypocalcemia independently of hypoparathyroidism. In contrast, other preoperative characteristics or intraoperative events increase risk only through a decrease in parathyroid function. Finally, hyperthyroidism did not have any influence on risk prediction in our population, in contrast with a previous report describing its association with hypocalcemia through the “hungry bone syndrome.”³⁰ Routine medical control of hyperthyroidism before thyroidectomy in our center may explain this apparent discrepancy.^30,31

The main strength of our study lies in the mathematical approach used, based on machine learning with multiple cross-validations. The resulting model allowed an individualized prediction of the postoperative hypocalcemia risk (PHR) based only on five simple pre- and intraoperative variables, that can be easily incorporated in the online calculator provided. To ensure strong external validity, our definition of post-thyroidectomy hypocalcemia was deliberately very broad, including all patients with biological hypocalcemia, irrespective of its symptomatic nature, depth, or duration. Future work may focus on the potential determinants of symptomatic hypocalcemia or prolonged hypoparathyroidism. Another limitation of our work lies in the absence of model testing on an independent external cohort. Therefore, the generalizability of our findings and algorithm warrants further validation in other centers.

In conclusion, in the present study, we leveraged machine learning to develop and validate a multivariable algorithm allowing the early and accurate prediction of postoperative hypocalcemia, combining three simple preoperative variables with intraoperative PTH measurement, which may guide early postoperative clinical management after total thyroidectomy. The proposed approach could help to individualize postoperative care and facilitate the implementation of outpatient thyroidectomy.

Supplementary Material

sla-280-835-s001.docx ^{(14.7KB, docx)}

DISCUSSANT

Elizabeth Nieveen van Dijkum (Amsterdam, the Netherlands)

In this single-center retrospective study, including 610 patients over a 29-month period, machine learning was used to develop a model predicting the risk of hypocalcemia after total thyroidectomy. The dynamics of perioperative PTHi measurements (pre-OP PTHi and PTHi drop in %) were found to be the most potent predictors in isolation. However, the association of perioperative PTHi with sex, age, and BMI showed the best performance for hypocalcemia prediction.

I have the following questions for the authors: First, the study aims to improve risk prediction and apply specific statistical techniques. You mention the need for a robust predictor, but how does your prediction model compare to other predictors already published?

Second, the thorough machine learning statistical analysis resulted in including 5 factors, with the most important factors being the drop in PTH and the PTH at the start of surgery. These factors are known and mostly used in existing models and not new. New factors, not known from earlier risk predictions are BMI and age, but these factors have very low predictive values. How valid are these factors, since the development cohort already had a difference in these BMI and age values? In addition, why was external validation not included, especially focusing on these two factors?

Finally, the predictor indicates low (<8%) intermediate (8%–19%) and high (>19%) risk for hypocalcemia. How can these values be used in patient care? Is a patient with a risk of 10% discharged? With or without calcium suppletion? What is the accepted number to treat (calcium supplementation) to prevent, for example, one readmission in relation to this risk predictor? Why was not use a simple high and low risk based on an accepted NNT?

Response From François Pattou (Lille, France)

Thank you very much for these interesting and important questions. To answer the first question, overall, the random forest algorithm performed slightly better that other exdisting linear models, including those based on peroperative PTH measurements. Of note, as illustrated in figure 1, the relation between PTH decrease and risk of hypocalcemia predicted by the model, varied non linearly with individual patient characatiristics.

Addressing your second question: The percentage of PTH decrease is crucial and significantly influences the statistics. When analyzed using linear regression, the mean values provide a strong prediction of PTH levels. However, linear regression can miss some patients, which is where nonlinear mathematics excels. It can identify information in the gaps that linear regression overlooks. This is the strength of machine learning – it is very efficient in capturing these subtleties.

To answer the last question: Safely discharging patients is the most important aspect of our work. Using the calculator should be able to assure patients, “You are safe to go back home tonight and do not need to send me blood calciu measurement m in one month.”Patients at higher risk (yellow and red categories ) are more challenging. Clinical judgment and context play significant roles here. In our practice, patients in the yellow or red categories stay an additional day, to measure the calcium the next day, and eventually prescribe treatment. However, in other settings where patients are generally discharged the same day, this infomation becomes crucial for selecting patients who must stay longer for calcemia monitoring and treatment.

Antonio D. Pinna (Weston, United States)

I am very curious to know if it is possible to include the application in an electronic medical record.

Response From François Pattou (Lille, France)

Sure! The algorithm only need to enter four variables(BMI, age, etc.). These plugins can be easily integrated into a medical record system to provide useful tools. We used a similar approach for predicting weight trajectories after bariatric surgery (https://bariatric-weight-trajectory-prediction.univ-lille.fr/) with an algorithm which can integrate thisinformation within electronic medical records.

Lawrence T. Kim (Chapel Hill, United States)

I have a rather mundane question related to the previous one. One of the barriers to this kind of work is that many of us, including myself, don’t know how to do the coding and other technical aspects involved in creating these techniques. Do you do it yourself? Did you learn how to do it, and if so, how do you get it done?

Response From François Pattou (Lille, France)

Thank you for this very important question. I don’t do it by myself; I’ve been closely collaborating with experts in machine learning for 5 years. This interdisciplinary work has involved a lot of intercative discussions. In health care, it’s crucial to use interpretable techniques. For example, it is important to understand why a lower BMI is a risk factor Surgeons don’t need to code themselves, but must invest significant time collaborating with data science experts What won’t succeed in health care is the “noninterpretable black box.” In constract to a phone navigation system that may guide you without any explanation, health care requires interpretability. Health-technology assessment agencies need to understand and evaluate such tools prior to their validation and reimbursement. There’s a difference between the vast majority of digital tools and numeric medical devices which have been validated for real care. Interpretability is the key.