Skip to main content
PMC home page
Journal of Neurological Surgery. Part B, Skull Base logoLink to Journal of Neurological Surgery. Part B, Skull Base
. 2022 Aug 25;83(6):635–645. doi: 10.1055/a-1885-1447

Machine Learning Models for Predicting Postoperative Outcomes following Skull Base Meningioma Surgery

Adrian E Jimenez 1, Jose L Porras 1, Tej D Azad 1, Pavan P Shah 1, Christopher M Jackson 1, Gary Gallia 1, Chetan Bettegowda 1, Jon Weingart 1, Debraj Mukherjee 1,✉
PMCID: PMC9653296  PMID: 36393884

Abstract

Objective  While predictive analytic techniques have been used to analyze meningioma postoperative outcomes, to our knowledge, there have been no studies that have investigated the utility of machine learning (ML) models in prognosticating outcomes among skull base meningioma patients. The present study aimed to develop models for predicting postoperative outcomes among skull base meningioma patients, specifically prolonged hospital length of stay (LOS), nonroutine discharge disposition, and high hospital charges. We also validated the predictive performance of our models on out-of-sample testing data.

Methods  Patients who underwent skull base meningioma surgery between 2016 and 2019 at an academic institution were included in our study. Prolonged hospital LOS and high hospital charges were defined as >4 days and >$47,887, respectively. Elastic net logistic regression algorithms were trained to predict postoperative outcomes using 70% of available data, and their predictive performance was evaluated on the remaining 30%.

Results  A total of 265 patients were included in our final analysis. Our cohort was majority female (77.7%) and Caucasian (63.4%). Elastic net logistic regression algorithms predicting prolonged LOS, nonroutine discharge, and high hospital charges achieved areas under the receiver operating characteristic curve of 0.798, 0.752, and 0.592, respectively. Further, all models were adequately calibrated as determined by the Spiegelhalter Z -test ( p >0.05).

Conclusion  Our study developed models predicting prolonged hospital LOS, nonroutine discharge disposition, and high hospital charges among skull base meningioma patients. Our models highlight the utility of ML as a tool to aid skull base surgeons in providing high-value health care and optimizing clinical workflows.

Keywords: meningioma, neurovascular structures, central nervous system, overall survival, outcomes

Introduction

Meningiomas are the most common type of primary intracranial tumor affecting the central nervous system (CNS). 1 2 They can be classified into two subtypes based on anatomical location: skull base and non-skull base. 3 While maximal resection of the tumor and its dural attachment is the standard surgical treatment for all meningiomas types, the utility of extent-of-resection as a meaningful prognostic variable for skull base meningioma patients specifically has recently been brought into question. 4 Further, the location of many skull base meningiomas adjacent to critical neurovascular structures often makes aggressive resection difficult or unfeasible, and puts patients at risk of serious postoperative complications. 5 While overall survival has not been shown to be significantly different between skull base and non-skull base meningiomas, patients with skull base tumors are approximately twice as likely to undergo retreatment of their meningiomas (either by surgery or radiotherapy) and have significantly shorter retreatment-free survival. 3 Therefore, being able to better predict the postoperative course of skull base meningioma patients specifically may aid in reducing patient morbidity and optimizing the delivery of high-value health care after surgery.

While predictive analytic techniques have been used to study meningioma postoperative outcomes generally, to our knowledge, there have been no studies that have investigated the utility of machine learning (ML) models in prognosticating outcomes among skull base meningioma patients specifically. The goal of the present study was to validate a workflow aimed at (1) developing predictive models for postoperative outcomes among skull base meningioma patients, specifically including prolonged hospital length of stay (LOS), nonroutine discharge disposition, and high hospital charges; and (2) validating the predictive performance of these models of out-of-sample testing data. We hope that this proof-of-concept study demonstrates the validity of using ML to develop effective prognostic tools for skull base meningioma patients.

Methods

Patient Selection and Recorded Variables

The present study was conducted using data from 265 patients who received surgical resection of their skull base meningiomas at a single academic institution between January 1 st , 2016 and December 31 st , 2019. Using Al-Mefty's anatomical classification system, the following meningioma subtypes were defined as “skull base”: tuberculum sellae, planum sphenoidale, olfactory groove, sphenoid wing/spheno-orbital, clinoidal, cavernous sinus, clival and petroclival, tentorial, cerebellopontine angle, foramen magnum meningiomas, meningiomas of the middle fossa floor, and temporal bone meningiomas. 6 Our Institutional Review Board (IRB), acting as a Health Insurance Portability and Accountability Act (HIPAA) Privacy Board, reviewed and approved the waiver of informed consent for this retrospective study (IRB00181593). Manual chart review of electronic medical records was used to obtain demographic and clinical information. Tumor size and location were determined using post-contrast magnetic resonance images, with tumor volume measured using tumor dimensions in axial ( x ), coronal ( y ), and sagittal ( z ) planes via the following formula: Inline graphic . An American Society of Anesthesiology physical status classification system (ASA) score was documented for each patient, and patient frailty was quantified using the 5-factor modified frailty index (mFI-5). 7 8 A symptomatic presentation was defined as a meningioma diagnosis on the basis of a workup prompted by any of the following symptoms: seizures, headaches, nausea/vomiting, diplopia, decreased hearing, vertigo, dysarthria, dysphagia, confusion, bladder incontinence, motor deficit, sensory deficit, language deficit, visual deficit, cognitive deficit, or gait deficit. Surgeon years of experience was defined as the number of years since a surgeon completed their residency training to the date of surgery, in line with prior research. 9 10

Regarding postoperative outcomes, prolonged hospital LOS and high hospital charges were both analyzed as dichotomous variables using a cutoff of the upper quartile of each outcome (>4 days for LOS and >$47,887 for hospital charges), as described previously. 11 12 13 14 15 16 For the present study, routine discharge disposition was defined as discharge to home (either with self-care or health care service assistance) and nonroutine discharge was defined as discharge to a rehabilitation facility, a skilled nursing facility, or a hospice facility. 17

Statistical Analysis

Data were collected using Microsoft Excel (version 2016, Microsoft Corp.) and statistical analyses were conducted using R statistical software (version 4.0.2, r-project.org). Bivariate analyses were conducted using Fisher's exact test and the Mann-Whitney U test for continuous and categorical variables, respectively. ML algorithms were trained using the Caret package. 18 For this study, elastic net logistic regression ML algorithms were used. Briefly, the elastic net is a statistical technique used to prevent model overfitting by applying a penalty function to regression β-coefficients. 19 Addition of this penalty function to either linear or logistic regression model coefficients allows for better predictive performance compared with ordinary least-squares regression coefficients by effectively removing model covariates that do not contribute to optimizing predictive performance, thereby creating more parsimonious models that perform better when evaluated on out-of-sample testing data. 19

Patient data was separated into training and independent holdout testing subsets based on an 70/30 ratio, respectively. Fivefold cross validation repeated 10 times was used to tune model hyperparameters on the 70% training dataset, and hyperparameter optimization was conducted using a random search. 20 Following training (i.e., hyperparameter optimization), the predictive abilities of the finalized models were evaluated on 30% holdout testing dataset. Models were compared based on their discrimination and calibration, quantified by the area under the receiver operating characteristic curve (AUROC) and by the Brier score, respectively. 21 22 An AUROC (also known as the c-statistic) of 0.70 is generally taken to indicate that a predictive model demonstrates clinically-useful discrimination. 21 For AUROC and Brier score metrics, 95% confidence intervals were obtained using 2,000 bootstrapped replicates while a 95% confidence interval for accuracy was calculated as described by Clopper and Pearson. 23 Spiegelhalter's Z -test was also used to assess for adequate calibration of the final models for prolonged LOS, nonroutine discharge, and high hospital charges, with p <0.05 indicating inadequate calibration. 22 Variable importance plots were also created for each model to depict the relative importance of each variable toward calculating the predicted outcome.

Results

Patient Demographics and Outcomes

Table 1 demonstrates the demographic and clinical characteristics of our patient cohort. Our cohort was comprised of patients with the following types of skull base meningiomas: 72 sphenoid wing meningiomas (no spheno-orbital meningiomas), 54 cerebellopontine angle meningiomas, 43 tentorial meningiomas, 27 tuberculum sella meningiomas, 20 cavernous sinus meningiomas, 15 olfactory groove meningiomas, nine meningiomas of the middle fossa floor, eight meningiomas of the temporal bone, six foramen magnum meningiomas, five planum sphenoidal meningiomas, three clinoidal meningiomas, and three clival/petroclival meningiomas. Overall, our cohort had a mean age (± SD) of 58.89 ± 12.91 years, was majority female (77.7%), and was mostly Caucasian (63.4%). Most patients had private insurance (68.7%), were married (69.8%), and underwent an elective resection of their skull base meningioma (92.1%). Most patients had WHO grade I tumors (94.0%), and the mean tumor size (± SD) of our cohort was 17.17 ± 20.99 cm 3 . A total of 47 (17.7%), 111 (41.9%), and 107 (40.4%) patients had tumors located in the anterior, middle, and posterior fossa of the skull base, respectively. The mean ASA and mFI-5 scores (± SD) for our patients were 2.64 ± 0.53 and 0.79 ± 0.84, respectively. The majority of our patient cohort had a symptomatic presentation (90.9%). The mean surgeon years of experience (± SD) among our cohort was 16.64 ± 10.71 years, with a small minority of surgeons utilizing an endoscopic endonasal approach for tumor resection (3.4%). The mean surgery duration (± SD) in our cohort was 4.84 ± 1.92 hours, while the mean hospital LOS (± SD) was 4.86 ± 7.18 days. Most patients had a routine discharge disposition postoperatively (88.7%), and the mean hospital charges (± SD) incurred among our patients were $44,740.88 ± $29,547.03.

Table 1. Patient demographic and clinical characteristics ( n  = 265) .

Characteristic n (%)
Mean age ± SD 58.89 ± 12.91
Sex
Female 206 (77.7)
Male 59 (22.3)
Race
White or Caucasian 168 (63.4)
Black or African-American, Asian, or Other 97 (36.6)
Insurance
Private 182 (68.7)
Medicare or Medicaid 83 (31.3)
Marital status
Married 185 (69.8)
Not married 80 (30.2)
Admission type
Elective surgery 244 (92.1)
Non-elective 21 (7.9)
WHO Grade
I 249 (94.0)
II/III 16 (6.0)
Mean tumor volume (cm 3 ) ± SD 17.17 ± 20.99
Tumor location
Anterior fossa 47 (17.7)
Middle fossa 111 (41.9)
Posterior fossa 107 (40.4)
Mean ASA score ± SD 2.64 ± 0.53
Mean mFI-5 score ± SD 0.79 ± 0.84
Hypertension requiring medication 138 (52.1)
Diabetes 46 (17.4)
Chronic obstructive pulmonary disease 13 (4.9)
Congestive heart failure 9 (3.4)
Functional status 4 (1.5)
Symptomatic presentation
Yes 241 (90.9)
No 24 (9.1)
Mean surgeon years of experience ± SD 16.64 ± 10.71
Surgical approach
Endoscopic endonasal resection 9 (3.4)
Craniotomy 256 (96.6)
Mean hospital LOS ± SD 4.86 ± 7.18
Prolonged hospital LOS (>4 d)
Yes 66 (24.9)
No 199 (75.1)
Discharge disposition
Non-routine 30 (11.3)
Routine 235 (88.7)
Mean hospital charges in U.S. dollars ($) ± SD $44,740.88 ± 29,547.03
High hospital charges (>$47,887.39)
Yes 66 (24.9)
No 199 (75.1)
Open in a new tab

Abbreviations: LOS, length of stay; SD, standard deviation.

Bivariate Analysis

Table 2 displays the results of our bivariate analysis assessing for significant relationships between patient demographic/clinical variables and our three postoperative outcomes of interest: prolonged LOS, nonroutine discharge, and high hospital charges. Regarding LOS, older patient age ( p  = 0.017), Medicare or Medicaid insurance status ( p  = 0.0021), non-elective admission ( p <0.0001), greater tumor volume ( p  = 0.0017), higher ASA score ( p <0.0001), greater mFI-5 score ( p  = 0.0032), less surgeon years of experience ( p  = 0.0045), and longer surgery duration ( p  = 0.017) were all significantly associated with prolonged hospital LOS. Regarding discharge disposition, older patient age ( p <0.0001), Medicare or Medicaid insurance status ( p  = 0.035), greater tumor volume ( p <0.0001), higher ASA score ( p <0.001), and higher mFI-5 score ( p  = 0.020) were all significantly associated with nonroutine discharge. Finally, non-Caucasian race ( p  = 0.012), Medicare or Medicaid insurance status ( p  = 0.032), non-elective admission ( p  = 0.0010), greater tumor volume ( p <0.0001), higher ASA score ( p  = 0.031), less surgeon years of experience ( p <0.001), and longer surgery duration ( p <0.0001) were all significantly associated with high hospital charges.

Table 2. Bivariate analysis of patient demographic/clinical characteristics and postoperative outcomes ( n  = 265) .

Characteristic Hospital LOS Discharge disposition Hospital charges
>4 d ( n  = 66) ≤4 d ( n  = 199) p -Value Non-routine ( n  = 30) Routine ( n  = 235) p -Value >$47,887.39 ( n  = 66) ≤$47,887.39 ( n  = 199) p -Value
Mean age ± SD 62.36 ± 13.17 57.73 ± 12.65 0.017 a 67.87 ± 11.58 57.74 ± 12.64 <0.0001 a 61.32 ± 14.49 58.08 ± 12.28 0.074
Sex
Male 13 (19.7) 46 (23.1) 0.61 4 (13.3) 55 (23.4) 0.25 15 (22.7) 44 (22.1) 1.00
Female 53 (80.3) 153 (76.9) 26 (86.7) 180 (76.6) 51 (77.3) 155 (77.9)
Race
Black or African-American, Asian, or Other 31 (47.0) 66 (33.2) 0.055 7 (23.3) 90 (38.3) 0.16 33 (50.0) 64 (32.2) 0.012 a
White or Caucasian 35 (53.0) 133 (66.8) 23 (76.7) 145 (61.7) 33 (50.0) 135 (67.8)
Insurance
Medicare or Medicaid 31 (47.0) 52 (26.1) 0.0021 a 15 (50.0) 68 (28.9) 0.035 a 28 (42.4) 55 (27.6) 0.032 a
Private 35 (53.0) 147 (73.9) 15 (50.0) 167 (71.1) 38 (57.6) 144 (72.4)
Marital status
Married 42 (63.6) 143 (71.9) 0.22 16 (53.3) 169 (71.9) 0.055 42 (63.6) 143 (71.9) 0.22
Not married 24 (36.4) 56 (28.1) 14 (46.7) 66 (28.1) 24 (36.4) 56 (28.1)
Admission type
Non-elective 16 (24.2) 5 (2.5) <0.0001 a 2 (6.7) 19 (8.1) 1.00 12 (18.2) 9 (4.5) 0.0010 a
Elective 50 (75.8) 194 (97.5) 28 (93.3) 216 (91.1) 54 (81.8) 190 (95.5)
WHO Grade
II/III 5 (7.6) 11 (5.5) 0.56 4 (13.3) 12 (5.1) 0.092 5 (7.6) 11 (5.5) 0.56
I 61 (92.4) 188 (94.5) 26 (86.7) 223 (94.9) 61 (92.4) 188 (94.5)
Mean tumor volume (cm 3 ) ± SD 24.34 ± 24.30 14.79 ± 19.25 0.0017 a 31.25 ± 23.94 15.37 ± 19.93 <0.0001 a 26.52 ± 25.88 14.07 ± 18.13 <0.0001 a
Tumor location
Anterior fossa 14 (21.2) 33 (16.6) Reference 4 (13.3) 43 (18.3) Reference 15 (22.7) 32 (16.1) Reference
Middle fossa 25 (37.9) 86 (43.2) 0.42 14 (46.7) 97 (41.3) 0.59 28 (42.4) 83 (41.7) 0.44
Posterior fossa 27 (40.9) 80 (40.2) 0.56 12 (40.0) 95 (40.4) 0.78 23 (34.8) 84 (42.2) 0.23
Mean ASA score ± SD 2.88 ± 0.41 2.56 ± 0.55 <0.0001 a 2.97 ± 0.32 2.60 ± 0.54 <0.001 a 2.77 ± 0.49 2.60 ± 0.54 0.031 a
Mean mFI-5 score ± SD 1.09 ± 0.97 0.69 ± 0.77 0.0032 a 1.20 ± 1.06 0.74 ± 0.79 0.020 a 0.98 ± 0.98 0.73 ± 0.78 0.099
Symptomatic presentation
Yes 63 (95.5) 178 (89.4) 0.21 29 (96.7) 212 (90.2) 0.49 63 (95.5) 178 (89.4)
No 3 (4.5) 21 (10.6) 1 (3.3) 23 (9.8) 3 (4.5) 21 (10.6)
Mean surgeon years of experience ± SD 13.26 ± 10.57 17.76 ± 10.54 0.0045 a 14.47 ± 9.22 16.92 ± 10.87 0.47 12.41 ± 10.15 18.05 ± 10.54 <0.001 a
Surgical approach
Endoscopic endonasal resection 3 (4.5) 6 (3.0) 0.69 0 (0.0) 9 (3.8) 0.60 4 (6.1) 5 (2.5) 0.23
Craniotomy 63 (95.5) 193 (97.0) 30 (100.0) 226 (96.2) 62 (93.9) 194 (97.5)
Open in a new tab

Note: Bold is used to highlight statistical significance.

Predictive Modeling Results

Table 3 displays the regularized coefficients for our fully-trained elastic net logistic regression models in addition to their respective odds ratios. Further, our elastic net models predicting prolonged LOS, nonroutine discharge, and high hospital charges had α values of 0.38, 0.017, and 0.95, respectively. While all patient demographic and clinical variables were utilized to predict nonroutine discharge disposition, elastic net regularization only selected the following variables for predicting prolonged hospital LOS: insurance status, admission type, tumor volume, ASA score, and surgeon years of experience. Further, elastic net regularization removed the following variables from logistic regression analysis when seeking to optimize the predictive accuracy of high hospital charges: patient age, patient sex, patient race, marital status, WHO grade, tumor location, ASA score, mFI-5 score, symptomatic presentation, and surgical approach.

Table 3. Elastic net logistic regression coefficients and odds ratios for models predicting LOS, discharge, hospital charges, and readmission.

Hospital LOS Nonroutine disposition Hospital charges
Characteristic β -coefficient Odds ratio β -coefficient Odds ratio β -coefficient Odds ratio
Age – – 0.44 1.55 – –
Sex
Male – – −0.30 0.74 – –
Female – – – –
Race
Black or African-American, Asian, or Other – – −0.19 0.83 – –
White or Caucasian – – – –
Insurance
Medicare or Medicaid 0.10 1.11 0.11 1.11 0.026 1.03
Private
Marital status
Married – – −0.15 0.87 – –
Not married – – – –
Admission type
Non-elective 0.26 1.30 −0.24 0.79 0.12 1.13
Elective
WHO Grade
II/III – – 0.12 1.13 – –
I – – – –
Tumor volume (cm 3 ) 0.039 1.04 0.41 1.51 0.38 1.46
Tumor location
Anterior fossa Reference Reference Reference Reference Reference Reference
Middle fossa – – 0.11 1.11 – –
Posterior fossa – – −0.076 0.93 – –
ASA score 0.20 1.22 0.30 1.36 – –
mFI-5 score – – 0.067 1.07 – –
Symptomatic presentation
Yes – – 0.052 1.05 – –
No – – – –
Surgeon years of experience −0.055 0.95 −0.15 0.86 −0.17 0.85
Surgical approach
Endoscopic endonasal resection – – −0.16 0.85 – –
Craniotomy – – – –
Open in a new tab

Importantly, our analysis demonstrated that Medicare or Medicaid insurance status (odds ratio [OR] = 1.11), non-elective admission (OR = 1.30), greater tumor volume (OR = 1.04 per 1 cm 3 increase), and higher ASA score (OR = 1.22 per 1 point increase) were all associated with higher odds of prolonged hospital LOS. Further, greater surgeon years of experience was associated with a lower odds of prolonged hospital LOS (OR = 0.95 per additional year of experience). Regarding discharge disposition, older patient age (OR = 1.55 per 1 year increase), Medicare or Medicaid insurance status (OR = 1.11), WHO grade II/III vs I (OR = 1.13), greater tumor volume (OR = 1.51 per 1 cm 3 increase), middle relative to anterior fossa tumor location (OR = 1.11), higher ASA score (OR = 1.36 per 1 point increase), higher mFI-5 score (OR = 1.07 per 1 point increase), and symptomatic presentation (OR = 1.05) were all associated with increased odds of nonroutine discharge. Male sex (OR = 0.74), non-Caucasian race (OR = 0.83), married marital status (OR = 0.87), non-elective admission status (OR = 0.79), posterior relative to anterior fossa tumor location (OR = 0.93), greater surgeon years of experience (OR = 0.86 per additional year of experience), and endoscopic endonasal approach relative to craniotomy (OR = 0.85) were all associated with decreased odds of nonroutine discharge. Finally, Medicare or Medicaid insurance status (OR = 1.03), non-elective admission (OR = 1.13), and greater tumor volume (OR = 1.46 per 1 cm 3 increase) were all associated with increased odds of high hospital charges. Greater surgeon years of experience (OR = 0.85 per 1 year increase) was the sole variable associated with decreased odds of incurring high hospital charges.

Table 4 displays the predictive performance metrics for our elastic net logistic regression models on our holdout validation datasets. Models predicting prolonged LOS, nonroutine discharge, and high hospital charges achieved AUROCs of 0.798, 0.752, and 0.592, respectively. ROC plots for all three models are displayed in Fig. 1 . The elastic net logistic regression model predicting prolonged LOS achieved an accuracy of 82.1% on its holdout dataset, while the model predicting nonroutine discharge achieved an accuracy of 89.9% and the model predicting high hospital charges achieved an accuracy of 73.4%. Further, the Brier scores of models predicting prolonged LOS, nonroutine discharge, and high hospital charges were of 0.15, 0.084, and 0.19, respectively. All three models demonstrated adequate calibration via the Spiegelhalter Z -test ( p >0.05). Variable importance plots for all three elastic net models are depicted in Fig. 2A–C .

Table 4. Elastic net logistic regression models predictive performance metrics and 95% confidence intervals on holdout validation sets.

Metric Hospital LOS ( n  = 78) Nonroutine discharge ( n  = 79) Hospital charges ( n  = 79)
AUROC 0.798 (0.662–0.900) 0.752 (0.581–0.906) 0.592 (0.445–0.731)
Accuracy (%) 82.1% (71.7%–89.8%) 89.9% (81.0%–95.5%) 73.4% (62.3%–82.7%)
Brier score 0.15 (0.11–0.19) 0.084 (0.037–0.13) 0.19 (0.14–0.24)
Spiegelhalter Z -test p -value 0.16 0.83 0.64
Open in a new tab

Abbreviation: LOS, length of stay.

Fig. 1.

Fig. 1

Open in a new tab

ROC curves for LOS, discharge, and hospital charges on holdout validation sets. LOS, length of stay; ROC, receiver operating characteristic curve.

Fig. 2.

Fig. 2

Open in a new tab

Variable importance plots for elastic net logistic regression models predicting (A) LOS, (B) discharge, and (C) hospital charges. LOS, length of stay.

Discussion

Prior Research

Previous research within the neurosurgical literature has utilized ML to predict postoperative outcomes such as LOS, discharge disposition, and hospital charges in brain tumor patients. A 2017 study by Muhlestein et al analyzed the impact of medical comorbidities on discharge disposition and LOS following craniotomy for brain tumor patients listed within the National Inpatient Sample. 13 The authors created ML ensemble models predicting discharge disposition and LOS >7 days with AUROC of 0.796 and 0.824, respectively. The authors also found that preoperative paralysis, fluid/electrolyte abnormalities, and other non-paralysis neurological defects most strongly influenced the ensemble model predicting prolonged LOS. 13 In a separate study, Muhlestein et al also developed ML models that directly predicted total charges for transsphenoidal surgery for pituitary tumors, and the investigators identified extended LOS, nonelective admission type, non-Southern hospital region, minority race, postoperative complication, and private investor hospital ownership as drivers of total charges and therefore as potential targets for cost-lowering interventions. 24 Within the orthopaedics literature, a 2018 study by Navarro et al used a naïve Bayesian ML algorithm to predict LOS and inpatient costs following total knee arthroplasty. The authors noted that their ML algorithm demonstrated high validity for predicting LOS and hospital charges, with AUROCs of 0.782 and 0.738, respectively. 25 While our prior work has utilized logistic regression models and inferential statistics to identify predictors of high-value care outcomes among skull base meningioma patients as well as to obtain predictive performance metrics for these models, the present study represents the first effort (to our knowledge) of applying ML methods to model these outcomes. 26

Present Study

The present work sought to apply an ML-workflow to prognosticating postoperative outcomes among skull base meningioma patients. Additionally, we also evaluated the discrimination and calibration of our predictive algorithms to assess whether preoperative patient demographic and clinical characteristics could effectively predict high-value care metrics. Overall, our two elastic net logistic regression models predicting prolonged hospital LOS and nonroutine discharge disposition demonstrated adequate discrimination (AUROC >0.70) and calibration (Spiegelhalter's Z -test p -value >0.05). Importantly, while our model predicting high hospital charges demonstrated adequate calibration, its AUROC of 0.592 suggests inadequate discrimination that would likely not be clinically useful. One potential reason for the our limited ability to predict high hospital charges preoperatively is that the charges incurred by a patient during their hospital stay are mainly influenced by intra- and postoperative variables such as surgery duration, postoperative complication, and total hospital LOS, as detailed in prior studies. 24 27 28 Our results highlight that while ML algorithms may be useful for predicting postoperative risk of prolonged LOS and nonroutine discharge disposition, the preoperative variables utilized in the present study are likely not sufficient for effectively predicting a patient's risk of incurring high hospital charges.

Further, our use of the elastic net logistic regression algorithm allowed us to calculate odds ratios for each model covariate to determine whether a given variable was associated with a higher or lower odds of a given postoperative outcome. Our results demonstrating that Medicare or Medicaid insurance, non-elective admission, greater tumor volume, higher ASA score, and less surgeon years of experience were all associated with increased odds of prolonged hospital LOS corroborates prior findings within the neurosurgical and spine surgery literature. 11 29 30 31 Further, the fact that older patient age, Medicare or Medicaid insurance, greater tumor volume, higher ASA score, and higher mFI-5 score are all associated with higher odds of nonroutine discharge also validates prior research findings. 32 33 34 35 Finally, the fact that Medicare or Medicaid insurance, non-elective admission, greater tumor volume, and less surgeon years of experience were all associated with higher odds in high hospital charges according to our predictive model is also in line with previous research. 16 30 36 37 38 39 40

Interestingly, several associations that were significant in bivariate analysis were excluded from the final predictive models during the training process (such as the significant association between older patient age and higher odds of prolonged LOS), and some associations that were not significant in bivariate analysis were included as inputs in the final elastic net models (such as the association between patient sex and discharge disposition). This also highlights the importance of differentiating between inferential statistics and predictive analytics. While inferential methods such as generalized linear models make use of probabilistic assumptions and hypothesis testing to provide a mathematical guarantee regarding the underlying structure and behavior of associations observed in datasets, predictive analytic methods such as ML algorithms focus mainly on achieving superior predictive performance (AUROC, calibration) on out-of-sample datasets with a lesser emphasis on representing the data-generating mechanism between model input and output. 41 Linear and logistic regression methods represent examples of inferential approaches where the probabilistic, stochastic structure of the models allows for calculation of meaningful confidence intervals and p -values that can be interpreted to yield insight regarding specific relationships between model inputs and outputs. Deep neural networks, on the other hand, are examples of algorithms that can achieve excellent performance metrics when predicting complex, non-linear relationships, but also provide little insight regarding the underlying data-generating relationship and thus how such predictions are being made. 42 43

The present study developed three predictive models using elastic net regularization, an ML algorithm that accomplishes data-driven variable selection and allowed us to train two models predicting prolonged LOS and nonroutine discharge disposition which demonstrated good calibration and discrimination on out-of-sample datasets. Elastic net logistic regression was also useful because we are able to calculate β-coefficients and odds ratios to gain some degree of understanding regarding how our model inputs were producing our model outputs. However, given that the elastic net regularization method does not have an underlying probabilistic structure like ordinary-least-square linear or logistic regression that would allow for the calculation of confidence intervals and p -values, it is important to keep in mind that we are limited regarding our inferences about the statistical relationships among model inputs and output. 44 45 Overall, our study demonstrates that ML methodology can be applied to prognosticate postoperative outcomes among skull base meningioma patients with reasonable predictive performance. However, investigators must be mindful about whether their priority is achieving superior predictive performance or attaining a better understanding of the underlying statistical relationships in their data. Further, given the mixed results of ML predictive performance relative to traditional statistical techniques within the medical literature, one cannot assume a priori that ML always leads to superior predictive performance compared with linear or logistic regression and therefore should empirically assess the performance of their algorithms to see if they attain acceptable levels of discrimination and calibration for their outcomes of interest. 46 47 48 49

Limitations

The present study is retrospective and is limited in its analysis of patient data from a single academic, medical institution during a restricted time period (2016–2019). The retrospective design of our study prevents us from commenting on any causal relationships that may exist between the variables that we analyzed. External validation of our findings in an independent cohort of skull base meningioma patients would be ideal to ensure the generalizability of our findings; this validation provides an avenue for future research. As all patients in this study were surgically managed, the model is not valid for patients who are treated only with non-surgical approaches such as radiotherapy. Additionally, the present study only estimated total hospital charges as opposed to costs. Hospital charges are initial hospital list prices for services while costs represent actual expenses incurred during a patient's hospitalization, and while charges and costs are not synonymous, we agree with previous investigators that charges may represent a useful proxy for costs. 24 Applying ML methods to estimate charges directly may serve as an avenue for future research efforts. Another important limitation is the small number of EEA cases in our patient cohort ( n  = 9). Additional research efforts incorporating greater number of EEA surgeries may be needed to definitively determine whether surgical approach is a useful prognostic variable to consider when using ML to predict high-value postoperative outcomes among skull base meningioma patients. Acknowledging these limitations, the present study has developed and internally validated predictive models that may be useful in optimizing postoperative outcomes and increasing the provision of high-value health care.

Conclusion

Our study developed three ML models predicting prolonged hospital LOS, nonroutine discharge disposition, and high hospital charges among skull base meningioma patients. Using preoperative demographic and clinical variables, our models predicting hospital LOS and nonroutine discharge disposition demonstrated adequate discrimination and calibration, and highlight the utility of ML as a tool to aid neurosurgeons in providing high-value health care and optimizing clinical workflows.

Funding Statement

Funding The authors received no financial support for the research, authorship, and/or publication of this article. The authors acknowledge assistance for clinical data coordination and retrieval from the Core for Clinical Research Data Acquisition, supported in part by the Johns Hopkins Institute for Clinical and Translational Research (UL1TR001079).

Conflict of Interest None declared.

Reporting Guidelines

The authors found no applicable reporting guidelines that would apply to this article. By following the EQUATOR reporting guidelines decision tree, ( http://www.equatornetwork.org/wp-content/uploads/2013/11/20160226-RG-decision-tree-for-Wizard-CC-BY-26 - February-2016.pdf), we found that none of the most popular checklists are appropriate for our study design.

References

  • 1.Ostrom Q T, Gittleman H, Fulop J.CBTRUS statistical Report: primary brain and central nervous system tumors diagnosed in the United States in 2008–2012 Neuro-oncol 201517(suppl 4):iv1–iv62. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Wang N, Osswald M. Meningiomas: overview and new directions in therapy. Semin Neurol. 2018;38(01):112–120. doi: 10.1055/s-0038-1636502. [DOI] [PubMed] [Google Scholar]
  • 3.Meling T R, Da Broi M, Scheie D, Helseth E. Meningiomas: skull base versus non-skull base. Neurosurg Rev. 2019;42(01):163–173. doi: 10.1007/s10143-018-0976-7. [DOI] [PubMed] [Google Scholar]
  • 4.Voß K M, Spille D C, Sauerland C. The Simpson grading in meningioma surgery: does the tumor location influence the prognostic value? J Neurooncol. 2017;133(03):641–651. doi: 10.1007/s11060-017-2481-1. [DOI] [PubMed] [Google Scholar]
  • 5.Chen C M, Huang A PH, Kuo L T, Tu Y K.Contemporary surgical outcome for skull base meningiomas Neurosurg Rev 20113403281–296., discussion 296 [DOI] [PubMed] [Google Scholar]
  • 6.DeMonte F, McDermott M, Al-Mefty O. Al-Mefty's Meningiomas. 2nd ed. Thieme;2011 [Google Scholar]
  • 7.Saklad M. Grading of patients for surgical procedures. Anesthesiology. 1941;2(03):281–284. [Google Scholar]
  • 8.Subramaniam S, Aalberg J J, Soriano R P, Divino C M.New 5-factor modified frailty index using American College of Surgeons NSQIP Data J Am Coll Surg 201822602173–181..e8 [DOI] [PubMed] [Google Scholar]
  • 9.Cahill P J, Pahys J M, Asghar J. The effect of surgeon experience on outcomes of surgery for adolescent idiopathic scoliosis. J Bone Joint Surg Am. 2014;96(16):1333–1339. doi: 10.2106/JBJS.M.01265. [DOI] [PubMed] [Google Scholar]
  • 10.Lau D, Deviren V, Ames C P. The impact of surgeon experience on perioperative complications and operative measures following thoracolumbar 3-column osteotomy for adult spinal deformity: overcoming the learning curve. J Neurosurg Spine. 2019;32(02):207–220. doi: 10.3171/2019.7.SPINE19656. [DOI] [PubMed] [Google Scholar]
  • 11.Dasenbrock H H, Liu K X, Devine C A. Length of hospital stay after craniotomy for tumor: a National Surgical Quality Improvement Program analysis. Neurosurg Focus. 2015;39(06):E12. doi: 10.3171/2015.10.FOCUS15386. [DOI] [PubMed] [Google Scholar]
  • 12.Lakomkin N, Hadjipanayis C G. Resident participation is not associated with postoperative adverse events, reoperation, or prolonged length of stay following craniotomy for brain tumor resection. J Neurooncol. 2017;135(03):613–619. doi: 10.1007/s11060-017-2614-6. [DOI] [PubMed] [Google Scholar]
  • 13.Muhlestein W E, Akagi D S, Chotai S, Chambless L B. The impact of presurgical comorbidities on discharge disposition and length of hospitalization following craniotomy for brain tumor. Surg Neurol Int. 2017;8:220. doi: 10.4103/sni.sni_54_17. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Kalakoti P, Missios S, Menger R, Kukreja S, Konar S, Nanda A. Association of risk factors with unfavorable outcomes after resection of adult benign intradural spine tumors and the effect of hospital volume on outcomes: an analysis of 18, 297 patients across 774 US hospitals using the National Inpatient Sample (2002-2011) Neurosurg Focus. 2015;39(02):E4. doi: 10.3171/2015.5.FOCUS15157. [DOI] [PubMed] [Google Scholar]
  • 15.Muhlestein W E, Akagi D S, Chotai S, Chambless L B. The impact of race on discharge disposition and length of hospitalization after craniotomy for brain tumor. World Neurosurg. 2017;104:24–38. doi: 10.1016/j.wneu.2017.04.061. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Khalafallah A M, Jimenez A E, Patel P, Huq S, Azmeh O, Mukherjee D. A novel online calculator predicting short-term postoperative outcomes in patients with metastatic brain tumors. J Neurooncol. 2020;149(03):429–436. doi: 10.1007/s11060-020-03626-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Brandel M G, Rennert R C, Wali A R. Impact of preoperative endovascular embolization on immediate meningioma resection outcomes. Neurosurg Focus. 2018;44(04):E6. doi: 10.3171/2018.1.FOCUS17751. [DOI] [PubMed] [Google Scholar]
  • 18.Kuhn M.Building predictive models in R using the caret package J Stat Softw 200828051–26.27774042 [Google Scholar]
  • 19.Zou H, Hastie T. Regularization and variable selection via the elastic net. J R Stat Soc Series B Stat Methodol. 2005;67(05):768. [Google Scholar]
  • 20.Bergstra J, Bengio Y. Random search for hyper-parameter optimization. J Mach Learn Res. 2012;13:281–305. [Google Scholar]
  • 21.Swets J A.Measuring the accuracy of diagnostic systems Science 1988240(4857):1285–1293. [DOI] [PubMed] [Google Scholar]
  • 22.Rufibach K.Use of Brier score to assess binary predictions J Clin Epidemiol 20106308938–939., author reply 939 [DOI] [PubMed] [Google Scholar]
  • 23.Clopper C J, Pearson E S. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika. 1934;26(04):404–413. [Google Scholar]
  • 24.Muhlestein W E, Akagi D S, McManus A R, Chambless L B. Machine learning ensemble models predict total charges and drivers of cost for transsphenoidal surgery for pituitary tumor. J Neurosurg. 2018;131(02):507–516. doi: 10.3171/2018.4.JNS18306. [DOI] [PubMed] [Google Scholar]
  • 25.Navarro S M, Wang E Y, Haeberle H S. Machine learning and primary total knee arthroplasty: patient forecasting for a patient-specific payment model. J Arthroplasty. 2018;33(12):3617–3623. doi: 10.1016/j.arth.2018.08.028. [DOI] [PubMed] [Google Scholar]
  • 26.Jimenez A E, Khalafallah A M, Lam S. Predicting high-value care outcomes after surgery for skull base meningiomas. World Neurosurg. 2021;149:e427–e436. doi: 10.1016/j.wneu.2021.02.007. [DOI] [PubMed] [Google Scholar]
  • 27.Little A S, Chapple K. Predictors of resource utilization in transsphenoidal surgery for Cushing disease. J Neurosurg. 2013;119(02):504–511. doi: 10.3171/2013.1.JNS121375. [DOI] [PubMed] [Google Scholar]
  • 28.Zacharia B E, Deibert C, Gupta G. Incidence, cost, and mortality associated with hospital-acquired conditions after resection of cranial neoplasms. Neurosurgery. 2014;74(06):638–647. doi: 10.1227/NEU.0000000000000342. [DOI] [PubMed] [Google Scholar]
  • 29.McKee S P, Yang A, Gray M. Intracranial meningioma surgery: value-based care determinants in New York State, 1995-2015. World Neurosurg. 2018;118:e731–e744. doi: 10.1016/j.wneu.2018.07.030. [DOI] [PubMed] [Google Scholar]
  • 30.Chapman E K, Doctor T, Gal J S. The impact of non-elective admission on cost of care and length of stay in anterior cervical discectomy and fusion: a propensity-matched analysis. Spine. 2021;46(22):1535–1541. doi: 10.1097/BRS.0000000000004127. [DOI] [PubMed] [Google Scholar]
  • 31.Ahn J, Iqbal A, Manning B T. Minimally invasive lumbar decompression-the surgical learning curve. Spine J. 2016;16(08):909–916. doi: 10.1016/j.spinee.2015.07.455. [DOI] [PubMed] [Google Scholar]
  • 32.Huq S, Khalafallah A M, Patel P. Predictive model and online calculator for discharge disposition in brain tumor patients. World Neurosurg. 2021;146:e786–e798. doi: 10.1016/j.wneu.2020.11.018. [DOI] [PubMed] [Google Scholar]
  • 33.Lakomkin N, Hadjipanayis C G. Non-routine discharge disposition is associated with post-discharge complications and 30-day readmissions following craniotomy for brain tumor resection. J Neurooncol. 2018;136(03):595–604. doi: 10.1007/s11060-017-2689-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Sastry R A, Pertsch N J, Tang O, Shao B, Toms S A, Weil R J. Frailty and outcomes after craniotomy for brain tumor. J Clin Neurosci. 2020;81:95–100. doi: 10.1016/j.jocn.2020.09.002. [DOI] [PubMed] [Google Scholar]
  • 35.Muhlestein W E, Akagi D S, Kallos J A. Using a guided machine learning ensemble model to predict discharge disposition following meningioma resection. J Neurol Surg B Skull Base. 2018;79(02):123–130. doi: 10.1055/s-0037-1604393. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Kidwai S M, Yang A, Gray M L. Hospital charge variability across New York State: sociodemographic factors in pituitary surgery. J Neurol Surg B Skull Base. 2019;80(06):612–619. doi: 10.1055/s-0038-1676839. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37.Hamill C S, Villwock J A, Sykes K J, Chamoun R B, Beahm D D. Socioeconomic factors affecting discharge status of patients with uncomplicated transsphenoidal adenohypophysectomy. J Neurol Surg B Skull Base. 2018;79(05):501–507. doi: 10.1055/s-0038-1635095. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Abou-Al-Shaar H, Azab M A, Karsy M, Guan J, Couldwell W T, Jensen R L. Assessment of costs in open microsurgery and stereotactic radiosurgery for intracranial meningiomas. World Neurosurg. 2018;119:e357–e365. doi: 10.1016/j.wneu.2018.07.161. [DOI] [PubMed] [Google Scholar]
  • 39.Alvin M D, Lubelski D, Alam R. Spine surgeon treatment variability: the impact on costs. Global Spine J. 2018;8(05):498–506. doi: 10.1177/2192568217739610. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Doumouras A G, Saleh F, Gmora S, Anvari M, Hong D. The value of surgical experience: excess costs associated with the Roux-en-Y gastric bypass learning curve. Surg Endosc. 2019;33(06):1944–1951. doi: 10.1007/s00464-018-6472-x. [DOI] [PubMed] [Google Scholar]
  • 41.Breiman L. Statistical modeling: the two cultures. Stat Sci. 2001;16(03):199–215. [Google Scholar]
  • 42.Hannun A Y, Rajpurkar P, Haghpanahi M. Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nat Med. 2019;25(01):65–69. doi: 10.1038/s41591-018-0268-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Senior A W, Evans R, Jumper J.Improved protein structure prediction using potentials from deep learning Nature 2020577(7792):706–710. [DOI] [PubMed] [Google Scholar]
  • 44.Lockhart R, Taylor J, Tibshirani R J, Tibshirani R. A significance test for the lasso. Ann Stat. 2014;42(02):413–468. doi: 10.1214/13-AOS1175. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Lee J D, Sun D L, Sun Y, Taylor J E. Exact post-selection inference, with application to the lasso. Ann Stat. 2016;44(03):907–927. [Google Scholar]
  • 46.Christodoulou E, Ma J, Collins G S, Steyerberg E W, Verbakel J Y, Van Calster B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol. 2019;110:12–22. doi: 10.1016/j.jclinepi.2019.02.004. [DOI] [PubMed] [Google Scholar]
  • 47.Marcus A P, Marcus H J, Camp S J, Nandi D, Kitchen N, Thorne L. Improved prediction of surgical resectability in patients with glioblastoma using an artificial neural network. Sci Rep. 2020;10(01):5143. doi: 10.1038/s41598-020-62160-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Bae S, Massie A B, Caffo B S, Jackson K R, Segev D L. Machine learning to predict transplant outcomes: helpful or hype? A national cohort study. Transpl Int. 2020;33(11):1472–1480. doi: 10.1111/tri.13695. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Machine Learning Consortium . Oosterhoff J HF, Gravesteijn B Y, Karhade A V. Feasibility of machine learning and logistic regression algorithms to predict outcome in orthopaedic trauma surgery. J Bone Joint Surg Am. 2022;104(06):544–551. doi: 10.2106/JBJS.21.00341. [DOI] [PubMed] [Google Scholar]

Articles from Journal of Neurological Surgery. Part B, Skull Base are provided here courtesy of Thieme Medical Publishers

RESOURCES