Developing Predictive Models to Anticipate Shunt Complications in 33,248 Pediatric Patients with Shunted Hydrocephalus Utilizing Machine Learning

Abstract

Introduction

Hydrocephalus is a common pediatric neurosurgical pathology, typically treated with a ventricular shunt, yet approximately 30% of patients experience shunt failure within the first year after surgery. As a result, the objective of the present study was to validate a predictive model of pediatric shunt complications with data retrieved from the Healthcare Cost and Utilization Project (HCUP) National Readmissions Database (NRD).

Methods

The HCUP NRD was queried from 2016 to 2017 for pediatric patients undergoing shunt placement using ICD-10 codes. Comorbidities present upon initial admission resulting in shunt placement, Johns Hopkins Adjusted Clinical Groups (JHACG) frailty-defining criteria, and Major Diagnostic Category (MDC) at admission classifications were obtained. The database was divided into training (n = 19,948), validation (n = 6,650), and testing (n = 6,650) datasets. Multivariable analysis was performed to identify significant predictors of shunt complications which were used to develop logistic regression models. Post hoc receiver operating characteristic (ROC) curves were created.

Results

A total of 33,248 pediatric patients aged 6.9 ± 5.7 years were included. Number of diagnoses during primary admission (OR: 1.05, 95% CI: 1.04–1.07) and initial neurological admission diagnoses (OR: 3.83, 95% CI: 3.33–4.42) positively correlated with shunt complications. Female sex (OR: 0.87, 95% CI: 0.76–0.99) and elective admissions (OR: 0.62, 95% CI: 0.53–0.72) negatively correlated with shunt complications. ROC curve for the regression model utilizing all significant predictors of readmission demonstrated area under the curve of 0.733, suggesting these factors are possible predictors of shunt complications in pediatric hydrocephalus.

Conclusion

Efficacious and safe treatment of pediatric hydrocephalus is of paramount importance. Our machine learning algorithm delineated possible variables predictive of shunt complications with good predictive value.

Introduction

Hydrocephalus is one of the most common neurologic pathologies treated by pediatric neurosurgeons [1, 2]. Cerebrospinal fluid (CSF) shunts are considered an effective treatment for hydrocephalus and have historically shown significant benefits in prolonging the lifespan of pediatric hydrocephalus patients [3]. However, CSF shunts are susceptible to complications including obstruction, infection, disconnection, and over/under drainage, with shunt failure rates quoted up to 50% within the first 2 years after placement [3–5].

In addition to patient morbidity related to these complications, there also exists a significant economic impact of pediatric hydrocephalus complications burdening the USA healthcare system with an estimated annual cost of 1–2 billion dollars [6–8]. A prior study found that shunt revision surgery accounted for 42.8% of CSF shunt procedures performed, which could be a key driver for the high costs associated with hydrocephalus diagnoses [6].

Therefore, the identification of predictive variables for shunt complications would allow for a more effective preoperative evaluation of complication risks in hydrocephalus patients. Using these evaluations, providers could better establish the need for a higher degree of postoperative care and surveillance and reduce time to follow-ups which may be advantageous in decreasing revision rates [9]. Furthermore, identifying predictive variables could help determine any potentially modifiable clinical risk factors to reduce these complications by enabling surgeons to address these factors pre- and intra-operatively [10–14].

In an effort to accomplish these goals, the authors developed a machine learning model for predicting shunt complications in pediatric patients utilizing a cohort of 33,248 patients from the Healthcare Cost and Utilization Project (HCUP) National Readmissions Database (NRD). The utilization of machine learning methods can handle multiple diverse inputs, such as variables provided by the NRD. This method improves predictive abilities over solely conventional univariate and multivariate analyses [15–20]. In this technical note, the authors utilized a classic training/validation/testing dataset split to internally and externally validate a guided regression model to predict shunt complications in a large pediatric cohort.

Methods

Data Source

In this study, the authors utilized the HCUP NRD with data collected from years 2016–2017. The NRD is a large database updated yearly, composed of nationally collated inpatient demographics, diagnoses, procedures, and readmissions. Hospital admissions are de-identified and are represented with a unique label to allow for accurate patient tracking. Between all years of NRD included in this study, we identified over 35 million patient discharges. All data regarding patient diagnoses and procedures were queried using International Classification of Diseases, Tenth Revision (ICD-10) codes (0016xJx & Z982).

Patient Selection

All patients with a new insertion or previous insertion of a CSF shunt for hydrocephalus were initially queried (n = 146,630), with the analysis dataset further limited to pediatric patients (<18 years of age; n = 33,248). Frailty status, defined by the Johns Hopkins Adjusted Clinical Groups (JHACG) frailty-defining criteria, was evaluated for each patient as a separate index reflective of comorbidity [21–23]. MDC classifications, which represent 25 diagnostic categories organized by organ system through ICD coding, were obtained for each patient through the NRD. These codes delineate the primary indication for admission into the hospital, thus the reason for initial hospital admission was not necessarily shunt complication.

The primary outcome of interest was evaluating shunt complication, defined as either (1) need for shunt revision or (2) the presence of shunt infection. Because shunt revision and shunt infection are common complications related to CSF shunt placement, they were consolidated into shunt complications. Shunt revision was defined as surgical revision or replacement of a ventricular shunt catheter. Shunt infection was defined as clinical evidence of infection related to the shunt including a positive CSF culture, purulent material from the shunt in association with fever, neurological decline, or increased white blood cell count. These CSF shunt complications were analyzed independently within one calendar year due to the limitations of the NRD query.

Statistical Analysis

All statistical analysis was carried out in RStudio (Version 1.2.5042), statistical tests were two-sided, and all p values <0.05 were defined as significant. The study database was divided into three subsets: 60% were used to create the training dataset (n = 19,948), 20% was used to create the validation dataset (n = 6,650), and the remaining 20% was used to create the testing dataset (n = 6,650).

Following univariate analysis, multivariable analysis was performed using the training dataset to identify significant independent predictors of shunt complications. All statistically significant variables were used to develop logistic regression models in accordance with published and validated techniques previously described in the literature [24, 25]. Parameters and hyperparameters were tuned using the validation dataset over 10 iterations, which indicated that all predictors with p values <0.20 may be considered within the final model to yield the best outcome. As a result, the final logistic regression model was developed using the testing set.

Post hoc receiver operating characteristic (ROC) curves were created using the pROC package following creation of the logistic regression model for shunt complications. The area under the curve (AUC) of each ROC was computed and served as a proxy for model performance. Variable importance was analyzed within each model in conjunction with Wald testing for model optimization. These variable importance metrics (VIMs) allow for estimation of the degree to which changes in our outcome were influenced by changes in individual predictor variables [26]. There was no evidence of statistically poor fit based on Hosmer-Lemeshow testing. In general, an AUC of 0.50 demonstrates a random guess, and AUC values greater than 0.70 defined as acceptable in the prediction of medical conditions [27].

Results

Patient Demographics

A total of 33,248 pediatric patients with shunts were identified for analysis. Demographic and surgical characteristics are reported in Table 1. The average age was 6.9 ± 5.7 years and 56.9% were male (Table 1). With respect to CSF shunt types, the majority of newly placed shunts were ventriculoperitoneal shunts, followed by ventriculoatrial shunts and ventriculopleural shunts, respectively. The most common initial MDC diagnosis upon hospital admission was disorders of the nervous system (45.2%) (Table 2).

Table 1.

Characteristics of patients retrieved in the Healthcare Cost and Utilization Project National Readmissions Database

Variable
All pediatric patients with shunt placement, n	33,248
Age, years	6.9±5.7
Sex, n (%)
Female	14,330 (43)
Male	18,918 (57)
Elixhauser comorbidity index	1.6±1.7
Mean number of diagnoses	11.6±7.3
Insurance, n (%)
Medicare	56 (0.2)
Medicaid	19,785 (60)
Private	11,618 (35)
Other	1,789 (5)
Median income by zip code, n (%)
Quartile 1	10,016 (30)
Quartile 2	9,538 (29)
Quartile 3	8,256 (25)
Quartile 4	5,096 (15)
Other	342 (1.0)
Hospital type, n (%)
Metropolitan non-teaching	1,667 (5.0)
Metropolitan teaching	31,307 (94)
Non-metropolitan	275 (0.8)
Discharge location, n (%)
Routine/home	29,119 (88)
Non-routine	4,120 (12)
Admission type, n (%)
Elective	9,360 (28)
Non-elective	23,888 (72)
Shunt type, n (%)
Current placement of ventriculoatrial shunt	261 (0.8)
Current placement of ventriculopleural shunt	137 (0.4)
Current placement of ventriculoperitoneal shunt	9,280 (28)
Previous shunt placement/other	23,570 (71)
Shunt complications, n (%)
Shunt complications during primary admission	3,487 (11)
Shunt complications during readmission	4,237 (13)
Any shunt complications (non-overlapping)	6,820 (21)

Table 2.

Major diagnostic category codes for initial hospital admissions (n = 33,248)

Major diagnostic category diseases and disorders	Patients, n (%)
Pre-system coding	54 (0.2)
Nervous system	15,026 (45)
Eye	61 (0.2)
Ear, nose, mouth, And throat	907 (2.7)
Respiratory system	3,764 (11)
Circulatory system	411 (1.2)
Digestive system	2,493 (8)
Hepatobiliary system and pancreas	92 (0.3)
Musculoskeletal system and connective tissue	2,345 (7)
Skin, subcutaneous tissue, and breast	497 (1.5)
Endocrine, nutritional, and metabolic system	1,178 (3.5)
Kidney and urinary tract	1,508 (4.5)
Male reproductive system	40 (0.1)
Female reproductive system	21 (0.06)
Pregnancy, childbirth, and puerperium	16 (0.05)
Newborn and other neonates (perinatal period)	1,603 (4.8)
Blood and blood-forming organs and immunological disorders	236 (0.7)
Myeloproliferative disorders (poorly differentiated neoplasms)	373 (1.1)
Infectious and parasitic disorders (systemic or unspecified sites)	1,203 (3.6)
Mental diseases and disorders	264 (0.8)
Alcohol/drug use or induced mental disorders	3 (0.009)
Injuries, poison, and toxic effect of drugs	684 (2.1)
Burns	23 (0.07)
Factors influencing health status and other contacts with health services	392 (1.2)
Multiple significant trauma	54 (0.2)

Multivariable Analysis

Following multivariable analysis using the training dataset, several predictor variables were identified. Variables found to be positively correlated with shunt complications included the following: increasing number of diagnoses during primary admission (OR: 1.05, 95% CI: 1.04–1.07) and having initial neurological diagnoses upon admission to the hospital (OR: 3.83, 95% CI: 3.33–4.42) (Table 3). Variables found to be negatively correlated with shunt complications included female sex (OR: 0.87, 95% CI: 0.76–0.99) and elective admissions (OR: 0.62, 95% CI: 0.53–0.72).

Table 3.

Multivariable analysis on the training dataset

Variable	Odds ratio	95% confidence interval	p value
Age	1.00	0.99–1.01	0.95
Female sex	0.87	0.76–0.99	0.039*
Decreasing frailty score (JHACG)	0.86	0.71–1.04	0.12**
Private insurance status	1.06	0.99–1.14	0.11**
Increasing median income quartile by zip code	0.99	0.93–1.06	0.86
Hospital type	0.91	0.70–1.20	0.51
Increasing number of diagnoses	1.05	1.04–1.07	<0.0001*
Malnutrition	1.04	0.74–1.45	0.81
Elective admission	0.62	0.53–0.72	<0.0001*
Nonroutine discharge	0.84	0.68–1.04	0.12**
Increasing hospital bed size	1.01	0.92–1.11	0.85
Neurological MDCs	3.83	3.33–4.42	<0.0001*

MDC, major diagnostic category.

*Statistically significant. All statistically significant variables were included in the final regression model.

**Not statistically significant but found to be a predictive variable upon tuning parameters/hyperparameters using the validation dataset.

Predictive Model Analysis

VIMs for all variables included in the final logistic regression model utilizing the testing dataset are reported in Table 4. Of these, increasing number of diagnoses, elective admissions, and initial admission for neurological disorders were found to be significant VIMs via Wald test. The ROC curve for the regression model demonstrated an AUC of 0.733 (Fig. 1). Decreasing frailty score, private insurance status, and nonroutine discharge variables were found to be predictive upon tuning parameters/hyperparameters of the validation dataset and were included in the predictive model. However, they were found not to be a VIM after Wald test assessment.

Table 4.

Variable importance in final regression model

Variable	Wald test	VIM
Female Sex	0.84	0.20
Decreasing frailty score (JHACG)	0.19	1.32
Private insurance status	0.42	0.80
Increasing number of diagnoses	<0.0001*	5.79
Elective admission	<0.0001*	4.45
Nonroutine discharge	0.25	1.15
Neurological MDCs	<0.0001*	12.81

JHACG, Johns Hopkins Adjusted Clinical Groups; VIMs, variable importance metrics.

*Statistically significant in the final regression model.

Fig. 1.

Post hoc receiver operating characteristic (ROC) curve for the prediction of shunt complications warranting readmission. The area under the curve (AUC) achieved here demonstrates acceptable prediction of shunt outcome using preoperative patient characteristics.

Discussion

CSF shunt complications, including mechanical failure, infection, as well as over/under drainage, represent a challenging scenario in pediatric patients with hydrocephalus. Approximately 15% of children may experience shunt failure within 30 days, and 30% of children will require surgical revision within the first year [28, 29]. Nearly 33% of shunt failures may be considered preventable, most often resulting from infection and proximal catheter malposition [13, 14]. In a single-center study spanning 15 years, Stone et al. [30] reported 84.5% of pediatric patients undergoing ventriculoperitoneal shunting required at least 1 shunt revision.

Utilizing machine learning, we developed a predictive model of shunt complications in pediatric patients using a cohort of 33,248 patients. The AUC value of the model (0.733) demonstrates accuracy in distinguishing patients at risk for shunt complications. To our knowledge, this represents the largest dataset of pediatric hydrocephalus patients examined in a study regarding shunt complications.

Machine Learning for Pediatric Hydrocephalus

Machine learning has proved to be a highly useful tool in medicine, allowing for detailed data analysis and disease state prediction. Particularly, when dealing with large quantities of patient data, machine learning algorithms may allow for the detection of patterns that may not have been previously evident. There are few reports in the literature of utilizing machine learning to predict pediatric shunt failure. Habibi et al. [31] utilized an artificial neural network in the prediction of ventriculoperitoneal shunt infections. These authors reported an AUC of 0.557 for a logistic regression model. Although our outcome has shunt complications rather than solely infections, our logistic regression demonstrated superior accuracy with an AUC of 0.733. This may be due to the methodological strengths of our study, including a larger sample size, examination of a wide variety of variables, and use of validation dataset. A recent study by Hale et al. [9] investigated support vector machine, naïve Bayesian, k-nearest neighbor, and artificial neural network algorithms for the prediction of shunt failure in 1,036 patients. The ANNs exhibited the highest AUC of 0.71 when taking into consideration a robust model of 38 clinical, radiological, surgical, and shunt-design variables. While the AUC of their approach was sufficient, their machine learning methods utilized non-parametric analysis and may not be best optimized for larger, parametric datasets often seen in patient variables. Logistic regression models, as a parametric method, typically have the highest potential predictive value for this type of data. As seen in our investigation, logistic regression performed quite well for the prediction of shunt complications using mostly parametric data from the NRD.

Identified Risk Factors for Shunt Complications Identified

This logistic regression model identified variables positively and negatively associated with shunt complications in pediatric hydrocephalus. Increasing number of diagnoses during the primary admission and having neurological diagnoses as the initial indication for hospital admission was associated with increased likelihood of shunt complications. These variables likely represent surrogates of patient complexity, and the current literature suggest that medically complex hydrocephalus patients are at increased risk for shunt complications [32]. Although these results are not novel, the large population size used in our model serves to validate this as a relevant risk factor when evaluating shunt complications.

Elective admissions and female sex were negatively associated with future shunt complications. Elective procedures may be associated with fewer shunt complications due to decreased severity of shunt failure symptoms. This is consistent with the prior literature, indicating that routine clinic visits after shunt placement yields better outcomes and lower the rates of revision surgery [33]. Conversely, patients who present with acute symptoms of shunt malfunction often undergo urgent shunt revision surgery. After hours, non-elective shunt surgery has been associated with increased shunt complication risk and the need for reoperation within 30 days [34].

We found some statistical evidence that female sex could be a novel factor that was associated with reduced risk of shunt complications in pediatric hydrocephalus. However, this did not become a VIM in the machine learning model. There are data in the adult neurosurgery literature that suggests males are at higher risk of both shunt infection and multiple revision surgeries [35]. However, several studies within the pediatric neurosurgery literature have not suggested sex as a potential risk factor for shunt complications [14, 36]. One possibility is that we included considerably more patients in our model than in previous reports (5–15x) and this increase in sample size was required to detect a significant difference. This warrants future consideration in future studies investigating the role of sex in pediatric shunt outcomes.

Unfortunately, several of the variables identified by our model are nonmodifiable. However, this is not an uncommon finding among studies examining risk factors for shunt failure [37]. Although they cannot be altered directly, these may be targeted through increased awareness, additional research investigations, and quality improvement initiatives. First, the variables included within our model can be incorporated into strategies to augment clinical decision making, especially in patients where clinical symptoms of shunt malfunction may be ambiguous or in patients who do not experience ventricular dilation with shunt malfunction [38]. Second, in patients with higher chances of having shunt complications when considering these data, neurosurgeons can opt for more frequent monitoring of the patients or potentially weigh risks and benefits of proceeding with endoscopic third ventriculostomy as an alternative procedure. Third, wider implementation of protocols to minimize shunt complications is necessary. The Hydrocephalus Clinical Research Network published a protocol in 2011 and 2016 on reduced shunt infection and has since been adapted into the Calgary Shunt Protocol [36, 39, 40]. Future studies can further investigate our identified risk factors for evaluation in a prospective clinical setting. Given the AUC of our logistic regression model in parametrically analyzing large datasets, our model should be applied in the identification and substantiation of risk factors of complications from other routine neurosurgical treatments.

Limitations

This study is not without limitations. First, we recognize our model was unable to identify modifiable patient variables. Our intention was to utilize machine learning algorithms to accurately analyze a large dataset and to identify possible clinical risk factors for shunt complications. We believe we achieved the former, however the data obtained from the NRD database, specifically the use of MDCs, lacked granularity to extract precise details from each patient. This prevented inclusion of specific peri- and intra-operative factors that may be critical predictor variables that could be potentially modifiable. This study was unable to account for pertinent demographical variables, radiological variables, information on the etiology of hydrocephalus, and information on surgical procedure. Comorbidity categories were broad and prevented determination of detailed epidemiology factors that may also be associated with shunt complications. Nonetheless, we believe that our results still contain some value for prognostication. Ideally, future studies will utilize our machine learning algorithm in a highly detailed dataset to better identify modifiable variables for shunt complications. Second, as a retrospective cohort study, the study is prone to bias. The NRD includes approximately 60% of the population of the USA and significant number of patients received the treatment from metropolitan teaching hospitals, selection bias may be present. National de-identified databases may also be subject to coding errors and differences in coding practices between participating hospitals. These databases’ direct extraction of patient-reported outcomes may be associated with shunt complications due to the lack of specific ICD-10 codes. Finally, the study is limited by its narrow time range (1 year). However, the dates were selected because mandatory ICD-10 coding began in late 2015, which allowed for the extraction of detailed codes for the purpose of analysis. The study was able to achieve greater granularity than studies using ICD-9 codes.

Conclusion

Despite their widespread use and clinical efficacy, complications in CSF shunts placed for pediatric hydrocephalus are common. In this technical note, we utilized machine learning to analyze a cohort of 33,248 pediatric hydrocephalus patients and developed a highly predictive model for shunt complications. Both an increased number of diagnoses and an initial neurological diagnosis upon hospital admission predicted higher rates of shunt complications. Elective shunt surgeries predicted fewer shunt complications. Further investigation is warranted to fully optimize the model and to understand the influence of these potential risk factors on CSF shunt complications in pediatric hydrocephalus. The algorithms showcased in this study can be expanded to analyze more granular data to identify modifiable factors in the future.

Acknowledgments

Authors Dr. Peter A. Chiarelli and Dr. Carli L. Bullis were not available to confirm co-authorship, but the corresponding author Dr. Shane Shahrestani affirms that authors Peter Chiarelli and Carli Bullis contributed to the paper, had the opportunity to review the final version to be published and guarantees authors Peter Chiarelli and Carli Bullis co-authorship status and the accuracy of the author contribution and conflict of interest statements.

Statement of Ethics

Ethical approval and consent were not required as this study was based on publicly available data.

Conflict of Interest Statement

The authors have no conflicts of interest to declare.

Funding Sources

There are no disclosures of funding for this study.

Author Contributions

The authors made the following contributions to this study: Shane Shahrestani and Nathan Shlobin: study conception, study design, data retrieval, data analysis, and manuscript drafting. Julian L. Gendreau: methodology, data analysis, and manuscript drafting. Nolan J. Brown and Alexander Himstead: data analysis, manuscript drafting, and visual data presentation. Neal A. Patel: manuscript revision and critical review. Noah Pierzchajlo and Sachiv Chakravarti: manuscript revision, critical review, and submission. Darrin Jason Lee, Peter A. Chiarelli, and Carli L. Bullis: critical review and commentary. Jason Chu: group supervisor and critical review.

Funding Statement

There are no disclosures of funding for this study.

Data Availability Statement

The data in this study were obtained from Healthcare Cost and Utilization Project (HCUP) National Readmissions Database (NRD) where restrictions may apply. The data that support the findings of this study are openly available in the HCUP NRD at https://hcup-us.ahrq.gov/nrdoverview.jsp.