Machine Learning to Predict Successful Opioid Dose Reduction or Stabilization After Spinal Cord Stimulation

Abstract

BACKGROUND:

Spinal cord stimulation (SCS) effectively reduces opioid usage in some patients, but preoperatively, there is no objective measure to predict who will most benefit.

OBJECTIVE:

To predict successful reduction or stabilization of opioid usage after SCS using machine learning models we developed and to assess if deep learning provides a significant benefit over logistic regression (LR).

METHODS:

We used the IBM MarketScan national databases to identify patients undergoing SCS from 2010 to 2015. Our models predict surgical success as defined by opioid dose stability or reduction 1 year after SCS. We incorporated 30 predictors, primarily regarding medication patterns and comorbidities. Two machine learning algorithms were applied: LR with recursive feature elimination and deep neural networks (DNNs). To compare model performances, we used nested 5-fold cross-validation to calculate area under the receiver operating characteristic curve (AUROC).

RESULTS:

The final cohort included 7022 patients, of whom 66.9% had successful surgery. Our 5-variable LR performed comparably with the full 30-variable version (AUROC difference <0.01). The DNN and 5-variable LR models demonstrated similar AUROCs of 0.740 (95% CI, 0.727-0.753) and 0.737 (95% CI, 0.728-0.746) (P = .25), respectively. The simplified model can be accessed at SurgicalML.com.

CONCLUSION:

We present the first machine learning–based models for predicting reduction or stabilization of opioid usage after SCS. The DNN and 5-variable LR models demonstrated comparable performances, with the latter revealing significant associations with patients' pre-SCS pharmacologic patterns. This simplified, interpretable LR model may augment patient and surgeon decision making regarding SCS.

ABBREVIATIONS:

AUROC

area under the receiver operating characteristic curve

DNNs

deep neural networks

LR

logistic regression

ML

machine learning

RFE

recursive feature elimination

SCS

spinal cord stimulation.

Opioid misuse is a public health crisis in the United States (US), accounting for almost 47 000 deaths in 2018 alone and more than 750 000 deaths since 1999,^1,2 without showing evidence of significant slowing according to data from the National Institute on Drug Abuse.³ The opioid epidemic has also resulted in significant cost to the US healthcare system, including more than $650 million in 2012 opioid-related hospital admissions⁴ and $78.5 billion in 2013 in total economic burden.⁵ Spinal cord stimulation (SCS), which we have previously associated with reduced opioid usage in a national cohort,⁶ represents a possible solution to help tackle the epidemic. Because SCS is a surgical solution with high up-front costs, its appropriate use relies on selecting patients with the highest chances of success⁷ and counseling patients about postsurgical expectations. Machine learning (ML) may enable these predictions.

SCS can effectively treat various chronic pain syndromes.⁷ In addition to improving quality of life and decreasing overall costs,^8,9 SCS is also associated with reduction in opioid use.^6,10 However, preoperatively, there are no objective criteria to predict who will most benefit. Given that up to 30% of patients fail to achieve long-term pain relief,¹¹ previous studies have attempted to identify factors associated with SCS success and found myriad potential variables, including psychiatric and medical comorbidities, pain etiology, surgical history, and medication history.^6,11-13

ML may allow translation of these associations into clinically useful predictions. ML has increasingly been applied within neurosurgery, spanning applications as diverse as predicting readmission after spine surgery, survival in patients with brain tumor, and outcomes after traumatic brain injury.^14,15 In the case of chronic pain, ML may be used to identify patients most likely to benefit from SCS, therefore informing clinician and patient decision making.

One of the most common algorithms used for ML is logistic regression (LR), a generalized linear model that, in the form used here, assumes a linear relationship between predictors and the log odds of the outcome.¹⁶ To uncover more complex, nonlinear associations, more advanced algorithms such as neural networks (with various numbers of “hidden layers” resulting in different model depths and complexities) may be used.¹⁷

In light of the American opioid crisis, understanding new solutions is of critical importance. Here, we present ML models to predict surgical success using the largest available cohort of SCS patients in the United States. We compare the efficacy of two ML strategies to predict successful reduction or stabilization of opioid usage after SCS: LR optimized with recursive feature elimination (RFE) and deep neural networks (DNNs), ultimately aiming to create a deployable model to augment patient and physician decision making.

METHODS

Ethical Approval and Reporting Guidelines

Institutional Review Board exemption was obtained for this study using deidentified data. This study followed the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis¹⁸ reporting guidelines.

Data Source and Cohort Identification

We used the IBM MarketScan databases, consisting of more than 200 million unique patients in the United States, to retrospectively identify the cohort. The starting data source and primary inclusion/exclusion criteria for this work are similar to that reported previously by our group.⁶ Patients had age ≥18 years, underwent SCS trial lead implant after January 1, 2010, and underwent permanent implantable pulse generator implant within 90 days of the trial and before December 31, 2015. Notably, patients were excluded if they took no opioids before SCS because the prediction of opioid dose change relies on preoperative use.

Data Availability Statement

The IBM MarketScan databases are available for purchase at https://www.ibm.com/products/marketscan-research-databases.

Outcome

We aimed to predict successful opioid dose reduction or stabilization (outcome = 1; “success”) vs opioid dose increase (outcome = 0; “failure”), measured 1 year after implantable pulse generator implant, representing a period when medication usage should stabilize postoperatively and one at which SCS outcomes are commonly assessed.¹⁹ This classification is based on clinically significant mobility between opioid dosage groups as defined by the Centers for Disease Control and Prevention guidelines.²⁰ Patients were grouped, both preoperatively (3-month period leading up to SCS trial implant) and postoperatively (10th, 11th, and 12th month after implantable pulse generator implant), according to average daily milligram morphine equivalents as follows: ≤20, 21-50, 51-90, and >90. For the group with >90 daily milligram morphine equivalents, the outcome was only considered successful if patients moved to a lower dose group. In other groups, opioid dose stabilization was considered successful because chronic pain patients often suffer from worsening pain and increasing medication requirements over time.²¹

Predictors and Data Preprocessing

Initially, 45 variables were included as potential predictors based on expert opinion of the authors, mostly related to medications, comorbidities, and SCS indications. Comorbidities included tobacco use and the 30 conditions comprising the Elixhauser Comorbidity Index²² (all derived from claims within one year after the SCS trial). SCS indications included back pain, limb pain, postlaminectomy syndrome, complex regional pain syndrome, neuritis/radiculitis, and degenerative disk disease.

For details on preprocessing, see Supplementary Methods, http://links.lww.com/NEU/D70.

Machine Learning Algorithms and Hyperparameter Tuning

We used two ML algorithmic strategies: (1) LR optimized with RFE and (2) DNNs with 2 to 5 hidden layers. Note that any neural network with more than 1 hidden layer is considered a deep (rather than shallow) neural network.²³ We tuned hyperparameters (Supplementary Methods, http://links.lww.com/NEU/D70) using a grid search in stratified 5-fold cross-validation over the entire data set, searching for the hyperparameter combination yielding the highest area under the receiver operating characteristic curve (AUROC).

Internal Validation Strategy and Model Performance Assessment

To acquire performance estimates for our modeling strategies that were unbiased by the hyperparameter tuning process, we used nested cross-validation with both inner and outer aspects comprising stratified 5-fold cross-validation. We assessed algorithms' discrimination using AUROC and calibration using calibration curves after isotonic calibration. The 95% CI for the AUROC was calculated using 2000 stratified bootstrap replicates of predictions from the validation folds. We assessed statistical significance of the AUROC difference between the two algorithms using the DeLong test.²⁴ The details are provided in Supplementary Methods, http://links.lww.com/NEU/D70.

RESULTS

Cohort Characteristics

In total, 7022 patients met inclusion criteria and 4699 (66.9%) had a successful outcome. Table presents values and definitions (when relevant) for all 30 predictors across the successful vs unsuccessful SCS groups. The largest differences were medication patterns. Patients who underwent successful surgery tended to use lower opioid doses before SCS and less frequently used long-term opioids (78.8% vs 95.4%), concurrent benzodiazepines (33.6% vs 43.4%), or other pain medications (59.7% vs 66.9%). They were also slightly less likely to exhibit polypharmacy (74.3% vs 77.1%). Overall, both groups had similar comorbidity profiles.

TABLE.

Patient Characteristics

Characteristic	No successful outcome (n = 2323)	Yes successful outcome (n = 4699)
Sex
Male	958 (41.2%)	1824 (38.8%)
Female	1365 (58.8%)	2875 (61.2%)
History of spine surgery in the past year	170 (7.3%)	326 (6.9%)
Pharmacologic
PreSCS opioid dose group
≤20 MME	447 (19.2%)	1908 (40.6%)
21-50 MME	378 (16.3%)	1443 (30.7%)
51-90 MME	244 (10.5%)	822 (17.5%)
>90 MME	1254 (54.0%)	526 (11.2%)
Long-term opioid usagea	2217 (95.4%)	3701 (78.8%)
Polypharmacyb	1792 (77.1%)	3490 (74.3%)
Concurrent benzodiazepine usagec	1009 (43.4%)	1581 (33.6%)
Other pain medications usaged	1554 (66.9%)	2807 (59.7%)
Chronic pain diagnoses
Back pain	1076 (46.3%)	2087 (44.4%)
Limb pain	225 (9.7%)	428 (9.1%)
Postlaminectomy syndrome	1114 (48.0%)	2156 (45.9%)
Complex regional pain syndrome	181 (7.8%)	342 (7.3%)
Neuritis/radiculitis	1205 (51.9%)	2453 (52.2%)
Degenerative disk disease	705 (30.3%)	1460 (31.1%)
Comorbidities
Valvular disease	156 (6.7%)	351 (7.5%)
Peripheral vascular disorder	197 (8.5%)	465 (9.9%)
Hypertension (uncomplicated)	1264 (54.4%)	2638 (56.1%)
Deficiency anemias	332 (14.3%)	581 (12.4%)
Chronic pulmonary disease	534 (23.0%)	1101 (23.4%)
Diabetes without chronic complications	516 (22.2%)	1084 (23.1%)
Diabetes with chronic complications	181 (7.8%)	388 (8.3%)
Hypothyroidism	367 (15.8%)	739 (15.7%)
Rheumatoid arthritis/collagen vascular disease	487 (21.0%)	950 (20.2%)
Fluid and electrolyte disorders	267 (11.5%)	454 (9.7%)
Solid tumor without metastasis	109 (4.7%)	274 (5.8%)
Obesity	317 (13.6%)	607 (12.9%)
Psychoses	1118 (48.1%)	1914 (40.7%)
Depression	797 (34.3%)	1412 (30.0%)
Drug abuse	259 (11.1%)	360 (7.7%)
Tobacco abuse	334 (14.4%)	567 (12.1%)
Sleep-disordered breathing	480 (20.7%)	1021 (21.7%)

MME, milligram morphine equivalents; SCS, spinal cord stimulation.

^aOpioid usage for >90 d within the 1-y pre-SCS period.

^bUsing more than 5 types of medications during the 3-mo period before SCS trial.

^cAt least 1 d of overlap between opioids and benzodiazepine prescriptions during the 3-mo period before SCS trial.

^dUsing nonsteroidal anti-inflammatory drugs, muscle relaxers, anticonvulsants, or analgesics for >90 d within the 1-y pre-SCS period.

Those with successful SCS had less intense medication patterns. Comorbidity profiles in both groups were similar.

Model Performances

We found similar AUROCs of 0.736 (95% CI, 0.723-0.749) for the 5-variable LR (see details below) and 0.740 (95% CI, 0.727-0.753) for the DNN (P = .25; Figure 1). Isotonic calibration was effective for both models, as indicated by the linear calibration plots for both the LR and DNN algorithms (Figure 2).

FIGURE 1.

Receiver operating characteristic curves for both the DNN and 5-variable LR models. There is no significant difference in the summative area under the receiver operating characteristic curve performance metric. Using the Youden index to render a decision threshold at probability = .64, the specificity is 54.3% and sensitivity is 88.8% for both models; in practice, however, we encourage clinicians to use the actual predicted probabilities rather than the binary prediction necessitated by arbitrary decision thresholds. AUC, area under the curve; DNN, deep neural network; LR, logistic regression.

FIGURE 2.

Calibration curves derived after isotonic calibration in nested cross-validation. Orange diagonal line at y = x indicates perfect calibration. Each of the 10 blue dots represents a bin with an equal number of predictions. Blue shading indicates 95% confidence interval. Regions where the blue line/dots are below vs above the orange line represent probability ranges where the model outputs probabilities that are too high or too low, respectively. Both the A, 5-variable recursive feature elimination logistic regression and B, deep neural network algorithms demonstrate effective calibration using this isotonic calibration technique, as visualized with a blue line closely overlying the orange line.

Effect of Number of Predictors in LR and Online Access to Model

Figure 3 demonstrates the effect of the number of predictors on the performance of the LR model, as seen through the RFE process. The optimal number of predictors was 8, although there is a notable plateau after using just 2 predictors. To create a model that would be simple to deploy while still giving a reasonably varied range of predictions (thus resulting in improved calibration vs the simplest possible models), we chose to optimize the RFE process to end with 5 predictors. The difference in AUROC (non-nested) between the 5-variable and 8-variable versions was <0.01 (Figure 3), justifying this decision to proceed with the more sparse model. Of these 5 key predictors, 3 were associated with lower probability of SCS success (higher pre-SCS dose group [P < .001], long-term use of opioids [P < .001], and concurrent use of benzodiazepines [P = .003]) and 2 were associated with higher probability of success (polypharmacy [P = .007] and history of tumor without metastasis [P = .120]). Pre-SCS dose group and long-term use of opioids exerted the highest influence on predictions (Figure 4). The simplified, calibrated 5-variable LR model can be accessed online at SurgicalML.com. By providing the 5 patient characteristics, a user can easily see the predicted chance of SCS success.

FIGURE 3.

AUROC visualized throughout the process of RFECV. Note the early plateau, illustrating that a model incorporating just 5 variables performs similarly to the 30-variable version. AUROC, area under the receiver operating characteristic curve; RFECV, recursive feature elimination by cross-validation.

FIGURE 4.

Relative predictor importance in the logistic regression model, as derived from cross-validation with the 5-variable version. Bars extending to the left (negative influence) indicate a reduced chance of success after spinal cord stimulation, whereas bars extending to the right indicate a higher probability of success. In the deployable model, patient predictions are made as the average of 5 different models (as developed during cross-validation), each of which may have slightly different coefficients for these 5 variables, but these relative influences remain similar. All of these influences were statistically significant at P < .05, except tumor without metastases. In the deployable model, we include all 5 variables because of the improved discrimination and calibration when doing so. RFE, recursive feature elimination.

DISCUSSION

Key Results

We present the first ML models to predict SCS success, with DNN and sparse LR models performing similarly well. This work and future extensions may enable more informed surgical decision making and patient prognostication.

Interpretation

Effect of Number of Predictors on Linear Model Performance

Importantly, RFE with LR demonstrated that using 5 predictors enabled performance comparable with 30 predictors—a drastic reduction mirrored in other clinical studies.^25,26 Here, we demonstrate that the key predictive variables relate more to medication patterns than comorbidities. One likely explanation is the similar comorbidity profiles of the successful and unsuccessful SCS groups. This speaks to the less invasive nature of SCS; for other aggressive surgeries, comorbidity profiles may be more important. Here, it is understandably difficult for any model to meaningfully improve its performance by incorporating data on these comorbidities because there is minimal signal to ascertain.

Notable Predictors in Simplified LR Model

In the 5-variable LR, the most influential predictors of reduced probability of success were higher pre-SCS opioid dose group and long-term opioid usage. These intuitive results align with our previous study, in which these 2 variables were most strongly associated with not stopping opioids.⁶ Although a less influential predictor, concurrent use of benzodiazepines was another logical pharmacologic parameter exerting negative influence on odds of success. We cannot make causal claims about these variables, but these results suggest that exploring preoperative weaning of opioids and benzodiazepines may increase probability of SCS success.

Less intuitively, we found that polypharmacy was associated with a small increase in SCS success probability. We hypothesize that polypharmacy patients may be sicker (ie, have more medical comorbidities and also more treatment-resistant pain) at baseline and accordingly have closer follow-up after surgery, thus leading to improved medication optimization and reduced opioid usage.

The only comorbidity found to be a useful predictor in the final model was solid tumor without metastasis. Although we cannot claim a causal relationship, future studies regarding SCS relief of oncologic pain may be worthwhile. Although a Cochrane review found no evidence to support or refute the use of SCS in patients with cancer-related pain, it did identify several trials in which patients with cancer reduced their analgesic use after SCS implantation.²⁷ As our understanding of the mechanism of action of SCS improves, future studies could elucidate the biological underpinnings of these associations.

Generalizability

Deep Learning vs LR

We present the first DNN to predict SCS success. Deep learning is attractive for this application because pain is multidimensional with a myriad of potential contributing factors. Unlike LR, DNNs have the ability to capture complex, nonlinear interactions. A 2020 review of ML applications in neurosurgery found that neural networks were superior to other algorithms in 86% of comparative studies.¹⁴ Thus, the application of deep learning to the widely prevalent condition of chronic pain represents a potential advance.

Nonetheless, with this particular data set, deep learning did not provide a significant advantage over LR. There are multiple possible reasons for this. First, DNNs require larger training data sets vs LR to reach their full predictive potential.¹⁷ Moreover, the DNN's performance may have been limited by the aforementioned issue that plagued the LR model—a lack of sufficient comorbidity variation for the algorithm to learn. In addition, although we did try 48 hyperparameter combinations, future efforts may expand this to potentially improve performance. Finally, it is possible that the important signal-carrying variables—here, patterns of medication use—do actually have linear correlations with the probability of SCS success, which can be equally well approximated with LR.

The equivalence of LR and neural network models has also been demonstrated in numerous ML applications, ranging from predicting outcomes after traumatic brain injury^15,28 to predicting the risk of breast cancer.²⁹ Therefore, it is essential to temper the promises of DNNs with the knowledge that, in some cases, the ease of developing, interpreting, and implementing LR models (without the “black box” of neural networks) could make them the superior choice. Each prediction task warrants this exploration on a case-by-case basis.

Improvements on Previous SCS Success Modeling Efforts

Previous efforts, including our own,⁶ identified variables correlated with SCS success without using them as part of true ML methodologies. The isolated factors are numerous, and there is little overlap in results between studies. Several potential explanations exist for this lack of consistency: (1) improvements in SCS technology over time³⁰; (2) variation in the definition and measurement of “success”³⁰; (3) variation in the type of SCS leads placed³¹; (4) small sample sizes or studies limited to single institutions^11,13; and (5) inherent difficulty in performing controlled trials because of lack of adequate controls.³² Moreover, by using ML to present calibrated models, we can calculate and discuss absolute probabilities of success (from both LR and DNN models) rather than just speaking of relative odds ratios from traditional statistical presentations of LR models.

Compared with our own earlier study,⁶ the current analysis (1) uses ML methodologies (eg, splitting data into training/testing/validation sets, focusing on performance assessment through AUROC, assessing calibration, etc.) and (2) presents a model that offers numerical predictions regarding the more realistic outcome of opioid dose stabilization/reduction (rather than odds ratios for complete weaning off opioids). Nonetheless, it is reassuring that the 2 most influential variables from our first study—pre-SCS opioid dosage group and long-term opioid usage—were again the 2 most important predictors in this study.

In one of the other few studies actually reporting a usable model for predicting SCS success, Burchiel et al³² used regression techniques based on a cohort of 40 patients from one institution. They present a model incorporating age, a depression inventory, and the evaluative subscale of the McGill Pain Questionnaire, ultimately rendering 88% accuracy in predicting success (>50% change in visual analog scale) at 3 months after SCS. We build on this by using more robust ML methodology, a larger and geographically comprehensive data set, and longer-term outcomes. However, this comparison does highlight the potential for using more granular clinical variables in future efforts.

Beyond SCS, there have been few studies using ML to predict changes in opioid usage patterns. One group used ML to identify factors associated with prolonged opioid use after anterior cervical discectomy and fusion³³ and, similar to us, found that duration of presurgical opioid use and use of other pain medications were associated with increased chances of long-term postoperative opioid use. These studies set the stage for continued learning on better algorithms and treatments to tackle the opioid crisis.

Limitations

This work has limitations inherent to all retrospective studies using large, insurance claims-based data sets, such as the lack of granular clinical detail and potential claims-based coding inaccuracies. Our cohort also represents heterogeneity in (1) the surgical technique (eg, including both percutaneous and paddle electrode placements) and (2) patient selection criteria used by different physicians (hence the need for our research), thus potentially leading to selection bias. Moreover, we define SCS “success” as reduction or stabilization in opioid use, which may not be a perfect proxy for pain relief; for some patients, outcomes based on direct pain or quality-of-life measurements (eg, using visual analog scale or the Patient-Reported Outcomes Measurement Information System [PROMIS] scale) may be more important. Finally, opioid prescribing practices may depend on physician type (eg, surgeon vs pain specialist vs primary care), which is not captured here.

CONCLUSION

We present the first ML-based models for predicting the reduction or stabilization of opioid usage after SCS. Both the LR and DNN models demonstrated comparable performance. The 5-variable and 30-variable versions of the LR model also performed similarly, opening an opportunity for deployment of a simple but effective model for predicting opioid weaning with SCS, based on presurgical measures and pharmacologic patterns. We hope these findings will help augment patient and clinical decision making regarding SCS. Future efforts should include incorporation of more granular clinical data, refinement of model architectures and hyperparameters, new algorithms, external validations, and prediction of additional outcomes.

Funding

This study did not receive any funding or financial support.

Disclosures

Dr Lad is a consultant for Abbott Laboratories, Boston Scientific, Higgs Boson Health, Medtronic, Minnetronix, Nevro, and Presidio Medical. Dr Adil and Mr Charalambous are consultants for Higgs Boson Health. Mr Dharmapurikar is employed by Higgs Boson Health. Ms Parente is a consultant for Nevro and Saluda.

Supplementary Methods. Subheadings in this section include the following: Software, Data Pre-Processing, Machine Learning Algorithms and Hyperparameter Tuning, Internal Validation Strategy and Model Performance Assessment, Calibration, and Deployable Model.

COMMENT

We commend the authors for their work introducing a machine learning (ML) model that can predict opioid use after spinal cord stimulator placement. The use of ML has exploded in medicine, with many using “big data” as an additional tool when making clinical decisions and predictions. However, machine learning is more than just “big data,” various algorithms use inputs and outputs to determine associations and draw conclusions. The work describes two ML algorithms, deep neural networks (DNNs) and logistic regression (LR) models, and provides a clear understanding of its application to the clinical question at hand. Both algorithms notably show similar efficacy in predicting opioid use in patients who undergo spinal cord stimulator placement. Predicting the success of surgical intervention has serious implications on how we utilize our healthcare resources and provide better value to patients. As the authors mention, the upfront costs of spinal cord stimulation can be high and thus its suitability relies on careful patient selection and counseling. Therefore, clinical judgment remains paramount when selecting patients for spinal cord stimulation. The authors show that ML models can provide additional information that helps inform our decision-making and counsel patients on expectations. To that end, the authors created a publicly accessible website that allows visitors to calculate predictions themselves and translate the insights from the scholarly work here into everyday practice. There are, of course, many steps left before we can claim that ML is positively influencing neurosurgical practice, but this work is an essential first step.

Nelson Sofoluke

Clemens M. Schirmer

Wilkes-Barre, Pennsylvania, USA