Narrative Review of Predictive Analytics of Patient-Reported Outcomes in Adult Spinal Deformity Surgery

Abstract

Study Design:

Narrative review

Objective:

Decision making in surgery for adult spinal deformity (ASD) is complex due to the multifactorial etiology, numerous surgical options, and influence of multiple medical and psychosocial factors on patient outcomes. Predictive analytics provide computational tools to analyze large data sets and generate hypotheses regarding new data. In this review, we examine the use of predictive analytics to predict patient-reported outcomes (PROs) in ASD surgery.

Methods:

A search of PubMed, Web of Science, and Embase databases was performed to identify all potentially relevant studies up to February 1, 2020. Studies were included based on the use of predictive analytics to predict PROs in ASD.

Results:

Of 57 studies identified and reviewed, 7 studies were included. Multiple algorithms including supervised and unsupervised methods were used. Significant heterogeneity was observed with choice of PROs modeled including ODI, SRS22, and SF36, assessment of model accuracy, and with the model accuracy and area under the receiver operating curve values (ranging from 30% to 86% and 0.57 to 0.96, respectively). Models were built with data sets of patients ranging from 89 to 570 patients with a range of 22 to 267 variables.

Conclusions:

Predictive analytics makes accurate predictions regarding PROs regarding pain, disability, and work and social function; PROs regarding satisfaction, self-image, and psychologic aspects of ASD were predicted with the lowest accuracy. Our review demonstrates a relative paucity of studies on ASD with limited databases. Future studies should include larger and more diverse databases and provide external validation of preexisting models.

Introduction

Adult spinal deformity (ASD) negatively affects patient quality of life to the same extent seen in other chronic diseases such as chronic heart and lung disease and severe osteoarthritis and rheumatoid arthritis.^1,2 ASD affects 32% of adults, and more than 60% of elderly adults in the United States suffer from spinal deformity.^3,4 As the population ages and life expectancies rise, ASD and related surgery is becoming increasingly common.^5,6 The implementation of value-focused programs that can identify cost variability has been shown to decrease spending and improve health outcomes.⁷ As such, tools that can predict patient-reported outcomes (PROs) in ASD have the potential to maximize value by identifying the appropriate surgery for each patient.

In spine surgery, predictive analytics has the potential to help identify both those patients most likely to benefit from surgery and those at highest risk for complications. Predictive analytics has previously been used to predict patient satisfaction after decompression for lumbar stenosis, functional outcomes after surgery for cervical spondylotic myelopathy and recurrent lumbar disc herniation, and poor outcomes after lumbar discectomy.^8-11 Within ASD surgery, predictive analytics has also been used to predict the need for blood transfusion, hospital length of stay, complications, pseudoarthrosis, and catastrophic costs,^12-16 as well as postoperative PROs.^17-23 Given the numerous variables that contribute to outcomes and patient satisfaction with ASD, predictive analytics provides a valuable tool to analyze large data sets with unclear links of variables to outcomes. As a result, there has been substantial heterogeneity in studies on this topic, which suggests the importance of a review summarizing the existing evidence. The aim of this study is to review the current literature on predictive analytics of PROs following ASD surgery.

Methods

A literature review was conducted according to Preferred Reporting Items for Systematic reviews and Meta-analysis (PRISMA) guidelines to identify all articles available in PubMed, Web of Science, and EMBASE databases as of February 2020 without limitation on starting date. Search syntax was built from terms related to “adult spinal deformity,” “predictive analytics,” and “patient reported outcomes.” Exact search terms (MESH, EMBASE, and Web of Science searches) are shown in Supplemental Figure 1.

Included studies used prediction models of standardized PROs following ASD surgery for both cervical and thoracolumbar deformities. Studies examining thoracolumbar deformity included patients with (1) sagittal vertical axis (SVA) ≥5 cm, (2) pelvic tilt (PT) ≥25°, or (3) thoracic kyphosis (TK) ≥60°.

Studies of adult cervical spine deformity included patients with radiographic evidence of (1) cervical kyphosis (C2-7 Cobb angle >10°), (2) cervical scoliosis (C2-7 coronal Cobb angle >10°), (3) C2-7 sagittal vertical axis (cSVA) >40 mm, or (4) chin-brow vertical angle (CBVA) >25°.

Studies were excluded if they did not include full text, were in languages other than English, or were animal studies.

Results

The PRISMA flow sheet for data collection is shown in Figure 1. A total of 109 articles were identified for review resulting in a total of 57 articles following duplicate removal, and 42 articles were excluded based on review of the title and abstract. The full text of 15 articles was then assessed for eligibility, resulting in the exclusion of 8 studies which either lacked full text, did not predict PROs, or had no PRO data. Final analysis included 7 retrospective cohort studies specifically utilizing predictive analytics for modeling of PROs in ASD surgery.

Figure 1.

Flow diagram demonstrating PRISMA identification, screening, eligibility, and inclusion. Eight studies were included in the final analysis.

Articles reviewed are summarized in Table 1. The median number of patients included in each study was 234 with a range of 89 to 570. There was a wide range of variables included with a median number of predictors of 46 (range 11-267). The most common PRO tools utilized were the Oswestry Disability Index (ODI), the Scoliosis Research Society Outcomes Questionnaire (SRS22), and the Short Form 36 (SF36). Numerous methods were used to develop predictive models including decision trees, conditional inference trees, gradient boosting machines, extreme gradient boosting tree, extreme gradient boosting linear models, random forest, generalized linear models, elastic net, elastic net regularized models, and unsupervised hierarchical clustering. The heterogeneity of the models and predictions prevented generalizations regarding conclusions. Various measurements were also utilized to assess computational modelling of the data including accuracy, area under the curve (AUC), and mean average error.

Table 1.

Predictive Analytics to Examine Patient-Reported Outcomes in Adult Spine Deformity Surgery^a.

Author, year	Patient-reported outcomes	Time period	Patients	Variables	Methods	Accuracy/AUC/outcome measure	Conclusion
Ames, 201921	SRS22r SF36 ODI	Pre-/postoperative to 1 or 2 years	561	150	Elastic net Gradient boosting machine Extreme gradient boosting tree Extreme gradient boosting linear models Random forest Elastic net regularized generalized linear models	35% to 80% 0.57-0.87 AUC	SRS-22r questions were accurately predicted; highest accuracy for questions regarding pain, disability, social/labor function
Ames, 201922	SRS22r SF36 ODI	Pre/postoperative to 1 or 2 years	570	75	Ordinary least squares Ordinary least squares with partitions Elastic net Gradient boosting machines Extreme gradient boosting tree Extreme gradient boosting linear models Random forest Generalized linear models	MAE 8% to 15%	Greatest improvement seen in PRO for patients with lowest baseline PRO
Ames, 201923	SRS22r SF36 ODI	2 years	570	22	Unsupervised hierarchical clustering	Not explicitly reported for PRO	3 patient clusters: young coronal, old primary surgery, old revision surgery; 4 surgery types: 3 column osteotomy, no osteotomy/IBF, IBF, SPO; greater improvement in PRO for old revision patients
Bortz, 201920	HQRL EQ5D NDI	3 months 1 year	138	267	Conditional inference trees	N/A	Patients undergoing nonroutine discharge after cervical deformity surgery had inferior EQ-5D outcomes, no difference in HRQL, NDI
Horn, 201919	NDI	1 year	89	11	Univariate and multivariate regression	0.86 AUC	In cervical deformity patients, poor outcome predicted by osteoporosis, worse baseline status, baseline PT >20°, >9 levels thoracic kyphosis, elevated C2-T3 SVA or global SVA
Scheer, 201818	ODI	1 year	198	43	Ensemble 5-bootstrapped decision-trees	86%, 0.94 AUC	Predictors of ODI MCID included gender, SRS score, back pain, SVA, PI-LL, primary vs revision
Oh, 201717	SRS22r SF36 ODI	2 years	234	46	Ensemble 5-bootstrapped decision-trees	86%, 0.96 AUC	10 patients misclassified as meeting MCID with model; patients meeting MCID had higher mean 2-year QALYs

Abbreviations: AUC, area under receiver operating curve; MAE, mean average error; PRO, patient-reported outcomes; IBF, interbody fusion; SPO, Smith-Perterson osteotomy; PT, pelvic tilt; SVA, sagittal vertical alignment; PI-LL, pelvic incidence to lumbar lordosis; MCID, mean clinically important difference; QALY, quality-adjusted life years.

^a Articles included in the review of the literature. Seven articles were identified for inclusion—5 for thoracolumbar ASD and 2 for cervical ASD. For thoracolumbar deformity, the most common PROs were the Scoliosis Research Society Health-Related Quality of Life Questionnaire (SRS-22r), Short Form Health Survey (SF-36), and Oswestry Disability Index (ODI). The most common PROs for cervical ASD included the Neck Disability Index (NDI), EQ5D, and Health-Related Quality of Life Questionnaire (HQRL). Follow-up ranged from 3 months to 2 years. Three articles by Ames et al had very similar patient numbers indicating that likely the same patients were used in each review.

Thoracolumbar Deformity

Oh et al¹⁷ conducted a multi-institutional retrospective review of prospectively collected database of 234 patients collected by the International Spine Study Group (ISSG) undergoing surgery for thoracolumbar ASD using boot-strapped decision trees to identify patients that would meet the minimum clinically important difference (MCID) in postoperative ODI. Their model included 46 variables and was reported to be 86% accurate with a 0.96 area under receiver operating curve (AUC) with internal validation using a 70:30 data split. Patients predicted to meet the ODI MCID had significantly greater quality adjusted life years (QALYs) gained at 2-year follow-up.

Similarly, Scheer et al¹⁸ utilized the same ISSG database to retrospectively review 198 patients with baseline ODI >30 and develop a model predicting patients who would meet the ODI MCID at 1 year. Their model incorporated 43 variables and was internally validated with a 70:30 data split to predict which patients would achieve the ODI MCID with 86% accuracy and 0.94 AUC. Major predictors of a positive outcome in this study included gender, lower preoperative SRS22 score and back pain rating, SVA, pelvic incidence to lumbar lordosis mismatch (PI-LL), and whether the patient was undergoing a primary surgery or a revision.

Ames et al²¹ developed models using a cohort of 561 from 2 independent ASD databases collected from 17 sites throughout the United States and Europe to predict patients’ responses to all 22 questions of the SRS-22r at 1 and 2 years follow-up. The models were developed using 6 different prediction algorithms (Table 1) and 150 variables. Internal validation was completed with an 80:20 data split, and the model with the maximum AUC was selected. Accuracy ranged from 35% to 80% for each question with an AUC of 0.57 to 0.87. The models most accurately predicted outcomes for questions assessing pain, disability, and social/labor functioning. They were least accurate for questions addressing satisfaction, depression/anxiety, and appearance. Low predictability for patient satisfaction was attributed to a low incidence of unsatisfied patients in the study.

A subsequent study²² utilizing the same database examined 570 patients and 75 variables to develop models capable of predicting a patient’s odds of achieving the MCID on the ODI, SF-36, and SRS-22r at 1 and 2 years follow-up using 8 modeling algorithms (Table 1). The goodness of fit for each model was assessed using mean average error (MAE) unlike the above-mentioned models, which utilized an accuracy and AUC measurement. The models underwent internal validation using an 80:20 data split with MAE ranged from 8% to 15%. Predictions generated by the models suggested that patients with low baselines PROs would experience the greatest gain in PROs; however, these same patients also had the highest rate of complications, suggesting them to be a high risk/reward population.

Ames et al²³ applied unsupervised hierarchical clustering to a combined ISSG/ESSG (European Spine Study Group) population to discover distinct patient clusters and groups of surgeries in an effort to predict surgical quality and PROs. Using 22 variables gathered from the 570 ISSG/ESSG data set, the authors identified 3 clusters undergoing surgery for ASD: young patients with a coronal deformity, old patients undergoing primary surgeries, and old patients undergoing a revision surgery. These clusters underwent 1 of 4 distinct surgery types: surgery with a 3-column osteotomy, surgery with an interbody fusion, surgery with a Smith-Peterson osteotomy, and surgery without osteotomies or interbody fusions. These data were used to predict the risk of surgical complications as well as ODI, SRS-22r, and SF-36 outcomes. The authors noted that the model generated from this analysis could function as a tool to accurately assess risks and benefits of interventions in patients with low baseline functional status. Similar to studies cited above, patients undergoing revision surgeries had the greatest potential for improvement in PROs but also the highest likelihood of complications.

Cervical Deformity

Two studies applied predictive analytics to surgery for cervical deformity. Predictive analytics was first successfully applied to cervical deformity patients by Horn et al using univariate and multivariate regression models to predict poor outcomes as measured by the Neck Disability Index (NDI). Their model included 89 patients and 11 variables and was found to be 86% accurate. Predictors of poor NDI scores included the presence of osteoporosis, a worse baseline functional status, baseline pelvic tilt (PT) >20°, greater than 9 levels of thoracic kyphosis, and an elevated C2-T3 SVA or global SVA >4cm.¹⁹ Bortz et al used conditional inference tree modeling to predict nonroutine discharge following cervical deformity surgery, defined as discharge to a rehabilitation centers rather than home. Their model found that patients who underwent nonroutine discharge had poorer overall health, as measured by the EuroQol-five dimensions (EQ-5D). However, there was no difference in health-related quality of life or NDI outcomes.²⁰

Discussion

The first studies published utilizing predictive analytics for ASD^17,18 employed supervised machine learning and bootstrapped decision trees. These models make predictions based on random inputs with regard to a target variable. The model then builds a branching, tree-like structure testing how well each variable predicts the outcome based on the fit of the variable with the outcome at each branching point. This continues iteratively until the final outcome is reached. Models are typically developed by splitting the data into a training and validation set, whereby 70% to 80% of the available data is used to create the model which is then validated using the remaining data. With time, the methods used to build models have become more complex. Newer models have been built using unsupervised learning and other more complex algorithms.²³

As all of the studies examined questionnaires regarding PROs for ASD (ie, SRS-22r, ODI, and SF-36), some qualitative observations can be made regarding the nature of the data presented. For example, the models were able to predict responses regarding pain, disability, and the patient’s work and social function with the highest accuracy. These questions have well-defined, objective metrics that are easily identified by the patient. These domains are also frequently the reasons motivating patients to seek treatment for spinal deformity. Responses to questions on disability have less variability both pre- and postoperatively and demonstrate lower day-to-day variability.

In contrast, one study demonstrated poorer predictions with responses to questions assessing a patient’s mental status (depression and/or anxiety), self-image, and overall satisfaction with ROCs (eg, “How do you look in clothes” and “In the last 6 months, have you felt so down in the dumps that nothing could cheer you up?” and “Are you satisfied with the results of your back management?” had the lowest AUCs of approximately 0.6).²¹ The cause of this is likely multifactorial. While these questions provide some representation of a patient’s day-to-day self-image and psychiatric state, the answers to the questions are prone to extreme variability throughout the course of a single day—who has not felt “down in the dumps” or not liked how they look in clothes? More data regarding a patient’s preoperative psychiatric state would be useful to develop accurate predictions regarding these questions. These data are likely not collected in a routine spine surgeon appointment and are likely not readily available through chart review—in fact, psychiatric records are frequently sealed from the rest of the patient’s electronic health records and special permissions are needed to access this information. Alternatively, this may indicate that these aspects of a patient’s life are not consistently addressed by ASD surgery. This is important information to keep in mind when counseling patients.

While the predictive models studied here accurately describe the data set from which they are derived, they are susceptible to bias. When a limited data set is used to train an algorithm with a high percentage of the data used for training, it is prone to bias and overfitting.²⁴ Many of the studies described here benefit from the use of a prospectively collected multicenter database, which should minimize this drawback. The use of prospectively collected data from diverse institutions is key to developing accurate, generalizable predictive models. Newer studies which eschewed the conventional decision tree model in favor of random forests and conditional inference trees are thought to be less susceptible to model overfitting.²⁴ Now, multiple algorithms are being tested on a given set to find the algorithm with the highest accuracy and greatest predictive power.

Unsupervised learning methods are also being explored to create predictive models for patient outcomes. The benefit of these models is that clustering and learning is done without active input from the researcher, reducing the potential for observer bias. This allows the algorithm to determine those variables most strongly related to the outcome(s) of interest and to detect previously unrecognized patterns in data.²⁵ The relationships found in data through unsupervised learning, however, are often not intuitive and require further investigation to determine their validity.

Predictive analytics have provided models that use preoperative inputs to accurately predict PROs 1 and 2 years postoperatively. The large number of preoperative variables that are associated with PROs (as many as 267 in one study)²⁰ suggests the importance of using these advanced statistical tools and large datasets to develop valid models.

In general, this review of the literature reveals a major paucity in research regarding the use of predictive analytics for predicting PROs in ASD surgery. Our search revealed only 7 articles written by 3 authors with similar cohort numbers in 3 studies indicating that the same or very similar cohorts may have been used in these studies. Studies incorporating alternate databases with more diverse patients are obviously necessary to create broadly generalizable conclusions. Additionally, all of the models examined were internally validated. External validation using different patient cohorts provides evidence that a model can be applied broadly across patients and suggests a stronger model. External validation of models further combats the problem of overfitting which can occur with models trained on large, complex datasets. Complex models which perform well on the training data set but poorly on new datasets as extra “noise” having nothing to do with the outcome of interest was included in the model.²⁶

More studies are needed regarding PROs for cervical deformity surgery to draw meaningful conclusions. Future studies would likely benefit from large patient databases such as those used by the ISSG and ESSG for thoracolumbar deformity.

Conclusion

Predictive analytics has been shown to provide accurate tools for predicting PROs in ASD surgery. This can improve ASD outcomes by preoperatively identifying patients most likely to have improvement in PROs and those at highest risk of complications, readmission, and reoperation. To date, studies completed have provided high accuracy regarding PROs for pain, disability, and work and social function, but not for satisfaction, self-image, and psychologic outcomes. These studies have been conducted on limited data sets, primarily obtained through the ISSG and ESSG. Furthermore, external validation of the models developed within these studies is necessary to demonstrate their validity and promote their use clinically. Initiatives to validate predictive tools and pool patient data will help develop the most accurate models and improve patient outcomes with ASD surgery.^27,28