CORR Insights®: Does the Presence of Missing Data Affect the Performance of the SORG Machine-learning Algorithm for Patients With Spinal Metastasis? Development of an Internet Application Algorithm

Where Are We Now?

Algorithms based in machine learning that are designed to anticipate prognosis and determine treatment plans are seeing wider use. In the field of spine surgery, these have been used to predict patient disposition after surgery, the risk of postoperative narcotic use, patient outcomes after surgery, and the prognosis for patients with epidural abscesses. The SORG Orthopaedic Research Group at Harvard has developed a number of these protocols and has made them available online (https://sorg.mgh.harvard.edu/predictive-algorithms/). This permits physicians to integrate these algorithms into their clinical decision-making process.

It can be difficult to know whether to recommend surgery for patients with spinal metastases. The SORG machine-learning algorithm (SORG-MLA) for metastatic disease is helpful because it can help determine which patients are likely to survive for 30 days, 90 days, or 1 year after surgery; that information factors heavily into the decision of whether or not to operate. This algorithm has been extensively tested and validated [5, 6, 10, 11]. However, in real-world situations, all the data necessary for the algorithm to work properly may not be readily available. This can be especially concerning for patients in whom the metastatic lesions are causing cauda equina syndrome or spinal cord compression, and for whom decisions must be made quickly to preserve neurological function.

The study by Huang et al. [4] in this month’s Clinical Orthopaedics and Related Research® evaluated whether the SORG-MLA remained accurate in the face of missing data. The authors used prior studies [5, 6, 10, 11] to ascertain which data points often are missing from patients’ charts in this setting, and then replaced it with artificially generated data using a missForest technique. The authors noted the albumin level and lymphocyte count were necessary to accurately predict patient survival as without them, the SORG-MLA algorithm did not provide an accurate survival prediction. Importantly, they demonstrated that the SORG-MLA still functions well when one to three data points are missing. This suggests that the SORG-MLA algorithm can still be a useful tool even when complete data are not readily available. As mentioned, the authors then used this set of findings to create their own internet-based application that can predict survival in patients with spinal metastatic disease even when some data are missing. This allows spinal surgeons and oncologists in other areas to access this algorithm and integrate it into their clinical decision-making, especially when making time-critical surgical decisions.

Where Do We Need To Go?

Predictive algorithms are only as good as the data used to create them. A review of 152 machine-learning algorithms found that 87% were at high risk of bias [7]. Factors contributing to bias included inadequate data, mishandling of missing data, and overfitting of the model. In the current study, data were obtained from patients at the National Taiwan University Hospital [4]. The data from many of the SORG-MLA studies come primarily from large, retrospective datasets at large academic centers. These data may not accurately represent patients from other geographic areas and patients treated at smaller, nonacademic institutions. More external validation studies are needed.

Retrospective data were used both in the current study [4] and in the original development of the SORG-MLA [7, 10]. These data may not represent changes in outcomes that arise from ever-evolving medical and oncological treatments. Recent studies have found that the SORG-MLA still performed well in a contemporary group of patients [2]. However, ongoing and repeated analyses will be necessary to ensure that this continues to be the case. In addition, prospective studies with a current set of patients can further validate the models.

The impact of missing data is not unique to the SORG-MLA for metastatic disease. The algorithm for patient outcomes after lumbar decompression surgery requires the imputation of PROMIS scores, which may not be collected by physicians in private or nonacademic practices. The effect of missing data on the performance of machine-learning algorithms is an important issue, and one that has not been extensively examined. The current study found that the SORG-MLA still performs well with missing data, and it identified which data points are necessary to have an accurate prediction. Other machine-learning algorithms must be similarly tested, and the algorithms adjusted if necessary to allow for calculations in the face of missing data and made available on the internet. This will facilitate meaningful, real-world use of these tools.

How Do We Get There?

As machine-learning predictive models continue to be developed, their impact on clinical practice will also grow. It is important for the clinicians who use these tools to understand how these algorithms are developed, the limitations of the data used in their creation, and how they can be best employed to improve patient care. Several papers have already been published in orthopaedic and spine journals to help educate clinicians on these important topics [3, 8, 9]. Continued education is necessary, including further journal publications, information and discussion sessions at national meetings, and the development of webinars or other educational materials from medical societies.

Machine-learning algorithms are most accurate when they have large pools of data from a wide range of patients. Large national patient registries from hip and knee arthroplasty societies are already being used to machine-learning algorithms [1]. I hope that the new American Spine Registry (https://www.americanspineregistry.org/) will provide additional data on a wide range of spinal conditions. It is important for this registry to include both academic and nonacademic locations around the country. This will allow for the widest array of data to be collected so that the models can be as inclusive and accurate as possible.

Finally, as machine learning expands, it is important to remember that in real-world practice, missing data are the norm, not the exception. That being so, developing machine-learning models that need the fewest possible data points to be accurate will be essential. The current study [4] demonstrated that doing so is both possible and helpful. Currently, many algorithms (including the SORG-MLA) are able to predict binary outcomes, such as mortality rate or risk of complications. As other algorithms are developed, they can be expanded to include factors that may be more concerning to patients, such as improved pain and quality of life. With this information, machine-learning algorithms will help us better anticipate prognostic endpoints that matter to our patients, and guide us toward better individual patient decisions in an era of personalized medicine.