Machine Learning for Suicide Research–Can It Improve Risk Factor Identification?

Machine learning is on the rise. According to Scopus (www2.scopus.com), the number of publications in medicine with machine learning in the title, abstract, or as a keyword during 2016 to 2018 increased from 1658 to 3904. In psychiatry, applications of machine learning are proposed to improve the accuracy of diagnosis and prognosis and determine treatment choice. At the same time, much of this research has given insufficient attention to high-quality methods, clinical applications, and ethical aspects. This is compounded by poor reporting of performative measures and misleading claims about the high accuracy of such approaches. In this issue of JAMA Psychiatry, the article by Gradus and colleagues¹ raises important questions about the place of machine learning in research and practice.

Machine learning is useful when analyzing several predictors, particularly if there are nonlinear observations and interaction terms that are difficult to theoretically conceptualize and practically model. A strength of the study by Gradus et al1 is that it uses a robust data set of patient-level predictors from Danish national health care registers without imposing or relying on theoretical models. The authors tested 1365 parameters based on 334 individual predictors, which were mostly associated with somatic and psychiatric diagnoses and by temporal proximity with the suicide event by being categorized into 4 points of 6, 12, 24, and 48 months. Another strength is the outcome of death by suicide. As the study illuminates risk factors rather than prediction, this is important, as risk factors for suicide are different from self-harm and suicidal ideation. For example, male sex is more highly associated with suicide, whereas female sex is a risk factor for suicide ideation and attempts.² The use of high-quality Danish register data means that suicide is ascertained with negligible missing data; however, the authors did not consider injuries of undetermined intent as an outcome, which could lead to possible misclassification. Based on 10 outcomes per predictor recommended by methodologists,³ the authors also had sufficient statistical power to investigate 14 463 suicides. A study of such large suicide numbers combined with hundreds of potentially modifiable predictors is an important advance for the field.

The most informative aspects of the study by Gradus et al¹ are in highlighting the importance of psychiatric disorders as risk factors for suicide. The analyses simultaneously accounted for somatic and psychiatric disorders and a few sociodemographic factors (eg, income, civil status, age bands, and being an immigrant). It reported that diagnoses, such as schizophrenia and adjustment disorders, and markers of psychiatric morbidity, such as antidepressants and antipsychotics, are strongly associated with suicide. This provides further evidence, in addition to findings from epidemiological and psychological autopsy studies, of the consistency and strength of modifiable risk factors. The advantage of machine learning in identifying predictors and their combinations opens up novel areas for future suicide research. These include the role of comorbidity and the heterogeneity within disorders. One clear implication of the article is that prediction models will likely improve their accuracy and usability if they are developed for specific diagnostic groups. Clinicians who treat people with schizophrenia, and the likely interventions they can offer to prevent suicide,⁴ will be different from those who assess and manage adjustment disorders, many of whom will only present to primary care and where psychological therapies will be more important. Another implication of the findings is that, despite many strong predictors in the final model, these predictors are not necessarily causal. Future research will need to investigate causal associations between predictors identified in such models and suicide. For example, the finding that diagnoses occurring in the 48 months before suicide were more important to prediction compared with 6 months does not negate the role of temporally close predictors; rather, it highlights that more research to examine timing and the chronicity of diagnoses associated with suicide is required.

The interpretation of these findings needs careful consideration to avoid the hype that frequently accompanies machine learning publications. Gradus et al¹ provide no comparison with simpler modeling, which will need to be done to contextualize the findings. The authors used classification trees and random forest models, both of which attempt to address limitations inherent to one another. Classification tree models, while more easily interpretable, are often unstable depending on the sample and variables available. Regression trees, while more stable, are often criticized for poor interpretability (“black box”). Despite these approaches, a recent review of 71 studies that compared machine learning prediction models with simpler regression ones found no improvement in the machine learning ones,⁵ suggesting that simpler models would be preferred by clinical services that would struggle to implement complex algorithms into computer systems and the clinical workflow. Notably, the sensitivity of the model in Gradus et al¹ is 32% based on a cutoff of the top 5% of men at risk, which means that for every 100 suicides, this model will detect 32 suicides. The high false-negative rate (68%) at the 5% cutoff will mean that this model will not be acceptable to clinicians without further improvements or if used without adjunctive clinical decision-making. Nevertheless, it is possible that unstructured clinical judgement may be similarly predictive of suicide and comparing these models with clinical decision-making will be informative.⁶ Even lowering the cutoff of the algorithm in Gradus et al¹ to 10% of men at risk, the sensitivity was 49%, meaning that more than half the suicides will be missed. For an outcome such as suicide, high false-positives may be tolerated if linked interventions are not harmful to the individual (although the wider resource implications need to be considered), but this will not be the case for high false-negatives. Additionally, calibration was not reported in Gradus et al,¹ which is an essential marker of performance as it reports the reliability of the probabilistic risk predictions⁷ rather than areas under the curve, sensitivities, and specificities, which aid in discriminating between risk groups (such as the top 5% vs the rest).

Nevertheless, the Gradus et al article stimulates research on how to improve these models. A prediction model can provide consistency, raise the ceiling of assessments, and enable the stratification of care transparently. Enhancing the model will come from optimizing the use of key predictors, such as age, which Gradus et al¹ used as a categorical variable (with 4 cutoffs at ages 30, 40, 50, and 60 years), whereas a continuous variable would likely provide better accuracy. Predictors based on other background factors, such as educational attainment, may further improve models in light of research demonstrating links between IQ and suicide.⁸ While population-based registers are advantageous for adequate statistical power, they lack finer-grained measurements of the psychological constructs associated with suicide (eg, hopelessness) and other time-sensitive environmental factors (eg, access to lethal means). Whether these add to incremental validity over simpler measures requires research.

As a necessary next step, Gradus and colleagues should make the algorithm available at no cost so that it can be independently validated, a key marker of validity for any prediction model. Future work needs to make available a prespecified protocol and clarify whether and how the final published study deviated from it. Reporting guidelines should be followed, such as the soon to be released TRIPOD-ML,⁹ including measures of discrimination and calibration, and journals could consider insisting on their completion on manuscript submission.

Going forward, the development of prediction models in specific, high-risk groups (eg, US Army soldiers after psychiatric treatment¹⁰) or those with certain psychiatric diagnoses (eg, schizophrenia spectrum disorders¹¹) will likely have benefits of higher accuracy, more acceptability to clinicians, and simpler linkage to interventions. How they can be most effectively linked to evidence-based treatments needs careful examination; without such links, prediction models are unlikely to reduce suicidality outcomes.