Predicting Suicide in the 30 Days Postdischarge from Psychiatric Hospitalization in Denmark Using Machine Learning

Abstract

Background:

Suicide risk is high in the 30 days postdischarge from psychiatric hospitalization but knowledge of the profiles of high-risk patients remains limited.

Aims:

To examine sex-specific risk profiles for suicide in the 30 days after psychiatric hospitalization using machine learning and Danish registry data.

Method:

We conducted a case-cohort study capturing all suicide cases occurring in the 30 days after psychiatric hospitalization discharge in Denmark from January 1, 1995 to December 31, 2015 (n=1,205). The comparison subcohort was a 5% random sample of all persons born or residing in Denmark on January 1, 1995 and who had a first psychiatric hospitalization between 1995 and 2015 (n=24,559). Predictors included diagnoses, surgeries, prescribed medications, and demographic information. The outcome was suicide death recorded in the Danish Cause of Death Registry.

Results:

For men, prescriptions for anxiolytics and drugs used in addictive disorders interacted with other characteristics in the risk profiles (e.g., alcohol related disorders, hypnotics and sedatives) that led to higher risk of postdischarge suicide. In women, there was interaction between recurrent major depression and other characteristics (e.g., poisoning, low income) that led to increased risk of suicide. Random forests identified important suicide predictors: alcohol related disorders and nicotine dependence in men and poisoning in women.

Conclusions:

Our findings suggest that accurate prediction of suicide during the high-risk period immediately following psychiatric hospitalization may require a complex evaluation of multiple factors for men and women.

Introduction

Suicide is a severe public health problem, with approximately 800,000 people dying by suicide each year globally.¹ The World Health Organization estimates that the global suicide rate is 10.6 per 100,000 person-years.² The immediate time after discharge from hospitalization for a mental health condition is a critical period during which suicide risk is high.^3,4 The rate of suicide in the first month after discharge from a psychiatric hospitalization is reported to be 2060 per 100,000 person-years.⁴ The suicide rate among persons in the first month after discharge from a psychiatric hospitalization is over 190 times the global suicide rate.^2,4 The immediate period after discharge from psychiatric hospitalization represents a unique opportunity for prevention to reduce suicide deaths among this vulnerable population.

Our understanding of the factors that predict heightened risk of suicide after psychiatric hospitalization remains limited. Previous studies documented increased postdischarge suicide risk among persons who have a history of self-harm, affective disorders, relationship problems, job loss, and are living alone.^5,6 Although many risk factors for suicide have been documented, a 2017 meta-analysis of the last 50 years of research on suicidal thoughts and behaviors found that we still cannot accurately predict who will die by suicide.⁷ Most studies examined a small number of risk factors, but accurate suicide prediction likely requires examination of hundreds of risk factors and their interactions. However, conventional parametric statistical techniques are not designed to examine large, highly correlated sets of predictors or to detect interactions among predictors without a priori specification. Machine learning methods can detect complex patterns and return useful algorithms for predicting suicide, thus enabling the development of novel suicide risk profiles that include constellations of predictors. Furthermore, the development of prediction models in high risk groups, such as persons who experienced psychiatric hospitalization, is likely to improve relevance and acceptability to clinicians who work with this population.¹¹

Kessler and colleagues developed algorithms to estimate suicide risk among United States (U.S.) Army soldiers post-hospitalization and in the U.S. veterans health administration system.^9,12 However, despite this contribution, several gaps remain. First, few studies have focused on predicting suicide in the relatively short 30-day window of interest to clinicians.⁷ One reason for this is the lack of an adequate sample size given that suicide is rare. Second, findings from U.S. Army members and Veterans may not be generalizable to the broader population of people who experience a psychiatric hospitalization. Third, little is known if risk profiles of suicide after psychiatric hospitalization differ among men and women. Previous research predicting post-hospitalization suicides using machine learning did not examine sex differences, which is likely due to the smaller proportion of women in the military than men. However, there may be different risk profiles of suicide in men and women since there are well-established sex differences¹³ in suicide risk and risk factor distributions.¹⁴

The purpose of this study was to identify novel interactions and variables that predict suicide in the 30 days after discharge from a psychiatric hospitalization in men and women. We leveraged Danish registry data captured over a twenty-year period encapsulating all diagnoses, surgeries, medication prescriptions, and demographic/social register information. We used machine learning methods including classification trees and random forests to achieve our dual goals of characterizing interactions between variables and elucidating novel predictors of suicide after psychiatric hospitalization.

Method

Study Sample.

The source population was all individuals born or residing in Denmark as of January 1, 1995. The start of the study period coincides with the switch from International Classification of Diseases, 8^th Revision to International Classification of Diseases, 10^th Revision (ICD-10) in 1994 and the start of reporting of all hospital outpatient clinic visits to the Danish National Patient Registry covering all Danish hospitals in 1995.¹⁵ We implemented a case-cohort design because it is an efficient approach for studying rare outcomes.¹⁶ We did not match cases and subcohort members to allow for maximum variability in the predictors in the analysis. Cases were all individuals who died by suicide and had been hospitalized for a psychiatric disorder within 30 days before their death between January 1, 1995 and December 31, 2015 in Denmark (n=1,205). The comparison subcohort was a 5% random sample of individuals in Denmark on January 1, 1995 and who had a first psychiatric hospitalization between January 1, 1995 and December 31, 2015 in Denmark (n=24,559). We included persons who experienced an inpatient hospitalization for the following disorders as recorded by two-digit ICD-10 codes from the Danish Psychiatric Central Research Register¹⁷ and Danish National Patient Registry¹⁵: mental disorders due to known physiological conditions (F01-F09), substance use disorders (F10-F19), schizophrenia (F20-F29), mood disorders (F30-F39), anxiety, dissociative, stress-related, and somatoform disorders (F40-F48), behavioral syndromes associated with physiological disturbances and physical factors (F50-F59), personality and behavior disorders (F60-F69), behavioral and emotional disorders with onset usually occurring in childhood and adolescence (F90-F98), and unspecified mental disorders (F99). We used central personal registry numbers that are unique individual-level identifiers assigned to all Danish residents to link data across Danish administrative and medical registries.¹⁸ We used the Danish Civil Registration System to randomly select comparison subcohort members.¹⁹

Outcome.

We obtained suicide cases using ICD-10 codes X60-X84 as recorded in the Danish Cause of Death Registry.²⁰ This registry records data on age of death, manner of death (e.g., natural, suicide), place of death, and autopsy results.²⁰ A validation study confirmed suicide as the cause of death for 92% of the deaths recorded as suicides.²¹

Predictors.

We examined the following variables as predictors in the machine learning models: age, marital status, immigration status, citizenship, family suicidal history (parent or spouse), employment, income, mental disorders, physical health disorders, surgeries, prescription drugs, and psychotherapy. We used the Danish Civil Registration System¹⁹ to obtain data on age, marital status, immigration status, generation of citizenship, and family suicidal history. We used the Integrated Database for Labor Market Research²² and Income Statistics Register to obtain baseline data on employment and income.²³ We ascertained psychiatric disorder diagnoses using two-digit ICD-10 codes from the Danish Psychiatric Central Research Register¹⁷ and Danish National Patient Registry.¹⁵ We also used the Danish National Patient Registry to obtain physical health diagnoses as recorded by second-level ICD-10 groupings. Surgery procedure codes from the Danish National Patient Registry were examined according to body system. We obtained data on prescription drugs from the Danish National Prescription Registry.²⁴ Prescription drugs for this study were coded according to level three Anatomical Therapeutic Classification codes. The Appendix contains all codes analyzed.

Statistical Analysis.

The time-varying predictors for both cases and the comparison subcohort members were defined 30 days after discharge from psychiatric hospitalization. Cases were persons who died by suicide in the 30 days after discharge. For cases, we dummy-coded variables to create time-varying predictors with intervals of 0–6, 0–12, 0–24, and 0–48 months before the date of suicide. To compute the prevalence of each predictor in the person-time that gave rise to cases for the comparison subcohort, we used the date 30 days after discharge to calculate the prevalence of predictors in the preceding 0–6, 0–12, 0–24, and 0–48 months prior to that date. For example, for a member of the comparison subcohort who was discharged from a psychiatric hospitalization on December 1, 2010, we calculated the prevalence of predictors 0–6, 0–12, 0–24, and 0–48 months before December 31, 2010 (i.e., December 31, 2010 is 30 days after the discharge date). Time intervals were chosen to be consistent with intervals used in previous research using machine learning for suicide prediction.^9,25–27 Age, immigration status, employment, and income at baseline were not coded as time-varying predictors. Predictors from all periods were evaluated simultaneously in the models.

We conducted data reduction to avoid overfitting. Overfitting arises when a model finds patterns that are unique to a specific dataset but are not generalizable to external samples.²⁸ We performed data reduction for men and women separately by: 1) removing rare predictors that had fewer than 10 observations in any cell of a 2×2 contingency table of the predictor and suicide⁹ and 2) removing predictors with negligible associations with suicide (unadjusted odds ratio between 0.9 and 1.1). We removed emergency department diagnoses due to their low positive predictive values.^29,30 The initial analytic dataset contained 2563 predictors. After data reduction, the final number of included predictors was 509 for men and 422 for women. The Appendix table provides the considered and retained predictors.

Given our interests in identify novel predictors and interactions that accurately predict suicide postdischarge from psychiatric hospitalization, we used recursive partitioning methods that automatize detection of interactions and provide metrics of predictor importance.³¹ First, we used classification trees, which are a nonparametric method that builds a decision tree based on predictors and their combinations that result in the highest probability of differentiating cases from non-cases. Classification trees can elucidate interactions among large sets of predictors without a priori specification and provide a visual depiction of risk factor constellations that predict suicide. However, classification trees are more vulnerable to overfitting than random forests.³² To decrease the risk of overfitting, we used 10-fold cross validation of classification trees. To increase visual interpretability, we set the maximum tree depth and minimum number of observations in any node to five. To address class imbalance, we used equal priors.³³ The risk of suicide was calculated for each identified combination of predictors. We used the R package rpart³⁴ to implement classification trees.

Second, we implemented random forests, which are a recursive partitioning method that comprises a set of decision trees generated using bootstrapped samples of the data. Each forest was built with 1000 trees and a minimum of 10 observations were needed to attempt a split. The number of variables sampled as split candidates at each node were 23 for men and 21 for women (i.e., square root of the total number of predictors for men and women; R package randomForest default). To address class imbalance, each individual tree was built using all suicide observations and an equally sized number of randomly selected non-suicide observations using the sampsize tuning parameter. We used 2-fold cross-validation to generate individual-level random forests predicted values. We calculated the mean decrease in accuracy of each variable, which represents the reduction in accuracy if a predictor were permuted.³¹ The larger the mean decrease in accuracy of a predictor, the more important it is for accurate prediction of suicide. We used the R package randomForest.³⁵ Although random forests provide metrics of predictor importance, they do not provide a visualization of interactions between variables as classification trees do. Thus, we leveraged the strengths of both classification trees and random forests to best serve our dual interests in identifying novel predictors and interactions that predict suicide after psychiatric hospitalization.

We evaluated prediction accuracy using receiver operating characteristics curve analysis conducted in 1000 bootstrap replicates to estimate the area under the curve (AUC) and its 95% confidence interval (CI). We also examined the sensitivity and specificity of the classification trees and random forests in detecting suicide. The analyses were conducted separately for men and women. We conducted sex-stratified analyses instead of including sex as a predictor in an analysis of the entire sample because the latter approach would not reveal sex differences in random forest variable importance, and in classification trees, would only display separate patterns of risk in men and women at the point that sex is chosen as a splitting variable but not earlier in the tree. Analyses were conducted in SAS, version 9.4 (SAS Institute Inc)³⁶ and R, version 3.5.2.³⁷ This study was determined to be exempt from review by the Boston University IRB approved by the Danish Data Protection Agency (record number 2015–57-0002).

Results

Table 1 shows the descriptive characteristics of the study sample. Among men, the mean ages were similar in the suicide cases and the comparison subcohort. Among women, suicide cases were on average younger than the comparison subcohort members (mean [SD]: 52 [16] vs 57 [24]). Across sexes, suicide cases and subcohort members had similar proportions of immigrant status and persons in a married or registered partnership. Suicide cases were less likely to be in the lowest income quartile compared with subcohort members.

Table 1.

Characteristics of the suicide cases and the subcohort, Denmark, January 1, 1995.

	Men		Women
Variable	Suicide cases (n = 762)	Comparison subcohort (n = 11966)	Suicide cases (n = 443)	Comparison subcohort (n = 12593)
Age, mean (SD), y	51 (16)	51 (21)	52 (16)	57 (24)
Marital status, N (%)
Married or registered partnership	249 (33%)	4024 (34%)	129 (29%)	3578 (28%)
Divorced/Widowed/Never married/single	513 (67%)	7901 (66%)	314 (71%)	8979 (71%)
Immigrant, N (%)	30 (4%)	498 (4%)	16 (4%)	485 (4%)
Income quartile, N (%)
<1	131 (17%)	3245 (27%)	95 (21%)	3223 (26%)
1 to <2	198 (26%)	2986 (25%)	88 (20%)	3193 (25%)
2 to <3	210 (28%)	2836 (24%)	111 (25%)	3095 (25%)
>3	223 (29%)	2899 (24%)	149 (34%)	3082 (25%)

Classification trees.

The highest risk of suicide was among men who were not prescribed antidepressants in the 48 months preceding psychiatric hospitalization, were diagnosed with alcohol related disorders in the preceding six months, were prescribed hypnotics and sedatives, had a poisoning diagnosis¹ in the preceding 48 months, and were prescribed anxiolytics in the preceding 12 months (n=20; suicide risk = 91%). However, men with the same risk profile (i.e., same combination of variables) who were not prescribed anxiolytics in the preceding 12 months (n=14) had a 0% risk of suicide. This result suggests that in this subgroup, there is an important interaction between this combination of variables and anxiolytics prescriptions. Another stark contrast in suicide risk is among men who were prescribed antidepressants in the prior 48 months and hypnotics and sedatives in the preceding 24 months, were diagnosed with cerebrovascular diseases in the preceding 12 months, and not diagnosed with poisoning in the preceding 48 months. Among men with this risk profile, those who were prescribed drugs used in addictive disorders in the preceding six months (n = 14) had a 72% risk of suicide. However, men with the same risk profile but who were not prescribed drugs used in addictive disorders in the preceding six months (n = 112) had a 0% risk of suicide. Figure 1 shows other combinations of predictors that conferred elevated risk of suicide in men (AUC = 0.80, 95% CI: 0.78, 0.81).

Figure 1. Classification tree depicting suicide predictors among male patients hospitalized for psychiatric disorders in Denmark, 1995 – 2015.

Note: ^aPoisoning by, adverse effect of and underdosing of drugs, medicaments and biological substances. Each shaded rectangular bin at the bottom (terminal node) represents the group of people with the characteristic profile in the branches above. Within the rectangular bins, n=the number of people who had the characteristic profile and risk=the proportion of people in that bin who died by suicide.

Among women who were hospitalized for psychiatric disorders, the highest risk of suicide was in women who were prescribed antipsychotics and had a poisoning diagnosis in the preceding 48 months (n=313, suicide risk=93%). The second highest risk group was women who were prescribed antipsychotics and anxiolytics and diagnosed with a specific personality disorder but were not diagnosed with poisoning in the preceding 48 months (n=172, risk=91%). Another interesting combination of variables was among women who had a poisoning diagnosis in the preceding 48 months, were not prescribed antipsychotics in the preceding 48 months nor anxiolytics in the preceding six months, and were not in the highest income quartile. Among women with this risk profile, those who had a recurrent major depressive disorder diagnosis in the preceding six months (n=22) had an 86% risk of postdischarge suicide whereas those who did not have recurrent depression (n=254) had a 10% risk of postdischarge suicide. Figure 2 shows other combinations of predictors and their associated suicide risks among women (AUC=0.83, 95% CI: 0.80, 0.86).

Figure 2. Classification tree depicting suicide predictors among female patients hospitalized for psychiatric disorders in Denmark, 1995 – 2015.

Note: ^aPoisoning by, adverse effect of and underdosing of drugs, medicaments and biological substances. Each shaded rectangular bin at the bottom (terminal node) represents the group of people with the characteristic profile in the branches above. Within the rectangular bins, n=the number of people who had the characteristic profile and risk=the proportion of people in that bin who died by suicide.

Random forests.

Among men who experienced a psychiatric hospitalization, 64% to 67% (fold1-fold 2) of the predictors had a mean decrease in accuracy above zero (mean=3.8, SD=3.4). Fifteen predictors were among the top 30 most important predictors in both folds (Figure 3). The most important variables for predicting suicide included age greater than 30 years, alcohol related disorders, nicotine dependence, major depressive disorder, unspecified dementia, antidepressants, reaction to severe stress, and adjustment disorders. The AUC for the random forest across folds was 0.82 (95% CI: 0.80, 0.83).

Figure 3. Variable importance of suicide predictors among male patients hospitalized for psychiatric disorders in Denmark from split sample cross-validation, 1995 – 2015.

The black dots represent the mean decrease in accuracy (MDA) value in fold 1 and the gray dots represent the MDA value in fold 2. The vertical line represents the average of the MDA values of all predictors with nonzero MDA values in folds 1 and 2 (3.8).

Among women who were hospitalized for a psychiatric disorder, 62% to 64% (fold1-fold 2) of the predictors had a mean decrease in accuracy above zero (mean=2.9, SD=2.2). Twelve predictors were among the top 30 most important predictors in both folds (Figure 4). The most important predictors of suicide among female patients hospitalized for psychiatric disorders included progestogens and estrogens in combinations, poisoning, being older than 60 years, receiving a state pension, antipsychotics, bipolar disorder, and major depressive disorder. The AUC for the random forest across folds was 0.85 (95% CI: 0.83, 0.87).

Figure 4. Variable importance of suicide predictors among female patients hospitalized for psychiatric disorders in Denmark from split sample cross-validation, 1995 – 2015.

The black dots represent the mean decrease in accuracy (MDA) value in fold 1 and the gray dots represent the MDA value in fold 2. The vertical line represents the average of the MDA values of all predictors with nonzero MDA values in folds 1 and 2 (2.9).

Operating characteristics of high-risk thresholds.

The predicted probabilities from cross-validated random forests were rank ordered and operating characteristics were computed among individuals in the top quintile of the predicted risk distribution. Men in the top 5%, 10%, and 20% of predicted risk accounted for 23%, 38%, and 59% of all suicide cases among men, respectively (specificity: 96%, 92%, and 82%). Women in the top 5%, 10%, and 20% of predicted risk accounted for 38%, 52%, and 71% of all suicide cases among women, respectively (specificity: 96%, 92%, and 82%).

Discussion

This study demonstrates the complexity of the patient profiles that carry a high risk of suicide in the 30 days after discharge from psychiatric hospitalization among men and women in a full population sample. Our findings build upon previous work demonstrating that accurate suicide prediction requires a complex combination of a large number of psychological, physical, and social factors, many of which are time-varying.

For men, there were important interactions between specific risk profiles by anxiolytics and drugs used in addictive disorders. In women, we found an interaction between poisoning and recurrent major depressive disorder that elevates the risk of suicide after psychiatric hospitalization. Poisonings may be capturing some non-fatal suicide attempts in these data, so these results may suggest that women with recurrent major depression and a history of non-fatal suicide attempts are at high risk for postdischarge suicide. These novel interactions should be replicated in future studies and examined in conventional designs aimed at quantifying the causal joint effects of these variables.

A surprising predictor of post-discharge suicide among men in the random forests was nicotine dependence. Nicotine dependence may serve as a proxy for smoking and health problems that are linked with chronic hypoxia and risk taking, which in turn are associated with an increased risk of suicide.^38,39 Furthermore, smoking may be a form of self-medication for emotional distress, which in itself is a risk factor for suicide.⁴⁰ Nicotine dependence may represent a potentially transdiagnostic suicide prevention factor since it appeared among the top most important predictors of postdischarge suicide above many psychiatric disorders. This finding is worthy of additional research to quantify its effect on suicide risk. It is also noteworthy that alcohol related disorders were important for accurate prediction of suicide in men but less so in women according to the random forests and classification trees. Alcohol related disorders improved prediction accuracy to a small extent in women but they were not among the top 30 predictors in both folds in cross validation of the random forests and they did not appear in the classification tree for women. Previous work suggests that male suicide decedents may be more likely to have had alcohol use disorders than female suicide decedents, which may explain our findings.⁴¹ A novel predictor identified among women was prescriptions for progestogens and estrogens in combination. An earlier study found lower suicide mortality in estrogen users compared with nonusers.⁴² However, this association may be explained by the selective prescription of postmenopausal estrogens to healthier women.⁴³ It is noteworthy that these predictors should be interpreted as risk markers and not causal risk factors given that our analyses were not intended to quantify the causal effect of any of these predictors, but rather to examine their contribution to accurate prediction of postdischarge suicide.

Our random forests’ operating characteristics are comparable to those of a previous study that used machine learning to predict postdischarge suicide among Veterans¹² which found that the 5% of patients with the highest predicted risk accounted for 32% of suicides in the one month after psychiatric hospitalization. In our study, persons in the highest 5% of predicted suicide risk accounted for 23% of all suicide deaths among men and 38% among women one month postdischarge. This suggests that a prevention program delivered to only 5% of hospitalized patients with the highest predicted risk could capture a large proportion of patients who would otherwise die by suicide.¹²

This study has several limitations. We were unable to observe more detailed social data. For many patients, the hospital may represent a respite from intolerably stressful situations, but upon discharge, patients are re-exposed to the same stressors that preceded their hospitalization such as social isolation, financial difficulties, problematic relationships, dependent care responsibilities, and other stressors.⁴⁴ We lacked data on important suicide risk factors such as trauma exposure, sexual minority status, and homelessness. Adequate representation of social conditions and acute emotional states in registry data remains an ongoing challenge. A second limitation is that there may be measurement error of variables, which may decrease model performance and distort variable importance in random forests.⁴⁵ For example, a diagnosis of poisoning by, adverse effect and underdosing of drugs, medications, and biological substances that may include non-fatal drug overdose including accidental, intentional self-harm, assault or undetermined intent, as well as poisoning due to adverse effects or underdosing. The broadness of this indicator makes it susceptible to measurement error. We were unable to conduct probabilistic quantitative bias analyses to examine the impact of measurement error due to computational capacity limitations of the analytic server. A third limitation is that although we were able to perform 10-fold cross validation for the classification trees, we were unable to do so for random forests. We instead performed 2-fold cross validation of the random forests because the analytic server was unable to conduct 10-fold cross validation of 1000 trees. The generalizability of these results to the U.S. remains unclear but our results are generally consistent with existing U.S.-based suicide findings.

Our results illustrate the complexity of the interactions between risk factors that elevate suicide risk in the immediate period after hospitalization discharge and the ways that they differ between men and women. We also highlight surprising, novel factors that emerged as important predictors for accurate classification of postdischarge suicide that are worthy of additional research.