Prediction of Sex-Specific Suicide Risk Using Machine Learning and Single-Payer Health Care Registry Data From Denmark

Key Points

Question

What are the sex-specific risk profiles for death from suicide in a general population sample?

Findings

In this case-cohort study of 14 103 individuals who died by suicide, risk profiles for suicide were different among men and women in a general population sample of 265 183 persons who did not die by suicide, with physical health more important to men’s than women’s suicide risk. Results suggest psychotropic medications and psychiatric disorders are important to suicide risk and, for many diagnostic variables and prescriptions, a longer vs shorter period of observation (eg, 48 vs 6 months prior to suicide) appeared to be more important.

Meaning

These findings suggest consistency with what is known about suicide risk but also potentially important, and understudied risk factors with evidence of unique suicide risk profiles among specific subpopulations.

Abstract

Importance

Suicide is a public health problem, with multiple causes that are poorly understood. The increased focus on combining health care data with machine-learning approaches in psychiatry may help advance the understanding of suicide risk.

Objective

To examine sex-specific risk profiles for death from suicide using machine-learning methods and data from the population of Denmark.

Design, Setting, and Participants

A case-cohort study nested within 8 national Danish health and social registries was conducted from January 1, 1995, through December 31, 2015. The source population was all persons born or residing in Denmark as of January 1, 1995. Data were analyzed from November 5, 2018, through May 13, 2019.

Exposures

Exposures included 1339 variables spanning domains of suicide risk factors.

Main Outcomes and Measures

Death from suicide from the Danish cause of death registry.

Results

A total of 14 103 individuals died by suicide between 1995 and 2015 (10 152 men [72.0%]; mean [SD] age, 43.5 [18.8] years and 3951 women [28.0%]; age, 47.6 [18.8] years). The comparison subcohort was a 5% random sample (n = 265 183) of living individuals in Denmark on January 1, 1995 (130 591 men [49.2%]; age, 37.4 [21.8] years and 134 592 women [50.8%]; age, 39.9 [23.4] years). With use of classification trees and random forests, sex-specific differences were noted in risk for suicide, with physical health more important to men’s suicide risk than women’s suicide risk. Psychiatric disorders and possibly associated medications were important to suicide risk, with specific results that may increase clarity in the literature. Generally, diagnoses and medications measured 48 months before suicide were more important indicators of suicide risk than when measured 6 months earlier. Individuals in the top 5% of predicted suicide risk appeared to account for 32.0% of all suicide cases in men and 53.4% of all cases in women.

Conclusions and Relevance

Despite decades of research on suicide risk factors, understanding of suicide remains poor. In this study, the first to date to develop risk profiles for suicide based on data from a full population, apparent consistency with what is known about suicide risk was noted, as well as potentially important, understudied risk factors with evidence of unique suicide risk profiles among specific subpopulations.

This case-cohort study examines the risk of suicide in Danish men vs women with use of machine-learning tools.

Introduction

Suicide is a worldwide contributor to mortality.¹ The age-standardized suicide rate of 13 per 100 000 per year in the United States has not decreased in the past 50 years.^2,3 Suicide was recently identified as 1 of only 3 causes of death that is increasing in the United States.⁴ The age-standardized suicide rate in Denmark has been comparable to that in the United States in the past 2 decades, but decreased to 9.2 per 100 000 as of 2016.²

In 2014, the US National Institute of Mental Health and the National Action Alliance for Suicide Prevention Research Prioritization Task Force published a list of research priorities.³ Key questions were identified, including “How can we better or more optimally detect [suicide] risk?”³ This call for research is not surprising given that decades of research have not resulted in improved suicide prediction in clinical settings.⁵ Sociodemographic characteristics, psychiatric diagnoses, and physical health diagnoses are well-documented risk factors for suicide.^6,7,8 However, most of the research that generated these findings used conventional null hypothesis testing statistical methods,⁵ and these methods are limited when the goal is accurate identification of a high-risk patient. For instance, logistic regression is not designed to examine large, highly correlated sets of predictors or to elucidate interactions among predictors without a priori specification. Additional research using flexible modeling procedures that capture the complexities of the causes of suicide is needed.

The shift in focus to supervised machine learning methods in psychiatry allows for the development of novel suicide risk profiles that include broad constellations of predictors. Machine learning methods have been used in some small, clinical civilian samples,^9,10,11 and larger samples of Army members, veterans, or civilian hospital patients.^12,13,14,15 To our knowledge, no study has used machine learning methods to examine suicide risk in a full civilian population. Thus, the goal of this study was to use population-based, prospectively recorded Danish medical and social registry data and supervised machine learning methods to identify risk profiles for death from suicide. Suicide risk differs by many demographic factors.¹⁶ Although all factors are important to examine, we chose to focus on sex differences for the purposes of this study, given well-established sex differences in suicide risk.^17,18,19

Methods

Study Sample

The source population for this study was all individuals born or residing in Denmark as of January 1, 1995, coinciding with the switch from International Classification of Diseases, 8th Edition, to International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) and the inclusion of outpatient visits in the registries (n = 5 078 382).²⁰ We used a case-cohort study design, consistent with prior machine learning studies of suicide death.^12,13,21,22 Individuals considered cases in this study died by suicide between January 1, 1995, and December 31, 2015 (n = 14 103). The comparison subcohort was a 5% random sample of living individuals in Denmark on January 1, 1995 (n = 265 183). Cases and subcohort members were unmatched to allow for maximum variability in predictors for the machine learning analyses. This work was deemed exempt from review by the institutional review board at Boston University because of deidentified data and approved by the Danish Data Protection Agency. Data were analyzed from November 5, 2018, through May 13, 2019.

Data Sources

Universal medical coverage is provided to all residents of Denmark through a tax-funded health care system.²⁰ More than 90% of the Danish population has at least 1 contact with the health care system in a given year.²³ Receipt of health care is recorded in medical and administrative registries at the national level.^24,25 The 10-digit Central Personal Register number, a unique personal identifier assigned to all residents of Denmark, can be used to merge individual data across 8 registries.

The Danish Civil Registration System was established in 1968. The data have been updated daily since 1989 and are widely accepted as accurate.^26,27,28 For this study we included sex at birth, age, immigration status (yes or no), generation of citizenship, family suicidal behavior (parent or spouse), and marital status.^26,27

The Income Statistics Register began in 1970 and contains variables related to income (eg, salary, private pension contributions).²⁹ The Population Education Register contains information on educational level attained for 96.4% of the Danish population.³⁰ The Integrated Database for Labor Market Research contains information about employment beginning in 1980.³¹

The Danish Psychiatric Central Research Register records all psychiatric inpatient, outpatient, and emergency department treatment data since 1995, including treatment dates and primary and secondary diagnoses.³² We evaluated psychiatric disorders according to 2-digit ICD-10 codes (eg, code F20 was used to capture schizophrenia and its subtypes; code F25 was used to capture schizoaffective disorder). Diagnoses in this registry are documented as high quality.^32,33

The Danish National Patient Registry includes treatment date and primary and secondary diagnostic codes, contact type (inpatient, outpatient, emergency), surgical procedure, and selected examination codes.^20,34 Diagnoses from this registry were included as second-level ICD-10 groupings (eg, codes C15-C26 included as 1 variable: malignant neoplasms of digestive organs). Surgery procedure codes were included according to body system (eg, nervous system surgeries). Two validation studies have found high correlations between the data contained in the registry and medical records.^20,35

The Danish National Prescription Registry has cataloged data on all prescription drugs sold in Danish pharmacies since 1994.⁴ The registry includes dispensing date, product name, and Anatomical Therapeutic Classification code. Data for this study were coded according to level 3 Anatomical Therapeutic Classification codes (eTable in the Supplement). Data in the registry are considered complete and valid from 1995.³⁶

The Danish Cause of Death Registry includes age at death and manner of death (eg, natural, suicide), place of death, and autopsy results. Suicide cases were identified via this registry via ICD-10 codes of X60 to X84.³⁷ In a validation study, 90% of the deaths registered as suicides were confirmed by experts.³⁸

Statistical Analysis

Some predictors (eg, natal sex) could be used in their registry-based form, while others (eg, diagnoses, medications) were dummy-coded to create time-varying predictors (ie, intervals of 0-6, 0-12, 0-24, and 0-48 months before the first day of the suicide month). To estimate the prevalence of each predictor in the person-time that gave rise to cases, we randomly selected a date for each member of the subcohort and evaluated the prevalence of predictors at the above time intervals in relation to that date. Predictors from all time points were evaluated simultaneously.

The data reduction process included elimination of rare predictors (≤10 observations^13,39) and predictors with negligible associations with suicide (unadjusted odds ratio of <0.9 or ≥1.1). We eliminated diagnoses occurring in the emergency department owing to low positive predictive value.^40,41 The initial analytic data set contained 2564 predictor variables. Following data reduction, the final number of included predictors was 1339. The eTable in the Supplement provides considered and retained predictors.

Given our dual interests in identifying novel predictor/interactions and developing algorithms that may accurately predict suicide, we used recursive partitioning machine learning methods that automatize detection of associations between predictors and outcomes and interactions among predictors and provide metrics of predictor importance.⁴²

First, we estimated classification tree (CART) models as an initial evaluation of the data structure.^43,44 Classification tree modeling is a nonparametric method that builds a decision tree based on predictors and their combinations that result in the highest probability of differentiating cases from noncases. Classification tree modeling was implemented using the R package, version 3.5.2 rpart (R Foundation), which uses a 10-fold cross-validation procedure.⁴⁵ To mitigate risk of model overfit and increase visual interpretability, maximum tree depth and minimum number of observations in any node (terminal or parent) were set to 10. Given the class imbalance (sample suicide rate, 5%), CART modeling was implemented using equal priors rather than the rpart default of priors proportional to the outcome frequency.^42,46 Risk of suicide was computed for each identified combination of predictors.

Second, we implemented random forest, which is less likely to produce an overfit model than CART modeling because random forest uses bootstrap aggregation, using the R package randomForest.⁴⁷ Each forest was built with 1000 trees with a minimum of 10 observations needed to attempt a split and 37 variables sampled as split candidates at each node (ie, square root of total number of predictors; randomForest default). Given the class imbalance, each tree was built using all observations of suicide plus a random equally sized number of subcohort observations using the sampsize tuning parameter.^48,49 Split-sample cross-validation was used to generate individual-level random forests–predicted values (the analytic server would not permit 10-fold cross-validation of 1000 trees). We used mean decrease in accuracy to evaluate each variable in terms of main effects and interactions across all trees.⁴²

Prediction accuracy was evaluated using receiver operating characteristic curve analysis conducted in 1000 bootstrap replicates to estimate area under the curve (AUC) and its 95% CI.⁵⁰ Additional operating characteristics were evaluated using high-risk subgroups and thresholds of predicted risk (eg, based on CART terminal nodes; random forests–predicted values). Sensitivity was prioritized in accordance with the goal of identifying suicide cases. Analyses were conducted separately for men and women in SAS, version 9.4 (SAS Institute Inc) and R, version 3.5.2.^51,52

Results

Of the 14 103 persons who were suicide cases, 10 152 were men (72.0%), with 130 591 men (49.2%) in the corresponding comparison cohort. The suicide cases included 3951 women (28.0%), with 134 592 women (50.8%) in the corresponding comparison cohort. Persons who died by suicide were slightly older than subcohort members (mean [SD] for men: 43.5 [19] years vs 37.4 [21.8] years; women: 47.6 [18.8] vs 39.9 [23.4] years), more frequently divorced (men: 1225 [12.1%] vs 8228 [6.3%]; women: 682 [17.3%] vs 10 266 [7.6%]), and in the second-income quartile (men: 2516 [24.8%] vs 19 980 [15.3%]; women: 1364 [34.5%] vs 34 114 [25.3%]) (Table).

Table. Characteristics of the Suicide Cases and the General Population Subcohort, Denmark, January 1, 1995.

Variable	Men		Women
	Suicide Cases (n = 10 152)	Comparison Subcohort (n = 130 591)	Suicide Cases (n = 3951)	Comparison Subcohort (n = 134 592)
Age, mean (SD), y	43.5 (18.8)	37.4 (21.8)	47.6 (18.8)	39.9 (23.4)
Marital status, %
Married or registered partner	4000 (39.4)	53 640 (41.1)	1665 (42.1)	53 856 (40.0)
Divorced	1225 (12.1)	8228 (6.3)	682 (17.3)	10 266 (7.6)
Single	4395 (43.3)	63 962 (49.0)	1103 (27.9)	55 238 (41.0)
Widow	467 (4.6)	3802 (2.9)	473 (12.0)	14 327 (10.6)
Unknowna	65 (0.6)	(959) 0.7	28 (0.7)	905 (0.7)
Immigrant, %	312 (3.1)	5698 (4.4)	167 (4.2)	5566 (4.1)
Income quartile, %
<1	1766 (17.4)	23 895 (18.3)	976 (24.7)	30 870 (22.9)
1 to <2	2516 (24.8)	19 980 (15.3)	1364 (34.5)	34 114 (25.3)
2 to <3	2387 (23.5)	23 962 (18.3)	962 (24.3)	31 841 (23.7)
3≥	2964 (29.2)	41 069 (31.4)	481 (12.2)	16 217 (12.0)
Unknowna	519 (5.1)	21 685 (16.6)	168 (4.3)	21 550 (16.0)

^aGiven the coverage of the Danish national registries, missing data were scarce. The few predictors with minimal missing data were imputed using the default approaches of rpart (surrogate variables) and randomForests (modal value).

CART Modeling

Among men, the highest risk for suicide was found among those not being treated by pharmacotherapy (eg, antidepressants, antipsychotics, or anxiolytics) and with a prior suicide attempt in the prior 4 years, and being in the second income quartile (n = 18; risk, 1.0). Similarly, men who received a prior diagnosis of poisoning by adverse effects or underdosing of drugs but did not have a coded prescription for antidepressants, antipsychotics, medications for addictions (eg, methadone), or hypnotics/sedatives in the prior 4 years had a risk of 0.42 (n = 251). Other combinations of importance are displayed in Figure 1 (AUC, 0.77; 95% CI, 0.77-0.78).

Figure 1. Classification Tree Depicting Suicide Predictors Among Men in Denmark, 1995-2015.

Each shaded rectangle at the bottom (terminal node) represents the group of people with the characteristic profile in the branches above. Within the rectangles, n indicates the number of people who had the characteristic profile and suicide indicates the proportion of people in that bin who died by suicide. AD indicates adjustment disorder; RSS, reaction to severe stress.

^aDrugs used in addictive disorders.

^bPoisoning by adverse effect and underdosing of drugs, medicaments (a substance used for medical treatment), and biological substances.

^cReaction to severe stress and adjustment disorders.

The highest risk among women was among those who did not have a recorded prescription for anxiolytics, antipsychotics, hypnotics or sedatives, or antidepressants, but made a suicide attempt in the previous 4 years (n = 16; risk, 1.0). Women who were prescribed hypnotic/sedatives and were diagnosed with poisoning by, adverse effect of, and underdosing of drugs in the 4 years before suicide had the next highest risk (n = 79; risk, 0.41). Other combinations of importance are displayed in Figure 2 (AUC, 0.87; 95% CI, 0.86-0.88).

Figure 2. Classification Tree Depicting Suicide Predictors Among Women in Denmark, 1995- 2015.

Each of the 2 columns of rectangles at the right (terminal node) represents the group of people with the characteristic profile in the branches at left. Within the rectangles, n indicates the number of people who had the characteristic profile and suicide indicates the proportion of people in that bin who died by suicide. MDD indicates major depressive disorder.

^aPoisoning, adverse effects, and underdosing of drugs, medicaments (a substance used for medical treatment), and biological substances.

^bDrugs used in addictive disorders.

^cβ-Lactam antibacterials, penicillins.

Random Forest

Among men, 90% to 91% (fold 1-fold 2) of the predictors had a mean decrease in accuracy above 0 (mean [SD], 7.8 [5.8]). Eighteen predictors were among the top 30 most important predictors in both folds (Figure 3). Removal of drugs used in addictive disorders 4 years before suicide from our models would have the largest association with accuracy. Other predictors in the top 30 included antidepressants, hypnotics/sedatives, and antipsychotics in the past 4 years, age, physical health diagnoses, and stress disorders. The AUC for the random forest across folds was 0.80 (95% CI, 0.79-0.81).

Figure 3. Variable Importance of Suicide Predictors Among Men in Denmark From Split Sample Cross-Validation, 1995-2015.

The blue dots represent the mean decrease in accuracy (MDA) value in fold 1 and the orange dots represent the MDA value in fold 2. The vertical line is the average of the MDA values of all predictors with nonzero MDA values in folds 1 and 2 (7.80).

^aPredictors that were in the top 30 predictors in folds 1 and 2 for men.

Among women, 87% to 89% (fold 1 – fold 2) of the total number of predictors had a mean decrease in accuracy above 0 (mean [SD], 4.6 [3.8]). Twenty-one predictors overlapped as a top 30 predictor in both folds, most of which involved psychiatric diagnoses and medications (Figure 4). Specifically, alcohol-related disorders, prior suicide attempts, drugs used in addictive disorders, schizophrenia, recurrent major depression, and stress disorders had the largest association with accuracy. The AUC for the random forest model across folds was 0.88 (95% CI, 0.88-0.89).

Figure 4. Variable Importance of Suicide Predictors Among Women in Denmark From Split Sample Cross-Validation, 1995-2015.

The blue dots represent the mean decrease in accuracy (MDA) value in fold 1 and the orange dots represent the MDA value in fold 2. The vertical line is the average of the MDA values of all predictors with nonzero MDA values in folds 1 and 2 (4.61).

^aPredictors that were in the top 30 predictors in folds 1 and 2 for women.

Operating Characteristics of High-Risk Thresholds

Cross-validated random forests–predicted probabilities were rank ordered and operating characteristics were calculated among individuals in the top quintile of the predicted risk distribution. Men in the top 5%, 10%, and 20% of predicted risk accounted (sensitivity) for 32.0%, 49.4%, and 65.9% of all male cases of suicide death, respectively (specificity, 97.1%, 93.1%, 83.6%). Women in the top 5%, 10%, and 20% of predicted risk accounted for 53.4%, 68.1%, and 81.0% of all female suicide deaths (specificity, 96.4%, 91.7%, 81.8%). The sensitivity among individuals in the top 5% of predicted risk was 6.4 times the expected value among men (32%/5%) and 10.6 times the expected value among women (53.4%/5%).

Discussion

We used supervised machine learning to develop sex-specific risk models for suicide using population-based data from the Danish national health care and social registries. From 2554 predictors, characterized across 4 different timeframes and spanning demographics, family history of suicide, previous suicidal behavior, psychiatric disorders, physical health disorders, and medications, we derived machine learning models that highlight important predictors and combinations.

Across all models, psychiatric disorders emerged as the most important predictors of suicide. This finding is largely consistent with what is known about psychiatric disorders and suicide risk.^7,53,54 However, some specific disorders that appeared in our results were novel. While schizophrenia and depression are well-established risk factors for suicide,^{17,55,56,57,58,59,60} there is some controversy in the literature with regard to whether stress disorders are a risk factor for suicide aside from depression.⁶¹ In these results, stress disorders are an important predictor of suicide among both men and women in models simultaneously evaluating depression. Furthermore, antidepressants, antipsychotics, hypnotics/sedatives, and medications used to treat addictions were important to the accuracy of predicting suicide across analyses. Although our results indicate that pharmacotherapy is important to suicide prediction, our noncausal models could not elucidate the direction or magnitude of associations.

Physical health diagnoses appeared to contribute more to suicide prediction for men than women. Conversely, psychiatric diagnoses and associated medications appeared to contribute more to the prediction of women’s suicide risk. It is possible that clinicians may more frequently assess for, recognize, and diagnose mental health symptoms among women, while these same symptoms may be attributed to somatic concerns among men.⁶² Previous machine learning work on suicide prediction among men relied on relatively healthy populations (eg, soldiers) and thus did not incorporate a detailed examination of physical health diagnoses.^12,13 We believe research will be needed to further examine physical health risk factors for suicide within a causal framework.

Limitations

This study has some important limitations worth noting. With regard to time-varying prediction, a longer period of observation before suicide (eg, diagnoses that occurred 48 months earlier) was more important to suicide prediction than a shorter period of observation before suicide (eg, diagnoses that occurred 6 months earlier). A limitation of this study is that nondiagnostic predictors occurring in the immediate days before suicide may be particularly important to risk (eg, acute interpersonal stress, loneliness), but these data are not available in the Danish registries. A challenge to epidemiologic studies of suicide in general is obtaining data on the sample size needed to study the causes of this relatively rare event (usually medical registry data), while also obtaining data on negative life factors not typically found in data sources of this size. It is likely that long-term diagnostic risk factors and acute life events and emotional states interact to most accurately characterize suicide risk. How to best capture long-term and acute risk factors in large, population-based samples has been an ongoing challenge to suicide epidemiologic research.

Another limitation to the present study is the reliance on only 2 machine learning classifiers: CART and random forest. We used these approaches to prediction for substantive reasons (ie, variable importance), but other classifiers or meta-classifiers (eg, super learning) could achieve better prediction performance.⁶³ Furthermore, despite our bootstrap and cross-validation approach, spurious findings are still possible. In addition, it is possible that observed sex differences are the result of differences in modeling; thus, these findings should be considered exploratory. Given our case-cohort study design, the risks presented in the CART terminal nodes may not reflect the true population risk. Results from this study should not be interpreted as causal effects. In addition, although there is a rigorous and thorough process for suicide death classification in Denmark (ie, a full inquest into the cause of death), suicide death misclassification is possible. The extent to which these results may generalize to populations outside of Denmark is unclear, although concerns about generalization may be assuaged given that our results are consistent with the existing, primarily US-based, suicide literature.

Conclusions

Despite decades of suicide research, our understanding of suicide risk remains poor. Machine learning may allow for the development of prediction models that can evaluate many relevant predictors of suicide simultaneously. To our knowledge, this study is the first to develop prediction models for suicide based on data from a full population. We found what appears to be consistency with what is known about suicide risk but also potentially important, understudied, predictors for future study. Results of this study can possibly be replicated in other novel data sets and used to inform the further development of general population prediction models for suicide.

Supplement.

eTable. Variables Included in the Analyses