Performance of a Prediction Model of Suicide Attempts Across Race and Ethnicity

Abstract

This study examines whether race disparities exist in the prediction of suicide attempts and if have they have detrimental effects on individuals and health care systems

Innovative prevention strategies are needed to reduce US suicide rates, which have been steadily increasing with a recent disproportionate increase among Black and Hispanic populations.¹ Predictive models of suicide risk have been developed using machine learning with electronic health records (EHRs).² Some models achieve the performance needed to cost-effectively target high-risk individuals.³ However, recent work applying a suicide-death prediction model found excellent performance at the population level (indexed by area under the receiver operating characteristic curve [AUC] = 0.82), but much lower AUC for the American Indian or Alaskan Native subsample (AUC = 0.60).⁴ We examined whether similar disparities exist in the prediction of suicide attempts, which are far more common and have detrimental effects on individuals and health care systems, even when nonfatal.

Methods

This diagnostic/prognostic study followed Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guidelines for prediction model validation; model development details are available elsewhere.⁵ A previously developed logistic regression prediction model of fatal and nonfatal suicide attempts⁵ was applied to an independent cohort of outpatient mental health encounters at Kaiser Permanente Northern California between October 1, 2016, and September 30, 2017, for patients older than 12 years. The Kaiser Permanente Northern California institutional review board approved this study and waived the need for informed consent. The model used only EHR data, including demographics, diagnoses, medications, and suicidality.⁵ Classification performance was assessed with AUC and sensitivity was calculated at 2 thresholds that yielded either 95% or 99% specificity in the full sample, which represents potentially realistic intervention thresholds given the burden of false positives on clinicians and health care systems.^3,6 Means with 95% CIs were estimated for each metric using bootstrapping (1000 replications).

Results

There were 1 408 682 encounters among 254 777 unique patients with mean (SD) age of 40.7 (18.7) years, including 164 920 women (64.7%), 89 857 men (35.3%), and 35 267 individuals with Medicare coverage (13.8%). The Table provides the distributions of self-reported race and ethnicity (US Census categories), suicide outcomes, and model performance metrics. There were 9093 visits followed by a suicide attempt within 90 days, which reflects a rate of 6.5 attempts per 1000 visits; across subgroups, this rate ranged from 4.6 (other/unrecorded race) to 19.3 (American Indian or Alaska Native). The Figure summarizes model performance statistics. The model achieved excellent overall classification, with mean AUC = 0.85 (95% CI, 0.84-0.85). Although there was variation across subsamples (AUCs 0.83-0.90), none of the AUCs were in the poor range previously observed for some subsamples in a suicide-death model.⁴ At 95% specificity, average sensitivity ranged from 28% to 48% and was higher for the White subsample vs most other groups; at 99% specificity, sensitivity ranged from 7% to 38% and was highest for the Native Hawaiian or Pacific Islander subsample and lowest for the Asian and other/unrecorded subsamples (Table).

Table. Observed Suicide Attempt Outcomes and Model Performance Metrics Across Race and Ethnicity in the Sample of All Outpatient Mental Health Encounters at Kaiser Permanente Northern California Between October 1, 2016, and September 30, 2017.

Race and ethnicity	No. (%)				Mean (95% CI)
	Visits	Visits followed by suicide attempt (per 1000 visits)	Unique patients	Patients with at least 1 suicide attempt	Area under the curve	Sensitivity at 95% specificity	Sensitivity at 99% specificity
Full sample	1 408 682 (100)	9093 (6.5)	254 777 (100)	1408 (0.6)	0.85 (0.84-0.85)	0.41 (0.40-0.42)	0.18 (0.17-0.19)
American Indian or Alaska Native	7671 (0.5)	148 (19.3)	1401 (0.5)	20 (1.4)	0.83 (0.80-0.86)	0.28 (0.22-0.36)	0.20 (0.14-0.27)
Asian	135 823 (9.6)	646 (4.8)	24 690 (9.7)	124 (0.5)	0.84 (0.82-0.85)	0.34 (0.31-0.38)	0.11 (0.09-0.14)
Black	122 378 (8.7)	717 (5.9)	21 204 (8.3)	111 (0.5)	0.84 (0.83-0.86)	0.39 (0.37-0.42)	0.18 (0.16-0.21)
Hispanic or Latino	199 595 (14.2)	1490 (7.5)	38 208 (15.0)	222 (0.6)	0.83 (0.82-0.85)	0.39 (0.37-0.42)	0.19 (0.17-0.21)
Native Hawaiian or Pacific Islander	6853 (0.5)	63 (9.2)	1433 (0.6)	10 (0.7)	0.90 (0.86-0.93)	0.48 (0.35-0.60)	0.38 (0.26-0.51)
Other or unrecorded race and ethnicitya	77 144 (5.5)	355 (4.6)	14 851 (5.8)	61 (0.4)	0.83 (0.81-0.85)	0.35 (0.30-0.41)	0.07 (0.04-0.10)
White	859 218 (61.0)	5674 (6.6)	152 990 (60.0)	860 (0.6)	0.85 (0.84-0.85)	0.44 (0.42-0.45)	0.19 (0.18-0.20)

^a This category includes individuals that self-identified as “other” or whose race and ethnicity status was not recorded in the electronic health record.

Figure. Receiver Operating Characteristic (ROC) Curve and Sensitivity of Suicide Attempt Prediction Model.

Discussion

The suicide-attempt model achieved excellent overall classification across all racial and ethnic groups. This contrasts with prior findings with a suicide-death model,⁴ where authors posited that poorer performance for some racial and ethnic groups may have resulted from a number of factors, including systemic barriers to utilization, practitioner bias, institutional discrimination, outcome misclassification, and low statistical power to detect race-specific risk factors given the rarity of the outcome. Although suicide-attempt models may be less vulnerable to the statistical power concern, the influence of all these factors warrant further investigation. A limitation of setting thresholds at high specificity is relatively high false-negative rates, which in this study varied across racial and ethnic groups; this underscores the importance of clinical workflows that always consider additional sources of information when assessing suicide risk. Ongoing efforts to incorporate social determinants and related factors into the EHR of integrated health care delivery systems may also be critical to improving suicide prediction models, particularly among historically underserved groups.

Supplement.

Data sharing statement.