Validating a Predictive Algorithm for Suicide Risk with Alaska Native Populations

Abstract

Introduction:

The American Indian/Alaska Native (AI/AN) suicide rate in Alaska is twice the state rate and four times the U.S. rate. Healthcare systems need innovative methods of suicide risk detection. The Mental Health Research Network (MHRN) developed suicide risk prediction algorithms in a general U.S. patient population.

Methods:

We applied MHRN predictors and regression coefficients to electronic health records of AI/AN patients aged ≥13 years with behavioral health diagnoses and primary care visits between October 1, 2016 and March 30, 2018. Logistic regression assessed model accuracy for predicting and stratifying risk for suicide attempt within 90 days after a visit. We compared expected to observed risk and assessed model performance characteristics.

Results:

10,864 patients made 47,413 primary care visits. Suicide attempt occurred after 589 (1.2%) visits. Visits in the top 5% of predicted risk accounted for 40% of actual attempts. Among visits in the top 0.5% of predicted risk, 25.1% were followed by suicide attempt. The best fitting model had an AUC of 0.826 (95% CI: 0.809–0.843).

Conclusions:

The MHRN model accurately predicted suicide attempts among AI/AN patients. Future work should develop clinical and operational guidance for effective implementation of the model with this population.

INTRODUCTION

Approximately 50,000 people die by suicide in the United States (US) annually, 2.5 times more than homicide.¹ Suicide is the 10^th leading cause of death, and the national suicide rate increased 35% between 1999 and 2019.¹ Significant disparities in suicide-related morbidity and mortality exist between groups by geography, race, ethnicity, and gender.^2,3

Alaska has one of the highest suicide rates in the US (28.51/100,000 in 2019), and some Alaskans are at higher risk than others. In 2019, the suicide rate among American Indian/Alaska Native (AI/AN) people in Alaska was 2.3 times the state rate and 4.7 times the US rate.¹ From 2015–2019, suicide rates among AI/AN males and females ages 18–34 years in Alaska were nearly 8 and 11 times higher than the respective US rates for this age group.¹ Despite decades of suicide prevention initiatives and research, Alaska’s AI/AN suicide outcomes have not improved.^4–8

Recent research demonstrates that healthcare systems are uniquely positioned to address suicide risk.⁹ The US Air Force multilevel suicide prevention program and Henry Ford Health System’s Perfect Depression Care Program significantly reduced suicide morbidity and mortality among health system users through systemic initiatives to identify people with high risk.^10–12 Most people who die by suicide make at least one healthcare visit in the year prior.¹³ At a tribal healthcare system in Alaska, 65% of AI/AN patients with a suicide-related healthcare visit saw a primary care provider in the preceding year, while <25% saw specialty behavioral health providers.⁶ These data indicate that healthcare systems, primary care clinics especially, are important settings for preventing suicide.

Many healthcare systems use self-report, depression screening tools such as the Patient Health Questionnaire (PHQ), item 9 of which can predict suicide risk for up to two years in the general population.^14,15 However, the PHQ was not found to predict suicide risk among AI/AN patients in Alaska.⁶ Many people who attempt or die by suicide neither disclose suicidality even when asked orshow behavioral warning signs prior to a suicide attempt, further challenging risk detection efforts that rely solely on self-report.^16,17 These factors are particularly relevant in populations in which maltreatment by government, research, and healthcare systems has cultivated mistrust and reticence to disclose distress or seek support outside the community. Furthermore, healthcare systems face obstacles to universal screening for depression or suicide and many primary care providers are often not adequately trained for these conversations.^18–21 Innovative, effective, and sustainable methods are needed to help healthcare systems identify and intervene with AI/AN individuals at risk for suicide.

Information collected in healthcare systems can be used to detect and stratify suicide risk in patient populations. Many factors routinely documented in the electronic health record (EHR), including certain diagnoses, history of suicidal behavior, and history of substance or behavioral health treatment, have been found to be positively, moderately associated with suicide risk. However, none of these factors alone can identify individuals with suicide risk or predict suicidal behavior, and few studies have examined multiple risk factors.²² Risk prediction models that use machine learning techniques can incorporate large volumes of data on numerous, co-occurring risk factors, calculate interaction effects, and yield greater prediction accuracy than self-report screening methods.²³ The Mental Health Research Network (MHRN), a consortium of 13 health systems serving over 12.5 million patients across 15 states,^24,25 recently used machine learning methods to develop algorithms that accurately predicted suicide attempts and deaths among people with behavioral health diagnoses in the general US population.²⁶ However, few studies have evaluated the differential predictive value of these algorithms by race or ethnicity. And fewer studies have determined if these models accurately detect suicide risk in populations with unique risk profiles and disparate burden of suicide morbidity and mortality, such as the AI/AN population in Alaska.

We applied two MHRN predictive algorithms for suicide attempt (hereafter called the “MHRN model”)²⁶ to EHR data in a tribal health system. Our goal was to determine if the MHRN model accurately detects and stratifies suicide risk in an AI/AN population. If accurate, this approach may be beneficial for identifying AI/AN patients with elevated risk and intervening to prevent suicide attempt and death.^8,27–32

METHODS

Setting

We applied the MHRN model developed for an Epic EHR database to the Cerner EHR at Southcentral Foundation (SCF), an Alaska Native-owned, non-profit healthcare system serving 65,000 AI/AN people in south central Alaska.³³ SCF services are funded through Indian Health Service (IHS), private insurance, and Medicaid/Medicare. SCF provides integrated behavioral health services at two large, urban primary care centers and several regional clinics, as well as 13 specialty behavioral health and substance use treatment programs. SCF also jointly owns and manages the Alaska Native Medical Center, 170-bed tertiary referral center for tribal health facilities statewide.

Approvals

Approvals were obtained from the Alaska Area Institutional Review Board, as well as the SCF and the Alaska Native Tribal Health Consortium (ANTHC) research review committees.³⁴. SCF and ANTHC compliance officers provided privacy consultations to ensure protection of patient data.

Sample

This study sampled outpatient visits, potentially including multiple visits for any individual patient. The sample consisted of outpatient primary care visits that occurred between October 1, 2016 and March 30, 2018.. Visits were included in the sample if the patient was: 1) aged 13 or older, 2) AI/AN, and 3) had a qualifying behavioral health diagnosis documented in the EHR. Qualifying diagnoses are listed in Table 1.

Table 1.

Characteristics of sampled primary care visits made by AN/AI patients ages 13 and older with behavioral health diagnoses at Southcentral Foundation October 1, 2016 through March 31, 2018

	All visits (n = 47413)		Visits with suicide attempt within 90 days (n = 589)
Characteristic	N	%	N	%
Patient gender (female)	30516	64.4	405	68.8
Patient age in years
13–17	2966	6.3	42	7.1
18–29	13027	27.5	218	37.0
30–44	13861	29.2	175	29.7
45–64	13349	28.2	134	22.8
65+	4210	8.9	20	3.4
Primary health insurance
Commercial	6522	13.8	20	3.4
Medicare	7160	15.1	67	11.4
Medicaid	22856	48.2	449	76.2
Indian Health Service	10561	22.3	53	9.0
Other	295	0.6	0	0.0
Charlson comorbidity index, past 1 year
0	30464	64.3	320	54.3
1	9571	20.2	134	22.8
2+	7378	15.5	135	22.9
Behavioral health diagnoses, past 5 years
Depressive disorder	27687	58.4	415	70.5
Anxiety disorder	30897	65.2	459	77.9
Bipolar disorder	3943	8.3	106	18.0
Schizophrenia spectrum disorder	4417	9.3	125	21.2
Personality disorder	7290	15.4	124	21.1
Post-traumatic stress disorder	5989	12.6	140	23.8
Other psychotic disorder	4339	9.2	102	17.3
Alcohol use disorder	18386	38.8	349	59.3
Drug use disorder	16343	34.5	347	58.9
Prior self-harm diagnoses
Prior 1 year	1949	4.1	232	39.4
Prior 5 years	3728	7.9	276	46.9
Behavioral health service use, past 5 years
Emergency department	21252	44.8	477	81.0
Hospitalization	14225	30.0	318	54.0
Behavioral health specialty clinic	8699	18.3	150	25.5
Psychiatric medication refills, past 5 years
Antidepressant	25381	53.5	401	68.1
Benzodiazepine	14747	31.1	278	47.2
Hypnotic	7028	14.8	116	19.7
Atypical antipsychotic	10266	21.7	267	45.3
PHQ* Item 9 score recorded
At index visit	9	<0.1	0	0.0
At any visit, past 1 year	762	1.6	8	1.4

^*PHQ, Patient Health Questionnaire

Data sources

Data were electronically extracted from SCF’s EHR for up to five years before each sampled visit. The MHRN model includes 102 predictors, including demographic (e.g., age, gender, type of insurance coverage), clinical (e.g., behavioral health and substance use diagnoses, dispensed psychiatric medications, past suicide attempts, other injuries or poisonings), and utilization characteristics (e.g., emergency department). A full list of MHRN predictors is provided in Appendix A. In addition, we collected EHR data on suicide attempts (hereafter called ‘suicide attempts’) within 90 days after each sampled visit. Consistent with the MHRN definition, a suicide attempt was identified as injury or poisoning coded in the EHR as resulting from intentional self-harm or having undetermined intent. Appendices B and C provide a list of diagnostic codes considered to indicate suicidal behavior and a link to behavioral health diagnostic codes, respectively

Statistical analysis

Descriptive statistics are presented as frequency and percentage. We used the MHRN model to predict risk of suicide attempt within 90 days after primary care visits made by AI/AN patients aged 13 and older with behavioral health diagnoses. We used the predictors and applied the assigned weights (regression coefficients) estimated in the MHRN model to comparable SCF EHR data. Because the original MHRN model was fit using logistic regression, linear predictions were on the logit scale; we applied the inverse logit function to calculate risk of suicide attempt after each visit as a proportion ranging from 0 to 1. We also categorized predicted risk into seven risk strata by percentile: <50th, 50th-75th, 75th-90th, 90th-95th, 95th-99th, 99th-99.5th, and >99.5th.

Classification performance, or accurate sorting of high and low risk, was assessed using receiver operating characteristic (ROC) curves and the associated area under the curve (AUC) or C-statistic. AUC values range from 0 to 1; a value of 0.5 indicates the model performs no better than chance while a value of 1 indicates perfect prediction. Results are presented as AUC with asymptotic normal 95% confidence intervals. We also assessed sensitivity within each predicted risk score percentile stratum by calculating the percent of observed suicide attempts within each pre-specified percentile stratum (column percent). Positive predictive value was evaluated by calculating the percent of observed suicide attempts among sampled visits in each pre-specified risk score percentile (row percent). Results are presented as percentages with exact binomial 95% confidence intervals to provide an estimate of error based on the visit sample captured over this study period.

We assessed performance characteristics for the MHRN model using two algorithms with differing numbers of variables. The least inclusive algorithm (Model 1) included predictors for age, sex, insurance, behavior health diagnoses, prescriptions, and prior service utilization. These variables reflect data typically accessible by insurers and health plans. The most inclusive algorithm (Model 2) adds race, ethnicity, neighborhood income, educational attainment; behavioral health diagnoses recorded on the day of the sampled visit; and PHQ-9 data recorded both prior to and on the day of the sampled visit. These variables may be more challenging to access and require linkage to external resources and “real-time” visit information. We chose to assess these two models because the least inclusive would be the easiest to implement in a health care setting like SCF and predicted risk could be calculated ahead of a scheduled visit. However, if the most inclusive model provided better predictive accuracy, then it may be worthwhile to access the additional data and calculate risk after a visit. Each model was developed independently by the MHRN and allowed for separate variable selection and coefficient estimation. Some MHRN predictor variables were not available at SCF, including: indicator for high-deductible insurance plan, neighborhood income, and educational attainment. Those variables were “omitted”, or set to zero, for the calculation of predicted risk of suicide attempt in the SCF sample and represent deviations from the MHRN algorithms. Other MHRN predictor variables were not applicable to the SCF sample; specifically, the SCF sample included only AI/AN people and no visits were covered by self-funded employer insurance. Predictor variables that had an interaction effect with AI/AN race in the MHRN models were included using the interaction regression coefficient estimate as the assigned weight. These omissions were not deviations from the MHRN model but reflect differences in the sampled populations. All analyses were conducted using Stata 15.1.³⁵

RESULTS

Sample characteristics

The sample included 47,413 primary care visits made by 10,864 unique patients. Of these patients, 168 people had a suicide attempt during the 18-month study timeframe. A total of 589 sampled visits (1.2%) were followed by suicide attempt within 90 days. The number of total visits per patient ranged from 1 to 88 with a median of 3 per patient (IQR: 1–5). Characteristics of the total sample of visits and visits with subsequent suicide attempts are shown in Table 1. More than half of all sampled visits were made by female patients (64.4%). A third of visits (33.8%) were made by patients ages 13 to 29 years. Medicaid was the most common insurance (48.2%). Anxiety (65.2%) and depressive disorder (58.4%) were the most common behavioral health diagnoses in the 5 years prior to the sampled visit. Many visits were preceded by behavioral health-related emergency department visits (44.8%) and prescription antidepressant refills (53.5%) in the prior 5 years.

Classification performance

ROC curves for model performance are shown in Figure 1. Model 1 had an AUC of 0.826 (95% CI: 0.809–0.843). The additional variables in Model 2 (AUC=0.820, 95% CI: 0.803–0.838) did not improve classification accuracy beyond that observed for Model 1.

Figure 1.

Receiver operating characteristic curves for predictive models of suicide attempt within 90 days following a primary care visit with a behavioral health diagnosis using data from sampled visits among Alaska Native patients at Southcentral Foundation Oct 2016 - Mar 2018 (n=47,413)

Note: Model 1 includes age, sex, insurance, and behavior health diagnoses, prescriptions, and service utilization prior to the day of the sampled visit; Model 2 includes Model 1 variables and ethnicity, behavioral health diagnoses recorded on the day of the sampled visit, and PHQ-9 data recorded prior to and on the day of the sampled visit

Sensitivity

Table 2 shows the percentage of all observed (actual) suicide attempts occurring in each predicted risk score percentile stratum. A higher percentage of observed suicide attempts in higher risk score strata (e.g., >95th percentile) compared to lower risk score strata (e.g., <50th percentile) indicates better classification accuracy. Percentages can be interpreted as sensitivity for each risk score strata or summed across multiple strata. For Model 1, 38.6% of observed suicide attempts occurred following visits with the highest 5% of predicted risk (>95th percentile). In contrast, less than 10% of suicide attempts occurred after visits with the lowest 50% of predicted risk for Model 1 (<50th percentile). Results were similar for Model 2; however, some accuracy was lost for visits with the highest 1% of predicted risk (>99th percentile; Model 1 accuracy 19.17%, Model 2 accuracy 14.77%).

Table 2.

Sensitivity for MHRN predictive models of suicide attempt within 90 days after a primary care visit among sampled visits at Southcentral Foundation Oct 2016 - Mar 2018 (n=47413)

	Model 1		Model 2
Risk score percentile	% of all attempts	(95% CI)	% of all attempts	(95% CI)
>99.5th	9.17	(6.96, 11.8)	6.79	(4.90, 9.13)
99th to 99.5th	10.0	(7.71, 12.7)	7.98	(5.92, 10.5)
95th to 99th	19.4	(16.2, 22.8)	26.1	(22.6, 29.9)
90th to 95th	8.32	(6.22, 10.8)	9.00	(6.81, 11.6)
75th to 90th	30.4	(26.7, 34.3)	22.6	(19.3, 26.2)
50th to 75th	13.2	(10.6, 16.2)	17.0	(14.0, 20.3)
<50th	9.51	(7.26, 12.2)	10.5	(8.17, 13.3)

Cells display percentage of all observed suicide attempts occurring in each predicted risk score percentile (column percent); MHRN = Mental Health Research Network; CI = confidence interval

Positive predictive value

Table 3 shows the percentage of observed suicide attempts among sampled visits in each predicted risk score percentile stratum. Agreement between observed risk and model-predicted risk within each stratum and greater separation of observed risk in the highest and lowest strata indicate better predictive performance. Percentages can be interpreted as positive predictive value for each risk score strata. For the two highest predicted risk score percentiles (99th-99.5th and >99.5th) in Model 1, >20% of visits were followed by a suicide attempt within 90 days. In contrast, in the lowest two predicted risk score percentiles (<50th and 50th-75th), <1% of visits were followed by a suicide attempt. Results were similar for Model 2; however, positive predictive value was lower in the two highest risk score percentiles.

Table 3.

Positive predictive value for MHRN predictive models of suicide attempt within 90 days after a primary care visit among sampled visits at Southcentral Foundation Oct 2016 - Mar 2018 (n=47413)

	Model 1		Model 2
Risk score percentile	% attempts among risk score percentile	(95% CI)	% attempts among risk score percentile	(95% CI)
>99.5th	22.8	(17.6, 28.7)	16.9	(12.3, 22.3)
99th to 99.5th	24.9	(19.5, 30.9)	19.8	(15.0, 25.5)
95th to 99th	6.01	(4.98, 7.18)	8.12	(6.93, 9.44)
90th to 95th	2.06	(1.53, 2.72)	2.24	(1.68, 2.91)
75th to 90th	2.52	(2.17, 2.91)	1.87	(1.57, 2.21)
50th to 75th	0.66	(0.52, 0.82)	0.84	(0.69, 1.03)
<50th	0.24	(0.18, 0.31)	0.26	(0.20, 0.34)

Cells display percentage of observed suicide attempts among sampled visits in each predicted risk score percentile (row percent); MHRN = Mental Health Research Network; CI = confidence interval

DISCUSSION

This study found that routinely-collected EHR data can be used to detect suicide risk among AI/AN patients. The MHRN model accurately detected and stratified risk of suicide attempts among AI/AN people with behavioral diagnoses following primary care visits. This model provides health systems serving AI/AN people an important opportunity to mitigate suicide-related health disparities in this population.

Model 1 and Model 2 performed similarly, with AUC of 0.826 and 0.820, respectively, though Model 2 was less accurate for those at highest risk of suicide (e.g. in the 99.5^th percentile). Within Model 2, some variables were not applicable or available for this population, including race, neighborhood median income, and education level. Had these latter two variables been available, they may have improved the performance of Model 2. Additionally, in a previous study with the SCF population, behavioral health conditions and PHQ-9 scores were not associated with suicide attempt, while utilization, opioid dispense, and poisoning/overdose diagnoses were associated.³⁶ This may explain in part why the inclusion of behavioral health conditions on the day of the visit and PHQ-9 scores, both prior and day of, in Model 2 did not greatly improve performance over Model 1 which already included utilization and prescriptions.

The striking accuracy of the MHRN model at SCF is notable given several important differences between SCF and the health systems in which the model was developed. First, healthcare visits by AI/AN people comprise a small proportion of the original sample in which these models were developed, whereas all visits in our sample were with AI/AN patients. Second, the MHRN model was developed with visits that occurred between 2009 and 2015, when both predictor diagnoses and outcome diagnoses were recorded using ICD-9 codes, whereas the SCF data was recorded using ICD-10 codes. Third, our data were recorded in and extracted from a Cerner database rather than the Epic databases used in MHRN health systems.

The proportion of visits followed by suicide attempt in this sample (1.24%) was approximately five times that found in the MHRN sample (0.26%) where the models were developed. Consequently, the proportion of visits followed by suicide attempt among the highest-risk patients was also several times higher in this sample (e.g., approximately 24% for visits above the 99^th risk score percentile in this sample, compared with approximately 6% for visits above the 99^th risk score percentile in the MHRN primary care sample). While some have questioned whether prediction models yield high enough positive predictive value to justify clinical implementation, we believe that a predicted near-term risk of suicide attempt exceeding 20% certainly warrants additional assessment and care planning.

Suicide risk prediction using EHRs could be an important component of strategies to prevent suicide in AI/AN communities. While this is the first study to assess a suicide risk algorithm using EHR data in a tribal setting, Haroz and colleagues applied a suicide risk prediction algorithm to a community-based suicide surveillance database to identify individuals at risk of suicide attempt up to 2 years after their initial visit.³⁷ These authors noted the need to extend this type of suicide risk prediction to health care systems serving tribal communities to increase the impact such interventions could have on AI/AN suicide outcomes nationally. This study demonstrates that the MHRN model has potential to strengthen current clinical practices and substantially improve suicide risk detection among AI/AN patients. It could be used to provide the support that AI/AN people with lived experience of suicide risk have noted as vital.⁷

Finally, these findings demonstrate that the MHRN model, developed within the Epic EHR database, can be applied to the Cerner EHR system showing the transferability of this risk prediction model. This is an important step toward the standardization of this model to work across diverse health systems.

Limitations

Some self-inflicted injuries or poisonings could have been mistakenly diagnosed as accidental, and some people attempting suicide may not have sought or received medical treatment within the tribal health system. In addition, some injuries or poisonings coded as having undetermined intent may have represented accidents, and some self-inflicted injuries or poisonings may have represented self-harm without intent to die. Random misclassification in either direction would introduce a conservative bias, yielding a falsely lower estimate of prediction model accuracy. Also, this algorithm can only include those individuals who make and attend healthcare appointments, which may not represent the entire AI/AN population in southcentral Alaska. Moreover, while several key risk factors for AI/AN suicide attempt were included in this algorithm, such as substance misuse and related health problems (e.g. PTSD, depression),⁷ important predictor variables that might more accurately predict suicide attempts for AI/AN people may be missing.

CONCLUSION

This study demonstrates that a predictive risk model for suicide attempt developed for a general U.S. patient population can accurately predict suicide risk in an AI/AN population using routinely-collected EHR data. This study also suggests that this model performs well across diverse EHR databases, even when certain data elements are not available. Future research should engage health system stakeholders to understand factors affecting the acceptability and feasibility of implementing a predictive algorithm for suicide attempt in primary care clinics serving AI/AN people.