This cohort study evaluates the association of use of a statistical model to evaluate suicide risk with improvements in evidence-based care and reductions in subsequent suicide-related behaviors in a Native American population.
Key Points
Question
How well does a statistical suicide risk model perform in a Native American community-based setting?
Findings
In this prognostic study of 400 Native American individuals in the White Mountain Apache Tribe’s suicide surveillance and case management system, implementation of a suicide risk model and related care improved risk identification and was associated with an extended reach of care and a reduced risk of suicidal behaviors among individuals identified as at risk for suicide and/or self-harm.
Meaning
The findings suggest that a statistical model to identify risk of suicide and an associated care model can be implemented in practice, but further testing is needed.
Abstract
Importance
There are many prognostic models of suicide risk, but few have been prospectively evaluated, and none has been developed specifically for Native American populations.
Objective
To prospectively validate a statistical risk model implemented in a community setting and evaluate whether use of this model was associated with improved reach of evidence-based care and reduced subsequent suicide-related behavior among high-risk individuals.
Design, Setting, and Participants
This prognostic study, done in partnership with the White Mountain Apache Tribe, used data collected by the Apache Celebrating Life program for adults aged 25 years or older identified as at risk for suicide and/or self-harm from January 1, 2017, through August 31, 2022. Data were divided into 2 cohorts: (1) individuals and suicide-related events from the period prior to suicide risk alerts being active (February 29, 2020) and (2) individuals and events from the time after alerts were activated.
Main Outcomes and Measures
Aim 1 focused on a prospective validation of the risk model in cohort 1. Aim 2 compared the odds of repeated suicide-related events and the reach of brief contact interventions among high-risk cases between cohort 2 and cohort 1.
Results
Across both cohorts, a total of 400 individuals identified as at risk for suicide and/or self-harm (mean [SD] age, 36.5 [10.3] years; 210 females [52.5%]) had 781 suicide-related events. Cohort 1 included 256 individuals with index events prior to active notifications. Most index events (134 [52.5%]) were for binge substance use, followed by 101 (39.6%) for suicidal ideation, 28 (11.0%) for a suicide attempt, and 10 (3.9%) for self-injury. Among these individuals, 102 (39.5%) had subsequent suicidal behaviors. In cohort 1, the majority (220 [86.3%]) were classified as low risk, and 35 individuals (13.3%) were classified as high risk for suicidal attempt or death in the 12 months after their index event. Cohort 2 included 144 individuals with index events after notifications were activated. For aim 1, those classified as high risk had a greater odds of subsequent suicide-related events compared with those classified as low risk (odds ratio [OR], 3.47; 95% CI, 1.53-7.86; P = .003; area under the receiver operating characteristic curve, 0.65). For aim 2, which included 57 individuals classified as high risk across both cohorts, during the time when alerts were inactive, high-risk individuals were more likely to have subsequent suicidal behaviors compared with when alerts were active (OR, 9.14; 95% CI, 1.85-45.29; P = .007). Before the active alerts, only 1 of 35 (2.9%) individuals classified as high risk received a wellness check; after the alerts were activated, 11 of 22 (50.0%) individuals classified as high risk received 1 or more wellness checks.
Conclusions and Relevance
This study showed that a statistical model and associated care system developed in partnership with the White Mountain Apache Tribe enhanced identification of individuals at high risk for suicide and was associated with a reduced risk for subsequent suicidal behaviors and increased reach of care.
Introduction
From 2010 to 2020, suicide was the 10th leading cause of death in the US.1 While there is significant heterogeneity, Native American individuals experience the highest suicide rate of all racial and ethnic groups in the US.2 Unlike other racial and ethnic groups, suicide deaths in the Native American population are disproportionately concentrated among adolescents and young adults aged 14 to 35 years.3 Moreover, these deaths are often underreported due to racial and ethnic misclassification of decedents and causes of death, resulting in an underestimation of the true burden.4 These rates and related challenges are a result of historical trauma and ongoing racism and discrimination, including chronic underfunding of the health system, resulting in limited access to culturally competent mental health care.
Native American communities are at the forefront of impactful approaches to solve the problem of suicide. These include the Celebrating Life (CL) system of the White Mountain Apache Tribe (WMAT). The CL system has 3 components: (1) community-wide active surveillance, (2) case management and referrals delivered by local community mental health specialists (CMHSs), and (3) community education. When comparing rates prior to CL implementation with those during CL implementation, suicide deaths decreased by 38% and attempts by 53%.5 Over time, the number of referrals to the CL program has grown. The increase in volume, combined with a small staff across a rural setting and limited access to specialty mental health services, has led to a need to prioritize more immediate follow-up care.
Yet, classification of suicide risk is complex.6 Clinical judgment has been consistently shown to be inaccurate.7,8,9 For example, 60% of patients who died by suicide were categorized by clinicians as low risk in the year before their death.10 Many health systems and programs are using brief, standard screening tools, but evidence for these tools is also mixed. Some studies have shown that screening surveys are accurate at estimating future suicide attempts and possibly deaths.11,12,13,14 However, other research has shown inaccuracies.15,16,17 Recent work has used machine learning to mine existing data or, in conjunction with collection of new data, to classify risk level.18 However, these models have rarely been implemented in practice.19,20,21
Meanwhile, despite Native American individuals facing the steepest disparities, few risk identification and classification approaches have been developed and validated specifically for this population. When models from other populations are applied to populations with different demographic and risk profiles, predictions can be inaccurate, with the potential to exacerbate disparities.22 Given the unique drivers of suicide risk in Native American populations23,24,25,26 and contextual factors that influence care,27 ensuring that risk classification approaches are accurate and feasible is paramount to reduce suicide-related disparities and promote health equity.
In partnership with the WMAT, we previously developed the Risk Identification for Suicide and Enhanced Care for Native American Populations (NATIVE-RISE) system, which includes a suicide risk model based on machine learning methods,28 and associated care pathways. In the current study, we aimed to (1) prospectively validate NATIVE-RISE’s embedded risk model and (2) evaluate the association of NATIVE-RISE with improving the reach of evidence-based care and reducing subsequent suicide-related behavior for high-risk individuals. To our knowledge, this is the first study within a Native American community to evaluate not only how a suicide risk model performs prospectively but also whether implementation of the model is associated with improved care and reduced risk for those most in need.
Methods
This prognostic study used data collected by the CL program from January 1, 2017, through August 31, 2022. Between January 1, 2017, and February 29, 2020, the model was calculated and stored in the CL program’s electronic record system but did not generate any notifications. As decided by CL program staff, suicide risk alerts were turned on starting March 1, 2020, in conjunction with biweekly team meetings to discuss high-risk cases and follow-up care. Because this study included a secondary analysis of deidentified data, it was determined to be non–human participants research as defined by US Department of Health and Human Services regulations (45 CFR §46.102). Approval for this project and publication of this article was granted by the White Mountain Apache Tribal Health Board and Tribal Council. The study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline.29
Background on Celebrating Life
Full details of the CL program can be found elsewhere.30 Briefly, it includes several key features. First, all community members who live or work on the reservation are required by tribal law to report individuals at risk of suicide to a central registry system. Reportable behaviors include suicidal ideation, self-injury, suicide attempts, suicide deaths, and binge substance use. Reports are made to the CL system using a standardized form called the Yellow Form. Second, for each report, a CMHS attempts to follow up with the individual. The purpose of the follow-up is to provide case management to help individuals access health and social services. A secondary aim is to verify the type of behavior and gather more understanding of the circumstances that led to that behavior using a standardized form known as the Pink Form. After the follow-up visit, CL staff may provide additional wellness checks, a form of brief contact intervention (BCI), and safety planning. Finally, CL also collects data using the Grey Form for any death by suicide in the community.
The CL program provides locally relevant data to guide ongoing suicide prevention efforts and has contributed, along with other efforts, to significantly reducing suicide rates.5 Over time, referrals to the CL system have grown. Given this growth, limited staff, and the large geographical area, there was a need for prioritization to help ensure staff reached those at the highest risk. Since 2017, The Johns Hopkins CL team has worked to enhance prioritization procedures through the development of a statistical model to identify those at highest risk28 and defining risk-concordant care.31 The risk model and associated care pathways are called NATIVE-RISE.
NATIVE-RISE
Briefly, the risk model in NATIVE-RISE was developed by examining 10 years of data from the Pink Forms and Grey Forms. Pink Form data are collected after an individual is reported to CL for having suicidal thoughts or behaviors. It includes data collected about the individual’s current living circumstances, mental health, social circumstances, and substance use history and information about the suicidal event. The Grey Form collects similar information but for individuals who have died by suicide and are reported to CL by local law enforcement and/or the health system. The risk model in NATIVE-RISE was developed to identify subsequent risk of suicide attempt or death in the 12 months after an individual has already had contact with CL. Across models evaluated, areas under the receiver operating characteristic curve (AUC) were large (>0.80).28 We selected lasso regression based on model performance and ability to be integrated into the data system. We then programmed this algorithm into the CL data system but did not allow for alerts to be triggered to allow for further validation prior to implementation. During this time, we did qualitative work to inform the implementation of the risk model.31 This resulted in the protocol as depicted in the Figure. After a Pink Form (ie, follow-up visit) is completed, the risk model generates an alert of risk status. As there are often clinical signs during the follow-up visits, the CMHS may also indicate that someone is at high risk even if the model indicates low risk. The critical role of CMHS insight, experience, and local knowledge also informed the selection of a cut point for the model that favored maximizing specificity.31 In practice, a CMHS and a supervisor meet biweekly to review high-risk cases flagged by the model or added based on CMHS feedback. All individuals determined to be at high risk for suicide receive regular follow-up wellness checks (ie, weekly for 4 weeks, then reducing in frequency depending on person’s status), a type of brief contact intervention, and safety planning.
Figure. Workflow That Incorporates Utilization Alerts Based on the Risk Identification for Suicide and Enhanced Care for Native American Populations Model.
Study Design
The implementation and evaluation of NATIVE-RISE focused on adults aged 25 years or older. We divided data into 2 cohorts: (1) individuals and events from the period prior to alerts being active and (2) individuals and events from the time after alerts were activated. The aim 1 analysis focused on a prospective validation of the risk model in cohort 1, while the aim 2 analysis compared the likelihood of repeated suicide-related events and the reach of BCI and wellness checks among high-risk cases between cohort 2 and cohort 1.
Participants
Cohort 1 included index cases (ie, new people with a suicide-related event) with or without subsequent contact with the CL system for suicide-related behaviors from January 1, 2017, to February 29, 2020. Cohort 2 included index cases with or without subsequent contact with the CL system for suicide-related behaviors from March 1, 2020, to August 31, 2022.
Statistical Analysis
Descriptive analyses were used to examine demographic characteristics of cohorts 1 and 2. The aim 1 analysis compared the odds of subsequent suicidal behaviors, as indicated by contact with the CL system, between individuals classified as low risk and high risk by the model at their index event. We used multivariate logistic regression controlling for the time under observation (ie, the time between the index event and February 29, 2020). We then compared this model with a multivariate logistic regression model controlling for the time under observation that examined the odds of subsequent suicidal behaviors by type of index event (eg, attempt vs ideation, binge substance use, and/or self-injurious behavior). The type of index behavior is often used as a proxy for risk.
For aim 2, we limited analyses to only individuals classified as high risk and compared the likelihood of subsequent suicidal behaviors and reach of BCI between cohorts 1 and 2. We used multivariate logistic regression models controlling for the time under observation (days between the index event and the end of the study period for the respective cohort), age at event, and index event type. Time under observation was measured continuously in days, while age and index event type were categorical. Age was categorized into bins of 5 years from ages 25 to 49 years or as 50 years or older. Dichotomous variables were generated for each type of index event. If a person did not have an event recorded after their index event, they were assumed to not have that event. Missing data were limited, with imputation based on previously collected data and/or use of complete case analysis.
Results are presented as odds ratios (ORs) with 95% CIs, with 2-sided statistical significance set to P < .05. In addition to the ORs, predicted probabilities are presented. We also present the adjusted AUC as a measure of diagnostic accuracy. Finally, we conducted several sensitivity analyses. The first focused on examining the impact of controlling for time under observation using a survival analysis and examining the hazard ratio of subsequent suicidal behaviors, comparing cohort 1 with cohort 2 (eFigure in Supplement 1). Because the COVID-19 pandemic impacted service utilization and led to businesses and schools being closed, we also examined whether our findings were consistent if events recorded from March 15, 2020, through July 31, 2020, were removed. If results were consistent without these observations, this would provide stronger evidence that our results captured the impact of the intervention rather than the impact of a concurrent event. All analyses were conducted using Stata, version 15 (StataCorp LLC).32
Results
Across both cohorts, 400 individuals identified as at risk for suicide and/or self-harm had 781 suicide-related events. The mean (SD) age was 36.5 (10.3) years. More than half of these individuals were female (210 [52.5%]), and 188 (47.0%) were male; data on sex were missing for 2 individuals (0.5%). Cohort 1 included 256 individuals (64.0%). Cohort 2 included 144 individuals (36.0%). Between cohort 1 and cohort 2, statistically significant differences were observed in the proportions of individuals whose index event was for suicidal ideation (102 of 256 [39.8%] vs 106 of 144 [73.6%]; P < .001) and for binge substance use (134 of 256 [52.3%] vs 28 of 144 [19.4%]; P < .001). While a higher proportion of individuals in cohort 1 than in cohort 2 had subsequent suicidal behaviors recorded in the CL system (101 of 256 [39.5%] vs 36 of 144 [25.0%]; P = .008), this finding may be attributed to the differing amount of time under observation between the 2 cohorts (Table 1).
Table 1. Demographic Characteristics of Participants in Cohorts 1 and 2.
| Characteristic | Participants, No. (%) (N = 400) | P value | |
|---|---|---|---|
| Cohort 1 (n = 255)a | Cohort 2 (n = 144)b | ||
| Sex | |||
| Female | 130 (51.0) | 79 (54.9) | .94 |
| Male | 123 (48.2) | 65 (45.1) | |
| Missing | 2 (0.8) | 0 | |
| Age category, y | |||
| 25-29 | 57 (22.4) | 52 (36.1) | .04 |
| 30-34 | 67 (26.3) | 36 (25.0) | |
| 35-39 | 39 (15.3) | 18 (12.5) | |
| 40-44 | 26 (10.2) | 11 (12.5) | |
| 45-49 | 31 (12.2) | 9 (6.2) | |
| ≥50 | 35 (13.7) | 13 (9.0) | |
| Index behavior | |||
| Suicidal ideation | 101 (39.6) | 106 (73.6) | <.001 |
| Suicide attempt | 28 (11.0) | 22 (15.3) | .17 |
| Self-injury | 10 (3.9) | 9 (6.2) | .39 |
| Binge substance use | 134 (52.5) | 28 (19.4) | <.001 |
| Risk classification based on model | |||
| Low | 220 (86.3) | 122 (84.7) | .48 |
| High | 35 (13.7) | 22 (15.3) | |
| Subsequent suicide-related events | |||
| No | 153 (60.0) | 108 (75.0) | .008 |
| Yes | 102 (40.0) | 36 (25.0) | |
January 1, 2017, to February 29, 2020, prior to active notifications based on the risk model.
March 1, 2020, to August 31, 2022, after implementation of active notifications based on the risk model.
Aim 1 Results
A total of 255 individuals in cohort 1 had index events between January 1, 2017, and February 29, 2020. Most index events (134 [52.5%]) were for binge substance use, followed by 101 (39.6%) for suicidal ideation, 28 (11.0%) for a suicide attempt, and 10 (3.9%) for self-injury. Among these individuals, 102 (39.5%) had subsequent suicidal behaviors. In cohort 1, the majority (220 [86.3%]) were classified as low risk, while 35 individuals (13.3%) were classified as high risk for suicidal attempt or death in the 12 months after their index event (Table 1).
Table 2 shows the odds of subsequent suicidal behaviors, comparing individuals classified as high and low risk while controlling for the time under observation in cohort 1. Overall, those classified as high risk had a 65.7% (95% CI, 48.4%-82.9%) chance of a future suicide-related event compared with those classified as low risk, who had a 35.5% (95% CI, 29.0%-41.9%) chance. This difference represents a greater odds of subsequent suicide-related events among those classified as high risk compared with those classified as low risk (OR, 3.47; 95% CI, 1.53-7.86; P = .003; AUC, 0.65). To further add context to this finding, after controlling for time under study, those with an index suicide attempt were no more likely to have subsequent suicide-related events compared with those with other index behaviors (OR, 1.86; 95% CI, 0.81-4.28; P = .18; AUC not applicable).
Table 2. Odds of Repeated Suicidal Risk Event After Index Event in Cohort 1a.
| Odds ratio (95% CI) | P value | |
|---|---|---|
| Risk status classified by algorithm | ||
| Low risk at index event | 1 [Reference] | .003 |
| High risk at index event | 3.47 (1.53-7.86) | |
| Index behavior | ||
| Suicidal ideation, binge substance use, or self-injury | 1 [Reference] | .13 |
| Suicide attempt | 1.91 (0.83-4.37) |
January 1, 2017, to February 29, 2020, prior to active notifications based on risk model. Controlled for time under observation.
Aim 2 Results
For aim 2, we focused on the 57 individuals classified as high risk across both cohorts, comparing the time when alerts were active (22 individuals [38.6%]; March 1, 2020, to August 31, 2022) with when they were inactive (35 individuals [61.4%]; January 1, 2017, to February 29, 2020). When alerts were active, there were only 6 individuals (27.3%) classified as high risk who had a subsequent suicide-related event compared with 24 (68.6%) when alerts were inactive. Overall, when alerts were inactive, high-risk individuals were more likely to have subsequent suicidal behaviors compared with when alerts were active (OR, 9.14; 95% CI, 1.85-45.29; P = .007) (Table 3) (power analyses for this analysis are provided in the eMethods in Supplement 1). Among high-risk individuals, during the time when alerts were inactive, the probability of a future suicide-related event was 74.0% (95% CI, 57.0%-91.1%), while when alerts were active, the probability was 23.8% (95% CI, 3.1%-44.4%). After removing the 33 events that occurred during the strictest COVID-19 lockdowns, individuals classified as high risk were still more likely to have a subsequent suicidal behavior when alerts were inactive compared with when alerts were active (OR, 5.47; 1.47-19.9; P = .01).
Table 3. Odds of Repeated Suicidal Behaviors and Receiving Follow-up Care Comparing the Inactive Period With the Period of Active Alerts Among 57 High-Risk Casesa.
| Odds ratio (95% CI) | P value | |
|---|---|---|
| Repeated suicidal behaviors | ||
| Alert activeb | 1 [Reference] | .007 |
| Alert inactivec | 9.14 (1.85-45.29) | |
| Receiving follow-up care | ||
| Alert activeb | 1 [Reference] | .005 |
| Alert inactivec | 48.60 (3.24-727.59) |
Controlled for time under observation, age, and type of index behavior.
March 1, 2020, to August 31, 2022.
January 1, 2017, to February 29, 2020.
We also found that prior to the alerts being active, only 1 of 35 individuals classified as high risk (2.9%) received a wellness check in addition to their initial case management visit. After the alerts were activated in conjunction with the biweekly meetings, 11 of 22 individuals classified as high risk (50.0%) received 1 or more wellness checks in addition to their initial Pink Form case management visit (2-3 BCIs per person total). Further, in our multivariate logistic regression, the odds of receiving a follow-up wellness check were significantly greater (OR, 48.60; 95% CI, 3.24-727.59; P = .005) when alerts were active compared with when they were inactive (Table 3). Prior to implementation, high-risk individuals had a 2.5% (95% CI, 0.0%-8.0%) chance of receiving additional wellness checks, while after implementation, the probability of additional wellness checks increased to 55.9% (95% CI, 28.1%-83.6%).
Discussion
We evaluated a statistical model for suicide risk identification and a linked care strategy aimed at reducing risk and improving the reach of evidence-based care in a Native American community setting. Findings from the prospective validation analysis suggest that individuals who were classified as high risk were 3.5 times more likely to experience subsequent suicidal behaviors compared with those classified as low risk. After implementing the risk identification model in practice, we also found that the risk of subsequent suicidal behaviors was significantly lower than prior to the model being implemented and that of BCIs were associated with higher rates of wellness checks (50% vs 2.9%) among high-risk individuals. Taken together, the preliminary evidence suggests that this model adds to our ability to accurately classify individuals at highest risk of suicide and is associated with reduced subsequent suicide risk and increased care for those at the highest risk.
To our knowledge, this is the first study to prospectively validate a machine learning–based risk algorithm and associated care model and evaluate its impact within a Native American community. Walsh and colleagues33 evaluated the performance of a suicide attempt prognostic model in an urban clinical care setting. Shaw et al34 applied the Mental Health Research Network (MHRN) model to electronic health records of Alaska Native primary care patients and found good performance, with an AUC of 0.83. However, a recent analysis of the MHRN model performance disaggregated by race suggested the model underperformed among Black and Native American patients compared with White, Hispanic, and Asian patients.35 Lower accuracy may translate into reduced care and contribute to further exacerbation of existing disparities. Our study is, to our knowledge, the first to prospectively validate a model developed specifically with Native American populations, with preliminary findings suggesting that the model continued to enhance our ability to detect those at elevated risk of suicide. Future research should continue to explore whether existing models are applicable in population subgroups and/or whether new models need to be developed.
For the prospective validation phase of our study in cohort 1, we found that those identified as high risk in our model had 3 times greater odds of having a subsequent suicide-related event compared with those classified as low risk. However, the AUC for the model was only 0.65, indicating only marginal discrimination. Small sample sizes may have also contributed to the low AUC. However, of note, the risk model substantially outperformed the alternative method based on the type of index behavior.
Most existing literature focuses on the development of predictive algorithms, with little attention paid to how to implement them in practice. A notable exception is the US Department of Veterans Affairs’ Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment (REACH-VET) program.36 A recent study evaluating the impact of REACH-VET indicated that the implementation of this program was associated with greater engagement and fewer mental health admissions, emergency department visits, and suicide attempts.37 These findings are consistent with our preliminary findings for NATIVE-RISE.
Given that our approach is centered on tribal data, it is vital to highlight the power of Indigenous data sovereignty.38 Using classification algorithms and connections to care for community members who are experiencing suicide-related behaviors is an approach that can be shared with other organizations serving Indigenous communities. In line with community-based participatory research, we have shared this work with our community partners and additional partners who are part of the Southwest Hub for American Indian Youth Suicide Prevention.
Limitations
Limitations of this work include small sample sizes, reliance on programmatic data, the lack of an experimental design, and findings that may not generalize broadly. First, our sample sizes were small, and further research is needed to confirm accuracy. Second, the analysis leveraged routinely collected programmatic data that rely on case manager documentation. Case managers have multiple responsibilities. Documentation likely undercounts care received by individuals. The observational nature of the study was particularly a concern when COVID-19 first arrived in the community, but the sensitivity analysis still supports our findings. In addition, our research focused on identifying individuals in a population that has already been identified as high risk for suicide within a tribal community that has prioritized suicide prevention locally. Results cannot generalize to all Native American individuals nor to other populations, such as patients seen in emergency departments.
Conclusions
This prognostic study showed that implementation of a statistical model and associated care system developed in partnership with the community-enhanced identification of Native American individuals at high risk for suicide was associated with reduced risk for subsequent suicidal behaviors and increased reach of care. This study was conducted using community-based participatory research, building on a long tribal-academic research partnership. Future work should continue to explore whether specific models are needed for certain subgroups of the population and focus on how best to implement prognostic models to enhance care for those most in need.
eMethods. Pre and Post Hoc Power Calculations
eFigure. Survival Analysis for Time to Subsequent Suicidal Events Comparing Time Periods Before the Alerts Were Active to the Period After Alerts Were Active
Data Sharing Statement
References
- 1.Hedegaard H, Curtin SC, Warner M. Suicide mortality in the United States, 1999-2019. National Center for Health Statistics data brief No. 398. February 2021. Accessed March 22, 2022. https://stacks.cdc.gov/view/cdc/101761 [Google Scholar]
- 2.Suicide Prevention Resource Center. Racial and ethnic disparities . Accessed January 12, 2021. https://sprc.org/scope/racial-ethnic-disparities
- 3.CDC WONDER . Centers for Disease Control and Prevention. Accessed January 6, 2021. https://wonder.cdc.gov/
- 4.Arias E, Schauman WS, Eschbach K, Sorlie PD, Backlund E. The validity of race and Hispanic origin reporting on death certificates in the United States. Vital Health Stat 2. 2008;(148):1-23. [PubMed] [Google Scholar]
- 5.Cwik MF, Tingey L, Maschino A, et al. Decreases in suicide deaths and attempts linked to the White Mountain Apache Suicide Surveillance and Prevention System, 2001-2012. Am J Public Health. 2016;106(12):2183-2189. doi: 10.2105/AJPH.2016.303453 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Franklin JC, Ribeiro JD, Fox KR, et al. Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol Bull. 2017;143(2):187-232. doi: 10.1037/bul0000084 [DOI] [PubMed] [Google Scholar]
- 7.Nock MK, Park JM, Finn CT, Deliberto TL, Dour HJ, Banaji MR. Measuring the suicidal mind: implicit cognition predicts suicidal behavior. Psychol Sci. 2010;21(4):511-517. doi: 10.1177/0956797610364762 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.King CA, Berona J, Czyz E, Horwitz AG, Gipson PY. Identifying adolescents at highly elevated risk for suicidal behavior in the emergency department. J Child Adolesc Psychopharmacol. 2015;25(2):100-108. doi: 10.1089/cap.2014.0049 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Hughes DH. Can the clinician predict suicide? Psychiatr Serv. 1995;46(5):449-451. doi: 10.1176/ps.46.5.449 [DOI] [PubMed] [Google Scholar]
- 10.Large M, Sharma S, Cannon E, Ryan C, Nielssen O. Risk factors for suicide within a year of discharge from psychiatric hospital: a systematic meta-analysis. Aust N Z J Psychiatry. 2011;45(8):619-628. doi: 10.3109/00048674.2011.590465 [DOI] [PubMed] [Google Scholar]
- 11.DeVylder JE, Ryan TC, Cwik M, et al. Assessment of selective and universal screening for suicide risk in a pediatric emergency department. JAMA Netw Open. 2019;2(10):e1914070. doi: 10.1001/jamanetworkopen.2019.14070 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Horowitz LM, Bridge JA, Teach SJ, et al. Ask Suicide-Screening Questions (ASQ): a brief instrument for the pediatric emergency department. Arch Pediatr Adolesc Med. 2012;166(12):1170-1176. doi: 10.1001/archpediatrics.2012.1276 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Ballard ED, Cwik M, Van Eck K, et al. Identification of at-risk youth by suicide screening in a pediatric emergency department. Prev Sci. 2017;18(2):174-182. doi: 10.1007/s11121-016-0717-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Posner K, Oquendo MA, Gould M, Stanley B, Davies M. Columbia Classification Algorithm of Suicide Assessment (C-CASA): classification of suicidal events in the FDA’s pediatric suicidal risk analysis of antidepressants. Am J Psychiatry. 2007;164(7):1035-1043. doi: 10.1176/ajp.2007.164.7.1035 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Rozel JS. Broken promise: challenges in achieving effective universal suicide screening. Acad Emerg Med. 2021;28(6):705-706. Published online December 22, 2020. doi: 10.1111/acem.14199 [DOI] [PubMed] [Google Scholar]
- 16.Simpson SA, Goans C, Loh R, Ryall K, Middleton MCA, Dalton A. Suicidal ideation is insensitive to suicide risk after emergency department discharge: performance characteristics of the Columbia-Suicide Severity Rating Scale Screener. Acad Emerg Med. 2021;28(6):621-629. doi: 10.1111/acem.14198 [DOI] [PubMed] [Google Scholar]
- 17.Carter G, Milner A, McGill K, Pirkis J, Kapur N, Spittal MJ. Predicting suicidal behaviours using clinical instruments: systematic review and meta-analysis of positive predictive values for risk scales. Br J Psychiatry. 2017;210(6):387-395. doi: 10.1192/bjp.bp.116.182717 [DOI] [PubMed] [Google Scholar]
- 18.Bernert RA, Hilberg AM, Melia R, Kim JP, Shah NH, Abnousi F. Artificial intelligence and suicide prevention: a systematic review of machine learning investigations. Int J Environ Res Public Health. 2020;17(16):5929. doi: 10.3390/ijerph17165929 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Gordon JA, Avenevoli S, Pearson JL. Suicide prevention research priorities in health care. JAMA Psychiatry. 2020;77(9):885-886. doi: 10.1001/jamapsychiatry.2020.1042 [DOI] [PubMed] [Google Scholar]
- 20.Reger GM, McClure ML, Ruskin D, Carter SP, Reger MA. Integrating predictive modeling into mental health care: an example in suicide prevention. Psychiatr Serv. 2019;70(1):71-74. doi: 10.1176/appi.ps.201800242 [DOI] [PubMed] [Google Scholar]
- 21.Tucker RP, Tackett MJ, Glickman D, Reger MA. Ethical and practical considerations in the use of a predictive model to trigger suicide prevention interventions in healthcare settings. Suicide Life Threat Behav. 2019;49(2):382-392. doi: 10.1111/sltb.12431 [DOI] [PubMed] [Google Scholar]
- 22.Rajkomar A, Hardt M, Howell MD, Corrado G, Chin MH. Ensuring fairness in machine learning to advance health equity. Ann Intern Med. 2018;169(12):866-872. doi: 10.7326/M18-1990 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Brockie TN, Dana-Sacco G, Wallen GR, Wilcox HC, Campbell JC. The relationship of adverse childhood experiences to PTSD, depression, poly-drug use and suicide attempt in reservation-based Native American adolescents and young adults. Am J Community Psychol. 2015;55(3-4):411-421. doi: 10.1007/s10464-015-9721-3 [DOI] [PubMed] [Google Scholar]
- 24.Wexler L, Chandler M, Gone JP, et al. Advancing suicide prevention research with rural American Indian and Alaska Native populations. Am J Public Health. 2015;105(5):891-899. doi: 10.2105/AJPH.2014.302517 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.O’Keefe VM, Tucker RP, Cole AB, Hollingsworth DW, Wingate LR. Understanding indigenous suicide through a theoretical lens: a review of general, culturally-based, and indigenous frameworks. Transcult Psychiatry. 2018;55(6):775-799. doi: 10.1177/1363461518778937 [DOI] [PubMed] [Google Scholar]
- 26.Gone JP, Hartmann WE, Pomerville A, Wendt DC, Klem SH, Burrage RL. The impact of historical trauma on health outcomes for indigenous populations in the USA and Canada: a systematic review. Am Psychol. 2019;74(1):20-35. doi: 10.1037/amp0000338 [DOI] [PubMed] [Google Scholar]
- 27.Payne HE, Steele M, Bingham JL, Sloan CD. Identifying and reducing disparities in mental health outcomes among American Indians and Alaskan Natives using public health, mental healthcare and legal perspectives. Adm Policy Ment Health. 2018;45(1):5-14. doi: 10.1007/s10488-016-0777-7 [DOI] [PubMed] [Google Scholar]
- 28.Haroz EE, Walsh CG, Goklish N, Cwik MF, O’Keefe V, Barlow A. Reaching those at highest risk for suicide: development of a model using machine learning methods for use with Native American communities. Suicide Life Threat Behav. 2020;50(2):422-436. doi: 10.1111/sltb.12598 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med. 2015;13:1. doi: 10.1186/s12916-014-0241-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Cwik MF, Barlow A, Goklish N, et al. Community-based surveillance and case management for suicide prevention: an American Indian tribally initiated system. Am J Public Health. 2014;104(suppl 3):e18-e23. doi: 10.2105/AJPH.2014.301872 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Haroz EE, Grubin F, Goklish N, et al. Designing a clinical decision support tool that leverages machine learning for suicide risk prediction: development study in partnership with Native American care providers. JMIR Public Health Surveill. 2021;7(9):e24377. doi: 10.2196/24377 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Stata Statistical Software. Computer program. Release 13.0. StataCorp LP. 2013.
- 33.Walsh CG, Johnson KB, Ripperger M, et al. Prospective validation of an electronic health record-based, real-time suicide risk model. JAMA Netw Open. 2021;4(3):e211428. doi: 10.1001/jamanetworkopen.2021.1428 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Shaw JL, Beans JA, Noonan C, et al. Validating a predictive algorithm for suicide risk with Alaska Native populations. Suicide Life Threat Behav. 2022;52(4):696-704. doi: 10.1111/sltb.12853 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Coley RY, Johnson E, Simon GE, Cruz M, Shortreed SM. Racial/ethnic disparities in the performance of prediction models for death by suicide after mental health visits. JAMA Psychiatry. 2021;78(7):726-734. doi: 10.1001/jamapsychiatry.2021.0493 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.McCarthy JF, Valenstein M, Kim HM, Ilgen M, Zivin K, Blow FC. Suicide mortality among patients receiving care in the Veterans Health Administration health system. Am J Epidemiol. 2009;169(8):1033-1038. doi: 10.1093/aje/kwp010 [DOI] [PubMed] [Google Scholar]
- 37.McCarthy JF, Cooper SA, Dent KR, et al. Evaluation of the Recovery Engagement and Coordination for Health–Veterans Enhanced Treatment Suicide Risk Modeling Clinical Program in the Veterans Health Administration. JAMA Netw Open. 2021;4(10):e2129900. doi: 10.1001/jamanetworkopen.2021.29900 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.US Indigenous Data Sovereignty Network . Accessed June 6, 2022. https://www.publichistoryproject.org/2021/11/04/united-states-indigenous-data-sovereignty-network/
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eMethods. Pre and Post Hoc Power Calculations
eFigure. Survival Analysis for Time to Subsequent Suicidal Events Comparing Time Periods Before the Alerts Were Active to the Period After Alerts Were Active
Data Sharing Statement