Racial/Ethnic Disparities in the Performance of Prediction Models for Death by Suicide After Mental Health Visits

R Yates Coley; Eric Johnson; Gregory E Simon; Maricela Cruz; Susan M Shortreed

doi:10.1001/jamapsychiatry.2021.0493

. 2021 Apr 28;78(7):1–9. doi: 10.1001/jamapsychiatry.2021.0493

Racial/Ethnic Disparities in the Performance of Prediction Models for Death by Suicide After Mental Health Visits

R Yates Coley ^1,^2,^✉, Eric Johnson ¹, Gregory E Simon ¹, Maricela Cruz ¹, Susan M Shortreed ^1,²

¹Kaiser Permanente Washington Health Research Institute, Seattle

²Department of Biostatistics, University of Washington School of Public Health, Seattle

Accepted for Publication: February 22, 2021.

Published Online: April 28, 2021. doi:10.1001/jamapsychiatry.2021.0493

^✉

Corresponding Author: R. Yates Coley, PhD, Kaiser Permanente Washington Health Research Institute, 1730 Minor Ave, Ste 1600, Seattle, WA 98101 (rebecca.y.coley@kp.org).

Author Contributions: Dr Coley had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Coley, Simon, Cruz, Shortreed.

Acquisition, analysis, or interpretation of data: Coley, Johnson, Simon, Shortreed.

Drafting of the manuscript: Coley, Shortreed.

Critical revision of the manuscript for important intellectual content: All authors.

Statistical analysis: Coley, Johnson, Cruz, Shortreed.

Obtained funding: Coley.

Administrative, technical, or material support: Shortreed.

Supervision: Coley, Shortreed.

Conflict of Interest Disclosures: Dr Shortreed reported having received grants from Syneos Health, which represented a consortium of pharmaceutical companies carrying out US Food and Drug Administration–mandated studies regarding the safety of extended-release opioids outside the submitted work. No other disclosures were reported.

Funding/Support: Research reported in this work was supported by the Mental Health Research Network with funding from grants U19 MH092201 and U19 MH121738 from the National Institute of Mental Health. Dr Coley is also supported by grant K12HS026369 from the Agency for Healthcare Research and Quality.

Role of the Funder/Sponsor: The funding sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Mental Health or the Agency for Healthcare Research and Quality.

Additional Contributions: Theresa Nguyen, LCSW, Chief Program Officer at Mental Health America, provided thoughtful feedback on the draft manuscript. Compensation was provided to Mental Health America for her time. She was not personally or directly compensated.

^✉

Corresponding author.

PMCID: PMC8082428 PMID: 33909019

Key Points

Question

Could implementation of suicide prediction models reinforce and worsen racial/ethnic disparities in care?

Findings

In this diagnostic/prognostic study, 2 prediction models for suicide within 90 days were developed and validated in a retrospective study of 13 980 570 outpatient mental health visits. Both models accurately predicted suicide risk for visits by White, Hispanic, and Asian patients, but performance was poor for visits by Black and American Indian/Alaskan Native patients and patients without race/ethnicity reported.

Meaning

This study suggests that implementation of either suicide prediction model would disproportionately benefit White, Hispanic, and Asian patients compared with Black and American Indian/Alaskan Native patients and patients with unrecorded race/ ethnicity.

Abstract

Importance

Clinical prediction models estimated with health records data may perpetuate inequities.

Objective

To evaluate racial/ethnic differences in the performance of statistical models that predict suicide.

Design, Setting, and Participants

In this diagnostic/prognostic study, performed from January 1, 2009, to September 30, 2017, with follow-up through December 31, 2017, all outpatient mental health visits to 7 large integrated health care systems by patients 13 years or older were evaluated. Prediction models were estimated using logistic regression with LASSO variable selection and random forest in a training set that contained all visits from a 50% random sample of patients (6 984 184 visits). Performance was evaluated in the remaining 6 996 386 visits, including visits from White (4 031 135 visits), Hispanic (1 664 166 visits), Black (578 508 visits), Asian (313 011 visits), and American Indian/Alaskan Native (48 025 visits) patients and patients without race/ethnicity recorded (274 702 visits). Data analysis was performed from January 1, 2019, to February 1, 2021.

Exposures

Demographic, diagnosis, prescription, and utilization variables and Patient Health Questionnaire 9 responses.

Main Outcomes and Measures

Suicide death in the 90 days after a visit.

Results

This study included 13 980 570 visits by 1 433 543 patients (64% female; mean [SD] age, 42 [18] years. A total of 768 suicide deaths were observed within 90 days after 3143 visits. Suicide rates were highest for visits by patients with no race/ethnicity recorded (n = 313 visits followed by suicide within 90 days, rate = 5.71 per 10 000 visits), followed by visits by Asian (n = 187 visits followed by suicide within 90 days, rate = 2.99 per 10 000 visits), White (n = 2134 visits followed by suicide within 90 days, rate = 2.65 per 10 000 visits), American Indian/Alaskan Native (n = 21 visits followed by suicide within 90 days, rate = 2.18 per 10 000 visits), Hispanic (n = 392 visits followed by suicide within 90 days, rate = 1.18 per 10 000 visits), and Black (n = 65 visits followed by suicide within 90 days, rate = 0.56 per 10 000 visits) patients. The area under the curve (AUC) and sensitivity of both models were high for White, Hispanic, and Asian patients and poor for Black and American Indian/Alaskan Native patients and patients without race/ethnicity recorded. For example, the AUC for the logistic regression model was 0.828 (95% CI, 0.815-0.840) for White patients compared with 0.640 (95% CI, 0.598-0.681) for patients with unrecorded race/ethnicity and 0.599 (95% CI, 0.513-0.686) for American Indian/Alaskan Native patients. Sensitivity at the 90th percentile was 62.2% (95% CI, 59.2%-65.0%) for White patients compared with 27.5% (95% CI, 21.0%-34.7%) for patients with unrecorded race/ethnicity and 10.0% (95% CI, 0%-23.0%) for Black patients. Results were similar for random forest models, with an AUC of 0.812 (95% CI, 0.800-0.826) for White patients compared with 0.676 (95% CI, 0.638-0.714) for patients with unrecorded race/ethnicity and 0.642 (95% CI, 0.579-0.710) for American Indian/Alaskan Native patients and sensitivities at the 90th percentile of 52.8% (95% CI, 50.0%-55.8%) for White patients, 29.3% (95% CI, 22.8%-36.5%) for patients with unrecorded race/ethnicity, and 6.7% (95% CI, 0%-16.7%) for Black patients.

Conclusions and Relevance

These suicide prediction models may provide fewer benefits and more potential harms to American Indian/Alaskan Native or Black patients or those with undrecorded race/ethnicity compared with White, Hispanic, and Asian patients. Improving predictive performance in disadvantaged populations should be prioritized to improve, rather than exacerbate, health disparities.

This diagnostic/prognostic study of outpatient mental health visits evaluates racial/ethnic differences in the performance of statistical models that predict suicide.

Introduction

Concern is increasing that the use of prediction models in health care will exacerbate existing disparities.^1,2 Clinical prediction models are frequently estimated using health records data, which reflect inequities in health care access, quality, and outcomes based on race/ethnicity, sexual orientation, gender identity, immigration status, and other aspects of patient identity. Given existing disparities, the presumption that historical care patterns reflect clinical need can perpetuate inequities.^3,4 Obermeyer et al⁵ investigated an algorithm to identify patients with complex care needs that used health care spending as a proxy for disease severity and found that it underestimated disease severity for Black patients, resulting in fewer services offered to Black patients and exacerbating inequitable allocation of health care resources. Barriers to accessing health care also impact the availability of data for model estimation. Prediction models will be more accurate for populations with better access to care.⁶

Prediction models that perform poorly within subpopulations may expose patients inaccurately identified as high risk to unnecessary or inappropriate care. One strategy to improve a prediction model’s true-positive rate within a subpopulation is to lower the threshold for intervention.³ However, if risk discrimination is poor, this strategy will also increase false-positive results. Costs, burdens, or adverse effects of overtreatment may also vary across populations. Marginalized communities that have historically received more invasive or intrusive interventions could be disproportionately harmed.

Suicide risk prediction models have been developed and validated in several settings and are now in regular use at the Veterans Health Administration, HealthPartners, and Kaiser Permanente.^7,8,9,10 Existing suicide prediction models are accurate on the population level, yet performance within subpopulations, including racial/ethnic subgroups, remains unexamined. The current study has several motivations for examining performance across race/ethnicity.¹¹ First, suicide mortality rates vary widely. Some disadvantaged populations have markedly lower rates—suicide death rates for Black, Hispanic, Asian, and Pacific Islander populations range from 6.7 to 6.9 per 100 000 people, whereas American Indian and Alaskan Native populations have a suicide death rate of 22.1 per 100 000 people.¹² Second, racial/ethnic inequities in access to high-quality care are particularly pronounced in mental health.¹³ Consequently, mental health diagnosis and treatment patterns (key predictors of suicidal behavior) vary by race/ethnicity because of persistent disparities in care received and subsequent differences in patient preferences.^{13,14,15,16,17,18,19,20,21} Third, suicide death is rare; the number of suicide deaths observed even in large data sets may be small for low-prevalence racial/ethnic groups, resulting in insufficient information available to identify race/ethnicity-specific risk factors and precluding estimation of separate prediction models. Fourth, potential harms of both false-negative errors (failure to intervene before suicide) and false-positive errors (delivery of unnecessary or potentially intrusive interventions) may vary across racial/ethnic groups. In this study, we evaluated racial/ethnic disparities in the performance of a risk algorithm trained on health record data to predict suicide death in the 90 days after an outpatient mental health visit.

Methods

Study Setting and Data Sources

Seven health systems (HealthPartners; Henry Ford Health System; and the Colorado, Hawaii, Northwest, Southern California, and Washington regions of Kaiser Permanente) participated in this diagnostic/prognostic study. Each system provides comprehensive mental health and general medical care to a defined population, has complete capture of utilization via electronic health record and insurance claims data, and maintains a research data warehouse that conforms to the Health Care Systems Research Network’s Virtual Data Warehouse model.²² Responsible institutional review boards for each health care system granted waivers of consent to use deidentified records data for this research. This study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline.

Participants and Examinations

Our study sample of 1 433 543 patients 13 years or older covered all 13 980 570 outpatient visits to a mental health specialist between January 1, 2009, and September 30, 2017 (or latest date cause-of-death data were available), with follow-up through December 31, 2017 (eTable 1 in the Supplement). Data analysis was performed from January 1, 2019, to February 1, 2021.

Measures and Definitions

Suicide predictors were extracted from health record and insurance billing information.²² Predictors included demographic characteristics (age, sex, race, ethnicity, and insurance type), comorbidities (measured by Charlson Comorbidity Index²³), mental health and substance use diagnoses, dispensed psychiatric medications, prior suicide attempts, prior mental health encounters (including emergency department and hospitalizations), and Patient Health Questionnaire 9 (PHQ-9) responses²⁴ (eTable 2 in the Supplement). Race and ethnicity were included as separate predictors. Clinical predictors were binary, indicating the presence of each diagnosis, prescription, or encounter in 3 overlapping periods before the visit: 90 days, 1 year, and 5 years. The PHQ-9 responses were summarized for the visit day and for the 90, 183, and 365 days before the visit. Separate predictors were defined for the first 8 items of the PHQ-9, a scale of depression symptom severity, and the ninth item, which specifically asks about suicidal ideation.^24,25

Race/ethnicity categories included White, Hispanic, Black, Asian, American Indian/Alaskan Native, Native Hawaiian/Pacific Islander, multiracial or other race, and unrecorded. Patients with Hispanic ethnicity were classified as such regardless of race for primary analyses; sensitivity analyses examined racial categories including Hispanic patients. Race/ethnicity data were routinely collected by self-report at clinic visits. Thus, race/ethnicity designations do not reflect biological or genetic underpinnings but instead represent patient-identified race/ethnicity in a health care setting. Patients may not have race/ethnicity recorded if they had fewer encounters during which this information was collected, they did not identify with any categories offered, or clinic staff did not inquire; available clinical data do not indicate why race/ethnicity data may be absent. Our analysis included patients with unrecorded race/ethnicity because they represent 6.1% of patients and 9.8% of suicides in the sample.

Suicide death within 90 days after a visit was defined by an International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) cause-of-death code for self-inflicted injury (codes X60–X84) or injury or poisoning with undetermined intent (codes Y10–Y34) in state mortality records.^26,27 Additional information on the data collection methods is in the eMethods in the Supplement.

Statistical Analysis

The training set for model development contained all visits for a random sample of 50% of patients stratified by race/ethnicity and suicide death; visits for the remaining 50% of patients constituted the validation set.

Prediction models were estimated using logistic regression with LASSO variable selection²⁸ and random forest.²⁹ Logistic regression models considered prespecified interactions between predictors (including prior diagnoses, suicide attempts, and PHQ-9 responses) and race and ethnicity (eTable 2 in the Supplement). Random forest is a tree-based method that explores interactions between predictors (including those with race and ethnicity) in estimating probability of an outcome. Event rates were too low to estimate separate models for each racial/ethnic subgroup.

Tuning parameter values for prediction models were selected using 10-fold cross-validation in the training data set (eTable 3 in the Supplement). Optimal values for each model corresponded to the highest area under the curve (AUC).³⁰ Prediction models were estimated in the training set using selected tuning parameters and used to obtain predictions in the validation data set.

We evaluated prediction models’ performance in the overall validation set and within subgroups defined by race/ethnicity. The AUC measured model discrimination. Sensitivity (true-positive rate) was estimated for global and race/ethnicity-specific thresholds defined at the 99th, 95th, 90th, and 75th percentiles of predictions in the training data set. Receiver operator characteristic curves were examined for racial/ethnic subgroups.

We also evaluated positive predictive value (PPV), a popular performance measure, because it incorporates true- and false-positive results, and the Brier score, which evaluates the accuracy of predictions. However, because both measures are influenced by event rate, neither was informative for comparing predictive accuracy across racial/ethnic subgroups given the variability in suicide rates. Additional details on this and other analytic decisions are in the eMethods and eTables 4 and 5 in the Supplement.

Results

Visit Characteristics

Our analysis included 13 980 570 visits by 1 433 543 patients. Most visits were made by White patients (58% of visits), followed by patients who were Hispanic (24% of visits), Black (8% of visits), Asian (4% of visits), Native Hawaiian/Pacific Islander (1% of visits), and American Indian/Alaskan Native (0.7% of visits) (Table 1; eTable 6 in the Supplement). Patients without race or Hispanic ethnicity recorded made up 4% of visits, and 0.3% of visits were made by patients recorded as multiracial or other race. Prior suicide attempts and inpatient hospitalizations with mental health diagnosis were most common among visits for American Indian/Alaskan Native patients (5% had attempts and 26% had hospitalizations) and White patients (4% had attempts and 26% had hospitalizations) and least common for patients without race/ethnicity recorded (2% had attempts and 11% had hospitalizations).

Table 1. Characteristics of Visits to Mental Health Specialists, 2009-2017.

Characteristic	No. (%) of visits
Characteristic	All (N = 13 980 570 visits)	White (n = 8 050 670 visits)^a	Hispanic (n = 3 326 284 visits)	Black (n = 1 159 557 visits)^a	Asian (n = 625 076 visits)^a	Native Hawaiian/Pacific Islander (n = 134 568 visits)^a	American Indian/Alaskan Native (n = 96 376 visits)^a	Multiple or other (n = 39 512 visits)^a	Not recorded (n = 548 527 visits)^a
Female	8 882 457 (64)	5 077 317 (63)	2 138 020 (64)	401 796 (64)	806 083 (70)	80 422 (60)	65 778 (68)	23 559 (60)	289 482 (53)
Age, y
13-17	1 536 021 (11)	716 425 (9)	530 244 (16)	68 768 (11)	106 984 (9)	21 540 (16)	11 521 (12)	6298 (16)	74 241 (14)
18-29	2 335 568 (17)	1 207 137 (15)	631 364 (19)	128 407 (21)	172 298 (15)	28 402 (21)	16 128 (17)	8135 (21)	143 697 (26)
30-44	3 557 819 (25)	1 847 430 (23)	989 044 (30)	182 963 (29)	300 149 (26)	37 792 (28)	24 379 (25)	11 855 (30)	164 207 (30)
45-64	4 963 692 (36)	3 130 525 (39)	948 341 (29)	186 090 (30)	466 181 (40)	38 942 (29)	35 917 (37)	10 675 (27)	147 021 (27)
≥65	1 587 470 (11)	1 149 153 (14)	227 291 (7)	58 848 (9)	113 945 (10)	7892 (6)	8431 (9)	2549 (6)	19 361 (4)
Medicaid insurance	870 831 (6)	354 792 (4)	310 834 (9)	44 273 (7)	104 247 (9)	27 200 (20)	9190 (10)	1898 (5)	18 397 (3)
PHQ-9 score recorded
At index visit	1 854 999 (13)	1 297 144 (16)	268 440 (8)	68 279 (11)	107 697 (9)	13 364 (10)	22 363 (23)	15 447 (39)	62 265 (11)
In past year	3 078 551 (22)	2 088 246 (26)	505 805 (15)	115 239 (18)	198 113 (17)	22 048 (16)	33 639 (35)	20 096 (51)	95 365 (17)
Diagnoses in past 5 y, including index visit
Depression	10 581 636 (76)	6 244 319 (78)	2 452 437 (74)	452 808 (72)	871 383 (75)	93 422 (69)	77 001 (80)	28 517 (72)	361 749 (66)
Anxiety	10 171 431 (73)	6 027 832 (75)	2 366 612 (71)	437 380 (70)	799 234 (69)	87 615 (65)	73 701 (76)	28 315 (72)	350 742 (64)
Bipolar depression	1 872 136 (13)	1 260 085 (16)	315 184 (9)	65 061 (10)	136 542 (12)	13 375 (10)	17 147 (18)	5798 (15)	58 944 (11)
Schizophrenia	581 422 (4)	309 366 (4)	127 367 (4)	39 947 (6)	78 266 (7)	6114 (5)	3259 (3)	1451 (4)	15 652 (3)
Other mental health condition	795 662 (6)	455 002 (6)	171 253 (5)	45 540 (7)	86 362 (7)	8492 (6)	6188 (6)	2954 (7)	19 871 (4)
AUD	2 206 221 (16)	1 373 972 (17)	494 211 (15)	52 267 (8)	159 953 (14)	22 052 (16)	17 531 (18)	6146 (16)	80 089 (15)
SUD	2 342 644 (17)	1 425 876 (18)	526 218 (16)	68 970 (11)	182 241 (16)	29 747 (22)	18 969 (20)	5803 (15)	84 820 (15)
Suicide attempt	581 872 (4)	358 361 (4)	136 241 (4)	24 237 (4)	38 550 (3)	5055 (4)	5157 (5)	1615 (4)	12 656 (2)
Encounters in past 5 y with mental health diagnosis
Inpatient	3 300 322 (24)	2 097 858 (26)	690 629 (21)	126 008 (20)	266 709 (23)	26 425 (20)	25 008 (26)	8207 (21)	59 478 (11)
Outpatient	12 785 622 (91)	7 455 900 (93)	2 995 042 (90)	561 061 (90)	1 054 388 (91)	120 972 (90)	89 042 (92)	34 755 (88)	474 462 (86)
Emergency department	4 729 434 (34)	2 746 504 (34)	1 190 459 (36)	188 624 (30)	413 681 (36)	39 595 (29)	34 967 (36)	11 037 (28)	104 567 (19)

Open in a new tab

Abbreviations: AUD, alcohol use disorders; PHQ-9, Patient Health Questionnaire 9; SUD, substance use disorders.

^{^a}

Patients with Hispanic ethnicity were classified as such regardless of race and excluded from other racial categories; thus White refers to visits by White non-Hispanic patients, Black refers to visits by Black non-Hispanic patients, and so on.

Suicide Outcomes

A total of 768 suicide deaths, 722 indicated as self-inflicted and 46 with undetermined intent, were observed within 90 days after 3143 visits (eTable 7 in the Supplement). The number of suicides was greatest for White patients (542 deaths), followed by Hispanic patients (80 deaths), patients without race/ethnicity recorded (75 deaths), and Asian patients (37 deaths). Fewer than 20 suicides were observed among patients who were Black, Native Hawaiian/Pacific Islander, or American Indian/Alaskan Native or were reporting multiple or other races (eTable 8 in the Supplement). Many suicides were preceded by multiple visits in the prior 90 days (median, 2; interquartile range, 1-5 visits). Suicide rates were highest for visits by patients without race/ethnicity recorded (5.71 per 10 000 visits), followed by visits from patients who were Asian (2.99 per 10 000 visits), White (2.65 per 10 000 visits), multiracial or other race (2.28 per 10 000 visits), American Indian/Alaskan Native (2.18 per 10 000 visits), Native Hawaiian/Pacific Islander (1.63 per 10 000 visits), Hispanic (1.18 per 10 000 visits), and Black (0.56 per 10 000 visits).

Predictive Performance

Suicide predictions had high discrimination in the validation set for logistic regression and random forest models and variable performance across racial/ethnic subgroups (Table 2). The AUCs were highest for visits from White, Hispanic, and Asian patients and lowest for visits by Black and American Indian/Alaskan Native patients and those with unrecorded race/ethnicity. For example, the AUC of the logistic regression model was 0.828 (95% CI, 0.815-0.840) for White patients compared with 0.640 (95% CI, 0.598-0.681) for visits from patients without race/ethnicity data recorded.

Table 2. Areas Under the Curve (95% CIs), Overall and Within Racial/Ethnic Subgroups, of Suicide Prediction Models^a.

Race/ethnicity	Logistic regression with LASSO	Random forest
Entire validation set	0.822 (0.811-0.832)	0.816 (0.806-0.825)
White	0.828 (0.815-0.840)	0.812 (0.800-0.826)
Hispanic	0.855 (0.822-0.885)	0.831 (0.799-0.859)
Black	0.775 (0.687-0.850)	0.786 (0.715-0.846)
Asian	0.834 (0.796-0.870)	0.882 (0.853-0.910)
American Indian/Alaskan Native	0.599 (0.513-0.686)	0.642 (0.579-0.710)
Not recorded	0.640 (0.598-0.681)	0.676 (0.638-0.714)

Open in a new tab

^{^a}

The 95% CIs were obtained from 1000 visit–level bootstrap samples. Prediction model performance is not reported for visits made by patients who were Native Hawaiian/Pacific Islander, multiracial, or other race because there were fewer than 10 visits with a suicide death within 90 days in the validation data set (eTable 9 in the Supplement).

Table 3 compares the sensitivity of global and race/ethnicity-specific thresholds defined at the 99th, 95th, 90th, and 75th percentiles of risk predictions. Sensitivities of global thresholds vary widely across racial/ethnic subgroups. For example, the 95th percentile threshold identifies nearly half of visits preceding suicide for White patients, whereas for American Indian/Alaskan Native and Black patients, it is no better than chance (a random 5% sample of visits). Sensitivity at global thresholds was improved in the random forest model for Asian patients but decreased for White patients and remained similar for other subgroups.

Table 3. Sensitivity (95% CIs), Overall and Within Racial/Ethnic Subgroups, for Suicide Prediction Models^a.

Percentile	All (N = 1543)^b	White (n = 1092)	Hispanic (n = 144)	Black (n = 30)	Asian (n = 85)	American Indian/Alaskan Native (n = 15)	Not recorded (n = 167)
Logistic regression with LASSO: global thresholds^c
≥75%	75.0 (72.9-77.1)	78.8 (76.5-81.2)	73.6 (66.6-80.6)	63.3 (46.7-80.0)	80.0 (70.6-88.2)	26.7 (6.7-53.3)	56.9 (49.1-64.1)
≥90%	54.9 (52.4-57.3)	62.2 (59.2-65.0)	54.9 (47.2-63.2)	10.0 (0-23.3)	44.7 (34.1-56.5)	6.7 (0-20.0)	27.5 (21.0-34.7)
≥95%	41.1 (38.6-43.5)	46.8 (43.7-49.7)	36.8 (29.9-45.1)	6.7 (0-16.7)	31.8 (22.4-42.4)	6.7 (0-20.0)	23.4 (16.8-29.9)
≥99%	15.4 (13.6-17.2)	18.4 (16.2-20.5)	5.6 (2.1-9.7)	3.3 (0-10.0)	8.2 (2.4-14.1)	0 (0-0)	12.0 (7.2-17.4)
Random forest: global thresholds
≥75%	71.4 (69.3-73.6)	73.5 (70.9-76.4)	66.0 (58.3-73.6)	56.7 (40.0-73.3)	90.6 (84.7-96.5)	33.3 (13.2-53.3)	59.3 (52.1-66.5)
≥90%	48.6 (46.2-51.2)	52.8 (50.0-55.8)	47.2 (39.6-55.6)	6.7 (0-16.7)	61.2 (50.6-71.8)	6.7 (0-20.0)	29.3 (22.8-36.5)
≥95%	38.0 (35.6-40.2)	40.6 (37.5-43.6)	38.2 (31.2-46.5)	3.3 (0-10.0)	60.0 (49.4-70.6)	6.7 (0-20.0)	20.4 (14.4-26.3)
≥99%	12.6 (10.9-14.2)	15.0 (12.9-17.3)	5.6 (2.1-9.7)	3.3 (0-10.0)	4.7 (1.2-9.4)	6.7 (0-20.0)	9.0 (4.8-13.2)
Logistic regression with LASSO: race/ethnicity-specific thresholds^d
≥75%	NA	76.4 (73.6-78.9)	80.6 (73.6-86.8)	70.0 (53.3-86.7)	83.5 (75.3-90.6)	26.7 (6.7-53.3)	35.3 (28.1-43.1)
≥90%	NA	59.6 (56.9-62.5)	65.3 (57.6-72.9)	56.7 (40.0-73.3)	44.7 (34.1-55.3)	6.7 (0-20.0)	25.1 (18.6-32.3)
≥95%	NA	45.5 (42.6-48.5)	47.2 (38.9-54.9)	6.7 (0-16.7)	32.9 (22.4-43.5)	6.7 (0-20.0)	22.2 (15.6-29.3)
≥99%	NA	17.1 (14.9-19.3)	9.7 (4.9-14.6)	3.3 (0-10.0)	8.2 (3.5-14.1)	0 (0-0)	9.0 (4.8-13.8)
Random forest: race/ethnicity-specific thresholds
≥75%	NA	72.3 (69.9-74.7)	77.1 (70.1-83.3)	90.6 (84.7-96.5)	76.7 (60.0-90.0)	13.3 (0-33.3)	49.7 (41.9-57.5)
≥90%	NA	52.1 (49.1-54.8)	48.6 (41.0-56.9)	61.2 (50.6-71.8)	36.7 (20.0-53.3)	6.7 (0-20.0)	26.3 (20.3-32.9)
≥95%	NA	37.8 (34.8-40.8)	43.1 (34.7-51.4)	52.9 (42.3-62.4)	3.3 (0-10.0)	6.7 (0-20.0)	18.6 (12.6-24.6)
≥99%	NA	13.8 (11.8-15.9)	6.9 (2.8-11.1)	4.7 (1.2-9.4)	3.3 (0-10.0)	6.7 (0-20.0)	9.0 (4.8-13.8)

Open in a new tab

Abbreviation: NA, not applicable.

^{^a}

Sensitivities are given as percentages. The 95% CIs were obtained from 1000 visit–level bootstrap samples. Prediction model performance is not reported for visits made by patients who were Native Hawaiian/Pacific Islander, multiracial, or other race because there were fewer than 10 visits with a suicide death within 90 days in the validation data set (eTable 10 in the Supplement).

^{^b}

Number of visits with suicide death within 90 days in validation data.

^{^c}

Global thresholds defined at given percentiles of the distribution of all risk predictions in the training data set.

^{^d}

Race/ethnicity-specific thresholds defined at given percentiles of the distribution of risk predictions among all visits from patients of the specified race/ethnicity within the training data set.

Defining race/ethnicity-specific thresholds improved sensitivity inconsistently for subgroups with poor sensitivity using global thresholds. For example, defining risk thresholds separately for visits by Black patients increased sensitivity of the random forest model at all percentiles examined (Table 3). For visits by American Indian/Alaskan Native patients and patients without race/ethnicity data recorded, however, sensitivity was lower or unchanged.

The Figure shows receiver operating characteristic curves for suicide risk predictions in each racial/ethnic group. For both models, if subgroup-specific thresholds were selected to equalize sensitivity across race/ethnicity, the false-positive rate would vary widely. For example, at 40% sensitivity, the false-positive rate from the logistic regression model is 4% for White patients, 9% for Black patients, and 28% for patients without race/ethnicity data recorded (eTable 11 in the Supplement).

Sensitivity analyses that evaluated prediction model performance in racial/ethnic groups including visits by Hispanic patients found similar results to primary analyses (eTables 12, 13, and 14 in the Supplement).

Discussion

This diagnostic/prognostic study examined the performance of 2 prediction models for suicide death within 90 days after an outpatient mental health visit. Although both models performed well at the population level, prediction was poor within some racial/ethnic subgroups. Clinical implementation of either model to target suicide prevention would provide less potential benefit to patients who are Black, are American Indian/Alaskan Native, or have no race/ethnicity data recorded compared with White, Hispanic, and Asian patients because a smaller proportion of eventual suicides would be identified.

One strategy to increase sensitivity within racial/ethnic subgroups is to define separate thresholds for intervention based on the distribution of predictions within each group.³ In the current analysis, because risk discrimination was poor for visits by Black and American Indian/Alaskan Native patients and patients without race/ethnicity data recorded, improved sensitivity was accompanied by higher false-positive rates. To intervene on the same proportion of eventual suicides across racial/ethnic subgroups, the proportion of visits receiving intervention must be considerably higher for Black patients, American Indian/Alaskan Native patients, and patients without race/ethnicity data recorded, exposing low-risk individuals in these populations to unnecessary and possibly intrusive interventions.

Relative benefits and harms of suicide prevention interventions vary by race/ethnicity. Additional attention from a mental health care professional may increase access to beneficial services and likely presents limited harm but could cause stigmatization or discrimination and damage patient-practitioner therapeutic alliances, particularly for patients from marginalized communities already less likely to trust or engage with traditional mental health care.³¹ Identifying high-risk patients could also initiate a cascade of interventions, including involuntary psychiatric hospitalization and contact with law enforcement, which pose a greater risk of harm to populations collectively referred to as BIPOC (Black, Indigenous, and people of color).^32,33,34

Suicide prevention interventions should be studied to understand their benefits and harms, and these benefits and harms may differ by race/ethnicity. Preferences about interventions should be elicited from patients, and stakeholders from affected communities should be engaged in determining if, and how, to implement a prediction model. Statistical decisions about what threshold to use for recommending intervention and which performance measure is most important to examine are intricately linked with decisions about what types of intervention a prediction model should guide. For example, when the PPV of a prediction model is low in a particular subgroup, it is important that the accompanying intervention pose minimal potential harm to those with false-positive results.

Health records data may poorly predict suicide death in some racial/ethnic groups for several reasons. First, systemic barriers to accessing affordable, culturally competent mental health care reduce utilization,¹³ resulting in sparser records data for BIPOC populations. In this analysis, prior mental health diagnoses and encounters were lowest among patients without race/ethnicity data recorded—missing information was likely related to low utilization and fewer encounters during which clinic staff asked about race/ethnicity—and predictive accuracy was particularly low in this subgroup.

Second, BIPOC populations are less likely to be diagnosed and treated for mental health conditions because of practitioner bias^19,35 and institutionalized discrimination.^13,20,36 For example, BIPOC populations are less likely to be screened^37,38 or receive high-quality treatment¹³ for depression and more likely to experience discrimination in health care settings.¹⁵ Lacking access to culturally responsive and appropriate treatment, BIPOC populations may be more likely to prefer nonmedical treatment for mental health conditions.^14,15

Third, outcome misclassification could diminish performance of suicide prediction models in racial/ethnic subgroups.^39,40 Suicide deaths may be incorrectly identified as unintentional, accidental, or homicide and vice versa. Suicide misclassification rates may vary by race/ethnicity,⁴¹ with greater risk of misclassification in groups with higher rates of violent deaths now classified as accidents or assaults.

Fourth, the association of mental health conditions with suicidal behavior may vary by race/ethnicity because of cultural differences that influence suicide risk and protective factors.^42,43 Risk modification is typically accommodated by stratifying model estimation or by including many interactions. However, because suicide death is a rare event, statistical power to identify race/ethnicity-specific risk factors is often low. Even in this sample of more than 13 million mental health visits, including more than 1 million visits by Black patients and almost 100 000 visits by American Indian/Alaskan Native patients, there were not enough events to estimate separate prediction models for each racial/ethnic subgroup. This limited statistical power is exemplified by the finding that a random forest prediction model, which uses a data-driven estimation procedure to explore interactions, did not outperform a logistic regression model with prespecified interactions. The suicide prediction models estimated in this study reflect the association between predictors and events for larger racial/ethnic groups (ie, White non-Hispanic patients). Performance for smaller populations (ie, Black and American Indian/Alaskan Native patients) suffered, likely because the association between predictors and the outcome diverged from population-level trends in these subgroups, and information was insufficient to accurately estimate race/ethnicity-specific associations.

Currently available methods maximize predictive performance over the entire sample and do not consider performance within subgroups. Poor performance within smaller subgroups, for example, visits for American Indian/Alaskan Native patients, has little influence on overall performance because this group represents a small proportion of the entire sample. New methods that account for performance in subgroups are needed to reduce racial/ethnic inequities.

Limitations

This study has limitations. Specifically, it examined suicide prediction for people engaged in outpatient mental health care in integrated health systems. This population is not representative of all people with mental health care needs and the settings where they receive care. BIPOC populations are less likely to benefit from prediction models deployed in this context because of barriers to mental health treatment, including low rates of insurance, patient mistrust after historical mistreatment of BIPOC populations, and a lack of culturally competent care.^{14,15,36,44,45} Addressing inequitable effects of clinical prediction models will require using data from a variety of settings and populations. To enable this, research networks with existing health records infrastructure and resources can partner with health care systems that serve more racially/ethnically diverse, lower-income, and uninsured or underinsured populations. Incorporating data sources beyond health records, including information on other social determinants of health⁴⁶ and traumatic or negative life events, including racial trauma,⁴⁷ could also increase suicide risk discrimination.^48,49

In addition, too few suicides were observed for multiracial patients to precisely estimate model performance. Multiracial identity was uncommon in our sample, suggesting that many multiracial people were included in monoracial categories or declined to self-report racial identity (eTables 8-10 in the Supplement).^50,51 Predictive performance for multiracial people is important because they report higher rates of mental illness than other racial/ethnic groups.⁵²

Conclusions

Increasing use of clinical prediction models has been met with justified concern about disparate performance and impact of algorithms trained on clinical data. This study found racial/ethnic differences in performance of 2 suicide prediction models that accurately identify high-risk patients in the overall population. These findings suggest that clinical implementation of either prediction model to guide suicide prevention interventions would pose less potential benefit and more harm to patients who are Black and American Indian/Alaskan Native or whose race/ethnicity is unrecorded compared with White, Hispanic, and Asian patients. Systemic barriers faced by Black and Indigenous people to accessing affordable, effective, and culturally responsive mental health care contribute to differences in model performance. Clinical use of the prediction models examined here would exacerbate these disparities.

Prediction models intended for clinical use should be assessed for performance in the patient populations in whom they will be used, including vulnerable groups in that population. Evaluating performance with precision may be challenging because of small sample size or low event rate, but local validation should be attempted regardless because performance differences may be detectable even in small samples. Health care system stakeholders, including practitioners, patients, and representatives from diverse communities, must carefully consider disparities in both benefits and harms posed when deciding whether and how to implement a prediction model. Researchers and health care systems are obligated to bridge the gap when potential inequitable impacts are uncovered, through either improving prediction model performance or ensuring effective, minimally harmful interventions for underserved populations. Otherwise, benefits of clinical prediction models will not be universally realized, and systemic modes of deprivation will be reinforced.

Finally, potential benefits and harms of using a prediction model within particular populations must be considered in the context of existing health inequities. BIPOC populations already face significant barriers to accessing mental health care and, as a result, have poorer outcomes. In this context, deploying a prediction model that provides less benefit to already underserved populations will widen this care gap. Health care systems should prioritize resources to reduce race/ethnicity-based inequities in care.

Supplement.

eMethods. Statistical Analysis Plan

eTable 1. Data Availability Dates for Participating Sites

eTable 2. List of All Predictors for Logistic Regression With LASSO Prediction Models

eTable 3. AUC Estimates From 5-Fold Cross-Validation for Tuning Parameter Selection for (A) Logistic Regression With LASSO And (B) Random Forest

eTable 4. Positive Predictive Value (95% CI), Overall and Within Racial and Ethnic Subgroups, of Suicide Prediction Models

eTable 5. Brier Scores, Overall and Within Racial and Ethnic Subgroups, of Suicide Prediction Models

eTable 6. Cross-tabulation of Patient Race and Ethnicity for Visits in Sample

eTable 7. Summary of Death by Injury or Poisoning With Suicidal or Undetermined Intent, Overall and by Race and Ethnicity

eTable 8. Summary of Suicide Death Outcome, Overall and Within Racial and Ethnic Subgroups

eTable 9. Area Under the Curve (95% CI) of Suicide Prediction Models Within Excluded Racial and Ethnic Subgroups

eTable 10. Sensitivity and False Positive Rate (95% CI) of Suicide Prediction Models for Excluded Racial and Ethnic Subgroups

eTable 11. False Positive Rate, Overall and Within Racial and Ethnic Subgroups, of Suicide Risk Predictions at Fixed Levels of Sensitivity

eTable 12. Area Under the Curve (95% CI) of Suicide Prediction Models Within Racial Categories Including Visits by Patients With Hispanic Ethnicity

eTable 13. Sensitivity (95% CI) of Suicide Prediction Models for Racial Categories Including Visits by Patients With Hispanic Ethnicity

eTable 14. False Positive Rate (95% CI) of Suicide Prediction Models for Racial Categories Including Visits by Patients With Hispanic Ethnicity

eReferences

Click here for additional data file.^{(582.8KB, pdf)}

References

1.Matheny M, Israni ST, Ahmed M, Whicher D. Artificial Intelligence in Health Care: The Hope, the Hype, the Promise, the Peril. National Academy of Medicine. 2020:94-97. [Google Scholar]
2.Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med. 2019;380(14):1347-1358. doi: 10.1056/NEJMra1814259 [DOI] [PubMed] [Google Scholar]
3.Rajkomar A, Hardt M, Howell MD, Corrado G, Chin MH. Ensuring fairness in machine learning to advance health equity. Ann Intern Med. 2018;169(12):866-872. doi: 10.7326/M18-1990 [DOI] [PMC free article] [PubMed] [Google Scholar]
4.Parikh RB, Teeple S, Navathe AS. Addressing bias in artificial intelligence in health care. JAMA. 2019;322(24):2377-2378. doi: 10.1001/jama.2019.18058 [DOI] [PubMed] [Google Scholar]
5.Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447-453. doi: 10.1126/science.aax2342 [DOI] [PubMed] [Google Scholar]
6.Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med. 2018;178(11):1544-1547. doi: 10.1001/jamainternmed.2018.3763 [DOI] [PMC free article] [PubMed] [Google Scholar]
7.McCarthy JF, Bossarte RM, Katz IR, et al. Predictive modeling and concentration of the risk of suicide: implications for preventive interventions in the US Department of Veterans Affairs. Am J Public Health. 2015;105(9):1935-1942. doi: 10.2105/AJPH.2015.302737 [DOI] [PMC free article] [PubMed] [Google Scholar]
8.Simon GE, Johnson E, Lawrence JM, et al. Predicting suicide attempts and suicide deaths following outpatient visits using electronic health records. Am J Psychiatry. 2018;175(10):951-960. doi: 10.1176/appi.ajp.2018.17101167 [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Kessler RC, Warner CH, Ivany C, et al. ; Army STARRS Collaborators . Predicting suicides after psychiatric hospitalization in US Army soldiers: the Army Study To Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry. 2015;72(1):49-57. doi: 10.1001/jamapsychiatry.2014.1754 [DOI] [PMC free article] [PubMed] [Google Scholar]
10.McCoy TH Jr, Castro VM, Roberson AM, Snapper LA, Perlis RH. Improving prediction of suicide and accidental death after discharge from general hospitals with natural language processing. JAMA Psychiatry. 2016;73(10):1064-1071. doi: 10.1001/jamapsychiatry.2016.2172 [DOI] [PMC free article] [PubMed] [Google Scholar]
11.Walsh CG, Chaudhry B, Dua P, et al. Stigma, biomarkers, and algorithmic bias: recommendations for precision behavioral health with artificial intelligence. JAMIA Open. 2020;3(1):9-15. doi: 10.1093/jamiaopen/ooz054 [DOI] [PMC free article] [PubMed] [Google Scholar]
12.Suicide Prevention Resource Center. Racial and Ethnic Disparities . 2020. Accessed March 30, 2020. https://www.sprc.org/scope/racial-ethnic-disparities
13.Alegría M, Alvarez K, Ishikawa RZ, DiMarzio K, McPeck S. Removing obstacles to eliminating racial and ethnic disparities in behavioral health care. Health Aff (Millwood). 2016;35(6):991-999. doi: 10.1377/hlthaff.2016.0029 [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Hines AL, Cooper LA, Shi L. Racial and ethnic differences in mental healthcare utilization consistent with potentially effective care: the role of patient preferences. Gen Hosp Psychiatry. 2017;46:14-19. doi: 10.1016/j.genhosppsych.2017.02.002 [DOI] [PubMed] [Google Scholar]
15.Sonik RA, Creedon TB, Progovac AM, et al. ; Health Equity Consortium . Depression treatment preferences by race/ethnicity and gender and associations between past healthcare discrimination experiences and present preferences in a nationally representative sample. Soc Sci Med. 2020;253:112939. doi: 10.1016/j.socscimed.2020.112939 [DOI] [PubMed] [Google Scholar]
16.Ahmedani BK, Stewart C, Simon GE, et al. Racial/ethnic differences in health care visits made before suicide attempt across the United States. Med Care. 2015;53(5):430-435. doi: 10.1097/MLR.0000000000000335 [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Coleman KJ, Stewart C, Waitzfelder BE, et al. Racial-ethnic differences in psychiatric diagnoses and treatment across 11 health care systems in the Mental Health Research Network. Psychiatr Serv. 2016;67(7):749-757. doi: 10.1176/appi.ps.201500217 [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Waitzfelder B, Stewart C, Coleman KJ, et al. Treatment initiation for new episodes of depression in primary care settings. J Gen Intern Med. 2018;33(8):1283-1291. doi: 10.1007/s11606-017-4297-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
19.Merced K, Imel ZE, Baldwin SA, et al. Provider contributions to disparities in mental health care. Psychiatr Serv. 2020;71(8):765-771. doi: 10.1176/appi.ps.201800500 [DOI] [PMC free article] [PubMed] [Google Scholar]
20.Smedley BD, Stith AY, Nelson AR, eds. Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care. National Academies Press; 2003. [PubMed] [Google Scholar]
21.Ahmedani BK, Perron B, Ilgen M, Abdon A, Vaughn M, Epperson M. Suicide thoughts and attempts and psychiatric treatment utilization: informing prevention strategies. Psychiatr Serv. 2012;63(2):186-189. doi: 10.1176/appi.ps.201100159 [DOI] [PMC free article] [PubMed] [Google Scholar]
22.Ross TR, Ng D, Brown JS, et al. The HMO Research Network Virtual Data Warehouse: a public data model to support collaboration. EGEMS (Wash DC). 2014;2(1):1049. doi: 10.13063/2327-9214.1049 [DOI] [PMC free article] [PubMed] [Google Scholar]
23.Charlson M, Szatrowski TP, Peterson J, Gold J. Validation of a combined comorbidity index. J Clin Epidemiol. 1994;47(11):1245-1251. doi: 10.1016/0895-4356(94)90129-5 [DOI] [PubMed] [Google Scholar]
24.Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606-613. doi: 10.1046/j.1525-1497.2001.016009606.x [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Simon GE, Rutter CM, Peterson D, et al. Does response on the PHQ-9 Depression Questionnaire predict subsequent suicide attempt or suicide death? Psychiatr Serv. 2013;64(12):1195-1202. doi: 10.1176/appi.ps.201200587 [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Bakst SS, Braun T, Zucker I, Amitai Z, Shohat T. The accuracy of suicide statistics: are true suicide deaths misclassified? Soc Psychiatry Psychiatr Epidemiol. 2016;51(1):115-123. doi: 10.1007/s00127-015-1119-x [DOI] [PubMed] [Google Scholar]
27.Cox KL, Nock MK, Biggs QM, et al. ; Army STARRS Collaborators . An examination of potential misclassification of Army suicides: results from the Army Study to Assess Risk and Resilience in Servicemembers. Suicide Life Threat Behav. 2017;47(3):257-265. doi: 10.1111/sltb.12280 [DOI] [PMC free article] [PubMed] [Google Scholar]
28.Tibshirani R. Regression shrinkage and selection via the Lasso. J R Stat Soc Series B Stat Methodol. 1996;58(1):267-288. doi: 10.1111/j.2517-6161.1996.tb02080.x [DOI] [Google Scholar]
29.Breiman L. Random forests. Machine Learning. 2001;45(1):5-32. doi: 10.1023/A:1010933404324 [DOI] [Google Scholar]
30.Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983;148(3):839-843. doi: 10.1148/radiology.148.3.6878708 [DOI] [PubMed] [Google Scholar]
31.Tucker RP, Tackett MJ, Glickman D, Reger MA. Ethical and practical considerations in the use of a predictive model to trigger suicide prevention interventions in healthcare settings. Suicide Life Threat Behav. 2019;49(2):382-392. doi: 10.1111/sltb.12431 [DOI] [PubMed] [Google Scholar]
32.Swanson J, Swartz M, Van Dorn RA, et al. Racial disparities in involuntary outpatient commitment: are they real? Health Aff (Millwood). 2009;28(3):816-826. doi: 10.1377/hlthaff.28.3.816 [DOI] [PubMed] [Google Scholar]
33.Criss D, Asmelash L. When a police wellness check becomes a death sentence. CNN. October 19, 2019. Accessed August 9, 2020. https://www.cnn.com/2019/10/19/us/wellness-check-police-shootings-trnd [Google Scholar]
34.McKernan LC, Clayton EW, Walsh CG. Protecting life while preserving liberty: ethical recommendations for suicide prevention with artificial intelligence. Front Psychiatry. 2018;9:650. doi: 10.3389/fpsyt.2018.00650 [DOI] [PMC free article] [PubMed] [Google Scholar]
35.Gara MA, Minsky S, Silverstein SM, Miskimen T, Strakowski SM. A naturalistic study of racial disparities in diagnoses at an outpatient behavioral health clinic. Psychiatr Serv. 2019;70(2):130-134. doi: 10.1176/appi.ps.201800223 [DOI] [PubMed] [Google Scholar]
36.Williams DR, Wyatt R. Racial bias in health care and health: challenges and opportunities. JAMA. 2015;314(6):555-556. doi: 10.1001/jama.2015.9260 [DOI] [PubMed] [Google Scholar]
37.Hahm HC, Cook BL, Ault-Brutus A, Alegría M. Intersection of race-ethnicity and gender in depression care: screening, access, and minimally adequate treatment. Psychiatr Serv. 2015;66(3):258-264. doi: 10.1176/appi.ps.201400116 [DOI] [PMC free article] [PubMed] [Google Scholar]
38.Stockdale SE, Lagomasino IT, Siddique J, McGuire T, Miranda J. Racial and ethnic disparities in detection and treatment of depression and anxiety among psychiatric and primary health care visits, 1995-2005. Med Care. 2008;46(7):668-677. doi: 10.1097/MLR.0b013e3181789496 [DOI] [PMC free article] [PubMed] [Google Scholar]
39.National Science & Technology Council . Strengthening the Medicolegal-Death-Investigation System: Improving Data Systems. National Science & Technology Council; 2016. [Google Scholar]
40.Owens DK, Sox HC. Biomedical decision making: probabilistic clinical reasoning. In: Shortliffe EH, Cimino JJ, eds. Biomedical Informatics: Computer Applications in Health Care and Biomedicine. Springer; 2014:67-107. doi: 10.1007/978-1-4471-4474-8_3 [DOI] [Google Scholar]
41.Rockett IR, Wang S, Stack S, et al. Race/ethnicity and potential suicide misclassification: window on a minority suicide paradox? BMC Psychiatry. 2010;10(1):35. doi: 10.1186/1471-244X-10-35 [DOI] [PMC free article] [PubMed] [Google Scholar]
42.Durant T, Mercy J, Kresnow MJ, Simon T, Potter L, Hammond WR. Racial differences in hopelessness as a risk factor for a nearly lethal suicide attempt. J Black Psychol. 2006;32(3):285-302. doi: 10.1177/0095798406290468 [DOI] [Google Scholar]
43.Coleman KJ, Johnson E, Ahmedani BK, et al. Predicting suicide attempts for racial and ethnic groups of patients during routine clinical care. Suicide Life Threat Behav. 2019;49(3):724-734. doi: 10.1111/sltb.12454 [DOI] [PMC free article] [PubMed] [Google Scholar]
44.Williams MT, Rosen DC, Kanter JW, eds. Eliminating Race-Based Mental Health Disparities: Promoting Equity and Culturally Responsive Care Across Settings. New Harbinger Publications; 2019. [Google Scholar]
45.Betancourt JR, Green AR, Carrillo JE, Owusu Ananeh-Firempong I. Defining cultural competence: a practical framework for addressing racial/ethnic disparities in health and health care. Public Health Rep. 2003;118(4):293-302. doi: 10.1016/S0033-3549(04)50253-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
46.Primm AB, Vasquez MJ, Mays RA, et al. The role of public health in addressing racial and ethnic disparities in mental health and mental illness. Prev Chronic Dis. 2010;7(1):A20. [PMC free article] [PubMed] [Google Scholar]
47.Butts HF. The black mask of humanity: racial/ethnic discrimination and post-traumatic stress disorder. J Am Acad Psychiatry Law. 2002;30(3):336-339. [PubMed] [Google Scholar]
48.Hatch SL, Dohrenwend BP. Distribution of traumatic and other stressful life events by race/ethnicity, gender, SES and age: a review of the research. Am J Community Psychol. 2007;40(3-4):313-332. doi: 10.1007/s10464-007-9134-z [DOI] [PubMed] [Google Scholar]
49.Andrews AR III, Jobe-Shields L, López CM, et al. Polyvictimization, income, and ethnic differences in trauma-related mental health during adolescence. Soc Psychiatry Psychiatr Epidemiol. 2015;50(8):1223-1234. doi: 10.1007/s00127-015-1077-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
50.Khanna N, Johnson C. Passing as black: racial identity work among biracial Americans. Soc Psychol Q. 2010;73(4):380-397. doi: 10.1177/0190272510389014 [DOI] [Google Scholar]
51.National Public Radio. When you’re mixed race, just one box is not enough [podcast]. The Race Card Series. April 2, 2013. Accessed August 27, 2020. https://www.npr.org/2013/04/02/175292625/when-youre-mixed-race-just-one-box-is-not-enough
52.American Psychiatric Association. Mental health disparities: diverse populations. Accessed August 26, 2020. https://www.psychiatry.org/psychiatrists/cultural-competency/education/mental-health-facts

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials