Estimated Average Treatment Effect of Psychiatric Hospitalization in Patients With Suicidal Behaviors: A Precision Treatment Analysis

Eric L Ross; Robert M Bossarte; Steven K Dobscha; Sarah M Gildea; Irving Hwang; Chris J Kennedy; Howard Liu; Alex Luedtke; Brian P Marx; Matthew K Nock; Maria V Petukhova; Nancy A Sampson; Nur Hani Zainal; Erik Sverdrup; Stefan Wager; Ronald C Kessler

doi:10.1001/jamapsychiatry.2023.3994

. 2023 Oct 18;81(2):135–143. doi: 10.1001/jamapsychiatry.2023.3994

Estimated Average Treatment Effect of Psychiatric Hospitalization in Patients With Suicidal Behaviors

A Precision Treatment Analysis

Eric L Ross ¹, Robert M Bossarte ², Steven K Dobscha ³, Sarah M Gildea ^4,⁵, Irving Hwang ⁴, Chris J Kennedy ⁶, Howard Liu ^4,⁵, Alex Luedtke ^7,⁸, Brian P Marx ^9,¹⁰, Matthew K Nock ^6,¹¹, Maria V Petukhova ⁴, Nancy A Sampson ⁴, Nur Hani Zainal ⁴, Erik Sverdrup ¹², Stefan Wager ¹², Ronald C Kessler ^4,^✉

¹Department of Psychiatry, Larner College of Medicine, University of Vermont, Burlington

²Department of Psychiatry and Behavioral Neurosciences, University of South Florida, Tampa

³VA Center to Improve Veteran Involvement in Care, Portland, Oregon

⁴Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts

⁵Center of Excellence for Suicide Prevention, Canandaigua VA Medical Center, Canandaigua, New York

⁶Department of Psychiatry, Massachusetts General Hospital, Boston

⁷Department of Statistics, University of Washington, Seattle

⁸Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Research Center, Seattle, Washington

⁹National Center for PTSD, VA Boston Healthcare System, Boston, Massachusetts

¹⁰Department of Psychiatry, Boston University School of Medicine, Boston, Massachusetts

¹¹Department of Psychology, Harvard University, Cambridge, Massachusetts

¹²Graduate School of Business, Stanford University, Stanford, California

Accepted for Publication: August 17, 2023.

Published Online: October 18, 2023. doi:10.1001/jamapsychiatry.2023.3994

^✉

Corresponding Author: Ronald C. Kessler, PhD, Department of Health Care Policy, Harvard Medical School, 180 Longwood Ave, Ste 215, Boston, MA 02115-5899 (kessler@hcp.med.harvard.edu).

Author Contributions: Dr Kessler had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Ross, Dobscha, Sverdrup, Wager, Kessler.

Acquisition, analysis, or interpretation of data: Ross, Bossarte, Dobscha, Gildea, Hwang, Kennedy, Liu, Luedtke, Marx, Nock, Petukhova, Sampson, Zainal, Wager, Kessler.

Drafting of the manuscript: Ross, Dobscha, Wager, Kessler.

Critical review of the manuscript for important intellectual content: All authors.

Statistical analysis: Ross, Hwang, Kennedy, Liu, Luedtke, Petukhova, Sverdrup, Wager.

Obtained funding: Kessler.

Administrative, technical, or material support: Bossarte, Gildea, Marx, Zainal, Kessler.

Supervision: Sampson, Wager, Kessler.

Conflict of Interest Disclosures: Dr Nock reported receiving publication royalties from Macmillan, Pearson, and UpToDate; consulting fees from Microsoft Corporation, the Veterans Health Administration, COMPASS Pathways, and for legal cases regarding a death by suicide; and holding stock options in Cerebral. Dr Nock is an unpaid scientific advisor for Empatica, Koko, and TalkLife. Dr Wager reported receiving grants from Google outside the submitted work. Dr Kessler reported receiving consulting fees from Cambridge Health Alliance, Canandaigua VA Medical Center, Holmusk, Partners Healthcare Inc, RallyPoint Networks Inc, and Sage Therapeutics; and holding stock options from Cerebral Inc, Mirah, PYM, Roga Sciences, and Verisense Health during the conduct of the study. No other disclosures were reported.

Funding/Support: The research reported in this publication was funded by the US National Institute of Mental Health (grant R01MH121478) and by the VA Center of Excellence for Suicide Prevention, Canandaigua VA Medical Center.

Role of the Funder/Sponsor: The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the Veterans Health Administration.

Data Sharing Statement: See Supplement 2.

^✉

Corresponding author.

PMCID: PMC10585585 PMID: 37851457

Key Points

Question

Can development of an individualized treatment rule identify patients presenting to emergency departments/urgent care with suicidal ideation or suicide attempts who are likely to benefit from psychiatric hospitalization?

Findings

A decision analytic model found that hospitalization was associated with reduced suicide attempt risk among patients who attempted suicide in the past day but not among others with suicidality. Accounting for heterogeneity, suicide attempt risk was found to increase with hospitalization in 24% of patients and decrease in 28%.

Meaning

Results of this study suggest that implementing an individualized treatment rule could identify many additional patients who may benefit from or be harmed by hospitalization.

Abstract

Importance

Psychiatric hospitalization is the standard of care for patients presenting to an emergency department (ED) or urgent care (UC) with high suicide risk. However, the effect of hospitalization in reducing subsequent suicidal behaviors is poorly understood and likely heterogeneous.

Objectives

To estimate the association of psychiatric hospitalization with subsequent suicidal behaviors using observational data and develop a preliminary predictive analytics individualized treatment rule accounting for heterogeneity in this association across patients.

Design, Setting, and Participants

A machine learning analysis of retrospective data was conducted. All veterans presenting with suicidal ideation (SI) or suicide attempt (SA) from January 1, 2010, to December 31, 2015, were included. Data were analyzed from September 1, 2022, to March 10, 2023. Subgroups were defined by primary psychiatric diagnosis (nonaffective psychosis, bipolar disorder, major depressive disorder, and other) and suicidality (SI only, SA in past 2-7 days, and SA in past day). Models were trained in 70.0% of the training samples and tested in the remaining 30.0%.

Exposures

Psychiatric hospitalization vs nonhospitalization.

Main Outcomes and Measures

Fatal and nonfatal SAs within 12 months of ED/UC visits were identified in administrative records and the National Death Index. Baseline covariates were drawn from electronic health records and geospatial databases.

Results

Of 196 610 visits (90.3% men; median [IQR] age, 53 [41-59] years), 71.5% resulted in hospitalization. The 12-month SA risk was 11.9% with hospitalization and 12.0% with nonhospitalization (difference, −0.1%; 95% CI, −0.4% to 0.2%). In patients with SI only or SA in the past 2 to 7 days, most hospitalization was not associated with subsequent SAs. For patients with SA in the past day, hospitalization was associated with risk reductions ranging from −6.9% to −9.6% across diagnoses. Accounting for heterogeneity, hospitalization was associated with reduced risk of subsequent SAs in 28.1% of the patients and increased risk in 24.0%. An individualized treatment rule based on these associations may reduce SAs by 16.0% and hospitalizations by 13.0% compared with current rates.

Conclusions and Relevance

The findings of this study suggest that psychiatric hospitalization is associated with reduced average SA risk in the immediate aftermath of an SA but not after other recent SAs or SI only. Substantial heterogeneity exists in these associations across patients. An individualized treatment rule accounting for this heterogeneity could both reduce SAs and avert hospitalizations.

This predictive analytics model develops a preliminary individualized treatment rule to estimate the association between psychiatric hospitalization and subsequent suicidal behaviors in patients at risk for suicide.

Introduction

The US suicide rate has increased 30.0% since 2000.¹ Mirroring this trend, emergency department (ED) visits for suicidality more than doubled in the past decade, now making up approximately 1.1% of all ED visits.² Treatment of patients presenting to EDs with suicidality is guided by an assessment of suicide risk. Hospitalization is the accepted standard of care for patients deemed at high imminent risk.^3,4 This risk assessment relies almost exclusively on clinicians’ expertise and judgment. Traditional risk scales do not substantially improve on clinical judgment,^5,6,7 and while some machine learning models have begun to surpass the accuracy of clinical risk prediction,⁸ these have not been widely implemented.

Another aspect of treatment decision-making that has even sparser evidence is the extent to which psychiatric hospitalization reduces the future risk of suicidal behavior in a patient with suicidality. It is unclear whether hospitalization reduces this risk.⁹ To our knowledge, the only 2 randomized trials ever carried out to compare hospitalization with discharge or intensive outpatient care for patients with suicidality found no benefits of hospitalization on either symptom burden or suicide risk.^10,11 Larger observational studies using various statistical methods to examine the same association nonexperimentally found that hospitalization is, for the most part, associated with either unchanged or increased suicide risk,^{12,13,14,15,16} although the latter might reflect failure to adjust adequately for adverse selection. More broadly, there are other substantial costs of hospitalization, including loss of autonomy, increased health care expenditures, and increased ED boarding times.^17,18

Considering the substantial costs and unclear effectiveness of psychiatric hospitalization in preventing suicidal behaviors, it is critical to find a means of targeting this intervention to the patients most likely to benefit and to avoid hospitalization in patients for whom it might be harmful. To date, there are no tools available to aid clinicians in doing this. To address this gap, we applied machine learning methods to observational (ie, nonexperimental) data from patients with suicidality presenting to Veterans Health Administration (VHA) EDs and urgent care (UC) to train a predictive analytics model that (1) estimated the average treatment effect of psychiatric hospitalization on the risk of a subsequent suicide attempt (SA), (2) evaluated whether heterogeneity exists in the association of psychiatric hospitalization with subsequent risk of SA, and (3) developed a preliminary individualized treatment rule (ITR) to identify subgroups of patients for whom hospitalization is likely to be associated with significantly decreased, significantly increased, and nonsignificant change in risk of SA.

Methods

Sample

We included all ED and UC visits of VHA patients in the US who presented with either suicidal ideation (SI) or SA and primary diagnoses of mental or substance use disorders according to the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) or International Statistical Classification of Diseases and Related Health Problems, Tenth Revision, Clinical Modification (ICD-10-CM) between January 1, 2010, and December 31, 2015. The study protocol was approved by the research ethics committees of the Veterans Administration Center of Excellence for Suicide Prevention and Harvard Medical School with a waiver of informed consent based on data being deidentified. Patients were followed up for 365 days after each visit. The sample was disaggregated into subgroups defined by primary presenting diagnosis (nonaffective psychosis, bipolar disorder, major depressive disorder, and any other ICD-9 CM or ICD-10-CM mental or substance use disorder [Other]) and level of suicidality (SI without a recent SA [SI only], SA in the 2-7 days preceding the visit [SA 2-7 days], and SA within 1 day of the visit [SA 1 day]). As a descriptive analytics study, the noneconomic aspects of the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) reporting guideline were used in reporting results.¹⁹

Treatment and Outcome Definitions

The focal treatment was psychiatric hospitalization. We defined a visit as resulting in psychiatric hospitalization if the patient was admitted with a primary diagnosis of an ICD-9-CM or ICD-10-CM mental or substance use disorder within 7 days of their initial ED presentation. We used this 7-day horizon to capture psychiatric admissions that occurred after a prolonged ED stay or brief medical admission. Due to difficulties in reliably identifying whether a psychiatric admission occurred for a small number of patients with current SA who were initially medically hospitalized, we excluded 783 such visits of nearly 200 000 visits in the analysis. The primary outcome was a new fatal or nonfatal SA identified within 12 months after the ED/UC visit (referred to herein as 12-month SA).

Data Sources

Nonfatal SAs were identified via either ICD-9-CM or ICD-10-CM codes for SAs in the VHA Corporate Data Warehouse²⁰ or by entries for SAs in the VHA Suicide Prevention Applications Network.²¹ The combination captures far more cases than by relying on medical records alone. Self-injuries with uncharacterized intent were not included in the definition. Fatal SAs were identified in the National Death Index.²²

Predictors of hospitalization and SAs were obtained from 3 sources: (1) structured predictors from the VHA Corporate Data Warehouse²⁰ (eAppendix in Supplement 1), (2) information about prior suicidal behaviors from the VHA Suicide Prevention Applications Network,²¹ and (3) a geospatial social determinants of health database (eTable 1 in Supplement 1) based on government data sources for the patient’s residential neighborhood (block group, census tract), county, and state.²³ Data linkage was made using Social Security numbers available for all VHA patients. Although the Corporate Data Warehouse shares the limitation of other electronic health records databases with respect to such issues as data completeness and coding consistency,²⁴ the VHA makes special efforts to address these challenges through ongoing training and quality monitoring activities.²⁵ The geospatial social determinants of health database, in comparison, is based on data aggregated from the Census Bureau’s Current Population Survey²⁶ and American Community Survey.²⁷

Predictors

We estimated treatment heterogeneity using baseline covariates that might reasonably be expected to predict SA among hospitalized and/or nonhospitalized patients based on a review of the literature described in the eMethods in Supplement 1. The databases included were used to operationalize 4 broad classes of these covariates: (1) psychopathologic risk factors (diagnoses, treatments, and suicidality) over the 5 years before the ED/UC visit (eAppendix in Supplement 1); (2) physical disorders and treatments during the same time period (eAppendix in Supplement 1), including injuries during the focal ED/UC visit and prescribed medications used before the focal visit classified by the US Food and Drug Administration as increasing suicide risk (eTable 2 in Supplement 1); (3) facility-level quality indicators (eg, inpatient staff turnover rates) (eAppendix in Supplement 1); and (4) indicators of social determinants of health at both the patient level (eg, ICD-9-CM and ICD-10-CM codes) and the geospatial level (eAppendix and eTable 1 in Supplement 1). These variables were selected based on a review of research on predictors of suicidal behaviors after psychiatric hospital discharge^28,29,30,31 and in more general patient samples.^32,33,34 Details about constructs, assessments, and rationale for inclusion are presented in the eMethods in Supplement 1. Missing values, which occurred in no more than 3.0% of records and only for geospatial variables, were imputed hierarchically using nearest-neighbor, rational, and median value imputations. Categorical predictors in all databases were 1-hot encoded as 0 to 1 dummy variables.

Statistical Analysis

All analyses were carried out using R, version 3.6.3 (The R Foundation for Statistical Computing). Data management was implemented with SAS, version 9.4 (SAS Institute Inc). Analyses were carried out from September 1, 2022, to March 10, 2023.

Estimating Average Treatment Effects

Although treatment effects cannot be estimated unequivocally with observational data, the analyses assumed as a first approximation that any nonrandom treatment assignment can be corrected by controlling for the baseline covariates described above.^35,36 The eMethods in Supplement 1 provides a discussion of this unconfoundedness assumption. Subsequent implementation of a pragmatic trial to evaluate the effect of using the current results to guide treatment assignment would be needed to avoid this provisional assumption.

A propensity score approach³⁷ was used to adjust for significant differences in baseline predictors between patients who were and were not hospitalized. Each visit i was assigned a weight of 1/p_i, where the propensity score p_i was the estimated probability of hospitalization for that presentation generated using the random forests (RF) machine learning method.³⁸ The expected outcomes for each visit given baseline covariates and each possible treatment assignment were estimated using a separate RF analysis. The estimated average treatment effect (ATE) of hospitalization in 12-month SA was then obtained via a doubly robust method,^39,40 combining information from these 2 RF analyses; the eMethods in Supplement 1 provides details. All steps were performed using generalized RF, a machine learning approach that expands on RF^41,42 with a focus on estimating treatment effects adjusting for measured confounders. To account for clustering due to some patients having multiple ED/UC visits, all visits for any patient in the training sample with multiple visits were included in a single fold of the 10-fold cross-validated internal subsampling procedure used to fit models. To account for possible biased ATE estimates in subgroups with extremely skewed p_i distributions, we calculated ATEs separately among hospitalized (ATE on the treated [ATT]) and nonhospitalized (ATE on the controls [ATC]) patients. All of these analyses used the grf R package.⁴³

Estimating Treatment Heterogeneity

The grf-defined treatment heterogeneity was the estimated conditional ATE (CATE); that is, the expected treatment effect for a patient conditional on the patient having a specific combination of baseline covariate values. The eMethods in Supplement 1 provides details. CATEs were first estimated in the 70.0% training sample and then evaluated in the 30.0% test sample. The evaluation entailed imputing CATEs from the training sample to the test sample, dividing the test sample into quintiles defined by these CATEs and estimating ATEs within each quintile using data from the test sample. The goal was to determine whether the ATE was highest in the quintile predicted to benefit most from hospitalization and lowest in the quintile predicted to benefit least from hospitalization. To rule out dependence on a specific software package, we estimated within-quintile ATEs using a different doubly robust approach described in the eMethods in Supplement 1.⁴⁴ A provisional ITR was then developed based on analysis of the CATE distribution.

Predictor Importance

Predictor importance in defining CATEs was examined using the kernel Shapley Additive Explanations (SHAP) method⁴⁵ implemented with the fastshap R package.⁴⁶ This method estimates the implications of changing a predictor from its observed score to the sample mean averaged across all logically possible permutations of other predictors. A higher mean absolute SHAP value suggests a more important predictor. Proportional mean absolute SHAP values (proportional SHAP) were calculated by dividing mean absolute SHAP values of classes and important predictors within classes by the mean absolute SHAP value of the entire model. Bee swarm plots were used to identify dominant directions and distributions of associations. The eMethods in Supplement 1 provides further details. The significance threshold was set to .05, and all tests were 2-sided and unpaired.

Results

Sample Characteristics

We identified 196 610 relevant visits made by 107 638 patients (an average of 1.8 visits/patient; range, 1-72; IQR, 1-2). Most visits were SI only (84.1% [165 307 of 196 610]) rather than SA 2 to 7 days (2.9% [5732 of 196 610]) or SA 1 day (13.0% [25 571 of 196 610]) (Table 1). Most patients were male (90.3% vs 9.7% women); median age was 53 (IQR, 41-59) years, and most patients were White (59.3%).⁴⁷ Nonaffective psychosis was the most frequent diagnostic category (61 522 visits) and bipolar disorder was the least frequent (34 071 visits). Distributions of patient characteristics were similar for visits resulting in hospitalization vs nonhospitalization except that SA 1 day was more common among hospitalized patients (15.0% [21 053 of 140 546] vs 8.1% [4518 of 56 064]).

Table 1. Patient Characteristics.

Characteristic	Patients, No. (%)
Characteristic	Total (N=196 610)	Hospitalized (n=140 546)	Not hospitalized (n=56 064)
Sex
Male	177 493 (90.3)	127 234 (90.5)	50 259 (89.6)
Female	19 117 (9.7)	13 312 (9.5)	5805 (10.4)
Age, y
Median (IQR)	53 (41-59)	52 (41-59)	53 (42-60)
Race and ethnicity^a
Hispanic	15 189 (7.7)	10 522 (7.5)	4667 (8.3)
Non-Hispanic Black	53 829 (27.4)	37 788 (26.9)	16 041 (28.6)
Non-Hispanic White	116 642 (59.3)	84 563 (60.2)	32 079 (57.2)
Other^b	10 950 (5.6)	7673 (5.5)	3277 (5.8)
Primary psychiatric diagnosis
Nonaffective psychosis	61 522 (31.3)	44 214 (31.5)	17 308 (30.9)
Major depressive disorder	60 790 (30.9)	45 509 (32.4)	15 281 (27.3)
Bipolar disorder	34 071 (17.3)	25 191 (17.9)	8880 (15.8)
Other	40 227 (20.5)	25 632 (18.2)	14 595 (26.0)
Level of suicidality
Suicidal ideation without recent attempt	165 307 (84.1)	116 446 (82.9)	48 861 (87.2)
Suicide attempt in past 2-7 d^c	5732 (2.9)	3047 (2.2)	2685 (4.8)
Suicide attempt in past day^d	25 571 (13.0)	21 053 (15.0)	4518 (8.1)
Prior (to past 7 d) lifetime suicide attempts
None	148 672 (75.6)	107 064 (76.2)	41 608 (74.2)
≥1	47 938 (24.4)	33 482 (23.8)	14 456 (25.8)

Open in a new tab

^{^a}

Race and ethnicity was included in the analyses as prior examinations have revealed racial and ethnic differences in psychiatric hospitalizations. Race and ethnicity are stored separately in the database. The categories used herein were created by cross-classifying the race and ethnicity variables. Individuals who identified as Hispanic or Latino ethnicity were categorized as Hispanic. All others were categorized using the race variable.

^{^b}

Other racial and ethnic groups included American Indian or Alaska Native, Asian, and Native Hawaiian or Other Pacific Islander. They were not separated into individual categories due to low numbers.

^{^c}

Before the emergency department/urgent care visit.

^{^d}

Within 1 day of the emergency department/urgent care visit.

Unadjusted Comparisons

A total of 71.5% (140 546 of 196 610) of the visits resulted in hospitalization, with the highest hospitalization rate (82.3% [21 053 of 25 571]) among patients with SA 1 day (Table 2). Twelve-month SA was 11.9% (95% CI, 11.8%-12.1%; P < .001) in the total sample. The highest 12-month SA (20.7%; 95% CI, 19.6%-21.7%; P < .001) was among patients with SA 2 to 7 days irrespective of hospitalization status. Hospitalization was not associated with 12-month SA in the total sample or in subgroups with SI only or SA 2 to 7 days. In contrast, for patients with SA 1 day, hospitalization was associated with a reduction in 12-month SA from 25.9% (95% CI, 24.6%-27.1%; P < .001) to 18.3% (95% CI, 17.8%-18.9%; P < .001) (difference, −7.5%; 95% CI, −8.9% to −6.1%; P < .001). Results were similar when further disaggregated by diagnostic subgroup (eTable 3 in Supplement 1).

Table 2. Unadjusted Distribution of the Outcome.

Variable	No. (%)	12-mo SA, % (95% CI)^a
Entire sample
Total	196 610	11.9 (11.8 to 12.1)
Hospitalized	140 546 (71.5)	11.9 (11.7 to 12.1)
Not hospitalized	56 064 (28.5)	12.0 (11.7 to 12.2)
Difference	NA	−0.1 (−0.4 to 0.2)
P value	NA	.60
Suicidal ideation without recent attempt
Total	165 307	10.4 (10.3 to 10.6)
Hospitalized	116 446 (70.4)	10.5 (10.3 to 10.7)
Not hospitalized	48 861 (29.6)	10.2 (9.9 to 10.4)
Difference	NA	0.3 (0.0 to 0.7)^b
P value	NA	.04
SA in past 2-7 d ^c
Total	5732	20.7 (19.6 to 21.7)
Hospitalized	3047 (53.2)	20.0 (18.5 to 21.4)
Not hospitalized	2685 (46.8)	21.5 (20.0 to 23.1)
Difference	NA	−1.6 (−3.7 to 0.5)
P value	NA	.14
SA in past d ^d
Total	25 571	19.7 (19.2 to 20.2)
Hospitalized	21 053 (82.3)	18.3 (17.8 to 18.9)
Not hospitalized	4518 (17.7)	25.9 (24.6 to 27.1)
Difference	NA	−7.5 (−8.9 to −6.1)^b
P value	NA	<.001

Open in a new tab

Abbreviations: NA, not applicable; SA, suicide attempt.

^{^a}

Nonfatal or fatal suicide attempt within 12 months (365 days) of the visit.

^{^b}

Significant at the .05 level, with 2-sided and unpaired testing.

^{^c}

Prior to the emergency department/urgent care visit.

^{^d}

Within 1 day of the emergency department/urgent care visit.

Estimated ATE

The estimated ATE of hospitalization after controlling nonparametrically for baseline covariates was consistent with the unadjusted results (Figure 1). Specifically, hospitalization was associated with a reduction in SA among patients with SA 1 day across all diagnoses. The ATEs ranged from −6.9% (95% CI, −9.7% to −4.1%; P < .001) among patients with nonaffective psychosis to −9.6% (95% CI, −13.5% to −7.7%; P < .001) among patients with bipolar disorder. In comparison, ATEs did not differ significantly from 0 for any other subgroup besides patients in the Other category with SI only (0.6%; 95% CI, 0.1%-1.2%; P = .02) and those with major depressive disorder with SA 2 to 7 days (−4.0%; 95% CI, −7.7% to −0.2%; P = .04). The ATT and ATC estimates (eTable 4 in Supplement 1) never differed from the ATE estimate by more than 1.1%, indicating good stability of the estimates.

Figure 1. — The “other” category indicates any other *International Classification of Diseases, Ninth Revision, Clinical Modification* or *International Statistical Classification of Diseases, Tenth Revision, Clinical Modification* mental or substance use disorder. MDD indicates major depressive disorder. Error bars indicate 95% CIs.

Estimated Heterogeneous Treatment Effects

Variation in ATE in the test sample across quintiles of predicted CATE was statistically significant in 9 of 12 subgroups (Figure 2; eTables 5-8 in Supplement 1). In the quintile in which ATE was predicted by training sample models to be highest, the estimated ATE was negative (ie, hospitalization was associated with reduced SA) across all 12 subgroups and statistically significant in 10 subgroups. In the quintile in which the benefit of hospitalization was predicted by the training sample models to be lowest, the estimated test sample ATE was positive (ie, hospitalization was associated with increased SA) across 10 of 12 subgroups and statistically significant in all 4 subgroups involving patients with SI only.

Overall, hospitalization was associated with reduced SA risk in 28.1% (SE, 0.2) of patients in the test sample and increased SA risk in another 24.0% of the patients (SE, 0.2) (eTable 9 in Supplement 1). The correlation between individual-level ATE and observed hospitalization was weak both overall (Pearson r = 0.09) and within separate diagnostic subgroups (Pearson r = 0.07-0.11) (eTable 10 in Supplement 1).

Simulated Effects of an ITR

A reasonable ITR based on these results would be to (1) hospitalize all patients in subgroups predicted to benefit significantly from hospitalization (ie, those with the highest predicted CATE), (2) avoid hospitalization in subgroups predicted to be harmed by hospitalization (ie, those with the lowest predicted CATE), and (3) defer the hospitalization decision to clinical judgment for the remaining patients. Observed hospitalization rates (SE) for these 3 groups were 73.6% (0.5%) for the first group, 66.8% (0.6%) for the second group, and 75.5% (0.6%) for the third group.

The simulated effects of this ITR are reported in Table 3. Assuming unconfoundedness, the ITR would reduce SA risk proportionally in the total population by 16.0% (95% CI, 12.6%-19.4%; P < .001) and reduce hospitalizations proportionally by 13.0% (95% CI, 12.7%-13.3%; P < .001). Both prevented SAs (19.5%; 95% CI, 15.3%-23.6%; P < .001) and averted hospitalizations (17.2%; 95% CI, 16.7%-17.6%; P < .001) would be highest among patients with SI only.

Table 3. Overall Estimated Effect of Implementing Individualized Treatment Rule vs Observed Treatment Decisions.

Variable	Primary presenting diagnosis
	Total		NAP		BD		MDD		Other^a
	% (95% CI)	P value	% (95% CI)	P value	% (95% CI)	P value	% (95% CI)	P value	% (95% CI)	P value
Prevented suicide attempts
Entire sample	16.0 (12.6 to 19.4)^b	<.001	20.0 (16.1 to 23.8)^b	<.001	18.1 (10.5 to 25.7)^b	<.001	8.8 (−0.5 to 18.1)	.06	13.3 (3.1 to 23.5)^b	.01
Suicidal ideation without recent attempt	19.5 (15.3 to 23.6)^b	<.001	24.5 (20.1 to 29.0)^b	<.001	22.2 (13.1 to 31.3)^b	<.001	10.2 (−1.4 to 21.8)	.09	14.8 (0.7 to 29.0)^b	.04
Suicide attempt in past 2-7 d^c	9.4 (−11.2 to 30.0)	.37	10.4 (−24.0 to 44.7)	.55	13.6 (−24.5 to 51.6)	.48	3.8 (−36.9 to 44.4)	.86	8.2 (−44.6 to 61.0)	.76
Suicide attempt in past d^d	5.9 (0.4 to 11.4)^b	.04	3.9 (−4.8 to 12.5)	.38	5.2 (−7.8 to 18.2)	.44	5.6 (−8.4 to 19.6)	.43	10.5 (−0.3 to 21.3)	.06
Averted hospitalizations
Entire sample	13.0 (12.7 to 13.3)^b	<.001	17.9 (17.2 to 18.5)^b	<.001	24.7 (23.7 to 25.7)^b	<.001	7.9 (7.4 to 8.3)^b	<.001	2.2 (1.9 to 2.6)^b	<.001
Suicidal ideation without recent attempt	17.2 (16.7 to 17.6)^b	<.001	22.2 (21.4 to 23.0)^b	<.001	31.6 (30.4 to 32.8)^b	<.001	10.4 (9.9 to 11.0)^b	<.001	5.7 (5.1 to 6.3)^b	<.001
Suicide attempt in past 2-7 d^c	−17.8 (−20.4 to −15.2)^b	<.001	−19.9 (−24.8 to −14.9)^b	<.001	−19.6 (−25.5 to −13.7)^b	<.001	−11.9 (−15.7 to −8.1)^b	<.001	−23.6 (−31.2 to −16.0)^b	<.001
Suicide attempt in past d^d	−5.4 (−6.0 to −4.9)^b	<.001	−4.0 (−5.0 to −3.1)^b	<.001	−4.9 (−6.1 to −3.6)^b	<.001	−3.7 (−4.6 to −2.9)^b	<.001	−10.0 (−11.6 to −8.4)^b	<.001

Open in a new tab

Abbreviations: BD, bipolar disorder; MDD, major depressive disorder; NAP, nonaffective psychosis.

^{^a}

The presenting diagnosis was any other mental or substance use disorder.

^{^b}

Significant at the .05 level, with 2-sided testing.

^{^c}

Prior to the emergency department/urgent care visit.

^{^d}

Within 1 day of the emergency department/urgent care visit.

Predictor Importance

Inspection of proportional SHAP values by predictor domain shows 3 broad trends (eTable 11 in Supplement 1). First, psychopathologic risk factors were the single most important class of predictors in 8 of 12 subgroups, with proportional SHAP values greater than 50% in 9 of the 12 subgroups. Second, facility-level quality indicators (proportional SHAP, 0.4%-9.9%) and physical disorders (proportional SHAP, 1.5%-48.8%) were consistently the least important predictors. Third, the importance of social determinants of health predictors varied markedly across suicidality subgroups, with proportional SHAP values of 6.8% to 29.4% for SA 1 day, 18.8% to 97.7% for SA 2 to 7 days, and 40.2% to 87.4% for SI only. More detailed analyses inspecting bee swarm plots (eFigures 1-12 in Supplement 1) and key predictor distributions in extreme quintiles (eFigures 13-22 in Supplement 1) showed the relationships to be highly complex, with relationships involving individual predictors varying jointly across disorders and levels of suicidality.

Discussion

In this retrospective analysis of VHA patients who presented to an ED/UC with suicidality, we used a machine-learning method to examine associations of psychiatric hospitalization with risk of subsequent SA. We found that hospitalization was associated with reduced average risk of subsequent SAs for patients in the immediate aftermath of an SA, but this risk was not substantially altered among patients with SI only or more remote SAs. There was substantial between-patient heterogeneity underlying these average associations, with hospitalization predicting significantly increased SA risk among some patients and significantly decreased risk among others.

To our knowledge, our study is the first to find that hospitalization is associated with reduced subsequent suicidal behavior among patients presenting to an ED/UC with suicidality. On first examination, this might seem to conflict with prior retrospective analyses that have found that hospitalization is associated with increased or unchanged risk of subsequent SAs.^{12,13,14,15,16} However, these prior studies used a distinct and heterogeneous population: patients presenting to EDs after self-harm with unspecified intent. Using an equally heterogeneous sample (combining SI and SAs), we similarly found no benefit of hospitalization; however, when we disaggregated patients with SAs from those with SI, we found that hospitalization was associated with a substantially reduced risk of subsequent suicidal behavior among patients with SA 1 day. This finding reinforces the importance of SAs in clinical decision-making,^3,4 especially when ED/UC visits are made in the immediate aftermath, and suggests that future research in this area should strive to disentangle presentations with SA from other manifestations of suicidality.

Our findings have immediate clinical implications for management of suicidality in ED settings. For patients presenting in the aftermath of an SA, clinicians could reasonably consider hospitalization the default approach in that it might be expected to substantially reduce the overall risk of subsequent SAs without increasing the risk among any identifiable patient subset. In contrast, for patients with suicidality other than in the immediate aftermath of an SA, hospitalization is not a justifiable default approach, as hospitalization is associated with an increased risk of subsequent SAs in 20.0% to 40.0% of patients and decreased risk in another 20.0% to 40.0%.

This heterogeneity in the association of hospitalization with subsequent SA risk clearly highlights the need for individualized treatment approaches and the limitations of any default treatment strategy. Compared with current clinical decision-making, we found that using our ITR to support decision-making might prevent as much as one-fifth of SAs while requiring one-sixth fewer hospitalizations. Given the nonexperimental nature of the analysis, broad implementation of this ITR would not be warranted unless it was confirmed in a pragmatic trial. Considering the increasing psychiatric bed shortage in the US,⁴⁸ escalating ED boarding problem,¹⁷ and limited success in reducing suicide rates,¹ the importance of carrying out such a trial is clear.

Limitations

Our work has 7 noteworthy limitations. First, findings may be biased by confounding due to unobserved determinants of the nonrandom probability of being hospitalized.

Second, the observed exposure (ie, hospitalization) and outcome (ie, SAs recorded in VA administrative databases) were proxies for unobserved variables of interest: in the case of the exposure, the clinical decision to hospitalize, which will not result in hospitalization if the patient refuses voluntary hospitalization and the clinician does not require involuntary hospitalization; in the case of the outcome, any SA regardless of whether treatment was obtained and, if obtained, occurred in the VA system or paid for by the VA. Classification errors in these observed measures could bias results. Detecting SAs for which treatment was not obtained would require follow-up surveys. Third, even in the absence of unmeasured confounders, residual bias could exist due to finite-sample errors in the RF or inherent limitations in the RF algorithm underlying a generalized random forest.

Fourth, our inclusion of nonfatal SAs that came to medical attention along with suicide deaths in our composite outcome measure is different from some prior work that focused on suicide deaths.¹³ This more inclusive outcome increased our statistical power at the expense of the outcome’s homogeneity and clinical importance, although some consequences (eg, future health care costs) are greater for nonfatal than fatal SAs.⁴⁹ Fifth, the results apply only to patients in the VHA system, who differ from patients in other sectors of the health care system in terms of both their characteristics and the services available to them.

Sixth, a wide range of clinical options exists as alternatives to psychiatric hospitalization (eg, conventional outpatient treatment, suicide-focused outpatient treatment, intensive case management, and partial hospitalization).⁴ We were unable to disentangle the estimated ATE of hospitalization separately from each of these options given the lack of information on which options were pursued for the patients who were not hospitalized. However, there is no technical impediment to estimating such more refined models if the comparator data are available.⁵⁰ This would be a valuable expansion of our analysis in future work.

Seventh, suicide risk is only one of many reasons for psychiatric hospitalization⁴⁷ and only one of many outcomes that hospitalization may affect. In recognition of these facts, an ITR of the sort developed herein should not be the sole arbiter of admission decisions even it were confirmed.

Conclusions

In this predictive analytics analysis of retrospective VHA data, we found that psychiatric hospitalization is associated with reduced average SA risk for patients in the immediate aftermath of an SA but not for those with SI only or more remote SAs. In each group, an ITR accounting for between-patient heterogeneity could identify substantial proportions of patients whose SA risk would be significantly increased or decreased by hospitalization. Implementing such an ITR has the potential to prevent SAs while requiring fewer hospitalizations than current care.

Supplement 1.

eMethods. Detailed Methods

eAppendix. Baseline Administrative Predictors

eTable 1. Social Determinants of Health: Geospatial Indicators

eTable 2. Medications Classified by FDA as Increasing Risk of Suicide as of September 2022

eTable 3. Unadjusted Suicide Attempt Outcomes by Diagnosis

eTable 4. Adjusted Estimated Average Treatment Effect by Primary Presenting Diagnosis

eTable 5. Estimated Heterogeneity of Treatment Effects Among Patients With NAP in the Test Sample Based on Predictions Made in the Training Sample

eTable 6. Estimated Heterogeneity of Treatment Effects Among Patients With BD in the Test Sample Based on Predictions Made in the Training Sample

eTable 7. Estimated Heterogeneity of Treatment Effects Among Patients With MDD in the Test Sample Based on Predictions Made in the Training Sample

eTable 8. Estimated heterogeneity of Treatment Effects Among Patients With Other Primary Disorders in the test Sample Based on predictions Made in the Training Sample

eTable 9. The Percentage of Patients the ITR Would Be Applied to (ie, Either a Recommendation to Hospitalize or a Recommendation Not to Hospitalize)

eTable 10. Pearson Correlation Between Individual-Level CATE and Whether the Patient Was Hospitalized

eTable 11. Mean Relative SHAP Values by Domain

eFigure 1. The 20 Variables With Highest SHAP_P in Predicting CATE in the Subsample of Patients With NAP and Suicide Attempt in the Past Day^a

eFigure 2. The 20 Variables With Highest SHAP_P in Predicting CATE in the Subsample of Patients With BD and Suicide Attempt in the Past Day^a

eFigure 3. The 20 Variables With Highest SHAP_P in Predicting CATE in the Subsample of Patients With MDD and Suicide Attempt in the Past Day^a

eFigure 4. The 20 Variables With Highest SHAP_P in Predicting CATE in the Subsample of Patients With Other and Suicide Attempt in the Past Day^a

eFigure 5. The 20 Variables With Highest SHAP_P in Predicting CATE in the Subsample of Patients With NAP and Suicide Attempt in the Past

2-7 days

eFigure 6. The 20 Variables With Highest SHAP_P in Predicting CATE in the Subsample of Patients With BD and Suicide Attempt in the Past

2-7 days

eFigure 7. The 20 Variables With Highest SHAP_P in Predicting CATE in the Subsample of Patients With MDD and Suicide Attempt in the Past

2-7 days

eFigure 8. The 20 Variables With Highest SHAP_P in Predicting CATE in the Subsample of Patients With Other and Suicide Attempt in the Past

2-7 days

eFigure 9. The 20 Variables With Highest SHAP_P in Predicting CATE in the Subsample of Patients With NAP and SI Without a Recent Suicide Attempt

eFigure 10. The 20 Variables With Highest SHAP_P in Predicting CATE in the Subsample of Patients With BD and SI Without a Recent Suicide Attempt

eFigure 11. The 20 Variables With Highest SHAP_P in Predicting CATE in the Subsample of Patients With MDD and SI Without a Recent Suicide Attempt

eFigure 12. The 20 Variables With Highest SHAP_P in Predicting CATE in the Subsample of Patients With Other and SI Without a Recent Suicide Attempt

eFigure 13. Any Psychotherapy in the 2 Years Prior to the ED/UC Visit

eFigure 14. Any Digestive System Disease in the 2 Years Prior to the ED/UC Visit

eFigure 15. Any External Cause of Morbidity in the 12 Months Prior to the ED/UC Visit

eFigure 16. Any Cocaine Abuse in the 2 Years Prior to the ED/UC Visit

eFigure 17. Any Chronic Pain in the 12 months Prior to the ED/UC Visit

eFigure 18. Hospitalized for Any Mental Health Disorder in the 5 Years Prior to the ED/UC Visit

eFigure 19. Any Housing or Economic Problems in the 2 Years Prior to the ED/UC Visit

eFigure 20. County Medicaid Eligibility Rate^a

eFigure 21. Number of Psychotherapy Treatment Visits in the 2 Years Prior to the ED/UC Visit

eFigure 22. Number of Remitted Alcohol Dependence Visits in the 5 Years Prior to the ED/UC Visit

eReferences

Click here for additional data file.^{(3.4MB, pdf)}

Supplement 2.

Data Sharing Statement

Click here for additional data file.^{(15KB, pdf)}

References

1.Garnett MF, Curtin SC, Stone DM. Suicide mortality in the United States, 2000–2020. NCHS data brief No. 433. March 7, 2022. Accessed March 3, 2023. https://stacks.cdc.gov/view/cdc/114217 [PubMed]
2.Owens PL, McDermott KW, Lipari RN, Hambrick MM. Emergency department visits related to suicidal ideation or suicide attempt, 2008–2017. Healthcare Cost and Utilization Project (HCUP): statistical brief 263. September 2020. Accessed March 4, 2023. https://hcup-us.ahrq.gov/reports/statbriefs/sb263-Suicide-ED-Visits-2008-2017.jsp
3.Jacobs DG, Baldessarini RJ, Conwell Y, et al. Practice guideline for the assessment and treatment of patients with suicidal behaviors. November 2003. Accessed March 2, 2023. https://psychiatryonline.org/pb/assets/raw/sitewide/practice_guidelines/guidelines/suicide.pdf
4.US Department of Veterans Affairs, US Department of Defense . VA/DoD clinical practice guideline: assessment and management of patients at risk for suicide: version 2.0-2019. May 2019. Accessed February 10, 2023. https://www.healthquality.va.gov/guidelines/MH/srb/VADoDSuicideRiskFullCPGFinal5088212019.pdf
5.Steeg S, Quinlivan L, Nowland R, et al. Accuracy of risk scales for predicting repeat self-harm and suicide: a multicentre, population-level cohort study using routine clinical data. BMC Psychiatry. 2018;18(1):113. doi: 10.1186/s12888-018-1693-z [DOI] [PMC free article] [PubMed] [Google Scholar]
6.Saab MM, Murphy M, Meehan E, et al. Suicide and self-harm risk assessment: a systematic review of prospective research. Arch Suicide Res. 2022;26(4):1645-1665. doi: 10.1080/13811118.2021.1938321 [DOI] [PubMed] [Google Scholar]
7.Wang Y, Bhaskaran J, Sareen J, Bolton SL, Chateau D, Bolton JM. Clinician prediction of future suicide attempts: a longitudinal study. Can J Psychiatry. 2016;61(7):428-432. doi: 10.1177/0706743716645287 [DOI] [Google Scholar]
8.Nock MK, Millner AJ, Ross EL, et al. Prediction of suicide attempts using clinician assessment, patient self-report, and electronic health records. JAMA Netw Open. 2022;5(1):e2144373. doi: 10.1001/jamanetworkopen.2021.44373 [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Large MM, Kapur N. Psychiatric hospitalisation and the risk of suicide. Br J Psychiatry. 2018;212(5):269-273. doi: 10.1192/bjp.2018.22 [DOI] [PubMed] [Google Scholar]
10.Jones G, Gavrilovic JJ, McCabe R, Becktas C, Priebe S. Treating suicidal patients in an acute psychiatric day hospital: a challenge to assumptions about risk and overnight care. J Ment Health. 2008;17(4):375-387. doi: 10.1080/09638230701504981 [DOI] [Google Scholar]
11.Waterhouse J, Platt S. General hospital admission in the management of parasuicide: a randomised controlled trial. Br J Psychiatry. 1990;156:236-242. doi: 10.1192/bjp.156.2.236 [DOI] [PubMed] [Google Scholar]
12.Steeg S, Emsley R, Carr M, Cooper J, Kapur N. Routine hospital management of self-harm and risk of further self-harm: propensity score analysis using record-based cohort data. Psychol Med. 2018;48(2):315-326. doi: 10.1017/S0033291717001702 [DOI] [PubMed] [Google Scholar]
13.Goldman-Mellor SJ, Bhat HS, Allen MH, Schoenbaum M. Suicide risk among hospitalized versus discharged deliberate self-harm patients: generalized random forest analysis using a large claims data set. Am J Prev Med. 2022;62(4):558-566. doi: 10.1016/j.amepre.2021.08.028 [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Kapur N, Steeg S, Turnbull P, et al. Hospital management of suicidal behaviour and subsequent mortality: a prospective cohort study. Lancet Psychiatry. 2015;2(9):809-816. doi: 10.1016/S2215-0366(15)00169-8 [DOI] [PubMed] [Google Scholar]
15.Steeg S, Carr M, Emsley R, et al. Suicide and all-cause mortality following routine hospital management of self-harm: propensity score analysis using multicentre cohort data. PLoS One. 2018;13(9):e0204670. doi: 10.1371/journal.pone.0204670 [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Carroll R, Corcoran P, Griffin E, et al. Variation between hospitals in inpatient admission practices for self-harm patients and its impact on repeat presentation. Soc Psychiatry Psychiatr Epidemiol. 2016;51(11):1485-1493. doi: 10.1007/s00127-016-1247-y [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Nordstrom K, Berlin JS, Nash SS, Shah SB, Schmelzer NA, Worley LLM. Boarding of mentally ill patients in emergency departments: American Psychiatric Association resource document. West J Emerg Med. 2019;20(5):690-695. doi: 10.5811/westjem.2019.6.42422 [DOI] [PMC free article] [PubMed] [Google Scholar]
18.Paksarian D, Mojtabai R, Kotov R, Cullen B, Nugent KL, Bromet EJ. Perceived trauma during hospitalization and treatment participation among individuals with psychotic disorders. Psychiatr Serv. 2014;65(2):266-269. doi: 10.1176/appi.ps.201200556 [DOI] [PMC free article] [PubMed] [Google Scholar]
19.Husereau D, Drummond M, Augustovski F, et al. ; CHEERS 2022 ISPOR Good Research Practices Task Force . Consolidated Health Economic Evaluation Reporting Standards 2022 (CHEERS 2022) statement: updated reporting guidance for health economic evaluations. BMC Public Health. 2022;22(1):179. doi: 10.1186/s12889-021-12491-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
20.US Department of Veterans Affairs . Health Services Research and Analysis: Corporate Data Warehouse (CDW). Accessed February 1, 2023. https://www.hsrd.research.va.gov/for_researchers/vinci/cdw.cfm
21.Hoffmire C, Stephens B, Morley S, Thompson C, Kemp J, Bossarte RM. VA Suicide Prevention Applications Network: a national health care system–based suicide event tracking system. Public Health Rep. 2016;131(6):816-821. doi: 10.1177/0033354916670133 [DOI] [PMC free article] [PubMed] [Google Scholar]
22.US Centers for Disease Control and Prevention . National Death Index. January 10, 2022. Accessed February 21, 2023. https://www.cdc.gov/nchs/ndi/index.htm
23.Wong MS, Steers WN, Hoggatt KJ, Ziaeian B, Washington DL. Relationship of neighborhood social determinants of health on racial/ethnic mortality disparities in US veterans—mediation and moderating effects. Health Serv Res. 2020;55(suppl 2):851-862. doi: 10.1111/1475-6773.13547 [DOI] [PMC free article] [PubMed] [Google Scholar]
24.Kohane IS, Aronow BJ, Avillach P, et al. ; Consortium For Clinical Characterization of COVID-19 by EHR (4CE) . What every reader should know about studies using electronic health record data but may be afraid to ask. J Med Internet Res. 2021;23(3):e22219. doi: 10.2196/22219 [DOI] [PMC free article] [PubMed] [Google Scholar]
25.US Department of Veterans Affairs VHA . VHA Health Information Management and Health Records, 1907.01— Health Information Governance. April 5, 2021. Accessed July 10, 2023. https://www.va.gov/vhapublications/ViewPublication.asp?pub_ID=9235
26.United States Census Bureau . About the current population survey. November 22, 2021. Accessed July 9, 2023. https://www.census.gov/programs-surveys/cps/about.html
27.United States Census Bureau . American Community Survey (ACS). June 20, 2023. Accessed July 11, 2023. https://www.census.gov/programs-surveys/acs
28.Bickley H, Hunt IM, Windfuhr K, Shaw J, Appleby L, Kapur N. Suicide within two weeks of discharge from psychiatric inpatient care: a case-control study. Psychiatr Serv. 2013;64(7):653-659. doi: 10.1176/appi.ps.201200026 [DOI] [PubMed] [Google Scholar]
29.Large M, Sharma S, Cannon E, Ryan C, Nielssen O. Risk factors for suicide within a year of discharge from psychiatric hospital: a systematic meta-analysis. Aust N Z J Psychiatry. 2011;45(8):619-628. doi: 10.3109/00048674.2011.590465 [DOI] [PubMed] [Google Scholar]
30.Park S, Choi JW, Kyoung Yi K, Hong JP. Suicide mortality and risk factors in the 12 months after discharge from psychiatric inpatient care in Korea: 1989-2006. Psychiatry Res. 2013;208(2):145-150. doi: 10.1016/j.psychres.2012.09.039 [DOI] [PubMed] [Google Scholar]
31.Troister T, Links PS, Cutcliffe J. Review of predictors of suicide within 1 year of discharge from a psychiatric hospital. Curr Psychiatry Rep. 2008;10(1):60-65. doi: 10.1007/s11920-008-0011-8 [DOI] [PubMed] [Google Scholar]
32.Bachmann S. Epidemiology of suicide and the psychiatric perspective. Int J Environ Res Public Health. 2018;15(7):1425. doi: 10.3390/ijerph15071425 [DOI] [PMC free article] [PubMed] [Google Scholar]
33.Klonsky ED, May AM, Saffer BY. Suicide, suicide attempts, and suicidal ideation. Annu Rev Clin Psychol. 2016;12:307-330. doi: 10.1146/annurev-clinpsy-021815-093204 [DOI] [PubMed] [Google Scholar]
34.O’Connor RC, Nock MK. The psychology of suicidal behaviour. Lancet Psychiatry. 2014;1(1):73-85. doi: 10.1016/S2215-0366(14)70222-6 [DOI] [PubMed] [Google Scholar]
35.Imbens GW, Rubin DB. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press; 2015, . doi: 10.1017/CBO9781139025751 [DOI] [Google Scholar]
36.Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41-55. doi: 10.1093/biomet/70.1.41 [DOI] [Google Scholar]
37.Desai RJ, Franklin JM. Alternative approaches for confounding adjustment in observational studies using weighting based on the propensity score: a primer for practitioners. BMJ. 2019;367:l5657. doi: 10.1136/bmj.l5657 [DOI] [PubMed] [Google Scholar]
38.Breiman L. Random forests. Mach Learn. 2001;45(1):5-32. doi: 10.1023/A:1010933404324 [DOI] [Google Scholar]
39.Funk MJ, Westreich D, Wiesen C, Stürmer T, Brookhart MA, Davidian M. Doubly robust estimation of causal effects. Am J Epidemiol. 2011;173(7):761-767. doi: 10.1093/aje/kwq439 [DOI] [PMC free article] [PubMed] [Google Scholar]
40.Hernán MA, Robins JM. Causal Inference: What If. CRC Press/Chapman & Hall; 2023. [Google Scholar]
41.Wager S, Athey S. Estimation and inference of heterogeneous treatment effects using random forests. J Am Stat Assoc. 2018;113(523):1228-1242. doi: 10.1080/01621459.2017.1319839 [DOI] [Google Scholar]
42.Athey S, Tibshirani J, Wager S. Generalized random forests. Ann Stat. 2019;47(2):1148-1178. doi: 10.1214/18-AOS1709 [DOI] [Google Scholar]
43.Tibshirani J, Athey S, Friedberg R, et al. Generalized random forests: package ‘grf’. May 11, 2023. Accessed February 1, 2023. https://cran.r-project.org/web/packages/grf/grf.pdf
44.Coyle JR. Package tmle3: the extensible TMLE framework. Accessed March 4, 2023. https://tlverse.org/tmle3/
45.Lundberg SM, Lee SI. A unified approach to interpreting model predictions. NeurIPS Proceedings. Accessed April 12, 2023. https://papers.nips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html
46.Greenwell B. fastshap: Fast Approximate Shapley Values, version 0.0.7. Accessed April 23, 2023. https://cran.r-project.org/web/packages/fastshap/index.html
47.Unick GJ, Kessell E, Woodard EK, Leary M, Dilley JW, Shumway M. Factors affecting psychiatric inpatient hospitalization from a psychiatric emergency service. Gen Hosp Psychiatry. 2011;33(6):618-625. doi: 10.1016/j.genhosppsych.2011.06.004 [DOI] [PubMed] [Google Scholar]
48.McBain RK, Cantor JH, Eberhart NK, Huilgol SS, Estrada-Darley I. Adult psychiatric bed capacity, need, and shortage estimates in California—2021. Rand Health Q. 2022;9(4):16. [PMC free article] [PubMed] [Google Scholar]
49.Ross EL, Zuromski KL, Reis BY, Nock MK, Kessler RC, Smoller JW. Accuracy requirements for cost-effective suicide risk prediction among primary care patients in the US. JAMA Psychiatry. 2021;78(6):642-650. doi: 10.1001/jamapsychiatry.2021.0089 [DOI] [PMC free article] [PubMed] [Google Scholar]
50.Nie X, Wager S. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika. 2020;108(2):299-319. doi: 10.1093/biomet/asaa076 [DOI] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials