Predictors of Intentional Self -Harm Among Medicaid Mental Health Clinic Clients In New York

Abstract

Background:

Behavioral health outpatients are at risk for self-harm. Identifying individuals or combination of risk factors could discriminate those at elevated risk for self-harm.

Methods:

The study population (N = 248,491) included New York State Medicaid-enrolled individuals aged 10 to 64 with mental health clinic services between November 1, 2015 to November 1, 2016. Self-harm episodes were defined using ICD-10 codes from emergency department and inpatient visits. Multi-predictor logistic regression models were fit on a subsample of the data and compared to a testing sample based on discrimination performance (Area Under the Curve or AUC).

Results:

Of N = 248,491 patients, 4,224 (1.70%) had an episode of intentional self-harm. Factors associated with increased self-harm risk were age 17–25, being female and having recent diagnoses of depression (AOR=4.3, 95%CI: 3.6–5.0), personality disorder (AOR=4.2, 95%CI: 2.9–6.1), or substance use disorder (AOR=3.4, 95%CI: 2.7–4.3) within the last month. A multi-predictor logistic regression model including demographics and new psychiatric diagnoses within 90 days prior to index date had good discrimination and outperformed competitor models on a testing sample (AUC=0.86, 95%CI:0.85–0.87).

Limitations:

New York State Medicaid data may not be generalizable to the entire U.S population. ICD-10 codes do not allow distinction between self-harm with and without intent to die.

Conclusions:

Our results highlight the usefulness of recency of new psychiatric diagnoses, in predicting the magnitude and timing of intentional self-harm risk. An algorithm based on this finding could enhance clinical assessments support screening, intervention and outreach programs that are at the heart of a Zero Suicide prevention model.

1. Introduction

Suicide rates in the United States have increased in the past decade (Curtin et al., 2016), with 48,344 deaths and 1.4 million suicide attempts in 2017 (United States Centers for Disease Control and Prevention, 2018). Suicide has emerged as the 10th leading cause of death for all ages and the 2nd for ages 10 to 44 (National Center for Injury Prevention and Control, 2016). In 2018, there were almost 500,000 emergency department visits as result of self-harm (United States Centers for Disease Control and Prevention, 2018). Besides causing economic and societal burden, intentional self-harm, a superordinate classification including both suicide attempts and non-suicidal self-directed injuries, is a leading predictor of subsequent suicide deaths (Bostwick et al., 2016; Olfson et al., 2017). The risk of suicide death is approximately 30 times higher among self-harm patients than in the general population (Cooper et al., 2005; Olfson et al., 2017). Almost 13% of the individuals who had attempted eventually die by suicide (Suominen et al., 2004). Though the risk of suicide death is higher within a year of initial attempt (Bostwick et al., 2016), a history of suicide attempt is considered as a crucial indicator of risk for suicide death (Suominen et al., 2004). Understanding the characteristics of individuals most at risk for suicide attempt and intentional self-harm is expected to increase identification, support treatment and to prevent completed suicide.

Past research has found that the majority of people who died by suicide or attempted suicide used health care services within one year prior to their death or attempt (Ahmedani et al., 2014; Fontanella et al., 2017; Kammer et al., 2021) underscoring the opportunity for suicide prevention within the healthcare system. The National Strategy for Suicide Prevention highlighted that inadequate suicide risk detection in healthcare settings was one critical component in need of improvement in order to reduce suicide deaths (U. S. Department of Health and Human Services, 2012). To improve suicide prevention, the National Action Alliance has endorsed the “Zero Suicide” (ZS) model (Labouliere et al., 2018; National Action Alliance for Suicide Prevention, 2011) which includes identification of patients at-risk for suicide as one of seven key elements necessary to target the aspirational goal of eliminating suicide deaths within healthcare settings.

A wealth of research has been dedicated to identifying risk factors for suicide and self-harm. Several demographic factors (male gender, age over 50 and white race), history of suicide attempts, recent discharge from a psychiatric inpatient unit with a depressive disorder diagnosis and certain psychiatric disorders (including major depressive disorder, substance use disorder, conduct disorder, anxiety disorder) are associated with increased risk of suicide death (Bostwick et al., 2016; Doshi et al., 2020; Haukka et al., 2008; Olfson et al., 2017, 2016; Suominen et al., 2004). When examining intentional self-harm with or without intent to die, demographic risk factors shift indicating higher risk for females and young adults (Chartrand et al., 2015; Mars et al., 2019, 2014). Though intentional self-harm includes both suicide attempt and non-suicidal self-harm, most research on self-harm cannot distinguish the prognostic implications of suicide attempt with clear intent to die from non-suicidal self-harm (Butler and Malone, 2013; Kapur et al., 2013; Mars et al., 2014). However, high suicidality and non-suicidal self-harm are significant independent predictors of a subsequent suicide attempt (Wilkinson et al., 2011). In fact, one study identified that an act of either type of self-harm is associated with a greatly increased risk of suicide over a 4-year follow-up (Cooper et al., 2005).

Understanding risk factors has not substantially improved efforts to identity and predict which individuals will engage in suicidal behavior (Franklin et al., 2017). Furthermore, clinical risk assessments to predict suicidal behavior (based on identified risk factors) perform only slightly better than chance (Tran et al., 2014), limiting the ability of clinicians to properly identify imminent suicidal behavior (Carter et al., 2017; Runeson et al., 2017). More recently, there is a growing effort to develop risk models to increase the accuracy of identification by leveraging large electronic health record and other administrative data using machine learning techniques to predict suicide attempts and death among different populations and time periods. Specifically, models have been developed for outpatient behavioral health patients (Simon et al., 2018), outpatient and inpatient populations (Barak-Corren et al., 2017), and among Veterans (Kessler et al., 2017). Using machine learning, Walsh et al. (2017), developed models based on EMR data from individuals with a history of self-harm in 7–720 days and Ribeiro and colleagues (2019) developed models using online survey respondents between 3 and 28 days(Ribeiro et al., 2019). Across models, a combination of substance abuse, diagnosis of mental disorder, recent hospitalization, and prior suicide attempt emerged as the strongest predictors for future suicidal behavior. A meta-analysis of 17 studies indicated that cumulative risk models, particularly short-term prediction, yielded significant improvements in the prediction of suicidal behaviors compared to risk assessments relying on risk factors alone (Belsher et al., 2019).

Although research has been conducted to predict suicidal behavior in different populations using different data sources, there is a gap in the literature in predicting intentional self-harm among the population with mental health diagnoses covered by Medicaid, a government-funded health care coverage program for low income individuals, families and children, that offers a full range of health services, including, but not limited to, doctor and clinic visits, emergency room visits and hospitalizations (New York State Department of Health, 2013) . Understanding suicide risk within the Medicaid population is important given the size (over 6 million people in New York (New York State Department of Health, 2018)) and that patients with severe mental illness have a history of higher utilization of public insurance (Rowan et al., 2013). Higher rates of suicide in Medicaid youth also confirm the importance of Medicaid as a “boundaried” suicide prevention setting (Fontanella et al., 2019; National Action Alliance for Suicide Prevention: Research Prioritization Task Force, 2014).

The aim of this study is to identify demographic and diagnostic risk factors for predicting future intentional self-harm among individuals served in mental health clinics using one year of New York State Medicaid claims data (New York State Office of Mental Health, 2019). We hypothesized that: risk scores built using measures readily available in Medicaid claims data including demographic (younger age, female gender, race, rural vs. urban domicile and low income vs. disability aid type), and behavioral health diagnostics variables (depressive disorders, bipolar disorder, schizophrenia, personality disorders and substance use disorders) would discriminate among those with and without an intentional self-harm event. In addition, we will test whether including information on the timing/recency of new behavioral health diagnoses may further improve the accuracy of the model.

2. Methods

2.1. Data source and study population

Data for the study population was extracted from the New York State Office of Mental Health (NYSOMH) database (New York State Office of Mental Health, 2019), a database that includes all claim and encounter data for Medicaid enrollees in New York State who received a mental health service. Mental health service in this context includes all service provided by any NYSOMH licensed clinic (New York State Office of Mental Health, 2021, 2020), including, but not limited to, assessment (initial and psychiatric diagnosis), psychotherapy (individual, family/-collateral and group), psychotropic medication treatment including injectable psychotropic medication administration with education and monitoring, crisis intervention and complex care management. The NYSOMH database includes billing information, including diagnosis codes for self-harm, not just from clinics, but also from emergency services, ambulances, medication prescriptions and hospitalizations. The eligible population (N = 248,491) included Medicaid eligible individuals ages 10 to 64, with a mental health clinic service between November 1, 2015 to November 1, 2016. Inclusion criteria included continuous Medicaid eligibility during this period (defined as not having an enrollment gap in Medicaid for longer than 45 days) (National Committee for Quality Assurance, 2018). Note that enrollment gaps can occur for many reasons, change in financial circumstances being one of them. Exclusions included individuals eligible for both Medicaid and Medicare as well as those 65 years old or older, who would primarily be eligible for Medicare, a health insurance program administered by the federal government for persons who are 65 years of age or older, or have certain disabilities (U.S. Centers for Medicare and Medicaid Services, 2020). Individuals eligible for Medicare were not included because most of their services would not be included in the Medicaid claims database. The Nathan Kline Institute and New York State Psychiatric Institute Institutional Review Boards reviewed the project and determined that the activity described does not meet the definition of human subject’s research as delineated by Department of Health and Human Services regulations [45 CFR Part 46] and Food and Drug Administration regulations [21 CFR Part 50 and 56].

2.2. Outcome measure

The outcome for each subject was an indicator (yes/no) for a suicide attempt /intentional self-harm episode during the 12-month period between November 1, 2015 to November 1, 2016. From here suicide attempt and intentional self-harm episode both together will be referred as Self-harm episode .The International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) diagnostic codes were used to define self-harm diagnoses following the Center for Disease Control guidance for identifying suicide attempts and intentional self-harm using ICD-10 data (Hedegaard et al., 2018b). Specifically, the list of self-harm diagnoses included codes from T36-T71 and X71-X83 and suicide attempt, ICD-10 code T1491. Although ICD-10 diagnosis codes for intentional self-harm do not allow any distinction between self-harm with and without intent to die (Hedegaard et al., 2018b), it is important note that they are closely related behaviors and previous literature indicates that most individuals who show up to an emergency department and receive a self-harm diagnosis had attempted suicide (Chartrand et al., 2015; Randall et al., 2017; Stanley et al., 2018). We have used an algorithm developed by Kammer and colleagues (Kammer et al., 2018) to define self-harm episodes. To ensure a single episode suicide attempt/self-harm event was not captured more than once by diagnoses given by different providers (e.g., potentially an outpatient provider, ambulance, emergency department, and inpatient) involved in responding to the crisis, only those self-harm episodes that indicated an emergency department visit or inpatient stay were included. Briefly, start and end dates for each self-harm episode were calculated for continuous dates of service from the emergency department visit dates of service and inpatient admission and discharge dates. For those who had more than one self-harm episode between November 1, 2015 to November 1, 2016 period, the most recent episode (closest to November 1, 2016) was used as index date, while the earlier self-harm episodes may have counted for the history of self-harm variable among the predictors.

For each patient, a past history of a self-harm episode was also defined as having a self-harm episode as defined above, within 3 months, and 1 year prior to their index episode, even if that crossed to the time before the November 1, 2015 date. For example, if a client had an index date (self-harm episode or first visit) on November 1,2015, their past self-harm information was established based on data from November 1, 2014 to October 31, 2015.

2.3. Index date

Each individual was assigned an index date between November 1, 2015 to November 1, 2016. Specifically, for individuals with one or more self-harm event, the index date was the last self-harm event during that period. For individuals without a self-harm event the index date was November 1, 2016.

2.4. Demographic and medicaid eligibility variables

Demographic characteristics included age (categorized into the following intervals: 10–16, 17–25, 26–39, 40–53 and 54–64, years), sex (male or female), race/ethnicity (White, African American, Asian, Hispanic, and Unknown) and region of domicile (rural /urban, based on county level population density larger than 1000 people per square mile in the 2010 census). Medicaid eligibility status was also included: all those receiving Social Security Income were considered as eligible for Medicaid due to a disability and all others were considered as belonging to the income aid category.

2.5. Psychiatric diagnosis

Based on the literature for diagnoses comorbid with self-harm, among the potential risk factors we included primary diagnosis of schizophrenia, depression, bipolar, personality disorder and substance related disorder. More precisely, a diagnosis was considered “present” if the relevant ICD code appeared as a primary diagnosis in any service claims within the 12 months prior to the index date. To examine the importance of the recency of the psychiatric diagnoses as a predictor of self-harm, given that a primary diagnosis may be a new diagnosis, or a longstanding, recurrent illness, individuals with each of the above diagnosis were categorized according to the days between the index date and the first record of the diagnosis (1–30 days, 31–90 days, 91–180 days, 181–365 days before the index date).

2.6. Statistical analysis

All analyses were conducted using SAS software, Version 9.4. First, the prevalence of individuals with self-harm was calculated for each category of every risk factor (variables defined in 2.4, and 2.5 above). Five multivariable logistic regression models were conducted to determine the adjusted odds ratio (AOR) and 95% confidence intervals, testing the hypothesis that demographics and diagnosis information can predict the risk of self-harm better than chance alone. Models of increasing complexity were fit, and the time frame of the predictors also was changed from 3 month to 1 year, to identify whether the addition of diagnosis and self-harm, and the increase of the time the predictors covered, increased the discrimination statistic of the model significantly. Model 1 included the demographic characteristics only. Model 2 included demographic variables and indicator variables for the five diagnoses of interest in the year prior to the index date (depression, schizophrenia, bipolar, personality disorder, substance use disorder). Model 3 added a new indicator variable for history of self-harm within 12 months prior to the index date. Model 4 used the same variables as model 2; however, it only considered primary diagnoses that were first observed within 3 months of the index date. Model 5 included the same variables and time frames as model 4 and added a history of a self-harm event within 3 months prior to the index date.

To build and validate the predictive models of self-harm, we randomly divided our study population into training (67%) and testing (33%) samples. Each individual risk factor had similar distribution in both samples with the average mean difference being 0.1%. Logistic regression with self-harm status as dependent variable were fit on the training sample for each model, and then the model coefficients were used for predicting the outcome in the testing sample. Receiver Operator Characteristic (ROC) curves were graphed for both training and testing samples based on predicted probability and observed event indicator for each subject to illustrate the sensitivity and specificity, or true positive and false positive rates, at different classification thresholds. The area under the ROC curve or concordance statistic (AUC or C-Statistics) was also estimated to measure the discrimination or the power of the model to separate those with and without self-harm, at different cutoff values of the probability. Taking values between 0 and 1, with larger values indicating better discrimination, and 0.5 being the neutral value, this measure reflects the proportion of pairs of individuals whose actual outcome and predicted probabilities agree among all possible pairs. The nonparametric approach was used to compare the AUC values. The standard errors and 95% confidence intervals were also calculated and presented.

3. Results

3.1. Prevalence

Approximately 1.7% (4224 out of 248,491) of the study sample had at least one episode of self-harm during the year. Among these, almost 14% (603 out of 4224) had a history of self-harm within 1 year before their index date.

Among 248,491 Medicaid clinic clients, there were 1207 episodes with a suicide attempt diagnosis (0.49%) and 3017 episodes with intentional self-harm diagnosis (1.21%), equivalent to a roughly 2:5 distribution of the two types of behavior in the combined outcome (see Table 1).

Table 1.

Prevalence and characteristics of suicide attempt/intentional self-harm in Medicaid population with mental health clinic services within the period Nov 01, 2015 to Nov 01, 2016.

Characteristics	N	Suicide Attempt /Self Harm		Suicide Attempt		Intentional Self Harm
		Episode	%	Episode	%	Episode	%
Total	248,491	4224	1.70%	1207	0.49%	3017	1.21%
Age 10–16	50,232	805	1.60%	258	0.51%	547	1.09%
17–25	37,642	1119	2.97%	327	0.87%	792	2.10%
26–39	60,664	1117	1.84%	298	0.49%	819	1.35%
40–53	57,869	842	1.46%	221	0.38%	621	1.07%
54–64	42,084	341	0.81%	103	0.24%	238	0.57%
Gender
Male	105,576	1500	1.42%	448	0.42%	1052	1.00%
Female	142,915	2724	1.91%	759	0.53%	1965	1.37%
Race
White	106,769	2316	2.17%	644	0.60%	1672	1.57%
African American	67,408	798	1.18%	246	0.36%	552	0.82%
Hispanic	26,237	339	1.29%	91	0.35%	248	0.95%
Asian	6768	102	1.51%	35	0.52%	67	0.99%
Other/unknown	41,309	669	1.62%	191	0.46%	478	1.16%
Region
Rural	82,645	2221	2.69%	584	0.71%	1637	1.98%
Urban	165,846	2003	1.21%	623	0.38%	1380	0.83%
Aid category
Disability	83,343	1224	1.47%	352	0.42%	872	1.05%
Income	165,148	3000	1.82%	855	0.52%	2145	1.30%
Diagnosis 12 months before index date1
Schizophrenia diagnosis	36,965	929	2.51%	262	0.71%	667	1.80%
1–30 days before	873	54	6.19%	18	2.06%	36	4.12%
31 −91 days before	2007	117	5.83%	36	1.79%	81	4.04%
91–180 days before	3629	135	3.72%	41	1.13%	94	2.59%
181–365 days before	30,456	623	2.05%	167	0.55%	456	1.50%
No schizophrenia diagnosis	211,526	3295	1.56%	945	0.45%	2350	1.11%
Depressive Disorder	113,836	2481	2.18%	734	0.64%	1747	1.53%
1–30 days before	3418	199	5.82%	71	2.08%	128	3.74%
31 −91 days before	7946	284	3.57%	90	1.13%	194	2.44%
91–180 days before	14,247	454	3.19%	115	0.81%	339	2.38%
181–365 days before	88,225	1544	1.75%	458	0.52%	1086	1.23%
No depressive Disorder	134,655	1743	1.29%	473	0.35%	1270	0.94%
Bipolar Disorder	39,595	1344	3.39%	375	0.95%	969	2.45%
1–30 days before	1182	55	4.65%	17	1.44%	38	3.21%
31 −91 days before	2797	131	4.68%	34	1.22%	97	3.47%
91–180 days before	4836	193	3.99%	59	1.22%	134	2.77%
181–365 days before	30,780	965	3.14%	265	0.86%	700	2.27%
No Bipolar Disorder	208,896	2880	1.38%	832	0.40%	2048	0.98%
Personality Disorder	5695	324	5.69%	105	1.84%	219	3.85%
1–30 days before	349	37	10.60%	11	3.15%	26	7.45%
31 −91 days before	758	58	7.65%	20	2.64%	38	5.01%
91–180 days before	1199	86	7.17%	25	2.09%	61	5.09%
181–365 days before	3389	143	4.22%	49	1.45%	94	2.77%
No Personality Disorder	242,796	3900	1.61%	1102	0.45%	2798	1.15%
Substance related Disorder	52,482	1520	2.90%	374	0.71%	1146	2.18%
1–30 days before	1505	81	5.38%	20	1.33%	61	4.05%
31 −91 days before	3407	126	3.70%	31	0.91%	95	2.79%
91–180 days before	6349	233	3.67%	60	0.95%	173	2.72%
181–365 days before	41,221	1080	2.62%	263	0.64%	817	1.98%
No Substance related Disorder	196,009	2704	1.38%	833	0.42%	1871	0.95%

¹Prevalence of suicide attempt/intentional self-harm were observed by the timing of diagnosis within 12 months period. Subjects can be assigned multiple diagnoses in the same time period and will be counted for each diagnostic category.

3.2. Demographic risk factors

Regarding age, the highest prevalence of self-harm was found among the 17–25 age group (2.9%) (see Table 1 and also Table 2 for odds ratios adjusted for all demographic and diagnostic covariates). Prevalence rate of self-harm was 35% higher among women (1.91%) than men (1.42%). With regard to race, higher prevalence of self-harm was found in white patients (2.17%) compared to other ethnic groups. Self-harm was more than twice as common among the patients in rural areas than urban (2.69% vs. 1.21%) and this difference was not explained away by adjusting for demographic or diagnostic differences. In terms of eligibility for Medicaid, those eligible for reasons of income had slightly higher prevalence of self-harm (1.82%) than those eligible because of disability (1.47%); however, after adjusting for other covariates the odds of self-harm for the two eligibility groups were not significantly different from each other.

Table 2.

Unadjusted and adjusted odds ratios of risk factors of suicide attempt/ intentional self-harm in Medicaid population with mental health clinic services within the year (Nov 1, 2015 to Nov 1, 2016).

Characteristics	Un adjusted	Model 1: Demographics		Model 2: Demographics + primary diagnosis
		AOR	(95% CI)	AOR	(95% CI)
Age
10–16	1.99	1.87	(1.6 - 2.1)	2.87	(2.50–3.28)
17–25	3.75	3.39	(3.0 - 3.8)	3.38	(2.98–3.84)
26–39	2.30	2.01	(1.8 - 2.3)	1.71	(1.51–1.95)
40–53	1.81	1.71	(1.5 - 1.9)	1.51	(1.33–1.72)
54–64	Ref	Ref		Ref
Gender
Male	0.74	0.74	(0.70–0.79)	0.70	(0.65–0.75)
Female	Ref	Ref		Ref
Race
White	Ref	Ref		Ref
African	0.54	0.71	(0.65–0.78)	0.68	(0.63–0.74)
American
Hispanic	0.59	0.75	(0.67–0.85)	0.80	(0.71–0.90)
Asian	0.69	0.92	(0.75–1.13)	1.02	(0.83–1.25)
Other/	0.74	0.86	(0.79–0.95)	0.91	(0.83–1.00)
Unknown
Region
Rural	2.26	1.92	(1.80–2.05)	1.87	(1.7–1.99)
Urban	Ref	Ref		Ref
Aid category
Disability	0.81	1.04	(0.97–1.11)	0.92	(0.86–1.0)
Income	Ref	Ref		Ref
Primary diagnosis
Schizophrenia diagnosis
1–30 days before	4.17			2.49	(1.86–3.35)
31 −91 days before	3.91			2.72	(2.23–3.32)
91–180 days before	2.44			1.82	(1.52–2.19)
181–365 days before	1.32			1.76	(1.61–1.94)
No schizophrenia diagnosis	Ref			Ref
Depressive Disorder
1–30 days before	4.71			4.26	(3.64–4.98)
31 −91 days before	2.83			2.49	(2.19–2.84)
91–180 days before	2.51			2.18	(1.96–2.43)
181–365 days before	1.36			1.37	(1.28–1.47)
No depressive Disorder	Ref			Ref
Bipolar Disorder
1–30 days before	3.49			2.10	(1.58–2.80)
31 −91 days before	3.52			2.38	(1.98–2.87)
91–180 days before	2.97			2.11	(1.80–2.46)
181–365 days before	2.32			2.11	(1.95–2.28)
No	Ref			Ref
Bipolar Disorder
Personality Disorder
1–30 days before	7.26			4.21	(2.92–6.05)
31 −91 days before	5.08			2.79	(2.10–3.71)
91–180 days before	4.73			2.88	(2.28–3.64)
181–365 days before	2.70			1.88	(1.58–2.25)
No	Ref			Ref
Personality Disorder
Substance related Disorder
1–30 days before	4.07			3.41	(2.69–4.32)
31 −91 days before	2.75			2.20	(1.82–2.65)
91–180 days before	2.72			2.39	(2.07–2.75)
181–365 days before	1.92			1.99	(1.84–2.15)
No Substance related Disorder	Ref			Ref

3.3. Psychiatric diagnosis as risk factor

The prevalence of self-harm was higher for each of the five primary psychiatric diagnoses examined (depression, schizophrenia, bipolar disorder, personality disorder, substance use disorder) than for those who did not have the diagnosis (Table 1). The time between the first diagnosis and the index date seemed to matter; specifically, prevalence was highest when the diagnosis was first observed within 30 days or 31–91 days prior to the index date. Adjusted odds ratios (see Table 2) show that, even after adjustment for demographics and co-morbid psychiatric disorders, very high risk persists for those with a new episode of the diagnosis within the last month of depression (OR=4.3, 95%CI: 3.6–5.0), personality disorder (OR=4.2, 95%CI: 2.9–6.1), and substance use disorder (OR=3.4, 95%CI: 2.7–4.3).

The highest prevalence of self-harm for any risk factor examined was estimated for those with a new diagnosis of personality disorder within the last 30 days (10.60%) (see Table 1); to study further we examined two different types of personality disorder, Borderline Personality Disorder (BPD) and Anti-social Personality Disorder (ASPD). These two disorders had high and comparable prevalence of self-harm (BPD: 7.48%, ASPD: 7.38%). After adjustment, those diagnosed with ASPD within the last month were at very high risk of self-harm (OR=9.47, 95% CI: 4.11- 21.84) and BPD was associated with high risk if it was first diagnosed within the last 3 months prior to the index date (OR=3.69, 95%CI: 2.57 - 5.31).

3.4. Prediction models

Five multi-predictor logistic regression models were fit with the same outcome variable (self-harm at any point during the 1-year study period). These models were built on the training sample (2/3 of the data) and compared based on the discrimination statistics (AUC or C statistic) on the remaining 1/3 of the sample – the testing sample (see Table 3). Model 1, based only on demographic variables, had the lowest AUC, although still significantly better than chance alone. Model 4, based on demographics and diagnoses indicator variables within the 3 months prior to the index date, had a higher discrimination statistic (testing sample AUC=0.86, 95%CI: 0.85–0.87). Addition of a history of self-harm episode during the last 3 months as a predictor (model 5) did not increase the discrimination statistic significantly (testing AUC=0.86, 95%CI: 0.85–0.87). Widening the time interval for the diagnostic indicator variables to the 12 months prior to the index date (model 2), reduced the performance significantly compared to model 4 (model 2: AUC=0.74, 95%CI: 0.73–0.75). Addition of history of self-harm episode within last 12 months in model 3 (model 3: 0.75, 95%CI: 0.75–0.76) didn’t help to improve the power of the model compared to model 4 and model 5, indicating that the recency of diagnoses has a significant and quantitative contribution to the predictive performance.

Table 3.

Discrimination statistics (AUC statistics) for five multi-predictor models for the suicide attempt/ intentional self-harm outcome in the training and the testing sample.

	Training Sample, n = 166,489					Testing Sample, n = 82,002
Model	Area	Standard Error	(95% CI)			Area	Standard Error	(95% CI)
Model1	0.66	0.01	(0.6524	–	0.672)	0.65	0.01	(0.6391	–	0.6679)
Model2	0.74	0.00	(0.7276	–	0.7466)	0.74	0.01	(0.7227	–	0.7498)
Model3	0.75	0.00	(0.7453	–	0.7643)	0.75	0.01	(0.7409	–	0.7679)
Model4	0.86	0.00	(0.8506	–	0.8666)	0.86	0.01	(0.8505	–	0.8728)
Model5	0.86	0.00	(0.853	–	0.8689)	0.86	0.01	(0.8533	–	0.8755)
Contrast
Model2 - Model1	0.07	0.00	(0.0657	–	0.0842)	0.08	0.01	(0.0683	–	0.0972)
Model3 - Model1	0.09	0.00	(0.0833	–	0.102)	0.10	0.01	(0.0864	–	0.1153)
Model4 - Model1	0.20	0.01	(0.1861	–	0.2068)	0.21	0.01	(0.1925	–	0.2239)
Model5 - Model1	0.20	0.01	(0.1885	–	0.2091)	0.21	0.01	(0.1952	–	0.2265)

Based on a pre-set cut-off value for the predicted probability of self-harm, chosen as the overall prevalence of self-harm in the current sample (1.7%), sensitivity and specificity for each model were calculated and compared across models (Fig. 1). Models 4 and 5 had a sensitivity of 70% and a specificity of 88%, outperforming all competitors on both measures (model 1 SE: 59%, SP: 64%; model 2: 65%, 71%; model 3: 62%, 75%).

Fig. 1.

ROC curves (graph of sensitivity against 1-specificity) for the five prediction models, in the training and the testing sample.

4. Discussion

In this study, we tested a series of predictive models of suicide attempts/ intentional self-harm using Medicaid claims data from approximately 250,000 mental health clinic clients across New York State during a single year. Using data readily available in claims and EMR data, a combination of demographic information and timing of primary behavioral health diagnosis, were good predictors of future intentional self-harm (AUC=0.86).

4.1. Risk factors

Receiving a new primary behavioral health diagnosis within the past 90 days after a period of not having any service for this primary diagnosis for nine months or more had greater predictive power than any other risk factors examined, including having this psychiatric diagnosis at any time within the past year. Furthermore, the study replicated findings from previous studies related to demographics and clinical factors, in that females, whites, young individuals aged 15–34 were at higher risk of self-harm and demonstrated how these differences in risk persist even after adjusting for all the other risk factors using adjusted odds ratios.

These findings are aligned with previous research on the general population where being female, aged 15–34, and being white have been associated with self-harm (Kessler et al., 1999; Olfson et al., 2015; United States Centers for Disease Control and Prevention, 2018). A recent publication on self-harm admissions using data from the National Trauma Data Bank found similar racial differences in risk, where whites were at highest risk; it also found that the age groups 15–24 and 25–34 had the highest risk for non-fatal self-harm (Hanuscin et al., 2018). Prior studies (Sorrell, 2020) report increased risk of fatal self-harm among the elderly; however, our analyses focused on non-fatal self-harm and our inclusion criterion of not using Medicare patients prevented us from estimating risk for those over 65 years old. We also found greater risk of self-harm in rural regions in New York, consistent with NCHS data that highlighted the growing gap in suicide mortality between regions from 1999 to 2017 (Hedegaard et al., 2018a). Importantly, our investigation extended these findings by showing that the regional gap in risk does not disappear after adjusting for demographics and behavioral health diagnosis, underscoring the increased risk for individuals living in rural areas. All mental health diagnoses examined conferred higher risk of self-harm, including personality disorder, bipolar disorder, substance related disorder, schizophrenia and depression, consistent with previous findings (Yeh et al., 2019). Similar to previous studies, we also found that patients with a personality disorder had higher risk of self-harm (Ansell et al., 2015; Trull et al., 2018) and this association was especially high if the diagnosis was recent.

4.2. Multi-predictor models

In addition to investigating risk factors for self-harm, our study identified using multi-predictor models a specific combination of demographic characteristics and new diagnoses within 90 days prior to the index date, with good discrimination (0.70 sensitivity and 0.88 specificity using a probability cutoff value 1.7%). While prior self-harm was an important predictor of subsequent self-harm, consistent with the literature (Olfson et al., 2017) the addition of this indicator variable (“history of self-harm within 90 days prior to index date”) did not substantially improve the predictive power of the model. This may indicate that demographics and underlying diagnosis, in particular a new psychiatric diagnosis, provides overlapping information with the history of self-harm episode to predict future events. Our study design may have reduced variability in terms of history of self-harm among the study sample due to the short time period considered. History of self-harm may have a significant predictive role when we use diagnosis history information over a longer time period, as reflected in model 2 and model 3.

The key contribution of our study is that it provides information on not just who is at highest risk of intentional self-harm among the Medicaid population with clinic based mental health services in New York State, but also when the period of highest risk occurs: a new primary psychiatric diagnosis after a period of not receiving any services for this diagnosis is a potent predictor of self-harm in the short term. Individuals with a new psychiatric diagnosis may include those with new onset of this condition, and those presenting with a relapse or exacerbation of a pre-existing condition. This finding agrees with a study from 2014 utilizing administrative data housed at the Manitoba centre for Health Policy which found that the risk of suicide or suicidal behavior is higher within 3 months after the initial mental health diagnosis (Randall et al., 2014). Our study suggests that individuals should be screened for suicide and self-harm risk when a new psychiatric diagnosis is given (especially depression, schizophrenia, bipolar, personality disorder, and substance use disorder), or when individuals return to psychiatric treatment after a period of 9 months or more, as the risk for self-harm may be greatest in the next 90 days. Simon and colleagues (Simon et al., 2018) examined the risk of suicide within 90 days following a mental health clinic visit, emphasizing the importance of estimating short-term risk of self-harm. Another recent study also developed algorithms to predict suicide with information from 1 week to 2 years before the suicidal event where the models with more recent information had better discriminatory power (Walsh et al., 2017). Our study enhances these findings by suggesting that emergence of a new psychiatric diagnosis is a more potent predictor than the presence of the psychiatric diagnosis in general. Future study is needed to determine the relative risk of a first episode of a new psychiatric condition, and a new episode of a pre-existing psychiatric condition.

4.3. Discrimination of predictive models

Self-harm is generally considered to be a low base rate event as evident from the prevalence of 1.7% in our study population. Models built for low prevalence events often have very low positive predictive value, while models restricted to higher risk populations may have higher positive predictive value. Our best model, which included demographic and new mental health diagnosis within 90 days prior to index date had a positive predictive value of 0.18 in conjunction with sensitivity of 0.54 for the top 5% of the population by risk score (see Table 4). This result suggests that, among individuals predicted to be high risk based on our models, approximately 1 in 5 would have an self-harm event within 90 days after a new mental health diagnosis, even though more than half of the subjects with self-harm are among those in the top 5% of the risk score. Our results compare well with the diagnostic accuracy of recent papers. For example, in the most acceptable model estimated by Simon and colleagues, the model demonstrated a PPV of 0.05 with a sensitivity of 0.44 (Simon et al., 2018).

Table 4.

Performance characteristics of the best predictive model at different cut-points.

Risk Score Percentile Cut points	PPV	NPV	Specificity	Sensitivity
>99.5th	57.7	98.58	99.79	16.95
>99th	42.57	98.73	99.4	25.83
>95th	18.44	99.18	95.85	54.24
>90th	11.18	99.35	90.99	65.63

Predictive modeling that assigns risk scores to mental health patients to predict negative outcome events such as suicide attempt or self-harm has emerged as an important goal in mental health research. Such algorithms, however, should not be used as a substitute for clinical assessments, rather, they should be used in conjunction with them. A high-risk score on our model could be used as one screening mechanism in outpatient health care settings in order to guide further assessment, especially when the timing information is taken into account (i.e., in the short term after a new primary psychiatric diagnosis after a period of not receiving any services for this diagnosis).

Our study has several limitations. The NYS Medicaid population is not representative of the overall population and therefore findings may not be generalizable to the entire population of the U.S. or other regions. Our sample was further restricted to clients with visits in a Mental Health Clinic, although data used in the models included those collected from non-clinic sources, thus our models may not generalize to predicting self-harm in the general NYS Medicaid population. In addition, administrative data underestimates the incidence of suicide attempts (Stanley et al., 2018). Claims data in particular are limited as they depend on the event being reported to Medicaid and may underestimate the true rate of suicide events among the Medicaid population. Medicaid data do not provide important demographic (marital status, living arrangements), socioeconomic (employment, education level), and non-diagnostic psychological (hopelessness, emotion dysregulation, etc.) information about the patient, all of which may be important risk factors. From the risk score usability point of view, the risk scores cannot be re-calculated automatically in real time using claims data because of a substantial claims data lag in centralized databases, however the data used in the predictive models are readily available in electronic medical records. As our suicide episode definition of outcome uses ICD-10 diagnosis codes for suicide attempt and intentional self-harm which do not allow any distinction between intentional self-harm with and without intent to die (Hedegaard et al., 2018b), our definition may include some recipients without intent to die. Our training and testing datasets were random samples, and thus the model needs to be validated in other samples. Future predictive modeling may consider other service settings and populations to validate and examine model performance. Another limitation is that we did not adjust for intensity of treatment in any of our models, although this could explain some of the observed differences - we left this for a future study.

Despite these limitations, our study has a number of strengths. We estimated the risk of self-harm during one year for individuals eligible for Medicaid, representing a population with lower income level. The model was estimated based on a very large sample of administrative data which is crucial for examining a low base rate phenomenon like self-harm risk. Medicaid claims data allowed us to use diagnostic information with greater precision between the time of new diagnosis and risk of self-harm, compared to the information available in some clinical studies without electronic health records.

5. Conclusion

In conclusion, we used demographic characteristics and diagnostic information from Medicaid claims data to identify individuals at risk of future self-harm. Our results highlight the importance of certain demographic and behavioral health diagnostic characteristics, and the timing of diagnosis, in obtaining accurate prediction. Such an algorithm could enhance clinical assessments to support screening for suicide risk. Specifically, if this model could be used to identify brief periods of markedly elevated risk, it would allow clinicians to increase frequency of screening, monitoring and follow-up efforts, and may have potential to prevent a large number of intentional self-harm events.