Using Recursive Partitioning to Predict Presence and Severity of Suicidal Ideation Amongst College Students

Abstract

Objective:

Predicting the presence and severity of suicidal ideation in college students is important, as deaths by suicide amongst young adults have increased in the past 20 years.

Participants:

We recruited college students (N = 5494) from ten universities across eight states.

Method:

Participants answered three questionnaires related to lifetime and past month suicidal ideation, and an indicator of suicidal ideation in a DSM-5 symptom measure. We used recursive partitioning to predict the presence, absence, and severity, of suicidal ideation.

Results:

Recursive partitioning models varied in their accuracy and performance. The best-performing model consisted of predictors and outcomes measured by the DSM-5 Level 1 Cross-Cutting Symptom Measure. Sexual orientation was also an important predictor in most models.

Conclusions:

A single measure of DSM-5 symptom severity may help universities understand suicide severity to promote targeted interventions. Though further work is needed, as similar scaling amongst predictors could have influenced the model.

Introduction

Machine learning algorithms are becoming more prevalent in suicide research^1,2 as traditional, theory-driven models tend to predict the presence of suicidal ideation and behaviors little better than chance.^3,4 Improving the identification of young adults with suicidal ideation is also becoming increasingly important, as death by suicide by young adults has trended upwards since the year 2000.⁵ In the present study, we aim to utilize machine learning, specifically recursive partitioning, to predict the presence, absence, and severity of suicidal ideation amongst a large sample of college students.

Overview of suicide and life-threatening behaviors.

Death by suicide is one of the leading causes of death amongst young adults aged 17 to 25 from the years 2017 to 2021, even though a small percentage of people that experience suicidal ideation go on to attempt or die by suicide.⁶ Previous research indicates that close to 24% of college students experience suicidal ideation, with over 9% of college students reporting a suicide attempt.⁷ It is important to understand the risk and protective factors that can influence suicidal thoughts and behaviors (STB). One theory, the interpersonal psychological theory of suicide (IPTS), posits that individuals have a separate desire and capability for suicide.⁸ The capability for suicide is believed to result from being repeatedly exposed to painful experiences (i.e., non-suicidal self-injury, intravenous drug use, etc.).⁸ The IPTS distinguishes the desire for suicide by two other constructs, thwarted belongingness and perceived burdensomeness.⁹ Thwarted belongingness refers to the unmet needs to belong with other individuals and perceived burdensomeness refers to the unmet need to contribute socially.^8–10

Risk factors for STB are related to the IPTS through thwarted belongingness and perceived burdensomeness. Individuals with certain mental health disorders associated with social withdrawal (i.e., depression, social phobias, psychotic disorders) are likely to experience increases in suicidal ideation due to thwarted belongingness and perceived burdensomeness.^11,12 Individuals with anxiety and increased impulsivity may also be at risk for high suicidal ideation due to the fear of being negatively evaluated by peers.¹³ Factors such as substance use might be related to suicidal ideation through perceived burdensomeness, with less associations to thwarted belongingness for most substances.^11,14 However, cannabis use specifically may be related suicidal ideation through an interaction with thwarted belongingness and perceived burdensomeness, such that cannabis use is related to suicidal ideation through perceived burdensomeness but only for individuals with high thwarted belongingness.¹⁵

In addition to mental health disorders, other stigmatized and minority groups may also be at greater risk for suicide due to thwarted belongingness and perceived burdensomeness.¹⁶ Individuals belonging to sexual and gender minorities (i.e., bisexual, lesbian, gay; trans, gender neutral, etc) experience greater suicidal ideation than individuals in the gender and sexual majority.^17–21 The relationship between sexual and gender minorities and STB could be due to the minority stress, or that stigma and discrimination cause a social environment that is detrimental to minorities and therefore causes increased mental health problems.²⁰ Suicidal ideation amongst sexual minority college students is correlated with both perceived burdensomeness and thwarted belongingness.¹⁹ One study examining the experience of bisexual, gay, and lesbian young adults with minority stress indicated that sexual minority stress at baseline was related to increased suicidal ideation at follow-ups through thwarted belongingness and perceived burdensomeness.¹⁷ Though, social support from others in the sexual minority community tends to decrease the relationship between thwarted belongingness and suicidal ideation; prompting a call for interventions targeting community building.²²

College interventions for STB tend to focus on gatekeeper training, or training individuals who are often in contact with students or family members in distress to identify students at risk for STB.^23,24 Identified students are then referred to campus assessment and treatment services. Though, the demand for college counseling services by far outweighs the supply of college counselors, resulting in long wait times to see counselors for therapy.²⁵ Reviews of gatekeeper programs show that they tend to improve the knowledge of suicide, increase suicide prevention self-efficacy, and decrease stigma about suicide amongst individuals trained.²⁶ Although gatekeeper training has demonstrated promising results, there is a continued need to adapt treatment to specific student populations (including sexual minorities) and synthesize the effects on outcomes.^24,27

Machine Learning and STB.

Traditional models identifying risk factors that may lead to STB tend to test a small number of predictors in isolation to assess theories, which may limit their ability to make accurate predictions.³ However, over the past 50 years, thousands of variables have been assessed as risk factors for suicide behaviors. The predictability of traditional models does not appear to have improved much in that time.³ One method of improving the classification and analysis of suicide behaviors and risk factors is incorporating machine learning models, specifically recursive partitioning, into suicide research.²⁸ Machine learning models can identify important predictors of STB among many potential risk factors, overcoming the weaknesses of focusing on only a few predictors. In their review, Heckler and colleagues²⁸ found that much of the work using machine learning to assess suicidal ideation examines text from social media through natural language processing. However, natural language processing models are not applicable to quantitative data, as they require text input. In another review, Burke and colleagues found that studies utilizing machine learning to examine STB had a wide range in the number of predictors included in the models; from the low 20s to over 1000 predictors. More recently, work has focused on examining important predictors of STB using several different machine learning models to find important, singular predictors of risks and protective factors.²⁹ However, recursive partitioning provides a unique machine learning approach that examines interactions between different risk factors.

Recursive Partitioning Overview

Recursive partitioning, often labeled decision, classification, or regression trees, is a machine learning algorithm that results in easily interpreted ‘trees’ due to their visual outputs.³⁰ These trees differ from other machine learning outputs, as the trees provide visual output for how variables interact, as opposed to variable importance metrics typically used in other machine learning algorithms that do not provide information on variable interactions within the models (i.e., black-box machine learning models).³¹ Decision trees use nonparametric regressions to establish the cut-point of a predictor that best approximates homogeneity between two groups on an outcome (a partition). The algorithm can use a predictor(s) multiple times in the same tree, recursively including all variables after each node until a stopping point is reached (e.g., a minimum node size). The utility of decision trees over traditional modeling in making predictions is established due to the nonparametric algorithm making no assumptions on the linearity of the relationship between predictors and outcomes³⁰ and the algorithm’s ability to use many predictors, potentially more predictors than participants.³²

Recursive Partitioning and suicide research.

Different applications of machine learning have been used in STB.³³ Decision trees have been implemented to predict STB amongst several different samples, including community samples^34–36, patients in primary care³⁷, and college students.¹ In one study assessing risk of suicidal ideation amongst adolescents and young adults ages 14–24, depression and a history of alcohol use were the best predictors in classifying risk of suicidal ideation.³⁸ Another study assessing suicidal ideation via decision trees in college students indicated that depression, two constructs related to nonsuicidal self-injury, and whether participants engaged in nonsuicidal self-injurious behaviors alone were the most important predictors of suicidal ideation.¹ Burke and colleagues¹ included race, gender, and age as predictors in their models but did not include other demographic factors, such as sexual orientation. Despite a growing body of literature utilizing machine learning to predict suicide behaviors, few studies have used recursive partitioning to examine STB among college students.^1,33 Moreover, few studies include sexual orientation as a predictor and explore whether it interacts with other risk factors.

Present Study

The present study’s first aim was to build upon existing literature using machine learning approaches to help determine risk profiles of suicidal ideation amongst a large sample of college students using decision trees. Our analyses will explore the main and interactive effects of many predictors to establish a decision tree that could be used for early identification of students who may be at risk for experiencing suicidal ideation. Our approach is exploratory, as is the nature of decision trees. Therefore, we do not make any a priori predictions about which predictor(s) will establish main and interactive effects in determining suicidal ideation.

Materials and Methods

Participants

A convenience sample of 5494 psychology students were recruited from 10 public universities in 8 states across the U.S. (AK, CA, CO, ID, NM, TX, VA, WA). Participants completed an online survey for partial course credit. A random missingness design was implemented to decrease participant burden. All participants were assigned to complete the Columbia Suicide Severity Rating Scale (see Trees 1–4). Participants were randomized to complete 9 or 18 secondary measures (50%), which included the DSM-5 Level 1 Cross-Cutting Symptom Measure. In our sample, 45.19% completed the DSM-5 Level 1 Cross-Cutting Symptom Measure (see Tree 5). Further information regarding participants and recruitment can be viewed in other manuscripts with this sample.^39,40 The Institutional Review Board approved these procedures of record and electronic informed consent was obtained from participants. See Tables 1 & 2 for the prevalence of each outcome and demographic variable (respectively), including information about missingness.

Table 1.

Outcome Variables

Lifetime Suicide (Y/N; n = 5059)
Yes	34.78%
No	57.30%
Missing	7.92%
Past month Suicide (Y/N; n = 5026)	39.75%
Yes	27.45%
No	64.03
Missing	8.52%
Lifetime Suicide Severity score (model n=5064)
0	54.64%
1	6.48%
2	6.04%
3	7.74%
4	5.66%
5	4.35%
6	7.26%
Missing	7.83%
Past Month Suicide severity score (model n=2060)
0	23.43%
1	6.17%
2	3.11%
3	2.80%
4	1.09%
5	0.53%
6	0.36%
Excluded/Missing	62.50%
DSM Cross-cutting Suicide score (model n = 2483)
0	34.26%
1	4.86%
2	3.57%
3	1.62%
4	0.89%
Excluded/Missing	54.81%
Total Sample Size	5494

Table 2.

Demographics.

Sex
Male	27.65%
Female	68.07%
Other	0.16%
Missing	4.10%
Gender
Male	25.90%
Female	63.11%
Transgender	0.42%
Other	1.57%
Missing	9.00%
Race/Ethnicity
American Indian/Alaska Native	2.38%
Asian	14.29%
Black/African American	11.52%
Native Hawaiian/Pacific Islander	1.02%
White	61.58%
Other	9.21%
Missing	5.27%
Ethnicity
Not Hispanic/Latino	68.24%
Yes, Mexican	21.56%
Yes, Cuban	0.45%
Yes, Puerto Rican	0.94%
Yes, Other	4.53%
Missing	4.28%
Marital Status
Never Married	91.24%
Married	3.46%
Separated	0.22%
Divorced	0.84%
Widowed	0.04%
Missing	4.20%
Sexual Orientation
Exclusively heterosexual	69.91%
Mostly heterosexual	13.75%
Equally hetero- and homosexual	6.53%
Mostly homosexual	1.87%
Exclusively homosexual	2.18%
Other	1.46%
Missing	4.30%

Note. Race/ethnicity were assessed with separate checkbox items (i.e., could select multiple options).

Measures

We included a total of 202 predictor variables in each of our models. For a brief description and complete list of all our predictors, see Supplementary Table 1. Many of our predictors are related to substance use (i.e., substance use history, emotion regulation, contexts in which individuals use in) and include other variables that may predict substance use. We included most of these variables for analyses as substance use tends to be related to suicidal ideation, and many factors predicting substance use (i.e., emotion dysregulation, impulsivity) are also closely related to STB.^{13,15,41–43} The number of predictors we used in our models falls in the low-to-middle range of studies using machine learning to predict STB.³³ For simplification and space, we present information on the measures that were included as outcomes and important measures that our models found as significant predictors below (Table 3). The median time to complete the survey was 42.675 minutes.

Table 3.

Descriptive Statistics of Outcomes and Important Predictors

Variable	M	SD	% Missing/Excluded
Columbia Suicide Severity Rating Scale – Lifetime	1.408	2.026	7.827%
Columbia Suicide Severity Rating Scale- Past Month	0.800	1.303	62.5%
DSM-5 Level 1 Cross Cutting Symptom Measure: Suicide	0.452	0.926	54.81%
Difficulties in Emotion Regulation Scale: Strategies subscale	2.142	0.997	54.21%
Difficulties in Emotion Regulation Scale: Nonacceptance subscale	2.293	1.161	54.22%
Difficulties in Emotion Regulation Scale: Goals subscale	2.640	1.197	54.19%
Index of Autonomous Functioning: Interest taking subscale	3.509	1.056	54.53%
Dysregulation Index	2.108	0.392	53.68%
DSM-5 Level 1 Cross Cutting Symptom Measure: Depression	2.802	2.258	54.55%
DSM-5 Level 1 Cross Cutting Symptom Measure: Psychotic Symptoms	0.569	1.333	54.57%
DSM-5 Level 1 Cross Cutting Symptom Measure: Repetitive Thoughts & Behaviors	1.312	1.888	54.59%

Note: Bolded items are outcome measures.

Suicidal Ideation.

We used a 12-item screener version of the Columbia Suicide Severity Rating Scale (CSSRS)⁴⁴ assessed on a binary response scale (0=no, 1=yes). Each item was administered to assess lifetime (CSSRS-L; 6 items, α, “In your lifetime, have you wished you were dead or wished you could go to sleep and not wake up?”) and past month (CSSRS-M; 6 items, α=.79 “In the past month, have you actually had any thoughts of killing yourself?”) suicidal ideation. We assessed outcomes of suicidal ideation via a dichotomized presence or absence of lifetime and past month suicidal ideation dichotomously (if any CSSRS items were endorsed for past month/lifetime). Severity of suicidal ideation was measured continuously by including the total CSSRS (lifetime and past month) as the outcome variable.

Mental Health.

We used the 23-item DSM-5 Level 1 Cross-Cutting Symptom Measure^45,46 to assess how much (or how often) participants have been “bothered by the following problems” in the past 2 weeks. Each item is answered on a 5-point response scale (0=None or not at all, 4=Severe or nearly every day), and the scale assesses 13 distinct mental health domains including depression (2 items, α=.85), anger (1 item), mania (2 items, α=.64), anxiety (3 items, α=85), somatic symptoms (2 items, α=.70), psychosis (2 items, α=.79), sleep problems (1 item), memory (1 item), repetitive thoughts and behaviors (2 items, α=.77), dissociation (1item), personality functioning (2 items, α=.83), substance use (3 items, α=.73), and suicidal ideation (1 item). The suicidal ideation item (“Thoughts of actually hurting yourself?) was used as an outcome (see Tree 5), whereas the remaining scales were entered as predictors. Although developed to aid clinical decision-making for clients seeking psychiatric services, Bravo et al. demonstrated that the scale has good psychometric properties in a college student sample similar to the present study.⁴⁷

Emotion Dysregulation.

Facets of emotion dysregulation were assessed using the 18-item Difficulties in Emotion Regulation Scale Short Form (DERS-SF)⁴⁸ assessed on a 5-point response scale (1=Almost Never (0–10%), 5=Almost Always (91–100%)). Like the original measure (DERS; Gratz & Roemer, 2004), this short form assesses six facets of emotion dysregulation: nonacceptance of emotional responses (3 items, α=.91, “When I’m upset, I feel guilty for feeling that way”), difficulty with goal-directed behavior (3 items, α=.95, “When I’m upset, I have difficulty getting work done”), impulsive control difficulties (3 items, α=.94, “When I’m upset, I become out of control”), lack of emotional awareness (3 items, α=.91, “When I’m upset, I acknowledge my emotions”, reverse-coded), limited access to emotion regulation strategies (3 items, α=.87, “When I’m upset, I believe there is nothing I can do to make myself feel better”), and lack of emotional clarity (3 items, α=.87, “I have difficulty making sense out of my feelings”).

Psychological Dysregulation.

Psychological dysregulation was assessed using the 43-item Dysregulation Inventory (DI)⁴⁹ assessed on a 4-point response scale (0=never true, 3=always true). This measure contains affective (“My mood goes up and down without reason”), behavioral (“I get very fidgety after a few minutes if I am supposed to sit still”), and cognitive (“I spend time thinking about how to reach my goals”) domains and was originally validated as a risk factor for the development of substance use disorder. The total score predicted substance use severity longitudinally among a child/adolescent sample, supporting its predictive validity.⁴⁹ The scale had high internal consistency in our sample (α=.914).

Demographics.

We administered a demographics questionnaire that includes age, sex, gender identity, year in school, race, ethnicity, marital status, military status, political spectrum, and sexual orientation. Sexual orientation was assessed on a 6-point continuous response scale (1=Exclusively heterosexual, 2=Mostly heterosexual, 3=Equally heterosexual and homosexual, 4=Mostly homosexual, 5=Exclusively homosexual, 6=Other). Age, year in school, and sexual orientation were entered as continuous variables, and all other demographic variables were entered as dummy-coded factors.

Statistical Analysis

First, we limited the number of predictors by removing item-level variables from each of the scales included in the models. We removed created variables that were examining combinations of contexts of using cannabis and alcohol (i.e., location of use, who they use with, contexts by biological sex), and kept only the initial contextual variables. We conducted recursive partitioning using the ‘rpart’ package in the R statistical environment.^50,51 The ‘rpart’ package uses a recursive partitioning algorithm that finds the variable that most accurately splits the data into two separate groups or nodes. For classification trees, splits aim to maximize the separation of the different classes (participants who reported STB and those who did not). For regression trees, the goal of a split is to minimize the variance of the dependent variable in each subset (participant severity of STB). The algorithm is then applied to best split each of these two groups of the decision tree, and the process is repeated.³⁰ This first tree is typically too complex and 10-fold cross-validation is used to prune the tree to a smaller size by comparing the cross-validation and minimum error of the tree. ⁵⁰ We did not set a maximum number of splits in the trees but did set the minimum node size to 20 to help prevent overfitting.

First, we used classification trees to examine the presence or absence of lifetime and past month suicidal ideation. Then, we used regression trees to examine the severity of each suicide outcome variable. Classification trees are used to build a tree with a binary outcome, while regression trees are used to build a tree with continuous outcomes. The same predictors were included in each model (see Supplementary Table 1). We calculated the area under the curve (AUC), sensitivity, specificity, and positive predictive value (PPV) of each classification tree model (Supplementary Table 2) and the Mean Squared Error (MSE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE) in each decision (regression) tree model (Supplementary Table 3).

In our models, sensitivity refers to the ability of the models to correctly classify a participant as having STB, specificity refers to the ability of the models to correctly classify a participant as not having STB, and PPV refers to the percentage of participants with a positive result who actually have STB. The MSE can be interpreted as the average squared differences between what the model predicted and the actual values. The RMSE is the square root of the MSE and can be interpreted as the average distance between the predicted and actual values in the same units as the dependent variables. Lastly, the MAE is the average absolute difference between the predicted and actual values. Lower values for the MSE, RMSE, and MAE indicate better model fit, though no true cutoff can be determined as they are dependent upon the scale of the outcome measure. Considering the scaling of our outcome variables (CSSRS: 0 – 6; DSM-SUI: 0–4), values under one for the MSE, RMSE, and MAE would likely indicate a good fit.

Regarding missing data, the rpart package removes cases where the dependent variable is missing. For rates regarding missing data on the outcomes, see Table 3. For missing data on predictor variables, the rpart package uses surrogate splitting. When a value for a predictor is missing, the algorithm examines all other predictors that are not missing and identifies the variable that has the strongest association with the variable being split and uses this value to determine the split.⁵⁰

Results

Classification Tree 1 (Presence of lifetime suicidal ideation).

We trained a classification tree to examine the presence and absence of lifetime suicidal ideation measured by the CSSRS-L version.⁴⁴ Figure 1 shows the equal weight pruned tree generated by recursive partitioning analysis; the percentages at the bottom of each node are the likelihoods that participants in that node would be in the group classified as having a lifetime history of suicidal ideation. Two splits were considered optimal by the model. Individuals that endorsed an exclusively heterosexual sexual orientation were classified into the no lifetime history of suicidal ideation group (30.1% chance of a participant in this node being classified as having suicidal ideation in their lifetime), and the not exclusively heterosexual group (58.6% percent change of a participant in this node being classified as having suicidal ideation in their lifetime) further split into those with more than a 2.2 on the goals subscale of the DERS (sample M = 2.64), which were classified as having no lifetime history (38.4% chance of being classified as having suicidal ideation in their lifetime), and those with less than 2.2 on the goals subscale, which were classified as having a lifetime history of suicidal ideation (63.6% chance of being classified as having suicidal ideation in their lifetime). The goals subscale of the DERS assesses a person’s difficulty engaging in goal directed behavior, with higher scores indicating greater difficulties. Classification tree 1 indicated that the probability of individuals classified with lifetime suicidal ideation did indeed have lifetime suicidal ideation (positive predictive value; PPV) was 0.694. The model’s achieved sensitivity, or the ability of our model to correctly identify all people who have a lifetime history of suicidal ideation, was 0.873 and the achieved specificity, or the ability of our model to correctly identify people who do not have a lifetime history of suicidal ideation, was 0.364. Lastly, our model resulted in an area under the curve (AUC) was equal to 0.626, indicating the probability of our model correctly distinguishing between the presence or absence of lifetime suicidal ideation is 62.6%. While in traditional models, an AUC of 0.626 is not considered great predictability, our achieved AUC does show a small improvement over historical AUCs in the STB research field (i.e., AUCs = 0.56–0.58).³

Figure 1:

Likelihood of Experiencing Suicidal Ideation in Lifetime

Classification Tree 2 (Past month suicidal ideation).

The best fitting model classifying the presence and absence of past month suicidal ideation as measured by the CSSRS-M⁴⁴ had four splits (Figure 2). Individuals that were exclusively heterosexual were classified into the no past month suicidal ideation group (22.4% chance of a participant in this node being classified as having suicidal ideation in the past month). Two more splits occurred for those that were not exclusively heterosexual (50.7% chance of being classified as having suicidal ideation in the past month): if individuals had a low score (less than 2.2, sample M =2.29) on DERS nonacceptance subscale, i.e., less difficulties in accepting their emotional distress, they were classified as having no past month suicidal ideation (30.7% chance of being classified as having suicidal ideation in the past month). For those indicating greater nonacceptance (DERS nonacceptance> 2.2), if they scored greater than 2.2 on the dysregulation inventory (sample M = 2.11), they were classified into the group with a presence of past month suicidal ideation (68.0% chance of being classified as having suicidal ideation in the past month). Furthermore, individuals had a dysregulation inventory score of less than 2.2 were further split by the interest taking subscale of the index of autonomous functioning with those scoring less than 3.7 (sample M = 3.51) classified as not having past month suicidal ideation (42.0%% chance of being classified as having suicidal ideation in the past month), and those with IAF interest taking subscale scores greater than 3.7 being classified as having past month suicidal ideation (58.0% chance being classified as having suicidal ideation in the past month). The interest taking subscale of the index of autonomous functioning measures the tendency for individuals to reflect on inner and outer events that facilitates insight to experiences. The model resulted in an AUC of 0.646, a PPV of 0.755, a sensitivity of 0.924, and a specificity of 0.300. The model metrics indicate that our overall model is slightly more accurate than the average range of models examining STBs in the past. However, the model did correctly identify individuals who do have STB (i.e., high sensitivity).

Figure 2:

Likelihood of Experiencing Suicidal Ideation in Past Month

Regression Tree 1 (Severity of lifetime suicidal ideation).

We used a regression tree to examine lifetime suicide severity measured by the CSSRS-L. Participants achieved an average CSSRS-L score of 1.41 (SD=2.03; See Supplementary Figure 1 for decision tree and Supplementary Table 3 for fit indices). Regression trees report the average lifetime suicide severity experienced by individuals at each node. For lifetime suicide severity scores, two variables contributed to the pruned decision tree: sexual orientation and the strategies subscale of the DERS. The strategies subscale assesses the belief that one can do little to regulate emotions, with higher scores indicating greater endorsement of the belief. Individuals (n=1362) that endorsed a sexual orientation other than exclusively heterosexual (>1.5) experienced an average CSSRS-L score of 2.456, which is higher than the sample average. Individuals (n=3,701) who endorsed being almost exclusively heterosexual (<1.5) experienced an average CSSRS-L score of 1.022, similar to the sample average. This group was split based on their DERS strategies subscale (sample M = 2.14). Individuals (n=2,908) who reported less difficulties implementing emotion regulation strategies (<2.17) experienced an average CSSRS-L score of 0.85, less than the sample mean (M = 1.408). Individuals (n=4,794) that reported problems implementing emotion regulation strategies (>2.17) experienced an average CSSRS-L score of 1.644, higher than the sample mean. Overall, the regression tree accounted for 9.85% of the variance in lifetime suicide severity rating. The average squared difference between the actual data and where the model predicted individuals would be (MSE) was equal to 3.62. The average difference between the observed CSSRS-L scores and the predicted values (RMSE) was equal to 1.90, and the absolute value of the difference between the actual CSSRS-L score and the predicted value (MAE) was equal to 1.55. Considering the scoring scale of the CSSRS-L (0 to 6), these numbers indicate a relatively poor fit of our model predicting the severity of lifetime suicidal ideation.

Regression Tree 2 (Severity of past-month suicidal ideation).

In the second regression tree, we trained a model to predict recent suicide severity via the CSSRS-M (Posner et al., 2011). Participants (N = 2060) reported an average CSSRS-M score of 0.80 (SD=1.30; See Supplementary Figure 2 for decision tree and Supplementary Table 3 for fit indices). Two variables contributed to the decision tree model predicting CSSRS-M scores: depression severity from the DSM-5 level 1 cross-cutting scale (DSM-DEP; American Psychiatric Association, 2013; Narrow et al., 2013) and the strategies subscale of the DERS. The initial split was based on the DSM-DEP (sample M = 2.80) score. Individuals (n=139) who reported greater depression (>6.5) experienced an average CSSRS-M score of 1.99. Individuals (n=1,921) with low to moderate DSM-DEP scores (<6.5) experienced an average CSSRS-M score of 0.71. This group split was based on their scores on the strategies subscale of the DERS. Specifically, individuals (n=1583) who endorsed weaker beliefs that they could not use strategies to regulate their emotions (<2.83) achieved an average CSSRS-M score of 0.59. Individuals (n=338) endorsed higher beliefs that they could not use strategies to regulate their emotions. (>2.83) achieved an average CSSRS-M score of 1.28. Overall, the decision tree accounted for 6.07% of the variance in past month’s suicide severity and achieved an MSE of 1.53, a RMSE of 1.23, and an MAE of 0.93. The CSSRS-M was scored on a 0 to 6 scale, and while an MSE and RMSE greater than one do not indicate the best fitting model, an MAE under one indicates that, on average, our model prediction was within one likert scale point from the actual participant value.

Regression Tree 3 (DSM-5 Cross-cutting measure suicide subscale).

Lastly, we trained a model to predict suicidal ideation symptoms from the DSM-5 cross-cutting scale (DSM-SUI). Participants (N = 2483) achieved an average DSM-SUI score of 0.45 (SD=0.93). Three variables contributed to the decision tree predicting DSM-SUI: severity of repetitive thoughts and behaviors from the DSM-5 cross-cutting scale (DSM-REP; sample M = 1.31), severity of psychotic symptoms from the DSM-5 cross-cutting scale (DSM-PSY; sample M = 0.57), and the DSM-DEP (sample M = 2.80) scale mentioned previously (Supplementary Figure 3). Individuals (n=1,950) that reported less repetitive thoughts and behaviors (DSM-REP<2.5) experienced an average score of 0.143 on the DSM-SUI scale. This group was split based on DSM-PSY scores. Individuals (n=104) that reported more psychotic symptoms (>1.5) experienced an average of 1.01 on the DSM-SUI scale. Individuals (n=1,846) that reported fewer psychotic symptoms (<1.5) experienced an average of 0.143 on the DSM-SUI. This group was split based on scores on the DSM-DEP scale. Individuals (n=1,846) that reported less depression (<4.5) achieved an average DSM-SUI score of 0.06. Individuals (n=271) that reported greater depression scores (>4.5) achieved average DSM-SUI scores of 0.60.

Returning to the top of the tree, individuals (n=533) that reported more repetitive thoughts and behaviors (>2.5) achieved average DSM-SUI scores of 1.41. This group was split based on scores on the DSM-PSY scale. Individuals (n=206) that reported higher psychotic symptoms (<2.5) achieved a DSM-SUI score of 2.05. Individuals (n=327) that reported lower psychotic symptoms (>2.5) experienced an average score of 1.01 on the DSM-SUI. This node was split based on DSM-DEP scores. Individuals (n=259) that reported lower DSM-DEP scores (<6.5) achieved an average DSM-SUI score of 0.77. Lastly, individuals (n=68) that reported higher DSM-DEP scores (>6.5) achieved an average DSM-SUI score of 1.96. Overall, the regression tree accounted for 39.40% of the variance in DSM-SUI scores, achieved an MSE of 0.46, a RMSE of 0.68, and an MAE of 0.37. Overall, these numbers indicate that the models achieved a good fit in predicting DSM-SUI scores.

Discussion

Approximately 41% of participants reported some history of lifetime suicidal ideation. Among those who reported lifetime suicidal ideation, approximately 38% reported suicidal ideation within the past month. We employed five recursive partitioning models, two classification models and three decision tree models, to identify the best predictors of lifetime and past month suicidal ideation. The classification trees had PPV’s ranging from 69% to 76% and AUC’s from 0.62 to 0.64. The accuracy of these models tend to be slightly lower than other machine learning models used in suicide research. ³³ Though, we did not have a variable that could account for past suicide attempts, which tends to strongly correlate with suicidal ideation and is included in other models examining STB. ^1,33

The decision trees were more variable in their accuracy, with R²’s ranging from 6% to 39%. The data predicts the presence/absence of STB better than its severity, a construct that may be harder to predict using cross-sectional self-report data. Though two of the regression trees MAE’s were within one scale unit of the outcome variables for the CSSRS-M and the DSM-5 cross cutting suicide measure. This result extends previous literature in which suicidal ideation severity outcome measures are often manually dichotomized to presence or absence of suicidal ideation.^28,33 Our classification tree models assessing the presence or absence of lifetime and previous month suicidal ideation tended to have good to great sensitivity and poor specificity, indicating a proportionally lower number of false negatives compared to false positives. However, given the strong relationship between suicidal ideation and suicide attempts⁵², clinicians would likely prefer an assessment that provides more false positives than false negatives.

Three of the five models shared the same initial split, sexual orientation. In each of the three models with sexual orientation as the initial split, individuals who were more likely to endorse a sexual orientation other than strictly heterosexual were at a greater risk of experiencing suicidal ideation. This result is line with the IPTS, in that sexual minorities tend to experience increased thwarted belongingness and perceived burdensomeness that put them at greater risks for STB. ^17,21,53 In each of the trees where sexual orientation was the first split, the second split related to emotion dysregulation via the difficult in emotion regulation scale (DERS). The strategies subscale presented as an important leaf node in the two models assessing the lifetime and past month (sexual orientation was not the initial split) severity of suicidal ideation via the CSSRS. The nonacceptance construct of the DERS was an important leaf node in the model classifying past month suicide. Lastly, the difficulty engaging in goal directed behavior construct of the DERS was an important leaf node in the model classifying the presence or absence of lifetime suicidal ideation.. Individuals with difficulty regulating emotions, especially negative emotions, tend to have a higher probability of being depressed and anxious, both important risk factors for suicidal ideation.^54,55 Additionally, our results regarding emotion regulation are in line with previous recursive partitioning models examining STB in college students, where affect regulation was a split in predicting past week suicidal ideation. ¹

Results surrounding sexual orientation in combination with emotion dysregulation indicate that there could be additional factors to consider when implementing suicide prevention programs on campus. Traditionally, these programs are conducted through gatekeeper training, or training someone regularly in contact with students on knowledge about suicide and processes to connect to campus services for referral. ²⁴ Though our study needs to be replicated with new samples, the results suggest that campus prevention methods could be tailored to sexual minority individuals. Specifically, gatekeeper training could have close ties to LGBTQ+ groups and centers on campus. Including these groups in gatekeeper training may also help mitigate the factors related to IPTS, as thwarted belongingness and perceived burdensomeness also tend to decrease with increased support from sexual minority community members. ²² In addition, college counseling services may consider providing emotion regulation strategy workshops to similar groups, or students that have already been identified as at risk to improve coping skills regarding difficult emotions.

Our best performing model, a decision tree examining suicidal ideation via the DSM-5 Level 1 cross-cutting symptom measure, accounted for approximately 39% of the variance in the outcome. Each node in the decision tree was from the DSM-5 cross-cutting measure: repetitive thoughts and behaviors, psychotic symptoms, and depression symptoms. The DSM-5 cross-cutting measure was created to assess symptom domains that tend to cut across different diagnoses. Given that we included a large number of predictor variables, a decision tree with variables stemming from a single measure was unexpected. While this result could be promising in reducing the number of measures needed to classify suicidal ideation, our DSM-5 cross-cutting measure decision tree could have been influenced by how the DSM-5 cross-cutting measure domains are all scaled from 0 to 4. Additionally, the identified predictors could also reflect the severity of overall mental health disorders, and thus a proxy for severity of suicidal ideation. Future work could compare the performance of machine learning models using features from the DSM-5 cross-cutting measure compared to other measures mapped onto the DSM-5 cross-cutting measure domains.

Limitations

Our study has several limitations worth considering. First, it measured the predictors and outcomes cross-sectionally. Thus, we cannot determine if our results will be replicated in longitudinal studies. Instead, our results likely only indicate important markers of suicidal ideation. Similarly, we assessed suicidal ideation, rather than suicide behaviors. Therefore, we cannot confirm whether the risk factors identified in our models would also influence suicide behaviors such as planning or attempts, and we did not have variables accounting for planning or previous attempts. Additionally, the model predicting the DSM-5 cross-cutting measure of suicidal ideation resulted in a decision tree that only had nodes from the DSM-5 cross-cutting measure. While this could potentially be viewed as a strength in limiting the number of measures needed to screen for suicidal ideation, the results could be due to the different components being assessed on the same scale. Future studies replicating our tree or validating the tree with a different dataset could help determine whether the item scaling in the DSM-5 cross-cutting measure influenced the recursive partitioning. Lastly, the specificity of our decision tree models was poor. While our achieved sensitivities were high, an ideal model would have a smaller number of both false positives and false negatives.

Future Directions

The use of machine learning applications to examine suicidal ideation is growing and could continue to expand in new directions. Our recursive partitioning models could be replicated or validated using another dataset. In addition, sex-stratified trees (or building different trees based on biological sex) could also help inform future precision medicine targets by examining differences in trees based on sex. Validation of our models in another dataset would allow us to understand our decision trees’ accuracy better. Extending our current work by including longitudinal data could help determine what predictors best predict suicidal ideation over time. Whether longitudinal data is used, future research should assess the deployment of machine learning models into the clinical world. That is, build predictive models, test and validate them, then deploy those algorithms to be used in mental health and medical settings, potentially making the jobs of individuals conducting diagnostic or clinical assessments less burdensome and more targeted. Lastly, using machine learning analytical methods and a range of variables limited to the IPTS and minority stress models may also provide information regarding nonparametric interactions between variables. This could help bolster theories and provide more targeted clinical implications for college campuses.

Conclusion

We used recursive partitioning to examine the presence or absence of lifetime and past month suicidal ideation, and the severity of suicidal ideation using three separate measures. The recursive partitioning models tended to err towards caution, resulting in decreased false negatives (sensitivity) and a larger proportion of false positives (specificity) and were overall less accurate than other machine learning models. Sexual orientation and facets of emotion regulation were the most prevalent predictors in the models, indicating a relatively good fit with existing literature on suicide and sexual minority stress. Machine learning approaches are promising in suicide risk assessment with easily interpretable visual results which may be particularly useful in the initial screening of college students. Further work is needed to replicate our findings and apply new machine learning approaches to assessing suicide risk.