A machine learning analysis of risk and protective factors of suicidal thoughts and behaviors in college students

Abstract

Objective:

To identify robust and reproducible factors associated with suicidal thoughts and behaviors (STBs) in college students.

Methods:

356 first-year university students completed a large battery of demographic and clinically-relevant self-report measures during the first semester of college and end-of-year (n = 228). Suicide Behaviors Questionnaire-Revised (SBQ-R) assessed STBs. A machine learning (ML) pipeline using stacking and nested cross-validation examined correlates of SBQ-R scores.

Results:

9.6% of students were identified at significant STBs risk by the SBQ-R. The ML algorithm explained 28.3% of variance (95%CI: 28–28.5%) in baseline SBQ-R scores, with depression severity, social isolation, meaning and purpose in life, and positive affect among the most important factors. There was a significant reduction in STBs at end-of-year with only 1.8% of students identified at significant risk.

Conclusion:

Analyses replicated known factors associated with STBs during the first semester of college and identified novel, potentially modifiable factors including positive affect and social connectedness.

Introduction

In the United States, age-adjusted suicide rates increased 33% between 1999 and 2017, particularly for the 15–24 age group, for whom suicide is the second leading cause of death.^1–3 Colleges are particularly affected, where the overall suicide rate is estimated at 7.5 per 100,000.^4,5 Moreover, a large study of nearly 14,000 first-year college students globally found that the 12-month prevalence rates of suicidal ideation, plans, and attempts were 17.2%, 8.8%, and 1.0% respectively.⁶ Additionally, the incidence rate of first onset of suicidal thoughts and behaviors (STBs) in college ranges between 4.6 and 6.4%, and appears to be larger than in the general population.⁷ Fortunately, college campuses present an ideal setting in which suicidality prevention efforts can be implemented. Understanding factors that are associated with increased or decreased STBs in college students will have important implications for informing these efforts.

Several factors likely contribute to STBs among students. First, college students evidence substantially higher rates of depression than the general population (30.6% vs. 9%).⁸ Second, while students with high comorbidity of mental illness prior to matriculation appear to be particularly affected,^9,10 an estimated 9% of depressed and 20% of anxious students are symptom free prior to matriculation.¹¹ The onset or exacerbation of depression and STBs during college likely relates to the fact that students face a multitude of transitional environmental challenges along social, financial, academic, and psychological domains.^12–14 Studies have observed an increase in acute stressors and perceived stress,^15,16 sleep disturbances,² hopelessness/helplessness,^12,17 loneliness or disconnection from others,^2,17 perceived burdensomeness,² poor parent-student relationships,⁴ and increased academic demands.^2,17 Together, these may adversely impact academic engagement and performance and/or lead to maladaptive coping (eg substance abuse, aggression, risky sexual behavior), which, in turn, may have further harmful effects on mental health and increase risk for STBs.^4,12,14

Although the number of students seeking help for serious emotional problems has increased, less than half of those who have considered suicide have sought professional help.¹⁸ Moreover, 80–95% of students who died by suicide never visited their college counseling center,¹⁹ and having STBs is associated with reduced odds of intention to seek treatment.²⁰ Finally, few colleges report being adequately equipped to address serious mental health issues.²¹ Therefore, it is not only necessary to improve assessment and early identification of students with STBs, but also to identify modifiable factors associated with STBs to in turn inform suicide prevention initiatives unique to this population.⁴

Despite several decades of research, our ability to identify individuals at risk for STBs remains limited. In fact, Franklin and colleagues²² have argued that minimal advances have been made in identifying reproducible, non-spurious factors that may correlate with increased or decreased risk for STBs. This is in part due to over-reliance on traditional statistical approaches inherently restricting how many different factors can be simultaneously examined. To address these issues and improve clinical decision-making, the use of sophisticated multivariate statistical analyses, such as machine learning (ML), has been proposed.^23,24 A review of 35 studies utilizing ML to examine suicide risk found greater prediction accuracy over the studies using traditional statistical methods, and furthermore, identified novel correlates of suicide.²³ However, the majority of these studies used available limited medical or university records or did not carry out comprehensive clinical and psychosocial assessments. Furthermore, as with studies employing traditional analyses, these studies did not examine factors that are associated with improvements in psychological well-being and resilience, and which may relate to decreases in STBs.

While findings point to specific risk factors for STBs (eg depression) in students using both traditional and ML statistical approaches, the limited number of explored variables impedes identification of broader factors that can be used in conjunction with each other to, upon further experimental investigation, inform prevention and intervention efforts. Therefore, this study used a data-driven ML framework to model demographic, clinical, and psychosocial correlates of STBs during the first semester of college. The overarching goals were (1) to examine whether algorithms developed based on these measures may offer utility in explaining variability in STB scores in college students, and (2) to identify robust and reproducible correlates of STBs that may be modifiable and thus could be targeted in subsequently experimentally validated individual and/or campus-wide prevention and intervention efforts. Additionally, we conducted an exploratory analysis using the same ML approach to predict STBs at the end of first year of college in this sample, which are reported in the Supplementary Materials.

Materials and methods

Participants

Participants were 356 non-treatment seeking undergraduate students (59.3% female) from a private, mid-Western university who voluntarily enrolled in a longitudinal study examining the impact of a brief, four-session resilience training course on mental health.²⁵ As part of this longitudinal study, 150 (42.13%) participants were assigned to the resilience training following the completion of baseline assessments. Participants completed demographic and self-report measures during their first semester of college (ie baseline) and again at the end of the second semester (ie “end-of-year”). Thirty-six percent withdrew from the study or were lost to follow-up at the end-of-year assessment, for a total of 228 for the end-of-year exploratory analyses. Group differences on baseline variables between participants who completed versus did not complete the end-of-year assessment are reported in the Supplementary Materials (Table S3). Given this attrition rate and a reduction in STB at end-of-year, we focus our analysis and results on the baseline assessment and report the exploratory end-of-year results in the Supplementary Materials.

Participants included in the current study overlap with, but are not identical to, the participants included in a previous publication reporting results from the resilience-focused clinical trial.²⁵ Exclusion criteria for the study included being under 18 years of age, not in the first year of college, or reporting significant mental (ie acute psychosis) or physical health problems requiring immediate medical attention. In accordance with federal and college regulations preventing international students from receiving research compensation, these students were excluded. All study procedures were approved by both the Western Institutional Review Board (IRB) (WIRB) and conducted in accordance with the World Medical Association Declaration of Helsinki. All students provided written informed consent prior to participation and were compensated for their time. The study was registered at the US National Institutes of Health (NIH) (ClinicalTrials.gov: #NCT02982070). The Consolidated Standards of Reporting Trials (CONSORT) diagram is provided in the supplemental Figure S1.

Measures

Measures were completed via secure survey links through Research Electronic Data Capture (REDCap). Risk for suicidal thoughts and behaviors (STBs) was measured using the Suicide Behaviors Questionnaire-Revised (SBQ-R), a self-report instrument that assesses the following, that is (1) history of suicidal ideation and suicide attempt (past month for the purposes of the present study; (2) frequency of suicidal ideation in the past month; (3) communication of suicidal behavior (eg “telling someone that you were going to commit suicide, or that you might do it”; and (4) self-perceived likelihood of future suicidal behavior.²⁶ SBQ-R is widely used and has been shown to be a reliable and valid instrument assessing suicidal ideation and behavior in students.^26–31 A cutoff score of ≥7 (range 3–18) is recommended to identify undergraduates at significant risk for suicide.²⁶ To ensure participant safety in the current study, a more conservative cutoff of ≥5 total score, or ≥4 on item #4, was used to identify participants for further assessment of ideation, plan, and intent by clinically-trained staff using the Columbia-Suicide Severity Rating Scale.³² The analysis and interpretation of the results, however, involved assessment of dimensional variability in SBQ-R scores. Participants also completed 27 clinically relevant demographic, medical history, substance use, positive and negative valence, trauma history, and resilience self-report measures, used to derive unique variables of interest in statistical analyses. Whether or not individuals completed the resilience training program²⁵ was also included as a potential predictor in the exploratory end-of-year analysis (Supplementary Materials). To account for a lack of variability in categorical variables, we collapsed across categories with a relatively small count (see Table 1).

Table 1.

Demographic and clinical characteristics.

Characteristics	n = 356	Range
Age, Mean (SD)	18.79 (1.08)	18.01–29.65
Gender, N (%)		–
Male	144 (40.45%)
Female	211 (59.27%)
Other	1 (0.28%)
Ethnicity, N (%)		–
Hispanic	47 (13.20%)
Non-Hispanic	309 (86.80%)
Race, N (%)a		–
White	237 (66.57%)
Black or African American	24 (6.4%)
American Indian or Alaska Native	12 (3.37%)
Middle Eastern/North African	3 (0.84%)
Asian Indian	7 (1.97%)
Chinese	3 (0.84%)
Japanese	0
Korean	2 (0.56%)
Other Asian	9 (2.53%)
Native Hawaiian or Pacific Islander	0
Some other race	8 (2.25%)
Multi-racial	51 (14.33%)
Resilience Training, N (%)		–
Yes	150 (42.13%)
No	206 (57.87%)
Annual parent or household income, N (%)b		–
$50,000 and less	113 (31.74%)
$50,000-$100,000	94 (26.40%)
$100,000-$150,000	63 (17.70%
$150,000 and over	86 (24.16%)
Place Resided before College		–
In-State	200 (56.18%)
Out-of-State	156 (43.82%)
First-Generation in College		–
First-Generation	51 (14.33%)
Not First-Generation	305 (85.67%)
Religion		–
Religious	263 (73.88%)
Non-Religious	93 (26.12%)
Importance of Religion	4.51 (2.12)	1.00–7.00
High school GPA, Mean (SD)	3.95 (0.43)	2.58–5.70
High School Class Sizec		–
<100	89 (25.00%)
100–300	93 (26.12%)
300–500	65 (18.26%)
500–700	40 (11.24%)
700–900	25 (7.02%)
>900	44 (12.36%)
Medical History
Traumatic Brain Injury, N (%) mild/moderate	109 (30.62%)	–
Psychotropic medication use, N (%), ≥ 1	126 (35.39%)	1.00–9.00
Psychotherapy treatment in past 3 months	37 (10.39%)	–
Number of current medical problems, ≥ 1	58 (16.29%)	0.00–5.00
Substance Use, M (SD)
Tobacco	0.85 (3.05)	0.00–28.00
Alcohol	3.02 (5.23)	0.00–34.00
Cannabis	1.12 (3.81)	0.00–31.00
Cocaine	0.03 (0.53)	0.00–10.00
Amphetamines	0.15 (1.89)	0.00–27.00
College experience
Number types of extra-curricular activities, ≥ 1	317 (89.04%)	0.00–8.00
Number types of academic help sought, ≥ 1	73 (20.51%)	0.00–3.00
Number of types of psych help sought, ≥ 1	60 (16.85%)	0.00–3.00
Satisfaction with education	5.63 (1.03)	2.00–7.00
Satisfaction with social experience	5.27 (1.48)	1.00–7.00
Major Declared	313 (87.92%)	–
Financial Aid type		–
Athletic	25 (7.02%)
Academic	297 (83.43%)
Need-Based Grant	150 (42.13%)
Work for pay	116 (32.58%)	–
Hours of work per week	3.80 (6.87)	0.00–40.00
College, N (%)		–
A&S College	77 (21.63%)
HS College	90 (25.28%)
Business College	43 (12.08%)
Eng&NS College	146 (41.01%)
Baseline Assessments, Mean (SD)
Suicide Behaviors Questionnaire – Revised	4.07 (2.04)	3.00–14.00
PROMIS Depression	53.23 (8.11)	34.20–78.20
PROMIS Anxiety	56.21 (9.10)	32.90–84.90
PROMIS Emotional Support	50.83 (8.44)	22.30–66.20
PROMIS Informational Support	53.41 (8.97)	23.20–69.80
PROMIS Social Isolation	50.86 (8.82)	31.80–73.10
PROMIS Sleep Disturbance	49.23 (8.88)	26.30–77.40
PROMIS Sleep-Related Impairment	53.88 (8.78)	26.20–81.20
NIH Toolbox Meaning & Purpose	48.17 (10.58)	15.30–68.50
NIH Toolbox Friendship	48.43 (10.72)	16.50–67.10
NIH Toolbox Self-Efficacy	45.16 (8.62)	20.80–68.30
NIH Toolbox Positive Affect	45.70 (9.44)	16.80–69.40
NIH Toolbox Perceived Stress	61.33 (6.42)	22.70–81.90
Connor-Davidson Resilience Scale 10	27.92 (6.76)	6.00–34.00
Freiburg Mindfulness Inventory	38.13 (7.51)	18.00–56.00
Emotion Regulation Scale
Suppression Score	4.06 (1.29)	1.00–7.00
Reappraisal Score	4.69 (1.09)	1.00–7.00
Vrana-Lauterbach Traumatic Events Scale
Number of traumas	1.49 (1.84)	0.00–11.00
Intensity of worst trauma	10.68 (4.64)	3.00–26.00

Abbreviations: GPA, grade point average; A&S, Arts and Sciences; HS, Health Sciences; Eng&NS, Engineering and Natural Sciences; PROMIS, Patient Reported Outcome Measurement Information System; NIH, NIH.

^aThe following Race variables were collapsed into one category for machine learning purposes: Black or African American, American Indian or Alaska Native, Middle Eastern/North African, Asian Indian, Chinese, Japanese, Korean, Other Asian, Native Hawaiian or Pacific Islander, Some other race, and Multi-racial, and entered into machine learning analysis with together with White variable.

^bAnnual parent or household income variables were collapsed into the following categories for machine learning purposes: <$100,000 and >$100,000.

^cHigh School Class Size variables were collapsed into the following categories for machine learning purposes: <500, 500–900, and >900.

Statistical modeling

The distributions of SBQ-R scores appeared right-skewed and thus were (natural) log-transformed and used as the dependent variables (Figure S3). Variables with less than 10% variability in the student population were excluded from the analysis. The model included 55 unique variables.²⁵ For additional information on the end-of-year model, please see the Supplementary Materials.

Different ML algorithms rely on unique assumptions and may result in different prediction accuracy, but no single algorithm is known to always outperform others on predictive accuracy. Although it is possible to incorporate the choice of algorithm in the training process, we chose to use the “wisdom of crowds” approach,³³ which combines predictions from multiple base learners (prediction algorithms). Specifically, we first utilized multiple “out of box” ML methods, followed by combining the predictions across methods by stacking or meta ensemble.^34–36

Each base learner model and the stacked model were built using nested-cross-validation (nCV), a layered approach to the traditional k-fold cross validation. Relative to other approaches, nCV effectively protects against overly optimistic estimates of model performance and guards against information leakage by keeping data used for model calibration, training, and model testing separate. The nCV procedure is executed across two loops, an inner loop and an outer loop. The inner loop is used to build base and stacked models, and the outer loop to evaluate model performance. The nCV procedure was repeated 100 times to quantify the variability of prediction accuracy. Given that nCV produces unbiased performance estimates regardless of sample size, it has been shown to be appropriate for use with small samples such as ours.^37–39

We applied four base learners in the inner loop for each training set, including elastic net,⁴⁰ support vector regression (SVR),⁴¹ random forest (RF),⁴² and k-Nearest Neighbors (knn).⁴³ For each base learner, the tuning parameter(s) were optimized by 5-fold cross validation (CV). Specifically, each training set was portioned into 5 distinct subsets, where 4 subsets were used for the training process to make predictions on the remaining subset. Optimal hyper-parameter values were chosen through random search⁴⁴ and the one-SE rule⁴⁵ using as the model performance metric. We obtained 4 sets of predicted values, one from each base leaner and their corresponding optimal hyper-parameter values.

Within the inner cross validation loop, each method produced a single best model and of the training sample (training R²). A stacked model was built by taking the arithmetic mean of predictions from each base learner, weighted by each model’s training R². In the outer loop, we applied the stacked model to predict the response in the corresponding validation set. Predicted values of the validation sets were combined and compared with the observed values to compute R². With 100 replications of partitions, we summarized the performance by the mean and 95% confidence interval of R².

Each base learner had a unique VI metric: absolute values of regression coefficients for elastic net, an “out-of-bag” mean square error obtained by permutation for RF, and a “filter” approach for SVR and knn wherein the response variable was regressed on each feature one at a time by a loess (Locally Weighted Scatter-plot Smoother), and the was computed as the variable importance metric. For each base learner, each feature was scaled to between 0 and 100 based off of its relative importance. We then combined across base learners to yield a stacked VI metric. The stacked VI for each feature was computed by taking the average importance across the four base learners, weighted by the relative performance of each model to favor stronger models. This produced a single set of VI values for each stacked model in the outer loop of nCV, which was then was averaged across folds to obtain a single VI estimate for each predictor.

One-hundred random partitions were used (ie 100 repeats of nested CV), and 95% confidence intervals for VI were taken as each variable’s mean importance ± 1.96 times its standard deviation. We computed Pearson correlation coefficients, 95% confidence intervals and FDR-corrected p-values for comparison purposes since massive univariate analyses are more common in the literature (Table 2). Analyses for prediction models were implemented using the caret package (version 6.0–76)⁴⁶ and partial-dependence plots by the “pdp” package,⁴⁷ and in R version 3.5.1. For additional information, please see the Supplementary Materials.

Table 2.

Demographic and clinical variables with both univariate correlation (Pearson, r and p-value FDR corrected) with Suicidal Behavior Questionnaire Score (log transformed) and variable importance (VI) in the stacked model.

Variable	Baseline
	r	p corr	VI
A&S College	0.18	0.00	10.04
Age	−0.03	0.74	6.80
Alcohol	0.10	0.08	5.68
Annual parent or household income	−0.11	0.08	5.96
Business College	−0.06	0.40	4.29
Connor-Davison Resilience Scale 10	−0.35	0.00	38.15
Eng&NS College	0.03	0.68	5.30
Female	0.03	0.64	4.26
Financial Aid – Academic	0.02	0.80	4.35
Financial Aid – Need-based	0.09	0.15	4.43
Grant
First-Generation in College	0.05	0.50	3.86
Freiburg Mindfulness Inventory	−0.40	0.00	52.25
Hispanic	0.01	0.90	2.74
Hours of work per week	n/a	n/a	n/a
HS class size <500	−0.01	0.92	3.95
HS class size >900	0.01	0.85	3.93
HS class size 500–900	−0.01	0.92	3.94
HS College	−0.16	0.01	12.59
HS GPA	−0.06	0.38	7.42
Importance of religion	−0.22	0.00	16.47
Intensity of worst trauma	0.24	0.00	29.05
Major Declared	−0.07	0.29	5.41
NIH Toolbox Perceived Stress	0.14	0.01	8.64
NIH Toolbox Friendship	−0.34	0.00	32.74
NIH Toolbox Meaning & Purpose	−0.44	0.00	74.45
NIH Toolbox Positive Affect	−0.45	0.00	67.49
NIH Toolbox Self-Efficacy	−0.32	0.00	32.61
Number of current medical problems	0.04	0.62	5.21
Number of traumas	0.20	0.00	21.87
Number of types of psych help sought	0.18	0.00	12.41
Number types of academic help sought	−0.06	0.40	8.00
Number types of extra-curricular activities	−0.19	0.00	17.92
Place resided before college	0.04	0.59	3.43
PROMIS Anxiety	0.35	0.00	42.70
PROMIS Depression	0.46	0.00	96.73
PROMIS Emotional Support	−0.23	0.00	17.77
PROMIS Informational Support	−0.32	0.00	35.53
PROMIS Sleep Disturbance	0.28	0.00	21.95
PROMIS Sleep-Related Impairment	0.18	0.00	10.39
PROMIS Social Isolation	0.43	0.00	69.40
Psychotherapy treatment in past 3 months	0.26	0.00	24.99
Psychotropic medication use	0.01	0.92	6.33
Race	−0.02	0.85	4.05
Reappraisal score	−0.31	0.00	34.12
Religion	−0.33	0.00	53.08
Resilience Training	n/a	n/a	n/a
Satisfaction with education	−0.18	0.00	9.01
Satisfaction with social experience	−0.24	0.00	13.68
Suicide Behaviors Questionnaire – Revised	n/a	n/a	n/a
Suppression score	0.12	0.03	7.25
Traumatic Brain Injury	0.09	0.12	6.39
Work for pay	0.14	0.01	12.95

Abbreviations: HS, high-school; GPA, grade point average; A&S, Arts and Sciences; HS, Health Sciences; Eng&NS, Engineering and Natural Sciences; PROMIS, Patient Reported Outcome Measurement Information System; NIH, NIH.

Order of variables are in alphabetic order.

Data availability

The R code used for analysis can be found on the open science framework at https://osf.io/wp6tn/. The data are not publicly available due to privacy and ethical restrictions but are available upon request.

Results

Using the SBQ-R cutoff score of ≥7,²⁶ 34 (9.6%) students were identified at significant risk for STBs (Table 1). Eighty (22.5%) and 92 (25.8%) students endorsed moderate to severe levels of depression (T > 60) and anxiety (T > 62), respectively.⁴⁸ Relative to baseline, there were significant decreases in severity of STBs reported at end-of-year [M(SD) = 3.36(.91); t(227) = 6.26, p < .001], with only 4 (1.8%) students meeting the cutoff score of ≥7. There were no differences in baseline SBQ-R scores between students who completed and did not complete end-of-year assessment [t(354) = 1.56, p = .12]. Given the reduction of STBs at end-of-year and limited variability (Figure S4), we focus our interpretation and discussion on the cross-sectional baseline model and report the prediction of end-of-year STBs in the Supplementary Materials. Symptoms of depression and anxiety also decreased at end-of-year relative to baseline [depression: M(SD) = 51.93(9.61), t(227) = 2.06, p = .041; anxiety: M(SD) = 54.49, (10.18), t(227) = 2.13, p = .034], although 48 (21.1%) and 50 (22.1%) students continued to endorse moderate to severe levels of depression and anxiety, respectively.

Machine learning

ML analysis identified a model with 49 variables that explained 28.3% of variance (95% CI: 28–28.5%) in baseline SBQ-R scores (Figure 1, Table 2). Of the top 15 variables with the highest importance, higher symptoms of depression and anxiety, feelings of social isolation, intensity of the worst trauma, sleep disturbance, and receiving mental health treatment in the last three months were associated with higher STBs, while higher scores in meaning/purpose, positive affect, trait mindfulness, trait resilience, perceived availability of helpful information and advice, trait-like use of cognitive reappraisal, friendships, self-efficacy, and identifying as religious were associated with lower STBs. Partial dependence plots for the 15 baseline variables with the highest importance are shown in Figure 2.

Figure 1.

Variable importance (VI) for models predicting suicidal thoughts and behaviors (STBs) in first year college students. The accompanying pie chart depicts percent of variance explained by the model and its 95% confidence interval. VI is based on the stacked ensemble. Variables with bars in red or blue have a positive or negative univariate correlation with suicidality, respectively. Error bars represent the 95% confidence interval, taken across partitions.

Abbreviations: Demographic variables: gender, religion, and participation in therapy in past three months; PROMIS Anxiety, Depression, Emotional Support, Informational Support, Sleep Disturbance, and Social Isolation scales; NIH Toolbox Friendship, Positive Affect, Meaning and Purpose, and Self-Efficacy scales; Freiburg Mindfulness Inventory (FMI); Emotion Regulation Questionnaire (ERQ): Reappraisal and Suppression scores; Connor-Davidson Resilience Scale (CDRISC-10); Suicidal Behaviors Questionnaire-Revised (SBQ-R); Vrana-Lauterbach Traumatic Events Scale (TES): total number of traumas and intensity score.

Figure 2.

Partial dependence plots showing the marginal strength of association for each of the 15 variables with the highest importance has on baseline SBQ-R score.

Abbreviations: PROMIS, Patient Reported Outcome Measurement Information System; NIH, NIH.

Discussion

The present study used a data-driven ML framework to identify variables associated with suicidal thoughts and behaviors (STBs) in first-year college students. Our study extends upon the previous literature by not only examining a large battery of demographic, clinical, and psychosocial measures as correlates of STBs, but also included factors that broadly constitute correlates of psychological well-being and resilience. This is an important endeavor as colleges turn their attention toward improving outcomes for their student populations, which are exhibiting increasing rates of psychopathology and alarming rates of STBs.

In this sample, 9.6% of students were considered to be at a serious risk for STBs. In line with previous research,^8,9 STBs in the present sample occurred in the context of considerable clinical symptomatology, where a fifth of the students reported moderate to severe levels of depression, and a quarter reported moderate to severe levels of anxiety. STBs decreased from the first semester of college to end-of-year, with only 1.8% continuing to report STBs above a recommended cutoff,²⁶ regardless of whether or not participants completed a brief resilience intervention in the first semester. This suggests that the first semester of college, as a phase of acute transition to college, may represent a particularly high-risk period for STBs, and is therefore an important time to focus on screening and prevention efforts aimed at reducing this risk and easing the transition to college. However, a fifth of students still reported moderate to severe levels of symptomatology at end-of-year, indicating a continued need for general mental health treatment options and programming for a large subset of college students.

Using a data-driven, atheoretical approach, our model identified correlates of STBs consistent with prior research and extended this literature through the identification of important factors associated with lower scores on measure of STB. While traditional statistical methods are better suited to model a handful of variables at a time using linear methods, ML methods allow for robust modeling of large number of factors and are not constrained by non-linearity, which is thought to be a more realistic reflection of the reality of complex phenomena such as STBs.⁴⁹ Although ML approaches are most often used to generate highly accurate “black box” prediction algorithms, this is not the only application of ML. Certain fundamental elements of ML (ie the use of rigorous cross-validation to more effectively guard against over-fitting, the application of a data-driven approach to model construction, and the use of algorithms that appropriately account for complexity and non-linearity) offer advantages over traditional modeling approaches, even when a black box prediction algorithm is not the end goal. The use of variable importance (VI) metrics allows for improved interpretability of ML models and enables investigators to pinpoint particularly important correlates using a data-driven atheoretical approach. While this application of VI is less widely used within the existing research on suicide, it has been applied successfully in other fields to aid biomarker discovery,⁵⁰ and can be particularly illuminating when multiple algorithms highlight the same features as most important.

Specifically, in our examination of baseline variables, the following were identified as the most important variables contributing to higher scores on a measure of STB risk assessment: depression; perceptions of being avoided by, excluded, and disconnected from others; anxiety; difficulty sleeping; number of traumatic events experienced and the intensity of the worst trauma; and engagement in treatment within the last three months. We hypothesize that the latter factor points to the severity of mental health problems, and the need to pursue psychological treatment.

Conversely, factors associated with lower scores on the STB risk assessment included: feeling that one’s life matters or makes sense; feelings that reflect pleasurable engagement with the environment (eg happiness, joy, enthusiasm, and contentment); self-identifying as religious; greater mindfulness skills; greater social connectedness (eg availability of friends or companions, feeling cared for, or a greater sense of belonging); a higher perceived quality of relationships that provide instrumental or informational support; greater self-perceived resilience and believing in one’s ability to manage and have control over meaningful events; and trait-like use of cognitive reappraisal (eg reframing the perception of stressful situations to reduce their emotional impact). The final overall model explained 28.3% variance. This amount of variance may not seem large given the number of examined factors. However, because we took a broad exploratory approach to the ML analysis, we included measures that may inherently have small relationships with STBs, but nevertheless represent factors that may be meaningful in understanding complex factors related to STBs. Moreover, given our use of nested-cross validation in model calibration and testing, it is likely that this estimate is stable and reproducible. Identification of these factors across different ML algorithms using stacking aids in determining the robustness of their importance in correlating with STBs. However, it is important to note that the importance of individual factors is relative to the model within which they are measured. As such, each identified factor should be considered within the context of other factors in the model, thus pointing to a set of important correlates of STBs.

Depression emerged as the most important predictor of STBs in our sample of college students, a finding that has been previously well described in this population.² Although STBs are a symptoms of major depressive disorder, particularly one that signifies its higher level of severity, they can be characteristic of a range of other disorders, including anxiety, psychotic, substance, and personality disorders.⁵¹ This points to the need for assessment of STBs not only in the context of major depression, but rather as a unique phenomenon in and of itself. Nevertheless, while data show that the majority of depressed individuals do not report or engage in STBs,⁵² STBs occur more often than any other symptom of major depression across episodes,^53,54 It is believed that, given its self-referential nature, suicidal ideation during a particular episode of depression enters the cognitive framework associated with depression, and is therefore more likely to be activated by subsequent experiences of low moods⁵⁵. Therefore, on college campuses, reported depressive states among students can be a crucial predictor of STBs, especially in those who have experienced them before.

Social connectedness, having meaning and purpose in life, experiencing positive affect, and self-efficacy emerged as factors related to lower STBs. These may be factors that lend themselves well to modification by pragmatic interventions. These findings are also in line with previous research. Across a number of studies, lack of social connectedness, loneliness, and perceiving oneself as a burden onto others are robust risk factors for heightened STBs, while reasons to live and hope repeatedly emerge as protective factors.^2,12,17 Further, having meaning and purpose in life has been shown to moderate STBs in depressed individuals,⁵⁶ and in the face of stress, may contribute to hope and optimism, which intrinsically have been found to serve a protective role against STBs.⁵⁷

While the emphasis on reducing negative affective symptoms in students at risk for STBs is necessary, our findings bring into focus other factors such as increasing meaning and purpose, and positive affect, which may protect against STBs.⁵⁸ However, current first-line interventions considered efficacious for depression and anxiety have only modest effects on positive affect.⁵⁹ Although we recently showed efficacy of a brief resilience intervention in reducing symptoms of depression and stress,²⁵ here, receiving the intervention was not an important predictor of end-of-year STBs nor did it have a direct relationship with end-of-year STBs. Therefore, using approaches that are known to reduce symptoms of depression by generating positive emotions may be necessary when treating these populations.⁶⁰ Nevertheless, future research is necessary to determine the impact of positive affect intervention on STBs. Finally, while learning adaptive coping skills is often part of existing interventions, they may not be universally implemented or made part of the university culture. Therefore, enhancing inclusivity and building peer and faculty/staff support systems, promoting access to and connections with on- and off-campus resources (eg mental health), and teaching adaptive coping skills (eg emotional reappraisal, mindfulness) through individual or campus-wide intervention programing may lead to broad effects on psychopathology and STBs. This is particularly important as help-seeking behaviors decrease with severity of STBs,⁶¹ and as nearly half of those who fail to seek help for emotional problems would prefer to instead talk with their friends and/or relatives.²⁰ Nevertheless, while these factors were identified as important with our ML algorithms, causality cannot be implied. Future experimental studies, including randomized clinical trials, will be necessary to examine the modifiable effect of these factors on reducing psychopathology and protecting against STBs.

Although the overall severity of STBs decreased from baseline to end-of-year, it is important to note that participants with increased STBs at baseline were also more likely to report increased end-of-year STBs. This finding supports the previously understood role of past suicidal thoughts and behaviors in predicting future risk and outcomes,⁶² as well as the high rates of persistence of STBs in this population.⁶ Therefore, students identified at risk for STBs early on warrant ongoing follow up and care in order to modify future risk. Interestingly, our analysis did not find demographic factors to relate to baseline STBs, which is similar to previous literature suggesting that STBs occur among college students independent of demographic and socioeconomic variables.⁶ Previous work has found evidence that low parental education level and difficult parental financial situation may relate to first onset of STBs in college,⁷ but these relationships were not identified in the current sample. Nevertheless, while demographic data are readily available to colleges, they may be insufficient for identifying those at highest risk of STBs. Supplementing data gathering efforts with a range of demographic, clinical, and psychosocial assessments may allow for better screening, predictive, and preventive utility.

Limitations and future research

Although we have used repeated nCV to increase the likelihood that our model will perform similarly when tested against new data and to reduce the risk of yielding overly optimistic estimates of proportion of variance explained or underestimates of model error, an external validation of both models with independent sample would be beneficial for further validation of these findings in college students. Second, the results presented here are cross-sectional. Therefore, causality cannot be inferred, and bi-directional relations between STBs and associated factors are possible. Third, the present sample consisted of first-year students at a private, midwestern university, which was of higher socioeconomic status (SES) and over-represented in regard to White and Native American populations. Although the employed methods were aimed at ensuring robustness of findings, future work is needed to establish whether findings indeed generalize to diverse student populations or other contexts, including large universities, community colleges, and high school. Further research is also needed to assess the predictive value of these factors in later college years and beyond. Fourth, while this study included a breadth of self-report measures relevant for mental health and STBs, there were potentially important variables that were not assessed, such as non-heterosexual orientation and trans-gender status.^6,7

Conclusion

This study used a data-driven framework to identify important variables constituting risk for or resilience from suicide thoughts and behaviors in college students. Several modifiable factors emerged as important correlates of STBs. Future studies should examine whether interventions designed to promote engagement with meaningful, important, and pleasurable activities, as well as enhance social connectedness, improve well-being and reduces the severity of suicidal thoughts and behaviors in student populations.