Using intensive longitudinal data to identify early predictors of suicide-related outcomes in high-risk adolescents: Practical and conceptual considerations

Abstract

Mobile technology offers new possibilities for assessing suicidal ideation and behavior in real- or near-real-time. It remains unclear how intensive longitudinal data can be used to identify proximal risk and inform clinical decision-making. In this study of adolescent psychiatric inpatients (N=32, ages 13–17, 75% female), we illustrate the application of a 3-step process to identify early signs of suicide-related crises using daily diaries. Using Receiver Operating Characteristic Curve (ROC) analyses, we considered the utility of 12 features—constructed using means and variances of daily ratings for six risk factors over the first two weeks post discharge (observations=360)—in identifying a suicidal crisis two weeks later. Models derived from single risk factors had modest predictive accuracy (Area Under the ROC Curve [AUC] 0.46–0.80) while nearly all models derived from combinations of risk factors produced higher accuracy (AUCs 0.80–0.91). Based on this illustration, we discuss implications for clinical decision-making and future research.

Introduction

The prevalence of youth suicide, which is the 2^nd leading cause of death for 10- to 19-year-olds (Centers for Disease Control and Prevention, 2019), is a major public health crisis. At the same time, there remain important gaps in our understanding of suicidal behavior and how to best prevent it. One such gap—a priority area identified by the National Action Alliance for Suicide Prevention (2014)—is improving the prediction of short-term suicide risk. Much of what is known about suicide risk is based on studies spanning across wide time intervals (months or years), which provides information about distal relationships but reveals limited information about who is at imminent risk and when (Bagge, Glenn, & Lee, 2013; Rudd et al., 2006). Yet, short-term risk prediction is a key concern for clinical providers (Glenn & Nock, 2014; Rudd et al., 2016). More research that incorporates proximal assessment is critically needed to provide insights about short-term precursors to suicidal ideation and behavior in order to facilitate early detection of risk as well as inform targeted interventions.

Technology-Assisted Strategies for Short-Term Suicide Risk Prediction

Mobile technologies offer novel opportunities for identifying early signs of suicide risk in a way that maximizes ecological validity and enables provision of timely support. Mobile technology can be used to assess suicidal ideation and behavior in real-time via Ecological Momentary Assessment (EMA) or near-real-time using daily diary assessments. These methods are characterized by repeated assessments in the person’s natural environment (Shiffman et al., 2008), involving short recall periods that reduce bias. A growing number of EMA and daily diary studies has expanded our understanding of the dynamic nature of suicidal thoughts (Czyz, Horwitz, Arango, & King, 2018; Hallensleben et al., 2018; Kleiman et al., 2017) and their time-dependent relationship with known risk factors ((Ben-Zeev, Young, & Depp, 2012; Czyz et al., 2018; Kleiman et al. 2017). However, it remains unclear whether and how EMA or daily diary data can be used to identify early signs of risk and guide targeted interventions.

Implications for Tailored Interventions

Identifying reliable proximal markers of suicide risk could inform personalized interventions provided at crucial moments in the individual’s natural environment. Specifically, validated indicators of proximal risk could lead to the development of adaptive interventions. Adaptive interventions provide empirically-based guidelines for clinical decision-making via decision rules that recommend if, when, how, and for whom an intervention should be delivered and adjusted at critical decision points (Lei, Nahum-Shani, Lynch, Oslin, & Murphy, 2012; Nahum-Shani et al., 2012). In an adaptive intervention, treatment is individualized based on tailoring variables—baseline (initial) as well as ongoing information from the individual used to guide whether and how to modify the treatment for the individual in terms of type, intensity, or dosage (Lei et al., 2012; Nahum-Shani et al., 2012; Nahum-Shani et al., 2017).

Although adaptive interventions have the potential to improve the effectiveness of interventions for adolescents at risk for suicide, there currently is insufficient empirical evidence to inform the construction of effective adaptive interventions for these youths. Specifically, there is a critical gap concerning the conditions that reliably represent early signs of elevated suicide risk to operationalize tailoring variables needed to identify the conditions in which treatment should be provided or modified and the conditions in which it should not. In this context, tailoring variables would reliably signal a change in suicide risk status providing an early indication of an impending crisis, which, in turn, would be used to guide the provision of an intervention addressing these early signs of elevated risk as soon as they occur.

Study Purpose

Although collecting daily and EMA data among individuals at risk for suicide has been shown to be feasible (Czyz, Nahum-Shani, & King, 2018; Davidson, Anestis, & Gutierrez, 2017; Husky et al., 2014), it remains unclear whether and how these types of assessments can be used to identify early signs of suicide risk and thus inform clinical decision-making. Rather than hypothesis testing, the goal of this proof-of-concept study is to illustrate the application of existing methods to derive information from daily diaries collected via mobile devices to identify early signs of suicide risk-related crises among high-risk adolescents. Based on this illustration, we report on the feasibility of this approach and discuss opportunities (including implications for identifying tailoring variables), challenges, and directions for future research.

Illustrative Example

For illustrative purposes, we use intensive longitudinal data collected from 32 adolescents who were followed with daily diaries for a month after discharge from psychiatric hospitalization and who provided information about suicide risk-related crises during this period. In this demonstration, we used data from daily assessments of clinically- and theoretically-grounded risk factors collected over the course of the initial two weeks post discharge to detect the occurrence of a distal outcome, operationalized in terms of whether or not adolescents experienced a suicidal crisis culminating in a suicide attempt or acute level of care. Consistent with the study’s focus, the models described in the study offer an illustration of an approaches for extracting suicide risk indicators from intensive longitudinal data rather than providing conclusive evidence about the risk indicators themselves.

For simplicity, we focused on two features (mean and variance) of six risk factors (hopelessness, connectedness to others, perception that one is a burden, emotional pain, self-efficacy to refrain from suicidal action, suicidal ideation duration) selected based on theoretical consideration of the following: the central role of psychache or psychological pain in suicide (Shneidman, 1993), the critical role of hopelessness in the cognitive framework of suicide (Wenzel & Beck, 2008), the interpersonal-psychological theory of suicidal behavior (Joiner, 2015; Van Orden et al., 2010), the three-step theory of suicide (Klonsky & May, 2015), and self-efficacy theory (Bandura, 1977; Czyz et al., 2014). Although constructing multiple features (e.g., mean, variance, minimum, maximum root mean square of successive differences) from these risk factors and applying different time scales (e.g., over 3 days, over 4 days, over 5 days, and so on) is possible, we considered the mean and variance over the first 14 days as a demonstration. In practice, additional features and time scales may be considered based on the guiding research questions. As an example, we demonstrate how this approach could be repeated using the same features using a different time scale (over the first 7 days).

Methods

Participants

Participants were adolescents (age 13–17) who were psychiatrically hospitalized due to last-month suicidal attempt and/or last-week suicidal ideation. Participants were recruited to take part in a pilot study of a brief psychosocial intervention with a daily survey follow-up component; the pilot trial focused of feasibility and acceptability of a safety planning intervention (Czyz, King, & Biermann, 2019). Exclusion criteria included: severe cognitive impairment or altered mental status, transfer to medical unit or residential placement, no availability of a legal guardian, and teen not having a cell phone with texting capability. Of those who met all eligibility criteria, 36 (76.6%) provided parental consent and teen assent. This study’s analytic sample was limited to 32 adolescents who continued in the study after discharge and for whom we had follow-up outcome data. The analytic sample was 75% female (n=24), with a mean age of 15.4 years (SD=1.37). The sample’s racial/ethnic distribution was as follows: 84.4% (n=27) Caucasian, 9.4% (n=3) African-American/Black, 9.4% (n=3) Asian, 6.3% (n=2) Hispanic, 3.1% (n=1) American Indian or Alaska Native, and 3.1% (n=1) Native Hawaiian or Other Pacific Islander. At baseline, 53.1% (n=17) participants had previously attempted suicide.

Procedures

Eligible adolescents who provided assent and parent consent completed a series of self-report surveys during hospitalization in addition to follow-up assessments (2-week online survey, 1- and 3-month phone assessments). In addition, adolescents were asked to complete one survey each evening for 28 days. A link to an online survey was automatically texted to participants’ phones between 5pm and 7pm. Compensation was up to $222, including $4 for each completed daily survey. For this study, we used data from daily diaries and the 1-month follow-up. The study was approved by the participating university’s Institutional Review Board.

Measures

Distal outcome (1-month follow-up).

The occurrence of a suicidal crisis was a binary outcome indicating whether or not adolescents experienced a suicide attempt or acute level of care (rehospitalization or ED visit) after the second week post-discharge and prior to the 1-month follow-up. Presence and dates of suicide attempts were assessed with the Columbia-Suicide Severity Rating Scale (Posner et al., 2011), a semi-structured interview. Presence, reasons for, and dates of psychiatric hospitalizations and ED visits since index hospitalization were also assessed.

Proximal risk factors (daily surveys).

We used daily surveys to assess theoretically-grounded risk factors based on the following measures. Adolescents responded in reference to the last 24-hours.

Self-efficacy to refrain from suicidal action.

Self-efficacy was assessed with an item from the Self-Assessed Expectations of Suicide Risk Scale (Czyz, Horwitz, & King, 2016). On a scale from 0 (“not at all confident”) to 10 (“completely confident”), adolescents rated: “How confident are you that you will be able to keep yourself from attempting suicide?”

Hopelessness.

Hopelessness was assessed on a 4-point scale using an item (“I see only bad things ahead of me, not good things”) from the Brief Hopelessness Scale (Bolland, McCallum, Lian, Bailey, & Rowan, 2001).

Connectedness and burdensomeness.

Using a 7-point scale (from “not at all true for me” to “very true for me”), adolescents rated the extent of their connectedness to others (“I am close to other people”) and their sense of burdensomeness (“The people in my life would be happier without me”). These questions were derived from the Interpersonal Needs Questionnaire (INQ) (Van Orden, Cukrowicz, Witte, & Joiner, 2012).

Psychological pain.

Participants rated the extent to which they felt miserable as a proxy for psychological pain. Responses were rated on a 5-point scale, from “very slightly or not at all” to “extremely.” This question was adapted from the Positive and Negative Affect Schedule for Children (PANAS-C) (Ebesutani et al., 2012).

Suicidal ideation duration.

Adolescents were asked daily: “At any point in the last 24 hours, did you have any thoughts of killing yourself?” An affirmative response was followed by a 5-point question assessing ideation duration (from “a few seconds or minutes” to “more than 8 hours/continuous”), modeled after the C-SSRS (Posner et al., 2011): “How long did these thoughts last?” We created a continuous scale from 0 (no ideation) to 5 (continuous ideation).

Data Analysis

A three-step iterative process was used to investigate whether and what type of daily survey information collected over the course of the initial two weeks post discharge is useful in identifying early those youth who are likely to experience a suicidal crisis after these initial two weeks and prior to the 1-month follow-up assessment (i.e. distal outcome). A total of 12 features—two features were constructed based on the mean of daily ratings and the variance of daily ratings over the first 14 days for each of the six risk factors of interest (psychological pain, burdensomeness, hopelessness, connectedness, self-efficacy to refrain from suicidal action, and duration of suicidal ideation)—were used in Receiver Operating Characteristic Curve (ROC) analyses as part of this iterative process. ROC is a nonparametric technique that is well-suited for using longitudinal data to construct tailoring variables (Steidtmann et al, 2013).

As part of this illustration, we focused on 12 features using a specific time scale (14 days). In practice, the three-step process could be repeated using time scales corresponding to different sequence of days (e.g., over the first 2 days, 3 days, 4 days, and so on) or pre-determined time scales meaningful in a specific intervention context (e.g., over the first 7 days post discharge when a decision to augment or not augment treatment is made). To demonstrate, the 3-step process was repeated for a different time scale (7 days) in supplemental analyses.

Step 1: Selection of risk factors for inclusion in prediction models of suicidal crises.

We conducted a series of logistic regressions (see Table 1), one for each risk factor, to investigate the extent to which each factor’s mean (of daily ratings), as well as the combination of its mean and variance (of daily ratings), was associated with the distal outcome. These analyses were used to determine whether a specific risk factor should be selected for further investigation in complex models. While using a univariate relationship between predictors and the outcome as a filter is not guaranteed to be optimal, this approach may nevertheless be useful in identifying risk factors that are most relevant in terms of clinical utility (effect size) while accounting for parsimony and implementation considerations. Using the univariate relationship in this setting is also inspired by the sparsity principle (Tibshirani, 2014) and the hierarchical ordering principle (Wu & Handa, 2009), suggesting that “decision making is based primarily on simpler effects, with more complex effects brought in as needed” (Collins et al., 2013).

Table 1.

Logistic regressions identifying suicide risk-related outcome based on risk factors considered individually

Model	Mean Psychological Pain	Variance Psychological Pain	Mean Hopeless- ness	Variance Hopeless- ness	Mean Burdensome- ness	Variance Burdensome- ness	Mean Connected- ness	Variance Connected- ness	Mean Self-Efficacy	Variance Self-Efficacy	Mean Duration of Suicidal Ideation	Variance Duration of Suicidal Ideation	AUC
1	2.73 (2.52)												0.76
2			2.40 (1.88)										0.69
3					0.97 (0.96)								0.52
4							1.25 (1.31)						0.46
5									0.55 (0.35)				0.80
6											2.06 (2.15)		0.70
7	2.19 (2.06)	2.85 (1.87)											0.79
8			2.41 (1.89)	1.34 (1.06)									0.69
9					1.01 (1.01)	0.64 (0.76)							0.58
10							1.12 (1.14)	0.31 (0.48)					0.62
11									0.48 (0.28)	0.81 (0.68)			0.79
12											2.06 (2.16)	0.99 (0.99)	0.70

Note. Each row corresponds to one logistic regression model. Non-blank cells indicate which features were considered in the same model. These cells present estimates of the odds ratio corresponding to the each listed feature with 95% confidence interval (CI) listed in [brackets]; values listed in parentheses represent standardized odds ratios. Bolded AUCs are significantly different (at 0.05 level) from 0.50. AUC = Area Under the Receiver Operating Characteristic Curve.

For models including the mean of daily ratings as the only feature in the model, risk factors were selected for further investigation if the odds ratio (OR) per one standard deviation of mean daily ratings was at least medium in magnitude (i.e. OR≥1.65 or OR≤0.61; see Pencina, D’agostino, Pencina, Janssens, & Greenland, 2012). The standardized ORs were obtained by first standardizing the predictors (as z-scores). Moreover, to identify those showing early signs of elevated suicide risk and those who do not, risk factors should also demonstrate sufficient sensitivity and specificity. Thus, we used the Area under the Receiver Operating Characteristic Curve (AUC) as an additional criterion for selecting a risk factor for further investigation. The AUC captures the average sensitivity over all values of false positive rates (i.e., 1-Specificity) of different cutoff points of predicted probability (Pencina, D’Agostino, D’Agostino, & Vasan, 2008), and it provides a measure of the usefulness of each separate model (across all possible cut-points); AUC ranges from 0.5 (no discriminative ability) to 1 (perfect discrimination). Hence, risk factors were selected for further investigation if their model’s AUC was at least medium in magnitude (i.e, AUC ≥ 0.64; see Pencina et al, 2012; Rice & Harris, 2005). We used an AUC threshold of at least 0.64 for the model with the mean alone or the model with both the mean and variance to minimize the possibility of prematurely excluding risk factors based on a single feature (i.e. the mean) alone. While we report for each model whether or not its AUC is different from 0.50 (at $\alpha$ = 0.05), we emphasize evaluation of models based on effect sizes and their potential practical and clinical utility.

Step 2: Constructing and evaluating prediction models of suicidal crises.

Risk factors that were selected in Step 1 for further investigation were subsequently examined in more complex models, wherein we simultaneously considered multiple risk factors and their features in a series of ROC analyses (Table 2). To facilitate interpretability in terms of clinical decision-making, we constrained the constructions of these models such that the mean of a given risk factor was always included as a feature (i.e. models including the variance included the corresponding mean). For example, low variance accompanied by low versus high mean might have different clinical implications (i.e. consistently low versus consistently high suicidal ideation). The clinical utility of the more complex model was evaluated based on the magnitude of the AUC. Striving to yield clinical utility, we considered AUC of at least 0.80 as adequate, as it corresponds to a very large effect size (Rice & Harris, 2005).

Table 2.

Logistic regressions identifying suicide risk-related outcome based on risk factors considered in combination

Model	Mean Psychological Pain	Variance Psychological Pain	Mean Hopelessness	Variance Hopelessness	Mean Self-Efficacy	Variance Self-Efficacy	Mean Duration of Suicidal Ideation	Variance Duration of Suicidal Ideation	AUC
1	6.69	15.81	0.02	0.01	0.25	0.62			0.89
2	9.96	4.13	0.14	0.40			1.26	0.64	0.82
3	0.32	3.48			0.26	0.87	3.20	0.31	0.89
4			0.04	0.06	0.13	0.83	8.62	0.22	0.91
5	10.54	4.37	0.13	0.10					0.82
6	2.59	4.02					1.19	0.58	0.80
7					0.39	1.13	2.98	0.24	0.86
8	0.72	6.81			0.42	0.59			0.86
9			1.20	2.10			2.05	0.88	0.66
10	2.59	4.02					1.19	0.58	0.80
11			0.57	0.16	0.39	0.87			0.81
12	2.34		0.05		0.41		3.25		0.87
13	4.33		0.20		0.61				0.86
14	0.39				0.44		2.57		0.81
15			0.13		0.38		3.39		0.87
16					0.59		1.75		0.83
17	1.44				0.63				0.84

Note. Each row corresponds to one logistic regression model. Non-blank cells indicate which features were considered in the same model. These cells present estimates of the odds ratio corresponding to each listed feature. Bolded AUCs are significantly different (at 0.05 level) from 0.50. AUC = Area Under the Receiver Operating Characteristic Curve.

Step 3: Construction of prediction rules for best-performing prediction models.

Finally, using models with largest magnitudes of AUC—”best-performing” models— we sought to determine cutoffs on predicted probabilities that maximized the discrimination between youth who would and would not experience a suicidal crisis by identifying a point on the ROC curve that yielded maximal sensitivity and specificity. Specifically, we used the “closest top left” criterion implemented in the R package pROC (Robin et al, 2011). The chosen cutoffs were then used to construct prediction rules (i.e. a rule that determines if a person would be classified as someone who would experience a suicidal crisis).

Analyses were performed in R using the pROC package (Robin et al, 2011). A subset of participant data was not held out prior to analyses (prior to performing Steps 1–3) to assess the performance of the best-performing models on an independent sample due to the modest sample size; this represents a necessary step for establishing the validity of the results and selected decision rules using the leave-out data to generalize the results.

Results

During the month following hospitalization, 5 (15.6%) adolescents reported an ED visit, 3 (9.4%) were psychiatrically rehospitalized, and 2 (6.3%) reported a suicide attempt. In total, 5 participants (15.6%) reported a suicide risk-related event. Of note, 60% of the five adolescents who experienced a suicide risk-related event had previously attempted suicide compared to 52% of those without the follow-up event. All these reported events occurred after the initial 14 days during which daily survey data were aggregated for analyses; the average number of days until a suicide-related event occurred within the observation period (i.e. after the initial 14 days and prior to the 1-month assessment) was 10.78 (SD=9.35) days. In addition, 360 (76.3%) of the daily surveys within the initial 14-day period were started and completed.

Results from Step 1.

Based on models considering risk factors one at a time (Table 1), we selected risk factors to be considered further in more complex models. Specifically, using standardized odds ratios estimated for Models 1–6 (models with the mean as the only feature) equivalent to at least a medium effect size (i.e. above 1.65 or below 0.61), we selected the following risk factors: psychological pain (OR = 2.52; Model 1), mean hopelessness (OR = 1.88; Model 2), mean self-efficacy (OR= 0.35; Model 5), and mean duration (OR= 2.15; Model 6). We also considered if AUC values for Models 1–6 (involving the mean alone) or their analogues in Models 7–12 (involving the mean and variance) were at least medium in magnitude (i.e. above 0.64). The results based on the AUC criterion were consistent with those based on the ORs and thus did not lead to removing any of the selected risk factors from consideration. Thus, subsequent analyses of possible combinations of risk factors and their features (means and variances) involved the constructs of psychological pain, hopelessness, self-efficacy, and duration of suicidal ideation.

Among the models considering single risk factors, the model yielding the largest AUC was Model 5 in Table 1 (AUC = 0.80; predictor: mean self-efficacy).

Results from Step 2.

Table 2 presents analyses of combinations of these four risk factors (three at a time or two at a time), which includes models involving means and variances (Models 1–10) as well as models involving means alone (Models 11–17). We did not include all four risk factors in the same model as the sample size was insufficient to obtain stable estimates of the model parameters. With the exception of Model 9 (AUC=0.66), AUCs ranged between 0.80–0.91 across the considered models (Models 1–17). The values of AUCs in Table 2 show that appropriately chosen combinations of risk factors substantially improved the detection of a suicidal crisis over considering each risk factor in isolation (Table 1). In addition, removing information on variability generally reduced AUC magnitude for equivalent models including only the mean.

Among the models considering risk factors simultaneously, the model with the largest AUC was Model 4 in Table 2 (AUC = 0.91; predictors: mean and variance of hopelessness, self-efficacy, and duration of suicidal ideation).

Results from Step 3.

Using the “closest top left” criterion, we identified a point on the ROC Curve corresponding to maximal sensitivity and specificity for the best-performing models. The optimal cutoff chosen for the overall best performing model (AUC = 0.91; Model 4, Table 2) corresponded to a sensitivity of 0.80 and a specificity of 0.96. The optimal cutoff for the best performing model considering each risk factor independently (AUC = 0.80; Model 5, Table 1) corresponded to a sensitivity of 0.60 and a specificity of 0.82. Table 3 presents prediction rules based on these best-performing models. It is important to note that these prediction rules correspond to a particular time scale (14 days) that was used as an illustrative example. In practice, the three-step process would be repeated using other times scales (e.g., over 2 days, over 3 days, over 4 days, and so on) until all possible time scales under consideration were evaluated, resulting in different prediction rules for any given time period. To demonstrate, we applied the three-step process using a different time scale (over 7 days; supplemental material).

Table 3.

Prediction rules for best performing models

	Complex model (Table 2, Model 4)		Individual model (Table 1, Model 5)
	AUC = 0.91		AUC = 0.80
Quantity (χ)	Estimate ($\hat{\beta }$)	SE	Estimate ($\hat{\beta }$)	SE
Intercept	19.90	11.74	2.59	2.05
Mean Hopelessness	−3.19	2.24
Variance Hopelessness	−2.87	7.89
Mean Self-Efficacy	−2.06	1.11	−0.60	0.30
Variance Self-Efficacy	−0.18	0.72
Mean Duration of Suicidal Ideation	2.15	1.16
Variance Duration of Suicidal Ideation	−1.54	2.12
Prediction Rules	IF $\frac{e^{X\widehat{\beta }}}{1+e{}^{X\hat{\beta }}}\ge 0.462$ THEN   Flag person as at-risk of suicidal crisis.		IF $\frac{e{}^{X\hat{\beta }}}{1+e{}^{X\hat{\beta }}}\ge 0.197$THEN   Flag person as at-risk of suicidal crisis. Equivalently, IF Mean self efficacy ≤ 6.66 THEN   Flag person as at-risk of suicidal crisis.

Note. AUC = Area Under the Receiver Operating Characteristic Curve. χ is a design matrix having number of rows equal to the number of study participants. For Table 2, Model 4 the columns of χ have values corresponding to 1’s, mean hopelessness, variance hopelessness, mean self-efficacy, variance of self-efficacy, mean duration of suicidal ideation, and variance of duration of suicidal ideation. For Table 1, Model 5 the columns of χ have values corresponding to 1’s, and mean self-efficacy.

Discussion

This proof of concept study capitalized on daily diary data to derive unique patterns in predictors serving as early indicators of an impending suicidal crisis among suicidal youth. While this study identified noteworthy patterns that signaled, with high accuracy, which teens are likely to experience a suicidal crisis in the month after hospitalization, it is important to highlight that its primary purpose was to describe an approach that can be used to extract these early indicators from intensely sampled risk factors rather than providing conclusive evidence about their accuracy (i.e. results from small samples can be skewed by atypical cases that may impact replicability). Thus, this study’s value lies in advancing the limited literature on proximal suicide risk by describing an approach used to derive markers of short-term risk from intensive longitudinal data. With this caveat in mind, this study’s findings point to the utility and feasibility of using intensive longitudinal data to produce valuable early markers of a suicidal crisis.

The results from this study’s illustrative example yielded several notable observations. First, nearly all individual predictors, which were constructed based on aggregated data from the first two weeks after discharge, did not meet the threshold of predictive accuracy (AUC of 0.80), defined in this study as indicating clinical utility. The only exception to this general pattern was the construct of self-efficacy. Second, for the most part, detection of future suicide risk-related outcomes was substantially improved when risk factors were considered in combination. Virtually all the considered combinations met the previously defined threshold of clinical utility. For example, the best-performing model (AUC=0.91)—i.e. model incorporating the mean and variance of hopelessness, self-efficacy, and suicidal ideation duration—had a corresponding sensitivity of 0.96 and specificity of 0.80. Though not conclusive, the robustness of these proximal indicators is notable given that suicide risk instruments (Huth-Bocks, Kerr, Ivey, Kramer, & King, 2007; Runeson et al., 2017) and known individual predictors (Franklin et al., 2016) assessed over longer intervals have shown modest clinical utility. Moreover, it is noteworthy that identifying early patterns of risk may not require assessing thoughts of suicide; the next best-performing model (AUC=0.89) did not include suicidal ideation duration. This is notable in light of the fact that some individuals may either not disclose or not experience suicidal thoughts until only moments before initiating suicidal behavior (Millner, Lee, & Nock, 2017). Future research should consider multiple pathways to early risk detection that may not require direct assessment of suicidal thoughts that could be subject to underreporting.

Third, models accounting for both the mean and the variance of the combined risk factors had generally higher levels of accuracy than equivalent models considering only the mean, which suggests that analysis of intensely sampled data will likely require moving beyond using single summary scores and considering relationships among different features of constructs to optimize suicide risk detection. Future studies should consider combinations of multiple features (mean, variance, minimum, maximum root mean square of successive differences) of dynamic constructs. Given that intensive longitudinal data collection may result in missing data, missingness could be included as a predictor. More complex feature construction methods, such as using latent profiles (Pettit, Silverman, Rey, Marin & Jaccard, 2016), could also be considered. Indeed, others have argued that deriving different indices of functioning from intensely collected data may be able to more meaningfully capture individuals’ experiences (Schneider & Stone, 2016; Stone, Broderick, Schneider, & Schwartz, 2012).

Given that a promising application of detecting early markers of suicide risk is to inform delivery of interventions, an important consideration is that the process used to arrive at these markers should ideally be guided by producing interpretable results, such as to secure providers’ willingness to act upon these results. While logistic regression produces interpretable prediction rules, other classification methods (e.g. neural networks, ensemble learning), which are often less interpretable, have the capacity to explore nonlinear relationships. The ROC-guided approach described in this study yielded models that were both interpretable (the practical meaning is readily apparent) and resulted in AUCs with meaningful magnitudes. Another practical concern has to do with balancing maximizing predictive accuracy with parsimony. Here, models with largest AUCs were relatively less parsimonious. Parsimonious models may be viable alternatives when reducing response burden is of high importance or, in context of adaptive interventions, when simple criteria used to determine for whom an intervention should be modified is desired (Almirall, Compton, Gunlicks‐Stoessel, Duan, & Murphy, 2012). Passive data collection (e.g., sensors, geolocation, activity) not requiring direct input from individuals—increasingly utilized to determine different aspects of mental health functioning (Mohr, Zhang, & Schueller 2017; Reinertsen & Clifford, 2018)—could be used to supplement assessments to address concerns about response burden. However, evidence regarding the validity and utility of passively-collected data in predicting suicide risk still needs to be established (Torous et al., 2018).

It is clear that mobile monitoring has substantial potential in generating clinically valuable indicators of suicide risk, which could facilitate timely interventions. There is evidence that daily diaries can be used to assess unfolding response to interventions after psychiatric hospitalization (Czyz et al., 2018) or to offer specific support based on a pre-defined response assessed daily (Kennard et al., 2018). The current findings demonstrate that intensive longitudinal data have value in constructing early indicators of suicide-related outcomes that may enable more personalized interventions. As described earlier, these indicators can be used to inform adaptive interventions by identifying tailoring variables (Lei et al., 2012; Nahum-Shani et al., 2012). For example, tailoring variables could take the form of an automated algorithm to “flag” near-term risk and guide clinical decision-making, such as prompting clinicians to recommend increasing intensity of treatment, augmenting treatment, or recommend higher level of care. Such an algorithm may be district for each time interval (time scale) under consideration, given that different time scales used to aggregate intensive longitudinal data may result in different best-performing models and prediction rules (as shown in supplemental tables). For example, each day of monitoring might have its own prediction rule derived from the three-step process. Identifying empirically-based tailoring variables is also critical for just-in-time adaptive interventions (JITAIs), a type of adaptive intervention addressing dynamically changing needs of an individual in their natural environment (Nahum-Shani et al., 2017). JITAIs and similar intervention designs have already shown promise in related areas of mental health (Schueller, Aguilera, Mohr, 2017). Additional research will be vital in validating early indicators of suicide-related outcome before they can be used in adaptive interventions to guide clinical practice.

Limitations and directions for future research.

First, we utilized aggregated data from select time scales (first 14 days; first 7 days in supplemental analyses) as an illustrative example, and future research should expand on this work by considering a broader array of features, risk factors, and time scales. Second, models beyond those tested in this study (e.g. models that capture additive relationships) should be considered in future work. Third, this study’s inpatient sample was comprised of mostly female and Caucasian adolescents, which limits the generalizability of the results. Fourth, the small sample size did not allow for more rigorous testing of the stability of the results; replicating these findings in larger samples is needed to independently validate the identified prediction rules. The modest sample size also limited our ability to explore more complex models (e.g., model based on means and variances of all four risk factors did not converge). Finally, more work will be needed to address practical implementation questions, such as establishing guidelines for responding to acute suicide risk, accounting for false negatives in prediction rules, and considerations related to privacy and integration of data with medical records.

In conclusion, this study illustrated the application of existing methods to identify early signs of suicide risk-related crises among suicidal youth. We know of no previous research on short-term indicators of suicide-related outcomes using intensive longitudinal data. These indicators could ultimately be used to construct tailoring variables and inform decision rules for suicide-specific adaptive interventions that guide when and for whom treatment intensification or augmentation might be needed. It is our hope that these findings will encourage additional research concerned with identifying proximal markers of suicide risk across different populations while improving upon these results by considering additional constructs, features, time scales, as well as data analytic approaches. Attention to practical issues such as interpretability, parsimony, and ethical considerations will be necessary to eventually encourage implementation in clinical settings. Research in this area has the potential to improve prediction of near-term risk as well as inform the development of tailored treatments for individuals at risk for suicide.