Abstract
“Skills to Enhance Positivity (STEP),” is a two-part positive affect program designed to decrease recurrent suicidal behavior in adolescents hospitalized due to suicide risk. Here, we describe the initial pilot phase in which the intervention was developed and modified based on a sample of 20 adolescent participants, ages 12–18 years old (mage=15.9, SD=1.5). STEP consisted of an in-person phase (four sessions) and a remote delivery phase (text messaging and phone calls). The inpatient sessions focused on psychoeducation of positive affect, mindfulness meditation, gratitude, and savoring. The remote delivery phase comprised of weekly phone calls and daily text messages to enhance mood monitoring and skills practice reminders. Average session attendance was 81%, and mean daily response rate to text messages was 73.6%, demonstrating high engagement. STEP was described as good or excellent by over 90% of parents and 100% of adolescents. Only one participant had a suicide attempt, and five were readmitted for suicidality in the following six months, fewer than comparable naturalistic studies. Although preliminary results are promising, larger randomized trials are needed to determine the efficacy of STEP in reducing suicidal behaviors.
Keywords: Adolescents, Suicidal Behaviors, Positive Affect, Mindfulness, Gratitude, Savoring
Suicide is the second leading cause of death in adolescents and accounts for more deaths than all natural causes combined (Xu, Murphy, Kochanek, & Bastian, 2016). Since 1980, the rate of suicide deaths among adolescents has increased by 11% (Minino, 2010). A 2011 CDC survey indicated that in a 12 month interval, 15.8% of U.S. high school students seriously considered suicide, 7.8% reported making at least one suicide attempt, and 2.4% reported making at least one suicide attempt that required medical attention (Kann et al., 2014). Based on epidemiologic studies, the lifetime prevalence suicide attempt rate among adolescents is in the range of 3.1–8.8% and the 12-month prevalence estimates are in the range of 7.3–10.6% (Nock et al., 2008). A preponderance of interventions for adolescent suicidality focus on crisis intervention, underlying psychiatric disorders, regulating negative affect, and reducing cognitive distortions. However, in a recent meta-analysis of cognitive behavioral therapy (CBT) to reduce suicidal behaviors, a significant treatment effect was observed for adult samples but not for adolescents (Tarrier, Taylor, & Gooding, 2008). Thus, alternative treatment approaches and treatment targets may be necessary to improve results.
One alternative approach is to augment treatment as usual with a focus on building resiliency through increasing positive affect. Low levels of positive affect has been found to uniquely contribute to suicide risk, independent of other factors. In a follow-up study of suicidal adolescents discharged from an inpatient psychiatric hospitalization, low positive affect prospectively predicted time to suicide events (attempts or readmissions) over 6 months of follow-up, even after controlling for other predictors, including depression severity and anhedonia (Yen et al., 2013). The effect of positive affect on suicidal ideation has also been demonstrated in older patients. In a study of 462 primary care patients ages 65 and over, lower positive affect distinguished suicide ideators from non-ideators, after controlling for age, gender, depression, negative affect, illness burden, activity, sociability, cognitive functioning and physical functioning (Hirsch, Duberstein, Chapman, & Lyness, 2007). Thus, targeting positive affect to reduce suicidal behaviors may hold promise.
There has been an emergence of interventions in the past decade that aim to increase positive affect. Most of these interventions involve exercises, such as writing letters of gratitude, counting one’s blessings or identifying three good things, performing random acts of kindness, meditating on positive feelings towards others, mindfulness meditation, savoring, and positive writing. Interventions involving multiple exercises, include Fordyce’s happiness intervention (Fordyce, 1977), Fava’s well-being-intervention (Fava, 1998), Frisch’s quality of life intervention (Frisch, 1998), and most notably, Seligman’s positive psychotherapy (PPT) (Seligman, Rashid, & Parks, 2006). Two recent meta-analyses of positive psychology interventions found that these interventions have small to moderate effects in enhancing well-being and decreasing depressive symptoms (Bolier et al., 2013; Sin & Lyubomirsky, 2009). However, the vast majority of these interventions were directed towards pre-clinical populations, with nearly half of these studies conducted on college populations. Only a handful of studies recruited from clinical or hospital settings (Emmons & McCullough, 2003; Fava, Rafanelli, Cazzaro, Conti, & Grandi, 1998; Fava et al., 2005; Seligman et al., 2006). Furthermore, to our knowledge, no study has examined the effect of positive affect interventions on suicidal behavior.
Fredrickson’s empirically-supported Broaden and Build theory can be extended to serve as a conceptual model for the association between positive affect and suicidal behavior. Whereas negative emotions have been theorized to serve an important and evolutionarily adaptive role in constricting attention to threatening stimuli (Frijda, Kuipers, & Ter Schure, 1989; Lazarus, 1991), the Broaden and Build model asserts that the function of positive affect may be to broaden attentional scope to help individuals be open to novel stimuli and social supports, and in turn broadening and building psychological and social resources (Fredrickson, 2001; Fredrickson & Branigan, 2005; Fredrickson, Cohn, Coffey, Pek, & Finkel, 2008; Fredrickson & Joiner, 2002; Fredrickson & Losada, 2005; Fredrickson, Tugade, Waugh, & Larkin, 2003). It has been demonstrated that inducing positive emotions in laboratory conditions results in improved problem solving and increased social support (Fredrickson et al., 2003). Each of these areas are critical to suicide prevention. Thus, based on the Broaden and Build theory of positive emotions, we developed an intervention that focuses on teaching skills to increase attention to positive affect and experiences, with the goal of decreasing suicidal behaviors.
We developed and refined a multi-modal, manualized, adjunctive positive affect intervention for high risk suicidal adolescents and their parents. Multiple modalities included: 1) an in-person phase to deliver the positive affect skills and strategies and to personalize the intervention; 2) a remote delivery phase of weekly phone calls and daily text messages to extend the reach of the intervention into participants’ home environments. Unlike other positive affect interventions, we began the implementation of our protocol on the adolescent inpatient psychiatric unit, targeting the adolescents at highest risk for suicidal behavior. Implementation of a text-messaging enhanced intervention in an acute setting is important to reach a segment of the clinical population that may be less inclined to remain engaged in ongoing outpatient therapy. Thus, the primary objective of this pilot intervention is to develop a positive affect skills based program for an acute setting, with technology assisted reminders to practice skills in participant’s in-vivo settings, and to examine preliminary indices of feasibility, acceptability, and clinical outcomes.
Methods
Participants
Participants were 20 patients recruited from an adolescent psychiatric inpatient unit, who were admitted due to concerns of suicide risk (e.g., attempt or ideation). Of these, 12 (60%) were admitted following a suicide attempt. Participants were between the ages of 12–18 (mage = 15.9, SD = 1.5), predominately female (75%), and non-Hispanic White (80%), living at home with a primary guardian, and proficient in English. Nearly a third (31.3%) had a gross annual family income of less than $50,000. Thirteen (65%) reported a lifetime suicide attempt, and eight participants (40%) reported additional previous hospitalizations, with average age of first hospitalization being 15.4 years (SD = 1.8). Seven had been hospitalized in the past due to suicidal behavior. Past abuse (physical, sexual, emotional, neglect) was reported by 8 (40%) participants, as ascertained by chart review and confirmed through interview with parents. Medical record review indicated that on admission, all participants presented with a mood disorder (55% MDD; 20% Depressive Disorder NOS; 25% Mood disorder NOS), three (15%) met criteria for a substance use disorder; and six (30%) presented with an anxiety disorder. Exclusion criteria, ascertained from medical chart review, included active psychotic disorder, autism disorder, cognitive deficits, and being a ward of the state. Participants were screened using medical charts and consultation with the attending psychiatrist.
Recruitment occurred from March 2014 to January 2015. Of the 43 participants who were deemed potentially eligible based on chart review, 24 patient/parent pairs were approached, and of those, 20 enrolled and signed parental consent and adolescent assent, resulting in an 83% response rate. (The discrepancy between numbers of eligible vs. approached participants is mostly due to very brief inpatient admissions, or difficulty in reaching parents for informed consent). Of the 20 that enrolled, one withdrew participation after the treatment phase, stating insufficient time to participate in the follow-up assessments. Feasibility results presented here are based on all 20 participants; acceptability, and preliminary efficacy are based on all available data.
STEP Intervention
Based on the Broaden and Build theory of emotion, we purport that increased attention to positive affect and experiences, directly and indirectly leads to decreases in suicidal thoughts and behavior. STEP focuses on providing psychoeducation on the function of positive emotions and three sets of skills: mindfulness meditation, gratitude, and savoring. Psychoeducation on the Broaden and Build theory and establishing the rationale for the intervention is critical to achieving initial buy-in from potentially skeptical adolescents. Mindfulness meditation is taught to help with mood monitoring and attentional awareness to positive affect and experiences. Gratitude and savoring were selected because they have been demonstrated to increase sustainable and eudaimonic (vs. transient, hedonic) positive affect in community and depressed adults (Bolier et al., 2013; Sin & Lyubomirsky, 2009). Participants were introduced to three practices within each of the three skills set: 1) mindfulness meditation (i.e., breathing mindfulness exercise, compassion meditation, mantra meditation); 2) gratitude (i.e., three good things, gratitude expression, random acts of kindness); and 3) savoring (i.e., sharing good things, mindfully attending to positive moments, journaling of positive experiences). See Table 1 for additional description of each skills practice.
Table 1.
Skills to Enhance Positive Affect
Meditation | Gratitude | Savoring |
---|---|---|
Compassion Meditation, also called loving-kindness mediation is a mindfulness exercise to develop the mental habit of compassion, for self and/or for others. To begin, identify 3–4 mantras that may be relevant (e.g. May I have peace, May I have acceptance, May I have compassion, May I have forgiveness). Inhale on “May I”; exhale on the mantra. | 3 Good Things is a journaling exercise in which one writes down three positive occurrences that happened throughout the day (every night). The goal of this exercise is to shift attention to positive experiences which can otherwise be forgotten or discounted. | Sharing good things is an exercise in which you share positive experiences with others. This could be something positive that happened at school (e.g. praise from a teacher, a compliment from a friend, or being thanked for something you did). It could be retelling a funny story, or sharing good news. It is intended to promote lingering and savoring of positive emotions while building personal and interpersonal resources. |
Mindful Breathing Meditation is an exercise to quiet the mind by focusing on one’s breathing. There are many ways to practice mindful breathing and it can be done anywhere. To begin, practice 10 deep breaths, making sure that the inhale is deep and expands the abdomen. | Gratitude exercises involve the expression of gratitude to someone either through a letter, phone call or visit. You can also express gratitude to yourself or to God. This exercise can be adapted to express gratitude using mental imagery in which you envision expressing gratitude to someone, or envision a circle of loved ones to invoke a sense of gratitude. It is okay to reflect on the gratitude without expressing it, however, the expression maximizes the effect of this exercise as it makes this exercise an interpersonal one in which positive emotions may get expressed back to you. | Savoring is an exercise in which one sustains pleasurable moments or experiences. This can be done with routine daily activities that we often take for granted (e.g., food, music, art, nature, pets)This can be extended to activity, conversations, friends, and happy memories, really focusing and concentrating on the positive sensations from those activities. |
Movement Meditation is an exercise that involves awareness of one’s body and bodily sensations. This can be practiced in a number of ways, such as mindful walking, dancing, yoga, or a simple stretch. Move in a carefree manner without judgment. | Acts of kindness is an exercise in which you do a random act of kindness for someone else, several times a week. This act of kindness should be something that you are not currently doing regularly (e.g. thank someone for listening to you, bring in the mail for an elderly neighbor, offer a seat to someone on the bus, etc.). | Journaling of positive memory or experience. In this exercise, find a time every day to write in a journal. Choose a positive memory or experience, recent or past that you want to reflect on and savor. Read and review the positive events you have written about. |
Participants were introduced to each of the three skills practice (e.g., breathing mindfully, three good things, journaling positive experiences) within each set of skills strategy (i.e., mindfulness medication, gratitude, savoring). However, they were asked to select only one from each set to practice as we believed that personalization by having adolescents choose specific skills practices helps achieve buy-in. Related to this, we sought to convey the message that there is individual variability in the effectiveness of skills, such that not all skills are effective for a particular individual, and that there are multiple ways of achieving mindfulness, gratitude and savoring. Thus, if a skill does not appear to be working for them, we encouraged participants to try a different one. This is particularly relevant to those that may resist learning skills or believe that skills are not helpful (e.g., “I’ve tried mindful breathing exercises, they don’t work for me”) to encourage them to try alternative methods (e.g., “Some people have a hard time with the stillness, maybe you might prefer a movement mindfulness exercise”). Another reason for the choice of one exercise is that we sought to not overwhelm adolescents with too many skills to learn and practice in a short amount of time, as typical duration of a hospital stay is six days, leaving only 3–4 days for the intervention after consent and baseline assessments are completed.
There were two phases of STEP, an in-person phase consisting of three individual sessions and one family session delivered on the inpatient unit or shortly after discharge, followed by a remote-delivery phase which consisted of one month of daily text messaging and weekly phone calls to facilitate practice of mood monitoring and positive affect skills.
In-person Phase
The in-person sessions commenced on the adolescent inpatient unit, and continued on consecutive weekdays. Whenever possible, all four sessions were completed during the patients’ hospital stay. All sessions were conducted by either the Principal Investigator or a psychology post-doctoral fellow with a Ph.D. in Counseling Psychology. The fellow was trained and supervised by the PI. Training included review of the treatment manual, listening to audio-recordings of the PI conducting the intervention, and individual meetings with the PI. Supervision involved the PI reviewing recorded sessions and weekly feedback. The STEP program was adjunctive; therefore, all participants received the usual care on the inpatient unit which includes individual sessions with the psychiatrist and therapy groups throughout the day.
Session 1 focused on building rapport, and assessment of the participants’ suicide risk and protective factors. Interventionists sought to ascertain the precipitants to the current suicidal crises, details surrounding the suicide attempt or ideation, and what factors protected them from dying by suicide. Session 1 also focused on providing psychoeducation on the functions of positive and negative mood affect, rationale for exercises to increase attention to positive affect and positive experiences, and the importance of mood monitoring. This was accompanied by a hand out that summarized the psychoeducational content. In session 2, three sets of strategies (mindfulness meditation, gratitude, and savoring) and specific skills within each strategy, were introduced. Multiple exercises within each strategy were taught so that patients could select the exercises that best suited their needs, selecting at least one exercise from each set of strategies. In session 2, the interventionist practiced each selected skill with the participant, and asked them to practice each of their selected skills the next day. Session 3 focused on reviewing the skills selected in the prior session, reviewing the skills practice, and problem-solving any obstacles to skills practice. If skills were not practiced, the interventionist guided the participant through another skills practice. This session also focused on getting the participant ready to practice skills in their home environment by discussing daily routines, potential obstacles, and identifying potential solutions. Finally, the interventionist reviewed with the participant what they should expect during the remote delivery phase of the intervention. The family session also occurred during the inpatient admission, at varying times depending on parent availability. In the family session, the goals of the STEP program were reviewed, including a review of psychoeducation on positive and negative affect, and the specific skills selected by the adolescent to practice. Parental support was enlisted to reinforce practice of the positive affect skills.
Remote-Delivery Phase
The remote delivery phase of the intervention began immediately after discharge. This was comprised of weekly telephone booster calls and daily text messages over four weeks post-discharge. The password protected text messages were sent through an automated system, at a time of day selected by the study participant. The first set of messages prompted for mood monitoring, followed by a request for the participant to select the type of skill (i.e., mindfulness meditation, gratitude, or savoring) that they would like to practice (“Choose the type of message you would like to receive right now: 1 = mindfulness; 2 = gratitude; 3 = savoring”). In response, the participant received a skill to practice (e.g., “share something positive with a friend or family member”). If the participant did not respond to the daily text query after 30 minutes, a random skills practice reminder would be automatically sent from a bank of text messages. Therefore, participants always received a text message reminder to practice a positive affect skill regardless of whether or not they responded to any of the text messages. STEP interventionists were instructed to log on to the text messaging portal to review responses on a daily basis. Responses which were outside of the expected range of values, or that were of particular concern, prompted a telephone call to the participant and/or parent. Participants were told that if they were feeling unsafe to talk to a parent or their therapist, of if not available, go to the nearest ER. For any texts that were sent that were not expected, an autoreply was sent “No one is reading messages right now. If you need help right now, please call your therapist or 911.” Weekly phone calls were scheduled to check in to assess suicide risk and to reinforce skills practice, and to make any adjustments to their skills practice if necessary.
Procedure
After hearing a detailed description of the study procedures, adolescent participants signed assent to participate, and parents signed informed consent. In addition, release of information forms for all area emergency departments were signed so investigators could ascertain whether any re-admissions due to suicide risk occurred during the follow-up interval, mitigating loss of data due to attrition.
Each adolescent and participating parent were administered separate baseline assessments. Assessments consisted of structured clinical interviews as well as self-report questionnaires. Upon completion of the baseline assessment, the in-person phase of STEP, (three individual sessions and one family session) commenced. Upon discharge, participants received text messages every day for one month that prompted for mood monitoring and delivered a positive affect skill to practice for that day.
A post-treatment assessment (after one month of daily text messages), was conducted with adolescent participants and parents separately. An in-person follow-up assessment which took place three months post-treatment, was also conducted. In addition, participants and parents received a six month (post-discharge) follow-up phone call to assess for the main clinical outcomes of the study, i.e., suicide attempts and readmissions. If recent suicidal behavior or ideation was reported during follow-up, research assistants were instructed to contact a study investigator for a safety check. All participants and parents were compensated for their time and ancillary costs associated with text messaging: $40 upon completion of the baseline assessment, $60 upon completion of post-treatment assessment, $50 upon completion of the follow-up assessment, and $10 upon completion of the follow-up phone call at six months post-discharge.
Assessments
Demographic information including sex, race, ethnicity, living arrangements, parents’ level of education and income, and clinical information such as treatment history, abuse history, and family history of psychiatric disorders, were assessed using the KSADS-PL Screen Interview (Kaufman et al., 1997). Patient’s psychiatric diagnoses were ascertained through medical chart review.
Suicidal behavior characteristics were assessed at baseline, post-treatment, and follow-up intervals using the Columbia Suicide Severity Rating Scale (C-SSRS) (Posner, Oquendo, Gould, Stanley, & Davies, 2007), a semi-structured interview that assesses for frequency and intensity of behaviors including suicide attempts, aborted and interrupted attempts, preparatory acts, and suicidal ideation, in the week prior and over lifetime. Non-suicidal self-injurious behaviors were also assessed on the C-SSRS. Suicidal ideation was assessed at baseline and post-treatment and follow-up intervals using the Suicide Ideation Questionnaire (SIQ) (Reynolds, 1985), a 30-item self-report instrument to assess frequency of suicide related thoughts. The SIQ has been demonstrated to be reliable (Cronbach’s α = .97; α = .88 in present sample) and has adequate concurrent and construct validity in an adolescent clinical population (Reynolds, 1991).
A modified version of the Client Satisfaction Questionnaire (CSQ) was administered to adolescent participants and parents (Attkisson & Zwick, 1982). Modifications were made to examine specific components of the STEP intervention including modality (i.e., individual session, family session, phone sessions, text messaging) and content (i.e., functions of positive affect, mindfulness, gratitude, and savoring). Responses were on a 4-point likert scale with higher scores corresponding to greater satisfaction. Cronbach’s alpha of the modified CSQ based on the present sample ranged between 0.80 (adolescent post-treatment) to 0.94 (parent post-treatment).
Data Analyses
The primary objective of this Phase 1a open pilot study was to develop the STEP intervention and to write up a treatment manual (Rounsaville, Carroll, & Onken, 2001). In addition we sought to examine feasibility, acceptability, and preliminary clinical outcomes of the intervention using engagement and satisfaction as proxies, respectively. Feasibility is operationalized based on the number of STEP sessions attended in addition to responses to the text messages, and acceptability is operationalized based on responses to satisfaction questions for STEP and its treatment components from both adolescent participants and parents. Descriptive statistics were examined for suicide attempts, behaviors and readmissions, and compared to findings from historical controls, i.e., naturalistic studies based on the same inclusion criteria and from the same recruitment site. Pre-post measures of suicidal ideation based on the SIQ were also examined.
Results
Feasibility
All 20 participants (100%) completed at least two of the four sessions; 16 (80%) completed three 3 sessions, and 9 (45%) completing all four in-person sessions. Sessions were generally 30–45 minutes in duration, and presented on different consecutive weekdays, unless we were aware of a pending discharge in which case, sessions were presented concurrently. Consistent with our prior experience in delivering adjunctive interventions in inpatient settings, incomplete delivery of sessions were either due to discharge from the inpatient psychiatric unit prior to completions (40%), or lack of availability of parent (15%). In anticipation of these problems, the STEP intervention was designed such that the majority of skills and rationale content were delivered in sessions 1 and 2. Furthermore, Session 3 (review, practice, and personalization to home environment) could be completed on an outpatient basis, and Session 4 is predominantly a review with the family. However, in the pilot study, all sessions were conducted while the participant was on the inpatient unit. When participants were discharged prior to the completion of the 4 sessions, attempts to have them return for the remaining sessions were unsuccessful, resulting in a number of no-shows or repeated re-scheduling. Participants were however, willing to speak by phone or continue with the text messaging component of the intervention. Therefore, in these situations we proceeded with enrolling the participant in the text messaging component of STEP prior to completion of the all four sessions.
With regard to the remote-phase delivery, we observed higher participation rates from texting vs. phone. Of the weekly phone calls, 60% of calls with adolescents and 56% of calls to parents were completed, with three adolescents and two parents (including one adolescent/parent set) completely nonresponsive to phone outreach. Responsiveness to text messaging was operationalized by daily response rate to text prompts. Participants were not compensated for any responses to text messages; thus data reflects naturalistic response tendencies. On average, participants responded on 72.4% of days (range: 20–100%). Two participants who did not respond to any phone outreach were highly responsive (95–100%) to text messaging. Furthermore, our program was such that even those who do not engage by responding to prompts still receive an automatic skills practice reminder after one hour of non-response. As one participant stated in follow-up feedback “I like getting the reminder texts but I don’t like having to respond.” For the most part, our analyses did not find any demographic or clinical characteristics that predicted response text messaging rates. The one exception was that participants with non-college educated parents had higher responses compared to those with college educated parents (t = 2.38; p = .04).
Acceptability
Table 2 provides a summary of the questions and responses regarding satisfaction with various components of STEP. Global ratings for quality of services and satisfaction with the program were high in both adolescents and parents, across both time points of post-treatment and follow-up. With regard to the modality of treatment, text messaging was rated as more helpful than weekly phone calls by both adolescents and parents. Parents uniformly responded that the joint family sessions were helpful, while a few teens reported that it didn’t help or in one case (at follow-up only), reported that the family session made things worse. With regard to content, psychoeducation on the functions of positive emotions was the most highly rated component, with gratitude skills being rated least helpful. Even so, 84–86% of respondents across both time points reported that gratitude skills helped somewhat or a great deal. During the remote delivery phase in which participants were asked to choose the type of skills they wanted to practice, mindfulness meditation was the most commonly selected skill (66%), compared to gratitude (20%) and savoring (14%). Feedback from participants and parents suggested that the text messaging was particularly helpful with comments such as: “send text messages longer than a month to make sure the practice of positive emotions becomes habitual” (from adolescent) and, “I think the text messages should continue through three months post hospitalization. Some days that may be the only support the child sees and it’s VERY important for her to be continually reminded that she has support and is not alone” (from parent).
Table 2.
Patient and parent satisfaction with STEP
Question | Response Options | Post-Tx | Follow-Up |
---|---|---|---|
| |||
How would you rate the quality of services you received? | Excellent | 8 (53%)a; 9 (64%)p | 4 (31%)a; 8 (62%)p |
Good | 7 (47%)a; 3 (17%)p | 9 (69%)a; 4 (31%)p | |
Fair | 0; 2 (14%)p | 0; 1 (8%)p | |
Poor | 0; 0 | 0; 0 | |
| |||
How helpful was it to learn about the possible functions of positive emotions? | A great deal | 9 (60%)a; 8 (57%)p | 6 (46%)a; 8 (62%)p |
Somewhat | 5 (33%)a; 6 (43%)p | 6 (46%)a; 4 (31%)p | |
Didn’t help | 1 (7%)a; 0 | 1 (8%)a; 1 (8%)p | |
Made things worse | 0; 0 | 0; 0 | |
| |||
How helpful was it to have a joint parent and adolescent meeting to review skills? | A great deal | 5 (36%)a; 9 (64%)p | 4 (22%)a; 8 (62%)p |
Somewhat | 6 (43%)a; 5 (36%)p | 7 (39%)a; 5 (38%)p | |
Didn’t help | 3 (21%)a; 0 | 1 (8%)a; 0 | |
Made things worse | 0; 0 | 1 (8%); 0 | |
| |||
How helpful was it to have weekly phone calls? | A great deal | 2 (13%)a; 5 (28%)p | 2 (15%)a; 7 (54%)p |
Somewhat | 7 (47%)a; 8 (44%)p | 6 (46%)a; 4 (31%)p | |
Didn’t help | 6 (40%)a; 1 (7%)p | 5 (39%)a; 2 (15%)p | |
Made things worse | 0; 0 | 0; 0 | |
| |||
How helpful was it to have text messages? | A great deal | 6 (40%)a; 6 (46%)p | 6 (46%)a; 8 (62%)p |
Somewhat | 7 (39%)a; 6 (46%)p | 6 (46%)a; 4 (31%)p | |
Didn’t help | 2 (13%)a; 1 (8%)p | 1 (7%)a; 1 (8%)p | |
Made things worse | 0; 0 | 0; 0 | |
| |||
How helpful were the mindfulness skills? | A great deal | 8 (53%)a; 8 (62%)p | 4 (31%)a; 5 (39%)p |
Somewhat | 5 (28%)a; 3 (23%)p | 8 (62%)a; 7 (54%)p | |
Didn’t help | 2 (13%)a; 2 (15%)p | 1 (6%)a; 1 (8%)p | |
Made things worse | 0; 0 | 0; 0 | |
| |||
How helpful were the gratitude skills? | A great deal | 4 (27%)a; 5 (36%)p | 4 (31%)a; 4 (31%)p |
Somewhat | 9 (60%)a; 7 (50%)p | 6 (46%)a; 6 (46%)p | |
Didn’t help | 2 (13%)a; 2 (15%)p | 3 (23%)a; 3 (23%)p | |
Made things worse | 0; 0 | 0; 0 | |
| |||
How helpful were the savoring skills? | A great deal | 10 (67%)a; 4 (29%)p | 7 (54%)a; 5 (42%)p |
Somewhat | 4 (27%)a; 8 (57%)p | 4 (31%)a; 5 (42%)p | |
Didn’t help | 1 (7%)a; 2 (14%)p | 2 (15%)a; 2 (17%)p | |
Made things worse | 0; 0 | 0; 0 | |
| |||
Overall, how satisfied are you with the program you received? | Very satisfied | 8 (53%)a; 8 (57%)p | 6 (46%)a; 8 (62%)p |
Mostly satisfied | 7 (47%)a; 6 (43%)p | 6 (46%)a; 4 (31%)p | |
Indifferent/mildly satisfied | 0; 0 | 1 (8%)a; 1 (8%)p | |
Quite dissatisfied | 0; 0 | 0; 0 |
Notes.
= adolescent;
= parent;
percentages based on valid responses
Preliminary Outcomes
Data on suicide events (attempts and readmissions due to suicide risk), using clinical follow-up interviews from the C-SSRS as well as medical records from all area emergency departments were also examined. In the year after discharge, only one study participant (5%) made a suicide attempt and five (25%) had been readmitted due to suicide risk. This is lower than the 19% who attempted suicide and the 37% readmitted to the adolescent inpatient unit due to suicide risk, observed in the Yen et al., (2013) study which recruited participants with the same inclusion/exclusion criteria from the same inpatient psychiatric unit and the same follow-up duration (Yen et al., 2013). Furthermore, large effect sizes were observed for suicidal ideation as assessed by the SIQ, with significant decreases from baseline (M = 114.69; SD = 26.58) to post-treatment (M = 59.92; SD = 50.22; d = 1.07), and from baseline to follow-up (M = 39.00; SD = 34.59; d = 2.97).
Discussion
Findings from this pilot development trial suggest that Steps to Enhance Positivity, a two-part intervention for suicidal adolescent inpatients, is, in general, feasible to administer and acceptable to adolescent patients and their parents. STEP differs from other positive affect interventions in that it is implemented during a hospital admission, a time of acute crisis and need, and targets suicidal behavior in the highest-risk adolescents during the initial months following inpatient psychiatric discharge, the period of highest risk for suicide attempts (Goldston et al., 1999; Meehan et al., 2006; Olfson et al., 2016). Furthermore, influenced by the Broaden and Build theory, it has a specific emphasis on increasing attention to positive affect and experiences. It also utilizes multiple modalities of skills delivery, specifically in-person sessions, phone calls, and text messaging, to extend the reach of treatment which is particularly relevant for the vulnerable subset of adolescents who experience barriers to conventional outpatient psychotherapy (James et al., 2010; Romansky, Lyons, Lehner, & West, 2003).
Our preliminary data suggests that if positive affect skills such as mindfulness meditation, gratitude, and savoring were regularly delivered in an inpatient setting, they would be well received: our inpatient sessions got very high satisfaction ratings from adolescents and parents. The completion of the in-person phase of STEP was, however, variable. Given the brief duration of inpatient days and the loss of potential treatment days to study related activities (e.g., obtaining informed consent/assent, baseline study assessment), this is not entirely surprising and was the basis for developing STEP such that all of the skills content were delivered within the first two sessions, which were attended by all participants. We developed the intervention so that sessions 3 and 4 could be completed on an outpatient basis; however, our experience was that once participants were discharged many were unlikely to return for an in-person session (e.g., repeated no-shows or cancellations). This represents a challenge in working with this population as they are often engaged in crisis-driven treatment. However, many participants and families who did not return to complete the in-person sessions were willing to check in by phone and participate in the text-messaging intervention. Nonetheless, our data suggests that improvements to feasibility are needed. For example, integrating STEP skill sessions into the standard inpatient group schedule, as opposed to it being delivered individually as part of a research protocol, will likely increase feasibility of skills delivery. While this would make the family session infeasible, our data suggests that adolescent participants had mixed reactions to the family session. Additionally, as the text-messaging component is automated and time limited (one month), with unexpected responses systematically flagged, it is feasible for a unit staff member to check the text messaging portal daily and triage flagged messages to bring to the attention of a mental health provider. This is a relatively low-cost intervention that can potentially reduce suicidal behavior and hospital re-admissions, and result in health care cost savings.
Engagement in the remote delivery phase, which includes weekly phone outreach and daily text messaging, was much higher. Corroborating our and others’ work suggesting that text messages may be more acceptable and disseminable for adolescents than other forms of mental health treatment, several participants who did not respond to phone calls were highly responsive to text messages (Aguilera & Berridge, 2014; Aguilera & Munoz, 2011; Ranney et al., 2014). On average, participants responded to the text messaging prompts on 72% of the days in which they received messages, and notably, were not incentivized for responding. Response rates were quite variable; however, engagement appeared to be generalizable across a number of demographic (e.g., age, sex, gender, race, ethnicity, income) and clinical characteristics (e.g., psychiatric diagnoses, past history of suicide attempts, abuse). The one exception in which those with parents without college education had higher rates of engagement than those with college-educated parents suggest that the reach of text messaging may perhaps be even more instrumental for patients of lower socio-educational background. The high engagement in text messaging is particularly important because, as stated by one of our parent participants, for many adolescents, on most days, this is the only intervention they receive. Moreover, as adherence to outpatient therapy is often tenuous due to pragmatic barriers such as transportation costs and parents’ work schedules, obstacles that many lower income families feel acutely, text-based skills reminders delivered to their in-vivo environment can be critical (Kazdin, Holland, & Crowley, 1997).
Our most notable clinical finding was the low rate of suicide attempts in our sample, after six months of follow-up. The 5% rate observed in the present study is substantially lower (vs. 19%) than that of another naturalistic study conducted with the same inclusion and exclusion criteria and at the same recruitment site (Yen et al., 2013) as well as other naturalistic studies. In a different study of 58 adolescents admitted to an inpatient unit for a suicide attempt, 3 months post-discharge, 45% reported continued suicidal ideation and 12% reported a repeat suicide attempt (Spirito, Valeri, Boergers, & Donaldson, 2003). In yet another study of 115 suicidal and non-suicidal adolescents, 16 participants (13.9%) reported suicide attempts within 3 months post-discharge and 19 participants (13% of the sample) reported suicide attempts within 6 months post-discharge (Prinstein et al., 2008). Similarly, our rate of inpatient readmission (25%) were lower than that observed in other naturalistic studies (37%) with similar inclusion and exclusion criteria (Yen et al., 2013). Furthermore, our rates for both suicide attempts and readmissions were obtained after a comprehensive review of medical records in the two major healthcare systems of our state. Thus, this is a promising finding, albeit based on a small sample, and warrants further examination in a randomized controlled trial.
There are a number of limitations to the current study. First, our sample was limited with regard to gender, racial, and ethnic diversity; larger trials with adequate representations of males, gender, racial and ethnic minorities are necessary to determine whether STEP is feasible and acceptable to the general population. Second, the present study did not have a control group, thus it is difficult to determine whether the improvements observed were in fact related to the intervention; however, our rate of suicide events was lower than that observed in other studies in our unit (Yen et al., 2013). Our findings suggest a need for further examination of positive affect mechanisms to determine if they underlie successful clinical outcomes of the positive skills intervention for suicidal adolescent inpatients, and suggests a need for adequately powered clinical trials. Finally, administration of STEP as a stand-alone research protocol rather than a program fully implemented in the context of an inpatient setting limited our ability to estimate of feasibility in a naturalistic setting.
In conclusion, the results obtained from this pilot study demonstrate promise for an adjunctive positive affect skills intervention with both in-person delivery of skills coaching supplemented with remote outreach via text messaging, in reducing suicidal behavior during the highest risk time of discharge from an inpatient psychiatric unit. Our sample was comprised of adolescents in acute crisis with variable histories of seeking and maintaining outpatient treatment. Therefore, skills that can be taught in acute care settings and maintained via remote outreach could extend the reach of current interventions, particularly among those less willing to engage in traditional outpatient therapy. The specific skills taught, mindfulness meditation, gratitude, and savoring had broad appeal to the adolescents. The problems that precipitate suicidal crises in adolescents are varied and unlikely to be resolved entirely through an increased focus on positive affect and experiences. However, our preliminary data suggests that when combined with treatment as usual, adjunctive practice of positive affect skills may hold promise in decreasing suicide attempts and readmissions in high risk suicidal adolescents
Acknowledgments
This work was supported by the National Institute of Mental Health under Grant R34 MH101272 to Dr. Yen.
Biographies
Shirley Yen, Ph.D. is an Associate Professor (Research) in the Department of Psychiatry and Human Behavior, Alpert Medical School, Brown University. Her research areas include assessment and treatment of suicidal behaviors in adolescents and young adults, borderline personality disorder, and bipolar disorder. She is the principal investigator of this study.
Megan Ranney, M.D. is Associate Professor in the Department of Emergency Medicine at Rhode Island Hospital, Alpert Medical School, Brown University; and Director and Founder of the Brown Emergency Digital Health Innovation program (www.brownedhi.org). Her research focuses on developing, testing, and disseminating digital health interventions for vulnerable emergency department patients.
Anthony Spirito, Ph.D. is a Professor in the Department of Psychiatry and Human Behavior at Alpert Medical School, Brown University. His primary area of research is in the treatment of adolescent depression, suicidality, and substance use. He has conducted both intensive and brief treatment protocols for adolescents with co-occurring problems.
Katherine Tezanos, B.A. is a first-year clinical psychology doctoral student at Teachers College, Columbia University. Her research interests include (1) Examining risk factors for suicide using a developmental lens; (2) Measurement and prediction of suicidal thoughts and behaviors among adolescents.
Christopher Kahler, Ph.D. is Professor and Chair of the Department of Behavioral and Social Sciences in the Brown University School of Public Health and an Associate Director of the Center for Alcohol and Addiction Studies. His research focuses on developing novel smoking cessation treatments and implementing interventions to reduce heavy drinking.
Adam Chuong, B.A. is currently in graduate student at the Rhode Island School of Design. He was a research assistant on this study, and hopes to bring his expertise in industrial design to technology based interventions for mental health to reduce treatment disparities.
Joel Solomon, M.D. is the Associate Medical Director of Child and Adolescent Services at Butler Hospital, and Clinical Assistant Professor in the Department of Psychiatry and Human Behavior at Alpert Medical School, in Providence, Rhode Island. His primary research interest is in reducing suicidal behaviors in adolescents.
Contributor Information
Shirley Yen, Alpert Brown Medical School of Brown University, Box G-BH, Providence RI 02906, Ph: 401-444-1915 / Fax: 401-444-1948, Shirley_Yen_PhD@Brown.edu
Megan L. Ranney, 55 Claverick St., Providence, RI 02903, Ph: 401-444-2557 / Fax: 401-444-2249, Megan_Ranney@Brown.edu.
Anthony Spirito, Alpert Brown Medical School of Brown University, Box G-BH, Providence RI 02906, Ph: 401-444-1929 / Fax: 401-444-1948, Anthony_Spirito@Brown.edu
Katherine Tezanos, Teachers College, Columbia University, 525 West 120th Street, New York, NY 10027, Ph: 212-678-7487 / Fax: 212-678-8105, kt2604@columbia.edu
Adam Chuong, Alpert Brown Medical School of Brown University, Box G-BH, Providence RI 02906, Ph: 401-444-1905 / Fax: 401-444-1948, adam.chuong@gmail.com
Christopher W. Kahler, Center for Alcohol and Addiction Studies and the Department of Behavioral and Social Sciences, Brown University School of Public Health, Box G-S121-4, Providence RI 02912, Ph: 401-863-6651 / Fax: 401-863-6697, Christopher_Kahler@Brown.edu
Joel Solomon, Butler Hospital, 345 Blackstone Blvd, Providence, RI 02906, Ph: 401-455-6200 / Fax: 401-455-6214, JSolomon@Butler.org
References
- Aguilera A, Berridge C. Qualitative feedback from a text messaging intervention for depression: benefits, drawbacks, and cultural differences. JMIR Mhealth Uhealth. 2014;2(4):e46. doi: 10.2196/mhealth.3660. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Aguilera A, Munoz RF. Text Messaging as an Adjunct to CBT in Low-Income Populations: A Usability and Feasibility Pilot Study. Prof Psychol Res Pr. 2011;42(6):472–478. doi: 10.1037/a0025499. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Attkisson CC, Zwick R. The client satisfaction questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome. Eval Program Plann. 1982;5(3):233–237. doi: 10.1016/0149-7189(82)90074-x. [DOI] [PubMed] [Google Scholar]
- Bolier L, Haverman M, Westerhof GJ, Riper H, Smit F, Bohlmeijer E. Positive psychology interventions: a meta-analysis of randomized controlled studies. BMC Public Health. 2013;13(1):119. doi: 10.1186/1471-2458-13-119. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Emmons RA, McCullough ME. Counting blessings versus burdens: an experimental investigation of gratitude and subjective well-being in daily life. J Pers Soc Psychol. 2003;84(2):377–389. doi: 10.1037//0022-3514.84.2.377. [DOI] [PubMed] [Google Scholar]
- Fava GA. Prevention of Recurrent Depression With Cognitive Behavioral Therapy: Preliminary Findings. Archives of General Psychiatry. 1998;55(9):816–820. doi: 10.1001/archpsyc.55.9.816. [DOI] [PubMed] [Google Scholar]
- Fava GA, Rafanelli C, Cazzaro M, Conti S, Grandi S. Well-being therapy. A novel psychotherapeutic approach for residual symptoms of affective disorders. Psychol Med. 1998;28(2):475–480. doi: 10.1017/s0033291797006363. [DOI] [PubMed] [Google Scholar]
- Fava GA, Ruini C, Rafanelli C, Finos L, Salmaso L, Mangelli L, Sirigatti S. Well-being therapy of generalized anxiety disorder. Psychother Psychosom. 2005;74(1):26–30. doi: 10.1159/000082023. [DOI] [PubMed] [Google Scholar]
- Fordyce MW. Development of a program to increase personal happiness. Journal of Counseling Psychology. 1977;24(6):511–521. doi: 10.1037/0022-0167.24.6.511. [DOI] [Google Scholar]
- Fredrickson BL. The role of positive emotions in positive psychology. The broaden-and-build theory of positive emotions. Am Psychol. 2001;56(3):218–226. doi: 10.1037//0003-066x.56.3.218. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fredrickson BL, Branigan C. Positive emotions broaden the scope of attention and thought-action repertoires. Cogn Emot. 2005;19(3):313–332. doi: 10.1080/02699930441000238. [doi] [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fredrickson BL, Cohn MA, Coffey KA, Pek J, Finkel SM. Open hearts build lives: positive emotions, induced through loving-kindness meditation, build consequential personal resources. J Pers Soc Psychol. 2008;95(5):1045–1062. doi: 10.1037/a0013262. doi:2008-14857-004 [pii] 10.1037/a0013262 [doi] [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fredrickson BL, Joiner T. Positive emotions trigger upward spirals toward emotional well-being. Psychol Sci. 2002;13(2):172–175. doi: 10.1111/1467-9280.00431. [DOI] [PubMed] [Google Scholar]
- Fredrickson BL, Losada MF. Positive affect and the complex dynamics of human flourishing. Am Psychol. 2005;60(7):678–686. doi: 10.1037/0003-066X.60.7.678. doi:2005-11834-001 [pii] 10.1037/0003-066X.60.7.678 [doi] [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fredrickson BL, Tugade MM, Waugh CE, Larkin GR. What good are positive emotions in crises? A prospective study of resilience and emotions following the terrorist attacks on the United States on September 11th, 2001. J Pers Soc Psychol. 2003;84(2):365–376. doi: 10.1037//0022-3514.84.2.365. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Frijda NH, Kuipers P, Ter Schure E. Relations among emotion, appraisal, and emotional action readiness. Journal of personality and social psychology. 1989;57(2):212. [Google Scholar]
- Frisch MB. Quality of Life Therapy and Assessment in Health Care. Clinical Psychology: Science and Practice. 1998;5(1):19–40. doi: 10.1111/j.1468-2850.1998.tb00132.x. [DOI] [Google Scholar]
- Goldston DB, Daniel SS, Reboussin DM, Reboussin BA, Frazier PH, Kelley AE. Suicide attempts among formerly hospitalized adolescents: a prospective naturalistic study of risk during the first 5 years after discharge. J Am Acad Child Adolesc Psychiatry. 1999;38(6):660–671. doi: 10.1097/00004583-199906000-00012. [DOI] [PubMed] [Google Scholar]
- Hirsch JK, Duberstein PR, Chapman B, Lyness JM. Positive affect and suicide ideation in older adult primary care patients. Psychology and Aging. 2007;22(2):380–385. doi: 10.1037/0882-7974.22.2.380. [DOI] [PMC free article] [PubMed] [Google Scholar]
- James S, Charlemagne SJ, Gilman AB, Alemi Q, Smith RL, Tharayil PR, Freeman K. Post-discharge services and psychiatric rehospitalization among children and youth. Adm Policy Ment Health. 2010;37(5):433–445. doi: 10.1007/s10488-009-0263-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kann L, Kinchen S, Shanklin SL, Flint KH, Hawkins J, Harris WA, Chyen D. Youth risk behavior surveillance—United States, 2013. 2014 [PubMed] [Google Scholar]
- Kaufman J, Birmaher B, Brent D, Rao U, Flynn C, Moreci P, Ryan N. Schedule for Affective Disorders and Schizophrenia for School-Age Children-Present and Lifetime Version (K-SADS-PL): initial reliability and validity data. J Am Acad Child Adolesc Psychiatry. 1997;36(7):980–988. doi: 10.1097/00004583-199707000-00021. [DOI] [PubMed] [Google Scholar]
- Kazdin AE, Holland L, Crowley M. Family experience of barriers to treatment and premature termination from child therapy. J Consult Clin Psychol. 1997;65(3):453–463. doi: 10.1037//0022-006x.65.3.453. [DOI] [PubMed] [Google Scholar]
- Lazarus RS. Emotion and adaptation. Oxford University Press on Demand; 1991. [Google Scholar]
- Meehan J, Kapur N, Hunt IM, Turnbull P, Robinson J, Bickley H, Appleby L. Suicide in mental health in-patients and within 3 months of discharge. National clinical survey. Br J Psychiatry. 2006;188:129–134. doi: 10.1192/bjp.188.2.129. [DOI] [PubMed] [Google Scholar]
- Minino A. Mortality among teenagers aged 12–19 years: United States, 1999–2006. NCHS Data Brief. 2010;(37):1–8. [PubMed] [Google Scholar]
- Nock MK, Borges G, Bromet EJ, Cha CB, Kessler RC, Lee S. Suicide and Suicidal Behavior. Epidemiologic Reviews. 2008;30(1):133–154. doi: 10.1093/epirev/mxn002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Olfson M, Wall M, Wang S, Crystal S, Liu SM, Gerhard T, Blanco C. Short-term Suicide Risk After Psychiatric Hospital Discharge. JAMA Psychiatry. 2016;73(11):1119–1126. doi: 10.1001/jamapsychiatry.2016.2035. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Posner K, Oquendo MA, Gould M, Stanley B, Davies M. Columbia Classification Algorithm of Suicide Assessment (C-CASA): classification of suicidal events in the FDA's pediatric suicidal risk analysis of antidepressants. Am J Psychiatry. 2007;164(7):1035–1043. doi: 10.1176/appi.ajp.164.7.1035. doi:164/7/1035 [pii] 10.1176/appi.ajp.164.7.1035 [doi] [DOI] [PMC free article] [PubMed] [Google Scholar]
- Prinstein MJ, Nock MK, Simon V, Aikins JW, Cheah CSL, Spirito A. Longitudinal trajectories and predictors of adolescent suicidal ideation and attempts following inpatient hospitalization. Journal of Consulting and Clinical Psychology. 2008;76(1):92–103. doi: 10.1037/0022-006X.76.1.92. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Ranney ML, Choo EK, Cunningham RM, Spirito A, Thorsen M, Mello MJ, Morrow K. Acceptability, language, and structure of text message-based behavioral interventions for high-risk adolescent females: a qualitative study. J Adolesc Health. 2014;55(1):33–40. doi: 10.1016/j.jadohealth.2013.12.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Reynolds WM. Suicide Ideation Questionnaire. Odessa, FL: Psychological Assessment Resources; 1985. [Google Scholar]
- Reynolds WM. Psychometric characteristics of the Adult Suicidal Ideation Questionnaire in college students. J Pers Assess. 1991;56(2):289–307. doi: 10.1207/s15327752jpa5602_9. [doi] [DOI] [PubMed] [Google Scholar]
- Romansky JB, Lyons JS, Lehner RK, West CM. Factors related to psychiatric hospital readmission among children and adolescents in state custody. Psychiatr Serv. 2003;54(3):356–362. doi: 10.1176/appi.ps.54.3.356. [DOI] [PubMed] [Google Scholar]
- Rounsaville BJ, Carroll KM, Onken LS. A stage model of behavioral therapies research: Getting started and moving on from stage I. Clinical Psychology: Science and Practice. 2001;8(2):133–142. [Google Scholar]
- Seligman ME, Rashid T, Parks AC. Positive psychotherapy. Am Psychol. 2006;61(8):774–788. doi: 10.1037/0003-066X.61.8.774. [DOI] [PubMed] [Google Scholar]
- Sin NL, Lyubomirsky S. Enhancing well-being and alleviating depressive symptoms with positive psychology interventions: a practice-friendly meta-analysis. J Clin Psychol. 2009;65(5):467–487. doi: 10.1002/jclp.20593. [doi] [DOI] [PubMed] [Google Scholar]
- Spirito A, Valeri S, Boergers J, Donaldson D. Predictors of continued suicidal behavior in adolescents following a suicide attempt. J Clin Child Adolesc Psychol. 2003;32(2):284–289. doi: 10.1207/S15374424JCCP3202_14. [DOI] [PubMed] [Google Scholar]
- Tarrier N, Taylor K, Gooding P. Cognitive-behavioral interventions to reduce suicide behavior: a systematic review and meta-analysis. Behav Modif. 2008;32(1):77–108. doi: 10.1177/0145445507304728. doi:32/1/77 [pii] 10.1177/0145445507304728 [doi] [DOI] [PubMed] [Google Scholar]
- Xu J, Murphy SL, Kochanek KD, Bastian BA. Deaths: Final Data for 2013. Natl Vital Stat Rep. 2016;64(2):1–119. [PubMed] [Google Scholar]
- Yen S, Weinstock L, Andover M, Sheets E, Selby E, Spirito A. Prospective predictors of adolescent suicidality: 6-month post-hospitalization follow-up. Psychological medicine. 2013;43(05):983–993. doi: 10.1017/S0033291712001912. [DOI] [PMC free article] [PubMed] [Google Scholar]