Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2019 Sep 1.
Published in final edited form as: Behav Ther. 2019 Jan 24;50(5):886–897. doi: 10.1016/j.beth.2019.01.004

Intervening on Thwarted Belongingness and Perceived Burdensomeness to Reduce Suicidality among Veterans: Subanalyses from a Randomized Controlled Trial

Nicole A Short 1, Lauren Stentz 1, Amanda M Raines 2, Joseph W Boffa 1, Norman B Schmidt 1
PMCID: PMC6703169  NIHMSID: NIHMS1012951  PMID: 31422845

Abstract

Suicide is a growing public health crisis among military veterans. Despite recent attention to this area, there are few empirically supported preventative interventions for suicidality among veterans. In the context of an empirically supported theoretical framework, the Interpersonal Theory of Suicide, the current study targeted suicide risk factors (i.e., perceived burdensomeness and thwarted belongingness) among a sample of 46 veterans selected from a larger clinical trial. Participants were randomized to receive either a newly developed computerized intervention aimed at decreasing perceived burdensomeness and thwarted belongingness, or participate in a repeated contact control condition. Results indicated a direct effect of the intervention on both perceived burdensomeness and thwarted belongingness. Temporal mediation analyses also revealed an indirect effect of condition on suicidality at Month-1 follow-up via reductions in perceived burdensomeness. The current results are the first to indicate that factors from the interpersonal theory of suicide can be reduced among veterans, and to demonstrate that these reductions in perceived burdensomeness lead to reductions in suicidality. Because of the brevity and computer delivery system, this intervention could be widely and rapidly disseminated among military veterans to reduce the public health burden of suicide in this population.

Keywords: suicide, suicidality, veterans, perceived burdensomeness, thwarted belongingness

Suicide is the tenth leading cause of death in the United States (CDC, 2016). Of particular concern, military veterans have become a focus of attention due to rising suicide rates for veterans returning from recent conflicts (i.e., Operation Enduring Freedom [OEF] and Operation Iraqi Freedom [OIF]). Indeed, military veterans are at roughly a 21% higher risk for suicide compared to civilian adults (VA, 2016). Furthermore, between 2001 and 2014, age-adjusted suicide rates increased by 32.2% among veterans compared to an increase of 23.0% among US Civilian Adults (VA, 2016). The problem of increasing suicide rates extends to active duty military personnel as active duty combat-exposed males are, for the first time, at greater risk for suicide than civilians (Kang & Bullman, 2008; Kaplan, Huguet, McFarland, & Newsom, 2007). Some unique factors of recent conflicts could help to explain this: multiple deployments, medical advancements allowing more personnel to survive combat-related injuries, and traumatic brain injuries (Schell & Marshall, 2008). In sum, both active duty military service members and veterans’ risk for suicide is greater than for civilians, and is growing, suggesting a need for attention toward theoretically-based interventions that could prevent suicide among veteran populations.

There is a growing body of empirically-supported suicide risk reduction treatments that have been studied among the general population. However, a very limited number have been tested in a military setting. Bryan and Rozen (2018) noted a total of only four such clinical trials of interventions whose primary outcomes involved targeting suicidal thoughts and behaviors that were developed in non-military settings and investigated among military personnel. Indeed, another recent review conducted by Harmon and Colleagues (2016) yielded a mere five studies that examined military suicide prevention programs and their outcomes, underscoring a lack of empirical evidence supporting the efficacy of enacted military suicide prevention programs. Furthermore, these studies typically did not use randomized controlled trials, precluding the ability to establish causal effects. Finally, the interventions often focused on training others to identify and address suicide risk status among their trainees, rather than directly intervening on theoretically driven risk factors for suicide among those at risk (e.g., James & Kowalski, 1996; McDaniel, Rock, & Grigg, 1990). This approach may be useful considering many in the military do not present for mental health treatment (Hoge et al., 2004); however, intervening on theoretically driven risk factors could be a way to increase the potency of current suicide risk prevention techniques. Some trials have integrated theoretically driven factors such as strengthening social support (Knox, Litts, Talcott, Feig, & Caine, 2003), and utilizing cognitive behavioral therapy (Rudd et al., 2015). Indeed, Bryan and Rosen (2018) noted interventions targeting proposed theoretical factors were particularly effective in reducing suicidality.

In addition to focusing on theoretical mechanisms underlying suicide risk among military personnel, it is also important to consider the dissemination potential of any suicide risk intervention. Recent research has highlighted the potential utility of computerized interventions in targeting a variety of psychological outcomes, including depression and anxiety (Andrews, Cuijpers, Craske, McEvoy, & Titov, 2010; McHugh & Barlow, 2010; Richards & Richardson, 2012). Few studies have examined whether computerized or technology-based interventions may reduce risk for suicide. One randomized clinical trial tested a 3-hour self-help intervention based on cognitive behavioral therapy and related variants (e.g., Dialectical Behavioral Therapy) targeting suicidal ideation, emotion regulation, thought challenging, and relapse prevention among a sample of participants with mild to moderate suicide risk. This active intervention produced small but significant effects on decreasing suicidal ideation compared to a waitslist control (van Spijker, van Straten, & Kerkhof, 2014). Although other studies have targeted help-seeking behavior by providing personalized feedback and psychoeducation (King et al., 2015) or used telehealth or email interventions among individuals at risk for suicide (i.e., Caring Letters; Luxton, June, & Comtois, 2013), no other studies to our knowledge have used technology-based interventions to directly target suicide risk.

One theory that may be relevant to suicide risk in the military is the interpersonal-psychological theory. The interpersonal theory is an empirically-validated model that identifies perceived burdensomeness (PB) and thwarted belongingness (TB) as key risk factors for suicidality (Joiner, 2005; Van Orden, Lynam, Hollar, & Joiner, 2006; Van Orden, Witte, Gordon, Bender, & Joiner Jr, 2008). PB is the interpretation that one is a burden or liability on those around them. TB is the sentiment that one’s existence is void of meaningful relationships with others. Experiencing feelings of PB and TB together produce suicidal ideation, but alone are insufficient to result in death by suicide. The theory posits that in addition to PB and TB, the individual must have the ability to inflict lethal self-harm. Joiner (2005) suggests that PB and TB may be malleable factors that change throughout one’s lifespan, while capability for suicide, particularly “practical capability,” is thought to be less amenable to reductions (though more recent research has questioned this; e.g., Klonsky & May, 2015).

Several studies have examined TB/PB and suicide risk among active duty service members. Indeed, constructs such as PB may be even more salient among veterans who often have co-occuring medical or mental health conditions that may impact their ability to fulfill responsibilities they formerly completed. Furthermore, the transition from active duty to retirement may exacerbate feelings of burdesomeness as individuals may feel they lost their “purpose” they were fulfilling during their time in service. These notions have been confirmed empirically. Using samples of personnel actively deployed to Iraq, Bryan, Clemans, and Hernandez (Bryan, Clemans, & Hernandez, 2012) found that PB was associated with increased suicidality. Similar findings were reported in another sample of US Air Force cadets who had recently completed basic training (Bryan, Morrow, Anestis, & Joiner, 2010). Extending these findings, Anestis, Khazem, Mohn, and Green (Anestis, Khazem, Mohn, & Green, 2015) reported that the interaction of PB and TB predicted suicidal ideation, plans, and preparations among individual’s serving in the US National Guard. Aspects of the interpersonal-psychological theory have also been supported in veteran samples. Using a qualitative approach, Brenner and colleagues (Brenner et al., 2008) coded qualitative statements and found themes to support the role of combat service in the development of PB and TB among OEF/OIF combat veterans, further underscoring the importance of these constructs within military populations. Building upon these findings, Monteith, Menefee, Pettit, Leopoulos, and Vincent (2013) reported an interactive effect of TB and PB on suicidal ideation among veterans entering psychiatric treatment. Though studies have noted more of an effect of PB on suicidality than TB (Chu et al., in press), findings collectively support the role of the interpersonal model with regard to suicidality in military samples (Monteith et al., 2013), indicating that intervening on PB and TB could be a promising direction to pursue to reduce their suicide risk.

In sum, suicide has become a public health crisis among military populations, and there is little empirical guidance as to preventative interventions that could reduce this burden. Furthermore, there is a wealth of evidence for PB and TB as predictors of suicidality, and growing evidence that these risk factors are relevant to military populations. Despite this, no studies have attempted to reduce PB and/or TB as a preventative intervention for suicide among veterans, or examined whether reductions in PB and/or TB are mechanisms underlying suicide risk reduction protocols. To address this gap in the literature, the current study tested the direct and indirect effects of a newly developed, computerized intervention directly targeting PB and TB against a repeated contact control among a sample of US military veterans who participated in a larger clinical trial. We hypothesized that the active PB/TB intervention would result in immediate self-reported reductions in PB and TB after the intervention compared to a repeated contact control. Furthermore, we hypothesized an indirect effect of condition such that those in the active condition would evidence greater reductions in Month-1 suicidality (Depression Severity Index – Suicidality Subscale) via Post-intervention reductions in PB and TB. Finally, we hypothesized this indirect effect would be specific to PB and TB, our intended treatment targets, rather than other general factors related to suicide (i.e., negative affect).

Methods

Participants

Participants consisted of community outpatients (N=303) recruited to participate in a randomized clinical trial examining the efficacy of a computerized intervention targeting risk factors associated with anxiety and mood disorders, and suicide (i.e., elevated PB or TB; Capron et al., 2012; Van Orden et al., 2010). A power analysis was used to determine the overall sample size, but not the subanalyses reported in the current manuscript. Data are part of a larger randomized controlled trial (Schmidt et al., in preparation), but the current results have not been reported elsewhere. To be eligible, participants were required to present with elevated risk on one or more risk factors based on scoring at or above the community mean for these measures (i.e., PB > 9 on the INQ-R, TB > 21 on the INQ-R, or anxiety sensitivity cognitive concerns > 9 on the Anxiety Sensitivity Index-3; Schmidt & Joiner, 2002; Van Orden, Cukrowicz, Witte, & Joiner Jr, 2012), be at least 18 years of age, and be proficient in English. Exclusion criteria included psychotic or bipolar spectrum disorders not treated with medication, or unstable psychiatric medication usage (i.e., had taken the same psychiatric medications for the past six weeks).

The current study only included veterans who participated in the active mood risk reduction treatment (n = 23) or the repeated contact control (n = 23) and completed both session 3 and Month-1 follow-ups, resulting in an overall sample for the current study of 46 (see Table 2 and Figure 1). We only included the active mood condition and the repeated contact control because the other two conditions focused on other risk factors for suicidality (i.e., anxiety sensitivity - cognitive concerns; Capron et al., 2012). The sample ranged in age from 23 to 79 years old (M=48.4, SD=13.70), and was primarily male (86.8%). About half (58.5%) identified as African-American, 37.7% as Caucasian/European-American, and 3.8% ‘Other’ (e.g., multiracial). Of these participants, 30.2% provided details of their military history because these items were added later on in the study. The majority of those who provided military details were involved in the Army (50.0%), followed by Navy (25.0%), Marine Corps (12.5%), Army Reserve (6.3%), and Air Force (6.3%), 56.3% of which reported having been deployed during their military career (two in OIF, two in OEF, two in Vietnam, one in Desert Storm, and one in Honduras). Two-thirds (66%) of the sample reported current psychiatric medication usage. The majority (86.8%) received at least one current psychiatric diagnosis (47.2% mood disorder, 30.2% PTSD, 64.2% anxiety disorder, and 19.9% substance use disorder). Nearly the entire sample (98.1%) self-reported some degree of active suicidal ideation as assessed by the Beck Scale for Suicidal Ideation, item 6 (this item assesses time spent “thinking about killing myself”; Beck, Kovacs, & Weissman, 1979).

Table 2.

Means, standard deviations, and zero-order correlations for all variables of interest

1 2 3 4 5 6 7 8
1. BL INQ-R TB -- -- -- -- -- -- -- --
2. BL INQ-R PB .71*** -- -- -- -- -- -- --
3. BL PANAS-NA .36** .48*** -- -- -- -- -- --
4. BL DSI-SS .53*** .55*** .45** -- -- -- -- --
5. Post INQ-R TB .62*** .43** .50*** .48** -- -- -- --
6. Post INQ-R PB .49** .53*** .59*** .56*** .67*** -- -- --
7. Post PANAS-NA .34* .42** .82*** .52*** .49** .65*** -- --
8. M1 DSI-SS .40** .41** .37* .86*** .39** .62*** .39** --
Active   M 36.16 15.16 26.04 1.44 31.83 11.88 19.75 .91
     SD 14.39 10.36 9.37 2.18 14.81 6.86 8.83 1.83
Control   M 37.59 17.19 26.48 .93 37.78 17.04 23.70 .78
     SD 13.19 10.40 11.51 1.94 13.21 10.68 11.93 2.13
Open in a new tab

Note. BL=baseline; Post=post-intervention; M1=month-1 followup; INQ-R=Interpersonal Needs Questionnaire – Revised; TB=Thwarted Belongingness PB=Perceived Burdensomeness, DSI-SS=Depression Severity Index – Suicide Scale;PANAS-NA=Positive Negative Affect Schedule – Negative Affect subscale

*

p < .05

**

p < .01

***

p < .001.

Figure 1.

Figure 1.

Open in a new tab

CONSORT Diagram.

Procedures

Participants were recruited from the community via advertisements targeted toward individuals interested in treatment for anxiety, depression, and suicide. We specifically recruited veterans using targeted flyers and by coordinating with local veterans organizations. Interested participants called to complete a brief telephone screen and were scheduled for a baseline appointment, to complete the Structured Clinical Interview for DSM-5- research version (SCID; First, Williams, Karg, & Spitzer, 2015), and self-report measures. They were then randomized to one of four conditions, two of which are focused on in the current study. Research assistants randomized participants using a random numbers table. All intervention sessions occurred in our laboratory once weekly for three consecutive weeks, and participants returned for a 1-month follow-up. Participants were compensated following each appointment. All participants provided written informed consent to participate in research and treatment, and all study procedures were approved by the University’s Institutional Review Board.

Experimental Conditions

Mood intervention condition.

The mood intervention condition comprised a psychoeducational and cognitive bias modification (CBM) component, both of which were fully computerized. Participants reported to the lab one per week over three weeks to complete the intervention. The intervention was designed to target TB and PB. To do so, the last author and doctoral students with experience treating depression consulted the literature regarding the IPT of suicide and their clinical experience to develop a list of common myths surrounding PB/TB (e.g., Sharing my problems with or being around others would be a burden to them; No one wants me around, etc.), which would be targeted in the intervention. Furthermore, this group consulted the literature and with military mental health providers to determine how PB/TB may relate to military service, and integrated some information specific to veterans. For example, the intervention discusses how being in the military can facilitate a sense of belongingness, which is often lost after one’s service is complete.

Overall, the psychoeducation component included education about depression and social interaction, as well as cognitive and behavioral techniques to combat these problems, presented in a 50-minute video complete with narration, animation, and interactive features. This was only given during the first intervention session. The psychoeducation was intended to mirror principles of cognitive behavioral therapy to correct problematic ideas related to PB/TB (e.g., “If you are around other people, you shouldn’t feel lonely”, “Talking to others about your problems makes you a burden”). The program presents social interaction as a human necessity, such as food or water, and that negative beliefs about being isolated or burdensome are often inaccurate. Information dispelling myths surrounding PB/TB were followed with a rationalization and examples of behavioral activation techniques (e.g., talking to a friend, sharing your feelings with someone you trust, volunteering). See Figure 2 for example screenshots.

Figure 2.

Figure 2.

Open in a new tab

Depiction of Mediation Results.* = p <.05. B = unstandardized regression coefficient. B1 refers to mediation models with Perceived Burdensomeness, B2 refers to paths with Thwarted Belongingness.

The CBM component was developed in accord with the positive CBM-I paradigm developed by Holmes and colleagues (2009). The CBM component was administered in each of participants’ three intervention sessions. Participants were presented with 100 scenarios across five training blocks, each block containing 20 scenarios, which were read aloud in E-Prime. Participants were provided headphones to listen to 10–13 second scenarios, with two seconds between each description. The scenarios are initially ambiguous scenario to train participants to generate positive outcomes. Participants began by completing emotionality ratings for ambiguous test paragraphs (“You have recently taken an important exam. Your results arrive with an unexpected letter of explanation about your grade”). Participants then completed the training phase, during which they were read ambiguous scenarios which resolve positively (e.g., “At your computer lesson you finish your work early so the lecturer gives you a new tasks to do. You don’t understand the task so you ask for help. The lecturer tells you that your request is the sign of being a good student.”). During the training phase, participants are instructed to imagine the events while listening to them, and then rate the vividness of the imagery (“How vividly could imagine the situations described?”). Following a five minute break, participants completed the testing phase, in which they rated the emotional valence of ambiguous test scenarios (e.g., “You buy a new outfit for a party. Other people’s reactions show how you look”).

Repeated-contact control.

Participants randomized to the RCC returned to the laboratory for their three weekly appointments according to the same schedule as the mood condition. During these appointments, participants completed suicide risk assessments with trained clinical psychology doctoral students or research assistants using techniques recommended by Chu et al. (2015). Appropriate preventative strategies were taken for individuals with elevated risk (e.g., safety plan development, means safety, etc.; Chu et al., 2015). It is important to note that similar procedures, such as repeated contact or even receiving letters have been shown to reduce suicidality (Carter, Clover, Whyte, Dawson, & Este, 2005; Fleischmann et al., 2008).

Measures

Structured Clinical Interview for DSM-5- Research Version (SCID).

The SCID (First et al., 2015) was used to evaluate participants’ diagnostic status. SCIDs were administered by trained clinical psychology graduate students. All SCIDs were reviewed by a licensed clinical psychologist to confirm accuracy. Agreement between interviewers for a random sample of SCID interviews from our laboratory resulted in high inter-rater agreement (i.e., over 80% with a kappa value of .86; n = 20; Schmidt et al., 2017).

Interpersonal Needs Questionnaire-Revised (INQR).

The INQ-R is a 15-item self-report questionnaire assessing PB and TB. The INQ-R was used to assess PB and TB at baseline and session 3. It has demonstrated strong psychometric properties (Van Orden et al., 2012). In the present sample, demonstrated excellent reliability at baseline and session 3 (α’s=.94 and .95, respectively). The PB scale exhibited excellent reliability at baseline and session 3 (α’s=.95 and .94, respectively). The TB scale exhibited good reliability at baseline and session 3 (α’s=.88 and .92, respectively). The INQ-R was used at baseline and session 3 to provide an assessment of PB and TB prior to and immediately following treatment.

Depressive Symptoms Inventory- Suicidality Subscale (DSI-SS).

The DSI-SS consists of four self-report items assessing suicidal ideation, plans, control of suicidal thoughts, and suicidal impulses; it has demonstrated strong psychometric properties in prior studies (Metalsky & Joiner Jr, 1997). The DSI-SS was administered at Baseline and Month-1. In the present sample, it demonstrated excellent internal consistency at baseline and month-one follow-up (α’s=.92 and .93, respectively). The DSI-SS was used at baseline and Month-1 to provide an assessment of suicidality prior to and one-month after treatment (to assess temporal mediation, with changes in PB/TB occurring prior to changes in suicidality).

Positive and Negative Affect Schedule- Negative Affect subscale (PANAS-NA).

The PANAS-NA consists ten items associated with trait negative affect. Participants were asked to indicate on a 5-point scale how much they had experienced a particular emotion (e.g., distressed, guilty) during the past week at Basline and Post-Intervention. The PANAS-NA scale is a psychometrically sound measure of negative affect (Watson & Clark, 1994). In the present sample it demonstrated excellent internal consistency (α’s for both were .94). The PANAS-NA was used at baseline and session 3 to provide an assessment of NA prior to and immediately following treatment.

Results

Descriptive statistics

First, means, standard deviations, and zero-order correlations were examined (Table 2). Mean burdensomeness and belongingness scores were higher than in a previous active duty sample (Bryan, Morrow, Anestis, & Joiner, 2009), but lower than in a sample of veterans with depression (Pfeiffer et al., 2014a). Mean levels of suicidality were higher than in an active duty military sample (Ribeiro et al., 2014). Correlations were in the expected directions, with all variables positively significantly correlated. There was no evidence of non-normality based on visual inspection of the data as well as skewness and kurtosis values (all skewness statistics < 2.0; all kurtosis statistics < 6.5). No threats to or violations of homoscedasticity or multicollinearity were found. T-tests revealed no significant differences between conditions across baseline variables, supporting equivalence of random assignment. Participants with missing data (n=8 of N=53, resulting in the current sample size of 46) were deleted listwise, as is default with the PROCESS Macro (Preacher & Hayes, 2004). There were no differences between completers and non-completers on TB/PB (p > .182), but there were differences between completers and non-completers in terms of suicidality (t [50] = 3.64, p = .001; completers M = 3.28, SD = 6.12, and non-completers M = .00, SD = .00).

Primary analyses

For all analyses, condition (Active=1, Control=0) was the independent variable, Post-Intervention scores for INQ-R TB/PB were mediators, and Month-1 DSI-SS scores were used as dependent variables, while baseline assessment of DSI-SS and the relevant INQ-R measure were used as covariates. Temporal mediation models were conducted using bias-corrected bootstrapping analyses with Preacher and Hayes’ macro for SPSS (Preacher & Hayes, 2008). These analyses utilized 5000 bootstrap resamples to compute values for the path(s) from condition to mediator (α) and the path(s) from mediator to outcome (ß), as well as the indirect effect (α ß), yielding 95% confidence intervals of the indirect effect based on the bootstrap distribution. All path estimates are presented as unstandardized coefficients by convention.

First, we conducted a temporal mediational model analyzing the indirect effect of condition on Month-1 suicidality via Post-Intervention burdensomeness, covarying for Baseline suicidality and PB (Figure 3). The entire model was significant and accounted for 77% of the variance in Month-1 suicidality scores (F (4, 40)=33.21, p < .001). First, the relationship between condition and burdensomeness (α path) was significant, with those in the mood condition experiencing significantly greater reductions in PB post-intervention compared to the control (B=−4.86, SE=1.97, p=.018). Next, the relationship between PB and Month-1 suicidality was not significant (B=.03, SE=.02, p=.169). Finally, the indirect effect of condition on Month-1 suicidality via PB was significant (B=−.17, 95% CI [−.56, −.002]). The direct effect of condition on Month-1 suicidality was not significant (c path; B=−.33, SE=.31, p=.292), nor was the effect of condition on Month-1 suicidality after accounting for burdensomeness (c’ path; B=−.16, SE=.33, p=.616). The ratio of the indirect effect to the total effect was 51.5%.

Figure 3.

Figure 3.

Open in a new tab

Example screenshots from the Perceived Burdensomeness/Thwarted Belongingness Intervention

Second, we examined a similar temporal mediational model analyzing the indirect effect of condition on Month-1 suicidality via Post-Intervention thwarted belongingness, covarying for Baseline suicidality and TB. The entire model was significant and accounted for 76% of the variance in Month-1 suicidality (F (4, 40)=32.02, p < .001). The effect of condition on Post-Intervention TB was significant (α path; B=−9.93, SE=4.59, p=.036). However, the effect of TB on Month-1 suicidality was not significant (b path; B < .01, SE=.01, p=.995). Furthermore, the indirect effect was not significant (B < .01, SE=.09, 95% CI[−.16, .23]). Neither the direct effect of condition on Month-1 suicidality (c path; B =−.35, SE=.33, p=.297), nor the direct effect of condition on suicidality after covarying for TB (c’ path; B=−.35, SE=.33, p=.296) were significant.

Specificity analyses

Finally, we examined negative affect as an alternative mediator to determine whether the indirect effect was specific to PB. The full model was significant and accounted for 77% of the variance in suicidality at Month-1 (F (4, 40)=33.26, p < .001). The effect of condition on negative affect showed a trend in the expected direction, with those in the active condition showing decreased Post-Intervention negative affect compared to those in the control (α path; B=−3.03, SE=1.73, p=.089). The effect of negative affect on Month-1 suicidality was not significant (b path; B=−.05, SE=.04, p=.094). The indirect effect of condition on suicidality through negative affect was not significant (B=.14, SE=.14, 95% CI[−.08, .48]). Neither the direct effect of condition on suicidality (c path; B=−.29, SE=.31, p=.362), nor the direct effect of condition on suicidality after coavarying for NA (c’ path; B=−.43, SE=.31, p=.181) were significant.

Post-hoc power analysis

Finally, due to the relatively low sample size, a post-hoc power analysis was computed in G*Power (Faul, Erdfelder, Lang, & Buchner, 2007) for the main effect of Condition on Post INQ-R burdensomeness. A post-hoc power analysis for mean differences between two groups was calculated, inputting Cohen’s d (.57), α = .05, and the sample size for each condition. Results yielded an observed power of .72.

Discussion

The present study investigated the effects of a brief, computerized intervention focused on interpersonal risk factors for suicide (i.e., PB and TB) vs. a repeated contact control on suicidality among clinical sample of veterans. To our knowledge, this is the first study to test whether PB and TB can be reduced among veterans, and whether these reductions would lead to reductions in suicidality. Furthermore, we utilized temporal mediation to confirm that reductions in PB preceded reductions in suicidality. First, findings revealed that our brief psychoeducational and CBM program significantly reduced both PB and TB among veterans. This is consistent with the theoretical view that PB and TB are malleable psychological risk factors (Van Orden et al., 2010), and with previous research finding that PB and TB are malleable among mixed samples of both civilians and veterans (Schmidt et al., In Preparation). However, the current findings expand upon this research by demonstrating that it is possible to reduce theoretically-driven suicide risks factor among veterans even with a brief computerized intervention. Finally, results were obtained when comparing our intervention to a repeated contact control, which, in itself, can be effective in reducing suicidality.

Second, results showed an indirect effect of condition on suicidality via PB. Again, this finding provides support for the interpersonal-psychological theory of suicide by being one of the first studies to demonstrate the causal role of PB in suicidality (Joiner, 2005), and the first to indicate this causal role among veterans. Furthermore, this finding is consistent with and expands upon several prior studies that have cross-sectionally demonstrated that PB is associated with suicide risk among veteran samples (Bryan et al., 2009; Pfeiffer et al., 2014a). The indirect effect of condition on suicidality via NA did not prove to be significant, serving as further evidence of specificity of PB as a mediator of suicidality. Taken together, findings provide further support for PB as a causal mechanism in suicidal ideation and behaviors.

Inconsistent with hypotheses, analyses revealed TB was not a mediator of suicide outcomes. While the relationship between condition and TB was significant, the indirect effect of condition on suicidality via TB was not. This is in contrast to studies finding that TB is associated with increased suicidality. However, these findings are consistent with more recent studies among veterans that have shown association between PB and suicidality, but not between TB and suicidality (Bell et al., 2017; Bryan et al., 2010; Monteith, Bahraini, & Menefee, 2017; Pfeiffer et al., 2014b). The notion that PB is more operative than TB in terms of predicting suicidality has also been supported by a recent meta-analysis across various populations (Chu et al., in press). Indeed, because PB is much less frequently endorsed than TB, any amount of PB may be associated with increased suicide risk (Bryan et al., 2012). On the other hand, TB is a much more common experience that may not always lead to suicidal ideation and behaviors. On the other hand, it is also possible that due to the relatively small sample size, we did not have the power to detect potentially effects of TB on suicidality. Indeed, a post-hoc power analysis supported this hypothesis, as power was below the recommended value of .80. Further research is needed to investigate whether TB plays a causal role in the development of suicidality. Finally, in contrast to previous findings in which only PB was decreased using our intervention in a mixed sample (Schmidt et al., In Preparation), the current results suggest that both PB and TB were decreased among veterans. Further research is needed to replicate these findings to determine whether TB is potentially more malleable among veterans in comparison to civilians.

The current findings also have clinical implications. Despite recent efforts, there is a dearth of empirically based suicide prevention tools, and even less options that can be easily disseminated to veteran populations. As such, results suggest that this brief, computerized intervention for TB and PB could be a promising direction to pursue to promote suicide prevention among veterans. In particular, during the transition from active duty to civilian life, many veterans may develop PB and TB. Specifically, during this time, veterans often experience emerging psychological symptoms (DiRamio, Ackerman, & Mitchell, 2008). Unfortunately, this distress is compounded by the fact that many veterans face strained or terminated relationships when they return home, or find that they miss the camaraderie of military life. Others feel as though they are unable to talk about their experiences in the military and find themselves becoming closed off from others, or feeling as though they are a burden on others (i.e., PB; DiRamio et al., 2008). Considering this, interventions such as this one could be useful in the transition from military to civilian life to reduce or prevent suicidality and potentially improve quality of life.

Results of the current study should be considered in the context of its limitations and directions for future research. First, not all participants in the sample had active suicidal ideation. Future research would benefit from assessing the effects of this intervention in a clinical sample of individuals with elevated suicidality or active suicidal ideation. Second, the majority of the sample was male. Although this is consistent with the demographics of the military and veterans overall (Patten & Parker, 2011), it is unclear if findings will generalize to female veterans. Third, results suggested that the intervention affected DSI-SS assessed suicidality, which comprises suicidal ideation, plans, and impulses. Although the DSI-SS is a common and validated measure of these aspects of suicidality (Joiner, Pfaff, & Acres, 2002), it is unclear if these results will translate to reductions in suicidal behaviors. Further research with large samples would be needed to directly test whether this intervention reduces the incidence of suicide attempts. Fourth, participants were followed up one month after treatment. Future research should include longer term follow-ups to determine the intervention’s durability and lasting impacts on suicide outcomes. Fifth, the veteran sample was relatively small, thus we may been underpowered to test some of the analyses. Future research should replicate these findings in larger samples. Sixth, some aspects of the PB/TB intervention were not necessarily specific to these constructs (i.e., the CBM portion). Although mediation analyses indicated PB was the mechanism accounting for treatment effects, it is possible there are other mechanisms relating to the CBM that we were unable to assess in the current study. Future research should consider these, or be more specific to PB/TB in all intervention components. Seventh, although the repeated contact control is a strength in that repeated contact may reduce suicidality (Carter, Clover, Whyte, Dawson, & Este, 2005; Fleischmann et al., 2008), it does not include other general treatment components that could account for participants improvement. As such, future research may benefit from including control conditions that include some kind of treatment component. Eighth, the current analyses did not use an intent-to-treat due to the preliminary nature of the study. Furthermore, there was evidence of no significant differences between completers vs. non-completers in terms of demographics and clinical characteristics, with the exception of suicidality. Yet in terms of suicidality, completers had significantly higher suicidality. Thus, concerns about missing data are somewhat mitigated given those with more severe suicidality actually were more likely to complete the study.

Despite these limitations, the current study provides a valuable step to the literature on suicide prevention in veteran samples. Our results suggest that a brief, computerized intervention targeting theoretically-based risk factors for suicide (i.e., TB and PB) reduces these risk factors, and, in turn, suicidality among veterans. Future research should continue to examine the effectiveness of this intervention across various measures of suicidal ideation and behavior, a longer follow-up period, and diverse samples of veterans, with the ultimate goal of determining whether targeting TB and PB is a novel and promising new target for reducing the significant public health burden of veteran suicide.

Table 1.

Sociodemographic and clinical characteristics of the sample

Control Active
% or M (SD) % or M (SD)
Age 38.59 (15.91) 36.37 (15.85)
Sex (% female) 49.2 55.2
Race
 White 42.9 59.7
 Black 47.6 26.9
 Asian 4.8 3.0
 Other 3.2 10.4
Ethnicity (% Hispanic) 3.2 11.9
Marital status
 Married or cohabitating 14.3 25.4
 Single 50.8 56.7
 Separated or divorced 30.1 16.4
 Widowed 4.8 1.5
Family Income
 Less than $10,000 31.7 26.9
 $10,000–25,000 34.9 19.4
 $25,000–40,000 12.7 16.4
 $40,000–75,000 7.9 25.4
 $75,000–100,000 6.3 4.5
 $100,000–150,000 4.8 6.0
 >$150,000 1.6 1.5
Education
 High school or equivalent 11.1 7.5
 Some college/2-year degree 65.1 54.3
 Four-year college degree 19.0 22.4
 Graduate school or higher 3.2 14.9
 ≥1 Psychiatric Disorder 93.7 94.0
Total Psychiatric Diagnoses 2.33 (1.37) 2.30 (1.38)
Anxiety Disorder 79.3 74.6
Mood Disorder 38.1 56.7
Posttraumatic Stress Disorder 20.6 34.3
Substance Use Disorder 22.3 16.4
Open in a new tab

Highlights.

  • A computerized treatment for suicide risk was tested

  • Treatment focused on perceived burdensomeness and thwarted belongingness

  • The intervention reduced suicide risk factors among veterans

  • Temporal mediation revealed an indirect effect on suicidality reductions via burdensomeness

Acknowledgments

This work was in part supported by the Military Suicide Research Consortium (MSRC), Department of Defense, and VISN 19 Mental Illness Research, Education, and Clinical Center (MIRECC) - award number: W81XWH-10-2-0181, but does not necessarily represent the views of the Department of Defense, Department of Veterans Affairs, or the United States Government. Support from the MSRC does not necessarily constitute or imply endorsement, sponsorship, or favoring of the study design, analysis, or recommendations. This study is registered at ClinicalTrials.Gov (Identifier: NCT01941862). The authors have no other conflicts of interest to disclose.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Andrews G, Cuijpers P, Craske MG, McEvoy P, & Titov N (2010). Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a meta-analysis. PloS one, 5(10). doi: 10.1371/journal.pone.0013196 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Anestis MD, Khazem LR, Mohn RS, & Green BA (2015). Testing the main hypotheses of the interpersonal–psychological theory of suicidal behavior in a large diverse sample of United States military personnel. Comprehensive Psychiatry, 60, 78–85. doi: 10.1016/j.comppsych.2015.03.006 [DOI] [PubMed] [Google Scholar]
  3. Beck AT, Kovacs M, & Weissman A (1979). Assessment of suicidal intention: the Scale for Suicide Ideation. Journal of Consulting and Clinical Psychology, 47(2), 343–352. doi: 10.1037/0022-006X.47.2.343 [DOI] [PubMed] [Google Scholar]
  4. Bell CM, Ridley JA, Overholser JC, Young K, Athey A, Lehmann J, & Phillips K (2017). The role of perceived burden and social support in suicide and depression. Suicide and Life-Threatening Behavior, 48,(1), 87–94. [DOI] [PubMed] [Google Scholar]
  5. Brenner L, Gutierrez P, Cornette M, Betthauser L, Bahraini N, & Staves P (2008). A qualitative study of potential suicide risk factors in returning combat veterans. Journal of Mental Health Counseling, 30(3), 211–225. doi: 10.17744/mehc.30.3.n6418tm72231j606 [DOI] [Google Scholar]
  6. Bryan CJ, Clemans TA, & Hernandez AM (2012). Perceived burdensomeness, fearlessness of death, and suicidality among deployed military personnel. Personality and Individual Differences, 52(3), 374–379. doi: 10.1016/j.paid.2011.10.045 [DOI] [Google Scholar]
  7. Bryan CJ, Morrow CE, Anestis MD, & Joiner TE (2009). A preliminary test of the interpersonal-psychological theory of suicide in a military sample. Personality and Individual Differences, 48, 347–350. doi: 10.1016/j.paid.2009.10.023 [DOI] [Google Scholar]
  8. Bryan CJ, Morrow CE, Anestis MD, & Joiner TE (2010). A preliminary test of the interpersonal-psychological theory of suicidal behavior in a military sample. Personality and Individual Differences, 48(3), 347–350. doi: 10.1016/j.paid.2009.10.023 [DOI] [Google Scholar]
  9. Bryan CJ, & Rozek DC (2018). Suicide prevention in the military: a mechanistic perspective. Current Opinion in Psychology, 22, 27–32. doi: 10.1016/j.copsyc.2017.07.022 [DOI] [PubMed] [Google Scholar]
  10. Capron DW, Fitch K, Medley A, Blagg C, Mallott M, & Joiner TE (2012). Role of anxiety sensitivity subfactors in suicidal ideation and suicide attempt history. Depression and Anxiety, 29(3), 195–201. doi: 10.1002/da.20871 [DOI] [PubMed] [Google Scholar]
  11. Carter GL, Clover K, Whyte IM, Dawson AH, & Este CD (2005). Postcards from the EDge project: randomised controlled trial of an intervention using postcards to reduce repetition of hospital treated deliberate self poisoning. British Medical Journal, 331(7520), 805–809. doi: 10.1136/bmj.38579.455266.E0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. CDC (Ed.). (2016). 10 Leading Causes of Death by Age Group, United States: - [Google Scholar]
  13. Chu C, Buchman-Schmitt JM, Stanley IH, Hom MA, Tucker R, Hagan CR, . . . E. JT (in press). The Interpersonal Theory of Suicide: A systematic review and meta-analysis of a decade of cross-national research. Psychological Bulletin. doi: 10.1037/bul0000123 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Chu C, Klein KM, Buchman-Schmitt JM, Hom MA, Hagan CR, & Joiner TE (2015). Routinized Assessment of Suicide Risk in Clinical Practice: An Empirically Informed Update. Journal of Clinical Psychology, 71(12), 1186–1200. doi: 10.1002/jclp.22210 [DOI] [PubMed] [Google Scholar]
  15. DiRamio D, Ackerman R, & Mitchell RL (2008). From combat to campus: Voices of student-veterans. NASPA Journal, 45(1), 73–102. doi: 10.2202/0027-6014.1908 [DOI] [Google Scholar]
  16. Faul F, Erdfelder E, Lang A-G, & Buchner A (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191. [DOI] [PubMed] [Google Scholar]
  17. First MB, Williams JBW, Karg RS, & Spitzer RL (2015). Structured Clinical Interview for DSM-5—Research Version (SCID-5 for DSM-5, Research Version; SCID-5-RV). Arlington, VA: American Psychiatric Association. [Google Scholar]
  18. Fleischmann A, Bertolote JM, Wasserman D, De Leo D, Bolhari J, Botega NJ, . . . Schlebusch L. (2008). Effectiveness of brief intervention and contact for suicide attempters: a randomized controlled trial in five countries. Bulletin of the World Health Organization, 86(9), 703–709. doi: 10.2471/BLT.07.046995 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Harmon LM, Cooper KR, Nugent WR, & Butcher JJ (2016). A review of the effectiveness of military suicide prevention programs in reducing rates of military suicides. Journal of Human Behavior in the Social Environment, 26(1), 15–24. doi: 10.1080/10911359.2015.1058139 [DOI] [Google Scholar]
  20. Hoge CW, Castro CA, Messer SC, McGurk D, Cotting DI, & Koffman RL (2004). Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care. New England Journal of Medicine, 351(1), 13–22. doi: 10.1056/NEJMoa040603 [DOI] [PubMed] [Google Scholar]
  21. Holmes EA, Lang TJ, & Shah DM (2009). Developing interpretation bias modification as a” cognitive vaccine” for depressed mood: imagining positive events makes you feel better than thinking about them verbally. Journal of Abnormal Psychology, 118(1), 76–88. doi: 10.1037/a0012590 [DOI] [PubMed] [Google Scholar]
  22. James LC, & Kowalski TJ (1996). Suicide prevention in an army infantry division: a multidisciplinary program. Military Medicine, 161(2), 97–101. doi: 10.1093/milmed/161.2.97 [DOI] [PubMed] [Google Scholar]
  23. Joiner TE (2005). Why people die by suicide. Cambridge, MA: Harvard University Press. [Google Scholar]
  24. Joiner TE, Pfaff JJ, & Acres JG (2002). A brief screening tool for suicidal symptoms in adolescents and young adults in general health settings: reliability and validity data from the Australian National General Practice Youth Suicide Prevention Project. Behavior Research and Therapy, 40(4), 471–481. doi: 10.1016/S0005-7967(01)00017-1 [DOI] [PubMed] [Google Scholar]
  25. Kang HK, & Bullman TA (2008). Risk of suicide among US veterans after returning from the Iraq or Afghanistan war zones. Journal of the American Medical Association, 300, 652–653. doi: 10.1001/jama.300.6.652 [DOI] [PubMed] [Google Scholar]
  26. Kaplan MS, Huguet N, McFarland BH, & Newsom JT (2007). Suicide among male veterans: A prospective population-based study. Journal of Epidemiology and Community Health, 61, 619–624. doi: 10.1136/jech.2006.054346 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. King CA, Eisenberg D, Zheng K, Czyz E, Kramer A, Horwitz A, & Chermack S (2015). Online suicide risk screening and intervention with college students: A pilot randomized controlled trial. Journal of Consulting and Clinical Psychology, 83(3), 630–636. doi: 10.1037/a0038805 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Klonsky ED, & May AM (2015). The three-step theory (3ST): A new theory of suicide rooted in the “ideation-to-action” framework. International Journal of Cognitive Therapy, 8(2), 114–129. doi: 10.1521/ijct.2015.8.2.114 [DOI] [Google Scholar]
  29. Knox KL, Litts DA, Talcott GW, Feig JC, & Caine ED (2003). Risk of suicide and related adverse outcomes after exposure to a suicide prevention programme in the US Air Force: cohort study. British Medical Journal, 327(7428), 1376–1381. doi: 10.1136/bmj.327.7428.1376 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Luxton DD, June JD, & Comtois KA (2013). Can postdischarge follow-up contacts prevent suicide and suicidal behavior? Crisis, 34(1), 32–41. doi: 10.1027/0227-5910/a000158 [DOI] [PubMed] [Google Scholar]
  31. McDaniel WW, Rock M, & Grigg JR (1990). Suicide prevention at a United States Navy training command. Military Medicine, 155, 173–175. doi: 10.1093/milmed/155.4.173 [DOI] [PubMed] [Google Scholar]
  32. McHugh RK, & Barlow DH (2010). The dissemination and implementation of evidence-based psychological treatments: a review of current efforts. American Psychologist, 65(2), 73–84. doi: 10.1037/a0018121 [DOI] [PubMed] [Google Scholar]
  33. Metalsky GI, & Joiner TE Jr (1997). The hopelessness depression symptom questionnaire. Cognitive Therapy and Research, 21(3), 359–384. doi: 10.1023/A:1021882717784 [DOI] [Google Scholar]
  34. Monteith LL, Bahraini NH, & Menefee DS (2017). Perceived burdensomeness, thwarted belongingness, and fearlessness about death: associations with suicidal ideation among female veterans exposed to military sexual trauma. Journal of Clinical Psychology, 73(12), 1655–1669. doi: 10.1002/jclp.22462 [DOI] [PubMed] [Google Scholar]
  35. Monteith LL, Menefee DS, Pettit JW, Leopoulos WL, & Vincent JP (2013). Examining the interpersonal–psychological theory of suicide in an inpatient veteran sample. Suicide and Life-Threatening Behavior, 43(4), 418–428. doi: 10.1111/sltb.12027 [DOI] [PubMed] [Google Scholar]
  36. Patten E, & Parker K (2011). Women in the U.S. Military: Growing Share, Distinctive Profile Retrieved 3/1, 2017, from http://www.pewsocialtrends.org/2011/12/22/women-in-the-u-smilitary-growing-share-distinctive-profile/
  37. Pfeiffer PN, Brandfon S, Garcia E, Duffy S, Ganoczy D, Kim HM, & Valenstein M (2014a). Predictors of suicidal ideation among depressed veterans and the interpersonal theory of suicide. Journal of Affective Disorders, 152, 277–281 doi: 10.1016/j.jad.2013.09.025 [DOI] [PubMed] [Google Scholar]
  38. Pfeiffer PN, Brandfon S, Garcia E, Duffy S, Ganoczy D, Kim HM, & Valenstein M (2014b). Predictors of suicidal ideation among depressed veterans and the interpersonal theory of suicide. Journal of Affective Disorders, 152, 277–281. doi: 10.1016/j.jad.2013.09.025 [DOI] [PubMed] [Google Scholar]
  39. Preacher KJ, & Hayes AF (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior Research Methods, Instruments, & Computers, 36(4), 717–731. doi: 10.3758/BF03206553 [DOI] [PubMed] [Google Scholar]
  40. Preacher KJ, & Hayes AF (2008). Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behavior Research Methods, 40(3), 879–891. doi: 10.3758/BRM.40.3.879 [DOI] [PubMed] [Google Scholar]
  41. Ribeiro JD, Witte TK, Van Orden KA, Selby EA, Gordon KH, Bender TW, & Joiner TE Jr (2014). Fearlessness about death: The psychometric properties and construct validity of the revision to the Acquired Capability for Suicide Scale. Psychological Assessment, 26(1), 115–126. doi: 10.1037/a0034858 [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Richards D, & Richardson T (2012). Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clinical Psychology Review, 32(4), 329–342. doi: 10.1016/j.cpr.2012.02.004 [DOI] [PubMed] [Google Scholar]
  43. Rudd MD, Bryan CJ, Wertenberger EG, Peterson AL, Young-McCaughan S, Mintz J, . . . Wilkinson E(2015). Brief cognitive-behavioral therapy effects on post-treatment suicide attempts in a military sample: results of a randomized clinical trial with 2-year follow-up. American Journal of Psychiatry, 172(5), 441–449. doi: 10.1176/appi.ajp.2014.14070843 [DOI] [PubMed] [Google Scholar]
  44. Schell TL, & Marshall GN (2008). Survey of individuals previously deployed for OEF/OIF. Invisible wounds of war: Psychological and cognitive injuries, their consequences, and services to assist recovery, 87–115. [Google Scholar]
  45. Schmidt NB, & Joiner TE (2002). Structure of the Anxiety Sensitivity Index psychometrics and factor structure in a community sample. Journal of Anxiety Disorders, 16(1), 33–49. doi: 10.1016/S0887-6185(01)00087-1 [DOI] [PubMed] [Google Scholar]
  46. Schmidt NB, Norr AM, Allan NP, Raines AM, & Capron DW (2017). Randomized clinical trial evaluating the efficacy of an automated intervention targeting anxiety sensitivity cognitive concerns for patients with suicidal ideation. Journal of Consulting and Clinical Psychology, 85(6), 596–610. doi: 10.1037/ccp0000195 [DOI] [PubMed] [Google Scholar]
  47. Schmidt NB, Raines AM, Short NA, & Allan NP (In Preparation). Brief interventions targeting interpersonal risk factors: Perceived burdensomeness mediates reductions in suicidality.
  48. VA. (2016). VA Suicide Prevention Program: Facts About Veteran Suicide, from https://www.va.gov/opa/publications/factsheets/Suicide_Prevention_FactSheet_New_VA_Stats_070616_1400.pdf
  49. Van Orden KA, Cukrowicz KC, Witte TK, & Joiner TE Jr (2012). Thwarted belongingness and perceived burdensomeness: construct validity and psychometric properties of the Interpersonal Needs Questionnaire. Psychological Assessment, 24(1), 197–215. doi: 10.1037/a0025358 [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Van Orden KA, Lynam ME, Hollar D, & Joiner TE (2006). Perceived burdensomeness as an indicator of suicidal symptoms. Cognitive Therapy and Research, 30(4), 457–467. doi: 10.1007/s10608-006-9057-2 [DOI] [Google Scholar]
  51. Van Orden KA, Witte TK, Cukrowicz KC, Braithwaite SR, Selby EA, & Joiner TE Jr, 117(2), 575 (2010). The interpersonal theory of suicide. Psychological Review, 117(2), 575–600. doi: 10.1037/a0018697 [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Van Orden KA, Witte TK, Gordon KH, Bender TW, & Joiner TE Jr (2008). Suicidal desire and the capability for suicide: tests of the interpersonal-psychological theory of suicidal behavior among adults. Journal of Consulting and Clinical Psychology, 76(1), 72–83. doi: 10.1037/0022-006X.76.1.72 [DOI] [PubMed] [Google Scholar]
  53. van Spijker BA, van Straten A, & Kerkhof AJ (2014). Effectiveness of online self-help for suicidal thoughts: results of a randomised controlled trial. PloS one, 9(2), e90118. doi: 10.1371/journal.pone.0090118 [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Watson D, & Clark LA (1994). The PANAS-X: Manual for the Positive and Negative Affect Schedule-Expanded Form. The University of Iowa: Ames. [Google Scholar]

RESOURCES