Skip to main content
VA Author Manuscripts logoLink to VA Author Manuscripts
. Author manuscript; available in PMC: 2025 Apr 1.
Published in final edited form as: J Affect Disord. 2024 Jan 12;350:7–15. doi: 10.1016/j.jad.2024.01.018

Impairment in recognition memory may be associated with near-term risk for suicide attempt in a high-risk sample

Catherine E Myers 1,2, Jill Del Pozzo 3,4, Rokas Perskaudas 3, Chintan V Dave 1,5, Megan S Chesin 6, John G Keilp 7, Anna Kline 8, Alejandro Interian 3,8
PMCID: PMC10922624  NIHMSID: NIHMS1962333  PMID: 38220108

Abstract

Introduction:

Prior work has implicated several neurocognitive domains, including memory, in patients with a history of prior suicide attempt. The current study evaluated whether a delayed recognition test could enhance prospective prediction of near-term suicide outcomes in a sample of patients at high-risk for suicide.

Methods:

132 Veterans at high-risk for suicide completed a computer-based recognition memory test including semantically-related and -unrelated words. Outcomes were coded as actual suicide attempt (ASA), other suicide-related event (OtherSE) such as aborted/interrupted attempt or preparatory behavior, or neither (noSE), within 90 days after testing.

Results:

Reduced performance was a significant predictor of upcoming ASA, but not OtherSE, after controlling for standard clinical variables such as current suicidal ideation and history of prior suicide attempt. However, compared to the noSE reference group, the OtherSE group showed a reduction in the expected benefit of semantic relatedness in recognizing familiar words. A computational model, the drift diffusion model (DDM), to explore latent cognitive processes, revealed the OtherSE group had decreased decisional efficiency for semantically-related compared to semantically-unrelated familiar words.

Limitations:

This study was a secondary analysis of an existing dataset, involving participants in a treatment trial, and requires replication; ~10% of the sample was excluded from analysis due to failure to master the practice tasks and/or apparent noncompliance.

Conclusion:

Impairments in recognition memory may be associated with near-term risk for suicide attempt, and may provide a tool to improve prediction of when at-risk individuals may be transitioning into a period of heightened risk for suicide attempt.

Keywords: Suicidality, suicide attempt, recognition memory, overgeneralization, drift diffusion model

1. Introduction

A key priority in predicting and preventing suicide is the ability to determine indicators of acute suicide risk among individuals considered at high risk for suicide (Rudd, 2008). Nearer-term suicide risk factors are highly understudied: less than five percent of studies in 50 years examine risk windows of less than 6 months (Franklin et al., 2017). Various neurocognitive impairments have been implicated in suicidality (Huber et al., 2019; Jollant et al., 2011; Keilp et al., 2013; O’Connor et al., 2000), particularly the transition from suicidal ideation (SI) to suicide attempt (SA), but much remains to be learned about the specific nature of such impairments and how neurocognitive impairments may contribute to near-term risk for suicide.1

Memory deficits are among the neurocognitive deficits reported in individuals with a history of SI or SA (e.g., Interian et al., 2020; Keilp et al., 2014; Keilp et al., 2013; Keilp et al., 2001; Richard-Devantoy et al., 2015), and brain areas associated with memory show altered connectivity in these patients (for review, Ballard et al., 2019). However, systematic reviews report small or negligible effect sizes in general memory when comparing individuals with a history of SA versus individuals with SI who have never attempted suicide, or comparing individuals with SI to non-suicidal controls (Lalovic et al., 2022; Saffer and Klonsky, 2018). For example, one meta-analysis found no difference between patients with history of SA and patients with SI (but no history of SA) on immediate recall (Richard-Devantoy et al., 2015). Contradictory findings regarding memory’s role in SA and SI may reflect the fact that the construct of memory is comprised of multiple domains, some of which may be more closely linked to suicidality than others. For example, autobiographical memory tends to be less specific and more general in patients with a history of SA (Richard-Devantoy et al., 2015), and also in patients with a history of recent SA who go on to make another SA within 1-month follow-up (Sidley et al., 1997).

Overgeneral autobiographical memory may reflect multiple factors, including a broader memory deficit in retrieving details of the specific context or setting in which information was acquired (Raes et al., 2006; Ramponi et al., 2004). For example, in a laboratory-based recognition memory test (RMT), participants first view a study list (e.g., common English words); after a short delay, participants view a test list that includes familiar (studied) items intermixed with novel items, and are asked whether each appeared on the study list. Since all the words are presumably familiar to adult speakers of English, the task requires memory for whether each specific word was recently experienced in the laboratory context. Participants can be scored for the accuracy with which they correctly endorse or recognize the familiar items (“hit rate”) as well as the accuracy with which they correctly reject the novel items (“correct rejection rate”). In some versions, such as the Related/Unrelated RMT (RU-RMT), the study and test lists include some words that are semantically related (e.g., types of fruit) and others that are unrelated (Interian et al., 2020; Savage et al., 2001). A common finding in RMTs with semantically related words is that participants tend to mistakenly endorse novel words from the same semantic category as study words (Deese, 1959; Roediger and McDermott, 1995); thus, after viewing LEMON, CHERRIES, and GRAPES on the study list, participants are more likely to mistakenly endorse a related novel word (ORANGES) than an unrelated novel word (VASE). One explanation for such so-called “false recognition” effects is that subjects recall the “gist” (category information) more accurately than the specific study items. In this way, recognition test performance may tap into some of the same issues that lead to over-general autobiographical memory.

One prior study of the RU-RMT in a group of Veterans recruited following an index suicide episode found that those with a recent (prior week) suicide attempt had a higher hit rate to related than unrelated words, compared to those with no suicide attempt in the prior year (Interian et al., 2020). An important question is whether such biases in recognition memory might precede, and therefore prospectively predict, upcoming SA.

With this goal in mind, the current study evaluated whether memory recognition performance could help predict an upcoming SA in a sample of high-risk Veterans, who were given the RU-RMT at their enrollment in a clinical treatment trial, and again at several points during follow-up. We coded outcomes in the 90-day window following each RU-RMT testing session, including any actual SAs, as well as other suicide-related behaviors (e.g., preparatory behaviors, suicide-related hospitalization). We hypothesized that poorer recognition memory performance would predict a higher likelihood of suicide attempt.

We also applied a computational model, the drift diffusion model (DDM) (Ratcliff, 1978; Ratcliff and McKoon, 2008; Ratcliff et al., 2016), to determine whether any observed differences in RU-RMT performance could be understood in terms of altered latent cognitive processes. The DDM assumes a decision-making process in which individuals accumulate evidence favoring each of two possible responses (e.g., familiar vs. novel in the RU-RMT), until enough evidence accumulates to trigger one response. These models attempt to fit the entire distribution of reaction times (RT) for correct and incorrect responses, and estimate individual-level latent cognitive parameters such as response bias, response caution, and decisional efficiency, that may help explain observed behavior (Myers et al., 2022). Linkage between DDM parameters and cognitive processes has been validated in a large number of studies, and there is also a growing literature linking DDM parameter estimates to neuropsychological substrates of decision-making (e.g., Gupta et al., 2022; Mueller et al., 2017; Mulder et al., 2012; Weigard and Sripada, 2021), including their potential utility as markers of upcoming risk for suicide attempt in high-risk individuals (Myers et al., 2023). Our exploratory hypothesis was that one or more DDM parameters would be predictive of SAs within the follow-up window, potentially elucidating specific cognitive processes altered in at-risk individuals entering a period of heightened risk for suicide attempt.

2. Methods

2.1. Participants and Procedures

This is a secondary analysis of data from 140 Veterans enrolled in a 12-month randomized control trial (RCT) of Mindfulness-Based Cognitive Behavioral Therapy for Suicide (MBCT-S). Full inclusion criteria and results from the RCT were previously published (Interian et al., 2021). In brief, Veterans were recruited from two Veterans Health Administration (VHA) medical centers in the northeastern US. Recruitment followed an index suicide-related episode that included an actual, interrupted, or aborted suicide attempt; preparatory behavior (e.g., obtaining access to method, writing a farewell note); or suicidal ideation resulting in acute hospitalization and/or engagement with VHA suicide prevention services.

For inclusion in the RCT, participants were required to have both (1) past-month suicidal ideation that resulted in suicide behavior or hospitalization, and (2) past-year actual, aborted, or interrupted suicide attempt (Posner et al., 2014) or placement on the VHA high-risk for suicide list. An actual suicide attempt was defined as self-directed violence carried out with suicidal intent and involving injury or potential for injury. Exclusions were made based on cognitive impairment that impacted informed consent, severe psychotic symptoms, disorganized or disruptive behavior, medical instability, and receipt of ≥2 mindfulness sessions during the previous year. As part of the RCT, all participants had access to a full range of standard mental health treatments within VHA for patients at high risk for suicide, which can include suicide safety planning, clinical monitoring, and attempts to engage in regular mental care (Katz, 2012; Stanley and Brown, 2012). Participants in the treatment condition also received MBCT-S.

At baseline (session 1), participants completed a clinical interview and self-report questionnaires. Prior suicide behavior and suicidal ideation severity were assessed using the Columbia Suicide Severity Rating Scale (C-SSRS; Posner et al., 2011), using published classification criteria (Interian et al., 2018). Past week suicidal ideation severity was assessed using the Beck Scale for Suicidal Ideation (SSI; Beck et al., 1979). Participants were followed for one year, including follow-up testing approximately 3 months (session 2) and 6 months (session 3) after baseline. Baseline and follow-up sessions included updated C-SSRS and SSI.

At each session, participants were also invited to participate in several computer-based tests of neurocognitive processes, including the RU-RMT described below; results from the other computer-based tests have been described elsewhere (Chesin et al., 2021; Interian et al., 2020; Myers et al., 2023). Four participants in the RCT declined to participate in the neurocognitive testing, and an additional two agreed, but RU-RMT data were not collected due to limitations of the testing session. There were no obvious demographic and clinical differences between these four individuals and those who did agree to participate in the neurocognitive testing. Additionally, as noted below, data from two participants were censored due to withdrawal or death from natural causes within 90d after enrollment. This left a sample of 132 unique participants who completed at least one session of RU-RMT testing, including 16 females (12.1%); mean age at baseline (session 1) was 46.2 years (SD 13.8). Table 1 summarizes key demographic and clinical variables for this sample. Twenty-five individuals (18.9%) had 1+ actual suicide attempt (ASA) during the one-year follow-up period post RCT-enrollment, and an additional 34 individuals (25.8%) had no ASA but 1+ other suicide-related event (OtherSE) such as aborted/interrupted attempt or preparatory behavior.

Table 1.

Sample characteristics, assessed at T1.

N %
Race/Ethnicity
White/Caucasian 58 43.9
Black/African American 38 28.8
Latino or Hispanic 29 22.0
Other/Mixed 7 5.3
Current Marital Status
Separated/divorced 58 44.3
Married/living as married 36 27.5
Never married 32 24.4
Widowed 5 3.8
N/A 1 -
Education
High school (or GED) or less 44 33.3
Some college 62 47.0
4-year college degree or higher 26 19.7
Employment Status
Employed 33 25.2
Unemployed 94 71.8
Compensated work therapy program 1 0.8
Full-time student 3 2.3
N/A 1 -
Lifetime history of traumatic brain injury (TBI)
No history of TBI 44 36.7
One or more TBI 76 63.3
N/A 12 -
Psychiatric diagnoses (lifetime history)
Major depressive disorder 93 70.5
Bipolar disorder 17 12.9
Generalized anxiety disorder 10 7.6
Social phobia 5 3.8
Obsessive-compulsive disorder (OCD) 4 3.0
Psychosis 6 4.5
Post-traumatic stress disorder (PTSD) 80 60.6
Substance use/abuse (excluding alcohol), past 30 days 58 43.9
Alcohol binge drinking, past 30 days 51 38.6

Note: “Other” race/ethnicity category includes Asian, Native American, Native Hawaiian. N/A=not available (or did not disclose). Percentages were calculated based on participants with non-missing data; may not sum to 100% due to rounding error. Psychiatric diagnoses are not mutually exclusive.

The study was approved by the medical center’s local Institutional Review Board (IRB; Protocol #1577256), and was performed in accordance with the principles stated in the Declaration of Helsinki and Federal guidelines for research on humans.

2.2. Related-Unrelated Recognition Memory Test (RU-RMT)

The RU-RMT (Interian et al., 2020) was adapted from Savage et al. (2001), and is a memory test in which subjects view a list of study words and, after a short delay, perform a yes-no recognition test involving both familiar (previously studied) words and novel words.

During the study phase (Figure 1A), words appear one-at-a-time on the screen for 1 sec, with an intertrial interval (ITI) of 1200 msec during which the screen is blank; each word is preceded by a 200 msec fixation cross. This is followed by a recognition phase (Figure 1B), in which words appear one-at-a-time and the subject presses the “V” key to indicate “Yes” if the word previously appeared in the study list, or the “N” key to indicate “No” if it did not. There is no time limit for responding; after the subject responds, there is a 500 msec ITI (blank screen) before the next word appears.

Figure 1.

Figure 1.

Examples of Screen Appearance During a Practice Trial. (A) Study phase, (B) recognition phase of the Related/Unrelated Recognition Memory Test (RU-RMT).

Subjects first complete a short practice phase in which they view six unrelated words and then are given a recognition test including the same six words and six novel words (word lists and other stimulus details appear in Supplemental Material). The task then proceeds to the study phase, which includes 40 words in four blocks of 10 words. The first and third blocks contain related words from two semantic categories: in block 1, five fruit (e.g., LEMON, GRAPES) and five musical instruments (e.g., PIANO, BANJO); in block 3, five items of clothing (e.g., VEST, SWEATER) and five office implements (e.g., PENCIL, RULER). The second and fourth block contain unrelated words (e.g., CLOWN, ATTIC). There is a delay of 10-15 minutes during which subjects performed other, unrelated tests, followed by the recognition phase which includes 120 words: the 40 study words, 20 novel related words (5 from each study category), and 60 novel unrelated words.

For each recognition trial, the stimulus word, category (familiar or novel; related or unrelated), subject response, and reaction time are recorded. For each data file, the overall percent accuracy is scored. To examine effects of semantic relatedness, and following Interian et al. (2020), we also calculated difference scores: a d-score for hits dhits, defined as percent correct for semantically-related familiar words minus percent correct for semantically-unrelated familiar words, and drejects, defined as percent correct for semantically-related novel words minus percent correct for semantically-unrelated novel words. Note that it would typically be expected that semantic relatedness might improve hit rate (leading to dhits>0) while also increasing false alarms (leading to drejects<0).

Of the 306 RU-RMT data files obtained from 132 participants, n=7 were excluded from analysis because the participant failed to reach criterion of at least 60% on the immediate recall test during Practice phase (n=7). Additionally, we excluded from analysis any data files with either <10% “familiar” responses overall (n=5) or <50% correct on the second half of the test (n=19), suggesting non-compliance and/or loss of motivation or attention as the task progressed. Finally, on the remaining set of 275 RU-RMT datafiles, data cleansing was performed on each data file to drop trials with RT <250ms as representing anticipatory responding, or RT > 10s as likely reflecting distraction; 95% of files had 2 or fewer trials dropped. All data files retained for analysis had at least 100 trials retained for analysis.

2.3. Outcome Classification

Suicide-related events that occurred during the 90 days after each testing session were determined from clinician-administered C-SSRS at each available testing session augmented by medical chart review to capture SI-related hospital admissions. Following Myers et al. (2023), each RU-RMT data file was assigned to one of three mutually exclusive outcome categories: (1) “ASA” if 1+ actual suicide attempt(s) occurred during the 90 days subsequent to the RU-RMT testing session; (2) “OtherSE” if there was no ASA during this time window but at least one other suicide-related event including interrupted/aborted suicide attempt, preparatory behavior, or suicide-related hospital admission (e.g., acute psychiatry admission related to suicidal ideation); or (3) “noSE” if neither SA nor OtherSE occurred within the 90-day follow-up window.

Of the 275 RU-RMT datafiles available for analysis, four could not be associated with a 90-day outcome due to censoring (including death from natural causes n=1; withdrawal from study n=1; study end within <90 days, n=2).

This resulted in a final set of 271 RU-RMT data files being retained for analysis, including 15 data files classed as ASA, from 13 unique individuals (two individuals had multiple ASAs). Another 29 data files from 27 unique individuals were classed as OtherSE; of these, 11 involved SI-related hospital admissions without suicide-related behavior; 5 involved interrupted/aborted attempts, and the remaining 13 involved preparatory behavior. The remaining 227 data files were classed as noSE within the 90-day follow-up window. Table 2 shows demographic and clinical information associated with each of the 271 RU-RMT files, by outcome group; note that participants contributing more than one data file are represented more than once in this table.

Table 2.

Key clinical and demographic variables, by outcome group, for the 271 RU-RMT data files.

noSE (n=227) OtherSE (n=29) ASA (n=15)
Frequency and %, within each outcome group N % N % N %
Randomized clinical trial
Treatment-as-usual (TAU) 106 46.7 13 44.8 12 80.0
MBCT plus TAU 121 53.3 16 55.2 3 20.0
Gender
Female 26 11.5 7 24.1 2 13.3
Education
High school (or GED) or less 70 30.8 9 31.0 6 40.0
Some college 107 47.1 14 48.3 7 46.7
4-year degree or higher 50 22.0 6 20.7 2 13.3
Clinical Diagnosis (lifetime)
Traumatic brain injury (TBI)* 135 (of 227) 64.3 13 (of 24) 54.2 8 (of 13) 61.5
Post-traumatic stress disorder (PTSD) 133 58.6 18 62.1 10 66.7
Bipolar disorder 25 11.0 4 13.8 1 6.7
Major depressive disorder 158 69.6 25 86.2 13 86.7
Mean and SD, within each outcome group Mean SD Mean SD Mean SD
Age (in years) 48.0 13.6 45.0 14.3 45.3 14.1
Lifetime prior ASAs 2.0 2.4 2.4 2.5 3.5 3.9
Beck Scale for Suicidal Ideation (SSI)* 6.8 (of 225) 9.2 10.0 8.5 15.9 9.8
Beck Depression Inventory (BDI)* 21.0 (of 222) 13.7 32.3 (of 28) 13.0 32.5 (of 14) 17.1

Note: Each of the 132 participants completed 1-3 RU-RMT testing sessions, resulting in multiple data files per participant, each of which was then classified according to outcome within the 90-days following that testing session. All scores are values at T1/baseline, except for SSI and BDI scores which were collected contemporaneous to each RU-RMT testing session.

*

Information unavailable for some cases; statistics calculated based on available data within each outcome group. ASA=actual suicide attempt; OtherSE=other suicide-related event excluding ASA; noSE=no ASA or OtherSE documented within follow-up period; RCT=Randomized clinical trial; MBCT-S=Mindfulness-based Cognitive Therapy adapted for Suicide.

2.4. Statistical Analysis

Generalized estimating equations (GEE) were used to test the effect of predictor variables on the multinomial outcome variable. GEEs are an extension of generalized linear models for analysis of repeated measures with non-normal response variables (Liang and Zeger, 1986; Shiffman, 2014; Zeger et al., 1988), and can model longitudinal designs while accounting for correlations between observations within individuals (Ballinger, 2004; Ma et al., 2012; Schober and Vetter, 2018). The outcome variable had three levels (noSE, OtherSE, ASA). Since each subject contributed data from up to 3 testing sessions, within-subject clustering was used and session modeled as a categorical factor. Effects were reported as odds ratios (OR) with 95% confidence interval (CI); threshold for significance was set at .05.

A GEE was used to evaluate whether overall percent accuracy added significantly to prediction of the outcome, over standard suicide risk variables (number of lifetime ASAs, SI at time of testing) as well as other covariates such as testing session number, age, gender, and receipt of study treatment during RCT. To explore the possible effects of semantic relatedness, the same methods were used: entering dhits and drejects into a GEE that also included the other risk variables and covariates.

GEE models were estimated using SAS Enterprise Guide (version 7.18, SAS Institute Inc., Cary, NC). Other data processing and analyses were conducted using R v. 4.1.0 (R Core Team, 2021). Graphs were produced using the ggplot package for R (Wickham, 2016), and tables using the gt package version 0.4.0 for R (Iannone et al., 2022). R script for analysis/simulation and SAS script for GEEs can be accessed at Open Science Framework (OSF): https://osf.io/3py8z/?view_only=13f9fc4dd08d44eea524ad18187767bf.

2.5. Drift Diffusion Model (DDM)

Full details of model-fitting and validation appear in the Supplemental Material. In brief, the DDM was fit to each RU-RMT data file individually, using Bayesian model fitting procedures in the Dynamic Models of Choice (DMC) package v. 190819 (Heathcote et al., 2019) and base R functions (R Core Team, 2021). We explored seven free parameters: boundary separation a (an indicator of response caution), relative starting point z (an indicator of bias favoring one possible response over another), non-decision time Ter (the portion of the total RT required to encode the stimulus and generate the response, in seconds), and four drift rates d representing decisional efficiency (speed of the evidence accumulation process) for semantically-related vs. semantically-unrelated familiar and novel words. For each data file, the medians of the posterior distributions for each parameter were used as point estimates. For consistency with behavioral results, the drift rates were converted to d-scores for drifthits, defined as drift rate for semantically-related familiar words minus drift rate for semantically-unrelated familiar words, and driftrejects, defined as drift rate for semantically-related novel words minus drift rate for semantically-unrelated novel words. These scores were subjected to the same group-level analyses (GEEs) as used for the behavioral data.

3. Results

3.1. Behavioral Data

Figure 2A shows RU-RMT overall percent accuracy, which was numerically lower in the ASA group than the noSE group. In the GEE model (Supplemental Table S3), lower RU-RMT accuracy (OR=.94 [95% CI=0.88-0.99], p=.027) and higher SI severity (OR=1.09 [95% CI=1.03-1.16], p=.004) were both predictive of ASA. Thus, every unit decrease in overall percent accuracy corresponded with a 6% increase of ASA risk while every unit increase in SSI corresponded with a 9% increase in ASA risk. None of the other predictors (session, age, gender, RCT treatment group, or number of prior ASAs) were significant of upcoming ASA (all p>.190). There were no significant predictors of the OtherSE outcome (all p>0.070).

Figure 2.

Figure 2.

Performance on RU-RMT Test, by outcome group. (A) Overall percent accuracy. (B) d-score for hits, defined as difference In percent accuracy for familiar semantically-related minus percent-accuracy for familiar semantically-unrelated words; d>0 Indicates expected benefit of semantic-relatedness in correctly recognizing familiar words. (C) d-score for correct rejects, defined as difference in percent accuracy for novel semantically-related minus percent accuracy for novel semantically-unrelated words; d<0 Indicates expected tendency to “falsely recognize” semantically-related novel words. ASA=actual suicide attempt within 90 days following RU-RMT testing; OtherSE=other suicide-related event (excluding ASA) within 90 days following RU-RMT testing; noSE=no suicide-related events within 90 days following RU-RMT testing. Columns show mean; error bars show standard errors.

A next question is whether the ASA group’s impairment was differentially affected by semantic relatedness or unrelatedness of the words. It would typically be expected to observe better performance (higher hit rate) on familiar words that are semantically related, compared to familiar words that are semantically unrelated; this is indeed reflected in dhits>0, as shown by all groups in Figure 2B. On the other hand, semantic relatedness would be expected to lead to more false alarms on novel words that are semantically related, compared to semantically unrelated novel words; this was also reflected in drejects<0, as shown by all groups in Figure 2C.

However, compared to the noSE reference group, the OtherSE group showed reduced dhits (OR=0.97 [95% CI=0.94-1.00], p=.033), while the ASA group did not (OR=1.00 [95% CI=0.96-1.05], p=0.857). This suggested that OtherSE experienced less of a benefit of semantic relatedness in recognizing familiar words. Neither the ASA nor OtherSE group differed significantly from the noSE group on drejects (all p>.500), although as expected SSI was associated with increased risk of ASA while participation in RCT MBCT-S treatment group was associated with reduced risk of ASA (see Supplemental Table S4).

3.3. Drift Diffusion Model

Figure 3 shows average of parameter estimates obtained from the DDM, for each outcome class. (Detailed results from DDM model-fitting and validation appear in Supplemental Material.) As expected, drifthits was generally positive, indicating more efficient evidence accumulation for familiar semantically-related than unrelated words, while driftrejects was generally negative, indicating less efficient evidence accumulation for novel semantically-related than unrelated words. Only drifthits was predictive of OtherSE (OR=0.42 [95% CI=0.23-0.77], p=0.005), such that every unit increase in drifthits was associated with 58% decrease in risk of OtherSE; no other DDM-related variables were predictive for the OtherSE group or ASA group, although again SSI was associated with increased risk of SSI while and participation in RCT MBCT-S treatment group were associated with reduced risk of ASA (see Supplemental Table S5).

Figure 3.

Figure 3.

Posterior Parameter Estimates from the Drift Diffusion Model (DDM7), for each outcome group. (A) Boundary separation (a), a measure of response caution; (B) relative starting point (z), a measure of response bias (shown as percent of a); (C) non-decision time (Ter), the part of the reaction time corresponding to stimulus encoding and motor execution; and (D-E) d-score of drift rates (d) for familiar words (related vs. unrelated) and novel words (related vs. unrelated); in each case, d>0 indicates larger drift rate (greater decisional efficiency) for semantically-related words. All parameters in arbitrary units except z (ranges 0..100 with 50 indicating no bias) and Ter (in seconds). Columns show mean; error bars show standard errors; other abbreviations as in Figure 2.

4. Discussion

Memory deficits have long been recognized as part of the clinical profile of suicidality (Richard-Devantoy et al., 2015). In a real-world, clinical sample of patients at high risk of suicide, the current study shows that memory deficits were predictive of suicide attempt. Specifically, overall accuracy on the RU-RMT was reduced in those with an upcoming ASA, but not those with other suicide-related events, such as interrupted/aborted attempts, preparatory behavior, or severe SI resulting in hospital admission, compared to individuals without upcoming suicide-related events.

4.1. Impaired recognition accuracy and upcoming suicide attempt

Impaired recognition accuracy significantly predicted risk of ASA, even after controlling for the predictive value of other relevant variables such as past-week SI and prior history of SA. The deficiency did not appear to reflect selective difficulty with semantically-related or non-related words. Thus, the ASA group’s performance seemed to reflect a more general recognition memory impairment rather than a specific change in semantic clustering.

A deficit in recognition memory may relate directly to core issues in suicidality, including impaired problem solving. For example, much of problem-solving involves the ability to search one’s memory for past episodes that are similar to the current situation, and to recall the responses and the outcomes that ensued. Thus, episodic memory forms a rich database from which to evaluate possible solutions to the current problem, while avoiding prior responses that were not productive. Reduced ability to recall specific prior episodes would severely hinder one’s ability to use past experience to inform present decision-making. Several prior studies have suggested that individuals who attempt suicide have over-general autobiographical memories, encoding gist at the expense of detail (Pollock and Williams, 2001); others have noted a link between social problem solving and over-general autobiographical memories in depression (Goddard et al., 1996, 1997). In the RU-RMT, it is assumed that all the stimulus words are familiar to the participant, and that the recognition test requires remembering which of the words were previously experienced in the specific context of the study phase. Poor recognition memory performance could therefore reflect insufficient ability to tag study words as having been experienced within the context of the study list, and/or to retrieve that contextual information during the recognition test. These results are consistent with clinical models of suicide that emphasize cognition (e.g., cognitive model of suicide; Wenzel and Beck, 2008) and the suitability of clinical interventions that directly target problem-solving to reduce risk (Gustavson et al., 2016).

To better understand the cognitive processes that might underlie the observed impaired recognition memory in the ASA group, we turned to a computational model, the DDM, to examine latent processes that are not directly evident from accuracy scores alone. However, no DDM variables encoding drift rate, response caution, response bias, or non-decision time were predictive of ASA. The drift rate is sometimes characterized as reflecting “decisional efficiency,” such that higher drift rates mean greater relative accessibility of information in long-term memory (Spaniol et al., 2006; Spaniol et al., 2008). Reduced decisional efficiency is emerging as a neurocognitive risk factor in a range of psychopathologies (for review, see Weigard and Sripada, 2021), while studies of healthy aging suggest that drift rate can change within subjects across the lifespan (Spaniol et al., 2006). Our own prior analysis of data from the current patient sample on a test of impulsivity and response inhibition (the go/no-go task) implicated reduced decisional efficiency in the ASA (but not OtherSE) group (Myers et al., 2023). Together with the current results, this suggests that memory (and response inhibition) might be among the cognitive phenotypes that could result from reduced decisional efficiency, and that might together combine to increase risk for suicidality.

4.2. Semantic relatedness and upcoming suicide-related events

In contrast to the ASA group, the OtherSE group did not show significantly reduced overall accuracy on the RU-RMT; however, they did show a reduction of the expected effects of semantic relatedness. While the OtherSE group did show the expected pattern of dhits>0 and drejects<0 (Figure 2B,C), reduced dhits was significantly predictive of OtherSE outcome, compared to the noSE reference group. When examined with the DDM, the OtherSE outcome was associated with reduced drifthits, which would be consistent with reduced efficiency in retrieving and acting upon those memories.

The ability to recall specific information, including contextual information, may form an important part of the cognitive profile associated with suicidality, and tests of this ability could provide a useful adjunct in attempting to identify individuals entering a period of risk for suicide-related events. If this were the case, it might be possible to observe increasing severity of recognition deficits, and of reductions in decisional efficiency, as an individual moves closer in time to a suicide attempt. Additional studies with more frequent testing intervals might be able to answer this question.

4.3. Study limitations

The current study had several important limitations. First, the study represents a secondary analysis of an existing dataset; as with any results obtained from secondary analyses, it will be important to replicate the findings in a new sample.

Additionally, the participants were all Veterans enrolled in a clinical trial, which may have modified their behavior across time. Indeed, participants in the MBCT-S treatment group had fewer ASAs during the RCT’s one-year follow-up than those in the control group (Interian et al., 2021). It would not be surprising if successful MBCT-S also modified participants’ cognition in a way that could affect behavior on the RU-RMT. In fact, prior studies have suggested that MBCT-S can modify attention, which (among other effects) can affect memory encoding (Chesin et al., 2016; Chesin et al., 2021). It would therefore be important to replicate the current results in a sample not explicitly enrolled in an RCT. Nevertheless, current results remained significant even after adjusting for treatment group effects, and time (testing session) in the multivariate GEE models. In fact, any study enrolling at-risk participants is likely to be complicated by the ethical necessity for interventions related to suicide prevention.

Other limitations of the current sample include the underrepresentation of females, which likely limited power to detect possible gender differences, and lack of information on psychotropic medication that could have influenced behavior on the RU-RMT. In fact, gender and medication status are part of a long list of demographic and clinical variables that could influence risk. Suicide is transdiagnostic, occurring across a range of mental health disorders; the current sample reflects that epidemiological reality, and suggests that robust neurocognitive effects may still be detected in those who will soon attempt suicide.

Another important concern of the current study is the number of RU-RMT data files that were excluded due to failure to master the practice phase, or apparent non-compliance or loss of motivation during the recognition phase. We should note that the general pattern of results was consistent when we considered stricter or laxer criteria, though at the expense of reduced sample size and power on the one hand, or floor effects on the other (results not shown). Still, it remains a concern that about 10% of the sample was not included in final analyses; this could reflect reduced cognitive ability in our sample, due to either demographic characteristics (e.g., low functional literacy), or clinical characteristics (e.g., suicidal mood, medication affecting attention and arousal, or other psychiatric conditions). For this reason, it would be valuable for further studies to consider simpler versions of the task. In particular, since overall accuracy emerged as predictive for ASA, it may be worth evaluating simpler word recognition tests, possibly with fewer words, shorter delay between exposure and recall, and/or less semantic relatedness as a possible source of interference.

Another key limitation is the low incidence of outcome events (particularly ASAs): Only about 5.5% of testing sessions were followed by an ASA within 90 days, although nearly 19% of participants experienced ASAs within the full one-year follow-up period post-enrollment in the original RCT. This follow-up attempt rate is comparable to that observed in similar intervention studies utilizing a one-year observation period (e.g., Miller et al., 2017). Still, the low frequency of attempts, even among a high-risk sample presents challenges for both clinical prediction and research studies.

Finally, a small number of participants in the RCT (4 of 140) declined to participate in the neurocognitive testing; while there were no obvious demographic or clinical variables differentiating this subset, it is possible that their cognitive performance and/or risk profile may differ in some important way from those who did agree to participate.

Despite these limitations, this study suggests the utility of using brief cognitive tests to determine acute suicide risk, and may thus contribute to assessment practices. Current prediction models, often relying on clinical decisions making using patient-reported data, are lacking in validity. More tests of the utility and additive predictive utility of cognitive tests in suicide risk assessment are needed to confirm these findings. If successful, such tests could be incorporated in clinical screening for suicide risk, and for tailoring interventions by identifying individuals who may be candidates for therapy to improve memory and problem-solving skills, to help them be better equipped to evaluate past experiences, identify effective solutions, and make informed decisions during times of crisis.

5. Conclusions

In summary, the current study suggested that a delayed recognition task is predictive of upcoming suicide attempt within the next 90 days, in a clinical sample of patients at high risk for suicide. While reductions in overall memory were associated with increased risk of ASA, reduced benefit of semantic relatedness in recognizing previously-presented words was associated with increased risk of other SE (excluding ASA). The ability to recall specific information, including contextual information, may form an important part of the cognitive profile associated with suicide attempt, and tests of this ability could provide a useful adjunct in attempting to identify individuals at heightened risk for upcoming suicide attempt, so that appropriate clinical resources can be targeted to those individuals at the time they are needed most.

Supplementary Material

1

Highlights.

  • Memory deficits form part of the clinical profile of suicidality.

  • Cognitive changes may prospectively predict suicide attempt in at-risk individuals.

  • Poor recognition memory predicted suicide attempt (SA) within next 90 days.

  • The prediction remained significant after controlling for other clinical variables.

  • Computational model suggested latent cognitive processes in those with upcoming SA.

Role of the Funding Source

This work was supported by the U.S. Department of Veterans Affairs through Health Services Research and Development Service #IIR 12-134 (IA) and Clinical Sciences Research and Development Service (CSR&D) Merit Award #I01 CX001826 (CEM). The contents do not represent the views of the U.S. Department of Veterans Affairs or the United States Government. The funder played no role in the study design, data collection, data analysis, interpretation of the data, writing of the report, or decision to submit the article for publication.

Footnotes

Declarations of Interest

All authors declare that they have no relevant relationships to disclose.

CRediT authorship contribution statement

Catherine E. Myers: Conceptualization, Methods, Software, Data Analysis, Writing-original draft, Writing-Review & editing, Funding Acquisition. Jill Del Pozzo: Data Analysis, Writing-original draft, Writing-Review & editing. Rokas Persaukas: Writing-Review & editing. Chintan V. Dave: Data Analysis, Writing-Review & editing. Megan S. Chesin: Writing-Review & editing. John G. Keilp: Conceptualization, Methods, Software, Writing-Review & editing. Anna Kline: Conceptualization, Methods, Funding Acquisition, Project Administration, Writing-Review & editing. Alejandro Interian: Conceptualization, Methods, Data Analysis, Writing-original draft, Writing-Review & editing, Funding Acquisition, Project Administration.

1

Abbreviations: SA=suicide attempt, SI=suicidal ideation, SE=suicide-related event, RCT=randomized clinical trial, RMT=recognition memory test, RU-RMT=Related/Unrelated Recognition Memory Test, MBCT-S=Mindfulness-based Clinical Therapy for Suicide, DDM=drift diffusion model, GEE=generalized estimating equations.

References

  1. Ballard ED, Reed JL, Szczepanik J, Evans JW, Yarrington JS, Dickstein DP, Nock MK, Nugent AC, Zarate CA, 2019. Functional imaging of the Implicit Association of the self with life and death. Suicide Life. Threat. Behav 49, 1600–1608. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Ballinger GA, 2004. Using generalized estimating equations for longitudinal data analysis. Organizational Research Methods 7, 127–150. [Google Scholar]
  3. Beck AT, Kovacs M, Weissman A, 1979. Assessment of suicidal intention: the Scale for Suicide Ideation. J. Consult. Clin. Psychol 47, 343–352. [DOI] [PubMed] [Google Scholar]
  4. Chesin MS, Benjamin-Phillips CA, Keilp J, Fertuck EA, Brodsky BS, Stanley B, 2016. Improvements in executive attention, rumination, cognitive reactivity, and mindfulness among high-suicide risk patients participating in adjunct Mindfulness-Based Cognitive Therapy: Preliminary findings. J. Altern. Complement. Med 22, 642–649. [DOI] [PubMed] [Google Scholar]
  5. Chesin MS, Keilp JG, Kline A, Stanley B, Myers C, Latorre M,St. Hill LM, Miller RB, King AR, Boschulte DR, Rodriguez KM, Callahan M, Sedita M, Interian A, 2021. Attentional control may be modifiable with Mindfulness-Based Cognitive Therapy to Prevent Suicide. Behaviour Research and Therapy 147, 103988. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Deese J, 1959. On the prediction of occurrence of particular verbal intrusions in immediate recall. J. Exp. Psychol 58, 17–22. [DOI] [PubMed] [Google Scholar]
  7. Franklin JC, Ribeiro JD, Fox KR, Bentley KH, Kleiman EM, Huang X, Musacchio KM,Jaroszewski AC, Chang BP, Nock MK, 2017. Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research. Psychol. Bull 143, 187–232. [DOI] [PubMed] [Google Scholar]
  8. Goddard L, Dritschel B, Burton A, 1996. Role of autobiographical memory in social problem-solving and depression. J. Abnorm. Psychol 105, 609–616. [DOI] [PubMed] [Google Scholar]
  9. Goddard L, Dritschel B, Burton A, 1997. Social problem-solving and autobiographical memory in non-clinical depression. Br. J. Clin. Psychol 36, 449–451. [DOI] [PubMed] [Google Scholar]
  10. Gupta A, Bansal R, Alashwal H, Kacar AS, Balci F, Moustafa AA, 2022. Neural substrates of the drift-diffusion model in brain disorders. Frontiers in Computational Neuroscience 15, 678232. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Gustavson KA, Alexopoulos GS, Niu GC, McCulloch C, Meade T, Areán PA, 2016. Problem-Solving Therapy reduces suicidal ideation in depressed older adults with executive dysfunction. Am. J. Geriatr. Psychiatry 24, 11–17. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Heathcote A, Lin Y-S, Reynolds A, Strickland L, Gretton M, Matzke D, 2019. Dynamic models of choice. Behavior Research Methods 51, 961–985. [DOI] [PubMed] [Google Scholar]
  13. Huber RS, Hodgson R, Yurgelun-Todd DA, 2019. A qualitative systematic review of suicide behavior using the cognitive systems domain of the research domain criteria (RDoC) framework. Psychiatry Res 282, 112589. [DOI] [PubMed] [Google Scholar]
  14. Iannone R, Cheng J, Schloerke B, 2022. gt: Easily create presentation-ready display tables, R package version 0.4.0 [Google Scholar]
  15. Interian A, Chesin M, Kline A, Miller R, Hill L, Latorre M, Shcherbakov A, King A, Stanley B, 2018. Use of the Columbia-Suicide Severity Rating Scale (C-SSRS) to classify suicidal behaviors. Archives of Suicide Research 22, 278–294. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Interian A, Chesin MS, Stanley B, Latorre M, St. Hill LM, Miller RB, King A, Boschulte DR, Rodriguez KM, Kline A, 2021. Mindfulness-based cognitive therapy for preventing suicide in military veterans: A randomized clinical trial. Journal of Clinical Psychiatry 82, 20m13791. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Interian A, Myers CE, Chesin MS, Kline A, Hill LS, King AR, Miller R, Latorre M, Gara MA, Stanley BH, Keilp JG, 2020. Towards the objective assessment of suicidal states: Some neurocognitive deficits may be temporally related to suicide attempt. Psychiatry Res. 287, 112624. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Jollant F, Lawrence NL, Olié E, Guillaume S, Courtet P, 2011. The suicidal mind and brain: A review of neuropsychological and neuroimaging studies. World Journal of Biological Psychiatry 12, 319–339. [DOI] [PubMed] [Google Scholar]
  19. Katz I, 2012. Lessons learned from mental health enhancement and suicide prevention activities in the Veterans Health Administration. Am. J. Public Health 102, S14–S16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Keilp JG, Beers SR, Burke AK, Melhem NM, Oquendo MA, Brent DA, Mann JJ, 2014. Neuropsychological deficits in past suicide attempters with varying levels of depression severity. Psychological Medicine 44, 2965–2974. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Keilp JG, Gorlyn M, Russell M, Oquendo MA, Burke AK, Harkavy-Friedman J, Mann JJ, 2013. Neuropsychological function and suicidal behavior: Attention control, memory and executive dysfunction in suicide attempt. Psychological Medicine 43, 539–551. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Keilp JG, Sackeim HA, Brodsky BS, Oquendo MA, Malone KM, Mann JJ, 2001.Neuropsychological dysfunction in depressed suicide attempters. American Journal of Psychiatry 158, 735–741. [DOI] [PubMed] [Google Scholar]
  23. Lalovic A, Wang S, Keilp JG, Bowie CR, Kennedy SH, Rizvi SJ, 2022. A qualitative systematic review of neurocognition in suicide ideators and attempters: Implications for cognitive-based psychotherapeutic interventions. Neurosci. Biobehav. Rev 132, 92–109. [DOI] [PubMed] [Google Scholar]
  24. Liang K-Y, Zeger SL, 1986. Longitudinal analysis using generalized linear models. Biometrika 73, 13–22. [Google Scholar]
  25. Ma Y, Mazumdar M, Memtsoudis SG, 2012. Beyond repeated measures ANOVA: Advanced statistical methods for the analysis of longitudinal data in anesthesia research. Reg. Anesth. Pain Med 37, 99–105. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Miller IW, Camargo CA, Arias SA, Sullivan AF, Michael H. Allen , Goldstein AB, Manton Anne P., Espinola JA, Jones R, Hasegawa K, Boudreaux ED, 2017. Suicide prevention in an emergency department population: The ED-SAFE study. JAMA Psychiatry 74, 563–570. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Mueller CJ, White CN, Kuchinke L, 2017. Electrophysiological correlates of the drift diffusion model in visual word recognition. Human Brain Mapping 38, 5616–5627. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Mulder MJ, Wagenmakers EJ, Ratcliff R, Boekel W, Forstmann BU, 2012. Bias in the brain: A diffusion model analysis of prior probability and potential payoff. Journal of Neuroscience 32, 2335–2343. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Myers CE, Dave CV, Callahan M, Chesin MS, Keilp JG, Beck KD, Brenner LA, Goodman MS, Hazlett EA, Niculescu AB, St. Hill L, Kline A, Stanley BH, Interian A, 2023. Improving the prospective prediction of a near-term suicide attempt in Veterans at risk for suicide, using a Go/No-Go task. Psychological Medicine 53, 4245–4254. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Myers CE, Interian A, Moustafa AA, 2022. A practical introduction to using the drift diffusion model of decision-making in cognitive psychology, neuroscience, and health sciences. Frontiers in Psychology 13, 1039172. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. O’Connor T, Rutter M, Beckett C, Keaveney L, Kreppner J, and the English and Romanian Adoptees Study Team, 2000. The effects of global severe privation on cognitive competence: Extension and longitudinal follow-up. Child Development 71, 376–390. [DOI] [PubMed] [Google Scholar]
  32. Pollock LR, Williams JM, 2001. Effective problem solving in suicide attempters depends on specific autobiographical recall. Suicide Life. Threat. Behav 31, 386–396. [DOI] [PubMed] [Google Scholar]
  33. Posner K, Brodsky B, Yershova KV, Buchanan J, Mann J, 2014. The classification of suicide behavior, In: Nock MK (Ed.), The Oxford handbook of suicide and self-injury. Oxford University Press, Oxford, UK, pp. 7–22. [Google Scholar]
  34. Posner K, Brown G, Stanley B, Brent DA, Yershova KV, Oquendo MA, Currier GW, Melvin GA, Greenhill L, Shen S, Mann JJ, 2011. The Columbia-Suicide Severity Rating Scale (C-SSRS): Initial validity and internal consistency findings from three multi-site studies with adolescents and adults. American Journal of Psychiatry 168, 1266–1277. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. R Core Team, 2021. R: A language and environment for statistical computing. Vienna, Austria. [Google Scholar]
  36. Raes F, Hermans D, Williams JMG, Demyttenaere K, Sabbe B, Pieters G, Eelen P, 2006. Is overgeneral autobiographical memory an isolated memory phenomenon in major depression? Memory 14, 584–594. [DOI] [PubMed] [Google Scholar]
  37. Ramponi C, Barnard PJ, Nimmo-Smith I, 2004. Recollection deficits in dysphoric mood: An effect of schematic models and executive mode? Memory 12, 655–670. [DOI] [PubMed] [Google Scholar]
  38. Ratcliff R, 1978. A theory of memory retrieval. Psychol. Rev 85, 59–108. [Google Scholar]
  39. Ratcliff R, McKoon G, 2008. The diffusion decision model: Theory and data for two-choice decision tasks. Neural Computation 20, 873–922. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Ratcliff R, Smith PL, Brown SD, McKoon G, 2016. Diffusion decision model: Current issues and history. Trends in Cognitive Sciences 20, 260–281. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Richard-Devantoy S, Berlim MT, Jollant F, 2015. Suicidal behavior and memory: A systematic review and meta-analysis. World Journal of Biological Psychiatry 16, 544–566. [DOI] [PubMed] [Google Scholar]
  42. Roediger H, McDermott K, 1995. Creating false memories: Remembering words not presented in lists. Journal of Experimental Psychology: Learning, Memory and Cognition 21, 803–814. [Google Scholar]
  43. Rudd MD, 2008. Suicide warning signs in clinical practice. Current Psychiatry Reports 10, 87–90. [DOI] [PubMed] [Google Scholar]
  44. Saffer BY, Klonsky ED, 2018. Do neurocognitive abilities distinguish suicide attempters from suicide ideators? A systematic review of an emerging research area. Clinical Psychology: Science and Practice 25, e12227. [Google Scholar]
  45. Savage CR, Deckersbach T, Heckers S, Wagner AD, Schacter DL, Alpert NM, Fischman AJ, Rauch SL, 2001. Prefrontal regions supporting spontaneous and directed application of verbal learning strategies: Evidence from PET. Brain 124, 219–231. [DOI] [PubMed] [Google Scholar]
  46. Schober P, Vetter TR, 2018. Repeated measures designs and analysis of longitudinal data: If at first you do not succeed — try, try again. Anesth. Analg 127, 569–575. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Shiffman S, 2014. Conceptualizing analyses of ecological momentary assessment data. Nicotine and Tobacco Research 16, S76–S87. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Sidley GL, Whitaker K, Calam RM, Wells A, 1997. The relationship between problem-solving and autobiographical memory in parasuicide patients. Behavioural and Cognitive Psychotherapy 25, 195–202. [Google Scholar]
  49. Spaniol J, Madden D, Voss A, 2006. A diffusion model analysis of adult age differences in episodic and semantic long-term memory retrieval. Journal of Experimental Psychology: Learming, Memory and Cognition 32, 101–117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Spaniol J, Voss A, Grady CL, 2008. Aging and emotional memory: cognitive mechanisms underlying the positivity effect. Psychol. Aging 23, 859–872. [DOI] [PubMed] [Google Scholar]
  51. Stanley B, Brown GK, 2012. Safety planning intervention: A brief intervention to mitigate suicide risk. Cognitive and Behavioral Practice 19, 256–264. [Google Scholar]
  52. Weigard A, Sripada C, 2021. Task-general efficiency of evidence accumulation as a computationally defined neurocognitive trait: Implications for clinical neuroscience. Biological Psychiatry: Global Open Science 1, 5–15. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Wenzel A, Beck AT, 2008. A cognitive model of suicidal behavior: Theory and treatment. Applied and Preventive Psychology 12, 189–201. [Google Scholar]
  54. Wickham H, 2016. ggplot2: Elegant graphics for data analysis Springer-Verlag, New York. [Google Scholar]
  55. Zeger SL, Liang KY, Albert PS, 1988. Models for longitudinal data: A generalized estimating equation approach. Biometrics 44, 1049–1060. [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1

RESOURCES