Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 May 1.
Published in final edited form as: Depress Anxiety. 2019 Feb 2;36(5):392–399. doi: 10.1002/da.22882

Research Domain Criteria scores estimated through natural language processing are associated with risk for suicide and accidental death

Thomas H McCoy Jr 1,2,3,*, Amelia M Pellegrini 1, Roy H Perlis 1,2
PMCID: PMC6488379  NIHMSID: NIHMS1007404  PMID: 30710497

Abstract

Background:

Identification of individuals at increased risk for suicide is an important public health priority, but the extent to which considering clinical phenomenology improves prediction of longer-term outcomes remains understudied. Hospital discharge provides an opportunity to stratify risk using readily available clinical records and details.

Methods:

We applied a validated natural language processing tool to generate estimated Research Domain Criteria scores for a cohort of 444,317 individuals drawn from 815,457 hospital discharges between 2005 and 2013. We used survival analysis to examine the association of this risk with suicide and accidental death, adjusted for sociodemographic features.

Results:

In adjusted models, symptoms in each of the five domains contributed to incremental risk (log rank p<0.001), with greatest increase observed with positive valence. The contribution of each domain to risk was time-dependent.

Conclusions:

Research Domain Criteria symptom scores parsed from clinical documentation are associated with suicide and illustrates that multiple domains contribute to risk in a time-varying fashion.

Keywords: suicide, accidental death, electronic health records, natural language processing, research domain criteria, survival analysis

INTRODUCTION

In 2016, suicide was the tenth most common cause of death, accounting for 44,965 of 2,744,248 reported deaths (Heron, 2018). With rates of suicide continuing to rise in the United States, identification of individuals at risk for suicide is an important public health priority (Rossen, Hedegaard, Khan, & Warner, 2018; Stone et al., 2018). Prior work on risk stratification has been wide ranging including potential biological correlates of suicide (Fanelli & Serretti, 2019; Le-Niculescu et al., 2013; Niculescu et al., 2015), stratification based using longitudinal medical claims data (Barak-Corren et al., 2017), as well as conventional diagnostic and demographic risk factor identification (Gvion & Levi-Belz, 2018; Hawton, Casañas I Comabella, Haw, & Saunders, 2013). Despite extensive work on risk stratification, suicide risk assessment tools are in limited use, and those that are in use are supported by limited evidence (Runeson et al., 2017). Moreover, while numerous individual risk factors for suicide have been identified (Chung et al., 2017; Turecki & Brent, 2015), little of this work has occurred in non-psychiatric populations (Ursano et al., 2018), even though more people see primary care doctors than psychiatrists in the month prior to suicide (Luoma, Martin, & Pearson, 2002).

The NIMH Research Domain Criteria (RDoC) were constructed to relate clinical presentation to neurobiology for use in transdiagnostic studies (Cuthbert, 2014; Insel et al., 2010; Insel & Cuthbert, 2015). The RDoC constructs enable a new approach to suicide risk stratification using features that may relate more closely to neurobiology. Specifically, they capture psychopathology associated with negative valence, positive valence, arousal, cognition, and social functioning. A recent review of suicide risk factors found that they span the NIMH’s RDoC domains and constructs (Glenn et al., 2018), with the preponderance relating to negative valence but effect sizes similar across domains. Unlike many previously identified risk factors (e.g. race, sex, or medical history), which are static and not modifiable (Franklin et al., 2017), RDoC domains are time varying and potentially modifiable, making them particularly attractive for use in risk stratification and identification (Glenn & Nock, 2014). Recognizing this potential, we sought to characterize suicide risk in terms of estimated Research Domain Criteria (eRDoC) domains, using a recently-validated approach based on application of natural language processing approach to electronic health records (McCoy Jr, Yu, et al., 2018). We have previously demonstrated that incorporating information from narrative clinical notes improved prediction of risk of suicide and accidental death in a very large cohort of consecutive hospital discharges (McCoy Jr, Castro, Roberson, Snapper, & Perlis, 2016) from two large general hospitals. The present work extends that work to include specific RDoC-informed phenotypes, rather than narrative documentation in general. Moreover, it seeks to quantify risk by examining such phenotypes for the first time in a general medical-surgical hospital population (McCoy Jr et al., 2015; McCoy Jr, Yu, et al., 2018). In particular, we hypothesized that multiple RDoC domains, and particularly positive and negative valence, would be associated with differential risk of suicide and accidental death in this large general hospital cohort.

MATERIALS AND METHODS

Study design and cohort derivation

This was a retrospective cohort study using electronic health records from two large academic medical centers. We examined all discharges following inpatient admission of adults ages 18–90 between 2005 and 2013 at two large New England hospitals. Data included sociodemographic features (age at admission, sex, race, and insurance type) and narrative hospital discharge notes. These data were extracted from the electronic health record and incorporated into a datamart using Informatics for Integrating Biology and the Bedside, or i2b2, server software (Murphy et al., 2007, 2010) for cohort management and health records data encoding.

Derivation of eRDoC scores

In prior work, we described derivation and initial validation of a tool for extracting estimates of the five RDoC dimensions from narrative text (McCoy Jr et al., 2018). In brief, a set of terms (tokens) were developed by clinical experts in consultation with the NIMH RDoC workgroup, then expanded in a search for semantically equivalent tokens using unsupervised machine learning, and then reduced to a set of terms that actually occur in psychiatric hospital clinical documentation. Then, in order to score each domain, the percent of domain-specific terms appearing in any given note is determined. That is, if 5 of 10 possible terms appear in a note, that note scores 5/10 or 0.5. The net result of this is an estimated domain symptom burden score for a clinical document. The software used for this scoring is freely available as open source for inspection at use (McCoy Jr, Yu, Cai, & Perlis, 2018). Full details on the development and validation of these estimates have been previously published as well as initial connection to underlying human genetics (McCoy Jr, Castro, et al., 2018; McCoy Jr, Yu, et al., 2018). For examples of tokens loading on each domain, see Table 1.

Table 1.

Examples of tokens (terms) loading on each RDoC domain.

RDoC Domain Example Terms
Arousal and Regulatory vigilance, sexual abuse, reactivity, theta, rem sleep
Cognitive impulse, attentive, attention, distracted, forget
Negative depressive disorder, panic attack, nervous, palpitation, worried
Positive drink, intoxication, gamble, with alcohol, heroin
Social misunderstand, inappropriate, threaten, engaged, empathic
Open in a new tab

RDoC = Research Domain Criteria

Outcome

For all individuals in the cohort and identified as deceased on the basis of the Federal social security registry, we queried public death certificate data available from the Massachusetts Department of Public Health in order to determine reported (coded) cause of death in terms of international classification of disease (ICD) codes. Consistent with prior work, recognizing that suicide may not be reliably coded when circumstances of death are indeterminate, the primary outcome of interest was suicide or accidental death, with planned secondary analysis examining suicide alone (McCoy Jr et al., 2016).

Analysis

In primary analysis, we examined time from discharge to suicide or accidental death, censoring at end of follow-up or death, using survival methodology. To facilitate interpretation, for individuals with multiple admissions during the risk period, we randomly selected one admission. Both visual inspection and formal testing of Schoenfeld residuals (Schoenfeld, 1980) indicated that multiple features violated the assumption of proportionality of hazards necessary to apply Cox regression. We therefore used Kaplan-Meier log-rank tests, stratified for key sociodemographic features (age, sex, insurance type), contrasting top and bottom quintile of each eRDoC domain score. Then, we captured these time dependent effects using parametric survival modeling and cross sectional logistic regression. For any Bonferroni correction (i.e., with nominal p<0.01) significant difference in survival between top and bottom eRDoC domain quintiles, we fitted a flexible parametric survival model to allow time varying effects and direct inspection of the hazard function through plotting (Royston & Parmar, 2002). Finally, to allow comparison of relative effects of each domain, we compared the ratio of cumulative events between the top and bottom quintile of eRDoC risk over the follow-up period. For example, if on a given day the top quintile had had 100 events and the bottom quintile had had 50 events the cumulative event rate on that day would be 2 (100/50).

The Partners HealthCare Human Research Committee reviewed and approved the study protocol and as a no-contact retrospective health care utilization study the requirement for informed consent was waived.

RESULTS

The cohort as a whole included 815,457 admissions for 444,317 individuals. Median follow-up was 1,793 days (IQR 963–2,760), representing a total of 2,262,588 person-years of follow-up. There were a total of 1,982 suicides or accidental deaths during follow-up, including 513 within 180 days of discharge, 1,137 within 2 years and 1,758 within 5 years of discharge. Characteristics of the cohort, including those who did or did not experience the endpoint, are summarized in Table 2. The outcome was observed most often in those who were male, those who were white, and those with public insurance (Table 2).

Table 2.

Characterization of study cohort overall and stratified by primary outcome (suicide or accidental death).

Overall Non-outcome Outcome
Subjects (n) 444317 442335 1982
Sex (Male (%)) 182326 (41.0) 181096 (40.9) 1230 (62.1)
Race (White (%)) 336922 (75.8) 335290 (75.8) 1632 (82.3)
Age (Mean (SD)) 53.15 (18.69) 53.12 (18.68) 58.39 (19.43)
Outcome of Suicide (%) 232 (0.1) 0 (0.0) 232 (11.7)
eRDoC Domain
 Arousal and Regulatory (Mean (SD)) 7.93 (5.49) 7.93 (5.49) 8.24 (5.05)
 Cognitive (Mean (SD)) 4.62 (4.19) 4.61 (4.19) 5.19 (4.56)
 Negative (Mean (SD)) 5.72 (4.40) 5.71 (4.40) 6.58 (4.70)
 Positive (Mean (SD)) 8.69 (6.31) 8.68 (6.31) 10.49 (7.07)
 Social (Mean (SD)) 3.56 (3.58) 3.56 (3.58) 4.39 (4.30)
Open in a new tab

eRDoC = estimated Research Domain Criteria

Kaplan-Meier survival curves for the cohort, comparing top versus bottom quintile for each eRDoC domain, are illustrated in Figure 1A-E, with shading indicating 95% confidence intervals. Importantly, to facilitate readability, the y-axis of each of these is restricted, reflecting the modest absolute risk for suicide in the cohort as a whole. Quintiles differed significantly (corrected p < .001) for all five eRDoC domains in analyses stratified for age, sex, and race. Plots of instantaneous hazard of the outcome in time-varying models adjusted for age, sex, and race are shown over the first year of follow-up in Figure 2A-E. Early hazard predominates in all five eRDoC domains but the highest and lowest risk quintiles hazard remains separated in longer term follow-up (Figure 2).

Figure 1A-E.

Figure 1A-E.

Figure 1A-E.

Figure 1A-E.

Figure 1A-E.

Figure 1A-E.

Open in a new tab

Kaplan Meier curves contrasting highest and lowest quintile of Arousal & Regulatory (A), Cognitive (B), Negative (C), Positive (D), and Social (E) eRDoC domain scores time to primary outcome.

Figure 2A-E.

Figure 2A-E.

Figure 2A-E.

Figure 2A-E.

Figure 2A-E.

Figure 2A-E.

Open in a new tab

Hazard curves from Royston-Parmar time-varying parametric survival model contrasting highest and lowest quintile of Arousal & Regulatory (A), Cognitive (B), Negative (C), Positive (D), and Social (E) eRDoC domain scores.

To facilitate comparison of the size of effect associated with stratification into the top versus the bottom quintile, we compared the ratio of cumulative events in the top versus the bottom quintile of each domain over the follow-up period (Figure 3A) with bootstrap 95% confidence intervals at selected time points (Figure 3B). The magnitude of effect is greatest for positive valence, although by 2 years similar risk is observed for negative valence. In aggregate, while Figure 1 illustrates modest absolute change in risk, Figure 3 illustrates substantial relative change in risk, with 3–5 fold greater hazard in the top compared to the bottom quintile for each domain.

Figure 3A-B.

Figure 3A-B.

Figure 3A-B.

Open in a new tab

Comparison between eRDoC domains (color) ratio of within domain ratio of cumulative event in the highest and lowest quintile of eRDoC domain score over time with LOESS curve (A) and within bootstrap 95% confidence intervals (B).

In sensitivity analysis (not shown), we also examined a more restricted outcome, the 232 explicitly documented suicides during follow-up. In this more limited analysis the direction of effect was similar to the primary analysis; however, the difference between highest and lowest risk quintile of the cognitive domain and arousal and regulatory domain was no longer statistically significant. In all five cases, by inspection, the period of greatest hazard was shifted earlier in the follow-up period. Likewise, a sensitivity analysis examining quartiles rather than quintiles led to similar results (see Supplemental Figure).

DISCUSSION

In this investigation of 444,317 hospitalized individuals across two large academic medical centers, we identified modest but significant associations between each of the five eRDoC scores and risk of suicide or accidental death over more than 2.2 million person-years of follow-up. Among these, greatest magnitude of risk was associated with positive valence, particularly during the first 1.5 years of follow-up; while in general risk peaked in the first 90 days and subsequently decayed for all domains, negative valence represented an exception to this pattern, with a gradual increase in risk after 1 year that plateaued by 2 years.

In a prior meta-analysis of suicide risk factors categorized according to RDoC domains, the majority were associated with negative valence (Glenn et al., 2018), although - consistent with our results - effect sizes for other domains were generally similar. Our results provide further support for the importance of negative valence, but also indicate the potential utility of further investigation of additional RDoC domains. More broadly, these results indicate the feasibility of characterizing these domains in non-psychiatric populations; while prediction of risk in known psychiatric cohorts is also important, the majority of suicide attempts occur among individuals who recently sought contact with non-psychiatrist caregivers (Luoma et al., 2002), and investigation of such cohorts has been less extensive (Ursano et al., 2018).

Perhaps less evident, but equally notable, we identified differential effects of these domains over time, particularly for negative valence. While the basic notion that short- and long-term suicide risk factors may be different is frequently discussed, our results indicate the complexity of this relationship. Although in general the risk associated with each of these domains diminishes with time, likely indicating that these measures are state- rather than trait- characteristics, the rate of decay varies. This complexity may help to guide future studies and interventions, with an effort to match assessments and interventions during the post-discharge period. That is, recognizing that the greatest contributors to risk change over time, multiple strategies to reduce suicide risk may be required depending on whether short, intermediate, or longer-term risk is being considered.

We note multiple important considerations in interpreting our result. First, the extent to which natural language processing efforts generalize across health systems remains to be established, and represents a challenge for all such studies. The eRDoC algorithm used does not incorporate negation (see, for example, Chapman et al., 2001), so ‘psychosis and ‘absence of psychosis’ would have the same impact on eRDoC score. Negation was not included in the original validation of the eRDoC algorithm (McCoy Jr, Yu, et al., 2018). Negation is an interesting avenue for future research; however, in experience with manual narrative note annotation suggests the majority of negated concepts appear in templated text such as review of systems and added little to classification (Filannino et al., 2017). Such templated text could also introduce bias if associated with particular clinicians caring for higher-risk patients; however, in this academic medical center based cohort, narrative notes are nearly always written by house officers who rotate on and off service, leading to essentially random distribution. Second, it bears emphasis that the estimated domain scores are not proxies or substitutes for actual assessment of individual symptoms or RDoC experimental paradigms; it is entirely possible that more focused assessment (for example, of cognitive symptoms) would identify larger effects. Indeed, the present study represents a starting point for improvement. Third, our outcome - suicide and accidental death - is drawn from death indices that may be prone to misclassification. Indeed, we combine these two groups because in prior work we have observed substantial elevation in risk of both outcomes among individuals with psychiatric diagnoses (McCoy, Jr et al., 2016). On the other hand, we would expect that such misclassification would lead us to underestimate effect sizes by introducing greater heterogeneity (i.e., if true suicides are classed as other causes of death, and vice versa).

Finally, while these models do improve risk stratification, they require further work before clinical deployment is likely to be useful: the goal of this report is to highlight the usefulness of natural language processing and RDoC domains, not to present an actionable risk model. While suicide remains among the most studied outcome of psychiatric illness, and its contribution to mortality continues to increase, the absolute rate of suicide in this clinical population remains low. On the other hand, this analysis also has relevant strengths. First, it reflects a large and highly generalizable pair of cohorts drawn from two different general hospital systems with distinct clinical cultures and catchment areas. While these are ‘open’ health systems insofar as participants may get care elsewhere, the mortality outcome is established by death certificates and national death index data. Second, the clinical measures are drawn from routine clinical care, and do not require any additional administration of research measures or paradigms, as such these scores could be delivered in a clinical setting without changes to clinical workflow. Finally, temporal dependency of results and the ability to target potentially modifiable eRDoC symptoms offers appealing avenues for future study as compared to fixed demographic traits.

CONCLUSION

These results indicate that estimates of dimensional psychopathology derived from natural language processing applied to hospital discharge documentation are associated with suicide and accidental death. They suggest the utility of pursuing more sensitive measures of such dimensions in order to better understand and predict this most feared psychiatric outcome. More broadly, they add to a growing literature supporting the use of narrative clinical notes to capture essential clinical data not reflected in standard administrative data.

Supplementary Material

Supp figS1

Supplemental Figure. Comparison between eRDoC domains (color) ratio of within domain ratio of cumulative event in the highest and lowest quartile (vs quintile) of eRDoC domain score over time with LOESS curve. eRDoC = estimated Research Domain Criteria.

ACKNOWLEDGMENTS

This study was funded by the National Institute of Mental Health and the Brain and Behavior Foundation. The sponsors had no role in study design, writing of the report, or data collection, analysis, or interpretation.

Footnotes

Conflict of Interest Statement

THM receives research funding from The Stanley Center at the Broad Institute, Brain and Behavior Foundation, and Telefonica Alfa. RHP holds equity in Psy Therapeutics; serves on the scientific advisory board of Genomind and Psy Therapeutics; and consults to RID Ventures. RHP receives research funding from NIMH, NHLBI, NHGRI, and Telefonica Alfa. AMP has no disclosures to report.

REFERENCES

  1. Barak-Corren Y, Castro VM, Javitt S, Hoffnagle AG, Dai Y, Perlis RH, … Reis BY. (2017). Predicting Suicidal Behavior From Longitudinal Electronic Health Records. The American Journal of Psychiatry, 174(2), 154–162. 10.1176/appi.ajp.2016.16010077 [DOI] [PubMed] [Google Scholar]
  2. Chapman WW, Bridewell W, Hanbury P, Cooper GF, & Buchanan BG (2001). A simple algorithm for identifying negated findings and diseases in discharge summaries. Journal of Biomedical Informatics, 34(5), 301–310. 10.1006/jbin.2001.1029 [DOI] [PubMed] [Google Scholar]
  3. Chung DT, Ryan CJ, Hadzi-Pavlovic D, Singh SP, Stanton C, & Large MM (2017). Suicide Rates After Discharge From Psychiatric Facilities: A Systematic Review and Meta-analysis. JAMA Psychiatry, 74(7), 694 10.1001/jamapsychiatry.2017.1044 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Cuthbert BN (2014). The RDoC framework: facilitating transition from ICD/DSM to dimensional approaches that integrate neuroscience and psychopathology. World Psychiatry, 13(1), 28–35. 10.1002/wps.20087 [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Fanelli G, & Serretti A (2019). The influence of the serotonin transporter gene 5-HTTLPR polymorphism on suicidal behaviors: a meta-analysis. Progress in Neuro-Psychopharmacology & Biological Psychiatry, 88, 375–387. 10.1016/j.pnpbp.2018.08.007 [DOI] [PubMed] [Google Scholar]
  6. Filannino M, Stubbs A, McCoy TH, Rosenquist JN, Brown HE, Perlis RH, & Uzuner Ö (2017). Symptom severity prediction from neuropsychiatric clinical records: Overview of 2016 CEGS N-GRID shared tasks Track 2. Journal of Biomedical Informatics, 75 Suppl, S62–S70. 10.1016/j.jbi.2017.04.017 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Franklin JC, Ribeiro JD, Fox KR, Bentley KH, Kleiman EM, Huang X, … Nock MK. (2017). Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research. Psychological Bulletin, 143(2), 187–232. 10.1037/bul0000084 [DOI] [PubMed] [Google Scholar]
  8. Glenn CR, Kleiman EM, Cha CB, Deming CA, Franklin JC, & Nock MK (2018). Understanding suicide risk within the Research Domain Criteria (RDoC) framework: A meta-analytic review. Depression and Anxiety, 35(1), 65–88. 10.1002/da.22686 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Glenn CR, & Nock MK (2014). Improving the short-term prediction of suicidal behavior. American Journal of Preventive Medicine, 47(3 Suppl 2), S176–180. 10.1016/j.amepre.2014.06.004 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Gvion Y, & Levi-Belz Y (2018). Serious Suicide Attempts: Systematic Review of Psychological Risk Factors. Frontiers in Psychiatry, 9, 56 10.3389/fpsyt.2018.00056 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Hawton K, Casañas I Comabella C, Haw C, & Saunders K (2013). Risk factors for suicide in individuals with depression: a systematic review. Journal of Affective Disorders, 147(1–3), 17–28. 10.1016/j.jad.2013.01.004 [DOI] [PubMed] [Google Scholar]
  12. Heron M (2018). Deaths: Leading Causes for 2016. National Vital Statistics Reports: From the Centers for Disease Control and Prevention, National Center for Health Statistics, National Vital Statistics System, 67(6), 1–77. [PubMed] [Google Scholar]
  13. Insel TR, Cuthbert BN, Garvey M, Heinssen R, Pine DS, Quinn K, … Wang P (2010). Research domain criteria (RDoC): toward a new classification framework for research on mental disorders. The American Journal of Psychiatry, 167(7), 748–751. 10.1176/appi.ajp.2010.09091379 [DOI] [PubMed] [Google Scholar]
  14. Insel TR, & Cuthbert BN (2015). Medicine. Brain disorders? Precisely. Science (New York, N.Y.), 348(6234), 499–500. 10.1126/science.aab2358 [DOI] [PubMed] [Google Scholar]
  15. Le-Niculescu H, Levey DF, Ayalew M, Palmer L, Gavrin LM, Jain N, … Niculescu AB (2013). Discovery and validation of blood biomarkers for suicidality. Molecular Psychiatry, 18(12), 1249–1264. 10.1038/mp.2013.95 [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Luoma JB, Martin CE, & Pearson JL (2002). Contact with mental health and primary care providers before suicide: a review of the evidence. The American Journal of Psychiatry, 159(6), 909–916. 10.1176/appi.ajp.159.6.909 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. McCoy TH Jr, Castro VM, Roberson AM., Snapper LA, & Perlis RH. (2016). Improving Prediction of Suicide and Accidental Death After Discharge From General Hospitals With Natural Language Processing. JAMA Psychiatry, 73(10), 1064–1071. 10.1001/jamapsychiatry.2016.2172 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. McCoy TH Jr, Castro VM, Rosenfield HR, Cagan A, Kohane IS, & Perlis RH. (2015). A clinical perspective on the relevance of research domain criteria in electronic health records. The American Journal of Psychiatry, 172(4), 316–320. 10.1176/appi.ajp.2014.14091177 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. McCoy TH Jr., Castro VM, Hart KL, Pellegrini AM, Yu S, Cai T, & Perlis RH (2018). Genome-wide Association Study of Dimensional Psychopathology Using Electronic Health Records. Biological Psychiatry, 83(12), 1005–1011. 10.1016/j.biopsych.2017.12.004 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. McCoy TH Jr, Yu S, Cai T, & Perlis RH. (n.d.). CQH Dimensional Phenotyper. Retrieved from https://github.com/thmccoy/CQH-Dimensional-Phenotyper
  21. McCoy TH Jr, Yu S, Hart KL, Castro VM, Brown HE, Rosenquist JN, … Perlis RH (2018). High Throughput Phenotyping for Dimensional Psychopathology in Electronic Health Records. Biological Psychiatry, 83(12), 997–1004. 10.1016/j.biopsych.2018.01.011 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Murphy SN, Mendis M, Hackett K, Kuttan R, Pan W, Phillips LC, … Chueh HC (2007). Architecture of the open-source clinical research chart from Informatics for Integrating Biology and the Bedside. AMIA Annual Symposium Proceedings, 548–552. [PMC free article] [PubMed] [Google Scholar]
  23. Murphy SN, Weber G, Mendis M, Gainer V, Chueh HC, Churchill S, & Kohane I (2010). Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). Journal of the American Medical Informatics Association: JAMIA, 17(2), 124–130. 10.1136/jamia.2009.000893 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Niculescu AB, Levey D, Le-Niculescu H, Niculescu E, Kurian S, & Salomon D (2015). Psychiatric blood biomarkers: avoiding jumping to premature negative or positive conclusions. Molecular Psychiatry, 20(3), 286–288. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Rossen LM, Hedegaard H, Khan D, & Warner M (2018). County-Level Trends in Suicide Rates in the U.S., 2005–2015. American Journal of Preventive Medicine, 55(1), 72–79. 10.1016/j.amepre.2018.03.020 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Royston P, & Parmar MKB (2002). Flexible parametric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects. Statistics in Medicine, 21(15), 2175–2197. 10.1002/sim.1203 [DOI] [PubMed] [Google Scholar]
  27. Runeson B, Odeberg J, Pettersson A, Edbom T, Jildevik Adamsson I, & Waern M (2017). Instruments for the assessment of suicide risk: A systematic review evaluating the certainty of the evidence. PloS One, 12(7), e0180292 10.1371/journal.pone.0180292 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Schoenfeld D (1980). Chi-Squared Goodness-of-Fit Tests for the Proportional Hazards Regression Model. Biometrika, 67(1), 145–153. 10.2307/2335327 [DOI] [Google Scholar]
  29. Stone DM, Simon TR, Fowler KA, Kegler SR, Yuan K, Holland KM, … Crosby AE (2018). Vital Signs: Trends in State Suicide Rates — United States, 1999–2016 and Circumstances Contributing to Suicide — 27 States, 2015. MMWR Morbidity and Mortality Weekly Report, 67(22), 617–624. 10.15585/mmwr.mm6722a1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Turecki G, & Brent D (2015). Suicide and suicidal behavior. The Lancet, 387(10024), 1227–1239. 10.1016/S0140-6736(15)00234-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Ursano RJ, Kessler RC, Naifeh JA, Herberman Mash HB, Nock MK, Aliaga PA, … Stein MB. (2018). Risk Factors Associated With Attempted Suicide Among US Army Soldiers Without a History of Mental Health Diagnosis. JAMA Psychiatry, 75(10), 1022–1032. 10.1001/jamapsychiatry.2018.2069 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supp figS1

Supplemental Figure. Comparison between eRDoC domains (color) ratio of within domain ratio of cumulative event in the highest and lowest quartile (vs quintile) of eRDoC domain score over time with LOESS curve. eRDoC = estimated Research Domain Criteria.

NIHMS1007404-supplement-Supp_figS1.tif (873.2KB, tif)

RESOURCES