Abstract
Background.
Emergency Departments (ED) are the first line of evaluation for patients at risk and in crisis, with or without overt suicidality (ideation, attempts). Currently employed triage and assessments methods miss some of the individuals who subsequently become suicidal. The Convergent Functional Information for Suicidality (CFI-S) 22 item checklist of risk factors, that does not ask directly about suicidal ideation, has demonstrated good predictive ability for suicidality in previous studies in psychiatric outpatients, but has not been tested in the real world-setting of emergency departments (EDs).
Methods.
We administered CFI-S prospectively to a convenience sample of consecutive ED patients. Median administration time was 3 minutes. Patients were also asked at triage about suicidal thoughts or intentions per standard ED suicide clinical screening (SCS), and the treating ED physician was asked to fill a physician gestalt visual analog scale (VAS) for likelihood of future suicidality spectrum events (SSE) (ideation, preparatory acts, attempts, completed suicide). We performed structured chart review and telephone follow-up at 6 months post index visit.
Results.
The median time to complete the CFI-S was three minutes (1st to 3rd quartile 3–6 minutes). Of the 338 patients enrolled, 45 (13.3%) were positive on the initial SCS, and 32 (9.5%) experienced a SSE in the 6 months follow-up. Overall, across genders, SCS had a modest diagnostic discrimination for future SSE (ROC AUC 0.63,). The physician VAS was better (AUC 0.76 CI 0.66–0.85), and the CFI-S was slightly higher (AUC 0.81, CI 0.76–0.87). The top CFI-S differentiating items were psychiatric illness, perceived uselessness, and social isolation. The top CFI-S items were family history of suicide, age, and past history of suicidal acts.
Conclusions.
Using CFI-S, or some of its items, in busy EDs may help improve the detection of patients at high risk for future suicidality.
Keywords: suicide, emergency departments, screening, prediction, triage, CFI-S
Introduction
With more than 4 million visits to the ED annually in the US for mental disorders, the ED is considered by many to be the first line of evaluation for patients at risk and in crisis, with or without overt suicidality (ideation, attempts).1 The weighted national estimate of patients with a diagnosis of “Suicide or Intentional Self-Harm” in the National Emergency Department Sample (NEDS) for 2013 was 1,411,770, patients, with 98.8% of those with suicidality as a first diagnosis discharged from the ED. In 2013, visits for suicidal ideation accounted for nearly 1 percent of all adult ED visits (108.3 million visits)2
To better predict and prevent suicides, emergency care providers need improved risk stratification tools for patients with overt or covert mental health crisis. Several tools-- PHQ93 and the ED-Safe Patient Safety Screener4 and the Suicide Behaviors Questionnaire—Revised (SBQ-R)5—have been created and validated for suicide screening. All of these include direct questioning about current or recent suicidal thoughts. None of these tools has been compared to physician gestalt or evaluated for their ability to predict rates of adverse suicide related events in patients who screen negative using the tool. There are yet no widespread clinically used simple objective tools to assess and track changes in suicidal risk without asking the individuals directly, although others in the field besides us are actively working on this problem and progress is being made (for example Nock and colleagues6,7, Boudreaux and colleagues8–10).Such tools are desperately needed, as individuals at risk may choose not to share their ideation or intent with others, for fear of stigma, hospitalization, or that in fact their plans may be thwarted.
The Convergent Functional Information for Suicidality (CFI-S) is a novel suicide risk instrument that comprises 22 questions and has shown good to excellent predictive value for suicidality in settings other than the emergency department (ED)11–13. The CFI-S is a checklist of risk factors for suicidality from a variety of domains including life satisfaction, mental health, physical health, environmental stress, addictions, cultural factors, and demographic information, and assigns a numeric point value for each response, 0 for absent or 1 for present. In essence, it is a “polyphenic” risk score, by analogy with polygenic risk scores. The tool was designed to be easy to score by self -administration, clinician administration, or based on medical records or next of kin information. Of note, it does not ask directly about suicidal ideation, as that is a delicate question in many non-specialized settings, and people who are truly suicidal might not share that information for fear of being stopped. .
We hypothesized that the CFI-S could be used in a heterogeneous sample of ED patients to identify high risk patients whose elevated risk was missed by both standard screening and by physician evaluation as measured by their gestalt impression of future risk.
We sought to test the accuracy of the CFI-S and physician gestalt VAS in a sample of urban ED patients, with a traditionally high proportion of non-Caucasians and low-income individuals. The reason for this is this population has a higher than average risk of suicidality14but with the lowest access to mental health services15. Thus, in this population, the need for accurate suicide risk assessment is compounded by the lack of current identification of non-overt suicidal ideation, and the need for pragmatic use of limited resources.
Methods
This study received approval from the Indiana University School of Medicine Institutional Review Board and all patients completed a written informed consent process before participation. Patients were eligible for inclusion if they were over 18 and were able to participate in the survey. Exclusion criteria included age less than 18, severe trauma or illness requiring emergent intervention or acute intoxication as reported by the clinical staff or patient. Patients were enrolled in the ED of the Sydney and Lois Eskenazi Hospital, an urban safety net hospital which has an annual volume of 95,000 visits. Between March of 2016 and April 2017 the CFI-S was prospectively administered to a sample of 338 ED patients. The patients were consecutive and non-selected, meaning the research staff enrolled patients one after the other without interruption, and without regard to the chief complaint. Research staff approached patients after their initial nursing and physician assessments were complete. They administered the CFI-S asking subjects for yes or no answers to all questions, documenting start and end times of the survey. Questions were asked exactly as written and requests for clarification from patients responded to with “please answer the question as you understand it.” Each answer of yes resulted in one point. The final CFI-S score is calculated by dividing the total number of affirmative answers by the total number of questions answered.
After the CFI-S was completed, research staff also approached the patient’s ED physician (board-certified emergency physicians, or emergency medicine residents in training) to obtain their physician clinical gestalt VAS regarding the patient’s risk of future suicide related outcomes over the following 6 months. Physicians were asked to assess their numeric probability (0–100%) that the patient would have a suicidality spectrum event (SSE), defined as repeat ED visit or admission for suicidal ideation, preparatory acts, suicide attempts, aborted or interrupted attempts or completed suicide in the 6 months following the patient’s index ED visit. This spectrum of severity approach is substantiated by our previous blood biomarker studies12,13, and is supportive of suicidality as its own free standing diagnosis, per the proposal of Oquendo and colleagues16. Physician gestalt VAS was obtained by clinicians making a vertical hash mark with a pen on a 10 cm horizontal line with 0 cm equaling 0% probability of suicide and 10 cm representing 100% probability. Research assistants also recorded subjects’ responses to the health system’s standard two question universal screening tool. This screen includes the questions: “Do you have any thoughts of hurting yourself or anyone else?” and “Do you feel hopeless or helpless?” An affirmative response to either question is considered a positive screen in the health system but only those answering yes to the first question were coded as positive suicide screen for the purposes of our analysis. Patients were followed prospectively for SSEs. Members of the research team, who were trained by the principal investigators, performed structured telephone and chart review follow up on all patients 6 months post index visit. For follow-up, study personnel dialed the telephone number that patients directly stated as the best way to contact them. The number was dialed at least three times on different days at different times of day. After 6 months, we also queried the CareWeb electronic interface to the Indiana Network for Patient Care (INPC) system for a suicidal event. The INPC represents over 100 separate healthcare entities in Indiana providing data including hospitals, health networks, and insurance providers. Chart reviewers were trained study personnel who individually read and considered any documented medical encounter for specific words suggesting a suicidal event. Ambiguous cases were resolved by consensus between two authors.
We treated the SCS, VAS and CFI-S as diagnostic tests with the primary outcome as SSE by 6 months. This work was done in accordance with the STARD guidelines for a diagnostic study17. Specifically, assessors of the primary outcome were blinded to the diagnostic test data.
Data were entered in REDcap and transferred to a spreadsheet for analysis. All data was z-scored by gender, to eliminate potential gender effects in the combined analysis. Diagnostic accuracy was assessed by 2×2 contingency table analysis to generate point estimates of sensitivity and specificity as well as the area under the receiver operating characteristic curve with associated 95% confidence intervals for each index. We compared areas under the receiver operating characteristic curve for gestalt VAS versus the CFI-S using the method of Hanley and McNeil.18 The optimal cutoff was chosen as the point on the receiver operating characteristic curve that conferred the peak diagnostic odds ratio, defined as the maximal likelihood ratio positive/likelihood ratio negative.19 Statistical analyses were performed with StatsDirect Statistical Software (v.3.0.187, Cheshire, England) and plots were made with as performed using GraphPad Prism version 7.00 for Windows, GraphPad Software, La Jolla California USA, www.graphpad.com
Results
As shown in Figure 1, we approached 367 patients, and 338 (92%) agreed to participate. Physician clinical gestalt VAS data were collected on 170 subjects.
Figure 1. Flow diagram of research subjects.
Number of patients approached, enrolled and with completed follow up.
Survey characteristics
The median time to complete the CFI-S was three minutes (1st to 3rd quartile 3–6 minutes). Two research personnel, blinded to each other’s results, administered the CFI-S twice to 10 patients approximately on hour apart and found that the repeated values were within 10% of the first value in 8 of 10 retests. The internal consistency of the CFI-S was good, as reflected by the Cronbach’s alpha of reliability =0.84 (95% lower confidence limit =0.81). Data were missing for one or more questions in 56 (16%) of the surveys.
Patient outcomes and predictive characteristics
Of the 338 patients, 32 (9.5%) experienced a SSE in the 6 months of follow-up. Table 1 outlines the demographics of our sample. The median age of those who experienced an SSE outcome was 39 (IQR 30–51) vs. 44 (IQR 31–54) in those who did not. ED revisits and psychiatric hospitalizations were the most common outcomes. Completed suicides did not occur in our sample in the 6 months follow-up.
Table 1: Demographic information.
Demographic characteristics of patients who experienced a SSE in 180 day follow up and those who did not.
| + SSE outcome | −SSE outcome | |
|---|---|---|
| Number | 32 | 306 |
| Average age (years) | 39 (IQR 30–51) | 44 (IQR 31–54) |
| Sex | ||
| Male | 59.4% | 47.4% |
| Female | 40.6% | 52.6% |
| Race/Ethnicity | ||
| White, non-Hispanic | 44.7% | 39.1% |
| White, Hispanic | 6.4% | 2.4% |
| Black, non-Hispanic | 34.0% | 28.9% |
| Asian, non-Hispanic | 0.0% | 0.7% |
| Not available | 14.9% | 28.9% |
Of the 32 subjects who experienced a suicide related outcome, 18 (56.3%) were not suicidal at first presentation as measured by the health system’s standard two question universal screening tool. This screen includes the questions: “Do you have any thoughts of hurting yourself or anyone else?” and “Do you feel hopeless or helpless?” For study purposes a positive screen required the first question to be answered affirmatively although anyone answering the second question affirmatively did have additional ED assessment for thoughts of self-harm per screening hospital protocol.
Figure 2, 3 and 4 examine predictive ability of SCS, VAS and CFI-S, for future SSEs.
Figure 2: CFI-S, physician gestalt VAS and SI screening.
Comparison of the median (interquartile range) for the CFI-S scores and physician gestalt VAS for patient who screened positive for suicidal thoughts in triage compared to those who did not.
Figure 3:
Receiver operating characteristic curve analysis for the CFI-S and physician gestalt for the criterion standard outcome of a Suicide Spectrum Event (SSE) in the 6 months following the index ED visit.
Figure 4. CFI-S and physician clinical gestalt VAS correlation.
The two measures are correlated. The CFI-S identifies as high risk patients that are not identified as such by the physician’s gestalt VAS.
Comparing the 32 patients who experienced a future SSE with the 306 who did not, the SCS showed a modest discrimination, with an ROC AUC of 0.63 (p= 3.91E-03). Physician VAS shows a higher ability to discriminate with an AUC of 0.75 (p=1.27E-04). CFI-S had the highest numeric AUC of 0.81 (p=5.11E-09). The optimal cutoff for Physician VAS was 1.2 and for the CFI-S was 0.65. To compare the AUCs for Physician VAS versus the CFI-S using the method of Hanley and McNeil first required restriction of data to patients with both values recorded, yielding 25 patients with SSE+ and 149 patients with SSE-. The AUCs for this restricted dataset were 0.78 (SEM 0.04) for the CFI-S and 0.77 (SEM 0.04) for the Physician VAS, yielding z=0.207, or P=0.82.
Figure 4 shows the individual items of the CFI-S that were most able to differentiate between SSE and non-SSE. The top items are history of psychiatric illness, perceived uselessness, and social isolation.
Similar analyses are shown for individuals who were negative on the SCS at the initial visit (Figures 3 and 4).
Discussion
In this sample of 338 consecutive ED patients, we found the current two question universal screen for suicidal intent failed to identify 18 (56.2%) patients who went on to have a SSE within 180 days. Those 18 patients represent 5.3% of our overall sample of unselected ED patients presenting for a wide variety of chief complaints. While several studies have found a high rate of undiagnosed depression in unselected patients,20 to our knowledge, this is the first study to use a tool to predict suicidality in an unselected ED cohort. According to CDC data, an estimated 1.3 million adults aged 18 or older (0.6%) attempted suicide in the past year1, and 43% of patients who complete suicide visit an ED at least once in the year prior to their death21.
Given the increased risk of ED patients and the relative insensitivity of universal screening, we were interested if emergency physician clinical gestalt VAS would be a better diagnostic tool to predict suicidal events. To our knowledge, this is the first study to assess the diagnostic properties of physicians’ clinical gestalt VAS for suicide related outcomes in the general ED population. Unfortunately, physician gestalt for adverse suicide related outcomes as measured by VAS shows only moderate diagnostic discrimination. In our experience ED physicians rely heavily on their clinical gestalt in combination with patients’ responses to direct questioning about suicidal thoughts or plans. Given the limited diagnostic ability of both of these tools, data from Table 1 provide a preliminary inference that emergency physicians could use the three most sensitive questions from the CFI-S (prior mental illness/ feelings of uselessness /social isolation) to improve our ability to detect future risk of suicidal behavior.
In addition to incorporating better risk assessments into our clinical assessment of suicidal risk, making a significant impact on morbidity and mortality related to undiagnosed suicidality will require the development and broad implementation of brief, structured risk assessment tools that obviate the need for patients’ to endorse directly present or recent suicidal thoughts. We found that the CFI-S can be given quickly and effectively in the ED setting, is an accurate predictor of adverse suicide related outcomes and avoids relying on patients’ self-report of current or recent suicidal thoughts.
The CFI-S took three minutes to complete, which although longer than it currently takes to complete our standard two question universal screening, is likely warranted given the potentially life-threatening consequences of a missed diagnosis.
Over half of patients who experienced suicide spectrum events in 180 day follow up were not identified by standard screening. It is both striking and concerning that current screening methods missed more than half of patients who would go on to experience a suicide related outcome.
Given the weakness of current screening tools as well as of physician gestalt VAS, the CFI-S has the ability to more effectively screen for high risk of suicidal outcomes and could help identify patients missed by current screening tools.
Possible future implications of this work include incorporation of more robust but still relatively time efficient risk stratification tools for all patients presenting to the ED and other high risk care settings. In a universal screening protocol, patients with a CFI-S >0.65 should be referred for urgent outpatient mental health evaluation and could be targeted in active community based ED follow-up programs focused on decreasing social isolation and supporting linkage to follow-up care. The 0.65 was determined as the cutoff of the CFI from the receiver operating characteristic curve that produced the highest diagnostic odds ratio (likelihood ratio positive/likelihood ratio negative).
Limitations
Our study sample was a convenience sample recruited from a single, urban safety net hospital, which may not represent a broad sample of patients in EDs across the US. Our criterion standard relied upon discovery from one or both of the following mechanisms: 1. Patients had to be able to recall and volunteer to report suicidal events on telephone interview, or, 2. Discovery of an accurate and clear description of a suicidal event documented by health care providers in medical records. It remains possible that these processes missed some suicidal events. Although all suicide spectrum events are important, those most dangerous (completed suicide) did not occur.
Conclusion
In this single center study that enrolled a heterogeneous sample of ED patients, current screening methods for suicidality failed to identify over half of those who went on to have a suicidal spectrum event in the next 180 days. Physician gestalt had moderate predictive accuracy. The CFI-S has good accuracy for detection of those who would have future events, without directly questioning patients about suicidal thoughts. These data imply that the CFI-S, or some of its component items notably related to mental health and stress, may be useful to improve detection of suicidality risk in emergency care, leading to targeted and personalized preventive strategies. The CFI-S was able to predict adverse suicide related outcomes in some patients who screened negative for suicidal ideation at ED presentation and who were considered low-risk by the physician evaluating them. We suggest that the broad implementation of CFI-S in ED settings may be complementary to current approaches and improve the detection of patients at high risk for future suicidal events.
Table 3:
Suicide spectrum events (SSE) that occurred in the 6 months following the original ED visit
| Number | |
|---|---|
| Total patients | 32 |
| Suicide Attempt | 10 (31.3%) |
| Psychiatric hospitalization for suicidality | 16 (50.0%) |
| Aborted/interrupted attempt | 11 (34.40%) |
| Preparatory Acts | 13 (40.6%) |
| ED visit for suicidal thoughts | 29 (90.1%) |
Table 4:
CFI-S individual items sensitivity and specificity for SSE in the 6 months of follow-up
| CFI-S Question | % answering yes | Sensitivity | Specificity |
|---|---|---|---|
| Q1. History of mental illness | 151 (44.7%) | 96.8% | 40.3% |
| Q2. Do you have poor compliance (i.e. not taking medications or keeping doctor’s appointments?) | 95 (28.1%) | 50.0% | 31.6% |
| Q3. Do you have a family history of suicide or suicide attempts in blood relatives? | 78 (23.1%) | 46.6% | 21.3% |
| Q4. Do you personally know somebody who has committed suicide? | 143 (42.3%) | 75.0% | 39.7% |
| Q5. CSF-5: Do you have a history of abuse physical, sexual, emotional, or neglect? | 130 (38.5%) | 62.5% | 36.5% |
| Q6. Have you had an acute/severe medical illness, including acute pain (‘I just can’t stand this pain anymore.’) within last 3 months? | 153 (45.3%) | 43.8% | 45.7% |
| Q7. Have you experienced acute stress within last 3 months (a major financial, professional, or personal loss or loss/death of a loved one?) | 198 (58.6%) | 80.6% | 57.1% |
| Q8. Have you experienced chronic feelings of uselessness or being a burden to your friends and/or family? | 146 (43.2%) | 87.5% | 38.9% |
| Q9. Do you have a history of introversion (keeping to yourself being a loner) and over-conscientiousness (obsessive), such as planning minute details or being highly organized? | 147 (43.5%) | 59.4% | 43.5% |
| Q10. Are you dissatisfied with how life turned out for you? | 118 (34.9%) | 73.3% | 32.7% |
| Q11. Do you feel hopeless about the future? | 84 (24.9%) | 64.5% | 22.1% |
| Q12. Are you currently abusing substances (drugs, alcohol, medications, painkillers)? | 83 (24.6%) | 40.6% | 23.1% |
| Q13. Have you attempted or threatened or called a suicide hotline in the past (before today)? | 92 (27.2%) | 59.4% | 23.9% |
| Q14. Do you lack religious beliefs? | 76 (22.5%) | 19.4% | 23.6% |
| Q15. Have you experienced rejection (in a relationship, in the family, at work) within the last three months? | 93 (27.5%) | 37.5% | 26.7% |
| Q16. Have you experienced chronic stress (i.e. lack of positive relationships, chronic hopelessness, social isolation)? | 134 (39.6%) | 80.6% | 36.0% |
| Q17. Do you have a history of impulsive behaviors related to anger, such as being in a rage, getting into physical fights, or seeking revenge? | 103 (30.5%) | 45.2% | 29.3% |
| Q18. Do you lack skills for coping with stress (i.e. have a habit of cracking under pressure)? | 106 (31.4%) | 59.4% | 29.9% |
| Q19. Do lack biological children or grandchildren? | 78 (23.1%) | 40.6% | 21.2% |
| Q20. Do you have history of command auditory hallucinations of self-directed violence (hearing voices telling you to harm yourself)? | 44 (13.0%) | 31.3% | 11.1% |
| Q21. Are you younger than 25 or older than 60? | 103 (30.5%) | 25.0% | 31.0% |
| Q22. Are you male? | 164 (48.5%) | 59.4% | 47.4% |
Acknowledgements:
We would like to thank the following medical students for help with the study: Luke Collins, Shelby Cuffley, Melissa Egert, Matt Gray, Kelly Grott Benjamin Judge, Dylan Long, Janette Magallanes, Erica Marburger, Richard Marcinko, Emeka Onwuzurumba, Michael Rice, Christopher Rodarte, Liz Rohn, Sarah Tepner, Leah Tucker, and Piiamaria Virtanen. This work was funded by NIMH/NHLBI K12HL133310 to JK, ABN and KB, and an Eli Lilly Foundation Physician Scientist Award to KB.
Footnotes
Conflict of Interest Disclosure:
KB, CD, JN, ABN, and JK report no conflict of interest related to this work. ABN is listed as an inventor on a patent application for suicide biomarkers filed by Indiana University.
Substance Abuse and Mental Health Services Administration, Results from the 2013 National Survey on Drug Use and Health: Mental Health Findings, NSDUH Series H-49, HHS Publication No. (SMA) 14–4887. Rockville, MD: Substance Abuse and Mental Health Services, 2014. Available at http://www.samhsa.gov/data/sites/default/files/NSDUHmhfr2013/NSDUHmhfr2013.pdf.
REFERENCES
- 1.Owens PL, M R, Stocks C. Mental Health and Substance Abuse-Related Emergency Department Visits among Adults. HCUP Statistical Brief #92 Agency for Healthcare Research and Quality, . [PubMed] [Google Scholar]
- 2.Owens PL (AHRQ) FKIWH, Heslin KC (AHRQ), Mutter R (SAMHSA), Booth CL, (SAMHSA), . Emergency Department Visits Related to Suicidal Ideation, 2006–2013. HCUP Statistical Brief Agency for Healthcare Research and Quality. [PubMed] [Google Scholar]
- 3.Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. Journal of general internal medicine 2001;16:606–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Boudreaux ED, Jaques ML, Brady KM, Matson A, Allen MH. The patient safety screener: validation of a brief suicide risk screener for emergency department settings. Archives of suicide research : official journal of the International Academy for Suicide Research 2015;19:151–60. [DOI] [PubMed] [Google Scholar]
- 5.Osman A, Bagge CL, Gutierrez PM, Konick LC, Kopper BA, Barrios FX. The Suicidal Behaviors Questionnaire-Revised (SBQ-R): validation with clinical and nonclinical samples. Assessment 2001;8:443–54. [DOI] [PubMed] [Google Scholar]
- 6.Glenn CR, Kleiman EM, Cha CB, Deming CA, Franklin JC, Nock MK. Understanding suicide risk within the Research Domain Criteria (RDoC) framework: A meta-analytic review. Depression and anxiety 2018;35:65–88. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Nock MK, Dempsey CL, Aliaga PA, et al. Psychological autopsy study comparing suicide decedents, suicide ideators, and propensity score matched controls: results from the study to assess risk and resilience in service members (Army STARRS). Psychological medicine 2017;47:2663–74. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Boudreaux ED, Camargo CA Jr., Arias SA, et al. Improving Suicide Risk Screening and Detection in the Emergency Department. American journal of preventive medicine 2016;50:445–53. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Boudreaux ED, Jaques ML, Brady KM, Matson A, Allen MH. The Patient Safety Screener: Validation of a Brief Suicide Risk Screener for Emergency Department Settings. Archives of suicide research : official journal of the International Academy for Suicide Research 2017;21:52–61. [DOI] [PubMed] [Google Scholar]
- 10.Miller IW, Camargo CA Jr., Arias SA, et al. Suicide Prevention in an Emergency Department Population: The ED-SAFE Study. JAMA psychiatry 2017;74:563–70. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Levey DF, Niculescu EM, Le-Niculescu H, et al. Towards understanding and predicting suicidality in women: biomarkers and clinical risk assessment. Molecular psychiatry 2016;21:768–85. [DOI] [PubMed] [Google Scholar]
- 12.Niculescu AB, Le-Niculescu H, Levey DF, et al. Precision medicine for suicidality: from universality to subtypes and personalization. Molecular psychiatry 2017;22:1250–73. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Niculescu AB, Levey DF, Phalen PL, et al. Understanding and predicting suicidality using a combined genomic and clinical risk assessment approach. Molecular psychiatry 2015;20:1266–85. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Li Z, Page A, Martin G, Taylor R. Attributable risk of psychiatric and socio-economic factors for suicide from individual-level, population-based studies: a systematic review. Social science & medicine (1982) 2011;72:608–16. [DOI] [PubMed] [Google Scholar]
- 15.Cummings JR, Wen H, Ko M, Druss BG. Geography and the Medicaid mental health care infrastructure: implications for health care reform. JAMA psychiatry 2013;70:1084–90. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Oquendo MA, Baca-Garcia E, Mann JJ, Giner J. Issues for DSM-V: suicidal behavior as a separate diagnosis on a separate axis. The American journal of psychiatry 2008;165:1383–4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ (Clinical research ed) 2015;351:h5527. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983;148:839–43. [DOI] [PubMed] [Google Scholar]
- 19.Glas AS, Lijmer JG, Prins MH, Bonsel GJ, Bossuyt PM. The diagnostic odds ratio: a single indicator of test performance. Journal of clinical epidemiology 2003;56:1129–35. [DOI] [PubMed] [Google Scholar]
- 20.Abar B, Holub A, Lee J, DeRienzo V, Nobay F. Depression and Anxiety Among Emergency Department Patients: Utilization and Barriers to Care. Academic emergency medicine : official journal of the Society for Academic Emergency Medicine 2017;24:1286–9. [DOI] [PubMed] [Google Scholar]
- 21.Da Cruz D, Pearson A, Saini P, et al. Emergency department contact prior to suicide in mental health patients. Emergency medicine journal : EMJ 2011;28:467–71. [DOI] [PubMed] [Google Scholar]