Which Chart Elements Accurately Identify Emergency Department Visits for Suicidal Ideation or Behavior?

Abstract

Objective:

In an emergency department (ED) sample, we investigated the concordance between identification of suicide-related visits through standardized comprehensive chart review versus a subset of three specific chart elements: ICD-9-CM codes, free-text presenting complaints, and free-text physician discharge diagnoses.

Methods:

Review of medical records for adults (≥18 years) at eight EDs across the United States.

Results:

A total of 3,776 charts were reviewed. A combination of the three chart elements (ICD-9-CM, presenting complaints, and discharge diagnoses) provided the most robust data with 85% sensitivity, 96% specificity, 92% PPV, and 92% NPV.

Conclusions:

These findings highlight the use of key discrete fields in the medical record that can be extracted to facilitate identification of whether an ED visit was suicide-related.

Introduction

Using electronic health records (EHRs) to identify and monitor patients at-risk for suicide has become more common after adoption of legislature like the Health Information Technology for Economic and Clinical Health Act (HITECH; US DHHS, 2009). This is particularly important because most individuals at-risk for suicide receive treatment at a hospital, emergency department, or primary care setting around the time of their suicide attempt (SA) or death by suicide (Gairin et al., 2003; Luoma et al., 2002; McCaig & Burt, 2005). However, a crucial step for effectively tracking suicide risk through EHR data is to first determine whether a healthcare encounter is suicide-related.

One frequently used method for studying various health-related research questions is chart review. Half of the articles in emergency medical services journals and 25% in peer-reviewed emergency medicine journals are chart review studies (Kaji et al., 2014). Due to the large amount of data in the healthcare system, researchers often use International Classification of Diseases (ICD) codes to identify visits associated with the health topic of interest (Chang et al., 2016). In regard to suicidal ideation (SI) or SA, there is limited information on how these visits are documented in the EHR (Anderson et al., 2015). Initial findings suggest that research that relies on ICD codes from the EHR to study suicide-related outcomes significantly underestimates both SI and SA cases (Anderson et al., 2015). A study on depressed primary care patients with documentation of SI in clinical notes found that only 3% had a corresponding ICD-9-CM code, while of the 86 patients with SA documented, only 19% had a corresponding ICD-9-CM code (Anderson et al., 2015).

In the absence of a universal formalized assessment of SI/SA at the time of a healthcare encounter, a complete chart review by trained raters should provide the most information about whether a healthcare encounter is related to SI/SA. However, individual and comprehensive chart review is a time-intensive and costly process. In this study, we sought to determine whether specific chart elements, which can be easily extracted from the EHR, representing both structured and unstructured (free-text) data, provide sufficient information to approximate results from a complete chart review of the index emergency department (ED) visit. We focused on three specific chart elements: (1) ICD-9-CM codes assigned by professional hospital coding staff for the purposes of claims and billing, (2) free-text presenting complaint, and (3) free-text physician discharge diagnosis.

Method

Setting, study design, and participant selection

Data for the current study were collected through a complete chart review of index visit records for ED patients presenting to the eight participating sites involved in the Emergency Department Safety and Follow-up Evaluation (ED-SAFE) study, a NIH-funded suicide prevention study (Boudreaux et al., 2013). Institutional review boards at all participating sites approved the study.

The participating sites reviewed a total of 3,776 charts. We initially examined 2,400 charts from randomly selected patients from the general ED population. However, there were only 2% of SI/SA cases detected. Although this is similar to the expected rate of SI/SA in the ED (Owens et al., 2017), we decided to enhance the sample for comparison purposes by including chart reviews from the 1,376 participants enrolled in the ED-SAFE study who were identified as having SI/SA during the index ED visit by research staff. Visits occurred between October 2009 and December 2013. All records were for patients 18 years and older. All chart review data were entered in a secure web-based data collection system (Research Electronic Data Capture (REDCap), Vanderbilt, TN).

Measurements or key outcome measures

Research personnel at each site completed a standardized abstraction form that included information such as patient demographics, ED presentation (e.g., chief complaint and triage code), documented mental health indicators, physician discharge diagnosis, and billing codes for the visit. Comprehensive chart reviews were conducted by trained chart abstractors at each site for documentation of whether the patient had any history, current or past, of SI/SA. Chart abstractors were instructed to use any documentation that occurred while the participant was in the ED. The ED stay was considered to have ended when the participant physically left the ED.

For the chart review of the index ED visit, documentation of SI/SA included any indication of suicidal ideation, suicide attempt, a history of suicide attempt, or reported suicidal thoughts. For the current study, if SI/SA was documented by the chart reviewers as “Yes, in the past week”, this was considered a “Yes” for active SI/recent SA via chart review. We repeated our analyses including both “Yes, in past week” and “Yes, no time specified” responses to SI/SA during the current ED visit. No significant differences were detected between the “Yes, in past week” group and the one including “Yes, no time specified”, so we defined a “recent” SI/SA visit as one occurring in the past week.

Three specific chart elements were tested as indicators of SI/SA-related ED visits derived from the complete chart review. First, we defined visits as suicide-related if there was an ICD-9-CM code for suicidal ideation (V62.84) or E codes of suicide and self-inflicted injury (E950-E959) in any diagnosis field as assigned by the site’s professional coding staff. This is the kind of data typically entered into claims databases. Then, two clinical researchers searched for terms in the free-text from the presenting complaint and the physician discharge diagnosis. These researchers did not have knowledge of information from elsewhere in the chart that would have influenced their coding of a visit as suicide-related; they assigned codes using a spreadsheet that listed only presenting complaints and physician discharge diagnosis. The free-text was coded as suicide-related if there was any term related to suicidal ideation or self-harm behavior with ideation or intent (e.g., intentional overdose, wrist laceration + suicidal ideation). For any presenting complaint where there was self-inflicted injury, but the intent was unclear (e.g., drug overdose with no documentation of suicidal ideation or intent), we made a decision not to code these as suicide-related visits. These guidelines were adapted from the procedures used during the complete chart review conducted during the ED-SAFE study. A randomly selected subset of 10% of cases were independently reviewed by two clinical researchers for accuracy. Kappa values were >.72, indicating a good level of agreement (Fleiss, 1981).

Data analysis

We compared identification of SI/SA through a complete chart review of the index ED visit to a subset of three specific chart elements (ICD-9-CM code, presenting complaint, physician discharge diagnosis). Percentages of SI/SA ED visits detected by these data sources were estimated using Stata 13.2 (StataCorp, College Station, TX). Agreement between our reference outcome, complete chart review, and the subset of three chart elements was calculated using sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and the kappa chance-corrected agreement statistic.

Results

A total of 3,776 charts were reviewed for patients with a median age of 39 (interquartile range, 27–52) years, 2,134 (57%) were female, 2,451 (65%) white, and 2,286 (61%) non-Hispanic. Using standardized chart review procedures, there were 916 charts (24%; 95%CI, 23–26) where SI only (no attempt) in the past week was documented and 387 (10%; 95%CI, 9–11) with SA in the past week.

We first examined identification of visits associated with suicide attempt (i.e., patient documented as engaging in intentional self-harm behavior with intent to die, not simply reporting suicidal ideation or intent). Of the 387 charts with SA documented in the past week, 30% (n=117; 95%CI, 26–35) would have been identified by an ICD-9-CM code associated with suicide or intentional self-inflicted injury, 35% (n=134; 95%CI, 30–40) by free-text in the presenting complaint, and 29% (n=112; 95%CI, 25–34) by free-text in the physician discharge diagnosis.

Of the 1,303 with SI or SA in the past week identified by complete chart review, 25% (n=327; 95%CI, 23–28) would have been identified by ICD-9-CM code associated with suicide or intentional self-inflicted injury, 73% (n=951; 95%CI, 71–75) by free-text in the presenting complaint, and 54% (n=708; 95%CI, 52–57) by free-text in the physician discharge diagnosis (Table 1). A combination of the three chart elements (ICD-9-CM code, presenting complaint, and discharge diagnosis) provided the most accurate identification of SI/SA identified via complete chart review (Table 2). There were, however, false positives (i.e., cases identified as SI/SA by the three chart elements that were not SI/SA-related according to the complete chart review) in 3% (n=96) of the total records and false negatives (i.e., the three chart elements did not identify SI/SA, but complete chart review did) in 5% (n=193) of the total records.

Table 1.

Concordance between identification of suicidal ideation or suicidal behavior via complete chart review and specific chart elements.

	Chart review indicates person is ideator OR attemptor
	No		Yes
	n	%	n	%
Individual chart elements
Presenting complaint is suicide related
No	2,401	97%	352	27%
Yes	72	3%	951	73%
Physician discharge diagnosis is suicide related
No	2,407	97%	595	46%
Yes	66	3%	708	54%
Discharge ICD code is suicide related
No	2,432	98%	976	75%
Yes	41	2%	327	25%
Combined chart elements
At least one of the three chart elements is suicide related
No	2,377	96%	193	15%
Yes	96	4%	1110	85%
Presenting complaint or Physician discharge diagnosis is suicide related
No	2,384	96%	215	17%
Yes	89	4%	1088	83%
Presenting complaint or Discharge ICD code is suicide related
No	2,387	97%	295	23%
Yes	86	3%	1008	77%
Discharge ICD code or Physician discharge diagnosis is suicide related
No	2,392	97%	539	41%
Yes	81	3%	764	59%

Table 2.

Comparison of chart elements for identifying cases with suicide ideation or behavior

	Proportion screened positive	Sensitivity	Specificity	PPV	NPV	Kappa
Individual chart elements
Presenting complaint; suicide-related	.27	73%	97%	93%	87%	0.74
Physician discharge diagnosis; suicide-related	.20	54%	97%	91%	80%	0.57
Discharge ICD code; suicide related*	.10	25%	98%	89%	71%	0.28
Combined chart elements
At least 1 of 3	.32	85%	96%	92%	92%	0.83
Presenting complaint or physician discharge diagnosis	.31	83%	96%	92%	92%	0.82
Presenting complaint or discharge ICD code	.31	77%	97%	92%	89%	0.77
Discharge ICD code or physician discharge diagnosis	.22	59%	96%	90%	82%	0.60

Discussion

As patient data become more readily available through the widespread use of EHR systems, there is an opportunity for clinicians and researchers to develop cost- and time-efficient means of identifying specific patient sub-populations. This is particularly relevant for high-risk patients like those at-risk for suicide, who may benefit from early detection and ongoing monitoring. The current study addresses an important research gap, namely whether specific chart elements can be used to identify SI/SA cases within the larger medical record. Similar to previous findings (e.g., Anderson et al., 2015), data from ICD-9-CM codes were poor indicators of SI/SA-related visits. Focusing on specific chart elements, most SI/SA cases (73%) could have been identified using free-text from the presenting complaint. However, when compared to the complete chart review, a combination of the three chart elements (ICD-9-CM codes, free-text from the presenting complaint and physician discharge diagnosis) identified the largest proportion of SI/SA visits (85%).

Studies using administrative data often rely on ICD codes (Kim, 2012), but as our findings show, ICD codes (including E and V codes) have low sensitivity and low negative predictive value when it comes to identifying suicidal ideation and behavior. This is particularly concerning as research relying on this type of administrative data may be significantly underestimating the prevalence of suicidal ideation or behavior in healthcare settings like the ED and there is no reason to believe this will improve with the implementation of ICD-10-CM.

Although our findings show that suicide-related diagnosis codes were underutilized in our patient population, we did find that this information is being documented in the presenting complaint and physician discharge diagnosis. Our findings add emphasis to the utility of focusing on presenting complaint and discharge diagnosis free-text in the detection of suicide-related visits. Automated processes using natural language processing (NLP) could search for relevant terms in the entire medical record (e.g., within free-text clinical notes), yet extracting all free-text data from an entire encounter in a HIPAA-compliant fashion can be challenging (Haerian, 2012; Liao et al., 2015). Our study lends additional support to evidence that free-text presenting complaints and discharge diagnoses can be used to identify a large proportion of SI/SA visits (Kuramoto et al., 2017). Further, the addition of ICD codes to the chart elements grouping only added an additional 2% (22 cases) beyond what presenting complaint and discharge diagnosis could detect. These results, combined with a careful study of false positive and false negative cases, may be beneficial for fine-tuning NLP algorithms or other machine learning processes directed at identifying SI/SA visits through EHR data.

In addition to tracking the acute and chronic conditions of patients, the utility of information gathered from the EHR has the potential to improve care by informing the clinical care process. Informatics tools are being integrated into primary care practices for treatment of medical health conditions (e.g., Kessler et al., 2017), such as decision support functionality built into EHR systems. This information could be used to generate alerts for a primary care provider, nurse manager, or mental health care professional. For example, alerts might warn them, in real time, of the occurrence of suicide-related ED visits so that they may promptly and appropriately follow up on these visits.

Limitations

For the purposes of this study, we were interested in investigating the utility of diagnosis codes as well as presenting complaint and discharge diagnosis free-text for identifying hospital records with documentation of suicidal ideation or behavior. Even though use of ED visits may limit the generalizability of findings to other clinical settings, the data were appropriate for our study goals and do provide information directly relevant to the clinical population of interest.

These data provided a unique opportunity to investigate the use of various chart elements for identifying suicidality, but findings from the free-text documentation should be interpreted cautiously. Due to the limitations of the data collected during the parent study, we were unable to examine a wider range of free-text data fields such as past medical history or problem list that are part of many EHRs. In addition, all of these sites were involved in a clinical trial (ED-SAFE) to improve screening for suicide in the ED. Although none of the ED-SAFE study procedures specifically focused on improving documentation in clinical text, simply increasing awareness regarding documentation for SI/SA may have resulted in increased mention of SI/SA in the presenting complaint and discharge diagnosis.

Conclusion

The large quantity of data available in the EHR presents a unique potential for identifying and monitoring suicide risk. Expanding the use of EHR data beyond ICD-9/10-CM codes has important implications including the development of automated methods to detect suicide ideation, behavior, and attempts in medical records. The current study provides valuable information on which types of chart data may be most useful for identification of SI/SA cases.