Evaluation of the National Electronic Injury Surveillance System – All injury program’s self-directed violence data, United States, 2018

Abstract

Introduction:

National estimates for nonfatal self-directed violence (SDV) presenting at EDs are calculated from the National Electronic Injury Surveillance System – All Injury Program (NEISS–AIP). In 2005, the Centers for Disease Control and Prevention and Consumer Product Safety Commission added several questions on patient characteristics and event circumstances for all intentional, nonfatal SDV captured in NEISS–AIP. In this study, we evaluated these additional questions along with the parent NEISS–AIP, which together is referred to as NEISS–AIP SDV for study purposes.

Methods:

We used a mixed methods design to evaluate the NEISS–AIP SDV as a surveillance system through an assessment of key system attributes. We reviewed data entry forms, the coding manual, and training materials to understand how the system functions. To identify strengths and weaknesses, we interviewed multiple key informants. Finally, we analyzed the NEISS–AIP SDV data from 2018—the most recent data year available—to assess data quality by examining the completeness of variables.

Results:

National estimates of SDV are calculated from NEISS–AIP SDV. Quality control activities suggest more than 99% of the cause and intent variables were coded consistently with the open text field that captures the medical chart narrative. Many SDV variables have open-ended response options, making them difficult to efficiently analyze.

Conclusions:

NEISS–AIP SDV provides the opportunity to describe systematically collected risk factors and characteristics associated with nonfatal SDV that are not regularly available through other data sources. With some modifications to data fields and yearly analysis of the additional SDV questions, NEISS–AIP SDV can be a valuable tool for informing suicide prevention.

Practical Applications:

NEISS-AIP may consider updating the SDV questions and responses and analyzing SDV data on a regular basis. Findings from analyses of the SDV data may lead to improvements in ED care.

1. Introduction

Every year in the United States nearly 50,000 people die due to suicide and nearly 500,000 present in emergency departments (EDs) for nonfatal self-directed violence (SDV) (also referred to as self-harm or self-inflicted injuries; Centers for Disease Control and Prevention National Centers for Injury Prevention and Control, 2020). This public health problem is worsening, as the age-adjusted rate of suicides and nonfatal SDV increased by 33% and 40%, respectively, between 2001 and 2018 (Centers for Disease Control and Prevention National Centers for Injury Prevention and Control, 2020). In 2018, suicide was the tenth leading cause of death in the United States (Centers for Disease Control and Prevention National Centers for Injury Prevention and Control, 2020). The direct and indirect costs for suicides and suicide attempts in the United States was estimated at $93.5 billion in 2013 (Shepard et al., 2016).

National estimates for nonfatal SDV presenting at EDs are calculated from the National Electronic Injury Surveillance System – All Injury Program (NEISS–AIP). NEISS–AIP is a collaboration between the U.S. Consumer Product Safety Commission (CPSC) and the U.S. Centers for Disease Control and Prevention (CDC) with the purpose of tracking first-time, nonfatal injury-related ED visits on all types and causes of injuries; deaths are excluded from the surveillance system. NEISS–AIP is a nationally-representative sample of 24-hour EDs with at least six beds. NEISS–AIP included 66 EDs when it started in 2000 and has decreased over time as more hospitals have dropped out than have been replaced; the 2018 sample included 59 hospitals.

CPSC and CDC train hospital coders to review ED medical records and abstract the necessary information on all injuries. CPSC manages and cleans the database with support from CDC. Select NEISS–AIP variables are available for querying through the WISQARS^™ website (https://www.cdc.gov/injury/wisqars/index.html) within one year. Within 3–4 years the public version of the NEISS–AIP dataset can be accessed free through the Inter-University Consortium for Political and Social Research website (https://www.icpsr.umich.edu/web/ICPSR/series/198/studies) (Fig. 1).

Fig. 1.

National Electronic Injury Surveillance System – All Injury Program data flow.

In 2005, CDC and CPSC added several questions on patient characteristics and event circumstances for all intentional, nonfatal SDV cases captured in NEISS–AIP. The additional questions along with the parent NEISS–AIP constitute the NEISS–AIP SDV surveillance system for purposes of this study.

While aspects of NEISS–AIP and its other special studies have been evaluated in the past (Davis, Annest, Powell, & Mercy, 1996; Jhung et al., 2007; Thompson, Wheeler, Shi, Smith, & Xiang, 2014), NEISS–AIP SDV data have not been evaluated, so little is known about their usefulness. In this project, we evaluated NEISS–AIP SDV in terms of overall quality and utility.

2. Methods

CDC defines a surveillance system as “the ongoing, systematic collection, analysis, interpretation, and dissemination of data regarding a health-related event.” These data are then used to inform prevention efforts “to reduce morbidity and mortality and to improve health” (Buehler, 1998; German, Horan, Lee, Milstein, & Pertowski, 2001; Teutsch & Thacker, 1995; Thacker, 2000). We used a mixed methods design to evaluate the NEISS–AIP SDV as a surveillance system through an assessment of 10 system attributes: usefulness, simplicity, flexibility, data quality, acceptability, sensitivity, predictive value positive, representativeness, timeliness, and stability (Table 1).

Table 1.

Definitions of surveillance system attributes (German et al., 2001).

Attribute	Definition
Usefulness	A public health surveillance system is useful if it contributes to the prevention and control of adverse health-related events, including an improved understanding of the public health implications of such events
Simplicity	The simplicity of a public health surveillance system refers to both its structure and ease of operation. Surveillance systems should be as simple as possible while still meeting their objectives
Flexibility	A flexible public health surveillance system can adapt to changing information needs or operating conditions with little additional time, personnel, or allocated funds
Data Quality	Data quality reflects the completeness and validity of the data recorded in the public health surveillance system.
Acceptability	Acceptability reflects the willingness of persons and organizations to participate in the surveillance system.
Sensitivity	Sensitivity refers to the proportion of cases of a disease (or other health-related event) detected by the surveillance system (Weinstein & Fineberg)
Predictive Value Positive	Predictive value positive (PVP) is the proportion of reported cases that actually have the health-related event under surveillance (Weinstein & Fineberg)
Representativeness	A public health surveillance system that is representative accurately describes the occurrence of a health-related event over time and its distribution in the population by place and person
Timeliness	Timeliness reflects the speed between steps in a public health surveillance system
Stability	Stability refers to the reliability (i.e., the ability to collect, manage, and provide data properly without failure) and availability (i.e., the ability to be operational when it is needed) of the public health surveillance system

We reviewed data entry forms, the coding manual, and training materials to understand how the system functions. To identify strengths and weaknesses of the surveillance system attributes, we interviewed multiple key informants, including CDC users, CPSC managers, and hospital and quality assurance coders of the data. Finally, we analyzed the NEISS–AIP SDV data from 2018—the most recent data year available—to assess data quality by examining the completeness of variables; specific variables included time of arrival at the ED, patient self-reported SDV intent (e.g., intent to die, intent to harm oneself, intent to escape), staff description/diagnosis, patient SDV risk factors (e.g., previous episodes of self-harm, depression, bipolar disorder, anxiety), use of alcohol at time of injury, use of recreational drugs at time of injury, substances used (if poisoning), and final disposition (if admitted or transferred).

3. Results

3.1. Attributes

NEISS–AIP SDV allows for the calculation of national estimates of SDV. In addition, the system captures SDV-related variables that are not regularly available through other data sources, such as the Healthcare Cost and Utilization Product – Nationwide Emergency Department Sample. Compared to surveillance systems that rely on administrative codes alone, this system relies on medical record review, and, as such, might capture more cases. One study found that SDV-related administrative codes are frequently not recorded because, in part, they tend to not be billable; as a result, SDV events would be undercounted even though often there is enough information in the medical record to identify the SDV (Stanley et al., 2018). Sensitivity and predictive value positive were difficult to assess due to a lack of a gold standard for comparison. However, quality control activities suggest more than 99% of the cause and intent variables were coded consistently with the open text field that captures the medical chart narrative. In addition, hospital reporting to CPSC is timely as it occurs within a week of the ED visit, but data are not usually analyzed until after the calendar year’s data have been cleaned and final weights have been assigned. Cleaning is completed about a year after data collection, which limits the ability to identify real-time changes in SDV-related trends. Findings from other system attributes can be found in Table 2.

Table 2.

Findings from the evaluation of attributes of the National Electronic Injury Surveillance System – All Injury Program Self-directed Violence surveillance system.

Attribute	Strengths	Weaknesses
Usefulness	Provides annual national SDV estimates. Captures SDV-related variables (e.g., risk factors, poisoning substances) that are not regularly available through other data sources.	Open-ended responses used for many SDV-related variables make analysis time-consuming.
Simplicity	Relatively few people involved in the data entry (generally just 1–2 coders per hospital). Overall system data flow appears straightforward.	NEISS-AIP (and by extension NEISS-AIP SDV) is not integrated with other data systems. Data are manually abstracted from medical records.
Flexibility	Relatively simple to add additional questions and responses onto existing NEISS-AIP structure, as evidenced by multiple additions that became effective in 2019.	Changes require approval by multiple departments at CPSC, updates to the electronic data abstraction forms and new training for staff, all of which require funds as well.
Data Quality	<1% of injuries had contradictory information in other dataset fields.	Certain variables have many observations with no information because it is unknown or missing from the ED medical records.
Acceptability	Hospital data are submitted complete and within seven days of ED visit, on average.	~1–3 hospitals drop out per year and need to be replaced (decline from 66 hospitals in 2000 to 59 in 2018).
Sensitivity	Not possible to assess because medical records were not available. Cases are captured through medical record review, which might identify more cases than would be captured by administrative codes alone.
Predictive Value Positive	Not possible to assess because medical records were not available. <1% of injuries classified as SDV had contradictory information in other dataset fields.
Representativeness	Representative of U.S. hospitals having 24-hour EDs and a minimum of 6 inpatient beds that serve the general population (excludes Department of Veterans Affairs hospitals and special-purpose hospitals (e.g., correctional facilities, psychiatric-only hospitals)).	Does not capture patients who do not seek medical treatment, those treated in physicians’ offices or urgent care facilities and those treated in hospitals excluded from the sampling frame.
Timeliness	Hospital data are submitted within seven days of ED visit, on average.	Weighted data cannot be queried until approximately 1 year after data collection. Public dataset not available until 3–4 years after data collection.
Stability	CPSC’s data reporting system is reliable. CDC’s WISQARS™ is reliably available for querying.	Laptop problems and turn-over of hospital coders can lead to data entry delays.

CPSC = Consumer Product Safety Commission.

ED = Emergency department.

NEISS–AIP = National Electronic Injury Surveillance System – All Injury Program.

SDV = Self-directed violence.

3.2. Data quality (Completeness)

In 2018, NEISS–AIP SDV recorded 8,752 unweighted cases treated in EDs for nonfatal SDV injuries. Some variables (e.g., sex, age) do not have missing or unknown values but others (e.g., race, location where injury occurred, use of alcohol, use of recreational drugs, blood alcohol concentration (BAC)) have unknown values for more than 20% of observations (Table 3). Some variables only offer open-ended responses (e.g., BAC, poisoning substances and their respective quantities). A few variables (e.g., patient risk factors, BAC, poisoning substances and their respective quantities, patient disposition at ED discharge) have large numbers of missing data because responses are not required.

Table 3.

Description, type and evaluation findings of select 2018 National Electronic Injury Surveillance System – All Injury Program Self-directed Violence surveillance variables (8752 observations).

Variable Description	Variable Type	Findings
Age (in years)	Numeric	100% of observations have age.
Sex	Multiple Choice	100% of observations have sex.
Race	Multiple Choice	6572 (75%) of observations have race.
Location where injury occurred	Multiple Choice	5556 (63%) of observations have location.
Time of arrival to ED	Numeric	8708 (99%) of observations have time of arrival.
How did the patient describe his/her intent to the staff, other people, or in a (suicide) note?	Multiple Choice	100% of observations have patient-described intent.
Other description of intent	Open-ended	501 (6%) of observations have “other” descriptions
How did the staff describe or diagnose the injury event (at the time of discharge)?	Multiple Choice	100% of observations have staff description of injury.
Other staff description or diagnosis	Open-ended	837 (10%) of observations have “other” descriptions/diagnoses.
Depression	Checkbox	5379 (62%) of observations have this risk factor.
One or more previous episodes of self-harm	Checkbox	3235 (37%) of observations have this risk factor.
Anxiety, panic attacks, post-traumatic stress disorder	Checkbox	2048 (23%) of observations have this risk factor.
History of other substance(s) abuse	Checkbox	1091 (13%) of observations have this risk factor.
Other psychological/psychiatric problem, e.g., schizophrenia	Checkbox	1000 (11%) of observations have this risk factor.
Bipolar disorder	Checkbox	786 (9%) of observations have this risk factor.
History of alcohol abuse	Checkbox	697 (8%) of observations have this risk factor.
Borderline personality disorder	Checkbox	199 (2%) of observations have this risk factor.
Other specified risk factor(s) (e.g., argument with loved one, abuse or neglect, death of a loved one, illness, money or legal problems	Checkbox	3254 (37%) of observations have this risk factor.
Please specify the other risk	Open-ended	3254 (37%) of observations have a specific “other” risk factor.
Was alcohol used by the patient at the time of the injury event?	Multiple Choice	6753 (77%) of observations have information related to alcohol use.
Blood alcohol concentration (BAC) level	Open-ended	2042 (23%) of observations have BAC levels.
Were recreational drugs (e.g., cocaine, heroin, marijuana, ecstasy) used by the patient at the time of the injury event?	Multiple Choice	6580 (75%) of observations have information related to recreational drug use.
If the self-harm method was poisoning, please record up to four medications, drugs or substances taken by the patient. (4 “Substance” variables)	Open-ended	6123 (70%) of observations have “Substance 1”. 1881 (22%) of observations have “Substance 2”. 709 (8%) of observations have “Substance 3”. 283 (3%) of observations have “Substance 4”.
Amount substance taken (4 “Amount” variables)	Open-ended	6123 (70%) of observations have “Amount 1” (pertaining to “Substance 1”). 1881 (22%) of observations have “Amount 2” (pertaining to “Substance 2”). 709 (8%) of observations have “Amount 3” (pertaining to “Substance 3”). 283 (3%) of observations have “Amount 4” (pertaining to “Substance 4”).
If the patient was admitted or transferred, please specify where s/he went	Multiple Choice	6453 (74%) of observations have information on patient disposition.

4. Discussion

NEISS–AIP SDV provides the opportunity to describe systematically collected risk factors and characteristics associated with nonfatal SDV that are not regularly available through other data sources, which, in turn, would be useful for prevention purposes. While this surveillance system has the potential to be useful, this evaluation suggests that there are challenges with many of its system attributes.

The NEISS–AIP SDV surveillance system attributes of simplicity and stability benefit from being a part of the larger NEISS–AIP surveillance system. Another system strength is the focus on medical record review to capture cases, which is likely more sensitive than if the system relied only on administrative codes. Despite these strengths, NEISS–AIP has its limitations, particularly because it is currently reliant on human resources to manually abstract information from ED medical records and then enter the data into the NEISS–AIP data collection system. CPSC is exploring machine learning to help automate data abstraction from electronic medical records, but currently NEISS–AIP is not integrated with other data systems like electronic medical records.

NEISS–AIP SDV has aspects that are timely, including data reporting from the hospitals to CPSC (within a week of the ED visit) and feedback from CPSC and CDC to hospitals flagging certain errors (within about a week). However, it takes nearly a year after the end of the calendar year for the weighted data to be available for internal use at CDC and for select variables to be available to the general public through WISQARS^™. Historically, the publicly available data set for NEISS–AIP was only available after 3–4 years. CDC is in the process of expediting the release of the public dataset that will allow for more timely analysis of the NEISS–AIP data by public health partners. The SDV data captured from the additional questions added in 2005 have not been and currently are not available to the public as we continue to evaluate their utility.

The usefulness of the NEISS–AIP SDV data requires further consideration due to a couple of system challenges. First, medical records that are incomplete or that do not require the same fields as NEISS–AIP SDV leads to unknown values being entered into NEISS–AIP SDV. For example, race is not always included in hospital ED records, which results in this variable frequently being missing and thus limiting the ability to look at associations between SDV and race.

In addition, many of the SDV-specific variables (e.g., patient self-reported intent, staff diagnosis, patient risk factors, BAC, poisoning substances and their respective quantities) use open text fields for large proportions of responses, making data entry time-consuming and data analysis difficult and inefficient. These variables should be examined to determine which questions and responses can be modified to reduce the amount of open-ended responses. For example, the poisoning substances variables could be displayed in a drop-down list by drug/substance class.

In addition to reducing the number of open-ended responses, there could be a review of which NEISS-AIP SDV variables should be maintained—either as is or with modifications—and which could be dropped because they are no longer relevant or because the data can be obtained elsewhere. For example, the NEISS–AIP SDV variable that captures staff description or diagnosis of the SDV injury is important to have a clinical assessment of self-harm intent but may need to be updated to avoid outdated terminology (e.g., suicide gesture) that has been in place since 2005. In addition, in 2019, NEISS–AIP added new variables to its core and modified the medical narrative field to collect more information regarding alcohol use, perhaps allowing for elimination of other alcohol use-related variables.

This evaluation was subject to at least two limitations. First, data were collected primarily through review of manuals and interviews with CPSC and CDC stakeholders. A standardized survey of all members of the NEISS–AIP surveillance team and a systematic review of medical records to validate the NEISS–AIP SDV data would have made this evaluation more robust. However, this was not possible due to time, financial, and planning constraints. Second, sensitivity and predictive value positive were difficult to assess due to a lack of a gold standard.

In summary, NEISS–AIP SDV is a unique surveillance system based on medical record review that collects SDV-related risk factors and characteristics that are not collected in other data sources. With some modifications to data fields and yearly analysis of the additional SDV questions, NEISS–AIP SDV can be a valuable tool for informing suicide prevention.

Biography

Daniel C. Ehlman ScD, MPH, is an Epidemic Intelligence Service (EIS) Officer assigned to the Division of Injury Prevention at CDC’s National Center for Injury Prevention and Control. His research interests include suicide and self-directed violence prevention. Dr. Ehlman has a Doctor of Science degree in Global Health Systems and Development from the Tulane University School of Public Health and Tropical Medicine.

Tadesse Haileyesus MS, is a Mathematical Statistician working in the National Center for Injury Prevention and Control at the CDC. He received a master’s degree in Statistics from the University of Georgia. He has authored and coauthored several scientific publications on a wide range of topics with a special focus on injury and violence prevention.

Robin Lee PhD, MPH, leads the Safety Promotion Team within CDC’s National Center for Injury Prevention and Control. Dr. Lee and her team are actively engaged in studying the best ways to prevent injuries and creating real-world solutions to keep people safe, healthy, and independent. Topics of interest include drowning and injuries that impact older adults. Dr. Lee has a Master of Public Health and a Doctorate in Epidemiology from the State University of New York at Albany. She has authored and coauthored numerous presentations and scientific publications and has received awards for her public health and volunteer service.

Michael F. Ballesteros PhD, is the Deputy Associate Director for Science of the Division of Injury Prevention, National Center for Injury Prevention and Control, CDC. His research interests include injury surveillance systems, unintentional injuries, and global health. Dr. Ballesteros received a PhD in Epidemiology and is a graduate of CDC’s Epidemic Intelligence Service (EIS) program.

Ellen Yard PhD, MPH, is an epidemiologist with the Suicide Prevention Team within CDC’s National Center for Injury Prevention and Control. The Suicide Prevention Team strives towards a vision of no lives lost to suicide; this is carried out through the use of data, science, and partnerships to identify and implement effective suicide prevention strategies to foster healthy and resilient communities across the United States. Dr. Yard has a Doctorate in Epidemiology from The Ohio State University. She has authored and coauthored numerous presentations and scientific publications in injury prevention.