Evaluation of Data Sources for Carbon Monoxide Poisoning Surveillance in New York City

Supplemental Digital Content is Available in the Text.

Abstract

Context:

Carbon monoxide (CO) exposure can be life-threatening. Suspected and confirmed cases of CO poisoning warranting health care in New York City (NYC) are reportable to the NYC Poison Control Center (PCC).

Objectives:

We evaluated 4 hospital-based sources of CO surveillance data to identify ways to improve data capture and reporting.

Design:

Suspected and confirmed CO poisoning records from October 2015 through December 2016 were collected from the NYC emergency department (ED) syndromic surveillance system, New York State Statewide Planning and Research Cooperative System (SPARCS) ED billing data, NYC PCC calls made from hospitals, and the Electronic Clinical Laboratory Reporting System (ECLRS). Syndromic and SPARCS records were person- and visit-matched. SPARCS and ECLRS records were also matched to PCC records on combinations of name, demographic characteristics, and visit information.

Setting:

Hospitals in NYC.

Participants:

Individuals who visited NYC hospitals for CO-related health effects.

Main Outcome Measures:

We assessed the validity of syndromic data, with SPARCS records as the gold standard. We matched SPARCS and ECLRS records to PCC records to analyze reporting rates by case characteristics.

Results:

The sensitivity of syndromic surveillance was 60% (225 true-positives detected among 372 visit-matched SPARCS cases), and positive predictive value was 46%. Syndromic records often missed CO flags because of a nonspecific or absent International Classification of Diseases code in the diagnosis field. Only 15% of 428 SPARCS records (total includes 56 records not visit-matched to syndromic) and 16% of 199 ECLRS records were reported to PCC, with male sex and younger age associated with higher reporting.

Conclusions:

Mandatory reporting makes PCC useful for tracking CO poisoning in NYC, but incomplete reporting and challenges in distinguishing between confirmed and suspected cases limit its utility. Simultaneous tracking of the systems we evaluated can best reveal surveillance patterns.

Carbon monoxide (CO) is a colorless, odorless gas that binds to circulating hemoglobin in place of oxygen, thus depriving the body of natural oxygen uptake. It is a by-product of incomplete combustion from burning fuel during fires, cooking, heating, or operation of combustion engines, such as cars or generators. Use of combustion devices such as boilers, furnaces, and gas-powered generators—typically during winter-month power outages or extreme weather1–4—and improper ventilation or functioning of cooking or heating appliances5 are risk factors for exposure and the most likely sources of exposure in New York City (NYC).6 CO toxicity can cause headaches, nausea, dizziness, and death. Exposure to 100 ppm of this “silent killer” for 1 hour is usually sufficient to produce symptoms7; a 30-minute exposure to 1200 ppm or repeated 8-hour exposure to more than 50 ppm can also cause adverse health effects.8 CO detectors are important for reducing risk of CO poisoning in the home.9,10

Each year, the NYC Fire Department responds to thousands of CO exposures of more than 1 ppm, and emergency departments (EDs) experience hundreds of visits for potential poisoning.11,12 The NYC Health Code requires health care providers and laboratories to report suspected, probable, and confirmed CO poisoning diagnoses and positive laboratory tests to the NYC Poison Control Center (PCC).13 The NYC Department of Health and Mental Hygiene (DOHMH) has shared access to data collected from NYC PCC calls in near real-time. Laboratory detection of carboxyhemoglobin (COHb), the molecule formed when CO binds to hemoglobin, in blood provides evidence of CO exposure and poisoning. Mild symptoms can appear when COHb reaches about 10%, which is thus used as the mandatory reportable test result level in NYC.8

We evaluated 4 sources of surveillance data for tracking suspected or confirmed CO poisoning in NYC, including (1) ED visits provided through Statewide Planning and Research Cooperative System (SPARCS) hospital billing data; (2) syndromic surveillance data from NYC EDs collected in near real-time by DOHMH; (3) records of calls made to the NYC PCC from NYC hospitals; and (4) COHb test results sent to DOHMH from the Electronic Clinical Laboratory Reporting System (ECLRS). None of these data sources have been evaluated for their ability to track CO health impacts in NYC, though evaluations from Maine and on the national scale have demonstrated the viability of using hospital and/or PCC data.14–17

This investigation aimed to (1) understand how well the 4 sources of surveillance data capture CO poisonings warranting health care in NYC; (2) evaluate underreporting of CO poisonings to PCC from the sources that are mandated to report; (3) examine PCC reporting by patient age, sex, length of stay, COHb test result, and hospital discharge disposition; and (4) validate and improve the CO syndrome definition used to identify potential cases in the syndromic surveillance system.

Methods

Record collection and matching

We assessed validity of a CO poisoning syndrome using an existing, internal data set that contained NYC ED data from both syndromic surveillance and SPARCS. SPARCS ED records were composed of those either treated and released from the ED or inpatients who came from the ED. ED records from syndromic and SPARCS were matched on patient age, sex, zip code of residence, admit date, admit time and Permanent Facility Identifier (PFI). Records had to match on all fields. The period of analysis was October 2015 through December 2016, providing the most recent data available across data sources and coinciding with hospital implementation of the International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) coding system (n = 4 325 808). Data from Department of Veterans Affairs (VA) hospitals were excluded from syndromic data because federal institutions do not submit data to SPARCS.

We applied the Centers for Disease Control and Prevention's National Environmental Public Health Tracking CO case definitions18 to records in SPARCS (ICD-10-CM code “T58” in principal or other diagnosis fields) but combined causes (eg, intentional, fire-related), because the matched syndromic data set did not distinguish cause. Syndromic chief complaint and diagnosis fields were searched for CO-related key words and both International Classification of Diseases, Ninth Revision (ICD-9) and International Classification of Diseases, Tenth Revision (ICD-10) diagnostic codes, respectively. The chief complaint field is usually patient-supplied, while the diagnosis field is provider-supplied after clinical assessment, though not always used. Although ICD-10 was in use during this time and the diagnosis field reflects a final diagnosis, this field is not a gold standard and ICD-9 codes could have been entered instead of ICD-10 codes. We used matched SPARCS records to identify missed or incorrectly flagged CO-related syndromic records, and we scanned these missed records for new key words to add to the syndromic inclusion and exclusion criteria.

CO-related records in PCC data were identified by substance code “0106000” or “carbon monoxide” text search and limited to calls placed by non-VA hospitals within NYC. Limiting to hospital calls facilitated matching to records from SPARCS. CO cases that were transferred to a second hospital were assigned records for both their initial and final point of care to improve chances of finding a match; hospital-based data sources contain records for both visits (see Figure 1, Supplemental Digital Content 1, available at http://links.lww.com/JPHMP/A848, PCC record collection process).

The ECLRS data set included records with a COHb test result of 10% or more. Duplicate records and tests taken within 7 days of the most recent test for the same patient (defined by the same first name, last name, and date of birth) were excluded, although 1 person could be exposed multiple times across the study period. We eliminated records from VA hospitals or hospitals outside of NYC (see Figure 2, Supplemental Digital Content 2, available at http://links.lww.com/JPHMP/A849, ECLRS record collection process).

The complete set of CO-related records from SPARCS—regardless of a match with syndromic—and ECLRS data were each matched to CO-related PCC records on age, sex, and date of visit (±2 days). Optional matching criteria included at least the first 2 letters of a patient's first and last names and PFI. ECLRS does not provide standardized PFI codes, but text searches of facility names were used to identify affiliated hospital groups that could be matched. When 2 or more records from one data set matched the same record in another data set, a best match was determined on the basis of the number and quality of the matching fields (see Figure 3, Supplemental Digital Content 3, available at http://links.lww.com/JPHMP/A850, hierarchical decision tree for record matching).

Analysis

Syndromic data to track CO poisoning were evaluated for sensitivity, specificity, and positive predictive value (PPV) using SPARCS as the gold standard. We calculated measures of PCC reporting completeness associated with SPARCS and ECLRS (COHb ≥10%). Syndromic was not assessed for PCC reporting completeness because not all CO-related records in this data set contain final clinical diagnoses. We assessed differences in both SPARCS and ECLRS cases reported to PCC by age, sex, length of stay, discharge disposition, and COHb test result using a chi-square test or Fisher's exact tests (where computationally feasible). This study protocol has been reviewed and approved by an institutional review board (IRB). Ethical approval for this research was obtained from the NYC DOHMH on November 15, 2019 (IRB protocol # 19-100).

Results

Aim 1

We identified 631 records flagged for CO in either syndromic or SPARCS data (n = 484 and 372, respectively) among records that matched perfectly on personally identifiable information and date of visit (Table 1). Using SPARCS as the gold standard, syndromic sensitivity was 60%, specificity was nearly 100%, and PPV was 46%. The most common reason for the syndromic case definition not detecting CO exposure or poisoning was a nonspecific or absent ICD code in the diagnosis field (see Table 4, Supplemental Digital Content 4, available at http://links.lww.com/JPHMP/A851, common ICD code pairings)

TABLE 1. Validity of Carbon Monoxide-related Syndromic ED Records to SPARCS ED Records, October 2015-December 2016.

	SPARCS		Total
	Yes	No
Syndromic
Yes	225	259	484
No	147	4 325 177	4 325 324
Total	372	4 325 436	4 325 808

Abbreviation: SPARCS, Statewide Planning and Research Cooperative System.

Aim 2

There were 422 calls made to PCC from hospitals, either by providers or their associated laboratories. An additional 56 CO-related records were identified in SPARCS that did not previously match to syndromic data (n = 428). Relatively few (15%) of all CO cases captured in SPARCS were reported to PCC (Table 2). Among COHb tests with results of 10% or more (n = 199), 16% were reported to PCC (Table 3). Use of the 12% cut point for surveillance would reduce the COHb data set to 126 records, of which 27 (21%) were reported to PCC.

TABLE 2. Analysis of Records Involving Carbon Monoxide Captured in SPARCS and Reported to PCC by Providers or Laboratories, October 2015-December 2016.

	PCC		Total
	Yes	No
SPARCS
Yes	64 (15%)	364 (85%)	428 (100%)
No	358	...	358
Total	422	364	786

Abbreviations: PCC, Poison Control Center; SPARCS, Statewide Planning and Research Cooperative System.

TABLE 3. Analysis of ECLRS Carboxyhemoglobin Results 10% or Higher Reported to PCC by Providers or Laboratories, October 2015-December 2016.

	PCC		Total
	Yes	No
ECLRS
Yes	32 (16%)	167 (84%)	199 (100%)
No	390	...	390
Total	422	167	589

Abbreviations: ECLRS, Electronic Clinical Laboratory Reporting System; PCC, Poison Control Center.

Aim 3

Analysis of reporting to PCC by patient demographics suggested a higher rate among those younger than 18 years and a lower rate among those aged 30 to 49 years in both ECLRS (P < .01) and SPARCS (P = .04) (see Table 5, Supplemental Digital Content 5, available at http://links.lww.com/JPHMP/A852, demographics of SPARCS records reported to PCC; Table 6, Supplemental Digital Content 6, available at http://links.lww.com/JPHMP/A853, demographics of ECLRS records reported to PCC). Male patients were reported more frequently as well (P = .001 and P = .18, respectively). Reporting rate did not differ by indicators of poisoning severity, including hospital stays lasting at least 1 day or discharge dispositions requiring follow-up care (P = .99 and P = .70, respectively), but higher COHb test results were reported more frequently (P = .09).

Aim 4

Key words added to inclusion criteria for the CO syndrome definition include the following: “inhail” with “co”; “monoxide”; “C02”; “tailpipe”; “tail pipe”; “exhaust pipe”; “exhaustpipe”; “monx”; “cohb”; and “cohgb.” The ICD-9 CO code (986) and Ecodes for toxic CO or gas exposure were also searched for in the diagnosis field to allow for delay in implementation of the ICD-10 system. Key words added to exclusion criteria include the following: “mexico”; “greco”; “carbonate”; “metacarbonol”; “copd”; “carbonic”; “bicarbon”; “co2 narcosis”; “c02 narcosis”; “co2 retention”; “c02 retention”; “code”; “tobacco”; “burn”; and any mention of tailpipe or exhaust pipe; and “pt co” or “patient co” without other mention of CO. These exclusion criteria were used because examination of the records clearly indicated the reason for erroneous flagging of the CO definition. “E986,” “Z986,” and “J986” were also added to the exclusion criteria as records with these codes were erroneously included because of the “986” term in the text search, although these terms will not need to be included in the future with improved adherence to the ICD-10 system. This combination of key words and exclusions added 25 and eliminated 20 records from the syndromic CO case count compared with the number of records generated by the syndromic definition that was previously in use. Syndromic sensitivity calculated using both chief complaint and diagnosis fields (60%) was higher than sensitivities calculated from both the diagnosis field alone (29%) and the chief complaint field alone (46%).

Discussion

Health Code–mandated reporting requirements are designed to make the NYC PCC the central repository for CO poisoning surveillance data. Near real-time availability of data along with accompanying information about the source of exposure makes it a valuable surveillance tool, with implications for development of intervention strategies. However, reporting from hospitals and laboratories was low. Syndromic surveillance and ECLRS data are also available in near real-time but have interpretational limitations. In addition, the 4 systems we examined capture data at different points along the care continuum, and changes in case status (suspected vs confirmed or ruled out) over time pose challenges for comparisons between systems.

Low syndromic sensitivity can be partially explained by nonspecific or absent key words. Providers may also use nonspecific diagnosis codes that correlate with symptoms of poisoning but do not explicitly mention CO, such as R51 (headache), or Z77 (general toxic exposure) (see Table 4, Supplemental Digital Content 4, available at http://links.lww.com/JPHMP/A851, common ICD code pairings). False-positives are also likely because not all records contain a final diagnosis ruling on CO exposure, making the syndromic system more appropriate for tracking suspected (not yet confirmed) cases. Accordingly, PPV was relatively low, although it is likely to be higher during winter months when the incidence of CO poisoning is higher.19

In a separate study of cardiovascular disease (CVD) syndromic surveillance, the chief complaint field only identified 33% of CVD cases defined by the diagnosis field—which gets closer to a final diagnosis.20 The diagnosis field is usually completed by providers after clinical assessment (although not used consistently), so we expected a syndrome definition that uses both chief complaint and diagnosis fields to have higher sensitivity than one using chief complaint alone. We observed this improvement, with sensitivity increasing from 46% to 60%.

Our estimates of syndromic sensitivity and PPV may be biased by syndromic-SPARCS matches that were missed. Because SPARCS data sets are defined by discharge year, a patient admitted in one year could have been discharged in the next. Discharges in the SPARCS 2017 data set were not available at the time of this analysis, so a patient with CO poisoning discharged in 2017 would not be included in our matched data set. At least 56 CO poisoning records were missed because of our requirement that syndromic records match perfectly to personal identifiers in SPARCS.

SPARCS was used as the gold standard for evaluating syndromic data because it reflects a completed clinical investigation. Thus, a CO-related diagnosis in SPARCS is more likely to reflect a true case. SPARCS data are also more standardized and complete across patients and hospitals. However, SPARCS is limited for surveillance purposes due to the several years of delay in release.

Elevated COHb levels can confirm CO poisoning, but selection of an appropriate cut point for surveillance can be challenging. Cigarette smoking can raise COHb levels,21,22 and a test may be administered even without suspected CO exposure,22 leading to false-positives at the lower end of detection. However, lower levels do not necessarily mean CO exposure has not occurred because levels decrease as time since exposure increases. Furthermore, the body clears CO more quickly if supplemental oxygen is administered.23 Nonsmokers who are exposed to CO may also develop symptoms of poisoning at lower COHb levels. The 2018 guidance from Council of State and Territorial Epidemiology (CSTE) defines a case as COHb of 12% or more in smokers or adults whose smoking status is unknown.22 We used a lower 10% cut point to be consistent with the NYC Health Code reporting level, but use of the 12% cut point for surveillance would increase the reporting rate by 5%. CSTE guidance defines “suspected” cases as COHb of 2.5% or more to take into account potential testing delay.22 Use of the 2.5% cut point for surveillance would add 9396 records to the COHb data set, of which only 20 were reported to PCC. An argument exists for tracking CO poisoning using a lower COHb level, but reporting rates would appear lower as a result.

The NYC Health Code does not specify who is mandated to report CO cases to PCC across hospitals, laboratories, and physician offices, which is one possible reason for underreporting. We are also unable to distinguish provider calls from laboratory calls in the PCC data without searching the notes field, so PCC records only indicate that care was sought and a report—regardless of a test result—was made. PCC records that did not match SPARCS records or COHb test results of 10% or more—or those that matched to COHb results of lower than 10%—could represent nondefinitive poisonings reported to PCC. The NYC Health Code asks for reporting of both suspected and confirmed CO cases, so inclusion of these cases is reasonable. Evidence for this type of reporting was confirmed in the PCC record disposition fields, which in some cases contained information that the CO exposure “was probably not responsible for effects” or “judged as nontoxic exposure.”

Even after limiting ECLRS and SPARCS data sets to records that were positive for CO, lack of standardized and consistent identifiers required allowing nonperfect matches with PCC. To identify whether further relaxing the matching criteria could provide more matches indicating a PCC report, PCC and SPARCS records were matched on original fields (date, sex, PFI, first 2 letters of the first and last names) plus an expanded age range of ±5 years. This modification resulted in just 2 additional perfect matches and 1 additional imperfect match, suggesting that more flexible match criteria could only marginally improve match rates.

The reasons for disparities in PCC reporting across patient groups are unclear. Children may receive more attention, or providers may be more cautious during care for children and thus more motivated to consult PCC. PCC consultation during the course of care can reduce the severity of poisoning,24 so we hypothesized that the urgency associated with severe cases would lead to more frequent PCC consultation. The data did not support this hypothesis.

We assessed the feasibility of combining PCC, ECLRS, and syndromic data to estimate overall burden of CO poisoning in NYC in near real-time but found that tracking only suspected cases is possible. The combined data cannot be used to track confirmed case counts because none of the component data sources contain standardized and reliable information on final diagnosis. For example, PCC records do not distinguish between reports of confirmed cases or consultative calls that are more likely to be associated with a suspected case. Challenges in interpreting lower COHb results have been discussed; thus, a level lower than 12% from the CSTE guidance, such as the 10% used here, is relevant for tracking suspected cases, especially when used simultaneously with other systems. Syndromic records may reflect only a suspected diagnosis early in the care continuum. The low PPV of syndromic data compared with SPARCS as the gold standard suggests that many syndromic records are more likely to be false-positive—or suspected—cases than confirmed.

Other administrative data sources might have been used to complement the ones we evaluated in this analysis.14–17 Data from CO poisoning–related emergency medical services and deaths are also important to track, although the latter counts are very low. While these sources may be useful for understanding circumstances of exposure and risk factors, they are less useful for understanding patterns over time and place.25

The data sources we evaluated (syndromic surveillance, hospital administrative billing records, reports to the NYC PCC and electronic COHb laboratory results) can be used in parallel to track CO exposure and poisoning in NYC. Combining data sources into one master data set is theoretically possible for estimating citywide burden of suspected cases, but confirmed cases cannot be tracked in this way because the syndromic and PCC data fields do not reliably or consistently reveal confirmed case status.

Implications for Policy & Practice

• The NYC PCC is the ideal central repository for CO poisoning surveillance data designed by Health Code–mandated reporting requirements, but reporting compliance is low. Raising awareness of the requirements—including who in the health care system is responsible for reporting, methods of reporting, and why it is useful—would improve the PCC data source.

• Communication back to providers about how surveillance data can be used to deepen understanding of CO poisoning hazards in NYC could also be helpful in improving provider buy-in.26