Evaluation of the Components of the North Carolina Syndromic Surveillance System Heat Syndrome Case Definition

Abstract

Objectives:

To improve heat-related illness surveillance, we evaluated and refined North Carolina’s heat syndrome case definition.

Methods:

We analyzed North Carolina emergency department (ED) visits during 2012-2014. We evaluated the current heat syndrome case definition (ie, keywords in chief complaint/triage notes or International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] codes) and additional heat-related inclusion and exclusion keywords. We calculated the positive predictive value and sensitivity of keyword-identified ED visits and manually reviewed ED visits to identify true positives and false positives.

Results:

The current heat syndrome case definition identified 8928 ED visits; additional inclusion keywords identified another 598 ED visits. Of 4006 keyword-identified ED visits, 3216 (80.3%) were captured by 4 phrases: “heat ex” (n = 1674, 41.8%), “overheat” (n = 646, 16.1%), “too hot” (n = 594, 14.8%), and “heatstroke” (n = 302, 7.5%). Among the 267 ED visits identified by keyword only, a burn diagnosis or the following keywords resulted in a false-positive rate >95%: “burn,” “grease,” “liquid,” “oil,” “radiator,” “antifreeze,” “hot tub,” “hot spring,” and “sauna.” After applying the revised inclusion and exclusion criteria, we identified 9132 heat-related ED visits: 2157 by keyword only, 5493 by ICD-9-CM code only, and 1482 by both (sensitivity = 27.0%, positive predictive value = 40.7%). Cases identified by keywords were strongly correlated with cases identified by ICD-9-CM codes (rho = .94, P < .001).

Conclusions:

Revising the heat syndrome case definition through the use of additional inclusion and exclusion criteria substantially improved the accuracy of the surveillance system. Other jurisdictions may benefit from refining their heat syndrome case definition.

Adverse heat-related outcomes are a highly preventable yet substantial public health burden. According to estimates, extreme outdoor heat is associated with a greater number of illnesses and deaths than any other weather-related exposure.^1,2 Extreme heat can result in adverse health outcomes, including neurologic damage, multiorgan failure, and death.^3–5 Awareness of symptoms and seeking medical care can mitigate or prevent the effects of severe adverse outcomes and the corresponding sequela. Preventing adverse heat-related outcomes can be achieved through individual behavioral modification and community-level adaptation measures. Syndromic surveillance systems can be used throughout the warm season and during extreme heat events to monitor trends in near-real time, modify or adjust public health messaging, and activate interventions (eg, cooling centers). Retrospective analysis of data on heat-related emergency department (ED) visits can be used to anticipate future ED usage, identify high-risk subpopulations, and evaluate interventions.

Within syndromic surveillance systems, the heat syndrome case definition comprises a combination of keywords and, in some jurisdictions, International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnostic and/or external cause-of-injury codes (E-codes).^6,7 The case definition of these cases and the criteria and keywords used to identify heat syndrome ED visits may vary by jurisdiction,⁶ and only a few studies have evaluated the keywords used in heat syndrome case definitions. As a result, the effectiveness of using heat syndrome case definitions to identify adverse heat-related outcomes related to ambient outdoor heat is unclear.

To improve the surveillance of heat-related ED visits in North Carolina, we evaluated the current heat syndrome case definition used by the state’s syndromic surveillance system, the use of additional inclusion and exclusion keywords, and the use of heat-related ICD-9-CM codes.

Methods

In North Carolina, heat-related syndromic surveillance is conducted with the North Carolina Disease Event Tracking and Epidemiologic Collection Tool (NC DETECT). All civilian hospital-affiliated EDs submit data to this system twice daily, and the system captures >99% of visits to those EDs.⁸ Authorized users (eg, public health professionals) can access the system data through interactive reports and line listings via a web-based application. Heat syndrome keywords are extracted from the chief complaints and triage notes (hereinafter, ED admission notes) with Structured Query Language (SQL).⁹ The system also has 11 diagnosis code fields and 5 E-code fields from which heat syndrome ICD-9-CM codes are pulled.

We obtained unpublished data from NC DETECT on ED visits from January 1, 2012, to December 31, 2014. We used the current heat syndrome case definition, which comprised keywords and heat-related ICD-9-CM codes, with additional heat syndrome keywords (Table 1) to identify and extract records from the system (Figure 1A). We included visits with an ICD-9-CM code of 992 or 992.0-992.9 (effects of heat and light) in any diagnosis code field and/or one of the following E-codes in any E-code field: E900, E900.0 (excessive heat due to weather conditions), E900.1 (excessive heat of manmade origin), or E900.9 (excessive heat of unspecified origin). We performed this work under contract to the North Carolina Division of Public Health for improving surveillance of heat-related illness; as such, institutional review board approval was not required.

Table 1.

Emergency department visits identified by heat syndrome case definition keywords, by keyword and the presence of a heat-related ICD-9-CM code, for all civilian hospital-affiliated emergency departments in North Carolina, January 1, 2012, to December 31, 2014^a

	Records, No. (%)
Transact-SQL Keywordb	Totalc	Presence of ICD-9-CM Coded
Current keywordse
heat exhaust* &! “explod*”	1028 (33.8)	503 (48.9)
overheat* | “over heat*”	646 (21.3)	181 (28.0)
“to hot” | “too hot”	594 (19.6)	76 (12.8)
heatstroke | “heat stroke”	302 (9.9)	110 (36.4)
“heat cramp*”	202 (6.7)	134 (66.3)
“heat related”	193 (6.4)	93 (48.2)
“hypertherm*”	58 (1.9)	14 (24.1)
“heat problem*”	7 (0.2)	3 (42.9)
heat sick*	3 (0.1)	0 (0.0)
sunstroke | “sun stroke”	3 (0.1)	1 (33.3)
“heat weak*”	1 (0.0)	0 (0.0)
Total	3037 (100.0)	1115 (36.7)
Additional keywordse
heat ex* &! “explod*”	646 (66.7)	334 (51.7)
“sun poison*”	184 (19.0)	2 (1.1)
“hot car”	34 (3.5)	7 (20.6)
“hot sun”	26 (2.7)	4 (15.4)
“heat injury”	25 (2.6)	12 (48.0)
“sun str*” | sunstr*	14 (1.4)	0 (0.0)
“heat str*” | heatstr*	12 (1.2)	4 (33.3)
“due to heat”	12 (1.2)	1 (8.3)
“heat ill*”	6 (0.6)	2 (33.3)
“heat emerg*”	6 (0.6)	3 (50.0)
“heat syn*”	3 (0.3)	2 (66.7)
“heat realted”	1 (0.1)	0 (0.0)
“heat prostration”	0 (0.0)	0 (0.0)
Total	969 (100.0)	371 (38.3)

Abbreviation: ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification.

^aData source: unpublished data, North Carolina Disease Event Tracking and Epidemiologic Collection Tool (NC DETECT).

^bIf a phrase is enclosed in quotes, the words had to be separated by a single space or a hyphen to be included.

^cPercentages may not total to 100 because of rounding.

^dHeat-related ICD-9-CM codes included 992, 992.0-992.9, E900, E900.0, E900.1, and E900.9.⁷

^e Current refers to keywords included in the heat syndrome case definition at the time of the study. Additional refers to keywords evaluated during the process of refining the heat syndrome definition. These keywords were not included in the current heat syndrome definition. SQL symbols refer to the following: * = wildcard, which means that any alphanumeric characters can follow (eg, “heat cramp*” will pick up “heat cramp” or “heat cramps”); &! = and not.

Figure 1.

Flowchart summarizing evaluation of the heat-related syndromic surveillance case definition for all civilian hospital-affiliated emergency departments (EDs) in North Carolina, January 1, 2012, to December 31, 2014: A, case identification; B, validation. Current refers to keywords included in the heat syndrome case definition at the time of the study. Additional refers to keywords evaluated during the process of refining the heat syndrome definition. Data source: unpublished data, North Carolina Disease Event Tracking and Epidemiologic Collection Tool (NC DETECT). E900.1 = E-code for excessive heat of manmade origins. Abbreviation: ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification.⁷

We evaluated the heat syndrome case definition to determine the number of ED visits identified by the case definition that were either not heat related or not related to ambient outdoor heat (Figure 1B). We classified heat syndrome ED visits as potential false positives if a burn diagnosis (ICD-9-CM code 940.0-949.9) was present or if the chief complaint was a burn. The chief complaint had to match exactly one of the following burn keywords: “BURN,” “burn,” “Burn,” “burn(s),” “Burn(s),” “BURN(s),” “BURNS,” “burns,” “Burns,” “(BURN),” or “(burn).” We also classified visits as potential false positives if the ED admission notes contained a non–heat-related keyword (Table 2). We manually reviewed the ED admission note fields for all potential false-positive ED visits. Because code E900.1 (excessive heat of manmade origin) is often used in other jurisdictions and heat-related illness studies as an exclusion criterion, we also manually reviewed all records with the E900.1 code. During the manual review process, we classified as true positive those ED visits with information indicating that the exposure was outdoor heat (eg, “mowing lawn,” “soccer practice,” “sitting outside”) or the patient believed that it was heat related (eg, “I got too hot”). We classified as false positive those ED visits with information indicating that the incident was not heat related or not related to ambient outdoor heat. Finally, we classified as true heat-related cases those ED visits in which a heat-related ICD-9-CM diagnosis code or E-code was present (Figure 1B).

Table 2.

TP and FP heat-related ED visits identified by non–heat syndrome case definition keywords and heat-related ICD-9-CM codes,^a by identification method, for all civilian hospital-affiliated EDs in North Carolina, January 1, 2012, to December 31, 2014^b,c

	Visits by Identification Method, No. (%)
	ICD-9-CM Code Only			Current Keyword Onlyd			ICD-9-CM Code + Current Keyword			Additional Keyword Onlyd			ICD-9-CM Code + Additional Keyword
Keyword	Total	TP	FP	Total	TP	FP	Total	TP	FP	Total	TP	FP	Total	TP	FP
Reviewed ED visits	222	70 (32)	14 (6)	249	39 (16)	200 (80)	55	37 (67)	2 (4)	18	9 (50)	2 (11)	15	5 (33)	0 (0)
Burne	19	2 (11)	3 (16)	121	1 (1)	120 (99)	2	2 (100)	0 (0)	5	0 (0)	2 (40)	1	0 (0)	0 (0)
Radiator or antifreeze	0	NA	NA	10	0 (0)	10 (100)	0	NA	NA	0	NA	NA	0	NA	NA
Grease, liquid, or oil	4	0 (0)	3 (75)	131	1 (1)	129 (98)	0	NA	NA	1	0 (0)	0 (0)	0	NA	NA
Shower	10	4 (40)	4 (40)	15	3 (20)	8 (53)	1	0 (0)	1 (100)	1	1 (100)	0 (0)	0	NA	NA
Water	65	63 (97)	0 (0)	83	27 (33)	51 (61)	26	26 (100)	0 (0)	10	7 (70)	0 (0)	4	4 (100)	0 (0)
Smoke inhalation	2	0 (0)	0 (0)	106	0 (0)	105 (99)	1	1 (100)	0 (0)	0	NA	NA	0	NA	NA
Fire	11	1 (9)	0 (0)	17	7 (41)	9 (53)	10	10 (100)	0 (0)	3	2 (67)	1 (33)	1	1 (100)	0 (0)
Heater	1	0 (0)	0 (0)	13	1 (8)	12 (92)	0	NA	NA	0	NA	NA	0	NA	NA
Hot tub, sauna, or hot spring	6	0 (0)	6 (100)	2	0 (0)	2 (100)	1	0 (0)	1 (100)	0	NA	NA	0	NA	NA
E900.1f	114	0 (0)	4 (4)	0	NA	NA	18	0 (0)	2 (11)	0	NA	NA	9	0 (0)	0 (0)

Abbreviations: ED, emergency department; FP, false positive; ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification; NA, not applicable; TP, true positive.

^aHeat-related ICD-9-CM codes included 992, 992.0-992.9, E900, E900.0, E900.1, E900.9.⁷

^bColumn numbers are mutually exclusive. If a record was not categorized as a TP or FP, it was unclassifiable (column not shown). Row numbers are not mutually exclusive. Stratified results by keyword or diagnosis do not sum to the total number of ED visits because of overlapping categories.

^cData source: unpublished data, North Carolina Disease Event Tracking and Epidemiologic Collection Tool (NC DETECT).

^d Current refers to keywords included in the heat syndrome case definition at the time of the study. Additional refers to keywords evaluated during the process of refining the heat syndrome definition. These keywords were not included in the current heat syndrome definition.

^eIncludes “BURN,” “burn,” “Burn,” “burn(s),” “Burn(s),” “BURN(s),” “BURNS,” “burns,” “Burns,” “(BURN),” “(burn),” or an ICD-9-CM code of 940.0-949.9. ED visits with burn keyword, n = 53; ED visits with burn diagnosis code, n = 60; ED visits with both, n = 35.

^fThe ED visits in this category were identified by the ICD-9-CM code E900.1 in the E-code fields and not by keywords in the chief complaint/triage note fields.

We summarized the case counts for each keyword and ICD-9-CM code, and we used the ICD-9-CM code–identified cases as a gold standard when calculating positive predictive value (PPV) and sensitivity. We used SAS version 9.4, Perl regular expression, to extract individual keywords and phrases.¹⁰ To determine if the distribution of keyword-identified ED visits was similar to the distribution of heat-related ICD-9-CM code–identified ED visits, we graphed the time series (ie, the number of identified ED visit counts per day) for the 2 distributions. We standardized the time series by dividing each daily count by the total number of counts for that year and calculated the Pearson correlation coefficient and corresponding P value between the 2 standardized series.

Results

Current Heat Syndrome Case Definition

We identified 8928 heat-related ED visits using the current heat syndrome case definition. Of these visits, 1922 (21.5%) were identified by a current heat syndrome keyword only, 5891 (66.0%) by a heat-related ICD-9-CM code only, and 1115 (12.5%) by both methods. We pulled heat-related keywords from both ED admission note fields; however, 5959 (66.7%) heat syndrome–identified ED visits did not have a triage note, and 69 (0.8%) chief complaint fields were blank. We pulled heat-related ICD-9-CM codes from both the diagnosis and E-code fields. Of the 8928 ED visits identified by the current heat syndrome case definition, 439 (5.0%) were missing diagnosis code information, and 3294 (36.9%) did not have E-code information. Of the 6723 ED visits with an injury diagnosis (ICD-9-CM codes 800-999), 1582 (23.5%) were missing E-code information.

Current Heat Syndrome Case Definition Keywords

Four phrases captured 2570 of 3037 (84.6%) keyword-identified ED visits: “heat exhaust,” “overheat,” “too hot,” and “heatstroke.” We assumed records with a heat-related ICD-9-CM code to be true heat-related ED visits (Table 1). Because manual review could not be conducted on all ED visits identified by the current heat syndrome case definition, we manually reviewed 410 ED visits with a non–heat-related keyword or a burn-related ICD-9-CM code. We classified 16 ED visits with a heat-related ICD-9-CM code as false positive, 12 of which were heat related but not related to outdoor heat. Among heat-related ED visits identified only by a heat syndrome keyword, 95% to 100% of those that also had a burn-related ICD-9-CM code, a burn keyword, or the keyword “grease,” “liquid,” “oil,” or “smoke inhalation” were classified as false positive. Only 3 records were identified by “smoke inhalation” alone (Table 2).

Additional Heat Syndrome Keywords

Through the use of additional keywords, we captured another 598 heat-related ED visits during the 3-year study period. A total of 371 ED visits previously identified only by heat-related ICD-9-CM code had at least 1 of the additional keywords in the ED admission notes. Of the 969 ED visits identified by additional keywords, 646 (66.7%) were identified by “heat ex” and 184 (19.0%) by “sun poison.” Of 184 ED visits, 2 (1.1%) with the keyword “sun poison” had a heat-related ICD-9-CM code. The additional keywords with the largest proportion of ED visits with a corresponding heat-related ICD-9-CM code were “heat ex” (334 of 646, 51.7%), “heat injury” (12 of 25, 48.0%), “heat emerg” (3 of 6), and “heat syn” (2 of 3, Table 1).

We manually reviewed 18 ED visits that were identified by the additional heat syndrome keywords only (Table 2) and classified 9 as true positives, 2 as false positives, and 7 as unclassifiable because there was not enough information in the ED admission notes.

Heat Syndrome Diagnostic Codes

Of the 7006 ED visits with a heat-related ICD-9-CM code, 6466 (92.3%) had an ICD-9-CM diagnosis code of 992-992.9 (effects of heat and light). Most of the 7006 heat-related ICD-9-CM coded ED visits were diagnosed as heat exhaustion (n = 4533, 64.7%), whereas only 260 (3.7%) were diagnosed as heatstroke or sunstroke—the most severe form of heat-related illness⁴ (Table 3). Most of the 540 ED visits identified by a heat-related E-code were caused by weather (E900.0; n = 371, 68.7%) or unspecified heat (E900.9; n = 149, 27.6%).

Table 3.

Heat-syndrome ED visits identified by heat-related ICD-9-CM code,^a by diagnosis codes and E-codes, for all civilian hospital-affiliated EDs in North Carolina, January 1, 2012, to December 31, 2014^b

	Visits by Code, No. (%)
Description (ICD-9-CM Code)	No E-codec	E900.0c,d	E900.1e	E900.9f	E900g	Total
Heatstroke and sunstroke (992.0)	99 (5.6)	128 (3.4)	7 (5.0)	18 (1.4)	8 (8.2)	260 (3.7)
Heat exhaustion (992.3-992.5)	959 (54.4)	2572 (68.6)	92 (65.2)	853 (68.1)	57 (58.2)	4533 (64.7)
Heat syncope (992.1)	162 (9.2)	159 (4.2)	12 (8.5)	35 (2.8)	7 (7.1)	375 (5.4)
Heat fatigue (992.6)	28 (1.6)	31 (0.8)	0 (0.0)	7 (0.6)	6 (6.1)	72 (1.0)
Heat cramps (992.2)	121 (6.9)	229 (6.1)	7 (5.0)	79 (6.3)	4 (4.1)	440 (6.3)
Heat edema (992.7)	4 (0.2)	1 (0.0)	0 (0.0)	3 (0.2)	0 (0.0)	8 (0.1)
Other specified/unspecified heat effects (992.8, 992.9)	391 (22.2)	259 (6.9)	9 (6.4)	109 (8.7)	10 (10.2)	778 (11.1)
No heat-related diagnosis code	0 (0.0)	371 (9.9)	14 (9.9)	149 (11.9)	6 (6.1)	540 (7.7)
Total	1764 (100.0)	3750 (100.0)	141 (100.0)	1253 (100.0)	98 (100.0)	7006 (100.0)

Abbreviations: ED, emergency department; ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification.

^aIncludes all ED visits with a heat-related diagnostic code. The codes are listed from most severe (heatstroke) to least severe (heat edema). If a record had >1 heat-related illness diagnosis code listed, only the most severe diagnosis code was used.⁷

^bData source: unpublished data, North Carolina Disease Event Tracking and Epidemiologic Collection Tool (NC DETECT).

^cPercentages may not total to 100 because of rounding.

^dExcessive heat due to weather conditions.

^eExcessive heat of manmade origins.

^fExcessive heat of unspecified origins.

^gExcessive heat.

In a manual review of 141 ED visits with an E-code for excessive heat of manmade origins (E900.1), we classified 6 (4.3%) as false positives, all of which were heat related but not related to outdoor heat (eg, sauna or shower). Of the other 135 cases, 32 did not have any keywords, and the rest were unclassifiable. For 14 of the 141 ED visits, the only heat-related ICD-9-CM code was E900.1; of those, 2 were false positives and 12 were unclassifiable.

Performance Measures

We considered the presence of a heat-related ICD-9-CM code (ie, diagnosis/E-code) to be a true heat-related ED visit. Therefore, based on the cases identified by the current heat syndrome case definition keywords (ie, test positive), the sensitivity was 15.9% (n = 1115 of 7006), and the PPV was 36.7% (n = 1115 of 3037). The correlation between keyword-identified ED visits and ED visits identified by heat-related ICD-9-CM codes was high (rho = 0.92, P < .001). After examining the additional keywords, we determined that “sun poison” should not be an inclusion keyword. With the additional keywords, we identified 9342 (n = 8928 + 598 – 184) total heat-related ED visits during the study period. During our evaluation, we manually reviewed 559 ED visits (n = 410 + 18 + 141 – 10 ED visits that had both ICD-9-CM code E900.1 and a nonheat keyword). We classified 218 as false positives, 160 as true positives, and 181 as unclassifiable. We identified most of the false positives using the chief complaint “burn” keyword (n = 84 of 218, 38.5%) or the keyword “grease,” “liquid,” or “oil” (73 of 218, 33.5%). The current heat syndrome case definition includes only 1 exclusion criterion (ie, heat exhaust* &! “explode*”). We identified additional exclusion criteria: chief complaint “burn” keyword; burn diagnosis codes; or the keywords “grease,” “liquid,” “oil,” “radiator,” “antifreeze,” “hot tub,” “hot spring,” or “sauna.” Using these exclusion criteria removed 6 true-positive ED visits and retained 30 false-positive ED visits. Only 11 ED visits were excluded based solely on the presence of a burn diagnosis (ie, no other exclusion keyword present). With the additional keywords and exclusion criteria, the number of heat-related ED visits identified increased from 8928 to 9132. Of those 9132 ED visits, 2157 (23.6%) were identified by a heat syndrome keyword only, 5493 (60.2%) by a heat-related diagnostic code only, and 1482 (16.2%) by both methods.

Based on the additional keywords and exclusion criteria, both the sensitivity (n = 1482 of 6975, 21.2%) and the PPV (n = 1482 of 3639, 40.7%) improved. With the refined heat syndrome case definition, the correlation between the keyword-identified and diagnostic code–identified case time series improved slightly (rho = 0.94, P < .001; Figure 2). After completing the evaluation, we manually reviewed an additional 400 randomly sampled heat-related ED visits identified by the refined heat syndrome case definition and identified 82 true positives (34 by keyword only, 23 by ICD-9-CM code only, 25 by both), 7 false positives (4 by keyword only, 3 by ICD-9-CM code only), and 311 that were unclassifiable (48 by keyword only, 222 by ICD-9-CM code only, 41 by both). For 84 of the 311 unclassifiable ED visits (43 by keyword only, 41 by ICD-9-CM code only), the only information available was 1 of the following heat phrases: “heat exposure,” “heat exhaustion,” “heat stroke,” “heat injury,” “heat related illness,” or “overheated.”

Figure 2.

Emergency department (ED) visits identified with the revised heat syndrome case definition, by identification method (keyword or International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] code⁷) and date, for all civilian hospital-affiliated EDs in North Carolina, January 1, 2012, to December 31, 2014. Data source: unpublished data, North Carolina Disease Event Tracking and Epidemiologic Collection Tool (NC DETECT).

Warm Season Results

To provide results comparable with other jurisdictions, we also restricted our results to the warm season (May 1–September 31), a period often used for heat-related illness surveillance.¹¹ Using the current heat syndrome case definition, we identified 7967 heat-related ED visits. Of these visits, we identified 1520 (19.1%) by a current heat syndrome keyword only, 5373 (67.4%) by a heat-related ICD-9-CM code only, and 1074 (13.5%) by both. The sensitivity of ED visits identified by the current heat syndrome case definition keywords was 16.7% (n = 1074 of 6447), and the PPV was 41.4% (n = 1074 of 2594). Using the refined heat syndrome case definition, we identified 8258 heat-related ED visits. Of these visits, we identified 1831 (22.2%) by a current heat syndrome keyword only, 4995 (60.5%) by a heat-related ICD-9-CM code only, and 1432 (17.3%) by both. The sensitivity of ED visits identified by the refined heat syndrome case definition keywords was 22.3% (n = 1432 of 6427), and the PPV was 43.9% (n = 1432 of 3263).

Discussion

Syndromic surveillance systems are used globally to monitor adverse heat-related outcomes in near-real time. However, to our knowledge, only 3 published conference abstracts have evaluated the components of jurisdictions’ heat syndrome case definitions.^12–14

The North Carolina syndromic surveillance system uses ED admission notes (ie, chief complaints and triage notes) and ICD-9-CM diagnostic codes and E-codes to identify heat-related ED visits. ED admission notes are typically included in the initial records for ED visits, with ICD-9-CM codes coming later as record updates. As a result, near–real-time surveillance relies heavily on the text of ED admission notes. Timeliness of assigning ICD-9-CM codes varies by hospital from 48 hours to 3 months. Most hospitals send at least 1 diagnosis code for most ED visits within 2 weeks.¹⁵ Given the variability of ICD-9-CM code receipt, identifying as many true-positive (or excluding as many false-positive) heat-related ED visits as possible through keywords is important. Even with the variability in timeliness, the inclusion of ICD-9-CM codes substantially improves the sensitivity of heat-related illness definitions when compared with a keyword-only approach. When using the data for retrospective surveillance, we found that a combination of ICD-9-CM codes and keywords provided the best results.

From January 1, 2012, to December 31, 2014, approximately 34.0% of heat-related ED visits were identified by the current heat syndrome case definition keywords. Using additional keywords and exclusion criteria increased the proportion of heat-related ED visits identified by a keyword to 39.8%. In our study, the sensitivity and PPV of ED visits identified by heat syndrome case definition keywords were 15.9% and 36.7%, respectively. These results were similar to the sensitivity (16%) and PPV (40%) observed in an evaluation of the New Jersey heat-related syndromic surveillance case definition based on data from May through September 2009-2011.¹² Adding keywords and exclusion criteria to the North Carolina heat syndrome case definition improved sensitivity to 21.2% and PPV to 40.7%.

To evaluate the performance measures, we used heat-related ICD-9-CM codes to classify true heat-related ED visits. Of the records selected for manual review, we found only a small proportion of false positives among diagnostic code-identified ED visits. Therefore, records with a heat-related ICD-9-CM code were most likely true heat-related ED visits. However, it may be incorrect to assume that the presence of a heat-related ICD-9-CM code is the gold standard. The keywords and the ICD-9-CM codes are based on 2 sources of information. The ICD-9-CM codes are from the clinician’s notes after the ED visit is completed, whereas the ED admission notes are reported by the nurses and the patient at arrival to the ED or shortly thereafter. Exposure information may not be required for treatment and, as a result, may not be included in the clinician’s notes, resulting in a heat-related diagnostic code not being assigned. The patient or the nurse may provide or record situational information in ED admission notes. Conversely, nonspecific keywords in the ED admission note, such as “syncope” or “headache” (without mention of heat), may lead to heat-related cases not being captured. The 2 sources of information, therefore, may provide unique but complementary information.

We did not use the phrase “sun poisoning” as an inclusion term when refining the heat syndrome case definition. This phrase appeared to capture ED visits for sunburn and very few ED visits for adverse heat-related outcomes (eg, thermoregulatory system failure). Although adverse heat-related outcomes and sunburn may be strongly correlated, they have different etiologies and require different approaches for public health education, prevention, and response.¹⁶

Most ICD-9-CM code–identified ED visits were for heat exhaustion, and the most frequently identified external cause was heat due to weather conditions. Of the 16 ED visits with a heat-related ICD-9-CM code that were classified as false positives, 6 had an E-code for excessive heat of manmade origins (E900.1), which is often not included or is explicitly excluded from heat-related illness definitions.^11,17–19 In our manual review of ED visits with code E900.1, we found ED visits that described patients working in their yard or sleeping in a non–air-conditioned room in the summer, as well as those in which the patients “got too hot at work,” with no indication of where they were working. To avoid capturing false positives and excluding true positives, exclusion keywords or diagnosis codes (eg, burn diagnosis) should be included with the heat-related diagnostic codes rather than excluding E900.1.

Limitations

This study had several limitations. First, owing to the large number of ED visits, manual review of ED admission notes for all identified ED visits was not feasible because of limited staffing resources and time. As such, we could not determine the number of false-positive ED visits for which manual review was not conducted. Furthermore, in many of the records, the ED admission note was either missing or did not provide enough information to determine if the case was a true positive or a false positive. To potentially address this issue, future studies should conduct a medical record review.

Conclusions

Evaluating the heat syndrome case definition and identifying additional inclusion and exclusion criteria improved the accuracy of the heat syndrome case definition and, in turn, should improve surveillance of adverse heat-related outcomes in North Carolina. The similar pattern over time of heat-related cases identified by keywords alone vs diagnosis codes and E-codes alone indicates that the unavailability of diagnostic codes (eg, within 48 hours of ED visit) will not change public health decisions based on syndromic surveillance system peaks and trends. This evaluation process may be useful for other jurisdictions that conduct heat-related syndromic surveillance.