Abstract
Objectives:
In Colorado, legalization of recreational marijuana in 2014 increased public access to marijuana and might also have led to an increase in emergency department (ED) visits. We examined the validity of using syndromic surveillance data to detect marijuana-associated ED visits by comparing the performance of surveillance queries with physician-reviewed medical records.
Methods:
We developed queries of combinations of marijuana-specific International Classification of Diseases, Tenth Revision (ICD-10) diagnostic codes or keywords. We applied these queries to ED visit data submitted through the Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE) syndromic surveillance system at 3 hospitals during 2016-2017. One physician reviewed the medical records of ED visits identified by ≥1 query and calculated the positive predictive value (PPV) of each query. We defined cases of acute adverse effects of marijuana (AAEM) as determined by the ED provider’s clinical impression during the visit.
Results:
Of 44 942 total ED visits, ESSENCE queries detected 453 (1%) as potential AAEM cases; a review of 422 (93%) medical records identified 188 (45%) true AAEM cases. Queries using ICD-10 diagnostic codes or keywords in the triage note identified all true AAEM cases; PPV varied by hospital from 36% to 64%. Of the 188 true AAEM cases, 109 (58%) were among men and 178 (95%) reported intentional use of marijuana. Compared with noncases of AAEM, cases were significantly more likely to be among non-Colorado residents than among Colorado residents and were significantly more likely to report edible marijuana use rather than smoked marijuana use (P < .001).
Conclusions:
ICD-10 diagnostic codes and triage note keyword queries in ESSENCE, validated by medical record review, can be used to track ED visits for AAEM.
Keywords: marijuana, syndromic, surveillance, ICD-10
Legalization of retail recreational marijuana in Colorado in 2012 led to an increase in public access to marijuana.1 To monitor trends and adverse health effects of marijuana use, local and state public health agencies are required by Colorado statute to establish methods to conduct marijuana-associated surveillance.2,3 Surveillance systems used to measure the burden on hospitals, emergency departments (EDs),4 and poison control centers5-7 related to marijuana use are present in Colorado; however, access to and interpretation of data attributing adverse health outcomes to marijuana exposure poses challenges of data timeliness, quality, and completeness. The use of syndromic surveillance data, which provide near–real-time data availability,8-10 is appealing as an alternative surveillance system.
Tri-County Health Department (TCHD) is the largest local public health agency in Colorado, serving approximately 1.5 million residents in 3 large counties in the Denver, Colorado, metropolitan area. In 2012, TCHD began participating in the Centers for Disease Control and Prevention (CDC) National Syndromic Surveillance Program (NSSP). Information is transmitted to the NSSP data platform in real time from 26 Denver-area hospitals; this information includes selected data fields from the medical record, including age, sex, residential ZIP code, race, ethnicity, chief complaint, triage note, and International Classification of Diseases (ICD) diagnostic codes. The Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE) processes medical record data from participating hospital EDs.11 Queries can be built in ESSENCE to identify ED visits of interest based on any combination of filters to these data fields. In syndromic surveillance, frequencies of ED visits of interest are calculated as trends over time to develop a baseline, from which statistical aberrations can be calculated to detect an outbreak or unusual cluster of illness.
DeYoung et al12 in 2017 described a method for applying a marijuana case definition to syndromic surveillance data to detect ED visits potentially related to marijuana use in Denver. Instead of reviewing the original medical record to determine whether their case definition accurately identified ED visits due to marijuana use, the authors used a review of the abbreviated chief complaint and triage note data fields available in ESSENCE as the reference standard. The authors concluded that this approach yielded a positive predictive value (PPV) of >95%.12 However, because of heterogeneous data quality and completeness in ESSENCE, this method has been reproached in syndromic surveillance of other conditions.13-15 We hypothesized that physician review of the entire ED visit would result in a more complete and accurate reference standard. The objective of our study was to determine whether ED visits related to acute adverse effects of marijuana (AAEM) were accurately identified by marijuana queries created in ESSENCE; we used physician review of the ED visit medical record as the reference standard.
Methods
Development of Queries to Detect AAEM Cases
We developed syndromic surveillance queries of various combinations of marijuana-specific International Classification of Diseases, Tenth Revision (ICD-10)16 diagnostic codes or keywords from the triage note and chief complaint data fields to identify ED visits that might be due to AAEM. Each keyword also had at least 1 negation term to minimize the capture of data on irrelevant ED visits; for example, we applied the negation term denies marijuana to the keyword marijuana so that a medical record would not be identified as representing a suspected case of AAEM when the term marijuana was present only in the phrase denies marijuana. We based the selection of ICD-10 diagnostic codes or keywords on previous work in this field to accurately identify ED visits for AAEM.4 We applied these queries to all ED visit records available in ESSENCE at 3 hospitals in the TCHD jurisdiction during 4 consecutive months in 2016-2017. We included any ED visit in ESSENCE during this time period that was identified by at least 1 query.
Hospital Selection
We selected the 3 hospitals for this analysis to represent diverse population demographic characteristics at varying geographic locations within the TCHD jurisdiction. Hospital A was a large, urban subspecialty hospital with >600 inpatient beds, hospital B was a suburban community hospital with about 100 inpatient beds, and hospital C was a suburban community hospital with about 200 inpatient beds. We applied the queries in ESSENCE to hospitals A and B during June 1–September 30, 2016, and to hospital C during March 1–June 30, 2017. The period analyzed for hospital C was later because this hospital began transmitting data to ESSENCE in 2017 and attained consistent data stream quality beginning in March 2017.
AAEM Case Definition
We defined an ED visit identified by at least 1 marijuana query in ESSENCE as a suspected AAEM case. We then classified an ED visit as a true AAEM case when the medical record documented that the ED health care provider’s clinical impression indicated that marijuana use was likely responsible for the patient’s presenting symptom. We defined a noncase as a visit in which the ED provider did not document that the patient’s illness was due to marijuana exposure or likely due to marijuana exposure.
Record Abstraction
One physician author (G.E.M.) reviewed the original electronic medical records of ED visits identified by the proposed queries as suspected AAEM cases to determine whether that patient visit met the AAEM case definition. The author recorded patient age, sex, and residential ZIP code; route and intentionality of marijuana use; marijuana toxicology testing results (if performed); and documentation of polysubstance use at the time of the ED visit using a record abstraction tool in EpiInfo.17 For each ED visit, the author also recorded any marijuana-specific ICD-10 diagnostic codes T40.7 (poisoning by cannabis), F12.1 (cannabis abuse), F12.2 (cannabis dependence), and F12.9 (cannabis use, unspecified) or keyword(s) “marijuana,” “THC,” “MJ,” “edible,” “cannab,” “smok AND weed,” “smok AND pot,” or “pot AND brownie” present in the chief complaint or triage note fields.
Data Analysis
We developed a novel approach to assess query performance. For each query, we calculated the PPV by using the case definition assigned by physician record review as the reference standard by using this equation:
Because we reviewed only ED visits identified as suspected AAEM cases, rather than the whole population of ED records during the specified period, we could not calculate or report the sensitivity, specificity, and negative predictive values of the queries. We calculated the frequencies of ICD-10 diagnostic codes and keywords present in the records for suspected AAEM cases and stratified them by case status (ie, true case or noncase). We identified the best-performing query as the one with the most parsimonious combination of diagnostic codes and keywords while achieving the highest average PPV and proportion of suspected AAEM cases identified of the total number of suspected AAEM cases (n = 422). We used the following equation.
We compared data on patient sex, age, geographic home (Colorado state resident or nonresident), polysubstance use, intentionality of marijuana use (intentional or accidental), and route of marijuana use by case status. We used the Pearson χ2 test of independence to examine the relationship between these variables and AAEM case status. To identify which diagnostic codes and keywords were most associated with true AAEM cases, we compared the frequencies of ICD-10 diagnostic codes and triage note keywords between true cases and noncases. We conducted all analyses by using SAS/STAT version 9.4,18 with P < .05 considered significant.
Query Application
After the evaluation of query performance, we applied the best-performing query to data in ESSENCE during July 1–December 31, 2017, from the 3 hospitals. We took this additional step to demonstrate the real-time utility of these data at a local health department to determine trends in local ED visits potentially due to marijuana exposure. The Human Subjects Review board at CDC and the Colorado Multiple Institutional Review Board reviewed the study design before data collection and determined that the study did not meet criteria for human subject research and waived review.
Results
Determination of PPV
During the study period, all 3 hospital systems had >95% required data fields (including ICD-10 diagnostic codes, triage notes, and chief complaints) available in ESSENCE. During the study period, hospital A reported 28 719 ED visits, hospital B reported 7297 ED visits, and hospital C reported 8926 ED visits, for a total of 44 942 ED visits (Figure 1). We applied 11 queries to these visits in ESSENCE. A total of 453 of 44 942 (0.97%) ED visits were identified by at least 1 query as a suspected AAEM case, of which 422 (93%) had an electronic medical record available for review.
Figure 1.
Number of eligible emergency department (ED) visits, number of ED visits identified as suspected cases of acute adverse effects of marijuana (AAEM), and number of true AAEM cases determined by chart review, Denver (Colorado) metropolitan area, 2016-2017. Abbreviation: ESSENCE, Electronic Surveillance System for the Early Notification of Community-Based Epidemics.
The overall PPV varied by hospital. At hospital A, 250 suspected AAEM cases were identified by at least 1 query, of which 114 were true AAEM cases, for a PPV of 46%. At hospital B, 39 suspected AAEM cases were identified by at least 1 query, of which 25 were true AAEM cases, for a PPV of 64%. At hospital C, 133 suspected AAEM cases were identified by at least 1 query, of which 49 were true AAEM cases, for a PPV of 37% (Table 1).
Table 1.
Number of true cases of acute adverse effects of marijuana (AAEM) and positive predictive value (PPV) identified by each query, by hospital, Denver (Colorado) metropolitan area, 2016-2017a
| Query No./Query Criteria | Hospital A | Hospital B | Hospital C | Total | Query Performanced | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| True Case,b No. (%) (n = 114) | Suspected Case,c No. (%) (n = 250) | PPV | True Case,b No. (%) (n = 25) | Suspected Case,c No. (%) (n = 39) | PPV | True Case,b No. (%) (n = 49) | Suspected Case,c No. (%) (n = 133) | PPV | True Case,b No. (%) (n = 188) | Suspected Case,c No. (%) (n = 422) | PPV | ||
| (1) ICD-10 codese only | 59 (52) | 69 (28) | 86 | 16 (64) | 19 (49) | 84 | 40 (82) | 102 (77) | 39 | 115 (61) | 190 (45) | 60 | 0.53 |
| (2) Triage note keywords onlyf | 85 (75) | 212 (85) | 40 | 17 (68) | 29 (74) | 59 | 27 (55) | 53 (40) | 50 | 129 (69) | 294 (70) | 44 | 0.57 |
| (3) Chief complaint keywords onlyf | 4 (3) | 5 (2) | 80 | 0 | 0 | 0 | 1 (2) | 1 (1) | 100 | 5 (3) | 6 (1) | 83 | 0.49 |
| (4) ICD-10 codese or chief complaint keywordsf or triage note keywordsf | 114 (100 | 250 (100) | 46 | 25 (100) | 39 (100) | 64 | 49 (100) | 133 (100) | 36 | 188 (100) | 422 (100) | 44 | 0.72 |
| (5) ICD-10 codese or triage note keywordsf | 114 (100) | 250 (100) | 46 | 25 (100) | 39 (100) | 64 | 49 (100) | 133 (100) | 36 | 188 (100) | 422 (100) | 44 | 0.72 |
| (6) ICD-10 codese or chief complaint keywordsf | 61 (54) | 72 (29) | 85 | 16 (64) | 19 (49) | 84 | 40 (82) | 102 (77) | 39 | 117 (62) | 193 (46) | 61 | 0.54 |
| (7) Chief complaint keywordsf or triage note keywordsf | 85 (75) | 213 (85) | 40 | 17 (68) | 29 (74) | 59 | 28 (57) | 54 (41) | 51 | 130 (69) | 296 (70) | 44 | 0.57 |
| (8) ICD-10 codese and chief complaint keywordsf and triage note keywordsf | 2 (2) | 2 (1) | 100 | 0 | 0 | 0 | 0 | 0 | 0 | 2 (1) | 2 (0) | 100 | 0.50 |
| (9) ICD-10 codese and triage note keywordsf | 30 (26) | 32 (13) | 94 | 8 (32) | 9 (23) | 89 | 18 (37) | 22 (17) | 82 | 56 (30) | 63 (15) | 89 | 0.52 |
| (10) ICD-10 codese and chief complaint keywordsf | 2 (2) | 2 (1) | 100 | 0 | 0 | 0 | 1 (2) | 1 (1) | 100 | 3 (2) | 3 (1) | 100 | 0.51 |
| (11) Triage note keywordsf and chief complaint keywordsf | 4 (3) | 4 (2) | 100 | 0 | 0 | 0 | 0 | 0 | 0 | 4 (2) | 4 (1) | 100 | 0.51 |
a Data are from 3 hospital emergency departments (EDs) in the Denver metropolitan area.
b A true case of AAEM was defined as an ED visit in which the medical record documented that the ED health care provider’s clinical impression was that the patient’s presenting illness was due to marijuana or likely due to marijuana.
c A suspected case of AAEM was defined as an ED visit identified by ≥1 query to the Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE).
d Query performance = (PPV + proportion of AAEM suspected cases identified)/2. Possible range: 0-1.0.
e International Classification of Diseases, Tenth Revision (ICD-10)16 codes: T40.7 (poisoning by cannabis), F12.1 (cannabis abuse), F12.2 (cannabis dependence), and F12.9 (cannabis use, unspecified).
f Inclusion: marijuana; THC; MJ; edible; cannab; smok and weed; smok and pot; pot and brownie. Negation: denies marijuana; denies MJ; denies THC; denies edible; denies cannab; marijuana and synthetic; marijuana and spice; marijuana and mamba; MJ and synthetic; MJ and spice; MJ and mamba.
Query Performance
The best-performing query was query 5, which combined ICD-10 diagnostic codes or triage note keywords. This query detected all 422 suspected AAEM cases, for a PPV of 44%, resulting in a query performance of 0.72 (Table 1). Query 5 had an identical performance index as query 4 but was more parsimonious and did not include keywords in the data field for chief complaint.
Fewer than 5% of true AAEM cases were detected by using keywords in the chief concern data field alone (query 3; Table 1). Any other query including keywords in the data field for chief complaint (queries 4, 6, and 7) also did not contribute substantially to the number of true AAEM cases detected. Queries 8-11, which required a combination of at least one ICD-10 diagnostic code and at least 1 keyword in the data field for either chief complaint or triage note, detected <40% of true AAEM cases in each hospital system.
Query performance varied by hospital. At hospitals A and B, query 1 (only ICD-10 diagnostic codes) had a PPV of 86% and identified more than half of the true AAEM cases. The same query at hospital C had a PPV of only 39%. Medical record review of the visits at hospital C showed that marijuana-specific ICD-10 diagnostic codes were applied to any visit in which toxicology screens were positive for cannabis, regardless of the reason for the ED visit.
The 2 ICD-10 diagnostic codes most frequently identified in both true AAEM cases and noncases were F12.1 and F12.9 (Table 2). However, compared with the diagnostic codes for noncases, the diagnostic codes significantly associated with an ED visit for true AAEM cases were T40.7 (100% vs 0%; P < .001) and F12.1 (77% vs 23%; P < .001).
Table 2.
Frequency of International Classification of Diseases, Tenth Revision (ICD-10) diagnostic codes and keywords in the triage note and chief complaint data fields, stratified by case status, for emergency department (ED) visits due to acute adverse effects of marijuana (AAEM), Denver (Colorado) metropolitan area, 2016-2017a
| Data Field | Data Element | AAEM Case Status | ||||
|---|---|---|---|---|---|---|
| Inclusionb | Negationc | Total Suspected Cases,d No. (%) | True Cases,e No. (%) | Noncases,f No. (%) | χ2 (P Value)g | |
| ICD-10 diagnostic code | T40.7 (poisoning by cannabis) | NA | 10 | 10 (100) | 0 | 12.8 (<.001) |
| F12.1 (cannabis abuse) | NA | 70 | 54 (77) | 16 (23) | 36.1 (<.001) | |
| F12.2 (cannabis dependence) | NA | 9 | 6 (67) | 3 (33) | 1.8 (.18) | |
| F12.9 (cannabis use, unspecified) | NA | 110 | 53 (48) | 57 (52) | 0.8 (.37) | |
| Triage note | Marijuana | Denies marijuana; synthetic; spice; mamba | 205 | 91 (44) | 114 (56) | 0.004 (.95) |
| THC | Denies THC | 19 | 12 (63) | 7 (37) | 2.8 (.09) | |
| MJ | Denies MJ; synthetic; spice; mamba | 51 | 20 (39) | 31 (61) | 0.7 (.41) | |
| Edible | Denies edible | 35 | 27 (77) | 8 (23) | 16.4 (<.001) | |
| Cannab | Denies cannab | 8 | 2 (25) | 6 (75) | 1.3 (.26) | |
| Smok AND weed | NA | 7 | 1 (14) | 6 (86) | 2.6 (.10) | |
| Smok AND pot | NA | 5 | 2 (40) | 3 (60) | 0.004 (.84) | |
| Pot AND brownie | NA | 1 | 1 (100) | 0 | 1.2 (.26) | |
| Chief complaint | Marijuana | Denies marijuana; synthetic; spice; mamba | 5 | 5 (100) | 0 | 12.7 (<.001) |
| THC | Denies THC | 1 | 0 | 1 (100) | 0.8 (.37) | |
| MJ | Denies MJ; synthetic; spice; mamba | |||||
| Edible | Denies edible | |||||
| Cannab | Denies cannab | |||||
| Smok AND weed | NA | |||||
| Smok AND pot | NA | |||||
| Pot AND brownie | NA | |||||
Abbreviation: NA, not applicable.
a Data are from 3 hospital EDs in the Denver metropolitan area.
b Inclusion codes and terms were ICD-10 diagnostic codes or keywords from the triage note and chief complaint data fields that were included in the syndromic surveillance algorithm.
c Negation terms were keywords and phrases from the triage note and chief complaint data fields that were suppressed in the syndromic surveillance algorithm to improve specificity of the algorithm.
d A suspected case of AAEM was defined as an ED visit identified by ≥1 query to the Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE).
e A true case of AAEM was defined as an ED visit in which the medical record documented that the ED health care provider’s clinical impression was that the patient’s presenting illness was due to marijuana or likely due to marijuana.
f A noncase of AAEM was defined as a visit in which the ED provider did not document that the patient’s illness was due to marijuana exposure or likely due to marijuana exposure.
g Using the Pearson χ2 test, with P < .05 considered significant.
In the data field for triage note, the most commonly identified keyword in the queries was marijuana (with exclusion terms denies marijuana, synthetic, spice, and mamba), which identified 205 suspected AAEM cases. However, the only keyword in the data field for triage note that was significantly associated with true AAEM cases was edible (with exclusion term denies edible), which detected 35 suspected AAEM cases, 27 (77%) of which were true AAEM cases and 8 (23%) of which were noncases (P < .001). All 5 records identified with the keyword marijuana in this field were determined to be true cases of AAEM (P < .001; Table 2).
Characteristics of Cases and Noncases
Among 188 true AAEM cases at the 3 hospitals, we found no significant differences in sex or age between cases and noncases. We found more male patients than female patients among cases and noncases. Most patients identified as true AAEM cases and noncases were aged 21-50. The proportion of true cases did not differ significantly by treatment facility (P = .09; Table 3).
Table 3.
Characteristics of true cases and noncases of emergency department (ED) visits for acute adverse effects of marijuana (AAEM), when available, Denver (Colorado) metropolitan area, 2016-2017a
| Factor | Case Status | χ2 (P Value)e | ||
|---|---|---|---|---|
| True Caseb (n = 188), No. (%) | Noncasec (n = 234), No. (%) | Total Suspected Cased (n = 422), No. (%) | ||
| Sex | 1.1 (.32) | |||
| Male | 109 (58) | 148 (63) | 257 (61) | |
| Female | 78 (41) | 86 (37) | 164 (39) | |
| Age, y | 1.3 (.52) | |||
| <21 | 33 (18) | 32 (14) | 65 (15) | |
| 21-50 | 133 (71) | 171 (73) | 304 (72) | |
| >50 | 22 (12) | 31 (13) | 53 (13) | |
| State residencef | 10.8 (.01) | |||
| Colorado | 148 (79) | 206 (88) | 354 (84) | |
| Non-Colorado | 32 (17) | 16 (7) | 48 (11) | |
| Marijuana route of usef | 16.5 (<.001) | |||
| Smoked | 63 (34) | 57 (24) | 120 (28) | |
| Eaten | 45 (24) | 8 (3) | 53 (13) | |
| Polysubstance usef | 3.7 (.05) | |||
| Yes | 60 (32) | 95 (41) | 155 (37) | |
| No | 126 (67) | 134 (57) | 260 (62) | |
| Intentionality of marijuana usef | 2.4 (.12) | |||
| Intentional | 178 (95) | 170 (73) | 348 (82) | |
| Accidental | 5 (3) | 1 (0) | 6 (1) | |
| Treatment facilityf | 9.4 (.09) | |||
| Hospital A | 114 (61) | 136 (58) | 250 (59) | |
| Hospital B | 25 (13) | 14 (6) | 39 (9) | |
| Hospital C | 49 (26) | 84 (36) | 133 (31) | |
a Data are from 3 hospital EDs in the Denver metropolitan area.
b A true case of AAEM was defined as an ED visit in which the medical record documented that the ED health care provider’s clinical impression was that the patient’s presenting illness was due to marijuana or likely due to marijuana.
c A noncase of AAEM was defined as a visit in which the ED provider did not document that the patient’s illness was due to marijuana exposure or likely due to marijuana exposure.
d A suspected case of AAEM was defined as an ED visit identified by ≥1 query to the Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE).
e Using the Pearson χ2 test, with P < .05 considered significant.
f Data were not always available in the ED visit. As such, the sum of these numbers does not always equal the total number of cases.
Patients identified as true AAEM cases were significantly more likely than patients identified as noncases to be non-Colorado residents (17% vs 7%, P = .01) and were significantly more likely to report ingesting edible marijuana products rather than smoking marijuana (24% vs 3%, P < .001). Intentionality of marijuana use did not differ significantly between cases and noncases, but only 6 patients reported accidental marijuana exposure (Table 3).
Application of Validated Query to ESSENCE
Of the 3 hospitals, Hospital A consistently had the highest frequency of suspected AAEM cases, and hospital B consistently had the lowest frequency of suspected AAEM cases (Figure 2).
Figure 2.
Number of emergency department (ED) visits possibly due to acute adverse effects of marijuana (AAEM) identified by query 5 in 3 hospital systems, Denver (Colorado) metropolitan area, July 1–December 31, 2017. Query 5 refers to an ED visit identified by having any of 4 International Classification of Diseases, Tenth Revision diagnostic codes16—T40.7 (poisoning by cannabis), F12.1 (cannabis abuse), F12.2 (cannabis dependence), and F12.9 (cannabis use, unspecified)—or marijuana-specific keywords in the triage note data field (inclusion terms: marijuana, THC, MJ, edible, cannab, smok and weed, smok and pot, pot and brownie; exclusion terms: denies marijuana, denies MJ, denies THC, denies edible, denies cannab, marijuana and synthetic, marijuana and spice, marijuana and mamba, MJ and synthetic, MJ and spice, MJ and mamba). The code for query 5 is available from the authors on request.
Discussion
Performing medical record review is critically important to accurately evaluate the performance of ESSENCE queries to detect true AAEM cases. In our study, the query using marijuana-specific ICD-10 diagnostic codes and keywords in the triage note data field outperformed queries based only on keywords in the chief complaint data field. However, even this best-performing query resulted in a modest PPV of 44%, which was lower than the >95% PPV estimates from a study that used a review of several data fields in ESSENCE as the AAEM case reference standard, rather than a medical record review for clinical impression.12 A strength of our study was the use of a high-quality standard for the case definition, namely documentation of the ED health care provider’s assessment that marijuana use was a likely cause for the ED visit. Another strength of this study was the use of data from 3 hospital systems with high-quality data extraction to ESSENCE, as demonstrated by the >95% data field completion. Our findings emphasize the importance of using a robust reference standard when evaluating the performance of queries to syndromic surveillance data.
The variation in PPV of queries by hospital might be related to hospital-specific coding and documentation practices. Differences in how hospital systems apply diagnostic codes is well documented.19,20 Our observation that query 1 (ICD-10 diagnostic codes alone) performed better at hospital A than at hospital C might illustrate these differences in coding practices. The observation that hospital C routinely applied a marijuana-specific ICD-10 diagnostic code whenever there was a positive result for a urine toxicology cannabinoid screen might in part explain the lower PPV (39%) of query 1 to detect AAEM cases at that hospital. Assigning ICD-10 diagnostic codes solely for a positive laboratory result that is unrelated to the clinical impression for the ED visit could result in an inflated denominator of suspected cases, thus lowering the PPV of the ICD-10 diagnostic code.
Marijuana tourism, or the practice of visiting an area with legalized or accessible marijuana, has been a well-documented source of revenue for Colorado21 since legalization of retail sales went into effect in January 2014. Our study suggests that marijuana tourists might be disproportionately using ED services for adverse health effects from marijuana exposure, reflected in the findings that non-Colorado residential status was significantly associated with true AAEM cases. It might be that marijuana tourists are less used to the effect of marijuana or less likely to use marijuana responsibly than Colorado residents, resulting in a disproportionate number of ED visits.22
Compared with smoked marijuana, edible marijuana has slower onset and longer-lasting effects.23 Users of edible marijuana can unknowingly ingest unsafe amounts of marijuana and subsequently experience extreme effects, leading to a visit to the ED. Our finding that patients who reported ingesting marijuana were significantly more likely than patients who reported smoking marijuana to be true AAEM cases is consistent with other research.24
Limitations
Our study had several limitations. First, because of the labor- and time-intensive approach of reviewing medical records, our validation process was limited to 3 hospitals in the TCHD jurisdiction. In addition, the frequency, pattern, and characteristics of marijuana-associated ED visits might differ from those in other US regions, particularly in areas where retail recreational marijuana is not legalized. Patients are also increasingly using alternative health care services, such as commercial urgent care centers, which were not represented in this analysis. Thus, the generalizability of the study findings is limited. Second, query performance in syndromic surveillance data depends on the quality and completeness of data that each hospital abstracts and sends to ESSENCE; hospitals that submit incomplete data limit the validity of any query used in ESSENCE to conduct surveillance. Thus, although the queries using the chief complaint data field performed poorly in our analysis, these queries might perform better in hospitals that transmit richer data on chief complaints. The utility of any particular data field in ESSENCE might therefore vary by facility and over time. Third, the query performance analysis was limited to PPV because the study was designed to evaluate only a subset of all ED visits, which made it impossible to calculate the sensitivity and specificity of each query. Finally, changes during the study period to the hospital ED ICD-10 diagnostic coding system, upgrade of ESSENCE, or secular changes affecting patients’ exposure to marijuana or their likelihood of presenting to the ED after experiencing adverse effects of marijuana might affect the validation results.
Conclusions
Methods for validating syndromic surveillance queries used for public health surveillance should be standardized to improve consistency and data interpretation. Our study demonstrated the benefits of using medical record review as a reference standard to verify syndromic surveillance queries and optimize query constructs used to identify potential cases of a disease or syndrome of illness. Validation was necessary before assuming that syndromic surveillance output was an accurate measure of community-level burden and trends in marijuana-associated adverse health outcomes. To optimize the validity and utility of syndromic surveillance to monitor ED visits related to adverse effects of marijuana, we recommend that hospitals apply ICD-10 diagnostic codes consistently and accurately so that these codes truly reflect the nature of the ED visit,19 rather than selecting codes only to increase billing reimbursement potential.20 Periodic quality checks of syndromic surveillance data should be performed to verify that no substantial changes in ICD-10 diagnostic coding or transmission patterns in ESSENCE have occurred that might affect the accuracy of surveillance output.
In our study, queries of ED visit data submitted through the ESSENCE syndromic surveillance system at 3 hospitals in Colorado that used marijuana-specific ICD-10 diagnostic codes and keywords in the triage note data field detected true cases of AAEM about half the time, although query performance varied by hospital. Application of validated queries in ESSENCE might enable detection of “outbreaks” of adverse health effects of marijuana resulting in ED visits.
Footnotes
Declaration of Conflicting Interests: The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The authors received no financial support for the research, authorship, and/or publication of this article.
ORCID iD: Grace E. Marx, MD, MPH 
https://orcid.org/0000-0003-2393-6947
References
- 1. Colorado Constitution Article XVIII, Section 16: personal use and regulation of marijuana. https://www.colorado.gov/pacific/sites/default/files/Section%2016%20-%20%20Retail.pdf. Accessed December 19, 2018.
- 2. Ghosh TS, Vigil DI, Maffey A, et al. Lessons learned after three years of legalized, recreational marijuana: the Colorado experience. Prev Med. 2017;104:4–6. doi:10.1016/j.ypmed.2017.02.021 [DOI] [PubMed] [Google Scholar]
- 3. Ghosh T, Van Dyke M, Maffey A, Whitley E, Gillim-Ross L, Wolk L. The public health framework of legalized marijuana in Colorado. Am J Public Health. 2016;106(1):21–27. doi:10.2105/AJPH.2015.302875 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Hall KE, Monte AA, Chang T, et al. Mental health-related emergency department visits associated with cannabis in Colorado. Acad Emerg Med. 2018;25(5):526–537. doi:10.1111/acem.13393 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Cao D, Srisuma S, Bronstein AC, Hoyte CO. Characterization of edible marijuana product exposures reported to United States poison centers. Clin Toxicol (Phila). 2016;54(9):840–846. doi:10.1080/15563650.2016.1209761 [DOI] [PubMed] [Google Scholar]
- 6. Wang GS, Roosevelt G, Le Lait MC, et al. Association of unintentional pediatric exposures with decriminalization of marijuana in the United States. Ann Emerg Med. 2014;63(6):684–689. doi:10.1016/j.annemergmed.2014.01.017 [DOI] [PubMed] [Google Scholar]
- 7. Wang GS, Hall K, Vigil D, Banerji S, Monte A, VanDyke M. Marijuana and acute health care contacts in Colorado. Prev Med. 2017;104:24–30. doi:10.1016/j.ypmed.2017.03.022 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Todkill D, Loveridge P, Elliot AJ, et al. Utility of ambulance data for real-time syndromic surveillance: a pilot in the West Midlands region, United Kingdom. Prehosp Disaster Med. 2017;32(6):667–672. doi:10.1017/S1049023X17006690 [DOI] [PubMed] [Google Scholar]
- 9. Goldman-Mellor S, Jia Y, Kwan K, Rutledge J. Syndromic surveillance of mental and substance use disorders: a validation study using emergency department chief complaints. Psychiatr Serv. 2018;69(1):55–60. doi:10.1176/appi.ps.201700028 [DOI] [PubMed] [Google Scholar]
- 10. Yoon PW, Ising AI, Gunn JE. Using syndromic surveillance for all-hazards public health surveillance: successes, challenges, and the future. Public Health Rep. 2017;132(suppl 1):3S–6S. doi:10.1177/0033354917708995 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Lombardo J, Burkom H, Elbert E, et al. A systems overview of the Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE II). J Urban Health. 2003;80(2 suppl 1):i32–i42. doi:10.1007/PL00022313 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. DeYoung K, Chen Y, Beum R, Askenazi M, Zimmerman C, Davidson AJ. Validation of a syndromic case definition for detecting emergency department visits potentially related to marijuana. Public Health Rep. 2017;132(4):471–479. doi:10.1177/0033354917708987 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Nolan ML, Kunins HV, Lall R, Paone D. Developing syndromic surveillance to monitor and respond to adverse health events related to psychoactive substance use: methods and applications. Public Health Rep. 2017;132(1 suppl 1):65S–72S. doi:10.1177/0033354917718074 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Coates RJ, Pérez A, Baer A, et al. National and regional representativeness of hospital emergency department visit data in the National Syndromic Surveillance Program, United States, 2014. Disaster Med Public Health Prep. 2016;10(4):562–569. doi:10.1017/dmp.2015.181 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Witkop C, Duffy M, Cohen L, Fishbein D, Selent M. Assessment of ESSENCE performance for influenza-like illness surveillance after an influenza outbreak—U.S. Air Force Academy, Colorado, 2009. MMWR Morb Mortal Wkly Rep. 2011;60(13):406–409. [PubMed] [Google Scholar]
- 16. World Health Organization. International Statistical Classification of Diseases and Related Health Problems. 10th revision https://www.who.int/classifications/icd/ICD10Volume2_en_2010.pdf. Published 2011. Accessed December 18, 2018.
- 17. Centers for Disease Control and Prevention. Epi Info. https://www.cdc.gov/epiinfo/index.html. Published 2011. Accessed December 18, 2018.
- 18. SAS Institute Inc. SAS/STAT Version 9.4 for Linux. Cary, NC: SAS Institute Inc; 2014. [Google Scholar]
- 19. Lloyd SS, Rissing JP. Physician and coding errors in patient records. JAMA. 1985;254(10):1330–1336. [PubMed] [Google Scholar]
- 20. O’Malley KJ, Cook KF, Price MD, Wildes KR, Hurdle JF, Ashton CM. Measuring diagnoses: ICD code accuracy. Health Serv Res. 2005;40(5 pt 2):1620–1639. doi:10.1111/j.1475-6773.2005.00444.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Light MK, Orens A, Lewandowski B, Pickton T. Market Size and Demand for Marijuana in Colorado. Boulder, CO: Colorado Department of Revenue; 2014. [Google Scholar]
- 22. Kim HS, Hall KE, Genco EK, Van Dyke M, Barker E, Monte AA. Marijuana tourism and emergency department visits in Colorado. N Engl J Med. 2016;374(8):797–798. doi:10.1056/NEJMc1515009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Barrus DG, Capogrossi KL, Cates SC, et al. Tasty THC: promises and challenges of cannabis edibles. Methods Rep RTI Press. 2016;2016(15):1279–1284. doi:10.2146/ajhp140523 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Kim HS, Monte AA. Colorado cannabis legalization and its effect on emergency care. Ann Emerg Med. 2016;68(1):71–75. doi:10.1016/j.annemergmed.2016.01.004 [DOI] [PMC free article] [PubMed] [Google Scholar]