Abstract
Background
Increasing epidemiologic and intervention research is being conducted on opioid overdose, a serious and potentially fatal outcome. However, there is little consensus on how to verify opioid overdose outcomes for research purposes. To ensure reproducibility, minimize misclassification, and permit data harmonization across studies, standardized and consistent overdose definitions are needed. The aims were to develop a case criteria classification scheme based on information commonly available in medical records and to compare it to reviewing physician clinical impression and simple encounter documentation.
Methods
In two large health systems, we developed a case criteria classification scheme for opioid overdose based on prior literature, expert opinion, and pilot testing with sample medical records. We then identified emergency department and hospital encounters (n=259) with at least one International Classification of Diseases–9–Clinical Modification (ICD-9-CM) codes suggestive of a pharmaceutical opioid or heroin poisoning. Physicians conducted structured medical record reviews to identify the proposed case criteria and generate a clinical impression and trained abstractors verified documentation. We then compared the case criteria classification scheme to clinical impression and encounter documentation.
Results
We developed a quantitative opioid overdose case criteria classification scheme which included three sets of major criteria and nine minor criteria (supporting documentation). The confirmation rates of the ICD-9-CM codes using the case criteria classification scheme, clinical impression, and encounter documentation ranged from 50.4% to 52.7% at one site and 55.5% to 67.2% at the second site. Discrepancies across approaches and sites related to differences in available records and documentation of clinical signs of overdose.
Conclusions
We propose a novel case criteria classification scheme for opioid overdose that could be used to rigorously and consistently define overdose across multiple research settings. However, prior to widespread use, further refinement and validation are needed.
Keywords: opioid analgesics, heroin, poisoning, epidemiology, methods
Introduction
Due to rising opioid overdose rates in the United States, Canada, and the United Kingdom,1–3 fatal and nonfatal opioid overdoses are increasingly important outcomes in clinical and epidemiological research. In the United States, numerous epidemiologic studies have relied on large electronic health record (EHR) and medical claims databases – which cover tens of millions of patients – to understand the risk factors for overdose.4–8 Although these studies have produced important, policy-informing results, large clinical databases have inherent limitations and improved methods to accurately and reliably identify overdose cases are needed.
To identify overdoses, EHR and claims-based studies typically use International Classification of Disease (ICD) codes, which are generated for billing and clinical care rather than research. Given that coding practices may be imprecise and variable across health systems, a proportion of the overdose outcome data will be misclassified as false negatives and false positives. Since opioid overdose is a relatively rare outcome, small amounts of outcome misclassification could have a significant impact on results. To minimize false positives, overdose cases have been identified and verified using a two-step process. First, researchers create a list of ICD codes to identify potential overdose cases in claims data. Second, researchers in settings with access to more detailed EHR data conduct medical record reviews to confirm true positive cases. Studies relying on medical record review report confirmation rates (positive predictive value) ranging from approximately 20 to 80 percent, depending on the health system and ICD codes employed.5,6,8–10
Another reason for inconsistency in confirmation rates is that studies have employed a wide range of approaches for verifying cases with medical record review.5,6,8–10 These approaches range from an abstractor confirming that overdose was documented in the medical record to clinical experts conducting a more involved case adjudication process. Although the former approach is efficient, it may not distinguish an overdose from other opioid-induced adverse events (e.g., ingestion without any signs of respiratory depression, over-sedation without loss of consciousness, and opioid withdrawal) and opioid misuse. In contrast, expert adjudication may be more rigorous and detailed, but it is resource-intensive and remains subjective. For intervention trials, an objective pre-specified outcome is imperative. At present, there are no established case criteria that can be consistently applied across studies to confirm the clinical syndrome associated with an opioid overdose. Without such criteria, the ability to interpret results across observational and interventional studies is limited.
For this study, we sought to develop and conduct an initial evaluation of an objective case criteria classification scheme that could be used to confirm potential pharmaceutical opioid and heroin overdoses in the medical record. Our goal was to propose a scheme that could be applied across disparate EHR-based research settings to confirm opioid overdose cases for observational and intervention studies. Our evaluation was designed to compare and contrast the case criteria classification scheme with two other confirmation approaches.
Methods
Design
We conducted a retrospective assessment of medical records in two health systems with distinct patient populations and EHR systems. The study design had two phases. In the first phase, we developed and refined criteria that could be used to confirm pharmaceutical opioid and heroin overdose cases identified in the EHR. A classification scheme to apply the case criteria was also developed. In the second phase, we evaluated the case criteria classification scheme by comparing it to two other approaches, neither of which represents a true “gold standard” (Figure 1). In this evaluation, we identified potential overdose cases using ICD codes in the two health systems. Physicians in each health system reviewed the medical records of potential overdose cases to abstract the pertinent clinical information that defined the various criteria of the case classification scheme, and generated a clinical impression about the event. To generate a clinical impression, physicians were instructed to make their own judgement about the event after reviewing the clinical documentation in the medical records. In parallel, non-clinician researchers independently examined the medical records of potential cases for encounter documentation of overdose due to opioids. Across the three approaches, confirmation rates were calculated and differences across the approaches were described.
Figure 1.
Approach to evaluating clinical case criteria classification scheme
Setting and Source Population
Our study was conducted at two sites selected because they represented different patient populations, health system organization, EHR systems, and approaches to medical coding: Kaiser Permanente Colorado (Site 1) and Denver Health Medical Center (Site 2). Site 1 is a large integrated health care organization with more than 30 primary care clinics and pharmacies serving approximately more than 600,000 members in the western United States. Site 1 does not operate its own hospitals but has access to most hospital records. Site 2 is a public healthcare system serving nearly 100,000 urban county residents. Site 2 includes a Level I Trauma Center, an acute care hospital, a system of linked federally qualified health centers, specialty care, and the paramedic system. The source population included all members 18 or older enrolled in site 1’s health plan for at least one day, or “empaneled” at site 2 between June 1, 2013, and April 1, 2014. At site 2, “empaneled” members were individuals who had at least one visit with a primary care provider during the prior 18 months, consistent with prior studies.11 Empanelment is a standard way in which individuals are identified and followed for health services research in health systems without formal membership enrollment, such as site 2.
Phase 1: Identifying Case Criteria to Develop a Classification Scheme
We first identified potential criteria that defined, characterized, and/or supported the diagnosis of the life-threatening clinical syndrome associated with an opioid overdose event. We then sought the input of study investigators and experts from a local, multi-institutional opioid research group consisting of at least one each from public health practice, epidemiology, internal medicine, emergency medicine, toxicology, and addiction medicine to identify additional criteria and/or refine the criteria. Criteria had to be identifiable in routinely collected medical records, such as paramedic reports, emergency department notes, hospital admission history and physical, discharge summaries, and laboratory records. In investigator team meetings, we discussed and iteratively refined the groupings of items into major and minor criteria. Major criteria defined or characterized the clinical syndrome of an opioid overdose (e.g., respiratory depression) whereas minor criteria were suggestive of the cause or the diagnosis (e.g., presence of paraphernalia, positive urine toxicology).
We then created a structured REDCap tool12 to abstract case criteria from the medical record. REDCap was used because it is a HIPAA compliant web-based application that can be used to securely manage patient-level health data across sites.12 To further refine the criteria, two physician investigators (I.A.B. and E.M.G.) conducted medical chart abstractions on eight potential overdoses identified using ICD-9-Clinical Modification (CM) codes for opioid overdoses at each site and the research team discussed the findings. This process resulted in further refinement of the classification scheme. Redundant criteria were eliminated or combined until we had a set of 18 criteria grouped within three major criteria (2–4 items each) categories plus minor criteria (9 items). Refinements of the criteria were made to accommodate the limited documentation available in the medical record. For instance, pinpoint pupils were rarely documented, thus pinpoint pupils was included as a minor criteria. In addition, the actual response to naloxone (e.g., increased respiratory rate) was rarely documented, leading us to omit explicit documentation of a response to naloxone from the criteria.
Finally, we created a standardized classification scheme that could be objectively applied to the case criteria to characterize the likelihood that the event was an opioid overdose: definite, probable, probably not, and definitely not. For example, if the event met at least one criterion from all three major categories or one criterion from two major categories plus two minor criteria, it was considered a “definite” opioid overdose. If an event met no major criteria, it was considered definitely not an overdose.
Phase 2: Evaluation
Sampling
At both study sites, we identified potential opioid overdoses in the emergency department or inpatient settings using ICD-9-CM poisoning codes suggestive of pharmaceutical opioid or heroin overdose generated from published sources (Appendix 1).5,13 We included various ICD-9-CM codes to allow comparisons across the three case confirmation approaches, from overdose codes that explicitly indicated opioids to overdose codes that did not mention opioids. We excluded codes in the outpatient setting because a preliminary review suggested that these were generally follow-up visits after an overdose event or had insufficient information to determine if an opioid overdose occurred. We also excluded codes that were infrequently used at both sites (e.g., E935.0, E935.1, E935.2: heroin, methadone, and other opiates “causing an adverse effect in therapeutic use”).14 Multiple codes may have been coded for each event. Events with diagnosis codes occurring within three days were considered as single events.
We used Cochran’s formula15 for calculating the sample size, with α = 0.05 and an assumed confidence interval width of 0.05 and a confirmation rate of 0.85 for ICD-9-CM codes which specified opioids and an assumed confidence interval width of 0.10 and a confirmation rate of 0.2 for ICD-9-CM codes which did not specify opioids. A medical record review was conducted on a random sample from each of the two groups.
Medical Record Reviews
For the case criteria classification scheme and clinical impression, Internal Medicine physicians at each site (I.A.B, E.M.G, S.C., and T.P.) conducted structured medical record reviews of the potential cases identified by ICD-9-CM codes (Figure 1). These physicians reviewed records for documentation of each of the case criteria. Medical records reviewed included paramedic notes, emergency department notes, hospital admission history and physical, discharge summaries, laboratory records, and telephone notes (e.g., requesting insurance authorization for inpatient admission), when available. Case criteria were entered into REDCap as yes, no, missing, or other. In 13 cases, two physicians independently reviewed the same event to assess reliability.
Physicians were asked to provide an overall clinical judgement of the primary cause of the event (clinical impression): definite, probable, probably not, definitely not a heroin or pharmaceutical opioid overdose, or insufficient information to classify/undetermined. Clinical information stored in EHRs can be variably organized and presented based on type of system and individual preferences. To ensure that each physician reviewed similar information in the EHR, physicians only judged the cause of the event after abstracting the case criteria. When both heroin and pharmaceutical opioids were involved in the event, physicians were asked to code both (e.g., definite heroin overdose and probable pharmaceutical opioid overdose). Given that overdoses may involve more than a single medication or substance, physicians documented the presence of other contributing substances. Finally, they judged the intent of the overdose (accidental, undetermined, or intentional).
For encounter documentation, trained non-clinician researchers reviewed all medical records within three days of the ICD-9-CM code date to determine if the reason for emergency department visit or the hospitalization was documented as an opioid overdose in the record.
Demographic information was derived from the EHR. Additional clinical data (e.g., highest level of care, disposition) were abstracted to provide context on the clinical severity and outcomes of identified overdoses.
Data Analysis
We compared the demographics of the study samples across sites using t-tests and chi-square tests. Data from case criteria were analyzed to determine if the events met the classification scheme for definite or probable pharmaceutical opioid or heroin overdoses. Definite and probable overdoses were considered true overdoses, whereas probably not or definitely not overdoses were considered false. Confirmation rates were calculated for each approach at each site separately. For each approach, the denominator was the total number of events identified by sampled overdose ICD-9-CM codes, while the numerator was the number of true overdoses. We also examined concordance between approaches by calculating Cohen’s kappa statistics. Finally, we qualitatively reviewed the abstractions and comments from instances where the findings from different approaches were discrepant.
This study was approved by the Colorado Multiple Institutional Review Board and the Kaiser Permanente Colorado Institutional Review Board. The requirement for informed consent was waived for this chart review study. We received a Federal Certificate of Confidentiality.
Results
Phase 1: Development of the case criteria classification scheme
The case criteria development process resulted in three categories of major criteria: (1) signs of respiratory depression, (2) altered mental status, and (3) opioid antagonist treatment receipt/response, as well as nine minor criteria, representing supporting documentation. Table 1 displays the proposed case criteria and classification scheme. All major criteria were not required to define an overdose case because EHR data typically does not contain adequate information to meet all criteria.
Table 1.
Proposed case criteria for confirming pharmaceutical opioid or heroin overdose events in health records and a case criteria classification scheme
| Case Criteria |
|---|
|
Major criteria 1: Signs of respiratory depression 1. Respiratory rate <10 or bradypnea or apnea at any time 2. Respiratory failure requiring mechanical ventilation/bag mask ventilation |
|
Major criteria 2: Altered mental status 1. Abnormal Glasgow coma score (<15) 2. Unresponsive/unconscious 3. Altered/sedated/confused/somnolent/stupor |
|
Major criteria 3: Opioid antagonist treatment receipt/response 1. Naloxone given by family, bystanders, emergency medical services or hospital personnel 2. Responded to naloxone in any form except sublingual* 3. Required more than 1 dose of naloxone 4. Required continuous naloxone |
|
Minor criteria (supporting documentation) 1. Encounter documents indicate opioid overdose 2. Witness or patient said the patient overdosed on opioids 3. Heroin/opioids/opioid pills/fentanyl patches around or on patient 4. Urine or blood test positive for opioids 5. Patient prescribed opioids 6. Patient known to use heroin or have history of heroin dependence 7. Patient is currently or recently in opioid agonist therapy/treatment or medication for the treatment of opioid use disorder 8. Needle / paraphernalia found at the scene or track marks / puncture wounds on patient 9. Miosis/pinpoint pupils |
| Classification scheme |
|
Definite opioid overdose ▪ At least 1 criterion within each of 3 major criteria categories, OR ▪ At least 1 criterion within each of 2 major criteria categories plus at least 2 minor criteria |
|
Probable opioid overdose ▪ At least 1 criterion within each of 2 major criteria categories with 0–1 minor criteria, OR ▪ At least 1 criterion within 1 major criteria category plus at least 1 minor criteria |
|
Probably not an opioid overdose ▪ 1 criterion within 1 major criteria category and no minor criteria |
|
Definitely not an opioid overdose ▪ No criteria in any major criteria category |
This exclusion was designed to exclude buprenorphine/naloxone products used in the treatment of opioid use disorder, which are administered via sublingual route
Phase 2: Evaluation
Study sample
We randomly sampled 259 emergency department and hospitalization visits linked to at least one ICD-9-CM code suggestive of pharmaceutical opioid or heroin overdose (131 at Site 1 and 128 at Site 2). Compared to Site 2 (Table 2), the study population at Site 1 was older (46.4 vs. 40.3 years; p=0.008), had a higher proportion of women (58.5% vs. 38.9%, p=0.002), and was more likely to be commercially insured (60.2% vs. 6.4%). At Site 2, there was a higher proportion of African Americans (12.7% vs. 5.7%), people of Hispanic ethnicity (29.4% vs. 6.5%), and people with Medicaid coverage (46.8% vs. 7.3%).
Table 2.
Demographic characteristics of patients with sampled International Classification of Diseases-9 Clinical Modification codes, by site (n=259)
| Characteristics | Site 1 (n=123a) | Site 2 (n=126b) | p-value |
|---|---|---|---|
| Age in years, | 0.008c | ||
| Mean (std dev) | 46.4 (20.4) | 40.3 (14.4) | |
| Median (25th, 75th percentile) | 46.0 (24.0, 64.0) | 37.0 (28.5, 51.0) | |
| Gender, n (%) | 0.002d | ||
| Male | 51 (41.5) | 77 (61.1) | |
| Female | 72 (58.5) | 49 (38.9) | |
| Race/ethnicity, n (%) | <0.001d | ||
| White | 89 (72.4) | 71 (56.4) | |
| Hispanic | 8 (6.5) | 37 (29.4) | |
| African American | 7 (5.7) | 16 (12.7) | |
| Other and Unknown | 19 (15.4) | 2 (1.6) | |
| Insurance typee, n (%) | <0.001d | ||
| Commercial | 74 (60.2) | 8 (6.4) | |
| Medicaid | 9 (7.3) | 59 (46.8) | |
| Medicare | 36 (29.3) | 23 (18.3) | |
| Indigent Plan Care | 0 | 6 (4.8) | |
| Others | 4 (3.3) | 19 (15.1) | |
| Unknown | 0 | 11 (8.7) | |
7 patients had more than 1 event during the study period (total events=131)
2 patients had more than 1 event during the study period (total events=128)
Compared the difference in means using the t-test
Compared using the chi-square test
Patients with more than 1 event had the same insurance at all visits
Confirmation rates
At Site 1, the confirmation rates for definite and probable opioid overdoses were 51.9%, 52.7%, and 50.4% for case criteria, clinical impression, and encounter documentation, respectively (Table 3). At Site 2, the respective confirmation rates were 67.2%, 63.3%, and 55.5%. Higher confirmation rates at Site 2 can be attributed to the availability of paramedic records and complete hospital records.
Table 3.
Opioid overdose confirmation rates by three approaches at two sites among potential cases identified using a range of International Classification of Diseases-9 Clinical Modification codes (n=259)
| Site 1 (n=131) | Site 2 (n=128) | |||
|---|---|---|---|---|
| Definite & probable opioid overdoses (n) | Confirmation rate (95% CI) | Definite & probable opioid overdoses (n) | Confirmation rate (95% CI) | |
| Case criteria classification scheme | 68 | 51.9 (43.4, 60.5) | 86 | 67.2 (59.1, 75.3) |
| Clinical impression | 69 | 52.7 (44.1, 61.2) | 81 | 63.3 (54.9, 71.6) |
| Encounter documentation | 66 | 50.4 (41.8, 58.9) | 71 | 55.5 (46.9, 64.1) |
Across all three approaches, 58.4% to 62.8% of the events were categorized as accidental, and heroin was judged to be responsible for 39.0% to 44.7% of the overdoses (Appendix 2). Poly-substance use and poly-pharmacy were common among opioid overdose cases (51.3%−55.8%).
Table 4 describes the distribution of the major and minor case criteria across the sites. Although a high proportion of medical records had evidence of altered mental status at both sites (79.7%−95.4%), signs of respiratory depression was less frequently documented, ranging from 36.2% to 57.0%. Receipt of or response to naloxone was documented in 47.8% to 72.1% of the encounters. Minor criteria were present in 98.5% to 100% of the events.
Table 4.
Case criteria met for cases confirmed by three approaches, by site
| Case criteria classification scheme n (%) | Clinical impression n (%) | Encounter documentation n (%) | ||||
|---|---|---|---|---|---|---|
| Criteria | Site 1 (n=68) | Site 2 (n=86) | Site 1 (n=69) | Site 2 (n=81) | Site 1 (n=66) | Site 2 (n=71) |
| Major criteria category 1: Signs of respiratory depression | ||||||
| 1. Respiratory rate <10 or bradypnea or apnea | 24 (35.3) | 45 (52.3) | 22 (31.9) | 40 (49.4) | 22 (33.3) | 35 (49.3) |
| 2. Respiratory failure | 14 (20.6) | 20 (23.3) | 10 (14.5) | 17 (21.0) | 9 (13.6) | 14 (19.7) |
| At least one major criteria in category 1 | 30 (44.1) | 49 (57.0) | 25 (36.2) | 44 (54.3) | 24 (36.4) | 38 (53.5) |
| Major criteria category 2: Altered mental status | ||||||
| 1. Abnormal Glasgow coma score (<15) | 4 (5.9) | 58 (67.4) | 3 (4.4) | 48 (59.3) | 2 (3.0) | 44 (62.0) |
| 2. Unresponsive/unconscious | 29 (42.7) | 41 (47.7) | 27 (39.1) | 36 (44.4) | 26 (39.4) | 33 (46.5) |
| 3. Altered/sedated/confused/ somnolent/stupor | 60 (88.2) | 80 (93.0) | 51 (73.9) | 67 (82.7) | 49 (74.2) | 57 (80.3) |
| At least one major criteria in category 2 | 64 (94.1) | 82 (95.4) | 55 (79.7) | 69 (85.2) | 53 (80.3) | 59 (83.1) |
| Major criteria category 3: Opioid antagonist treatment receipt/response | ||||||
| 1. Naloxone given | 33 (48.5) | 60 (69.8) | 32 (46.4) | 55 (67.9) | 31 (47.0) | 48 (67.6) |
| 2. Responded to naloxone | 30 (44.1) | 50 (58.1) | 29 (42.0) | 48 (59.3) | 28 (42.4) | 42 (59.2) |
| 3. Required more than 1 dose of naloxone | 15 (22.1) | 20 (23.3) | 14 (20.3) | 20 (24.7) | 14 (21.2) | 19 (26.8) |
| 4. Required continuous naloxone | 8 (11.8) | 4 (4.7) | 8 (11.6) | 4 (4.9) | 8 (12.1) | 4 (5.6) |
| At least one major criteria in category 3 | 34 (50.0) | 62 (72.1) | 33 (47.8) | 57 (70.4) | 32 (48.5) | 49 (69.0) |
| No major criteria | 0 | 0 | 11 (15.9) | 8 (9.9) | 10 (15.2) | 8 (11.3) |
| Minor criteria | ||||||
| 1. Encounter documents indicate opioid overdose | 56 (82.4) | 67 (77.9) | 63 (91.3) | 72 (88.9) | 62 (93.9) | 63 (88.7) |
| 2. Witness or patient report | 49 (72.1) | 62 (72.1) | 52 (75.4) | 69 (85.2) | 52 (78.8) | 56 (78.9) |
| 3. Heroin/opioids/opioid pills/fentanyl patches around or on patient | 6 (8.8) | 13 (15.1) | 7 (10.1) | 15 (18.5) | 6 (9.1) | 13 (18.3) |
| 4. Toxicology positive for opioids | 24 (35.3) | 27 (31.4) | 27 (39.1) | 25 (30.9) | 26 (39.4) | 22 (31.0) |
| 5. Patient taking prescription opioids | 48 (70.6) | 27 (31.4) | 44 (63.8) | 23 (28.4) | 41 (62.1) | 20 (28.2) |
| 6. Patient known to use heroin or have history of heroin use disorder | 12 (17.7) | 55 (64.0) | 15 (21.7) | 56 (69.1) | 13 (19.7) | 49 (69.0) |
| 7. Patient is currently/recently in medication assisted treatment | 6 (8.8) | 8 (9.3) | 8 (11.6) | 8 (9.9) | 7 (10.6) | 7 (9.9) |
| 8. Needle / paraphernalia, fresh track marks / puncture wounds | 2 (2.9) | 25 (29.1) | 2 (2.9) | 27 (33.3) | 2 (3.0) | 23 (32.4) |
| 9. Miosis/Pinpoint pupils | 13 (19.1) | 49 (57.0) | 11 (15.9) | 45 (55.6) | 11 (16.7) | 41 (57.8) |
| At least one minor criteria | 67 (98.5) | 86 (100.0) | 69 (100.0) | 81 (100.0) | 66 (100.0) | 70 (98.6) |
For reliability testing of clinical impression, there was complete agreement between the two physician abstractors at Site 2, whereas there was a lack of agreement in one out of the eight medical records at Site 1. One abstractor indicated the case was a probable pharmaceutical opioid overdose whereas the other indicated there was insufficient information to judge.
There was discordance in the cases confirmed across approaches, with kappa values of 0.50 to 0.61 (Table 5). We identified reasons for discrepancies between approaches by reviewing the discordant chart abstraction documents and comments (Appendix 3). When the case criteria were consistent with an opioid overdose but clinical impression was inconsistent (n=18), it was generally because other medications or substances were involved, such as digoxin or lithium. Physician abstractors were reluctant to conclude that opioids were the primary cause of the overdose; in contrast, the case criteria did not restrict the definition to events in which other substances could have been the primary cause. When the case criteria classification was negative for an overdose but clinical impression was positive (n=19), it was often due to limited documentation of the clinical signs consistent with overdose, such as low respiratory rate. For example, documentation was sometimes sparse for heroin overdoses among people who injected drugs. In such a case, the overdose was clearly documented in the medical record but the event could not meet case criteria. In other instances, patients received treatment with naloxone in the pre-hospital setting and fully recovered by the time they arrived in the emergency department. If paramedic reports were unavailable, there was insufficient information documented in the emergency department notes to meet case criteria but it was clear to the reviewing physician that an opioid overdose had occurred. Other discrepancies are described in Appendix 3.
Table 5.
Comparison of application of the three approaches (n=259)
| Number cases (%) | Kappa (95% CI) | ||
|---|---|---|---|
| Clinical Impression | Case Criteria | ||
| Opioid Overdose | No Opioid Overdose | 0.50 (0.42, 0.59) | |
| Opioid Overdose | 131 (50.6) | 19 (7.3) | |
| No Opioid Overdose | 18 (7.0) | 59 (22.8) | |
| Insufficient Information | 5 (1.9) | 27 (10.4) | |
| Encounter Documentation | Case Criteria | ||
| Opioid Overdose | No Opioid Overdose | 0.57 (0.47, 0.67) | |
| Opioid Overdose | 118 (45.6) | 19 (7.3) | |
| No Opioid Overdose | 36 (13.9) | 86 (33.2) | |
| Clinical Impression | Encounter Documentation | ||
| Opioid Overdose | No Opioid Overdose | 0.61 (0.53, 0.69) | |
| Opioid Overdose | 131 (50.6) | 19 (7.3) | |
| No Opioid Overdose | 5 (1.9) | 72 (27.8) | |
| Insufficient Information | 1 (0.4) | 31 (12.0) | |
Discussion
We developed and assessed a novel case criteria classification scheme for pharmaceutical opioid and heroin overdose. Our classification scheme was founded on prior literature, refined by a multi-disciplinary group of clinical and research experts, and evaluated in two distinct health systems serving demographically diverse populations. Although opioid overdose may appear to be a relatively clear clinical syndrome in practice, it can be interpreted on a spectrum from over-sedation to death. We therefore believe our standardized classification scheme has the potential to increase validity and reproducibility across future epidemiological studies and intervention trials.
As the incidence of overdose continues to increase,16 it is imperative that overdose be measured as a primary or secondary outcome in studies of opioid use or opioid use disorders. However, individual studies seldom have populations large enough to achieve adequate statistical power for overdose outcomes. This implies that multi-site studies or meta-analyses using a standard overdose definition are needed. For example, in a meta-analysis of extended-release injectable naltrexone that considered opioid overdose as a potential adverse event of naltrexone treatment,17 Jarvis and colleagues concluded that there was inconsistency and a lack of rigor in how opioid overdose was assessed and reported by most existing observational studies and trials. Thus, the evidence on whether extended-release naltrexone is associated with an increased risk of opioid overdose is inconclusive. A rigorous, standardized case definition would enhance the ability to interpret findings across studies, providing a stronger evidence base to inform clinical guidelines and care.
When we examined a range of diagnostic codes expected to have variable confirmation rates, confirmation rates across the three approaches were similar. However, there was substantial discordance in the cases confirmed across approaches. Compared with encounter documentation and clinical impression, there were discordant findings in 21.2% to 26.6% of the cases classified using case criteria. Such discrepancies suggest that each approach may be subject to differing levels of misclassification bias, which could be differential or non-differential with respect to exposure. Without a true “gold standard” definition for overdose, the potential impact of the misclassification on results from overdose research studies cannot be thoroughly evaluated.
Our case classification scheme represents an initial step towards a consensus definition for research, but we suggest researchers carefully consider availability and completeness of data prior to selecting this method to ascertain cases. The case criteria classification scheme was designed for use in health systems research using detailed information commonly recorded by health professionals in medical records. We found that some potential overdose encounters had insufficient data to classify events using clinical criteria. This suggests that the scheme may be difficult to implement in community settings or other settings with few medical records available. In these settings, encounter documentation may be more appropriate.
Further, we refined the scheme to reduce false negatives by accommodating limitations in real-world clinical documentation, such as poor documentation of pinpoint pupils and specific response to naloxone. In other health systems with different documentation patterns, it may be appropriate to modify the scheme to include pinpoint pupils as a major criteria or require response to naloxone as part of the third major criteria. Such modifications would not detract from the goal of enhancing rigorous overdose reporting. Finally, we used ICD-9-CM codes to identify potential events; our scheme should be evaluated using ICD-10-CM codes.
In our medical record reviews, we systematically guided physicians through a medical record review of potential overdose cases to identify the case criteria. Although rigorous, physician-led medical record reviews tend to be resource-intensive. Future research should focus on how to refine the classification scheme so that it can be used by non-experts to reliably abstract the pertinent overdose-defining criteria from the medical records.
Future studies could formally validate the accuracy and reliability of the classification scheme. For instance, in a prospective study, potential overdose cases presenting to care could undergo systematic testing and documentation by a trained specialist independent of the treating medical professionals. This would lead to a registry of confirmed overdose cases, which could be compared with cases identified using our classification scheme in context of routine clinical care and documentation practices. In such a study, sensitivity and specificity could be calculated to assess accuracy and performance.
Our case criteria classification scheme represents a novel approach to confirm opioid overdoses in epidemiological and intervention studies that has the potential to improve the scientific rigor and reproducibility of future overdose research.
Acknowledgements
We wish to thank Shane Mueller, MSW for project coordination and data collection; Ted Palen, MD, PhD, Ruth Bedoy, and Melanie Stowell, MSc for data collection; Kerry Broderick, MD, Jason Hoppe, DO, Joseph Frank, MD, MPH, and Andrew Monte, MD for providing expertise and feedback on overdose definitions; and Kristin Breslin, MS for providing data and analysis assistance.
Funding: Funding for this study was provided by the National Institute on Drug Abuse of the National Institutes of Health under Award Number R34DA035952, R01DA042059 and the Colorado Clinical & Translational Sciences Institute (CCTSI) with the Development and Informatics Service Center (DISC) grant support (NIH/NCRR Colorado CTSI Grant Number UL1 RR025780). The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Appendix 1.
International Classification of Diseases-9 Clinical Modification (ICD-9-CM) Diagnostic Codes Used to Identify Potential Overdoses
| ICD-9-CM Diagnostic Code Description | ICD-9-CM Code |
|---|---|
| Codes which specifically mention opioids | |
| Poisoning by opiates and related narcotics | 965.0 |
| Poisoning by opium (alkaloids), unspecified | 965.00 |
| Poisoning by heroin | 965.01 |
| Poisoning by methadone | 965.02 |
| Poisoning by other opiates | 965.09 |
| Accidental poisoning by heroin | E850.0 |
| Accidental poisoning by methadone | E850.1 |
| Accidental poisoning by other opiates | E850.2 |
| Codes which did not specifically mention opioids | |
| Poisoning by unspecified analgesic and anti-pyretic | 965.9 |
| Analgesics, antipyretics, and antirheumatics | E950.0 |
| Other sedatives and hypnotics | E950.2 |
| Tranquilizers and other psychotropic agents | E950.3 |
| Other specified drugs and medicaments | E950.4 |
| Unspecified drug or medicament | E950.5 |
| Poisoning by analgesics, antipyretics, and antirheumatics, undetermined whether accidentally or purposely inflicted | E980.0 |
| Poisoning by other sedatives and hypnotics, undetermined | E980.2 |
| Poisoning by tranquilizers and other psychotropic agents, undetermined | E980.3 |
| Poisoning by other specified drugs and medicinal substances, undetermined | E980.4 |
| Poisoning by unspecified drug or medicinal substance, undetermined | E980.5 |
Appendix 2.
Clinical characteristics of definite or probable opioid overdoses based on case criteria classification scheme, clinical impression and encounter documentation
| Characteristics | Case criteria classification scheme (n=154) | Clinical impression (n=150) | Encounter documentation (n=137) |
|---|---|---|---|
| Intent by abstractor judgment | |||
| Accidental | 90 (58.4) | 94 (62.7) | 86 (62.8) |
| Undetermined | 41 (26.6) | 36 (24.0) | 30 (21.9) |
| Intentional/Suicide | 20 (13.0) | 20 (13.3) | 19 (13.9) |
| Not an overdose | 3 (2.0) | 0 | 2 (1.5) |
| Opioid responsible for the overdosea | |||
| Heroin | 60 (39.0) | 67 (44.7) | 61 (44.5) |
| Oxycodone | 34 (22.1) | 33 (22.0) | 30 (21.9) |
| Morphine | 20 (13.0) | 20 (13.3) | 21 (15.3) |
| Hydrocodone | 15 (9.7) | 19 (12.7) | 19 (13.9) |
| Methadone | 17 (11.0) | 15 (10.0) | 16 (11.7) |
| Other | 24 (15.6) | 24 (16.0) | 23 (16.8) |
| None | 10 (6.5) | 0 | 1 (0.7) |
| Other contributing drugs/substancesa | |||
| Benzodiazepines | 36 (23.4) | 35 (23.3) | 30 (21.9) |
| Alcohol | 25 (16.2) | 24 (16.0) | 23 (16.8) |
| Cocaine | 15 (9.7) | 13 (8.7) | 10 (7.3) |
| Marijuana | 7 (4.6) | 6 (4.0) | 5 (3.6) |
| Selective serotonin reuptake inhibitors | 4 (2.6) | 5 (3.3) | 5 (3.6) |
| Others | 45 (29.2) | 34 (22.7) | 33 (23.4) |
| At least one other substance | 86 (55.8) | 77 (51.3) | 72 (52.6) |
| Outcome was death | 3 (2.0) | 3 (2.0) | 2 (1.5) |
| Highest acuity setting of care | |||
| Intensive care unit | 42 (27.3) | 31 (20.7) | 30 (21.9) |
| Inpatient | 43 (27.9) | 39 (26.0) | 35 (25.5) |
| Observation unit | 34 (22.1) | 35 (23.3) | 33 (24.1) |
| Emergency Department | 33 (21.4) | 42 (28.0) | 36 (26.3) |
| Paramedic | 1 (0.6) | 1 (0.7) | 1 (0.7) |
| Not applicable or missing | 1 (0.6) | 2 (1.3) | 2 (1.5) |
| Disposition | |||
| Discharged home | 113 (73.4) | 112 (74.7) | 103 (75.2) |
| Transfer to psychiatric unit | 14 (9.1) | 13 (8.7) | 14 (10.2) |
| Jail or Prison | 5 (3.3) | 6 (4.0) | 4 (2.9) |
| Other | 16 (10.4) | 12 (8.0) | 11 (8.0) |
| Not applicable or missing | 6 (3.9) | 7 (4.7) | 5 (3.6) |
| Referred to:a | |||
| Outpatient psychiatric services | 20 (13.0) | 21 (14.0) | 20 (14.6) |
| Outpatient substance use disorder treatment | 9 (5.8) | 11 (7.3) | 10 (7.3) |
| Patient instructed to call for: | |||
| Primary care appointment | 36 (23.4) | 32 (21.3) | 27 (19.7) |
| Outpatient psychiatric services | 6 (3.9) | 6 (4.0) | 4 (2.9) |
| Outpatient substance use disorder treatment | 20 (13.0) | 19 (12.7) | 15 (11.0) |
| Other | 13 (8.4) | 12 (8.0) | 12 (8.8) |
| Not applicable or missing | 66 (42.9) | 63 (42.0) | 61 (44.5) |
Patients may belong to more than 1 category
Appendix 3:
Reasons for discrepancies across approaches
| Discrepancy | Reasons |
|---|---|
| Case criteria classification: yes Clinical impression: no |
• Other medications commonly involved (e.g., digoxin, lithium, ibuprofen, benzodiazepine) and opioids not considered the primary cause of the event, even if opioids involved • Clinical impression was uncertain: “probably not” an opioid overdose |
| Case criteria classification: no Clinical impression: yes |
• Limited documentation in encounters prevented meeting the criteria but historical features strongly suggested heroin overdose, e.g., in person who had a history of intravenous heroin use or had repeated heroin overdoses • Patients treated in pre-hospital setting and recovered by time seen in emergency department • Patients being observed after ingestion of too many pills |
| Case criteria classification: yes Encounter documentation: no |
• Other medications commonly involved (e.g., tramadol, valproic acid, hydroxyzine, zolpidem) therefore opioids not the primary cause of the event even if involved |
| Case criteria classification: no Encounter documentation: yes |
• Limited documentation in encounters prevented meeting the criteria but opioid/heroin overdose was documented • Family member reported the event was an overdose but the patient showed no signs by the time they were seen in the emergency department, patient being observed • Patients treated in pre-hospital setting and recovered by time seen in emergency department |
| Clinical impression: yes Encounter documentation: no |
• Missing key documents such as paramedic report, emergency department note, or hospital discharge summary • Documentation in another part of the medical record, such as a telephone note or a pre-authorization but could not be verified in the encounter itself |
| Clinical impression: no Encounter documentation: yes |
• Polypharmacy • Patient reported they had an overdose, e.g., took too many opioid medications, but did not show signs of overdose |
References
- 1.Rudd RA, Seth P, David F, Scholl L. Increases in Drug and Opioid-Involved Overdose Deaths - United States, 2010–2015. MMWR Morb Mortal Wkly Rep. 2016;65(5051):1445–1452. [DOI] [PubMed] [Google Scholar]
- 2.Special Advisory Committee on the Epidemic of Opioid Overdose. National report: Apparent opioid-related deaths in Canada (January 2016 to June 2017) Web-based Report. 2017; https://www.canada.ca/en/public-health/services/publications/healthy-living/apparent-opioid-related-deaths-report-2016-2017-december.html. Accessed March 23, 2018.
- 3.Office of National Statistics. Deaths Related to Drug Poisonings in England and Wales, 2016 Registrations. 2017; https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/deathsrelatedtodrugpoisoninginenglandandwales/2016registrations. Accessed March 23, 2018.
- 4.Larochelle MR, Liebschutz JM, Zhang F, Ross-Degnan D, Wharam JF. Opioid prescribing after nonfatal overdose and association with repeated overdose: a cohort study. Ann Intern Med. 2016;164(1):1–9. [DOI] [PubMed] [Google Scholar]
- 5.Dunn KM, Saunders KW, Rutter CM, et al. Opioid prescriptions for chronic pain and overdose: a cohort study. Ann Intern Med. 2010;152(2):85–92. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Rowe C, Vittinghoff E, Santos GM, Behar E, Turner C, Coffin PO. Performance measures of diagnostic codes for detecting opioid overdose in the emergency department. Acad Emerg Med. 2017;24(4):475–483. [DOI] [PubMed] [Google Scholar]
- 7.Reardon JM, Harmon KJ, Schult GC, Staton CA, Waller AE. Use of diagnosis codes for detection of clinically significant opioid poisoning in the emergency department: A retrospective analysis of a surveillance case definition. BMC Emerg Med. 2016;16:11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Chung CP, Callahan ST, Cooper WO, et al. Development of an algorithm to identify serious opioid toxicity in children. BMC Res Notes. 2015;8:293. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Green CA, Perrin NA, Janoff SL, Campbell CI, Chilcoat HD, Coplan PM. Assessing the accuracy of opioid overdose and poisoning codes in diagnostic information from electronic health records, claims data, and death records. Pharmacoepidemiol Drug Saf. 2017;26(5):509–517. [DOI] [PubMed] [Google Scholar]
- 10.Pfister GJ, Burkes RM, Guinn B, et al. Opioid overdose leading to intensive care unit admission: Epidemiology and outcomes. J Crit Care. 2016;35:29–32. [DOI] [PubMed] [Google Scholar]
- 11.Hambidge SJ, Ross C, Shoup JA, et al. Integration of data from a safety net health care system into the Vaccine Safety Datalink. Vaccine. 2017;35(9):1329–1334. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–381. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Injury Surveillance Workgroup (ISW7). Consensus recommendations for national and state poisoning surveillance. Atlanta, GA: The Safe States Alliance; April 2012. [Google Scholar]
- 14.National Center for Health Statistics, Centers for Medicare and Medicaid Services. International classification of diseases, Ninth Revision, Clinical Modification, Fifth Edition. Washington: US Government Printing Office; 1999. [Google Scholar]
- 15.Cochran W Sampling Techniques. New York, NY: John Wiley & Sons; 1977. [Google Scholar]
- 16.National Center for Health Statistics. Provisional counts of drug overdose deaths, as of 8/6/2017. Centers for Disease Control and Prevention; 2017. [Google Scholar]
- 17.Jarvis BP, Holtyn AF, Subramaniam S, et al. Extended-release injectable naltrexone for opioid use disorder: a systematic review. Addiction. 2018;113(7):1188–1209. [DOI] [PMC free article] [PubMed] [Google Scholar]