Development of an algorithm to identify inpatient opioid‐related overdoses and oversedation using electronic data

Abstract

Purpose

To facilitate surveillance and evaluate interventions addressing opioid‐related overdoses, algorithms are needed for use in large health care databases to identify and differentiate community‐occurring opioid‐related overdoses from inpatient‐occurring opioid‐related overdose/oversedation.

Methods

Data were from Kaiser Permanente Northwest (KPNW), a large integrated health plan. We iteratively developed and evaluated an algorithm for electronically identifying inpatient overdose/oversedation in KPNW hospitals from 1 January 2008 to 31 December 2014. Chart audits assessed accuracy; data sources included administrative and clinical records.

Results

The best‐performing algorithm used these rules: (1) Include events with opioids administered in an inpatient setting (including emergency department/urgent care) followed by naloxone administration within 275 hours of continuous inpatient stay; (2) exclude events with electroconvulsive therapy procedure codes; and (3) exclude events in which an opioid was administered prior to hospital discharge and followed by readmission with subsequent naloxone administration. Using this algorithm, we identified 870 suspect inpatient overdose/oversedation events and chart audited a random sample of 235. Of the random sample, 185 (78.7%) were deemed overdoses/oversedation, 37 (15.5%) were not, and 13 (5.5%) were possible cases. The number of hours between time of opioid and naloxone administration did not affect algorithm accuracy. When “possible” overdoses/oversedations were included with confirmed events, overall positive predictive value (PPV) was very good (PPV = 84.0%). Additionally, PPV was reasonable when evaluated specifically for hospital stays with emergency/urgent care admissions (PPV = 77.0%) and excellent for elective surgery admissions (PPV = 97.0%).

Conclusions

Algorithm performance was reasonable for identifying inpatient overdose/oversedation with best performance among elective surgery patients.

KEY POINTS.

• Inpatient opioid‐related overdoses/oversedation can be identified with reasonable accuracy by the algorithm developed in this study, using electronically available health care data.

• Overdose/oversedation events that occur in inpatient settings can be differentiated from overdoses occurring in the community that are subsequently treated in health care settings.

1. INTRODUCTION

To help address the problem of opioid‐related overdose and death,1, 2, 3, 4 methods are needed to accurately identify and differentiate overdoses that occur outside hospitals but which are subsequently treated in hospitals, versus overdose/oversedation that occurs following administration of opioids in inpatient settings. Such differentiation is necessary to adequately conduct public health surveillance, target interventions, and assess outcomes of inpatient quality improvement initiatives addressing oversedation.5 The latter is particularly true given the complexity of providing appropriate inpatient pain management when conflicting goals must be balanced, including preventing negative consequences of inadequately treated perioperative pain,6 reducing risk for opioid oversedation,7, 8, 9 and maintaining patient satisfaction.7 Large health care databases offer opportunities to develop, assess, and implement the tools necessary to achieve these goals.

The work reported here was conducted as part of a comprehensive study that developed algorithms to identify and classify opioid‐related overdoses using electronic health records (EHRs) and claims databases (see Green et al, companion paper10). Findings of the larger study showed that the algorithm developed identified few inpatient opioid‐overdose events.10 While this was a positive finding regarding accuracy for detecting opioid‐related overdoses that occur in the community and are treated in health care settings, it suggested that a new tool was needed to detect inpatient overdoses/oversedation. The aim of the work reported here was to develop and test such an algorithm.

2. METHODS

2.1. Study population and data sources

The study population included all patients with hospitalizations, whether admitted through emergency or urgent care, or for elective surgery, in Kaiser Permanente Northwest (KPNW) facilities from 1 January 2008 to 31 December 2014. KPNW is an integrated health plan serving Oregon and Southwest Washington, providing comprehensive inpatient and outpatient medical care. At study end, KPNW was serving about 500 000 members that broadly reflected the service‐area population.

Data sources included health plan administrative records, admission type (eg, elective surgery or emergency/urgent care departments), EHRs, and electronic inpatient clinical records including clinical laboratory test results and clinical notes from patient encounters. Records included procedures completed and medications prescribed and administered. Medication administration records, including dates and times, were linked to inpatient encounter records. All inpatient departments were included, as were emergency and urgent care departments, because opioids are administered by professionals in all these settings.

2.2. Algorithm development

Initial development began with three hypothesized algorithms for identifying overdose/oversedation among individuals receiving inpatient medical care: (1) naloxone administered following inpatient opioid administration, (2) primary opioid poisoning or opioid‐withdrawal diagnostic codes following inpatient opioid administration, and (3) secondary opioid‐overdose or opioid‐withdrawal diagnostic codes following opioid administration. Hypothesized algorithms were based on work in the parent study that validated methods for identifying and classifying opioid‐related overdoses,10 previous related work,11 and existing literature addressing opioid‐related overdoses and oversedation (see, for example, previous studies12, 13, 14, 15). Using KPNW's electronic databases, we randomly selected 10 to 15 events using each method and audited the corresponding medical records to assess whether or not the event was an overdose/oversedation.

The approaches that relied on diagnostic codes were found to lack accuracy and reliability because events indicative of opioid overdose or withdrawal based upon chart review were typically not coded for overdose/oversedation, even in cases when clinical notes clearly indicated overdose/oversedation was related to administered opioids.

We also found that the proposed method using naloxone administration to identify opioid‐overdose/‐oversedation events had limitations. First, naloxone was routinely used as part of electroconvulsive therapy (ECT), resulting in false positives for overdose/oversedation. Second, events in which inpatient opioid administration was followed by hospital discharge and readmission with naloxone administration were typically overdoses occurring in the community. Moreover, we found that some naloxone administrations occurred many days following opioid administration, most often when transdermal patches, drips, or pumps were administered, making administration periods difficult to determine when only initial administration time was recorded. This raised the question of whether or not a time‐based cut‐point might be necessary to identify actual overdose/oversedation events; that is, whether algorithm performance would deteriorate as the number of hours between opioid administration and subsequent naloxone administration increased.

On the basis of this preliminary assessment, we moved the following algorithm forward for testing: (1) Identify all inpatient events (including those occurring in emergency department/urgent care visits) during which naloxone was administered following opioid administration within 275 hours of opioid administration; (2) exclude events with ECT procedure codes; and (3) exclude events with inpatient opioid administration followed by hospital discharge, then readmission followed by naloxone administration. We evaluated performance of this algorithm in a larger sample, stratified by the elapsed time between opioid and naloxone administration. The unit of analysis was the outcome of naloxone administration, though chart auditors reviewed all available data for the inpatient stay as needed to make a determination regarding outcome. Data S1 contains instructions for implementing the algorithm.

2.3. Sampling

In total, the algorithm identified 871 candidate overdose/oversedation events, ranging from 0 to 275 hours between opioid and naloxone administration with the exception of one outlier (naloxone administration 480 h following opioid administration). That outlier was dropped from the study. Among the remaining 870 events, there were 823 individuals with one event, 20 individuals with two events, and one individual each with three and four events. Data were skewed, with mean naloxone administration 9.87 hours (SD = 24.38) following opioid administration (median = 4 h; mode ≤ 1 h). We chart audited a random sample of 235 (27%) of the 870 suspected algorithm‐identified events.

2.4. Chart audits

Trained chart auditors with extensive experience with studies of opioids and opioid‐related overdoses reviewed the 235 randomly sampled candidate cases to determine if algorithm‐identified events were actual inpatient cases of opioid‐related overdose/oversedation. All auditors had worked on the parent study, which included training and collaboration with clinicians for determining whether events were opioid‐related overdoses or not.10 The lead auditor, who supervised audits of more than 5000 putative opioid‐related overdoses in the parent study, was responsible for training and meeting with auditors to resolve any discrepancies. Auditors were not physicians, but determinations were based solely on clinician documentation in the record, not clinical assessment. Clinicians were available for consultation when it was difficult to make a determination. For each event, two chart auditors not involved in audits used for algorithm development reviewed all inpatient, emergency room and urgent care visits for the patient on the identified date and completed keyword searches for “Narcan” and “naloxone” within hospital discharge summaries to determine day, time, and circumstances surrounding naloxone administration. If keyword searches did not produce adequate information, auditors opened the full encounter report and reviewed all history and physical information, emergency room/urgent care notes, progress notes, and any additional documentation. Charts were reviewed until adequate information was collected to make the following determinations on the basis of clinician documentation of symptoms and response to naloxone: (1) confirmed inpatient opioid overdose/oversedation, (2) possible inpatient opioid overdose/oversedation, or (3) not an opioid overdose/oversedation. Discrepancies between chart auditors were resolved with discussion and review by additional study staff. Specific criteria for auditor determinations were as follows:

• Confirmed opioid‐related inpatient overdose/oversedation: Chart documented that patient was overly sedated, received too large a dose of opioids as part of the anesthesia process, experienced an overdose, or became somnolent/apneic after opioid administration. Effects of opioids were clearly documented as having been reversed by naloxone.

• Possible opioid‐related inpatient overdose/oversedation: Naloxone was administered, but auditors were unable to determine if naloxone improved the condition, or there was uncertainty as to whether the patient's medical condition improved as a result of the naloxone or as a result of concomitant medications/treatments given at the same time.

• Not an opioid‐related inpatient overdose/oversedation: Naloxone clearly did not resolve the problem, or documentation indicated the event was an opioid‐related adverse drug reaction (eg, itching after opioid administration), not overdose/oversedation.

3. RESULTS

Of the 235 algorithm‐identified suspect overdose/oversedation events, 185 (78.7%) were determined to be confirmed cases; 37 (15.5%) were determined not to be an overdose/oversedation; and 13 (5.5%) were coded “possible” cases. All had at least some descriptive clinical information. Table 1 shows results in two ways: (1) including “possible” cases as confirmed overdose/oversedation and (2) including “possible” cases with events deemed not to be overdose/oversedation. The table includes overall results for each method and by admission type (elective surgery admission and emergency or urgent care department admission).

Table 1.

Results of chart audits for inpatient stays identified as suspected overdose/oversedation using medication administration records, by admission type, and by different approaches to addressing events deemed “possible” overdoses by chart auditors

	Confirmed Overdose	Not an Overdose	Total	PPV
				95% Confidence Interval
Elective surgery admissions	73	3	76	0.96
				0.88‐0.99
Emergency and urgent care department encounters	112	34	146	0.77
				0.69‐0.83
All inpatient encounters	185	37	222	0.83
				0.78‐0.88
“Possible” overdoses coded as “overdose”
Elective surgery admissions	86	3	89	0.97
				0.90‐0.99
Emergency and urgent care department encounters	112	34	146	0.77
				0.69‐0.83
All inpatient encounters	198	37	235	0.84
				0.79‐0.89
“Possible” overdoses coded as “not an overdose”
Elective surgery admissions	73	16	89	0.82
				0.72‐0.89
Emergency and urgent care department encounters	112	34	146	0.77
				0.69‐0.83
All inpatient encounters	185	50	235	0.79
				0.73‐0.84

Abbreviation: PPV, positive predictive value.

Across admission types, when events coded as “possible” overdose/oversedation were included with confirmed events, positive predictive value (PPV) was very good (84%; 95% CI, 0.79‐0.89). Performance was reasonable (PPV = 77%; 95% CI, 0.69‐0.83) for emergency and urgent care admissions and excellent (PPV = 97%; 95% CI, 0.90‐0.99) for elective surgery admissions. When “possible” events were coded conservatively as “not opioid‐related overdose/oversedation,” overall PPV = 79% (95% CI, 0.73‐0.84), and for elective surgery, PPV = 82% (95% CI, 0.72‐0.89). Algorithm performance was best among elective surgery admissions. Increasing hours between time of opioid administration and naloxone administration up to 275 hours did not affect the likelihood that algorithm‐identified events were overdoses/oversedation.

4. DISCUSSION

This preliminary evaluation of a simple algorithm to identify inpatient opioid‐related overdose/oversedation using electronic data found that the algorithm performed well. Performance was best for elective surgeries; performance among patients with admissions through emergency and urgent care departments was adequate. Although there is a wealth of data that can be mined from electronic records, and a multitude of variables that could be used to identify opioid overdose/oversedation events, our goal was to develop a simple, parsimonious algorithm that could be ported to other sites with similar data that would be easily available. Although research developing a more complex algorithm would likely have value, it would also be more likely to produce an algorithm with limited utility in other systems.

The finding that there was little overlap of overdose/oversedation resulting from inpatient opioid administration with overdoses identified using the diagnostic code‐based algorithm we developed for community settings (see companion paper)10 indicates that events identified by each algorithm are nearly mutually exclusive. The finding that performance did not change as number of hours increased between opioid administration and naloxone administration supports the value of the algorithm for identifying events, whether or not they appear close in time to recorded opioid administration.

We are aware of one previous study that assessed the value of using naloxone administration alone to identify oversedation among postoperative patients, limiting the evaluation to within 72 hours of surgery.12 To our knowledge, the work reported here, although preliminary in nature, is the most comprehensive assessment of a method for identifying opioid‐related overdose/oversedation events resulting from administration of opioids in any inpatient setting, across a longer elapsed time period between opioid administration and naloxone administration. In short, taken together, our algorithms are able to identify and differentiate community‐occurring overdoses treated in health care settings10 from overdoses/oversedation events that are the result of opioids administered in inpatient settings. As such, they show promise for public health surveillance and assessment of interventions to reduce community‐occurring overdoses and inpatient oversedation.

4.1. Limitations

Our work is preliminary in nature, and we did not have a sample of individuals without identified events that would have allowed calculation of sensitivity, specificity, and negative predictive value. We also included some cases that had been reviewed in early efforts to develop the inpatient algorithm. These events were all independently rereviewed by different auditors, but this remains a weakness that could have led to inflated model performance. Our model identifies, but does not differentiate, oversedation and overdose. Additional research identifying clinical precursors to oversedation and overdose could differentiate these types of events and aid quality improvement efforts. Events identified by the existing algorithm provide an opportunity to complete such research. This algorithm only identifies cases involving naloxone administration when some clinical situations resulting from oversedation, such as those leading to intubation, may not include naloxone administration. Such missed cases, if identified in future research, would negatively affect algorithm sensitivity and indicate a source of bias. In addition, though naloxone may be administered without effect in some cases, even when opioids contribute to clinical decline, the algorithm in its current form would correctly identify these cases as opioid‐related oversedation. Future research is warranted to assess sensitivity, specificity, and negative predictive value of the existing algorithm. With further validation, this simple algorithm should be useful in most hospital settings with electronic records.

ETHICS STATEMENT

This study was reviewed, approved, and monitored by the Kaiser Permanente Northwest Institutional Review Board for the Protection of Human Subjects.

CONFLICT OF INTEREST

All authors have received research funding from the Opioid Postmarketing Requirement Consortium (OPC), which funded this project. This project was conducted as part of a Food and Drug Administration (FDA)–required postmarketing study of extended‐release and long‐acting opioid analgesics (https://www.fda.gov/downloads/Drugs/DrugSafety/InformationbyDrugClass/UCM484415.pdf). The OPC is composed of companies that hold NDAs of extended‐release and long‐acting opioid analgesics and, at the time of submission, the included the following companies: Allergan; Assertio Therapeutics, Inc; BioDelivery Sciences, Inc; Collegium Pharmaceutical, Inc; Daiichi Sankyo, Inc; Egalet Corporation; Endo Pharmaceuticals, Inc; Hikma Pharmaceuticals USA Inc; Janssen Pharmaceuticals, Inc; SpecGX, LLC; Pernix Therapeutics Holdings, Inc; Pfizer, Inc; and Purdue Pharma, LP. The study was designed in collaboration between OPC members and independent investigators with input from FDA. Investigators maintained intellectual freedom in terms of publishing final results. This study was registered with ClinicalTrials.gov as study NCT02667197 on 28 January 2016.

Dr Green, Mr Sapp, and Ms Janoff received funding from Purdue Pharma, LP for related research, prior to the present study. Dr Green has provided research consulting to the OPC. Kaiser Permanente Center for Health Research (KPCHR) staff were responsible for study design, analysis, publication decisions, publication content, and manuscript preparation, though OPC members provided comments on the manuscript. KPCHR authors made all final decisions about manuscript content. Drs DeVeaugh‐Geiss and Coplan were employees of Purdue Pharma, LP at the time this research was completed.

Supporting information

Data S1. Supporting information

Green CA, Hazlehurst B, Brandes J, et al. Development of an algorithm to identify inpatient opioid‐related overdoses and oversedation using electronic data. Pharmacoepidemiol Drug Saf. 2019;28:1138–1142. 10.1002/pds.4797