Identifying naloxone administrations in electronic health record data using a text-mining tool

Abstract

Background:

Effective and efficient methods are needed to identify naloxone administrations within electronic health record (EHR) data to conduct overdose surveillance and research. The objective of this study was to develop and validate a text-mining tool to identify naloxone administrations in EHR data.

Methods:

Clinical notes stored in databases between January 2017 and March 2018 were used to iteratively develop a text-mining tool to identify naloxone administrations. The first iteration of the tool used broad search terms. Then, after reviewing clinical notes of overdose encounters, we developed a list of phrases that described naloxone administrations to inform iteration two. While validating iteration two, additional phrases were found, which were then added to inform the final iteration. The comparator was an administrative code query extracted from the EHR. Medical record review was used to identify true positives. The primary outcome was the positive predictive values (PPV) of the second iteration, final iteration, and administrative code query.

Results:

Iteration two, the final iteration, and the administrative code had PPVs of 84.3% (95% confidence interval [CI] 78.6–89.0%), 83.8% (95% CI 78.6–88.2%), and 57.1% (95% CI 47.1–66.8%), respectively. Both iterations of the tool had a significantly higher PPV than the administrative code (P<0.001).

Conclusions:

A text-mining tool improved the identification of naloxone administrations in EHR data from less than 60% with the administrative code to greater than 80% with both versions of the tool. Text-mining tools can inform the use of more sophisticated informatics methods, which often require significant time, resource, and expertise investment.

Introduction

Over 47,000 Americans died as a result of opioid overdose in 2017.¹ Overdoses of prescribed and illicit opioids account for the highest number of nonfatal drug poisonings resulting in emergency department (ED) visits and hospitalizations among all drug classes.² In response, the United States Surgeon General and other public health officials have called for widespread distribution of naloxone,^3,4 a competitive mu-opioid receptor antagonist used to reverse the central nervous system and respiratory depression caused by an opioid overdose.

Retrospectively identifying naloxone administrations in electronic health record (EHR) data may facilitate clinical surveillance efforts, allow for evaluations of overdose education and prevention programs, and support observational or interventional research in patients who are prescribed opioids. However, for longitudinal cohort studies, identifying patients in the EHR who have been administered naloxone – rather than those who have received a naloxone prescription – is challenging. Inpatient medication administration records (MAR) have been used,^5–7 but these records may be inaccessible by researchers, unreliable if the administration is not documented on the MAR, or ineffective if naloxone is administered before hospital arrival. Alternatively, administrative billing codes may also be used. However, these codes have consistently demonstrated poor validity to identify other medication exposures (positive predictive values ranging 49–61%), and to the best of our knowledge, the accuracy of such codes to identify naloxone administrations has never been evaluated.^8–10 Using these codes could therefore lead to biased estimates of real-world naloxone use.

Identifying naloxone administrations in the EHR is further complicated by the fact that healthcare providers often document medication administrations such as naloxone in free-text clinical notes, which are not discrete fields and thus difficult to extract from the EHR. Biomedical informatics techniques such as text mining offer an attractive opportunity to identify information embedded within information-rich clinical notes.^11,12 Simple text-mining techniques may represent an alternative or necessary precursor to more sophisticated natural language processing (NLP) or machine learning methods, which may require significant time, resource, and expertise investment to develop and implement. These resources may not be available to agencies engaged in evaluating public health programs. As such, text mining methods have been widely used for health services research, including to extract depressive symptoms,¹³ evaluate pharmacogenomic associations from published literature,¹⁴ and identify adverse drug events.¹⁵

However, to date, approaches to query free-text clinical notes to identify naloxone administrations have not been developed. For this report, we developed and validated a text-mining tool to identify naloxone administrations within clinical notes contained in EHR data.

Methods

Study design and population

We identified a cohort of individuals enrolled in the Kaiser Permanente Colorado (KPCO) health plan between January 2017 and March 2018. To identify naloxone administrations broadly, no population exclusion criteria were used. KPCO is an integrated healthcare delivery system that cares for over 630,000 members residing within urban and rural areas of Colorado. Data from hospital, ED, telephone, medical office encounters, medication dispenses, and procedures are stored within KPCO’s electronic administrative and claims databases. Clinical notes are stored in KPCO’s EHR. This study was approved by the Kaiser Permanente Colorado Institutional Review Board.

Overview of procedures

A text-mining tool was developed and refined iteratively using encounters within the EHR (Figure 1). Encounters for the text-mining tool iterations were extracted from the EHR, then the encounters were de-duplicated by medical record number and encounter date to avoid multiple results for a single encounter. No additional data cleansing, integration, or noise correction methods were undertaken. Psychiatry notes were not included. Each resulting encounter was validated with medical record review to confirm naloxone administration.

Figure 1:

Development of text-mining tool

After development of the text-mining tool, encounters with a code that identified naloxone administration per the Healthcare Common Procedure Coding System (HCPCS) were extracted, de-duplicated, and reviewed in the medical record using the same process to serve as a comparative standard to the text-mining tool.

Development of the text-mining tool

The development of the tool occurred in four steps as described in greater detail below.

Step One of Tool Development: Iteration One

Iteration one of the tool consisted of all encounters (i.e., hospitalizations, ED visits, office visits, telephone notes, e-mails) containing the phrases, “Narcan,” “naloxone,” or “NLX” (Figure 1). The number of results from this query (greater than 3,400) was far in excess of the expected number of overdoses or naloxone administrations within the time period. To confirm, a random subset of 100 encounters were selected for medical record review to identify true and false positives.

For this step of tool development and all subsequent steps, one investigator reviewed medical records to identify true and false positives. The investigator determined whether naloxone was administered in each encounter by reviewing ED notes, hospital discharge summaries, inpatient progress notes, medication administration records, and other clinical texts to find documentation that naloxone was given to the patient by emergency services personnel, bystanders, caregivers, law enforcement personnel, or health professionals. The investigator recorded the person or persons who administered naloxone to the patient in each encounter to quantify the tool’s ability to capture naloxone administrations in healthcare and non-healthcare settings.

If naloxone was administered, the encounter was classified as a “true positive” (TP); if naloxone was not administered, the encounter was classified as a “false positive” (FP). A separate category of “indeterminant” was created if the investigator could not determine whether naloxone was given in the encounter, or if there was no information available in the EHR. Examples of phrases that deemed each encounter as a TP or FP are shown in Table 1.

Table 1:

Examples of text to classify encounters as true positive and false positive.

Classification	Example documentation
True Positive	“She has received Narcan x2 with improvement in mental status.”
	“Narcan in the field, now on Narcan gtt at the hospital.”
	“EMS gave 0.5 mg of Narcan…”
	“With EMS dose of Narcan he is awake alert and conversational.”
False Positive	“Continue Narcan 4 mg prescription as needed for opioid-related overdose.”
	“Patient denied Narcan enroute.”
	“Patient was talking and mentating appropriately although appeared to be demonstrating a toxidrome consistent with heroin. She was not therefore, given Narcan.”
	“Morphine (see comments): Extremely sensitive, very sleepy, received Narcan last time (4 doses of Narcan)”

ED=emergency department; EMS=emergency medical services; gtt=infusion

Of the 100 randomly-selected encounters that underwent medical record review for iteration one, only four encounters were TPs. The remainder (n=96) were FPs because the word “naloxone” was not associated with an administration of the medication but was instead recorded in medication and allergy lists, hospital orders, and specialist recommendation notes for patients on chronic opioid therapy.

Step Two of Tool Development: Targeted phrase development

To pursue a more targeted list of phrases to include in the text-mining tool, all encounters with an International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) code of T40.x (“poisoning by, adverse effect of narcotics and psychodysleptics”) were extracted from the medical record. A random selection of 100 encounters underwent medical record review to identify true naloxone administrations according to the same criteria used in Step 1.

While performing medical record review, the investigator compiled a list of specific phrases that were used by clinicians and health care personnel to document a naloxone administration. The list was consolidated to a targeted list of phrases to develop iteration two (next step) of the text-mining tool.

During this step of the tool development, two patterns were identified that impacted the tool. First, naloxone administrations were documented in inpatient and ED encounters. Second, clinicians referred to naloxone as its branded, trade-name (Narcan^®), and mentions of “naloxone” typically occurred in conjunction with a “Narcan^®” mention in the same encounter, referred to buprenorphine-containing products, or occurred in standardized patient instructions, all which did not reflect an administration. Therefore, the decision was made to restrict subsequent iterations to inpatient or ED encounters and to use the term “Narcan” in all phrases of the text-mining tool.

Step Three of Tool Development: Iteration Two

In step three, for iteration two of the tool, all inpatient and ED records were queried for clinical notes containing any phrase from the list identified during step two. We performed a medical record review of all identified encounters to classify TPs and FPs using according the same criteria used in steps one and two. During medical record review, additional phrases that could also identify naloxone administrations were identified and integrated into the final iteration of the tool.

Step Four of Tool Development: Final Iteration

In step four, for the final iteration of the tool, all inpatient and ED records were queried for clinical notes containing any phrase from the list identified during steps two and three. Each encounter underwent medical record review to identify TPs and FPs according to the same criteria used in the prior steps. During medical record review for this step, no new phrases were identified, so the decision was made to end development of the tool with this iteration.

HCPCS code comparator

Encounters with an administrative code that identified naloxone administration were extracted to serve as a comparison for the text-mining tool. The code in the HCPCS that identifies the administration of “injection, naloxone hydrochloride, per 1 mg” is “J2310.” Each encounter associated with this code underwent medical record review to identify TPs and FPs according to the same criteria in the text-mining tool development.

Positive Predictive Value

To compare the accuracy of the text-mining tool iterations and the HCPCS code, the positive predictive value (PPV) was calculated for iteration two of the text-mining tool, the final iteration of the text-mining tool, and the HCPCS code query. Dividing the number of TP encounters by the total number of encounters yields the PPV ($\text{PPV}=\frac{\mathit{\text{True Positives }}\left(TP\right)}{\left(\mathit{\text{True Positives}}+\mathit{\text{False Positives }}\left[FP\right]\right)}$. Iteration one was not included in the analysis due to the large number of results.

The Wilson Score method was used to calculate 95% confidence intervals (CI) for the PPVs.¹⁶ Two-sample t-tests were used to test for significant differences in the PPV between: 1) Iteration two and the HCPCS code; 2) Final iteration and the HCPCS code; and 3) Iteration two and the final iteration. An alpha level of 0.05 was used for significance. Analyses were performed in SAS v.9.4 (SAS Institute Inc., Cary, NC).

Results

Iteration One

Iteration one broadly included any clinical note that contained “naloxone,” “Narcan,” or “NLX.” A large number of FPs (96 of 100 encounters) resulted for a variety of reasons. Naloxone mentions occurred in medication lists where patients were prescribed naloxone for emergency use at home without a subsequent administration (n=40). Additionally, the word “naloxone” is also associated with buprenorphine-naloxone, a medication used for substance use disorder treatment (n=9). Specialists may also recommend naloxone in consultation notes without a resulting administration of naloxone (n=17). Naloxone is routinely ordered for inpatient or outpatient medical procedures, and a provider may order naloxone for “as needed” use without a subsequent administration (n=28). Finally, one patient had naloxone as an allergy listed on their profile, and for another patient, naloxone was included in the allergy field to describe an “allergy” to an opioid medication from a prior overdose event.

PPV

No encounters were classified as “indeterminant” for the text-mining tool iterations or the HCPCS code. The PPVs of iteration two and the final iteration were similar (Table 2). Applying iteration two of the tool resulted in 204 encounters, 172 of which were TPs (PPV 84.3%, 95% CI 78.6–89.0%). Applying the final iteration resulted in 241 unique encounters, 202 of which were TPs (PPV 83.8%, 95% CI 78.6–88.2%).

Table 2:

Positive predictive values of billing code and text-mining tool to identify naloxone administrations.

Method	Encounters	True Positives	False Positives	PPV (95% CI)
Tool: Iteration Two	204	172a	32a	84.3%a (78.6, 89.0%)
Tool: Final Iteration	241	202a,b	39a,b	83.8%a,b (78.6, 88.2%)
Billing code J-2310	105	60	45	57.1% (47.1, 66.8%)

^aP<0.001 compared to billing code

^bP>0.05 compared to iteration two

CI = confidence interval; PPV = positive predictive value

Within the study period, the HCPCS code was attached to 105 encounters. Of these, 60 encounters were TPs (PPV 57.1%, 95% CI 47.1%, 66.8%; Table 2). Almost half of the encounters were FPs (n=45, 43.9%), most frequently because naloxone may have been ordered, but not administered, as a part of routine clinical practice (e.g., for as-needed reversal of anesthesia in operative settings).

Both the second and final iterations of the tool had a significantly higher PPV than the HCPCS code (P<0.001); whereas, the PPV was not significantly different between iteration two and final iteration of the tool (P>0.05).

Person administering naloxone

Naloxone was administered by a variety of individuals, and each query method varied in detection of who administered naloxone (Figure 2). Iteration two, the final iteration, and the HCPCS code most commonly identified naloxone administrations by emergency medical services personnel (i.e., paramedic, emergency medical technicians) and healthcare personnel (i.e., emergency department or inpatient physicians, nurses, or other personnel).

Figure 2: Party administering naloxone, by query method.

*“Other” includes a combination of Caregiver or bystander only; EMS and healthcare personnel; and caregiver and EMS administrations. EMS = emergency medical services.

Discussion

We developed a text-mining tool that incorporated data queries in conjunction with medical record review to identify naloxone administration. The PPV for a text-mining tool was significantly greater than that of the HCPCS code. The final iteration of the text-mining tool can be used to identify cohorts of patients who may have received naloxone in community, ED, and inpatient settings, allowing clinicians and researchers to evaluate naloxone expansion programs, perform clinical surveillance, or conduct research on real-world opioid and naloxone use.

While we evaluated three iterations of the tool, the development process could have continued past three iterations. The decision to stop refining the tool after the third iteration was largely due to the modest increase in TPs (172 to 202) and slight decrease in PPV (84.3% to 83.8%) observed from the second to final iteration of the tool. This study serves as necessary preparatory work for more sophisticated biomedical informatics techniques, such as NLP, which can more seamlessly integrate the nuances of spelling, grammar, and sentence syntax but require more time and resources. Our method to develop a text-mining tool can be adapted to other clinical settings to identify other types of exposures from free-text clinical notes in the EHR.

As reinforced with the findings of this study, administrative codes have consistently identified medication exposures with poor accuracy (PPV ranging 49–61%).^8–10 The accuracy of such a code for a medication exposure relies upon a medical billing specialist: 1) identifying that the medication was administered from clinical documentation, and 2) entering the correct administrative code that reflects the exact medication administered. There is no administrative code for intranasal naloxone, only injectable naloxone. In contemporary clinical practice, intranasal naloxone is utilized more often than injectable naloxone because it is easier to administer, equally efficacious, and reduces risk of needlesticks.¹⁷ It is possible that intranasal delivery of naloxone occurred but was not detected via the HCPCS code. Studies that rely solely on HCPCS codes may underestimate the prevalence of naloxone administration and contribute to unmeasured misclassification bias.⁸ Text-mining tools such as the one developed in this study may provide superior detection of medication administrations.

Biomedical informatics techniques such as text mining or NLP have been previously used in pharmacoepidemiology and pharmacovigilance studies to identify medication exposures from unstructured clinical text.^18–21 For example, the “Medication Challenge” engaged international teams to develop rule-based, machine learning, and hybrid NLP tools to define medication exposures (i.e., doses, routes of administration, frequencies, duration, indication) from hospital discharge summaries. Most of the developed tools used rule-based logic to identify routes, doses, and frequencies with high accuracy. Although all rule-based tools are limited to querying text according to the prespecified rules, our rule-based text-mining tool identified naloxone exposure with a relatively low FP rate.

Once developed and validated, text-mining tools benefit researchers by reducing the time required to perform manual record review to identify events that are difficult to abstract using conventional methods. The ability to mine vast amounts of data using biomedical informatics tools also increases the investigator’s ability to perform observational research or program evaluations.²² Although we did not test for differences in time saved using the text-mining tool, the accuracy of the text-mining tool in this study suggests that biomedical informatics may efficiently and effectively identify medication exposures beyond the status quo (i.e., administrative codes or patient report). For overdose research and surveillance, applying biomedical informatics tools may improve the identification of individuals who are at-risk for or have experienced an overdose event, which could improve risk stratification tools and facilitate comparative effectiveness research.

A strength of our study was the use of a large database with clinical and administrative data for members of an integrated healthcare delivery system. This database allows researchers to identify a cohort of patients, follow them longitudinally, and capture their medical encounters in the outpatient, ED, and inpatient settings. However, the ability to identify member encounters occurring outside of the health system may be limited. In addition, we did not validate our findings within another health system, which would improve generalizability. Furthermore, in circumstances where the exact timing of naloxone administration may need to be strictly defined, all methods should be validated with manual chart review, as we observed several instances with the text-mining tool and the HCPCS code queries in which the resulting encounter was communicating a naloxone administration that may have occurred days or weeks previously. Finally, while the tool described in this study achieved a relatively high PPV greater than 80%, misclassification bias is still possible, and caution should be taken when applying this tool given that it is not 100% accurate.

Additional performance statistics (i.e., sensitivity, specificity, and negative predictive value) were not calculated because we could not review encounters that were not flagged by the text-mining tool to calculate false negative and true negative rates.²³ Because of this, our work should not be interpreted in the context of identifying encounters in which naloxone was not administered (true negatives) or encounters in which naloxone was administered but not identified by the tool (false negatives). Future research may be able to validate and improve upon the tool to increase PPV or calculate sensitivity, specificity, and negative predictive value.

Our final iteration could be improved. Although additional phrases were found during review of encounters from iteration two but not for review of encounters from the final iteration, additional steps to identify non-standard language in other sources (e.g., paramedic notes) should be taken in the future. As time passes and naloxone use occurs more frequently in the community, the phrases used in the tool will need to be re-evaluated to include additional terminology. Future tools may need to include phrases that include the term “naloxone.” Applying the tool to outpatient visits may also be relevant for clinical surveillance purposes. Future developments could incorporate formal information extraction techniques such as named-entity recognition, relation extraction, and others.²⁴ Finally, qualitative analysis of the false positives from the current tool provide several opportunities to refine the tool, such as including negation terms (e.g., excluding a note with a phrase “no dose of Narcan was given”), excluding mentions of “buprenorphine-naloxone,” or exclusion of mentions of naloxone in allergy fields.

Conclusions

This study demonstrated that a text-mining tool could be developed to more accurately identify naloxone administrations in EHR data than an administrative code. The application of biomedical informatics tools to pharmacoepidemiologic research is crucial to improving methods of identifying medication exposures for clinical and research purposes.