Extraction of Active Medications and Adherence Using Natural Language Processing for Glaucoma Patients

Abstract

Accuracy of medication data in electronic health records (EHRs) is crucial for patient care and research, but many studies have shown that medication lists frequently contain errors. In contrast, physicians often pay more attention to the clinical notes and record medication information in them. The medication information in notes may be used for medication reconciliation to improve the medication lists’ accuracy. However, accurately extracting patient’s current medications from free-text narratives is challenging. In this study, we first explored the discrepancies between medication documentation in medication lists and progress notes for glaucoma patients by manually reviewing patients’ charts. Next, we developed and validated a named entity recognition model to identify current medication and adherence from progress notes. Lastly, a prototype tool for medication reconciliation using the developed model was demonstrated. In the future, the model has the potential to be incorporated into the EHR system to help with realtime medication reconciliation.

Introduction

The rapid adoption of electronic health records (EHRs) has generated large-scale clinical data that has been re-used for many purposes, including patient phenotyping,¹ pharmacovigilance,^{2, 3} comparative effectiveness research,⁴ clinical decision support,^{5, 6} and quality improvement and research.⁷ Although secondary use of EHR shows many benefits such as improved healthcare quality, reduced healthcare costs, and effective clinical research,^{8, 9} there are many challenges that still need to be addressed. One of the biggest challenges is the accuracy and completeness of EHR data, specifically medication information.¹⁰

The accuracy of medication data is crucial for patient safety, quality of care, and clinical research. Inaccurate or incomplete medication records can lead to polypharmacy, adverse medication interactions, and decreased data reliability in research.¹¹ The medication list is a structured record of a patient’s medication data which is populated automatically by electronically prescribed medications or manually through medication reconciliation.¹² However, the EHR system may not always capture medication data correctly or prevent errors in the medication list.¹³ Previous studies have shown that medication lists frequently contain errors, including duplicated documentation of medications, outdated discontinued prescriptions in the medication list, and missing medications prescribed elsewhere.^{12, 14-18} In addition, prior studies show that physicians direct very little attention to EHR medication lists, and instead spend most time reviewing the impression and plan section.^{19, 20} It seems reasonable to expect that medications recorded in narrative notes are more reliable and can be helpful with medication reconciliation. Medication reconciliation is a process to create and maintain patients’ most current and accurate list of medications.^{21, 22}

However, manual reviewing progress notes for medication data extraction in EHR is time-consuming and labor-intensive. Natural language processing (NLP) is a promising strategy for capturing medication information from the free-text progress note. With advancements in machine learning and the large text corpora available in EHR, NLP has been successfully used to process free-text EHR data, for deep contextualized word representations,²³ information extraction,²⁴ and semantic analysis.²⁵ Named entity recognition (NER) is a sub-task of information extraction, which seeks to identify words or phrases into pre-defined categories with specific labels.²⁶ Over the past years, the NER technique has been applied to extract medication information, such as drug names, frequency, dosage, adverse drug events, adherence, etc., from free-text documents.^27–31 For example, a conditional random field (CRF) model was used to develop a NER model to detect medication attributes and adverse drug events.²⁸ Also, bidirectional long short-term memory (LSTM) model was used for named entity recognizing for medication information.^{27, 29} More recently, pre-trained deep learning models were widely used for biomedical information extraction.^{31, 32} However, to our knowledge, there has not been a well-developed NLP tool to identify a list of current medications for a specific disease such as glaucoma and help with medication reconciliation.

The purpose of this study is to develop a NER model for extracting patients’ current ophthalmologic medication and adherence from free-text notes for glaucoma patients. Glaucoma is characterized by progressive degeneration of the optic nerve and irreversible visual field loss, and it is the leading cause of irreversible blindness worldwide.³³ The majority of glaucoma patients are treated using medical therapy, and the accuracy of medication documentation is crucial in glaucoma management ³⁴ However, the accuracy of glaucoma medication documentation is unclear. In addition, glaucoma patients’ medication non-adherence rate has been reported to vary from 24% to 59%.^{35, 36} Therefore, a reliable method to assess glaucoma patients’ current ophthalmologic medication and adherence is needed.³⁷ Finally, the reliability of medication data is important for glaucoma research, such as prediction models for disease progression. In this study, we first manually reviewed patient charts for discrepancies in medication documentation between medication lists and progress notes. Next, we trained and tested a NER model for extracting current medication from progress notes and evaluated its accuracy. Finally, we demonstrated an approach for medication reconciliation using the NER model on small sample progress notes.

Methods

This study was approved by the Institutional Review Board at Oregon Health and Science University (OHSU). OHSU is a large academic medical center in Portland, Oregon. This study was conducted at Casey Eye Institute, OHSU’s ophthalmology department serving all major ophthalmology subspecialties. The department performs over 130,000 outpatient examinations annually and is a major referral center in the Pacific Northwest and nationally. In 2006, OHSU implemented an institution-wide EHR (EpicCare; Epic Systems, Verona, WI) to handle all ambulatory practice management, clinical documentation, order entry, medication prescribing, and billing.

The study contains three phases (1) Explore medication discrepancies between the medication list and the progress note for glaucoma by manually reviewing charts; (2) Develop a NER model to extract patients’ current ophthalmologic medication and medication adherence from progress notes for glaucoma patients and (3) Apply the NER model to perform medication reconciliation.

1. Manual Chart Review of Medication Lists and Progress Notes

Progress notes and medication list data from EHR were extracted for 150 randomly selected Casey Eye Institute patients with encounter ICD10 diagnosis codes related to glaucoma from January 23, 2019, to September 28, 2020. The patient’s most recent office visit notes were manually reviewed by three independent reviewers. The medications recorded in the narrative notes were abstracted and compared to the EHR medication list at the time of visit. All ophthalmologic medications and over-the-counter (OTC) medications (e.g., artificial tears) were collected. All medications listed in the notes but not on the medication list or vice versa were labeled. Cross-validation among the three reviewers was conducted by using a subset of 20 encounter notes (96.4% agreement).

2. NER Model for Extracting All Ophthalmic Medications

We sampled a dataset with 507 progress notes from office visits at the Casey Eye Institute from January 01, 2019, to December 31, 2019, with encounter ICD10 codes associated with glaucoma. The dataset was constructed by random stratified sampling from all ophthalmology visits according to the department and primary provider name. The documents were manually annotated for nine categories: Drug Name, Route, Frequency, Dosage, Strength, Duration, Adverse Drug Event (ADE), Adherence, and Current Medication Use. All medication names, including generic names, brand names, and abbreviations, were sourced from publicly available online resources and glaucoma specialists. An open-source tool (Doccano; Open source: Doccano; 2018) was used to annotate the documents.³⁸ Due to the limited number of ADE entities, we discarded this category and kept the other eight entities. Figure 1 displays an example of the annotation. The annotated dataset was randomly split into 75% for training and 25% for testing. A 10% randomly sampled subset of documents from the training data was used as a validation set for turning the hyperparameters. Table 1 presents the description of the datasets and annotation statistics.

Figure 1.

Example of note annotation by an open-source tool. Medication drug name, strength, dosage, frequency, route, duration, current use, and adherence are identified.

Table 1.

Distribution of annotated entities and number of progress notes in training and testing datasets.

Named Entities	Train	Test	Total
Drug	2029	505	2534
Frequency	1722	411	2133
Route	1666	371	2037
Dosage	201	40	241
Duration	35	15	50
Strength	168	31	199
Adherence	132	48	180
Current Medication Use	725	185	910
Number of Notes	381	126	507

We used named entity recognition, a sub-type information extraction technique, to extract medication information and adherence from clinical notes. The NER model was developed in Python 3.7.6 using the spaCy library.^39,40 The spaCy library is a free open-source library for NLP. The architecture of spaCy’s NER model is based on convolutional neural networks which uses a word embedding strategy using sub-word features and “Bloom” embeddings.^41,42 In this study, the training task contains 200 epochs with experiments with multiple hyperparameter settings. Different learning rates (initial at le—2, le—3, 2e—3, le—4, 2e—4) were tested and adjusted by two optimizers: Adaptive Moment Estimation (Adam) and stochastic gradient descent. We use a decaying dropout rate (0.5 - 0.35; le-3) to avoid overfitting. Also, we experimented with different batch compounding sizes and regularization schemes to optimize the model. The results of the NER model’s extraction for the test set were determined by comparing the manually annotated and the NER model’s extracted entities. The model performance was evaluated by using F1 score, precision, recall, and the micro-averaged score, which aggregates the contributions of all categories to calculate the average metrics.⁴³

3. Medication Reconciliation Using NER Model for Current Medications

Finally, we developed a prototype medication reconciliation tool using the optimized NER model. For this purpose, we are focusing only on medications the patient is currently using as documented in progress notes. Figure 2 demonstrates an example of medication reconciliation using our prototype tool. First, our NER model extracted the patient’s medications and “Drug Use” label from the l50 sample progress notes which were manually reviewed in phase 1. The “Drug Use” label identified which medications that the patient was currently taking. Next, the current medications were standardized based on RxNorm Ingredient (IN).⁴⁴ Finally, the standardized medications were compared to the manually identified medications from phase 1. Both ophthalmologic medications and over-the-counter (OTC) medications (e.g., artificial tears) were included. All medications listed in the notes but not on the medication list or vice versa were flagged.

Figure 2.

Example of medication reconciliation using the developed NER model

Results

1. Manual Chart Review of Medication Lists and Progress Notes

The randomly sampled 150 patients’ notes and medication lists contained a total of 450 medications, including glaucoma eye drops, mydriasis eye drops, antimicrobials, corticosteroids, and OTC medications. Prescription medications were most common (n = 355; 79%), followed by OTC medications (n = 95; 22%). Around 57% of patients had at least one medication mismatch for all categories in their records. However, only 36% of patients had at least one medication mismatch for prescription medications (Figure 3). Nearly 66% of medications (n = 298) could be reconciled between the progress notes and medication list. Around 34% (n = 152) of medications are mismatched for various reasons, including medications prescribed by clinicians from different institutions, medications with duplicated prescriptions, medications that were prescribed and entered in the medication list but not recorded in the progress note, and old medications that were not discontinued in the medication list. Figure 4 displays the distribution of medication mismatches among the two categories in the EHR by location. The most frequent mismatch was found with prescription medications (55%) followed by the OTC medications (45%). The OTC medications were more commonly recorded in the progress notes but not entered into the medication list. In contrast, mismatched prescription medication more often appeared in the medication list but not in the progress notes.

Figure 3.

Medication documentation mismatches were stratified based on the number of mismatches that occurred per patient for prescription (blue), OTC (green), and all medications (yellow).

Figure 4.

Summary of medication mismatches across 150 patients.

2. NER Model for Extracting Current Ophthalmic Medication

The custom NER model was trained with 381 progress note documents that were manually annotated with eight named entities and then tested on 126 progress notes. Table 2 presents the overall micro-averaged and per-entity performance for the optimal NER model on test data (126 progress notes). The overall performance of the NER model across all categories was F1 score = 0.955, Precision = 0.951, and Recall = 0.957. Higher performance was observed on medication-related entities: Drug, Name, Route, Frequency, Dosage, and Strength, compared to patient’s behavior-related entities: Adherence and Current Medication Use. An error analysis was performed for false negative and positive on Drug Name, Adherence, and Current Medication Use to recognize the source of error predictions. Several causes of errors were identified, such as different wordings for medication adherence, mislabeled current medication use and drug name due to similar sentence structure, eye exams or warm compress mislabeled as drug name, and misclassification when entity information was contained in a short sentence. (Table 3).

Table 2.

The results of the NER model on the test dataset

Entities	Performance on Test Data
	Precision	Recall	F1-Score
Drug	0.971	0.971	0.971
Frequency	0.972	0.969	0.970
Route	0.948	0.986	0.966
Dosage	0.987	0.998	0.991
Duration	1.000	0.600	0.749
Strength	0.969	0.997	0.982
Adherence	0.803	0.758	0.779
Current Medication Use	0.899	0.919	0.909
Average (micro)	0.951	0.957	0.955

Table 3.

Error analysis from NER predictions related to Drug Name, Current Medication Use, and Adherence labels

Error category	Example	Explanation
Mislabeled Current Medication Use	“Urgent add on - Last seen Dr. X on X/X/XXXX”	Unexplained error, “Urgent add on” was labeled as Current Medication Use
	“encouraged PFATs at least BID OU - discussed to space at least 5 mins from glaucoma drops”	“Encouraged” was mislabeled as Current Medication Use due to the similar sentence structure
Mislabeled Adherence	“- History inconsistent drop adherence”	There are many different wordings for medication adherence, and Adherence label was not assigned
	“No eye pain/discomfort but patient admits to forgetting his drops frequently.”
Mislabeled Drug Name	“Cont warm compresses BID ou”	“warm compresses” was mislabeled as Drug Name due to a similar sentence structure
	“Vision has been good. Just using OTC readers.”	“Using” was mislabeled as Current Medication Use, and OTC readers was mislabeled as Drug Name due to the similar sentence structure

3. Medication Reconciliation Using the NER Model

The prototype medication reconciliation tool identified 408 current medications from the 150 progress notes that were manually reviewed in phase 1. After standardizing the medications to RxNorm, 14 medications were removed for a final list of 394 medications. Among the 394 medications, there were 379 medications matched with the manually abstracted current medications. The prototype tool achieved a good performance of F1 score = 0.969, Precision = 0.959, and Recall = 0.979.

Discussion

In this study, we explored medication discrepancies in the EHR data and evaluated the performance of a custom NER model’s applicability to extract current medication for glaucoma patients. We also used the developed NER model in a proof-of-concept application to perform medication reconciliation in a subset of our patients. The key findings from our study were (1) Medication discrepancies in patient charts were found to be present in a large proportion of office visits; (2) The custom NER model can accurately extract current medication and adherence for glaucoma patients; (3) The NER model can be used to reconcile the medication documentation.

The first key finding is that medication discrepancies were found to be present in a large proportion of office visits. Our study shows that approximately twenty percent of medications prescribed to glaucoma patients had at least one discrepancy between the medication list and the progress note. Overall, more than one-third of patients in this study had at least one medication mismatch between both data sources. These inconsistencies in the EHR medication records may increase the risk of medication errors⁴⁵ and affect the reliability of research that relies on this data. These findings are similar to other studies, including a study for microbial keratitis demonstrating 76.9% of medication agreement between progress notes and medication lists¹² and another study for inflammatory bowel disease reporting 78.6% of medications agreement between clinical narrative and medication list.¹⁵ The findings from these studies indicate that the accuracy of the medication list is a common problem. An accurate tool for medication reconciliation of medication lists and further qualitative studies to understand the causes of medication data discrepancies is needed.

The second key finding is that our NER model can accurately identify current medication and adherence from progress notes from outpatient glaucoma visits. In our study, the model reached a micro-averaged F1 score of 0.955 across all categories. The NER model was developed to recognize eight categories from free-text progress notes, including drug names (including generic, brand, and abbreviation names), the route of administration, prescription frequency, the dosage of the drugs, drug strength, duration, medication adherence, and current medication use. The NER model could accurately identify medication-related entities (except duration) but showed lower performance on patient behavior-related entities, such as adherence and current medication use. The difference could be ascribed to the limited number of training cases and the higher variety of wordings. As shown in Table 1, there are only 35 annotated duration entities and 132 annotated adherence entities in the training data. In addition, the words and phrases to indicate adherence and current medication use are various, and some of these phrases are located in different sentences than the medications. Nevertheless, the most common error of drug name identification is mislabeling other terms such as “warm compress” or “OTC readers” as a drug name due to similar sentence structure. For example, “warm compress left eye PRN” or “Vision has been good. Just using OTC readers.” In these cases, these mislabeled drugs will easily be filtered out of the results in practice during the conversion to RxNorm names.

Finally, our NER model can be used to reconcile medication documentation. As shown in the phase one study, we can manually abstract the medication records from their progress notes to compare with their medication list. Similarly, the NER model was able to recognize common medications as well as identify text related to current medication use. This is the first study, to our knowledge, to develop NLP models to recognize current medication use from free-text progress notes. With the ability to identify the current medication use, we are able to capture the whole picture of current medications for the target patients and reconcile it with their medication list. As previously mentioned, the medication reconciliation between progress notes and medication lists was only reported from 76.9% to 79.6% for three different diseases, including microbial keratitis, inflammatory bowel disease, and glaucoma.^{12, 15} And more than one-third of patients had at least one discrepancy for ophthalmic prescription medications.¹² In our study, the NLP tool can correctly identify current medications for glaucoma patients on 150 sample progress notes (F1 score = 0.969). Figure 2 displays an example of medication reconciliation using the NLP tool. In this prototype tool, we focused on reconciling the drug names since physicians did not always record the other attributes, such as route, frequency, and dosage along with the medications. In future work, we plan to extend this medication reconciliation method to use the information from both narrative progress notes and medication lists to construct a current medication list for glaucoma patients.

Our study has limitations future work may address. First, some of the entities are naturally less frequently recorded in the progress notes that affect the performance of the NER model. For example, text related to drug duration appeared much less frequently than other entities, such as drug name, route, and frequency. Thus, it is challenging to train the model correctly recognize these entities. A similar finding was reported in another study.³¹ Second, the model was trained on a set of notes for glaucoma patients from a single institution; it is unclear if the model can be generalizable to other subspecialties within ophthalmology or other healthcare systems. Finally, the application of the custom NER model for medication reconciliation is a proof of concept. We conducted the test of medication reconciliation using the NER model on a limited number of samples. Our intention is to extend and replicate these study methods to different specialties and institutions to increase the generalizability of our model. In the future, the custom model could be incorporated into the EHR system to help with medication reconciliation.

Conclusion

Discrepancies in medication documented in the medication and in progress notes were observed in more than one-third of encounters for glaucoma patients. Inaccurate medication lists in the EHR may affect the reliability of the research or clinical decision support using this data. Since physicians often record current medication information in the progress notes this data could be used for medication reconciliation. In this study, we developed an NLP model to accurately identify current medication information from free-text EHR data that can be applied to perform automated medication reconciliation; the performance of the model is similar to the best performing published NLP models for medication extraction studies.^{27–31, 46} This has implications in improving the data quality and usefulness for medication data in both research and clinical care.