Identification of Documented Medication Non-Adherence in Physician Notes

Abstract

Medication non-adherence is common and the physician’s awareness of it may be an important factor in clinical decision making. Few sources of data on physician awareness of medication non-adherence are available. We have designed an algorithm to identify documentation of medication non-adherence in the text of physician notes.

The algorithm recognizes eight semantic classes of documentation of medication non-adherence. We evaluated the algorithm against manual ratings of 200 randomly selected notes of hypertensive patients. The algorithm detected 89% of the notes with documented medication non-adherence with specificity of 84.7% and positive predictive value of 80.2%. In a larger dataset of 1,000 documents, notes that documented medication non-adherence were more likely to report significantly elevated systolic (15.3% vs. 9.0%; p = 0.002) and diastolic (4.1% vs. 1.9%; p = 0.03) blood pressure. This novel clinically validated tool expands the range of information on medication non-adherence available to researchers.

Introduction

Physician behavior is an important determinant of quality of care and clinical outcomes^{1, 2}. It is, in turn, influenced by many factors³. In order to improve quality of care by influencing physician behavior it is important to understand these factors and how they affect patient care.

Medication non-adherence is common among patients⁴ and its association with adverse clinical outcomes is well documented⁵. Physician surveys show that awareness of the patient’s non-adherence may change the treatment they prescribe to the patients⁶. However, the magnitude of this effect is not well studied.

One of the reasons for the paucity of research in this area is the difficulty in obtaining information on physician awareness of their patients’ medication adherence. Many studies of medication adherence draw upon the data on timing of prescription refills to estimate adherence⁷. However, this information is not routinely available to physicians and consequently may not be optimal for modeling their behavior.

Physicians are expected to record all of the information obtained during a patient encounter, including their perception of medication adherence, in their notes. As electronic medical records become more common, notes are frequently available in digital format that is amenable to computational analysis. In recent years a number of successful systems for analysis of narrative medical documents have been reported^8–11; however, none of them analyzes the text for documentation of patient compliance. In our previous work analyzing text of physician notes we successfully identified documentation of patient non-adherence to provider recommendations (including treatment, follow-up schedule and others) and found that it was associated with an increased rate of emergency department visits¹². In this paper we describe the design of the algorithm to detect documentation of medication non-adherence in the text of provider notes. We report on the algorithm’s validation with respect to the accuracy of the text analysis and relationship to clinical outcomes.

Materials and Methods

The goal of this project was to develop software that could identify documentation of non-adherence to the medication regimen in the text of narrative notes in the electronic medical record. For the purpose of this study non-adherence was defined as lack of compliance with provider recommendations with regards to medications. It included patients discontinuing medications permanently or temporarily as well as using at an incorrect dose or frequency. Self-discontinuation of medications due to side effects was included if this was the first time the provider found out about it. Self-discontinuations of medications due to side effects in remote past were excluded. Discontinuations of medications as directed by a physician were excluded as well.

Algorithm

The algorithm used for identification of documentation of non-adherence in the text of the notes is schematically represented in Figure 1. The software takes plain text files containing multiple documents as input. The documents are split into sentences and three classes of semantic heuristics are sequentially applied to each sentence: a) continuous heuristics b) discontinuous heuristics and c) discontinuous heuristics with negative qualifiers. Continuous heuristics specify every word in the phrase to be detected (e.g. “does not take”), discontinuous heuristics specify some but not all of the words (e.g. “refuses [any words] and to take”), and discontinuous heuristics with negative qualifiers also provide a counter-heuristic that must not match (e.g. “stopped her medication” but not “I stopped her medication”). The order of application of the heuristics was chosen based on the expected relative precision: while discontinuous heuristics are more flexible, they inherently have a degree of unpredictability that lowers their specificity. The software implements a total of 87 heuristics that can be classified into eight semantic classes (Table 1). Multiple non-overlapping heuristics can be matched to a single sentence.

Figure 1.

Algorithm for Identification of Documentation of Medication Non-Adherence

Table 1.

Semantic Classes of Heuristics Representing Documentation of Medication Non-Adherence

Semantic Class	Does not take	Refuses	Inconsistent	Non-Compliant
Examples	Has not been taking his glyburide. Unable to afford her medications.	Hesitant to try insulin. He is unwilling to take beta-blockers.	She takes her Lipitor only rarely.	Poor medication compliance.
Semantic Class	Uncertain	Difficulty	Counseling	Not Tolerated
Examples	He is not sure what medications he should be taking.	Patient struggles with taking her insulin.	I emphasized the importance of taking his blood pressure medications	Unable to tolerate lisinopril due to cough.

Evaluation

We evaluated the accuracy of the software on a dataset of 200 primary care physicians’ notes from the electronic medical record system of two large academic hospitals. In order to ensure high event frequency necessary to achieve statistical power required for the evaluation, the notes were randomly selected from a set of documents that recorded a significantly elevated blood pressure (> 150/95 mm Hg) and no anti-hypertensive medication intensification (based on physician surveys that indicate that patient non-adherence is a common reason not to intensify treatment⁶). Blood pressure values and anti-hypertensive medication intensification were computationally abstracted from the notes using a previously validated technique¹³. Each note was manually analyzed by two independent reviewers (senior pharmacy students), who did not participate in the development of the algorithm, to identify all sentences that represented documentation of the patient’s non-adherence to their medication regimen. Sentences where the reviewers did not reach an agreement were re-analyzed to determine the consensus designation. The software output was compared to the consensus rating to determine sensitivity (recall), specificity and positive predictive value (precision) of the software for detection of notes with documented medication non-adherence..

In order to validate the software output clinically, we analyzed a set of 1,000 randomly selected primary care physicians’ notes on hypertensive diabetic patients with documented blood pressure. It has been shown that medication non-adherence identified from other data sources is associated with higher incidence of elevated blood pressure¹⁴. We therefore analyzed the relationship of documented medication non-adherence identified by the software in the note and the frequency of significantly elevated blood pressure (> 150/95) recorded in the same note to test the hypothesis that the data generated by our software conforms to this previously established association.

Statistical Analysis

Cohen’s kappa was used to assess inter-reviewer agreement. Normal approximation was used to calculate confidence intervals for sensitivity, specificity, and positive predictive value. Fisher exact test was used to estimate the significance of differences in fractions of visits with significantly elevated blood pressure between notes with and without documentation of medication non-adherence.

IRB

The study protocol was reviewed and approved by Partners Human Research Committee.

Results

The final consensus of the manual review identified a total of 82 notes with documented medication non-adherence among the 200 evaluated notes. The kappa statistic for agreement between the two reviewers at the note level was 0.673 (95% CI 0.565 – 0.781) indicating good agreement. In these 82 notes manual review consensus identified 150 sentences (range 1–6 per note) that recorded documentation of medication non-adherence. Eighty (53.3%) of these sentences were identified by both reviewers in the initial analysis

Sensitivity, specificity and positive predictive value of the software results are shown in Table 2. The algorithm correctly identified 89% of the notes with documented medication non-adherence. Examples of phrases correctly and incorrectly rated by the algorithm are listed in Table 3. Many of the missed phrases included misspellings and information distributed over several sentences or requiring the broad context of the note for correct interpretation. Common false-positives included documentation of patient not taking their medication due to physician instructions or not taking medications that were not necessarily ordered / recommended by a medical provider.

Table 2.

Accuracy of Identification of Notes with Documented Medication Non-Adherence

Sensitivity	89.0% (± 6.0%)
Specificity	84.7% (± 5.5%)
Positive Predictive Value	80.2% (± 6.3%)
Negative Predictive Value	91.7 (± 4.7%)

95% confidence interval is given in parentheses

Table 3.

Examples of Phrases Correctly and Incorrectly Rated by the Algorithm

True Positives	He has run out of pain medications. With regard to hyperkalemia, he did not increase the Kayexalate as instructed. He may have missed a few doses.
False Positives	She felt warm but has not taken her temperature. She does not take any medications for relief of her hernia symptoms. Last TSH was very high 10/99, reflecting being off of Synthroid for her scan.
False Negatives	Takes meds but none today. I had asked her to increase her lisinopril dosing to 40 mg a day. She presents today stating that she has not done this. Withut meds today. Also compliance may be an issue.

In order to establish clinical validity of the data generated by the algorithm we analyzed 1,000 notes with any documented blood pressure level to test the hypothesis that patients with documented non-adherence would be more likely to have elevated blood pressure. None of these notes were previously analyzed in the development of the algorithm. The algorithm detected a total of 633 phrases in 365 notes documenting medication non-adherence. The frequency of phrases documenting medication non-adherence per note was distributed exponentially (Figure 2). Phrases stating that the patient does not take the medication were the most frequent (Figure 3) followed by phrases that referred to the patient refusing a change in their medication regimen. In the entire set of 1,000 notes, the average blood pressure documented in the note was 130.5 mm Hg systolic and 74.1 mm Hg diastolic. Notes that had documented medication non-adherence identified by the software were more than 1.5-fold likely to have systolic blood pressure >150 mm Hg (15.3% vs. 9.0%; p = 0.002) and more than two-fold likely to have diastolic blood pressure >95 mm Hg (4.1% vs. 1.9%; p = 0.03) recorded.

Figure 2.

Distribution of Frequency of Phrases Documenting Medication Non-Adherence per Note

Figure 3.

Distribution of Non-Adherence Heuristic Classes in 1,000 patient notes

Discussion

Few means to study patient medication adherence information are currently available to researchers. Most published studies have used physician surveys^{6, 15, 16} which do not allow large-scale investigations. In this project we have investigated feasibility of using computational analysis of the text of physician notes as a novel source of data on physician awareness of patient medication adherence.

Our analysis has shown that documentation of medication non-adherence is lexically diverse and is represented by multiple semantic classes. Nevertheless the algorithm we designed was able to achieve high sensitivity detecting nearly 9 in 10 notes with documented medication non-adherence while maintaining acceptable specificity and precision. Many of the instances of documentation of non-adherence the algorithm missed required advanced semantic analysis including information distributed across multiple sentences or interpretation of the context (e.g. a record of “non-compliance” without specifically referring to medications but located in a paragraph discussing pharmacologic treatment of the patient’s hypertension). Common false positives included documentation of patients not taking over the counter medications or discontinuing medications according to their provider’s recommendations. Crucially, we were able to show that the data generated by the algorithm correlates well with expected patient outcomes (higher frequency of elevated blood pressure) providing a level of clinical validation in addition to factual validation against the human interpretation of the text (which can also be fallible).

Data obtained using the algorithm could be used for several purposes. Researchers could utilize it for large-scale studies evaluating, for example, how physicians ascertain medication non-adherence or the effect of physicians’ perception of patient medication adherence on their clinical decision making. Operationally, if no other sources of information on patient medication adherence are available (as is frequently the case), it could be used to identify high-risk patients who could potentially benefit from more intensive clinical interventions (e.g. a phone call from a pharmacist).

Our study has several limitations. The data we analyzed came from primary care practices of two academic medical centers in Massachusetts and may not be applicable to other healthcare settings. The inter-reviewer agreement for the manual review to identify documentation of medication non-adherence was less than what could have been expected. This may confound the interpretation of the evaluation of the algorithm’s accuracy. However, lower levels of inter-reviewer agreement are common for medical documents¹⁷ and likely reflect the complex nature of these documents. Per design requirements, our algorithm identified documentation of past medication non-adherence and medication discontinuation due to side effects, but other researchers may require different definitions which our algorithm in its current form is not able to provide.

Conclusion

We have designed and validated the first, to our knowledge, algorithm that identifies documentations of medication non-adherence in the text of physician notes. The algorithm showed high accuracy and was also validated clinically. This is an important step in enhancing the armamentarium of health services researchers with novel tools that let them utilize previously inaccessible sources of data.