An Analysis of Two Sources of Cardiology Patient Data to Measure Medication Agreement

Abstract

Errors and incompleteness in electronic health record (EHR) medication lists can result in medical errors. To reduce errors in these medication lists, clinicians use patient self-reported data to reconcile EHR data. We assessed the agreement between patient self-reported medications and medications recorded in the EHR for six medication classes related to cardiovascular care and used logistic regression models to determine which patient-related factors were associated with the disagreement between these two information sources. From our 297 patients, we found self-reported medications had an overall above-average agreement with the EHR (? = .727). We observed the highest agreement level for statins (? = .831) and the lowest for other antihypertensives (? = .465). Agreement was less likely for Hispanic and male patients. We also performed an in-depth error analysis of different types of disagreement beyond medication names, which revealed that the most frequent type of disagreement was mismatched dosages.

Introduction

The Kaiser Family Foundation estimates that around 3.8 million drugs were dispensed in 2018, and the CDC estimates that almost half of the United States population is on at least one prescription medication.^1,2 With every medication that is prescribed and filled, there is a risk of an adverse drug event (ADE), which is defined as "any injuries resulting from medication use, including physical harm, mental harm, or loss of function."³ ADEs may contribute to negative patient health outcomes, including emergency department visits, prolonged hospital stays, medication non-adherence and increases in hospital admissions overall.^4-7 ADEs also lead to increased costs and utilization of healthcare resources, which negatively impacts both patients and healthcare providers. Specifically, ADEs cause about 650,000 adults and children to visit the emergency department each year, with 27% of all adult emergency department visits leading to inpatient admittance.^8,9 For hospitals, ADEs are costly in terms of time and money. The Institute of Medicine states the cost of ADEs for the healthcare industry is $3.5 billion per year.¹⁰ Specifically, one study found on average, these events cost hospitals almost $3500 per patients and extend inpatient stays by three days.¹¹

Previous research identifies some potential failure points in the healthcare delivery process that may lead to ADEs, including medication choice, prescription writing, formulation, medication dispensing, administration (by providers or patients), and therapy monitoring.^12,13 Some of these errors are more likely to occur with incorrect or insufficient medication-related information in electronic health records (EHR). If a medication is not recorded on a patient's medication list, for example, a provider may be more likely to choose or write a prescription for another medication that results in overdosing or a drug-drug interaction. A medication list that is out-of-date may lead to the dispensing of drugs that should no longer be taken, as well as the self-administration by patients of medications that their providers don't want them to take. Medication lists issues also stem from the fragmented medical record system within the US, leading clinicians to depend on patient self-reported data for more accurate health histories.

Several approaches are used to mitigate the risk of ADEs stemming from inaccurate and incomplete medication lists. Direct patient-revision of medication lists during the pre-check-in process and pharmacist intervention have both been found to improve medication list completeness and accuracy.^14-16 The most popular approach is the process of medication reconciliation, "a formal process for creating the most complete and accurate list possible of a patient's current medications and comparing the list to those in the patient record or medication orders."¹⁷ This process is a preventive measure for ADEs, allowing safe clinical decisions¹⁸ and can be summarized as the collection, comparison, and usage of two lists: current medications according to the patient and prescribed medications.¹⁹ Ideally, medication reconciliation results in a complete and accurate medication list, as well as an improvement in a patient's understanding of their medication regimen.

The widespread adoption of medication reconciliation and the other efforts described above have led to more accurate medication lists^19,20 and higher patient literacy,²¹ yet medication list data quality problems still exist.²² Some factors that contribute to problems with the medication list have been identified. Infrequent medication counseling, for example, makes it less likely that a medication list will be up to date with respect to a patient's knowledge of their current medications. Studies have also demonstrated that factors like hospital setting (i.e., primary care) and low patient comprehension result in decreased amounts of medication reconciliation completed inpatient encounters.²³ Conversely, decreasing medication complexity has been shown to improve reconciliation.²⁴

Medication reconciliation has been explored in several contexts^23,25,26 with various data sources^19,20, but in most cases, medication reconciliation continues to be an in-person activity led and mediated by the provider. There has been promising work assessing the feasibility and usefulness of patients engaging more directly with the reconciliation process to improve the accuracy and completeness of medication lists, often utilizing patient portals and other forms of health information technology.^27-30 Further work is needed, though, to understand the medication-related and patient-related factors that are associated with the quality of patient self-report. In this study, we measured the agreement between patient-reported medication information and EHR medication list data and assessed what factors affect this agreement. We also conducted an in-depth error analysis to understand the types of disagreements that may occur and their frequency.

Methods

We collected patient-reported medication information using a REDCap survey and measured the agreement between these data and EHR medication list data. Specifically, we looked at six standard medication classes related to cardiovascular care: angiotensin-converting enzyme inhibitors (ACE inhibitors), angiotensin II receptor blockers (ARBs), antithrombotics, beta-blockers, statins, and included a sixth category: other antihypertensives (including diuretics, calcium channel blockers, etc.). Focus on cardiovascular diseases is a good contender for reconciliation analysis since these conditions affect almost half of American adults (48.0%),³¹ with one in every four deaths being due to heart disease.³²

We used Cohen's Kappa, F-measure, and descriptive statistics to evaluate and characterize agreement between the patient self-report of medication class and the EHR medication list. Further analyses with logistic regression models were employed to measure several patient-related factors' influence on the agreement level. Types of disagreements were further classified and assessed using descriptive statistics.

This study was conducted at Oregon Health & Science University (OHSU) in Portland, Oregon. OHSU is an academic medical center in Portland, Oregon, that includes two hospitals and multiple ambulatory care clinics. The primary care population at OHSU includes approximately 78,000 patients. Our data set consisted of two data sources: 1) patient self-report of several clinical concepts related to cardiovascular care and health via a REDCap survey and 2) medication list data and demographic data pulled from the EHR. This study was approved by OHSU's Institutional Review Board (#00017632).

Survey

Invitations to complete the REDCap³³ survey were sent via email to 1,700 eligible patients who met the following inclusion criteria: between 18 and 89 years of age, English as a preferred language, and had at least one outpatient visit with the OHSU Knight Cardiovascular Institute between 2/11/2017 and 2/12/2018.¹⁸ All survey items were developed in close collaboration with cardiovascular clinicians. Within the survey, patients were asked whether they were currently taking medications in the six classes described above. Likert-like response items were used to allow patients to denote the confidence of their answer using a 5-point scale, ranging between "definitely not" and "definitely yes"). These responses were dichotomized: "maybe yes" or "definitely yes" were considered affirmative. Patients who responded affirmatively to a medication question were then prompted to provide a free text response with more details about the mediation name and dosage. An example is shown in Figure 1.

Figure 1.

Two example survey items allowing participants to self-report mediation usage. The lower item, which allows a free-text response, is only shown when the participant replies affirmatively ("maybe yes" or "definitely yes") to the preceding question.

EHR Data

Key demographic concepts (age, sex, race, and ethnicity) and active medication list data were queried from the OHSU Epic instance via the Integrated Care Coordination Information System (ICCIS), a population management system.³⁴ Medications in the six target classes were queried from the system using relevant value sets from the Value Set Authority Center.³⁵ The specific value sets used to extract medications within the six drug classes of interest were: Aspirin and Other Antiplatelets (2.16.840.1.113883.3.464.1003.196.12.1211), Low Intensity Statin Therapy (2.16.840.1.113762.1.4.1047.107), Moderate Intensity Statin Therapy (2.16.840.1.113762.1.4.1047.98), High Intensity Statin Therapy (2.16.840.1.113762.1.4.1047.97), ACE Inhibitor or ARB (2.16.840.1.113883.3.526.3.1139), Beta Blocker Therapy (2.16.840.1.113883.3.526.3.1174), Anti-Hypertensive Pharmacologic Therapy (2.16.840.1.113883.3.600.1476), and Diuretics (2.16.840.1.113883.3.666.5.829).

Analysis Methods

We used descriptive statistics and logistic regression to characterize medication agreement between patient self-report and the EHR medication list. For these analyses, we looked only at the structured responses about whether patients believed they were taking a medication class or not. Free text responses were then used to assess more granular agreement.

Agreement assessments: We employed several metrics to determine the agreement of self-reported medication class use with the EHR medication list. For assessing the overall agreement, we used Cohen's Kappa and F1 Scores. We also used Cochran's Q to determine whether the proportion of agreement between medication classes was equal. In these analyses, we only explored agreement in terms of the presence or absence of data with the EHR and patient survey. Likert responses were converted to a binary outcome to match the EHR data. Negative responses (Untrue, Maybe not, and Definitely not) were denoted with a zero (0) and affirmative responses (Maybe yes and Definitely yes) were denoted with a one (1). To determine which factors may contribute to differences in agreement, Cohen's Kappa was calculated for age, ethnicity, and sex. Age was converted into six age ranges for easier interpretation of the agreement results and to normalize the age distribution. Race was excluded due to its low sublevel counts.

Statistical modeling: To determine which factors contribute to the agreement of the two data sources, we employed logistic regression models, implemented using the R package stats.³⁶ Our outcome agreement was determined by the concordance of our two data sources. True positives and negatives became agreement (1), and false positives and negatives became disagreement (0). All regression models used three variables sex, age, and medCount. medCount is the total amount of medications found within the patient's medication list. We use this factor as a proxy measure for medical complexity. For our continuous variables, age, and medCount, we used the min-max scaling standardization.

True positive exploration: Because our agreement analysis only looks at the presence or absence of medication classes to calculate true positives, we conducted a further investigation of medication class true positives within our data with a free-text analysis of patient-reported medication details. We used a random sample of 80% of the EHR structured text for this analysis and matched these data to the patient survey free-text responses. Three main categories: active ingredient, drug name, and drug name and dosage, were used to discern where discrepancies within the true positives originated. We assumed patients would be less likely to know the exact dosage of their medications,³⁷ so dosage was only explored if the drug name was correctly identified.

Results

We received complete responses from 298 participants, for a response rate of 17.5%. One participant was omitted due to incomplete demographic data for a final sample of 297. Participants were 52.8% female, 94.6% white, 96.7% non-Hispanic, and had an average age of 60.4 years (range 19 – 87). At least one medication in the six classes of interest was self-reported by 80.9% of patients, and 85.9% had at least one present in the EHR, creating a 6.00% difference in the presence of information. The average number of medication classes reported by patients was 2.18 ± 0.171, and the average number, according to the EHR, was 2.67 ± 0.192.

Table 2 shows the presence of medication classes in each data source. Antithrombotics were the most reported medication class for patients and the EHR and ARBs were the least reported in both sources.

Table 2. Percentages of patients on each medication class according to patient self-report and EHR, as well as the agreement between the two. Percentages are out of n=297.

Med Class	Survey	EHR	Kappa (95% CI)	F1 Score
ACE Inhibitors	22.5%	29.9%	0.724 (0.633, 0.815)	0.848
Antithrombotics	62.1%	60.1%	0.717 (0.632, 0.801)	0.903
ARBs	14.8%	20.8%	0.726 (0.621, 0.832)	0.774
Beta-blockers	37.9%	48.3%	0.735 (0.658, 0.813)	0.848
Other Antihypertensives	29.2%	50.7%	0.465 (0.364, 0.566)	0.664
Statins	50.3%	56.7%	0.831 (0.768, 0.895)	0.922

Agreement Trends

We assessed medication class agreement with two statistics, Cohen's Kappa and F1 Scores, which are summarized in Table 2. Figure 2 shows the huge proportion of true negatives within most medication classes, the absence of the medication class in the survey and EHR, so we also decided to calculate the F1 scores to see the relationship between agreement and discordance. Overall, the two sources had an agreement of 0.727 (95% CI: [0.695, 0.759]), an above-average score.

Figure 2.

Counts of agreement types between the two data sources

Other antihypertensives had the lowest Kappa (0.465, 95% CI: [0.364, 0.566]) and F1 Scores (0.664), while statins had the highest Kappa (0.831, 95% CI: [0.768, 0.895]) and F1 scores (0.922). The difference in trends was not explained by the number of true negatives present, with ARBs having the highest amount.

Differences in agreement are seen with demographic factors, with age and sex having the strongest association. When we divided age into age ranges, we saw varied differences in Kappa scores. The other category had the lowest agreement for four out of six age ranges and statins had the highest agreement for three age ranges (40's, 50's and 60's). For sex, we saw females had a higher average agreement (κ = 0.705, 95% CI: [0.657, 0.753]) than males (κ = 0.686, 95% CI: [0.636, 0.735]). In ethnicity, we saw higher average agreement in non-Hispanics (κ = 0.732, 95% CI: [0.611, 0.854]) than Hispanics (κ = 0.706, 95% CI: [0.671, 0.740]). Kappa scores were not computed for race due to insufficient sample sizes.

Statistical models

We used logistic regression models to determine the influence of different patient-level factors on the agreement between self-report and the EHR medication list. A Cochran's Q test confirmed that agreement differed across medication classes, so we ran a separate model for each medication class. We ran the same model for each medication class with agreement as the dependent variable and sex, age, and medCount as the independent variables. Table 3 shows the results of the regression analyses. Sex was the most influential factor for antithrombotics and ACE-inhibitors, both being significant. Even though both medication classes see a significant effect, the size of the effect is different. Men on ACE-inhibitors are .433 times less likely to see agreement than women, while men on antithrombotics are 2.70 times more likely to see agreement than women. Age was the most influential factor for angiotensin receptor blockers and statins but was only significant for the former. For age, a coefficient less than one indicates a certain percent decrease in the outcome (agreement between the two data sources). For every one year increase in age, there is a 5.50% decrease in the odds that patients who reported being on an ARBs will agree with their EHR record. Medication count was the most influential factor for beta-blockers and other antihypertensives but was only significant for the latter. For each medication added to a patient's total medication list, patients who reported taking an other antihypertensive show a 5.90% decrease in the odds that their medication will match with the EHR.

Table 3. Medication class regression analysis results.

Med Class	Sex(M)	Age	medCount
ACEs	0.434*	0.982	0.966
Antithrombotics	2.70*	0.986	1.03
ARBs	0.984	0.945**	0.943
Betas	0.886	0.992	0.954
Other	0.584	0.989	0.941**
Statins	0.646	0.977	1.03

^* Significant at 0.05

^** Significant at 0.001, results are exponentiated and unscaled

True Positive Exploration

Structured responses to the medication class questions that were identified as true positives within our random sample were manually reviewed in order to validate patient responses. Table 4 shows the proportions of patients who correctly described the components of the medications reported. Active ingredient refers to the primary biologically active ingredient in a medication, and drug name refers to the full name of the medication (brand names and generic names were treated equivalently), which can include the main ingredient. If patients correctly identified the correct drug name, we also looked at the dosage. The total amount of matched medications per class denotes the amount of true positives patients correctly identified the active ingredient over half of the time (58.4%) and identified the proper medication name less than half of the time (48.7%). Drug name and dosage information was correctly identified 21.4% of the time. We also looked for spelling errors and found that most people spelled their medications correctly (90.6%).

Table 4. An in-depth analysis of errors found within the patient survey data^*.

Med Class	Active Ingredient	Drug Name	Drug Name and Dosage	N
ACE Inhibitors	53.1%	32.7%	18.4%	20
Antithrombotics	84.2%	84.2%	26.3%	110
ARBs	85.7%	78.6%	14.3%	15
Beta-blockers	35.7%	31.0%	21.4%	50
Other Antihypertensives	69.7%	54.1%	22.9%	43
Statins	50.7%	50.7%	23.2%	70
Total	58.4%	48.7%	21.4%	308

^* Percent Correct Compared to Structured EHR data

Discussion

This study explored the factors that influence the agreement between the EHR medication list and patient-reported medications for five standard medication classes within the cardiology specialty. The more medications were present in the EHR (85.9%) than were self-reported by patients (80.9%). Within medication classes, we only saw the equivalent presence of medications in antithrombotics. Kappa and F1 scores revealed varying degrees of agreement within demographic factors and medication classes. This variation is partially explained by errors examined within patient-reported data, mostly accounted for by dosage errors.

Overall, the data showed an above-average agreement with medication classes having no apparent pattern. Demographic variables also revealed similar results with sex, ethnicity, and age. Using age ranges for our Kappa analysis, we saw the other category having the lowest agreement three times, with the lowest agreement class for people in their 40s and 50s being ARBs. As a patient's age increased, we saw fluctuating decreases in agreement, where the lowest average Kappa score belonged to patients above 80 (κ = 0.635, 95% CI: [0.492, 0.778]).

Our regression analyses continued to show no clear pattern in our data but showed significance for some variables. The other category had the smallest change in agreement for every added medication count but was the only one to be significant. The small change could be explained by the within-category variation happening, meaning many of the medications found in this category could be uncommon treatments for cardiac diseases. ARBs had the coefficient with the largest effect (2.70), but the large coefficient can partially be explained by having the greatest amount of true negatives (n = 232).

The true positive validation partially explains the agreement variability in our study. More patients were likely to know the active ingredient of their medication (58.4%) than the drug name (48.7%), yet most of these medications were spelled correctly (90.6%). Patients on beta-blockers were least likely to identify the correct active ingredient (35.7%) or drug name (31.0%). Most errors stemmed from self-reported medications with correct names and incorrect dosages (21.4%).

Learning about medication reconciliation with live patient data was crucial to illustrate an accurate representation of the completeness of each data source. Patient surveys are an important data source that allows patients to reflect on their current health history and improves the accuracy of EHR records. This study showed patient surveys provide the opportunity for EHR records to close the 6.00% difference of information and improve medication lists to ensure their accuracy for long term patient-centered care. Repetition and efficiency of these surveys have the potential to increase the presence and accuracy of medications found in both sources above a .727 agreement rate, which is considered moderate agreement,³⁸ to reach a strong agreement (> .80) for this clinically essential data type. Improvement in this process may have huge health, cost, and time benefits for the patient, provider, and healthcare institution.

Limitations and Future Work

There were several limitations of this study, the largest being its generalizability. The study used a racially and ethnically homogeneous set of patients within a specialty (cardiology). Another limitation came from our small sample size. A larger sample would have allowed us to include and analyze factors that could contribute to agreement through more complex regressions and more robust machine learning algorithms. The temporality of medication list data causes another limitation as it cannot ensure complete accuracy of EHR data. To mitigate this limitation, we pulled EHR data to match the survey completion dates to represent the most current data. There was also a limitation in using one EHR record as it limits the medication list to a patient's memory at any one visit and ones prescribed at the institution. One EHR record does not reflect the totality of a patient's health history, so future work would seek multiple data sources to provide a comprehensive picture of a patient's current medications from numerous healthcare institutions' EHR data, claims data, and pharmacy records of dispensed prescriptions.

Conclusion

In this paper, we sought to identify factors that influence medication reconciliation between our two data sources. We analyzed clinical and demographic factors to observe variability in agreement. Preliminary analysis displayed differences, but no one pattern described the trends observed. Our regressions analysis found that most of the differences were not significant, but some significant findings (i.e., the other antihypertensives category) did point to the usefulness of patient-reported data. Our patient survey data provided the opportunity to validate the true positives within our dataset and explore how patient inaccuracies can affect agreement. In the future, we plan to assess additional sources of medication data to analyze new factors contributing to agreement, ultimately proving why we must improve medication reconciliation processes for more accurate health histories within the EHR.

Figures & Table

Table 1. Demographics of 297 study participants. Data are presented as percentages unless otherwise stated.

Average Age (years/range)		60.8 (19 - 87)
Sex
	Female	158 (52.8)
	Male	139 (46.5)
Race
	Asian/Chinese/Japanese/Korean/Pacific Islander	4 (1.34)
	Black/African American	2 (0.67)
	White/Caucasian	283 (94.3)
	Unknown	9 (3.01)
Ethnicity
	Hispanic	7 (2.34)
	Non-Hispanic	289 (96.3)
	Declined	2 (0.67)
Average Agreement between Data Sources		1.99 (0.167)
Average Reported Medication Classes		2.18 (0.171)
Average Medication Classes in EHR		2.67 (0.192)