Evaluation of an Inpatient Computerized Medication Reconciliation System

Abstract

We designed the Pre-Admission Medication List (PAML) Builder medication reconciliation application and implemented it at two academic hospitals. We asked 1,714 users to complete a survey of their satisfaction with the application and analyzed factors associated with user efficiency. The survey was completed by 626 (36.5%) users. Most (64%) responders agreed that medication reconciliation improves patient care. Improvement requests included better medication information sources and propagation of medication information to order entry. Sixty-nine percent of admitting clinicians reported a typical time to build a PAML of <10 min. Decreased reported time to build a PAML was associated with reported experience with the application and ease of use but not the average number of medications on the PAML. Most users agreed that medication reconciliation improves patient care but requested tighter integration of the different stages of the medication reconciliation process. Further training may be helpful in improving user efficiency.

Introduction

Medication errors are common in inpatient care ¹ and can potentially lead to adverse drug events. ² Many medication errors causing adverse drug events are the result of system failures in medication prescription and administration rather than of random events. ² In order to improve the quality of patient care in our hospitals, it is important to address medication safety on a systematic basis. ³

A large percentage of medication errors occurs at transitions of care. ⁴ Recognizing this, the Joint Commission for Accreditation of Healthcare Organizations (JCAHO) mandated that starting on January 1, 2006, all health care organizations that prescribe medications “accurately and completely reconcile medications across the continuum of care” and specifically that at every transition of care accurate lists of medications pretransition and posttransition be compiled and compared to each other. ⁵

Paper-based medication reconciliation has been shown to reduce the number of medication errors. ⁶ However, it is a resource-intensive process that does not take advantage of information already present in existing computerized resources and cannot be easily integrated with subsequent stages of patient care (e.g., ordering medications in institutions with computerized provider order entry). Computerized medication reconciliation systems have been shown to lead to improved patient satisfaction ⁷ and possibly improved reconciliation, ⁸ but the optimal design and implementation of these systems are not known. We therefore assessed clinicians' attitudes toward a computerized medication reconciliation system recently designed and implemented in our multi-hospital system, ⁹ their compliance with the medication reconciliation process as implemented in this system, and the factors that affect their efficiency and compliance.

Methods

The computerized medication reconciliation system we evaluated is centered around a web-based application, the Pre-Admission Medication List (PAML) Builder. Screenshots of the application and a brief description of the workflow can be found in the Appendix. For this study we analyzed data from users creating PAMLs for patients admitted between August 1, 2006, (3 weeks prior to the first survey email) and December 31, 2006.

PAML Builder User Survey

We conducted a survey of all active (at least five times) users of the PAML Builder in August to September 2006, starting 4 weeks after the software was completely implemented at two academic hospitals. Survey responders were asked to rate their frequency of using the application, its ease of use, the time they usually spend to build a PAML, and their agreement that medication reconciliation improves patient care, and to list desired improvements to the computerized medication reconciliation process. Most of the survey questions either used standard Likert scales or allowed for free-text entry of answers.

User Efficiency and Compliance Analysis

At both hospitals where the PAML Builder was implemented, house staff play the main role in admitting patients to the hospital and documenting their preadmission medications. We therefore focused our analysis of user efficiency and compliance with the medication reconciliation process on the survey responders who stated that they were interns or residents.

Self-reported (in the survey) average time required to construct a PAML was used as the measure of provider efficiency in using the application. We analyzed the relationship of this outcome variable with the following potential predictors: (1) the average number of PAML medications on the PAMLs edited by the user prior to administration of the survey, (2) the user's self-reported experience with the PAML Builder application, and (3) ease of use of the PAML Builder application as reported by the user on the survey.

In order to complete medication reconciliation on admission and reconcile the PAML with admission orders, users are required to enter a planned action on admission (e.g., “Continue”, “Discontinue”) for each medication record on the PAML. The fraction of the medication records on a particular PAML that had a planned action on admission documented was used as the measure of provider compliance with the medication reconciliation process. PAMLs built by the house staff survey responders for patients admitted after the survey (between August 21, 2006 and December 31, 2006) were used in this analysis. We analyzed the relationship of this outcome variable with the following potential predictors: (1) the number of all PAMLs edited by the user on the day they edited the PAML being analyzed; (2) whether the PAML was started between 9:00 pm and 7:00 am; (3) self-reported ease of use of the PAML Builder; (4) the user's reported perception of how medication reconciliation improves patient care; (5) the reported time it usually took the user to build a PAML; (6) the number of medications on the PAML, and (7) the user's historical compliance with this metric over 3 weeks prior to the survey. Details of statistical analysis of user efficiency and compliance can be found in the Appendix.

Institutional Review Board

The Partners HealthCare System institutional review board granted expedited approval of this study and waived the need for informed consent.

Observations

We identified 41,587 PAMLs constructed on patients admitted during the study period. On average, each PAML had 3.7 medications, 68.7% of which had a planned action on admission recorded.

Among the medication records with a documented planned action on admission, 63.7% were continued as written (see Appendix for details). Only 4.3% of medications were discontinued. In the majority of PAMLs, users either recorded planned action on admission for all (63.3%) or none (34.0%) of the medications on a particular PAML.

PAML Builder User Survey

We e-mailed the invitation to complete the survey to 1,714 active PAML Builder users. Six hundred twenty-six users (36.5%) completed the questionnaire. Nearly half of the responders were nurses, 31.9% were physicians, and 10.4% were other prescribing providers, including nurse practitioners and physician assistants. More detailed information on clinicians who completed the questionnaire can be found in the Appendix.

Thirty-nine percent of all surveyed users reported satisfaction with the PAML Builder application, and 32.1% were neutral (see Appendix for details on distributions of survey responses). Nearly half the responders agreed that creating a PAML helps them reconcile medications on admission. Sixty-four percent of the responders agreed that medication reconciliation improves patient care. Of the 209 house staff, nurse practitioners and physician assistants who completed the survey, 68.9% reported that it usually took them <10 min to complete a PAML.

Free-text answers to the question “PAML Builder would be easier to use if:” were manually classified into several categories (▶).

Table 1.

Table 1 Responses to the Question “PAML Builder would be easier to use if:”

Category	Number of Answers (% of the Total)
	Prescribers	Nonprescribers
Total number of responders to the question	137	96
Order entry integration	45 (33)	2 (2.1)
Interface	24 (18)	11 (12)
Better clinician compliance	2 (1.5)	19 (20)
Better medication information sources	15 (11)	4 (4.2)
Fewer annoying prompts	14 (10)	1 (1.0)
Better training	5 (3.6)	7 (7.3)
Exposure too limited to judge	2 (1.5)	8 (8.3)
The application was faster	8 (5.8)	1 (1.0)
Nurses entered the medications	2 (1.5)	6 (6.3)
Patients knew their medications	0 (0)	7 (7.3)
Other	11 (8.0)	17 (18)

Distribution of answers to the free-text question “PAML Builder would be easier to use if:”. All answers were manually classified into one of the above categories.

The most common categories were integration with medication order entry and improvements to the user interface such as detailed audit trail, predictive entry of medication-specific doses and frequencies, larger field of view, and context-sensitive help. Responders also noted the importance of better compliance with the medication reconciliation process by other team members, improved preadmission medication information sources, and better training.

User Efficiency in Using the PAML Builder

Acquisition and documentation of medication history is a cornerstone of medication reconciliation, but can be time consuming. We therefore analyzed the predictors of the time typically required to build a PAML as reported by 118 house staff in the survey. Both the user's reported experience with the PAML Builder and ease of use of the application were associated with shorter time to build the PAML (p = 0.013 and p = 0.005, respectively). The average number of medications on the PAMLs edited by the user prior to the administration of the survey was not significantly related to the reported time to build a PAML.

User Compliance with the Medication Reconciliation Process

We examined the predictors of the users' compliance with recording a planned action on admission for each PAML medication in 3,071 PAMLs constructed by 101 house staff survey responders after the survey administration. The users' historical compliance with documentation of planned action on admission was derived from 997 PAMLs they edited during the three weeks prior to the survey.

The main predictor of the fraction of medications with a recorded planned action on admission on a given PAML was the PAML author's performance on the same metric prior to the administration of the survey (p < 0.0001). The reported ease of use of the PAML Builder application, time it usually takes to build a PAML, perceived positive effect of medication reconciliation for patient care, greater total number of PAMLs edited by the user that day, or PAML initiation at night were not significantly associated with a change in compliance.

Discussion

In this study we present the results of the evaluation of a computerized medication reconciliation system implemented at two academic hospitals. We investigated the users' assessment of the system and the factors linked to users' effectiveness and compliance with the expected workflow. Several important messages can be drawn from our evaluation:

• 1 While most users agreed that computerized medication reconciliation improves patient care, there were several common themes in their recommendations for improvements in the design. In the user survey, some of the most frequent requests focused on enhancements of the inflows and outflows of the medication information for the reconciliation process. In particular, users requested improvements to medication information sources and propagation of the documented preadmission medication information to the medication order entry. An important implication for the design of computerized medication reconciliation systems, as well as electronic medical record systems in general, is that medication information needs to be entered in a coded format rather than as free text, permitting interoperability between different medication information sources, medication reconciliation applications, and order entry systems. Furthermore, encoding ideally should be done using commonly accepted standards to allow future information exchanges between multiple applications without a need for customized mapping.

• 2 Provider effectiveness in using electronic medication reconciliation system (as estimated by the reported average time spent on building a PAML) depended less on the workload (e.g., the average number of medication records on the PAMLs they had built) and more on their familiarity and experience with the application. Although it is possible that the design of the application was not sufficiently optimized and required a significant learning effort, this finding was unexpected for a relatively simple single-screen application with a limited number of menu options. It is possible therefore that user efficiency could be improved by further training, which some of the users requested in their survey responses. Organizing training for medical professionals who work changing shifts around the clock and frequently have little time to spare from patient care is always a challenging task. Our findings emphasize its importance for success of a new application, particularly one that implements a novel workflow.

• 3 User compliance is frequently a challenge for novel workflows. In our study a measure of compliance with the medication reconciliation process was not predicted by indicators of the users' familiarity with the application, belief about usefulness of medication reconciliation for patient care, efficiency in using the software, or their workload, but only by the users' historical compliance with the process. Possible solutions therefore may include individual user feedback to noncompliant users and/or “hard stops.”

Our study had several limitations. The majority of the PAML Builder users did not complete the questionnaire. Therefore, if the users who did not respond to the questionnaire held opinions different from the ones who did, our findings would be biased. Offering a payment to each of the users who completed the survey could have increased the response rate. Our study focused the analysis of user efficiency in using the application and compliance with the medication reconciliation process on the house staff who play the main role in obtaining medication histories in many academic medical centers, including ours. However, in other settings this role may be played by attending physicians, nurses, or pharmacists. Therefore the findings of our study may not be universally applicable. We based our analysis of user efficiency in using the software on user responses to the survey, which may have been affected by recall bias. Many patients in the study were healthy women being admitted for delivery, leading to a relatively low average number of preadmission medications. However, most patients were being admitted to the general medical and surgical floors, making our findings generalizable to other health care settings.

In summary, this evaluation of a novel computerized medication reconciliation system demonstrated the importance of integration of medication reconciliation with upstream and downstream applications and comprehensive user training. Most survey responders agreed that medication reconciliation helps improve patient care.