Development and Evaluation of a Checklist for Medication Order Review by Pharmacists

Abstract

Background:

To create a checklist of the tasks that a pharmacist must perform during medication order review in the hospital setting and to evaluate the utility of and pharmacists’ satisfaction with the checklist.

Methods:

An evidence-based checklist for medication order review was developed, with items related to order urgency, verification of patients’ identity, therapeutic review, and actionable items. Pharmacists were educated about the checklist, and it was made available at 2 community hospitals in an urban setting. Pharmacists completed a nonvalidated satisfaction survey and participated in focus groups or interviews within 3 months after implementation of the checklist. Qualitative descriptive theory was used to identify themes within the data. Near-miss occurrence reports for the 3 months before and after implementation of the checklist were quantified.

Results:

Of 16 pharmacists who were involved in the implementation phase, 14 participated in focus groups or an interview, and 11 responded to the survey. All respondents felt that the primary role of the checklist was for training. They felt that the checklist could be useful when reviewing high-alert or unfamiliar medications or therapy for patients with complex medications. The checklist was most helpful when it was used as a reminder, on an as-needed basis. Nine (82%) of the 11 survey respondents indicated that the checklist standardized the process of medication order review, the same number felt that it prevented accidental omission of critical checks, and 8 (73%) felt that it improved patient safety. Education was necessary to reinforce the purpose of the checklist and its self-check nature. There was no difference in the number of near misses in the pharmacy between the 3-month periods before and after implementation of the checklist.

Conclusion:

Pharmacists participating in the study felt that a checklist for medication order review had a role in training new pharmacists and standardizing processes.

INTRODUCTION

A checklist is a tool to guide users in accurately completing a multistep task. It serves to prevent omissions, prioritize actions, standardize procedures, and minimize reliance on memory.^1,2 Many industries have successfully used checklists to reduce errors. Aviation, aeronautics, and product manufacturing all require the recall and performance of complex, multistep tasks and have used checklists to decrease human error and ensure that quality standards are maintained.^3,4 Like these industries, many health care professions involve work that requires the recall and performance of complex, multistep tasks for the safe provision of care to patients. Medical checklists have been employed in operating rooms and intensive care units to prevent errors, improve interprofessional communication, decrease postoperative complications, and decrease in-hospital mortality.^5–7 The successes achieved thus far warrant consideration of more widespread application of checklists in health care.

Medication order review by pharmacists is a multistep process in which pharmacists evaluate orders (prescriptions) for safety, efficacy, and appropriateness by examining drug- and patient-related factors.⁸ Medication order review is a composite of multiple tasks, including verifying information, applying critical thinking skills, and making appropriate decisions, which lends itself to application of a checklist.⁹ Following such a review, when the pharmacist has deemed the prescription appropriate for dispensing, the order is entered into the pharmacy information system. Pharmacists must effectively and efficiently complete the process of medication order review in an environment that often presents internal and external distractions.⁹ In the context of medication order review by pharmacists, a checklist could be used as a prompt to verify that all critical checks have been performed and to ensure that optional check items are not forgotten. An item is considered critical if failure to check it could lead to an error every time medication order review is performed.¹⁰ An item is considered optional if its status as a critical check is situational, depending on the medication itself or the medical condition being treated. Checklists may be useful in medication order review, because they offer a moderate level of guidance, being flexible enough to allow the user to use his or her own judgment.³

This project was conducted at the Winnipeg Regional Health Authority (WRHA), a publicly funded organization that provides acute, long-term, and community health services for about 800 000 people. Pharmacy personnel within the WRHA Regional Pharmacy Program consist of 150 and 200 full-time equivalent unionized pharmacists and technicians, respectively, who provide pharmacy services throughout 8 hospitals with a total of about 2000 beds. Within the WRHA facilities, training of pharmacists for medication order review occurs on a site-specific basis. Because the WRHA Regional Pharmacy Program is a large employer, the organization has many new, casual, and part-time staff, and checklists have been described as particularly helpful in familiarizing these types of staff with a protocol.² Although it is likely that each WHRA pharmacist performs the steps of medication order review each time he or she checks a prescription, no standardized process has been adopted or encouraged for training or educational purposes.

In the context of medication order review, a checklist could be used to ensure that all tasks necessary for safe and effective review are completed. In addition, a checklist for medication order review by pharmacists in institutional practice could move hospital pharmacies toward addressing Accreditation Canada’s Managing Medications standard 21.0, which requires a risk management program to reduce medication-related errors and sentinel events.^11,12 A hospital pharmacy checklist for medication order review could reduce the frequency of errors and near misses (hazardous situations that could have led to an adverse event for a patient had they not been intercepted)¹³ through standardization of the processes involved. To date, the role of checklists in medication order review in the pharmacy has not been evaluated.

The purposes of this study were to create a checklist for medication order review by pharmacists, to evaluate the usability of and pharmacists’ satisfaction with the checklist, and to determine the effect of the checklist on the number of near misses reported by pharmacists.

METHODS

This study was a prospective mixed-method project consisting of the development, implementation, and evaluation of an evidence-based checklist for medication order review. The checklist was evaluated by means of a survey, focus groups and interviews, and quantification of near-miss occurrence reports. The checklist was implemented and evaluated over the period May to November 2009.

Study Population

The study population consisted of all pharmacists at 2 community hospitals (a total of 16 pharmacists) in the WRHA. In this health authority, pharmacists are responsible for review and entry of medication orders, tasks that are performed either in a centralized dispensary or on the wards. Only pharmacists employed by the WRHA at the time the checklist was implemented, the evaluation survey was distributed, and focus groups or interviews were conducted were eligible to participate. Pharmacy managers were excluded from the study to avoid potential bias during focus groups.

Development of the Checklist

Development of the checklist was based on the results of a task analysis of medication order review by pharmacists at sites throughout the WRHA, as well as a review of the literature on checklist design and consultations with pharmacists who perform this type of review. Information gathered from the task analysis was integrated with accepted principles of checklist design and a published prescribing checklist designed for physicians^10,14 to create the initial version of the checklist. Practising pharmacists (n = 27) at the 2 study sites and at 2 nonstudy sites, also within the WRHA, reviewed and agreed upon the checklist items and pilot-tested the checklist for functionality. The checklist contained items related to order urgency, verification of patients’ identity, therapeutic review (for safety and efficacy), and actionable items (Table 1), and was designed for general purpose use on all wards for all types of patients.

Table 1.

Checklist for Medication Order Review by Pharmacists^*

Order urgency

Patient location

Type of order (new admission, transfer, post-op, etc.)

Order triaged as urgent, emergent, routine, or general ward stock

Verification of patient’s identity

Medical record number

Name

Age

Sex

Allergies

Disease states/problems

Therapeutic review

Date and time

Review right and left sides of order sheet

Doctor (signature present, consult service suggestions, appropriate prescriber)

Formulary/restrictions/product availability

Complies with medication order writing standards

Contraindications

Laboratory tests/levels (e.g., SCr, K)

Renal function

Dose (weight, renal function, hepatic function, normal range)

Dosage form

Route (appropriate, available)

Schedule/frequency (appropriate, ± food, spacing with other medications)

Administration (NPO, rate, administration policies for IV medications)

Duration (number of days)

Start date / start time

Stop date / stop time

Directions

Interim quantity

Number of labels to be printed

Drug interactions

Duplication (same drug or therapeutic duplication)

Discontinue existing orders replaced by new order

Comments/notes

Actionable items

Order laboratory tests/levels (please specify)

Pharmacist intervention/clarification chart note required

Documentation in pharmacy information system for technicians, pharmacists, or nurses

Communicate with other health care professional

Refer to clinical pharmacist for follow-up

Medication counselling required

Sign name/initial

K = potassium, NPO = nothing by mouth, SCr = serum creatinine.

^*Adapted with permission. © 2009 Winnipeg Regional Health Authority.

Implementation

The checklist was implemented at 2 community hospitals within the WRHA. The implementation phase consisted of an education session and dissemination of the checklist to the 16 study participants. To maximize uptake of the checklist, pharmacists were educated (by means of a didactic lecture) about the common causes of errors, the environmental and human factors that may contribute to errors, the purpose and safety benefits of checklists, and application of the checklist concept to medication order review, including how the checklist was developed and its function and contents. Copies of the checklist were posted at each computer terminal used for medication order review, and the pharmacists were asked to use the checklist as a self-check when performing this type of review. The pharmacists were asked to use their professional judgment to determine which situations and medication orders would most benefit from use of the checklist and were given examples of situations for which they might wish to use the checklist (including new admissions and unfamiliar medications). Pharmacists were not asked to use the checklist for every order, nor were they asked to document use of the checklist.

Measurement

The checklist was evaluated within 3 months after implementation. The evaluation consisted of a satisfaction survey, focus groups and interviews, and quantification of near-miss occurrence reports. The survey consisted of a set of statements, based on a literature review and field-tested for readability, designed to evaluate the usability of and pharmacists’ satisfaction with the checklist. For each statement, respondents were asked to indicate their level of agreement using a 5-point Likert scale (strongly agree, agree, neutral, disagree, strongly disagree). Hard copies of the 19-item unvalidated satisfaction survey were delivered to the study pharmacists, who completed and returned the surveys anonymously. Focus groups and interviews were conducted, on a voluntary basis, 2 to 3 months after implementation, during regular shifts. Interviews were used for pharmacists who were unavailable to participate in a focus group but who wanted to participate in the evaluation phase of the project. Participants were recruited by email. The focus groups and interviews were semistructured and were based on open-ended discussion questions (Box 1). Each focus group or interview was conducted by the same 2 investigators (L.D.M., C.B.R.), who had no supervisory or direct working relationship with any of the participants. One of these investigators (L.D.M.) facilitated the focus group discussions, which were audio-recorded and transcribed verbatim, while the other investigator (C.B.R.) took notes.

Box 1. Question Guide for Focus Groups and Interviews.

In your opinion, what is the purpose of a medication order review self-check checklist?

How has the self-check checklist affected your workflow?

Tell me about any potential reasons why someone might or might not want to use a medication order entry self-check checklist.

How has the self-check checklist influenced your approach to medication order review?

In what kinds of situations would a medication order review self-check checklist be useful?

In what kinds of situations, if any, have you used the medication order review self-check checklist?

Can you give me an example of a specific time when the self-check checklist has been particularly useful? In this instance, what was your course of action?

What areas of improvement for systemic processes within the pharmacy department has the self-check checklist helped you identify? What action, if any, did you take in raising these issues to your peers?

What kinds of positive things could happen as a result of implementing the self-check checklist into other WRHA pharmacies? What about negative things?

What would you change about the education session? What would you keep the same?

Is there anything that we haven’t discussed yet that you feel is important?

All hospitals in the WRHA employ nonpunitive, voluntary occurrence reporting systems that can be used for any type of error or near miss. To determine if the checklist affected identification of near misses, the number of near-miss occurrence reports generated by pharmacy staff (using a standardized regional form) in the 3 months before and the 3 months after implementation was determined at both sites where the checklist was implemented. It was assumed that an increase in the number of near-miss occurrence reports after implementation would suggest that use of the checklist led to increased awareness of near misses within the pharmacy. Near-miss occurrence reports generated during corresponding time periods from a similar-sized community hospital pharmacy not involved in the project were quantified as an additional control.

The University of Manitoba Bannatyne Campus Research Ethics Board approved this project. Study participants consented to participate in the survey, focus groups, and interviews, and were made aware that their participation was voluntary. Because of the small sample size, no identifying or demographic information was collected from participants.

Data Analysis

For analysis of the survey results, the percentage of respondents who agreed with each of the statements was determined using Microsoft Excel.

Following transcription of recordings of the focus groups and interviews, the 3 investigators independently reviewed the transcripts and used qualitative descriptive theory and content analysis to identify common words and phrases. These words and phrases were used to form codes, and the codes were grouped into common themes. With this method, the goal is to present a comprehensive descriptive summary of the data in everyday terms for the events described and in the way that best suits the data, without applying a specific theory of analysis.¹⁵ Transcript coding was performed independently by each investigator, and the final themes were generated through consensus. Transcripts were reviewed sequentially according to the timing of the focus groups and interviews. Upon review of the final transcript, the investigators determined that theme saturation had been achieved and that no further interviews were required.

RESULTS

Survey

Surveys were distributed to the 16 pharmacists, of whom 11 completed the survey (69% response rate). The majority of respondents felt that the checklist served to prevent the accidental omission of critical checks, improved overall patient safety, and standardized the process of medication order review (Table 2). Just over half of respondents indicated that their awareness of near-miss events increased after using the checklist (Table 2). However, few respondents reported using the checklist for nonroutine or unfamiliar medications or for high-risk medications (Table 2). When questioned about frequency of checklist use on the most recent shift, 8 (73%) of 11 respondents reported not using the checklist, 1 respondent (9%) had used it once during the shift, and 2 respondents (18%) had used it twice or more. Respondents indicated that the concept of the checklist was a good one, but they noted that it was difficult to implement in practice and that it reduced efficiency (Table 2).

Table 2.

Pharmacists’ Responses to Survey

Survey Statement	No. % in Agreement* (n= 11)
The checklist is easy to read.	11 (100)
I understand how to use the checklist.	11 (100)
The checklist follows a logical order.	7 (64)
Using the checklist does not take up too much of my workspace.	5 (45)
The checklist is easy to use.	6 (55)
The checklist is useful.	6 (55)
It was easy to incorporate the checklist into my current routine for MOR.	1 (9)
Using the checklist does not make me less efficient at MOR.	0 (0)
I use the checklist to review high-risk medications.	2 (18)
I use the checklist for nonroutine/unfamiliar medications.	2 (18)
I use the checklist for new admission orders.	3 (27)
The checklist has prevented me from committing an error.	2 (18)
The checklist increased my awareness of near-miss events.	6 (55)
The checklist improves patient safety.	8 (73)
The checklist prevents the accidental omission of critical checks.	9 (82)
The checklist standardizes MOR.	9 (82)
The education session provided adequate instruction on how to use the checklist.	9 (82)
The education session was a valuable use of time.	8 (73)
I would recommend the self-check checklist implementation to be accompanied by an education session	10 (91)

MOR = medication order review.

^*Respondents rated their agreement with each statement using a 5-point Likert scale.

The data represent the sum of ratings of “agree” and “strongly agree”.

Themes from Focus Groups and Interviews

Of the 16 pharmacists invited, a total of 14 participated in 4 focus groups and 1 interview (with each focus group and the interview lasting 15–45 min). Each focus group consisted of 2 to 4 pharmacists. The participating pharmacists had different levels of experience, and together, they represented the majority of pharmacists on staff at the 2 hospitals. Five themes emerged, and the results are discussed in the context of these themes.

Training of New Practitioners and Students

All study participants felt that the primary role of the checklist was as a training tool for new practitioners and students. Participants also felt that having the checklist available to new practitioners would make the transition to practice easier, since new practitioners might not yet have developed a systematic approach to medication order review. As one study participant stated, “Having [a checklist] in black and white . . . implies that it’s not just you or another pharmacist telling a new pharmacist that this is important. [The checklist] formalizes that this is an expectation. This is the standard of professional practice.”

Medication Safety

Study participants identified improved safety of medication order review as one of the primary reasons for using the study checklist. They acknowledged that some components of medication order review might be missed or overlooked and that the checklist was intended to reduce such omissions. As one study participant stated, “Having a clear list of things we are supposed to look for available would help to reduce the number of potential errors.” Several study participants commented that checklists have improved safety in numerous other professions and could therefore improve patient safety in medication order review.

Study participants also identified having a standard process for medication order review as a reason for using the checklist. As one study participant said when commenting on the checklist, “People know exactly what’s expected of them in the dispensary and what they should be checking on an order when they start out.” Several participants felt that having a formalized checklist emphasized the importance of the medication order review process more than reliance on individual professional judgment.

Practicality

Study participants did not report any significant effect on workflow caused by the checklist, as they did not refer to it for every order. However, they theorized that if they were to use the checklist for every order, there would be a substantial negative impact on pharmacy workflow. One study participant commented, “It would be disruptive if you stopped and consulted it on every order.” Study participants agreed that for the checklist to be practical in performance of medication order review, it would have to be used on an as-needed basis.

Usefulness

Study participants described several situations in which use of the checklist would be valuable. High-alert or unfamiliar medications and patients with complex medical situations receiving multiple medications were identified several times as examples for which pharmacists felt that a checklist would minimize errors. One study participant stated, “If there is a number of new medications, I’d want to use [the checklist] so that I don’t miss anything.” Some study participants felt that the checklist could function as a “refresher”; for example, a pharmacist might look at the checklist before starting a shift, glance at it from time to time, and refer to it when necessary.

Several study participants did not feel that the checklist was useful for their own practice. When discussing unfamiliar medications, one study participant commented, “A pharmacist should still not need the checklist because he or she should still be automatically going through this process.” These opinions were largely from experienced pharmacists who felt that the checklist represented a paper version of current pharmacist practice.

Education

Study participants found the education session useful for defining the purpose of the checklist, the rationale for its use, and how it should be used. Participants felt that some form of education was necessary to implement the checklist. As one study participant stated, “When you are first confronted with this checklist, you might be tempted to think, ‘Well, I don’t need this. I already have my own process, and big deal.’ But, if you see how it applies to other industries, then that’s more evidence that we should be taking this seriously.”

Near-Miss Reporting

At 1 of the 2 implementation sites, no near-miss occurrence reports were generated in the 3 months before implementation of the checklist, and 1 near-miss occurrence report was generated in the 3 months after implementation. Information about near misses was not available for the second implementation site. At the control site, no near-miss occurrence reports were generated in either time period.

DISCUSSION

To the authors’ knowledge, this is the first study to develop and evaluate an evidence-based checklist for medication order review by pharmacists in a hospital setting. Study participants felt that the primary purpose of the checklist was as a training tool. They also felt that the checklist would encourage development of a systematic and comprehensive approach to medication order review and could be useful for practising pharmacists during review of high-alert or unfamiliar medications or review of therapy for patients with complex medications, on an as-needed basis. Despite a perception of checklist utility, especially for training purposes, study participants did not report referring to the checklist frequently, and no change in the number of pharmacy near-misses was found after implementation of the checklist (although data for this component of the study were incomplete). It is possible that checklist implementation as a component of training could address Accreditation Canada’s requirement for a coordinated risk management program to reduce medication-related errors and sentinel events and should therefore be considered by hospital pharmacy departments.¹¹ Factors to consider in implementing such a checklist include the importance of educating pharmacists and use of the checklist for training new staff.

Participants in the current study felt that a checklist would encourage standardization of care, and they recognized the importance of adequate medication order review for high-alert medications, despite the fact that only a minority adopted the checklist to review such orders. The low rate of use of the checklist may have reflected the study sample: most participants were experienced individuals with established behaviour for medication order review. Participants may have perceived their personal process for medication order review to be efficient, safe, and precise. Additionally, most factors needed to facilitate behavioural change (specifically, a clearly understood desired end state, readiness for change, and broad support and reinforcement for change) take time to develop within health care institutions.¹⁶

The education of pharmacists is a key component to successful implementation of a checklist for medication order review. The pharmacists in this study identified the education session as a necessary component of checklist implementation, essential for obtaining compliance and for fully communicating expectations. It is therefore recommended that implementation of similar checklists at other institutions include an education component. The education session provided a venue for explaining that the checklist was meant to serve as a self-check tool, to be used at the discretion of the individual pharmacist. This information was positively received, as participants felt that an expectation of checklist use for every order would be unrealistic. The literature on checklist design recognizes that the likelihood of a checklist affecting practice depends upon the commitment of users to employ the checklist and recommends education as a means of getting that commitment.^10,14 In the current study, education was provided in the form of a didactic lecture highlighting the benefits of checklists in other industries, the components of the checklist for medication order review, and the expectations of pharmacists using the checklist; this form of education was well received by the study participants. In a previous study,¹⁷ physicians and nurses went through an education program before implementation of a required pre-operative checklist. As participants gained experience with the checklist and encountered situations in which use of the checklist caught errors, compliance rose from 75% to 100%.¹⁷ This increase in compliance coincided with an increase in the self-reporting of near misses, which was believed to be due to the individuals involved becoming better able to identify errors through their use of the checklist.¹⁷ However, the possibility of surveillance bias and/or the Hawthorne effect may have influenced these findings. There is a need for education of pharmacists and continued reinforcement of the benefits of a standardized process to aid in the development of a systematic and comprehensive approach to medication order review.

The use of the checklist for medication order review as a training tool for new practitioners may, over time, contribute to greater uptake of a standardized process for this type of review. The study participants recognized the importance of having a systematic and comprehensive approach to medication order review and felt that using the checklist as a training tool would aid the development of such an approach. Several of the pharmacists participating in this study felt that they had developed their own systematic approach to medication order review, before implementation of the checklist. However, they also acknowledged that without a standardized training process, there is a risk that some new practitioners may not develop a process for medication order review sufficient to provide the high-quality patient care expected of pharmacists. Incorporating the checklist into the training process will ensure that all new practitioners receive similar training. A checklist ensures that something the trainer is thinking about, but not saying out loud, is not lost to the trainee. Most importantly, a checklist formalizes what is expected of a new practitioner performing medication order review. Although the majority of study participants did not adopt the checklist during the study period, they indicated that they would use it as a training tool to ensure that new pharmacists developed a standardized process for medication order review that would lead to sufficient review of high-alert medications.

In this study, no change in the number of near misses reported by pharmacists was observed after implementation of the checklist. The small sample size of this study, and the resulting lack of power to detect a difference, may have contributed to this result. Sax and others¹⁷ detected an increase in the number of near misses reported after implementation of a surgical checklist and participation in a training intervention. However, their study involved 857 participants, whereas ours had only 16 participants. Our result may also have been related to the fact that near misses are significantly underreported in health care.^18–21 Even if additional near misses were identified using the checklist, a lack of reporting would have contributed to a failure to detect the change. In our study, the phenomenon of underreporting seems possible, because 55% of survey respondents indicated increased awareness of near-miss events secondary to checklist use, but there was no commensurate increase in the number of near-miss reports. It is possible that simply knowing about a standardized process of medication order review or seeing the checklist posted near a workplace computer terminal contributed to some steps in behavioural change toward an improved safety culture, including critical reflection about medication order review.

This study had several limitations. Responder bias is a potential limitation for both the survey and the focus groups and interviews; however, the response rate for both types of evaluation was high. The pharmacists who were asked to use the checklist also contributed to its development, as numerous pharmacists at several study and nonstudy sites within the WRHA reviewed and provided feedback on early versions of the checklist. Pharmacists’ contributions to checklist development might have facilitated buy-in and implementation at the study sites, which might in turn limit the generalizability of the study findings. The small sample size and underreporting of near misses may have prevented detection of an association between use of the checklist and the number of near-miss reports. However, sample size was not likely a limitation in the qualitative analysis, given that theme saturation was achieved.

CONCLUSIONS

We developed an evidence-based checklist for medication order review by pharmacists. Pharmacists participating in this study felt that the checklist had value as a training tool for new practitioners, as a way to encourage the development of a systematic and comprehensive approach to medication order review, and as a way to ensure standardized medication order review among pharmacists. Participants felt that a checklist for medication order review has a role in the training of new pharmacists and the standardization of processes for medication order review. Hospital pharmacy departments could consider implementing this checklist as one way to address Accreditation Canada’s requirement for a risk management program to reduce medication-related errors. Emphasizing the potential benefits of the checklist to pharmacists, using the checklist as a training tool, and sharing near misses identified through use of the checklist could contribute to its adoption.