Evaluation of a Hospital Pharmacist Annual Competency Program

Abstract

Purpose: The purpose of this project was to evaluate a pharmacist annual competency evaluation (PACE) program for pharmacists at a single site. The results of this evaluation will be utilized to understand the effectiveness of PACE and provide suggestions for programmatic improvement. Methods: The primary outcome of this study was to evaluate the change in pharmacist composite self-efficacy (PSE) scores before and 1 month after the PACE program. The composite score was composed of self-efficacy related to 5 different clinical tasks. The 5 tasks selected were advanced cardiac life support, chemotherapy, dofetilide, patient-controlled analgesia pump, and alteplase (tPA). Secondary outcomes included evaluating each component of the composite score at 1 month, changes in composite score and components at 1 year, and changes in the generalized self-efficacy (GSE) survey at 1 month. Last, themes were identified from a feedback questionnaire after completion of PACE. Results: For the primary endpoint, PSE composite scores increased by 12.6% from pre-PACE to post-PACE at 1 month (79.6 ± 12.2 vs 89.7 ± 5.8; P < .001). All components of the composite PSE score showed a statistically significant improvement from baseline to 1 month. At 1 year, tPA was the only clinical task that was statistically different from baseline. GSE scores increased by 11.4% from pre-PACE to post-PACE (31.6 ± 3.6 vs 35.4 ± 2.8). No pharmacists required remediation. Conclusion: The competency assessment program led to improvements in pharmacist self-efficacy in dealing with low-volume, high-risk clinical pharmacy tasks. Future studies could assess various forms of competency assessment and timing of these programs to determine the most effective way to ensure pharmacist competency.

Introduction

Competence is defined as a condition of being well-qualified, fit, or capable when performing a task.¹ One strategy for measuring competence is using a competency-based assessment (CBA) program. CBAs have been developed to assess knowledge, clinical skills, and professionalism of both trainees and licensed professionals in health care. Furthermore, CBAs can be used to evaluate an individual’s competence during all phases of professional development to meet specific individual goals.² Competence can be assessed using a combination of performance-based tools, such as simulation-based labs, objective structured clinical examinations, multiple choice questions, and oral examination.³

Although there is extensive literature on CBA programs in health care, particularly in the medical and nursing fields, there is a paucity of literature on the evaluation of CBA programs within the field of pharmacy. Literature has shown effectiveness of CBA programs within pharmacy for licensure examination, relicensure, and competence of pharmacy students.^4-6 Limited literature within the hospital pharmacy setting exists evaluating CBA programs, and mostly focuses on training programs for specific clinical tasks.^7,8 Therefore, it is unclear whether CBA programs are an effective method to evaluate pharmacist’s knowledge, clinical skills, and confidence within the hospital setting.

The purpose of this project was to evaluate a pharmacist annual competency evaluation (PACE) program for pharmacists at a large urban hospital. Evaluation of the PACE program will occur by measuring pharmacist self-efficacy, or pharmacist confidence in their ability to be successful in performing specific tasks, as higher levels of self-efficacy have been associated with both higher levels of motivation and work performance.⁹ The results of this evaluation will be utilized to understand the effectiveness of a pharmacist competency program and provide suggestions for programmatic improvement.

Methods

Research Design

A quasi-experimental, mixed methods evaluation was conducted at Ascension St. John Hospital, a 772-bed community teaching hospital in Detroit, Michigan, from 2017 to 2018. Pharmacists were identified via employment records within the pharmacy department. Pharmacists who were currently employed at St. John Hospital and Medical Center (SJHMC) in a full-time, part-time, or contingent position were included in this study. Pharmacists were excluded if they participated in the creation of all competency stations or did not complete the baseline competency assessment. Institutional review board approval was obtained prior to conducting the study.

The primary outcome of this study was to evaluate the change in pharmacist composite self-efficacy (PSE) scores before and 1 month after the PACE program. The composite score was composed of self-efficacy related to 5 different clinical tasks. Secondary outcomes included evaluating each component of the composite score at 1 month, changes in composite score and components at 1 year, and changes in the generalized self-efficacy (GSE) survey at 1 month. A preplanned subgroup analysis was done to evaluate the effect of residency training on changes in PSE. Last, themes were identified from a feedback questionnaire after completion of PACE.

Precompetency Phase

The PACE program encompassed 3 phases: a precompetency, competency, and postcompetency phase (Figure 1). During the precompetency phase, pharmacists completed a baseline knowledge-assessment quiz for each clinical task along with the PSE and GSE. The baseline quiz consisted of 5 questions that were validated by the competency work group and was delivered through the online learning management system. After baseline knowledge was assessed, pharmacists were provided with educational materials. These materials included a handout that was developed using primary and tertiary literature along with hospital policies that were pertinent to each clinical task. If pharmacists initially did not achieve 100% on the baseline quiz, they retook the quiz as many times as needed to achieve 100% prior to competency day.

Figure 1.

Pharmacy annual competency evaluation structure.

Note. ACLS = advanced cardiac life support.

Competency Phase

Competency station topics were selected by the competency work group (N.F., M.D.-S., C.G.) based on pharmacist-involved clinical tasks at our institution. Pharmacists practice in a decentralized pharmacy model in which they are responsible for order verification, attending code blues, and managing of anticoagulation, patient-controlled analgesia, and antibiotic pharmacokinetic dosing. Stations were chosen if clinical tasks were likely to cause direct patient harm if done incorrectly and could be encountered in clinical practice, although it would be unlikely to be encountered on a daily basis. The 5 competency stations selected were the following: advanced cardiac life support (ACLS), chemotherapy, dofetilide, patient-controlled analgesia pump, and tPA. Pharmacists had received training in these areas on hiring to ensure baseline competence.

Each competency station was developed by a team of 2 to 4 pharmacists. Pharmacists were chosen for each station based on expertise. Each team started development of the station by writing objectives for the clinical task (Table 1). Educational materials were developed to ensure objectives could be met. Competency stations ranged from 10 to 15 minutes in length and pharmacists rotated every 15 minutes. Stations consisted of clinical cases that were developed using objectives for the clinical task and designed to mimic a real-life situation. A list of acceptable answers was developed for each station based on the task. Stations were completed through computer-based order entry, skill demonstration, verbal explanation, or written response. All pharmacists were given the same patient case.

Table 1.

Competency Station Objectives for Pharmacists.

Competency station	Competency objectives
Advanced cardiac life support	● Provide the correct medication administration sequence according to the 2015 AHA ACLS Guidelines ● Recommend the correct vasopressor therapy (medication and initial rate) following a cardiac arrest ● Compound any ACLS medication in a timely manner
Chemotherapy	● Correctly input a chemotherapy order based on a written chemotherapy prescription ● Correctly undergo the proper procedure for double-checking a chemotherapy order and entering the order into hospital computer-based electronic medical system
Dofetilide	● Provide an appropriate initial dofetilide dose ● Provide the appropriate dose adjustment or recommendation based on changes from baseline QTc ● To explain the appropriate monitoring parameters and contraindications for dofetilide use
Patient-controlled analgesia	● Provide an appropriate initial PCA dosing for opioid-tolerant patients ● Explain when it is appropriate to use a PCA with a continuous infusion regimen ● Correctly convert an opioid-tolerant patient to an appropriate oral analgesic regimen ● Provide nonopioid analgesic recommendations that are most appropriate for an individual patient
tPA	● Provide the correct dosing and infusion time of tPA for any indication (stroke, acute massive PE, STEMI, or code situation) ● Appropriately explain the contraindications for the use of tPA ● Correctly compound and verify a compounded tPA order

Note. AHA = American Heart Association; ACLS = advanced cardiac life support; PCA = patient-controlled analgesia; PE = pulmonary embolism; QTc = QT(corrected); STEMI = ST-elevation myocardial infarction.

Competency at each station was assessed using an evaluation checklist. The evaluation checklist was a list of clinical tasks within a competency station in which pharmacists were responsible for completing. An example checklist is displayed in Table 2. The tasks were listed in order in which they were to be completed. Tasks that were determined to be more critical, in that if done incorrectly could lead to direct patient harm, were depicted with an asterisk. Pharmacists were required to complete 90% of the tasks correctly including 100% completion of critical steps to pass the competency. This approach was chosen after discussion of potential cutoffs with a group of 10 pharmacists. Pharmacists were provided with all evaluation checklists for each competency station prior to assessment. Pharmacists involved in the development of a station were assessed at all stations except the one they assisted in developing. Stations were “open note” to simulate the typical pharmacist work environment. No punitive action was taken if a pharmacist was unable to complete a competency station successfully on the first attempt. Remediation was planned to occur at a later date if pharmacists did not pass the competency station and consisted of pharmacists attempting completion of the same competency station.

Table 2.

Example Checklist: Advanced Cardiac Life Support (ACLS Competency.

Associate name		Date
Skill Demonstration	Validation Date: __________ Validator Initials: _________ Criteria Met?	Remedial Validation Date: __________ Validator Initials: _______ Criteria Met?
Part 1: ACLS Medication Management
1. Identify the appropriate cardiac rhythm via EKG.	Yes No	Yes No
2. *To recommend the appropriate medication, dose, and administration based on the EKG and ACLS algorithm pathway.	Yes No	Yes No
3. To draw up/compound a medication appropriately.	Yes No	Yes No
Part 2: Post Cardiac Arrest Medication Management
4. *Recommend the correct vasopressor and initial dose.	Yes No	Yes No
5. To explain the correct monitoring parameters for the selected vasopressor.	Yes No	Yes No
Critical Thinking Questions	Criteria Met?	Criteria Met?
6. What are examples of reversible causes of a cardiac arrest? H’s and T’s	Yes No	Yes No
7. What are the concentrations for norepinephrine and epinephrine bags?	Yes No	Yes No

Instructions:

● Using this checklist, validate skills 1 to 5 with the associate.

● *Critical step.

If all criteria are met, competency has been demonstrated. Sign and date the associate’s ACLS Competency Sheet.

● If both critical steps (indicated by “*”) are unmet, the associate has not demonstrated competency. Return the checklist so that it can be used for the remedial validation and encourage review of the ACLS resources available on MyLearning.

Note: EKG = electrocardiogram.

Postcompetency Phase

Following completion of all competencies, the PSE survey was completed by pharmacists at 1 month and prior to the next competency day 1 year later. Last, an anonymous feedback questionnaire was sent to pharmacists after completion of the competency day.

Program Assessment

Pharmacist self-efficacy was evaluated using the GSE scale (Table 3) and PSE survey (Table 4). Schwarzer and Jerusalem’s validated GSE was used to assess a general sense of perceived self-efficacy.¹⁰ The GSE is a 10-question survey designed using a 4-point scale (1 = not at all true, 4 = exactly true). The GSE composite score ranged from 10 to 40 points, with a score of 40 indicating a higher GSE and a score of 10 indicating a lower GSE. As GSE does not evaluate self-efficacy of pharmacists performing specific clinical tasks, the PSE was used to provide a more accurate representation of pharmacist self-efficacy.

Table 3.

Generalized Self-Efficacy Survey.

Generalized Self-Efficacy Scale
1. I can always manage to solve difficult problems if I try hard enough.  a. Not at all true  b. Hardly true  c. Moderately True  d. Exactly True
2. If someone opposes me, I can find the means and ways to get what I want.  a. Not at all true  b. Hardly true  c. Moderately True  d. Exactly True
3. It is easy for me to stick to my aims and accomplish my goals.  a. Not at all true  b. Hardly true  c. Moderately True  d. Exactly True
4. I am confident that I could deal efficiently with unexpected events.  a. Not at all true  b. Hardly true  c. Moderately True  d. Exactly True
5. Thanks to my resourcefulness, I know how to handle unforeseen situations.  a. Not at all true  b. Hardly true  c. Moderately True  d. Exactly True
6. I can solve most problems if I invest the necessary effort.  a. Not at all true  b. Hardly true  c. Moderately True  d. Exactly True
7. I can remain calm when facing difficulties because I can rely on my coping abilities.  a. Not at all true  b. Hardly true  c. Moderately True  d. Exactly True
8. When I am confronted with a problem, I can usually find several solutions.  a. Not at all true  b. Hardly true  c. Moderately True  d. Exactly True
9. If I am in trouble, I can usually think of a solution.  a. Not at all true  b. Hardly true  c. Moderately True  d. Exactly True
10. I can usually handle whatever comes my way.  a. Not at all true  b. Hardly true  c. Moderately True  d. Exactly True

Table 4.

Pharmacist Self-Efficacy Survey.

tPA

1. I am self-confident that I can dose tPA correctly for each indication (dose, bolus vs infusion dose) with the appropriate infusion time.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

2. I am self-confident on the contraindications of tPA.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

3. I am self-confidence that I know the proper use of tPA in a code blue situation.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

Chemotherapy

1. I am self-confident that I can correctly input a chemotherapy order from a written prescription without assistance.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

2. I am self-confident that I can verify that a chemotherapy order is correct based on a patient’s individual characteristics (weight, BSA etc).

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

3. I am self-confident that I am able to follow the correct procedure when verifying a chemotherapy order.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

ACLS

1. I am self-confident that I am able to provide the correct medication administration sequence during a cardiac arrest according to the 2015 AHA Guidelines.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

2. I am self-confident I am able to recommend the correct vasopressor therapy (medication and initial rate) following a cardiac arrest.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

3. I am self-confident that I can compound a product in a timely manner during a code situation.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

Dofetilide

1. I am self-confident that I am able to provide an appropriate initial dofetilide dose.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

2. I am self-confident that I am able to provide the appropriate dose adjustment or recommendation based on an individual’s change in baseline QTc or renal function.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

3. I am self-confident that I can explain the appropriate monitoring parameters and pharmacist responsibilities for patients on dofetilide.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

Patient-controlled analgesia

1. I am self-confident that I am able to provide an appropriate initial PCA dosing for opioid-tolerant patients.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

2. I am self-confident I can explain the appropriate use of a continuous infusion PCA regimen and know how to dose it.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

3. I am self-confident I can correctly convert an opioid-tolerant patient to an appropriate oral analgesic regimen from a PCA regimen.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

4. I am self-confident that I am able to provide nonopioid analgesic recommendations that are most appropriate for an individual patient.

Cannot do             Moderately certain can do         Highly certain can do

0    10    20    30    40    50    60    70    80   90   100

Note. ACLS = advanced cardiac life support; BSA= body surface area; PCA = patient-controlled analgesia.

The PSE survey was developed using Bandura’s Model and used as a guide for question structure.¹¹ A total of 16 items were given in the PSE. The survey was based on a 100-point scale, ranging in 10-unit intervals from a score of 0 (“cannot do”) to 100 (“highly certain can do”). We used a scale ranging from 0 to 100, with a 10-unit interval. This format provides a stronger predictor of performance than smaller interval scales, such as 1- or 5-interval scales, as the responses are distributed over a wider interval.¹¹ Reliability was measured using Cronbach’s alpha. Content validity was performed to assess the validity of the PSE survey by an expert. The correlation of GSE and PSE score will be used to assess convergent validity. Following completion of all competencies, the PSE survey was completed by pharmacists at 1 month and prior to the next competency day 1 year later. Last, an anonymous feedback questionnaire was sent to pharmacists after completion of the competency day.

Descriptive statistics were used to characterize quantitative data. Paired student t test was used for the primary outcome to compare changes in mean PSE from pre- and 1 month post-PACE. Across all other time points, repeated measures analysis of variance (ANOVA) or mixed ANOVA was used. Interviews were analyzed using a thematic approach. Themes from the feedback questionnaire were identified, coded, and agreed upon by 2 investigators (N.F., C.G.). Thematic analysis was conducted by reading through the responses and identifying potential codes, similar to substantive coding in grounded theory.¹²

Results

Sixty-two clinical pharmacists met the inclusion criteria with 50 pharmacists being included. Eighteen of the 50 included pharmacist had at least 1 year of residency training. Two pharmacists were excluded as they assisted in development of all competency stations, 4 had missing PSE data, and 6 were unable to attend competency assessment dates. Reasons for pharmacist being unable to attend competency dates include medical leave (3) and other employment commitments (3). At baseline, 1 month, and 1 year, reliability of the PSE was as follows: tPA (0.843, 0.933, 0.894), chemotherapy (0.914, 0.940, 0.973), ACLS (0.915, 0.909, 0.924), dofetilide (0.967, 0.824, 0.994), PCA (0.882, 0.903, 0.981).

For the primary endpoint, PSE composite scores increased by 12.6% from pre-PACE to post-PACE at 1 month (79.6 ± 12.2 vs 89.7 ± 5.8; P < .001). All components of the composite PSE score showed a statistically significant improvement from baseline to 1 month (Figure 2). At 1 year, there was no overall difference in PSE scores compared with baseline (82.8 ± 12.4 vs 78.1 ± 13.9, respectively; P = .114) and tPA was the only clinical task that was statistically different. No difference in PSE score change after 1 month was seen between those that had residency training or did not (10.3 vs 9.7, P = .58). GSE scores increased by 11.4% from pre-PACE to post-PACE (31.6 ± 3.6 vs 35.4 ± 2.8). GSE and PSE change were positively correlated (0.57, P < .01). No pharmacists required remediation.

Figure 2.

Changes in pharmacist self-efficacy (PSE) self-efficacy score.

Note. ACLS = advanced cardiac life support; PCA = patient-controlled analgesia.

Twenty-one pharmacists responded to the anonymous feedback questionnaire after PACE. The majority of pharmacists felt that PACE was very beneficial (61.9%) or somewhat beneficial (33.3%). One pharmacist (4.8%) responded that PACE was not very beneficial. Pharmacist felt that PACE made them more knowledgeable, improved their confidence, and provided them with high-quality learning material. Pharmacists frequently mentioned preference of this format over lecture and online tests. Pharmacists thought the day could be organized better and disliked having to do competencies during their shift. Most comments around organization were in reference to scheduling and logistics surrounding staffing. Pharmacists also felt like they could use more time at each station. Some pharmacists disliked that stations allowed open notes, although others liked the lower stress environment that open notes allowed.

Discussion

Pharmacists who completed the PACE program showed improvement in self-efficacy in completing low-volume, high-risk clinical pharmacy tasks. Interestingly, we saw improvements despite starting at a relatively high task-specific self-efficacy (~80 out of 100) at baseline. We were surprised that many of the pharmacists stated they preferred this method over lecture and quizzes. We believe the overall success of this program was a result of strategies employed during the program design phase. Many of these strategies were identified through communication with nurse educators at our institution and review of the medical literature. First, we were transparent in what we believed pharmacists needed to accomplish and provided the competency checklist prior to the competency day. Second, we involved our pharmacists in the creation of the stations. We believe this was critical to make the program sustainable past the first year and to obtain buy in. Third, in addition to policies each station provided an educational handout. This approach could be applied to the creation of future competency stations for pharmacist tasks we did not evaluate, such as pharmacokinetic dosing.

Despite improvements seen at 1 month, most stations self-efficacy declined over the course of the year. Competency may need to be reassessed more frequently than on a yearly basis as self-efficacy for most tasks returned to baseline after 1 year. The frequency of program occurrence needs to be balanced with institutional needs. Although we split the work among many pharmacists, it would be challenging to conduct the program more frequently than every 6 months. Feedback obtained from our post-PACE survey will lead to changes in some strategies in the future. The biggest adjustment will be scheduling competency evaluation during shift changes, increasing the pharmacist overlap time period on the competency day, and building competency day into the staffing schedule. We hypothesize this will address concerns surrounding the organization of the day and ensure time is not taken away from patient care. In addition, we will ensure each station is designed to have an extra 5 minutes of time to ensure ample time is provided for each station.

Our study mostly agrees with previous studies that discuss the development of pharmacist competency programs. One study focused on improving antimicrobial stewardship competency in 17 hospital pharmacists.⁸ This study found improvements in knowledge after completion of the program, although changes in self-efficacy were not measured. The program was delivered over a 4-month period and was primarily delivered through a lecture-based format. At the end of the study 94% of pharmacists thought the program should be offered annually and 65% thought the program should occur more frequently. Of note, 5 of 17 pharmacists required remediation. This was likely because the skills assessed were new to some pharmacists. In our study, the skills assessed were something pharmacists were already expected to be able to complete, which may explain why no pharmacists needed to undergo remediation. Our approach is not intended to assess new information, but instead ensure competence of pharmacists for tasks they have previously completed. Last, this study agrees with our finding that suggest competency programs may need to occur more frequently than on a yearly basis.

Improvements in knowledge and comfort were also seen in a study before and after a pharmacist code blue training program.⁷ Material in this program was delivered through a lecture format. Improvements were seen 1 week later; however, longer term follow-up was not performed. Remediation was not mentioned within the article. The main difference between our study and the previously mentioned literature was the focus on a specific task, while our program focused on a variety of clinical tasks. In addition, the purpose of the previously published programs was to teach, where our program was to ensure competence.

This study had limitations. First, self-efficacy changes were not compared with a control group; therefore, we cannot compare our results with pharmacists who did not undergo a competency assessment program. However, it would have been difficult to prevent cross-contamination of groups as pharmacists could have communicated with one another. Second, not all pharmacist participated, which may have influenced study. Third, the PSE survey used in this study was not previously validated. However, Bandura recommends measuring task-specific self-efficacy and we used his model to develop the PSE.¹¹ In addition, the PSE correlated with the GSE, which is a validated measure of self-efficacy and was reliable. Fourth, competence was not measured at 1 month and 1 year, although this was not our primary outcome. Last, results of competency day may vary by institution staffing model and may be dependent on the hospital.

Conclusion

The competency assessment program led to improvements in pharmacist self-efficacy in dealing with low-volume, high-risk clinical pharmacy tasks. Future studies could assess various forms of competency assessment and timing of these programs to determine the most effective way to ensure pharmacist competency.