Learning Outcomes and Behavioral Changes With a Pharmacy Continuing Professional Education Program

Abstract

Objective

To examine the results of an interactive curricular-based 3-month long pharmacy continuing professional education (CPE) program on short- and long-term learning outcomes and behavioral changes of current and potential preceptors.

Methods

A CPE program was developed that covered specific disease states and teaching skills. The goals of the course were to provide knowledge and skills needed to precept the College's new advanced rotations, and to attract more pharmacists to serve as preceptors. The course included pre-readings and 3-hour long workshops over a 3-month period of time. Learning and behavior were assessed by a pretest and posttest and follow-up survey.

Results

Ninety-nine pharmacists completed the course. Fifty participants completed the assessments and were included in the analysis, yielding a usable response rate of 52%. However, only 30 participants completed the follow-up survey instrument, resulting in a response rate of 30%. There was a significant increase in test scores between the pretest and posttest, but a significant decline in test scores between posttest and the follow-up survey.

Conclusions

An interactive, curricular-based pharmacy CPE program is effective in increasing learning, but participants may not maintain the acquired knowledge over time. The program was not an effective mechanism to attract pharmacists to serve as preceptors.

INTRODUCTION

Continuing professional education (CPE) is a thriving enterprise across many disciplines and professions.^1-5 Professionals have a plethora of options to choose from regarding educational content as well as delivery systems.^1-5 These programs may include attendance at live conferences and seminars, or independent home study via journals, CDs, or web-based programs. For most health-care practitioners, CPE is closely linked to maintaining licensure, and therefore embedded in professional practice. Pharmacists were first required to participate in mandatory CPE in 1967 in Florida and Kansas.⁶ Now, mandatory CPE is required by 51 boards of pharmacy in the United States as a prerequisite for relicensure.⁷ During the last reporting year, the Accreditation Council for Pharmacy Education (ACPE), reported that 18,773 unique programs were offered to pharmacists and over 2 million pharmacists participated in these programs (e-mail from ACPE, March 22, 2005).

One of the standards set by ACPE in their “Criteria for Quality” focuses on learning assessment.⁸ The criterion calls for evaluation mechanisms that allow the participants to assess their own learning; however, long-term learning outcomes, changes in practice behaviors, and patient outcomes are infrequently monitored and reported.

The extant literature on the effectiveness of CPE interventions is mixed and limited to specific disciplines. Continuing medical education has the largest and most robust body of evidence examining the effectiveness of CPE on learning, practice change, and patient outcomes. In a recent review article, Davis noted that educational interventions have a positive effect on physician behavior and almost half of the studies reviewed reported an improvement in patient health care outcomes.⁹ However, this review examined only studies that were randomized controlled trials. Only 14 studies met the criteria and were included in the review. What is particularly noteworthy in this review is that the evidence suggests how courses are delivered is critical in recognizing positive outcomes. Programs that used interactive teaching methods were more likely to see a positive outcome than programs that used more traditional lecture methods.⁹

The literature on outcomes of CPE in other professions is sparse. In pharmacy, few studies exist that examine the cognitive gains, behavioral changes, and patient outcomes of pharmacy CPE programs. The results are mixed. Studies have found, as a result of participation in pharmacy CPE programs, significant changes in emphatic responses to written stimuli and role-play, and an increase in documentation of clinical interventions and monitoring of patient outcomes.^10,11 Barner and Bennett examined the effect of a 9-month pharmaceutical care certificate course and found an increase in participants' confidence in the ability to provide pharmaceutical care and an increase in the ability to identify and resolve drug-related problems and patient monitoring, but no significant differences in the frequency of actually performing these tasks.¹² Chen et al reported short-term increases in pharmacists' knowledge and attitudes toward diabetes as a result of a pharmacy CPE program.¹³ Fjortoft and Schwartz reported short- and long-term cognitive gains from a pharmacy CPE program, but no behavioral or practice changes.¹⁴ Engel, Nutescu and Joseph reported that many participants of a anticoagulation clinic certificate program went on to develop those services at their own site after program completion, but it is unclear if this was a statistically significant number or if the services were in the planning stages at the participants' home institution prior to their participation in the certificate course.¹⁵

Other studies have examined pharmacists' preferences and perceptions of pharmacy CPE. Pharmacists rated CPE programs as effective in increasing their knowledge of the subject taught, but did not find CPE effective in changing their clinical practice behavior.¹⁶ Pharmacists have also reported that they are far more interested in CPE than certificate courses or post-baccalaureate academic programs.¹⁷

The evidence regarding the effectiveness of pharmacy CPE suggests that pharmacy CPE is effective at changing knowledge and attitudes, but its impact on changing behavior has yet to be sufficiently documented. In addition, its larger effect on patient outcomes is not known. The objective of this study is to examine the results of an interactive curricular-based 3-month long pharmacy CPE on short- and long-term learning outcomes and behavioral changes.

METHODS

An interactive, curriculum-based 3-month pharmacy CPE was developed and offered by the Chicago College of Pharmacy at Midwestern University. The title of the program was “Advanced Training for Advanced Patient Care.” The goals of the program were: (1) to provide pharmacist preceptors with the knowledge and skills needed for the Colleges' new advanced rotations, and (2) to attract pharmacists to serve as preceptors. The course consisted of pre-readings, four 3-hour workshops that were interactive in nature and utilized cases to illustrate concepts learned from the readings, and pre- and post-learning assessments. The course covered a number of topics grouped for the 4 workshops: asthma and smoking cessation, anticoagulation and hypertension, diabetes and dyslipidemia, and teaching skills. The topics were chosen to update pharmacists on key disease states covered in required rotations. The workshop on teaching skills provided pharmacists with key teaching skills to enable them to be more effective preceptors. That workshop covered the teaching process and the changing role of the preceptor. Objectives for each of the topics were developed and included basic comprehension, application, and evaluation of knowledge content. The program was offered free of charge to participants and was supported by College funds. Upon completion of the program, each participant received 20 contact hours of CPE. The course was offered twice in 2003, once in mid-March through early May and again in mid-September through early November.

Recruitment of participants began about 8 weeks prior to each scheduled program. A brochure was developed that described the program and was mailed to all registered pharmacists in the greater Chicago area. About 7500 brochures were mailed. Ninety-nine pharmacists enrolled in and completed the program. Sixty-two of the participants were currently serving as preceptors, and the rest were potential preceptors. Fifty-seven pharmacists participated and completed the March-May course and 42 pharmacists participated and completed the September-November course.

The pharmacists were given a preassessment at the beginning of 3 of the 4 workshops. The fourth workshop focused on teaching skills and was not included in the analysis of learning gains due to the different nature of the content. The preassessments were each 10 questions about basic practice knowledge areas covered in the workshops and pre-readings. The same assessment was administered at the conclusion of each of the 3 workshops. Each question had a right or wrong answer, and participants received a score, which reflected the number of correct answers. A follow-up survey instrument that was coded to match the pre/post assessments was mailed to all participants. This survey instrument was administered in June 2004. The follow-up survey instrument included a modest monetary incentive. The follow-up survey included all of the questions from the 3 workshop assessment, plus questions regarding the participants demographics and service as a preceptor. A reminder mailing was sent 2 weeks later. Given that the first survey was lengthy and may have been a deterrent to responding, an abbreviated survey was sent on the second mailing, which just included demographic questions and questions regarding whether they were preceptors.

Data analysis was performed using SPSS for Windows (version 11.5, SPSS, Inc, Chicago, Ill, 2002). Repeated measure ANOVA is the preferred statistical test for comparing data over 3 points in time for dependent groups (pretest, posttest, follow-up survey). However, given that the sample for the third time or those who completed the entire follow-up survey instrument was only 30, or roughly one third of the original sample, 2 paired t tests were conducted. Descriptive statistics were used to describe the sample demographic characteristics.

RESULTS

Thirty participants responded to the first mailing, which was the long survey instrument that included all the learning assessment questions from the workshops. The response rate at this point was 31%. An additional 20 participants responded to the second mailing, which was the short survey instrument that asked only demographic questions and questions regarding precepting activities. The total response rate then increased to 52%. Chi square analysis was conducted on all the demographic variables to compare respondents to phase 1 of data collection and respondents to phase 2 of data collection. No statistically significant differences were found between the 2 groups in gender, years in practice, education and training, and primary practice setting. Therefore, respondents from both phases of data collection were analyzed as one group.

A description of the sample is presented in Table 1. The majority of the sample were female, practiced in corporate community settings, and had been in pharmacy practice 21 years or more. The first paired t test on the pre/post workshop assessments (time 1 and time 2) indicated successful learning had occurred for each of the 3 workshops. Posttest scores were higher than pretest scores for all 3 workshops. The results of this analysis are described in Table 2. The second paired t test (time 2 and time 3) indicated a statistically significant decline in average scores for all 3 workshop content areas. The results of this analysis are described in Table 3.

Table 1.

Demographics of Participants in a Survey Regarding a Continuing Professional Education Program

Table 2.

Comparison of Pretest and Posttest Scores of Participants in a Pharmacy Continuing Professional Education Workship

*Score range 0-10

†p < 0.05

Table 3.

Comparison of Scores on the Posttest and Follow-up Survey (N = 30)

*Score range 1-10

†p < 0.05

The second goal of the program was to attract new pharmacists to serve as volunteer preceptors. Out of the 99 pharmacists who attended the program, 37 were not preceptors. Out of that 37, only 4 have since volunteered to serve as a preceptor. Reasons cited by the participants revolved primarily around time. Respondents cited that they did not have time to precept students due to staffing responsibilities or because their company or institution did not provide them with enough time for this service. The results of this analysis are described in Table 4.

Table 4.

Most Important Reasons Stated by Pharmacists for Not Serving as Preceptors (N = 37)^*

^*Total who responded to this item

DISCUSSION

These data indicate that pharmacists have short-term cognitive gains from an interactive curricular-based pharmacy CPE program. Participants, as a group, had a statistically significant increase in assessment scores from the pre-workshop assessment to the post-workshop assessment. However, participants then had a decrease in score average as a group from posttest to follow-up survey. The follow-up survey was administered 8 to 13 months after course completion. While the scores did not revert to the pre-assessment level, there was still a significant decrease from the posttest level. This suggests that the respondents were not successful in long-term retention of all that was learned from program participation. There are several possible explanations for this. First, adults are motivated to participate and attend CPE programs for a variety of reasons. There are several typologies that attempt to categorize reasons for participation. The most well-known and earliest model for explaining adult participation in CPE was developed by Robert Houle in 1961 and suggests that adults participate in CPE because they are either goal-oriented (participants are using education as a means of achieving a specific goal), activity-oriented (participants are motivated to attend for the social interaction), or learning-oriented (participants attend for the joy of learning for its own sake).¹⁸ Pharmacists, like other health professionals, are mandated to complete CPE. Pharmacist choices of which program to attend or which monograph to read may be driven by cost, convenience, topic, and/or need. Lastly, the assessment questions may not have adequately assessed what the participants wanted to learn and learned, but rather, what the faculty thought was important.

The questions in the learning assessments were all multiple choice or fill in the blank. Some questions assessed recall and comprehension of concepts taught, others assessed application and evaluation of concepts taught. For example, “The goal blood pressure for patients with hypertension and diabetes is:________(fill in the blank).” Another example is “The goal INR range for a patient with a mechanical mitral valve replacement on warfarin is:_________(fill in the blank). The majority of the respondents practiced in a community setting. These pharmacists may not be in a practice environment that reinforced specific content from the program, for example, the content on anticoagulation. Therefore, since they did not use this information, they did not have the ability to recall it on the follow-up survey.

The secondary goal of the program was to recruit more preceptors. This goal was minimally achieved. Out of the potential 37 new preceptors, only 4 served as preceptors in the following year. The data suggest that barriers to precepting exist on 2 levels. The first level is at the individual pharmacist level. Pharmacists cited lack of confidence and skills as reasons for not volunteering to precept pharmacy students. Some pharmacists simply do not believe they have the knowledge base and the skills needed to precept doctor of pharmacy students. The second level of barriers concerns the pharmacists' employer or institution. These data suggest that pharmacists believe that their institutions do not support precepting students. This is evident in the statements regarding lack of time or other responsibilities preventing them from volunteering to precept pharmacy students. Whether this conclusion is real or perceived is not known. Regardless of whether this is a perception or reality, colleges of pharmacy must grapple with this issue and work with institutions to help create environments where precepting students is valued.

Given the extant literature and the results of this study, there is some evidence that learning occurs as a result of pharmacy CPE, but still little evidence that pharmacy CPE can affect practice change or patient outcomes. If indeed pharmacy CPE serves to protect the public's well-being and safety, CPE providers and the profession need to measure the effects of CPE activities on practitioner performance and behavior. This kind of assessment requires thoughtful planning and a long-term commitment to evaluating program outcomes.

Adults learn within the context of real-life problems in real-life settings. The theory of situated cognition states that one cannot separate the content and the learning process from the situation in which the learning is presented or applied.¹⁹ Practice-based learning is being explored in continuing medical education and is centered around the idea that physicians are continuously learning and improving while in their place of practice.²⁰ The underlying assumption, which is supported by evidence presented earlier, is that learning that results in sustained knowledge does not occur in isolated faculty content-driven CPE programs. This is not news to pharmacists. In their study of pharmacists' lifelong learning behaviors, Hanson and DeMuth described the most common learning activity reported by pharmacists as “communicating with one's peers.”²¹ It is not clear from the study where this communication occurs or what is the content of the communication. However, while it is not explicitly stated, the implication is that pharmacists learn by discussing real life problems with their colleagues in real life situations.

Finally, the International Learning Unit (ILU) has been recently developed as a new mechanism by which to provide evidence of participation in CPE programs. The ILU is defined as an outcomes-based evaluation composed of 5 content items for 0.1 learning units. A pass rate of 80% is required.²² Whether or not the ILU will be widely adopted by CPE providers and organizations is not known, but it is a radical departure from the existing method of measuring “seat time” as evidence of learning.

The study has limitations. The low number of participants who completed the long-term follow-up survey instrument, which included all the learning assessment questions, was disappointing and affected the analytic plan. In addition, given that the survey instrument was mailed to participants' homes and was essentially a “take-home” test, it is not known whether the respondents answered the questions independently, without using any resources.

The program studied was offered twice in 2003, once in the spring and once in the fall. It is not known whether the study sample participated in the spring or fall course. The follow-up survey instrument was administered in summer 2004. For some respondents, this was 8 months after course completion, while for other respondents, this was 13 months after course completion. The time differential could have impacted scores on the follow-up survey.

Lastly, while the content of the program was not modified from the spring to the fall, one faculty member who was involved in the development and teaching of the course left the University and was replaced with another faculty member. This person followed the same delivery methods; nevertheless, the learning environment was changed.

CONCLUSION

These results suggest there are cognitive gains from continuing professional educational experiences, but those gains declined over time. The secondary goal of signing up program participants to serve as preceptors for pharmacy students was minimally achieved. Reasons cited for this included both individual reasons and institutional reasons.

The results of this study raise more questions regarding the effectiveness of CPE on learning that lasts, behavioral or practice changes, and patient outcomes. Clearly, CPE as a mechanism to ensure pharmacists' continued competence is being and should continue to be scrutinized.