Long-term Effectiveness of Online Anaphylaxis Education for Pharmacists

Abstract

Objective. To evaluate the long-term effectiveness of an Australasian Society of Clinical Immunology and Allergy (ASCIA) anaphylaxis e-learning program compared to lectures or no training.

Design. A controlled interrupted-time-series study of Australian pharmacists and pharmacy students who completed ASCIA anaphylaxis e-learning or lecture programs was conducted during 2011-2013. Effectiveness was measured using a validated test administered pretraining, posttraining, and 3 and 7 months after training.

Assessment. All learning groups performed significantly better on all posttests compared to the pretest, and compared to a control group (p<0.001). The proportion of e-learners achieving the minimum standard for anaphylaxis knowledge improved from 45% at pretest to 87% at 7 months.

Conclusion. The ASCIA e-learning program significantly increased anaphylaxis knowledge. The high proportion of participants achieving the minimum standard at 7 months indicates long-term knowledge change.

INTRODUCTION

Anaphylaxis is a severe, progressive allergic reaction that is rapid in onset and may cause death.¹ The incidence of anaphylaxis has dramatically increased over the past decade,^2-9 with more cases occurring in the community setting than in the hospital setting.¹⁰ Early diagnosis of anaphylaxis and treatment with adrenaline is essential to prevent fatalities, and deaths are more common in patients with a history of asthma.^10-14 Adrenaline is internationally recognized as the first-line treatment for anaphylaxis, with auto-injector devices universally recommended as first aid for anaphylaxis occurring in the community setting. Prescriptions for adrenaline auto-injector devices should be accompanied by a device-specific emergency action plan.^11,14-19

In Australia, pharmacists supply adrenaline auto-injectors to patients who present a physician’s prescription, or to those patients without a prescription when an individual therapeutic need is established by the pharmacist. In addition, pharmacists sell these devices to Australian schools and childcare services to facilitate emergency treatment.^20-23 With each distribution, pharmacists should educate patients (or their agents) about anaphylaxis, confirm they have a device-specific ASCIA Action Plan for Anaphylaxis, and advise them regarding the correct use and storage of the adrenaline auto-injector.^14,20,24 Pharmacists also provide collaborative care (usually with a family physician or specialist physician) to patients with comorbid conditions including asthma, offer advice about and sell medicines for the treatment of allergies, and are sometimes called upon to provide first aid for patients with acute anaphylaxis. Changes to devices in Australia, including the addition of Anapen in 2010 and the change of EpiPen to a new-look device in 2011, highlighted the potential for patient confusion and the importance of up-to-date pharmacist advice. Therefore, pharmacists need to have a thorough knowledge of anaphylaxis as well as adrenaline auto-injectors.

In 2011, the Australasian Society of Clinical Immunology and Allergy (ASCIA) launched “ASCIA Anaphylaxis e-training for pharmacists” to meet the need for accurate, consistent anaphylaxis education. This e-learning package complemented existing ASCIA anaphylaxis e-training programs for schools and childcare services and other health professionals. The importance of ensuring that this e-training is effective at increasing anaphylaxis knowledge is paramount to reducing the risk of fatal anaphylaxis in the community. Long-term effectiveness is of prime importance because the incidence of anaphylaxis is increasing and errors in management because of waning knowledge may result in a poor outcome for the patient.

Effectiveness studies of e-learning in health professionals’ education indicate e-learning is as effective as traditional methods at increasing knowledge immediately after training.^25-29 However, there is little evidence to support the long-term effectiveness of e-learning to enhance knowledge, or to meet a minimum knowledge requirement, such as a minimum pass score. In this study, we sought to evaluate the immediate and long-term impact of ASCIA Anaphylaxis e-training for pharmacists on anaphylaxis knowledge, compared to ASCIA anaphylaxis lecture training or no training. We hypothesized that ASCIA Anaphylaxis e-training for pharmacists would be as effective as ASCIA anaphylaxis lecture training at increasing short and long-term knowledge, meeting a minimum standard for anaphylaxis knowledge, and teaching the steps required for adrenaline auto-injector device administration. We also hypothesized that both programs would be superior to no training.

DESIGN

This controlled, interrupted time-series study was conducted in Australia between August 2011 and April 2013. The University of Western Australia Human Research Ethics Committee gave ethics approval for the study in July 2011.

Intervention participants were eligible if they were pharmacists or pharmacy students within Australia. Pharmacists included professionals registered with the Pharmacy Board of Australia (PBA) and pharmacy interns who held provisional registration as a pharmacist with PBA and who were completing practice hours under the direct supervision of a registered pharmacist. Pharmacy students were individuals enrolled in an approved course of study in the field of pharmacy at an Australian university. Control participants were students of medicine or pharmacy at the University of Western Australia.

All participants were recruited using a convenience approach. E-learning participants were recruited from across Australia while registering online for ASCIA anaphylaxis e-learning between September 2011 and May 2012. Lecture participants were recruited while attending ASCIA anaphylaxis lectures in Perth, Western Australia, between August and September 2011. As the e-learning and lecture participants were separated by both place and time, randomization to either intervention arm was not possible. Control participants were recruited while attending regular university lectures and tutorials in Perth, Western Australia, in September 2012. The aims, objectives, relevance of the study, and option to participate were explained, and all participants provided written, informed consent prior to enrollment in the study (e-learning participants gave consent by selecting an “I Agree” checkbox online). Participants also completed a short demographic survey, which included the variables gender, age group, main job in pharmacy, type of control student, postal code of main workplace, and graduation year.

ASCIA Anaphylaxis Training for Pharmacists

The training program was developed by ASCIA in consultation with the Pharmaceutical Society of Western Australia, the Pharmaceutical Society of Australia, the Pharmacy Guild of Australia, and the Society of Hospital Pharmacists of Australia. The training was advertised as an accredited continuing professional development (CPD) activity with these organizations, as well as through professional newsletters, magazines, and websites. Table 1 provides an overview of the training. Briefly, e-learning and face-to-face lecture programs consisted of the same 4 modules, each designed to take 15 minutes to complete. E-learning was presented as a series of slides using Metamorphosis software (Easy Authoring, Sydney, Australia). Face-to-face lectures were delivered as Microsoft PowerPoint slides.

Table 1.

An Overview of ASCIA Anaphylaxis Training for Pharmacists

E-learning participants were allowed to complete training at their own pace, although it was recommended that all modules and tests be completed within a 2-week period. Explanatory notes for slides accompanied the e-learning program to ensure equivalence with spoken material presented in face-to-face lectures. Participants were encouraged to obtain their own trainer adrenaline auto-injector devices and practice the steps required for their administration while completing the program.

Lecture participants attended one of three 1-hour, face-to-face lectures. To ensure consistency across lectures, a dedicated ASCIA-approved lecturer (a clinical immunology/allergy medical specialist) delivered all lectures in the study. Participants were provided with trainer adrenaline auto-injector devices for the duration of the lecture only, and a hands-on activity was included to demonstrate the steps required for administration.

Completion of a posttest was a requirement for CPD credits in both programs. Understanding the correct answer is considered part of the learning experience, and e-learning participants received brief and immediate online feedback (as part of the learning program) on their test results, including the correct answers to questions. Lecture participants were able to access the correct answers from researchers in the lecture room after completing the posttest. Neither group received a link to or copy of the test answers, nor were answers provided at the 3-month or 7-month follow-up tests. For students, the training did not form part of any university assessment.

Control participants attended a lecture on women’s health, participated in a discussion session on professional pharmacy practice, or completed a pharmacy-dispensing laboratory session. All control interventions lasted 60 minutes.

EVALUATION AND ASSESSMENT

Knowledge gain was assessed using a 12-question test, the Anaphylaxis Training Pharmacist Assessment Tool (AT-PAsT), which we developed and validated prior to use in the study.³⁰ We used a combination of multiple-choice, yes/no, and order-the-steps questions to measure knowledge of the prevention, identification, and management of anaphylaxis in the community setting. An expert group of 10 allergy and immunology physicians and 2 clinical pharmacists developed the test questions and assessed content validity. Modifications to wording and content changes were made to 2 questions. Face validity was evaluated in a group of 15 pharmacists and 5 pharmacy students, and all agreed they understood the questions and response options. This test was pilot tested on a group of 67 pharmacists who attended an ASCIA anaphylaxis lecture in Adelaide, South Australia, in July 2011. Although the test demonstrated a significant improvement in knowledge scores after the lecture (8.2-11.2 points, paired t test; p<0.001), 4 questions did not show response change and thus may have overstated knowledge (McNemar test; p>0.5). These questions were redeveloped, reviewed by the expert group and pharmacists for content and face validity, and incorporated into the final version of the AT-PAsT.

The test was administered immediately before training and immediately after training, then 3 and 7 months after training. To reduce practice effect, the questions and their response options were reordered on each test. Participants in the e-learning group completed the pretest and posttest online as part of the e-learning program. Participants in the lecture and control groups completed the pretest and posttest on paper in the lecture or tutorial room. Pharmacy students completed the 3-month follow-up test on paper. All other tests were completed through the online research suite Qualtrics (Qualtrics, Utah). When follow-up tests were due, participants received an e-mail notification and up to 5 e-mail reminders. The follow-up tests remained accessible for 2 weeks. Three prizes (cinema tickets or retail vouchers), with a maximum value of AU$100, were provided as an incentive to complete each of the follow-up tests. Of the participants who completed the 3-month and 7-month follow-up tests, 1 winner from each group (e-learning, lecture or control), was drawn at random. There were no other incentives provided in the study.

As there were no reliable estimates for expected standard deviation in score, we did not conduct a priori sample-size calculations. However, a post hoc power calculation, using the 7-month posttest sample size of 30 in the e-learning group and 50 controls with an observed standard deviation of 1.4 points, showed that the study had 86% power to detect a difference in score of 1 point between groups at the 5% level of significance. Calculations for all other sample sizes in the study groups yielded power estimates between 86% and 100% for between-group and within-group comparisons.³¹

Analysis

All analyses were performed using SPSS version 21 (IBM, New York), and reported as 2-sided p-values with a 5% level of significance. A linear mixed-effects model with post hoc pairwise analysis was used to evaluate changes in short-term and long-term knowledge within and between learning and control groups. We specified score as the dependent variable, with group (e-learning, lecture pharmacists, lecture pharmacy students, or control) and test (pretest, posttest, 3-month and 7-month tests) as covariates. We compared models with and without demographic covariates (gender, age group, main job in pharmacy, type of control student, postal code of main workplace, and years since graduation). As the majority of the sample was from Western Australia, we converted the postal code of main workplace to 2 geographic areas, Western Australia or all other Australian states. Analyses were restricted to participants who had valid, non-missing data for all variables in the model.

We compared the proportion of participants within and between learning groups who, at each test, achieved the minimum standard for anaphylaxis knowledge (score≥9 out of 12) and correctly ordered the steps for EpiPen and Anapen device administration. The Pearson chi-square test was used for between-group comparisons and the McNemar test was used for within-group comparisons. Data for individual answers to the device-ordering questions for the e-learning group were not available for the pretest and posttest (only the overall scores were available). Therefore, we could only make comparisons between the 3-month and 7-month tests in the e-learning group.

Results

We recruited 383 participants (277 intervention and 106 controls) to the study (Table 2). There was significant diversity across all 4 groups based on demographic variables (p<0.001). E-learning and lecture pharmacists groups were similar by age group and years since graduation, but differed by gender, main job in pharmacy, and location of main job (Table 2). Completion rates across the 4 tests ranged from 100% at posttest, to 47.2% at 7 months (Figure 1), and were similar between groups (p=0.91 at 7 months).

Table 2.

Participant Characteristics by Intervention and Control Group at Pretest (count and %)

Figure 1.

Study groups, participation and completion rates by group and test.

Mean knowledge scores were significantly different by group and test (p<0.001, Table 3). With all demographic variables in the model, there were no significant differences in score by age group (p=0.28), main job in pharmacy (p=0.06), type of control student (p=0.082), state of main workplace (p=0.96), or years since graduation (p=0.56). Score initially differed significantly by gender (p=0.05); however, this effect was lost when non-significant variables were removed from the model (p=0.06).

Table 3.

Mean Anaphylaxis Training Knowledge Assessment Score by Group and Test^a

Figure 2 and Table 3 show mean AT-PAsT scores by group and test. There was a significant and sustained improvement in anaphylaxis knowledge after training in all learning groups (paired t tests, p<0.001 for all comparisons). Mean scores improved by 3.3, 2.8, and 4.6 points immediately after training in the e-learning, lecture pharmacists, and lecture pharmacy students groups, respectively, but decreased in the control group. Mean scores decreased significantly from posttest scores in all learning groups at the 3-month test (a respective score decrease of 1.6, 1.4, 1.7 points). At 7 months, mean scores improved and were above the minimum standard in all learning groups. There were no significant changes in mean score in the control group after posttest.

Figure 2.

Mean anaphylaxis knowledge assessment scores by group and test.

Figure 3 shows the change in mean AT-PAsT scores by group over time. All learning groups performed significantly better on all posttests compared to control (p<0.001 for all comparisons). E-learning and lecture pharmacist participants had similar scores across all tests except posttest, where e-learning scores were slightly higher (0.65 points, p=0.04). Lecture pharmacy students had the greatest gains in knowledge of all learning groups, yet lower scores. It was not possible to compare e-learning scores for pharmacy students with lecture pharmacy students’ scores, as only 3 pharmacy students completed the e-learning program.

Figure 3.

Change in mean anaphylaxis knowledge assessment scores by group over time. E-learning scores were similar to lecture pharmacist scores at pre-test (p=0.62), 3-month test (p=0.79) and 7-month test (p=0.31). Control scores were significantly lower than all intervention scores after training (p<0.001).

There were significant and sustained improvements in the proportion of learners achieving the minimum standard for anaphylaxis knowledge after training (Table 4). Less than 46% of e-learning and lecture pharmacists achieved the minimum standard before training; however, 7 months after training, over 80% achieved this standard. The improvement in the proportion of lecture pharmacy students achieving the standard was almost tenfold: from 6.7% pretest to 61.8% at 7 months.

Table 4.

Learners Achieving the Minimum Standard for Anaphylaxis Knowledge and the Correct Device Administration Steps by Group and Test.

Although there were sizeable gains in the proportion of lecture participants who passed the device-ordering questions after training, these gains were not sustained over time (Table 4). At 7 months, 63.3% of e-learning participants and 61.5% of lecture pharmacists correctly ordered the 4 steps for both EpiPen and Anapen, an improvement of around 15% in each group from pretest.

DISCUSSION

Pharmacists play a vital role in the management of anaphylaxis patients. Easily accessible, effective anaphylaxis education is essential to fulfil this role.^32-35 However, there is little evidence of the effectiveness of anaphylaxis training for pharmacists. We evaluated the e-learning program, ASCIA Anaphylaxis e-training for pharmacists, and measured its effectiveness in terms of knowledge change.

This education program was associated with significant and sustained improvements in anaphylaxis knowledge. Short-term knowledge gains (on average, a 39% improvement in mean score) were similar to immediate gains seen in other pharmacy e-learning effectiveness studies.^36-39 Persistence of knowledge 7 months after training was high: almost 90% of e-learners achieved at or above our minimum standard for anaphylaxis knowledge, compared to 45% of the same learners before training. Thus, the results add long-term effectiveness to the existing body of e-learning pedagogical research,^25-29 and more importantly, demonstrate that this education program is effective long-term. ASCIA Anaphylaxis e-training for pharmacists was as effective as lecture training, and significantly more effective than no training, at improving short-term and long-term anaphylaxis knowledge in pharmacists. We were unable to demonstrate effectiveness of this e-learning program in pharmacy students due to low numbers of student participants. Even so, lecture training was effective at improving short-term and long-term anaphylaxis knowledge in pharmacy students, and other research has demonstrated short-term effectiveness of e-learning in pharmacy students in different subject areas.^38,40-42 Therefore, it is likely that this e-learning program would also be effective for pharmacy students. There was no change in anaphylaxis knowledge in those who did not receive training. This is consistent with the broader literature for short-term e-learning effectiveness.^26,29 However, as far as we know, this is the first study to demonstrate long-term differences in an e-learning group compared to a group who did not receive training.

An essential part of anaphylaxis education for patients is hands-on training in the use of adrenaline auto-injectors. Although pharmacists are ideally placed to deliver this training, there is evidence that the majority of anaphylaxis patients do not receive it.^24,43,44 People who do not know how or when to use their adrenaline auto-injector may elect not to do so in an emergency, or may incorrectly activate the device.⁴⁵ Devices and procedures change over time, and there is a constant need to improve pharmacists’ skills in this area, so they can better train those at risk of anaphylaxis.^{13,35,43,44,46} Approximately two-thirds of e-learners in our study were able to correctly order all of the steps required for both EpiPen and Anapen administration 7 months after training. Lecture participants achieved results similar to those for e-learners, even though they had hands-on practice with devices during training. Although long-term device recall was poorer compared to anaphylaxis knowledge, other research has shown device recall may wane over time.^47,48 In a group of physician trainees, only one-third accurately demonstrated devices 6 months after training.⁴⁸ In our study, the complexities of the different devices, lack of regular experience with them, and the fact they were new to many pharmacists at the time of training may have impacted pharmacists’ long-term recall. As the participants were geographically diverse, we did not evaluate device demonstration as a skill. Thus, while knowledge of device administration steps improved at 7 months, application of this knowledge was not assessed.

Strengths and limitations

This study has a number of strengths. The training program and assessment test were developed using a rigorous approach and validated prior to use. We included 2 comparator groups in our study: traditional lecture training and no training. Further, we conducted 3 posttraining tests, with a follow-up period considerably longer than that of other e-learning effectiveness studies. Retention rates were high: almost all participants completed the posttest, and around 50% completed all 4 tests. This compares favorably with response rates to e-mailed surveys (where the average response rate is 33%).⁴⁹ The study had sufficient power to detect a mean score difference of at least 1 point within and between groups. Finally, there was no duplication in recruitment of pharmacists to intervention groups (pharmacists who participated in the e-learning group could not participate in the lecture group and vice versa).

However, we did not randomize participants to intervention or control groups, and as we adopted a convenience method of recruitment, the study may have been affected by selection bias. The lack of randomization would only affect between-group comparisons. Nevertheless, generalization of the e-learning results may be limited to people with a high comfort level with learning via the Internet and/or who have experience using multimedia online. Given that the study sample represented well-educated professionals who had daily exposure to Internet-related technologies, we expected knowledge and use of the Internet to be high in this population. Lecture participants also were required to show a high level of comfort with Internet use, as they were required to complete all follow-up tests online. Further, the vast literature evaluating e-learning programs, the increasing delivery of online education, a historical early acceptance of technology in the pharmacy profession (all suggesting pharmacists are confident Internet users), and the difficulties achieving a true random sample in online research may have combined to reduce the effect of selection bias in our study.^50-52 In addition, we evaluated ASCIA Anaphylaxis e-training for pharmacists in a context where learners now define their education strategies (eg, choosing rather than being recruited to undertake this program),⁵³ which may have provided real-world evidence for effectiveness.

The control group did not include pharmacists and began the study at a different time than the intervention groups. We chose to use students as controls because we could ensure that they did not receive inadvertent exposure to anaphylaxis training and thus contamination during follow up. Nonetheless, we acknowledge that control scores were significantly lower than intervention scores at pretest. This ultimately impacted pairwise comparisons and may have distorted the magnitude of the difference between training and no training. Moreover, the control scores did not change over time, despite participants completing the same test questions on 4 occasions. This may have been because of/the result of lack of interest in the topic, lack of perceived relevance to practice, fatigue from completing multiple tests, or a true effect.

We used the same 12 questions for each of the 4 tests. There was the potential for a learning effect from the test itself, although we did attempt to control for practice effect, and it was unlikely given there was no change in control scores. Although we did not adjust for multiple comparisons in the analyses, we do not consider this to be a limitation. The key effectiveness measure—long-term knowledge change—was assessed in 3 post hoc tests (e-learning, lecture training, or no training groups, comparing 7-month tests and pretests), and the magnitude of the change in knowledge at all tests was large. Therefore, with low numbers of multiple comparisons, an effect size of practical relevance, and very low p-values (p<0.001), there was no need for adjustment.⁵⁴

Finally, we acknowledge that this training may not have been wholly responsible for knowledge demonstrated at 7 months. There is the potential for academic dishonesty with tests completed remotely. However, participants were de-identified and study incentives were not dependent on scores, so we consider the motivation to deceive was low. Although exposure to alternate anaphylaxis information over time (eg, through general media or through self-study) may have confounded the results, knowledge gain across learner groups was consistent (with no gain in the control group) over 7 months.

Implications and recommendations

ASCIA Anaphylaxis e-training for pharmacists is part of a group of e-learning packages available to pharmacists and other health professionals, school and childcare workers, and the general community throughout Australia and New Zealand. Since 2011, more than 760 pharmacists, 4600 health professionals, 130 000 school and childcare workers, and 1100 members of the general public, have completed this training.⁵⁵ The key messages in each of these programs are equivalent, and the language used in each program is appropriate for the intended learner. Despite the success in implementation, ASCIA anaphylaxis e-training programs have not previously been evaluated for effectiveness. The study demonstrates that ASCIA Anaphylaxis e-training for pharmacists is effective at increasing and maintaining long-term anaphylaxis knowledge across a demographically and geographically diverse population of pharmacists.

Because accurate and current anaphylaxis knowledge is an essential part of anaphylaxis management, the question of when to retrain should be considered. As the majority of e-learners met the minimum standard for anaphylaxis knowledge 7 months after training, it is difficult to define a retraining interval based on declining knowledge. An additional follow-up evaluation of the same participants at 18-24 months may be a realistic timeframe. For pharmacy students in the era of the flipped classroom, the addition of this e-learning program would increase their anaphylaxis knowledge while allowing them to actively practice with adrenaline auto-injector devices. Investigating the effectiveness of the e-learning program in this context would be useful.

Pharmacists have been identified as an underutilized resource for providing anaphylaxis education and device training at the time of adrenaline auto-injector supply.^43,44 One-third of e-learners in the study did not correctly order the steps for EpiPen and Anapen device administration, and this may impact the quality of advice provided with these devices. Covert or overt simulation-based research is required to determine what happens at the time of adrenaline auto-injector distribution in pharmacies, as a measure of translation of anaphylaxis learning to practice. Research options include simulated patient methodology to assess device demonstration and anaphylaxis knowledge, or the use of overt simulation (for example, using mannequins) to investigate the pharmacist’s response to acute anaphylaxis.

SUMMARY

Regular education updates are required for pharmacists to maintain current knowledge about the prevention and treatment of anaphylaxis and how to supply and use adrenaline auto-injectors. ASCIA Anaphylaxis e-training for pharmacists increased anaphylaxis knowledge long-term. Knowledge gains were similar to ASCIA lecture training and superior to no training. This e-learning program offers a convenient, effective, no-cost option for pharmacists to improve and maintain their anaphylaxis knowledge. Future evaluations should seek to define an interval for retraining and investigate translation of anaphylaxis knowledge to practice.