Spaced learning versus traditional learning for pharmacists implementing β-lactam therapeutic drug monitoring

Abstract

Purpose:

No consistent recommendations exist on how best to educate pharmacists to facilitate implementation of new evidence-based practices. This study aimed to determine whether spaced learning was more effective than traditional education when learning a new evidencebased practice (β-lactam therapeutic drug monitoring, or TDM).

Summary:

β-lactam TDM was implemented in a phased fashion at Mayo Clinic in Rochester. Two go-lives took place, separated by 6 months, which allowed for a natural experiment to test education delivery methods (traditional education in the first go-live group of pharmacists vs spaced learning in the second go-live group of pharmacists). Each group received the same base educational package, with pharmacists in the spaced learning group also offered monthly in-person sessions for 6 months before go-live to discuss key considerations and cases with experts. There was no difference between the spaced learning group and the traditional education group when assessing appropriate use of TDM as defined by (1) appropriate patient selection; (2) correct ordering of laboratory testing; (3) appropriate clinical action; and (4) completed documentation. Assessments of secondary knowledge transfer also showed no difference between the groups. Pharmacists in the different education groups had similar satisfaction scores related to the effect and utility of the education. Both spaced learning and traditional education were part of a comprehensive implementation plan for β-lactam TDM, which might have influenced the outcomes. Spaced learning for future pharmacist education efforts needs further study.

Conclusion:

As part of a greater β-lactam TDM implementation effort, altering the education delivery method did not impact the degree of knowledge transfer.

β-lactam antibiotics are the backbone of treatment in acutely ill patients with infections, particularly in those with sepsis.^1,2 However, because of widespread pharmacokinetic variability in critically ill patients, the dose-response relationship for β-lactam antibiotics is unpredictable. Use of standard doses recommended in the package insert can result in highly variable drug concentrations.³ Therapeutic drug monitoring (TDM) has been suggested as the future of individualizing β-lactam therapy, but uptake remains limited. Multinational surveys have indicated that β-lactam TDM is used in only 5% to 20% of critical care practices.^4–7 A possible reason for incomplete or unsuccessful uptake of β-lactam TDM could be inadequate education of clinicians.^8–13 In our formative mixed-methods work, a multinational group of critical care and infectious disease health professionals broadly acknowledged the relevance of β-lactam TDM to their practice but cited a lack of familiarity with the supporting literature and a lack of comfort with utilizing results to guide treatment decisions as barriers to successful implementation.¹³ These results have been reproduced using qualitative methods, including insights from pharmacist stakeholders.⁸

No consistent recommendations exist on the best approach for inter-professional education about new evidence-based practices like β-lactam TDM. Clinicians have reported a need for more education beyond the current standard of seminars, patient cases, and open forums.^8,14 Research on learning and retention strategies in higher education has demonstrated that spaced learning (delivery of small amounts of information repeatedly over time) confers greater knowledge gain and retention than a single learning experience,^14–17 which aligns with theory on preventing cognitive overload.^15,16 These concepts have only rarely been evaluated in the healthcare setting.¹⁸ The purpose of this article is to describe the application of education structured in a spaced learning framework to support implementation of β-lactam TDM at a large academic medical center.

Methods

Setting and participants.

In June 2022, β-lactam TDM was implemented in the intensive care units (ICUs) at Mayo Clinic, an academic medical center located in Rochester, MN. Implementation of β-lactam TDM at the study center has previously been described in detail.¹⁹ Practice leaders were engaged to support the project, resources were identified, and local drug assays for cefepime, piperacillin, and meropenem were developed and validated.²⁰ Clinical workflows were created to specify the recommended population for TDM, desired pharmacokinetic/pharmacodynamic targets, and approach to blood specimen collection and interpretation of results. Finally, educational strategies and supportive materials were developed to provide substantive and varied information to diverse stakeholders involved with β-lactam TDM.

To constrain clinical and laboratory volumes and facilitate continuous workflow refinement, β-lactam TDM was deployed in a phased approach (phase 1, select adult ICUs; phase 2, all adult ICUs). This phased strategy separated implementation by discrete pharmacist workgroups, which cover specific ICUs without overlap (eg, the cardiothoracic surgery ICU pharmacist team was in phase 1 while the neurosciences ICU pharmacist team was in phase 2). Phase 1 pharmacists (those in the first wave of adult ICUs where β-lactam TDM was implemented) received traditional education approximately 1 month before the phase 1 go-live. Phase 2 pharmacists (ie, those in the remaining adult ICUs with delayed implementation and infectious diseases pharmacists) received spaced learning over 6 months before the phase 2 go-live.

Educational resource suite.

A suite of educational resources was created to support implementation of β-lactam TDM (Table 1). All tools, including an eLearning module (Appendix A), traditional lectures, a frequently asked questions (FAQ) document, an infographic (Figure 1), contact information for clinical champions, and clinical case review podcasts, were archived on an institutional web-based repository available to Mayo Clinic staff. Further specifics on the educational suite have previously been published.¹⁹

Table 1.

Educational Suite

Education item	Description
Online module	A 15- to 20-minute asynchronous eLearning module on Mayo’s Learning Management System that covered evidence for β-lactam TDM, operational considerations, and practice problems. Developed in partnership with the Learning Solutions Center at Mayo Clinic. The content was subdivided into 4 tracks according to clinical role: physicians/advanced practice providers, pharmacists, nurses, and laboratory personnel. The module was assigned as required education for pharmacists and was optional for all other stakeholders. To successfully complete the education module, pharmacists were required to obtain a passing score (80% or higher) on a knowledge check, which included application of learned principles to a patient case.
Traditional lectures	PowerPoint presentations lasting 45-60 minutes provided to multidisciplinary audiences explained the foundational importance of β-lactam TDM, including justifying the need for β-lactam TDM and key considerations for implementation. These lectures are archived in perpetuity on an internal web-based interface for review by any Mayo Clinic staff member. Objectives of the delivered lecture included to (1) describe the pharmacokinetic differences observed in critically ill patients; (2) justify the need for β-lactam optimization in the critically ill using preclinical and clinical data; (3) explain the preferred β-lactam TDM target; and (4) outline the key operational considerations with the Mayo Clinic β-lactam TDM program.
FAQ document	A FAQ document addressed common clinical and process questions in 3 topic areas: patient and drug selection, specimen collection, and assay interpretation. This document is revised as needed and serves as a just-in-time resource for pharmacists.
Infographics	Contained key points on operational and clinical considerations to be used as a quick reference when performing β-lactam TDM. These infographics are revised or new ones are created as needed, and they serve as a just-in-time resource for pharmacists.
Clinical champions	Pharmacists from each ICU and infectious diseases pharmacists received additional training to answer questions for colleagues during implementation, which were captured via a Qualtrics form. Clinical champions were available to answer questions from colleagues outside their respective ICUs and tracked questions they received and information provided on the Qualtrics form.
Podcasts	Four 15- to 25-minute podcasts were developed with external clinical experts. The first was a 25-minute pros/cons debate focused on “Is β-lactam TDM worth it?”. Three 15-minute real-world case presentations discussed challenging clinical scenarios (eg, difficult-to-penetrate locations, extremes of weight, empiric vs microbiologically confirmed sources of infection, interpreting high drug levels). Recordings were accompanied by show notes with timestamps.

Abbreviations: FAQ, frequently asked questions; ICU, intensive care unit; TDM, therapeutic drug monitoring.

Figure 1.

Example infographic provided to pharmacists.

Educational resource suite deployment.

A traditional presentation of education was developed to support pharmacists involved in the first phase of implementation. Pharmacists in this group were asked to complete an eLearning module approximately 1 month before go-live. Other resources, including traditional lectures, the FAQ document, and infographics, were made available for independent review. A pharmacist champion for each ICU was identified, who received 3 additional 30-minute training sessions conducted by project leaders and served as a resource for team members.

In the second phase, pharmacists received spaced learning over 6 months before the phase 2 go-live. Similar to in the first phase, pharmacists were requested to complete the eLearning module before go-live. Traditional lectures, the FAQ document, and infographics were available for independent review according to pharmacists’ preferences, and pharmacist champions were similarly available as a resource. In this second phase, pharmacists also received educational exposure once monthly during team meetings over the 6 months leading up to the phase 2 go-live. These education-based team meetings allowed for in-depth discussion of key facets of the β-lactam TDM program, including the rationale for the evidence-based practice, patient selection, and monitoring parameters. Patient cases curated from the initial implementation phase were selected by project staff members and presented with support from clinical practice experts with extensive experience in β-lactam TDM management (sample case in eAppendix). Learners were offered the cases in advance of the presentations to encourage independent review and allow practice with clinical support tools, specifically the institutional β-lactam TDM calculator. During the case reviews, the invited clinical practice expert reviewed considerations that would be necessary for appropriate management, providing learners with the opportunity to ask questions about clinical decision-making and justification. Case presentations were recorded and saved as podcasts to support independent review if desired. The spaced learning group in the second phase was also sent summaries of key concepts monthly via email before the phase 2 go-live. Figure 2 provides an overview of the education and assessments in the 2 phases.

Figure 2.

Implementation phases: patient populations, patient locations, and pharmacist learning group assignments. Phase 1 education (traditional education only) spanned the period from 1 month before to 1 month after go-live, whereas phase 2 education (spaced learning) spanned the period from 1 month before to 6 months after the phase 1 go-live. FAQ indicates frequently asked questions.

Evaluation of educational suite.

To assess for adequate transfer of knowledge after education, the appropriateness of use of β-lactam TDM was examined using electronic health record queries of real-world patient cases. The traditional learning group included 33 critical care pharmacists who completed 54 β-lactam TDM encounters for 41 unique patients in the 4 months after phase 1 go-live (June to September 2022). The spaced learning group included 33 critical care and infectious diseases pharmacists who completed 11 β-lactam TDM encounters for 9 unique patients in the 4 months after phase 2 go-live (November 2022 to February 2023).

The cases were reviewed independently and in duplicate by 2 study team members for the clinical and operational appropriateness of β-lactam TDM use. Cases were each evaluated according to 4 factors: (1) appropriate patient selection; (2) appropriate ordering of β-lactam levels; (3) appropriate clinical action (dose adjusted or not); and (4) appropriate documentation in an electronic health record progress note. The 65 TDM encounters (54 in the traditional education group and 11 in the spaced learning group) served as the basis for assessing appropriate use of β-lactam TDM using electronic health record queries. Interrater reliability ranged from substantial agreement to perfect agreement (Appendix B). Generally, pharmacists in both groups successfully selected appropriate patients, ordered β-lactam levels appropriately, and took appropriate clinical action (>80% for all domains in both groups) (Appendix C). The traditional education group more appropriately documented β-lactam TDM in progress notes than the spaced learning group.

As an additional proxy for knowledge, the number of queries received by pharmacist champions was recorded in a centralized database in the 4 months after each group’s go-live (phase 1, June to September 2022; phase 2, November 2022 to February 2023). Twenty-eight pharmacist champion queries were recorded for the traditional education group compared to 12 for the spaced learning group. Most queries were operational rather than clinical in both groups.

Pharmacist performance on the eLearning module assessment (number of attempts to complete the module, time to complete the module, and score on the final attempt) was reviewed to evaluate the education the groups received. In this analysis, 35 (53%) of the pharmacists to whom the module was assigned completed the eLearning module (traditional education group, n = 19; spaced learning group, n = 16). Pharmacists in the traditional education group averaged 1.4 attempts to complete the module (range, 1-3 attempts) with an average final score of 93% (by design, pharmacists were required to achieve a score of 80% or higher to pass). Pharmacists in the spaced learning group averaged 1.4 attempts to complete the module (range, 1-3 attempts) with an average final score of 94%. The time to complete the eLearning module was 34 minutes 8 seconds in the traditional education group compared to 39 minutes 4 seconds in the spaced learning group. Notably, whether pharmacists were required to complete the eLearning module (whether it was mandatory or optional) was subject to the discretion of their supervisor and was not determined by the implementation team.

Finally, web-based surveys were used to assess self-reported knowledge and confidence with β-lactam TDM. Baseline surveys were sent 1 month before phase 1 go-live (June 2022). A follow-up survey was sent 1 month after each group’s go-live date (phase 1: baseline, June 2022; follow-up, July 2022; phase 2: baseline, June 2022; follow-up, December 2022). Surveys were distributed to pharmacists through email with electronic reminders sent on days 8 and 10 and the survey closed on day 11 (Qualtrics, Provo, UT). Individuals consented to the use of their data for research by clicking on the survey link. The response rate for the baseline survey was 83% (n = 55/66). No differences were observed at baseline between the groups in self-reported knowledge and confidence. At follow-up, 86% of pharmacists in the traditional education group and 100% of pharmacists in the spaced learning group agreed that the β-lactam TDM education provided was sufficient for the pharmacists to implement β-lactam TDM in clinical practice. Eight-six percent of pharmacists in the traditional education group and 93% of pharmacists in the spaced learning group agreed that they would recommend the β-lactam TDM education to others.

All analyses were performed using SPSS software (IBM SPSS Statistics for Windows, Version 29.0.2.0; IBM Corporation, Armonk, NY).

Discussion

We describe approaches for both traditional and spaced learning education when providing education to pharmacists for β-lactam TDM implementation. No substantial differences in appropriate use of β-lactam TDM were noted between the 2 methods of education. Similarly, the groups did not differ in the number and type of questions raised about the evidence-based practice, eLearning module results, or self-reported knowledge and confidence.

Using the framework of mapping,²² the first step in implementation of any evidence-based practice is to identify barriers. Clinicians have identified insufficient education as a key barrier to implementation of β-lactam TDM.^8–13 To date, educational delivery techniques for implementation of β-lactam TDM have not been described.

Our innovative approach to education included spaced learning, a technique that has had relatively limited application in healthcare practice. The concept of spaced learning comes from traditional classroom learning and is often linked to memorization.²³ While memorization sits lowest on Bloom’s taxonomy for knowledge, it serves a purpose in building a foundation to reach higher skills such as application, analysis, and synthesis of information, which was our ultimate educational objective.²⁴ Spaced learning has been described in healthcare in the context of medical residencies, using some form of flashcards followed by a test.^25–28 To our knowledge, this is the first study to use spaced learning in a real-world setting (outside of formal training), analyzing knowledge transfer rather than test results.

In addition to education, behavior change principles influence implementation. In one study, Nilsen and colleagues²⁹ mapped change in clinician behavior onto Coetsee’s response to change framework.³⁰ Most clinicians in the study responded to change with “indifference” or “passive resistance.” Factors that may preclude a higher level of clinician response (eg, “commitment” or “involvement”) include a lack of ownership or support in the change process. In the β-lactam TDM implementation effort, we leveraged peer “champions” in both groups. Leadership and involvement of peers likely contributed to a favorable response to change, as evidenced by use of β-lactam TDM, engagement in educational sessions, and willingness to ask questions of champions.

Time management and time spent on tasks other than patient care may influence patient outcomes and satisfaction.^31,32 Education and competencies are an additional task with which pharmacists (and other healthcare workers) are faced, which could impede patient care.^31,32 Thus, when education is provided, it must be valuable and worth the pharmacist’s efforts.^33,34 On the basis of our survey assessing satisfaction and confidence, pharmacists receiving spaced learning would recommend this style of education, which suggests to the authors that the additional time spent on β-lactam TDM was deemed valuable. Additionally, given the relatively low completion rate for the eLearning module despite high satisfaction, strong consideration should be given to making key education mandatory when feasible.

Key limitations included discrepancies in the use of β-lactam TDM between the groups. Pharmacists in the traditional education group had a higher volume of β-lactam TDM encounters in the first 4 months after go-live. This is likely a function of early enthusiasm about a new tool in practice and the patient populations in the first set of ICUs where TDM was used. A high proportion of patients in these ICUs experience profound organ dysfunction and need treatment with extracorporeal support devices such as extracorporeal membrane oxygenation and continuous kidney replacement therapy. These individuals may experience more pharmacokinetic variability,³⁵ which may motivate greater use of β-lactam TDM relative to other patient populations. We observed that patients in the spaced learning group had less electronic health record documentation of β-lactam TDM findings. This result may be due to the small sample size or the frequency with which the antibiotics were de-escalated in the spaced learning group, resulting in pharmacists not documenting β-lactam TDM, for reasons that are unclear. Studying spaced learning in healthcare during an active implementation can also influence study findings. We employed a Plan-Do-Study-Act (PDSA) cycle framework to continually improve upon β-lactam TDM as it was being used.³⁶ PDSA not only resulted in improvements in the workflow but also enhanced educational materials to include key information learned during implementation. Providing this updated education was necessary to support a successful implementation but may have impacted metrics. Despite these limitations, the relatively limited literature on the educational delivery techniques applied in healthcare makes these data innovative and informative.

Conclusion

Our examination of the application of different educational strategies to implementation of innovate workflows highlights the importance of a carefully crafted implementation plan for new evidence-based practices. Further study is needed to evaluate whether spaced learning provides a relative advantage compared to traditional education in the healthcare setting.

Supplementary Material

Appendix A

Content pathways were created to support role-specific education of the learner to recognize their responsibilities in the process along with the responsibilities of other stakeholders.

Supplementary material is available with the full text of this article at AJHP online.

KEY POINTS.

• Ensuring appropriate integration and implementation of a newly created clinical practice, such as β-lactam therapeutic drug monitoring, requires a robust education plan.

• In the implementation of this education plan, pharmacists’ retention and application of knowledge pertaining to the new practice was not significantly improved with a spaced learning strategy.

• The optimal educational method for training pharmacists on new evidence-based practices needs further study.

Data availability

The data underlying this article are available in the article and its supplementary material.