Evaluation of an Interprofessional Training Program to Improve Cancer Drug Therapy Safety

PURPOSE:

Drug therapy for cancer is a high-risk, high-volume clinical intervention that requires interprofessional teams. Given the complexity of anticancer drug therapy and safety concerns, an interdisciplinary team developed a novel training program for oncology registered nurses and pharmacists to improve cancer drug safety.

METHODS:

Participants completed preworkshop learning assessments and received access to web-based modules on six topics: hazardous drug handling, drug extravasation, hypersensitivity reaction management, sepsis recognition, immune checkpoint inhibitor toxicities, and oral oncolytic adherence. In a 7-hour workshop, participants applied module content in interactive exercises and high-fidelity simulations. Preworkshop and postworkshop questionnaires assessed changes in knowledge and confidence in each topic. Program satisfaction and changes to clinical practice or policies were assessed 3 months after the workshop.

RESULTS:

Two hundred ninety-two nurses and 82 pharmacists applied to participate, and 103 (35%) and 44 (54%) have participated, respectively. Long-term follow-up data were available on 133 (90%) participants. Change scores in confidence to meet program objectives increased between pre- and postworkshop (range of increase 0.6-0.8, P < .01). Knowledge scores increased significantly between pre- and postworkshop (average improvement of 3.2 points, P < .01). Overall program satisfaction was high (mean 5.0, standard deviation [0.2] on a five-point scale). Seventy-seven (60%) reported that they had made at least one clinical practice or institutional policy change at 3 months.

CONCLUSION:

An interprofessional education program with online modules, in-person interactive sessions, and simulation activities is a promising strategy to deliver cancer drug safety content to practicing oncology clinicians.

INTRODUCTION

Cancer drug therapy is a high-volume, high-risk clinical intervention that requires interprofessional clinical teams. An estimated 23 million adult visits occur annually for cancer drug therapy in the United States; 19 million (84%) are in ambulatory settings.¹ Drug therapies now span traditional chemotherapeutic agents, biologics, and immunotherapies. To address concerns over the variability in cancer drug therapy delivery, the Oncology Nursing Society and ASCO jointly issued safety standards.^2,3 The adoption rate of these standards is unknown, and only 305 practices (of thousands) participate in a voluntary, fee-based certification process by ASCO's Quality Oncology Practice Initiative.⁴ The Hematology/Oncology Pharmacy Association has published standards for hematology/oncology pharmacy practice,⁵ and the American Society of Health-System Pharmacist has also published safety guidelines for handling hazardous drugs.⁶ Previous work has shown that knowledge and adherence to safety standards is suboptimal.^7,8 In particular, gaps exist in adherence to evidence-based practices for safe handling of hazardous drugs, management of oncology emergencies, and extravasations, supporting patients who are prescribed oral oncolytics and implementing clinical practice change in oncology settings.

Historically, healthcare systems devote substantial financial resources to orienting new employees to their organizations. Far fewer resources support professional development of experienced healthcare professionals. However, overall median institutional spending on all healthcare employee training has decreased by 50% between 2015 and 2018.⁹ Furthermore, orientation programs and educational programs are often discipline-specific, which limits the opportunity for collective learning across multiprofessional teams. To improve the safety of care delivery and outcomes of patient care, it is essential to embrace interprofessional clinical training.

In this context, the purpose of the Multi-Professional Oncology Safety and Simulation Training program was to develop and deliver an evidence-based educational program for oncology nurses and pharmacists to deliver safe and effective care for patients receiving cancer drug therapies. Intended program outcomes included participant satisfaction, increased knowledge and confidence to manage drug therapy care, and intention-to-change clinical practice in their workplace.

METHODS

Program Recruitment, Selection, and Initiation

The project team developed a multimodal strategy to recruit participants. Social media activities included Facebook and Twitter postings across the university's School of Nursing, College of Pharmacy, and Cancer Center's accounts, as well as affiliated organizations. Using purchased mailing lists from professional nursing and pharmacy organizations, high-quality color postcards were mailed to oncology nurses (n = 6,608) and pharmacists (n = 2,222). Statewide mailing lists were used in waves, first targeting contiguous states to Michigan and then expanding in subsequent campaigns to include states with large numbers of oncology professionals. State and regional medical oncology and oncology nursing chapter leaders received an e-mail and poster to distribute to their members. Two health professional organizations also delivered e-mail blasts to their members. All marketing materials directed potential applicants to a dedicated program website, where applicants completed a 21-item survey to provide background information about themselves, their practice setting, and their interest in the program.

Selection criteria included practicing registered nurse or pharmacist in the United States, proportion of time spent in direct care of patients with cancer, and proportion of time spent in managing aspects of care specific to cancer drug therapy. To assure that content was delivered to clinicians practicing in underserved areas, emphasis was placed on applicants who practiced in a historically distressed area, as measured by the area deprivation index.¹⁰ To promote team-based learning, participants were encouraged to sign up as teams of nurses and pharmacists who worked together in the same facilities and additional preference was given to applications from pharmacist-nurse dyads (or larger teams). The program manager notified participants of their acceptance and placed individuals not selected on a waiting list for future programs.

Once accepted, participants received access to the Canvas Learning Management System (Instructure, Salt Lake City, UT), which housed all assessment activities, learning materials, and archives. Program staff held optional web-based sessions to troubleshoot the learning management system and answer questions before the workshop. Reminder e-mails about workshop logistics and outstanding preworkshop learning modules were sent 4 weeks before the workshop and again 7 days before the workshop. The program director's institutional review board determined that the project was exempt from human subjects review.

Program Development

Our program's purpose was to deliver a high-quality training program for practicing oncology nurses and pharmacists to improve their ability to deliver cancer drug therapy safely. Thus, we designed a program with multiple educational approaches. First, we surveyed learners to assess baseline knowledge and current practice. These survey data were used to tailor onsite workshop content. Second, to engage learners and to generate a shared understanding of key concepts, we provided video presentations and selected seminal readings to review before workshop attendance. Third, we structured a 7-hour in-person workshop to enable learners to apply principles from the prework to their practice setting. Workshop activities included team-based exercises, facilitated discussion, and high-fidelity simulation activities. After the in-person workshop, learners received access to supplemental digital resources, including a module on stimulating clinical practice change, archived lectures, handouts, and access to a discussion board to ask questions of faculty and workshop attendees.

To encourage workshop attendance and survey completion, participants received a $200 gift card upon arrival to the workshop, an additional $50 after their day-of-workshop evaluation was completed, and an additional $50 when the three-month follow-up survey was completed. National Cancer Institute–funded educational programs provide similar incentives to participants to offset costs associated with travel, time away from family, and work rearrangements. To minimize the potential for social desirability bias, evaluation data were separated from requisite information needed to process incentives. Participants received seven interprofessional continuing education contact hours from the National Center for Interprofessional Practice and Education for participation.

Prelearning Modules

Before the workshop, participants completed prelearning activities using the learning management system, and these included self-assessment of knowledge of the ASCO/Oncology Nursing Society safety standards; hazardous drug handling; extravasation prevention and management; supporting patients on oral oncolytic therapy; and recognition and management of urgent oncology situations (sepsis, hypersensitivity, and immunotherapy toxicities). After participants completed these assessments, the learning management system provided access to brief learning modules on these topics. After each module, participants received prompts to bring copies of their relevant policies and procedures to the in-person workshop for discussion with the faculty and other learners.

Didactic Workshop Content

During the in-person workshop, interprofessional faculty, composed of doctorally prepared nurses and pharmacists, led participants through applied case-based discussions on the content areas. The workshop was held in a newly constructed educational facility with interactive technology–rich classroom space with active learning pods and a 13,000-square-foot state-of-the-art simulation center.

Faculty used findings from the preworkshop assessments to tailor their presentations to address frequently identified knowledge or practice gaps. Faculty led brief presentations to reinforce key content, discuss clinical practice controversies, and provide students with resources for additional learning. Interactive discussion and tabletop exercises facilitated group learning and active knowledge acquisition. To facilitate networking among students and faculty, faculty led office hours during break times and lunch to answer participant questions or critique existing policies and procedures.

Simulations

After the morning session, participants engaged in three different simulation experiences—each lasting 45 minutes—followed by structured debriefing conversations. Faculty with subject matter expertise developed different simulation experiences to provide participants with a range of options based on their interest areas. These simulations were adherence to oral oncolytics, management of a chemotherapy spill and coincident vesicant extravasation, management of a hypersensitivity reaction, differentiation between sepsis and immune checkpoint inhibitor toxicity, and appropriate procedures for donning and doffing personal protective equipment for handling hazardous drugs. Faculty drafted each simulation to enable nurses and pharmacists to engage in their respective clinical roles.

Subject matter faculty partnered with a certified simulation educator to design the simulation(s) associated with their content area. Program staff led draft simulation exercises to identify needed modifications. Final simulation guides included room setup instructions, instructor notes, time sequences, role cards, and structured debriefing questions. Before each simulation session, faculty reviewed the simulation exercises and huddled with the simulation team to make final adjustments. Simulation exercises were tape-recorded to inform the program's quality improvement efforts.

The oral oncolytic simulation used a standardized patient. The hazardous drug handling simulation included video review of correct technique, structured feedback, and group discussion. The remaining simulations were situated in a chemotherapy infusion setting with simulation mannequins and participants playing roles of clinicians or family members.

Each scenario started with a prebriefing to orient the participants to the simulation objective and answer any questions they might have. As most participants had little, if any simulation experience, cue cards and other cognitive aids were used to assist learning. Each simulation was run by a faculty member with content expertise and simulation technician if a simulator was used.

After each simulation, a 20-minute structured debriefing session occurred. We used a plus/delta debriefing approach with advocacy and inquiry style questions to ensure that learners could close their personal learning gaps.¹¹ Facilitators met with their groups after each simulation in a classroom to debrief. The debriefing started with overall discussion of how participants felt about the scenario. Then, participants moved to a discussion on “what went well” and “what would they do differently” in the same situation. The use of advocacy and inquiry questions helped get at any underlying behaviors. This style of questioning allows the facilitator to make an observation of something that occurred during the simulation and ask the participant to provide input to what they were thinking at that time. This has been shown to be an effective method of simulation debriefing. During the debriefing sessions, participants were encouraged to discuss their clinical policies and experiences to identify areas for future improvement. An interactive lunch session provided participants an opportunity to consider principles congruent with the responsible conduct of research, as applied to clinical oncology settings.

Evaluation Methods

We applied Kirkpatrick's four-level framework¹²—satisfaction, learning, behavior change, and outcomes—to inform our evaluation plan. Survey measures were adapted from an existing National Cancer Institute–sponsored educational program led by one of the co-authors (G.H.M.).

Satisfaction was measured using a five-point Likert scale, extremely satisfied, somewhat satisfied, neither, somewhat dissatisfied, and extremely dissatisfied. Participants were asked about their overall program satisfaction and their satisfaction with each of the four key content areas. Usefulness of the content and applicability to their clinical area were also evaluated using a five-point scale of strongly agree to strongly disagree in each content area.

Learning was evaluated through two measures. The first measure was a knowledge test assessed at the time of workshop enrollment, which was repeated 3 months after workshop completion. We examined the number and percentage of questions answered correctly. The second learning measure was a survey of self-confidence measured before and immediately after the workshop. The self-confidence survey consisted of seven statements related to workshop content and skills. The five-point scale ranged from not at all confident to completely confident.

Behavior change was assessed 3 months after workshop completion with a structured questionnaire. Participants reported whether they had reviewed their institutional policies and whether they had completed practice changes in specific content areas.

Data analysis was composed of descriptive statistics of each outcome measure. To assess changes in learning, we calculated paired t-tests to compare preworkshop knowledge and confidence scores with postworkshop scores, with a significance level set at P < .05. We also calculated within-subjects Cohen's d effect sizes for these two outcomes.

In addition to surveys, the last 30 minutes of the workshop enabled participants and faculty to debrief on lessons learned and insights from the day and suggest improvements to the training program.

RESULTS

Of 292 nurse and 82 pharmacist applicants, 103 and 44 attended the first three workshops and 189 and 38 were placed on waiting lists. We examined the degree to which the program reached participants of diverse backgrounds. From the first three workshops, 43 distinct teams of nurses and pharmacists participated from 58 different institutions (Table 1). To date, fourteen percent of participants are of color. A substantial proportion of participants practice in either a rural (14.6%) or a socially disadvantaged (17.1%) area. Participants have substantial oncology clinical experience (mean [standard deviation]: 12.4 [9.9] years and range across all participants: less than 1 year to 38 years). Participants live in 13 states.

TABLE 1.

Workshop Participant Characteristics

Satisfaction was assessed for the overall program and by each specific content area and evaluated on a scale from 1-5, with 5 being highly satisfied. The results are shown for each profession and for the entire cohort (Table 2, n = 147). Participants were highly satisfied with all content areas with extravasation and urgent oncology situations among the highest. Both nurses and pharmacists were equally satisfied across specific content areas and with the overall program. One participant reported that they were somewhat dissatisfied with the program; all others reported that they were extremely (n = 135) or somewhat (n = 11) satisfied.

TABLE 2.

Program Satisfaction at Workshop Conclusion

Learning Objective Assessment

We assessed learner-rated self-confidence in delivering cancer drug therapy safely at two time points: at the time of program enrollment and again immediately after the workshop (Fig 1). Both pharmacists and nurses showed significant improvement between the time of enrollment and after completing the workshop, with an average rating improvement of 0.6-0.8, on a five-point Likert scale, all P < .01. Cohen's calculated effect size for the entire cohort was 0.78, a large effect size. Significant improvements in knowledge scores were observed in pharmacists (average improvement of 2.8 points), nurses (average improvement of 3.4 points), and the entire cohort (average improvement of 3.2 points; all P < .01; Cohen's calculated effect size for the entire cohort was 1.07, reflecting a change in knowledge scores that exceeded one standard deviation, Fig 2). Knowledge scores before or after the workshop were not significantly associated with years of clinical experience (results not shown).

FIG 1.

Self-reported confidence in delivering cancer drug therapy safely (N = 147). Confidence to deliver cancer drug therapies safely was evaluated on a five-point Likert scale, with 1 = not at all confident to 5 = completely confident. A paired t-test was used to evaluate changes at the time of workshop enrollment and immediately following the in-person workshop.

FIG 2.

Comparison of knowledge scores before and 3 months after the workshop (N = 133). Nurse and pharmacist participants completed a 25-item knowledge assessment at the time of workshop enrollment and 3 months after workshop completion. Average knowledge scores are shown for nurses, pharmacists, and all participants before and after the workshop.

Practice Change

Data on practice change experiences were available for 129 participants (88% of all participants) who completed the three-month assessment (Table 3). Of these participants, 112 (87%) reviewed at least one institutional policy and 77 (60%) reported making at least one practice change. The most frequent policy changes involved hazardous drug handling (48%), followed by extravasation (26%), urgent oncology situations (18%), and oral oncolytic management (14%). A notable number (range 39%-67%) involved a group process, as opposed to individual policy review or practice change.

TABLE 3.

Reported Changes in Clinical Policies After 3 Months (n = 129)^a

Verbal workshop debriefings and open-text comments identified that some content was more pertinent to certain roles. After our first workshop debriefing, we modified simulation activities. Specifically, we provided details as to specific nursing and pharmacy roles in each simulation, surveyed participants for their simulation activity preferences, and facilitated wayfinding within the simulation center. The simulation educator also oriented participants to what they could expect to see and experience in the simulation activities.

DISCUSSION

Cancer drug therapy is among the most clinically complex treatments to deliver. Increasingly, a larger number of drugs—often in novel combinations—are delivered to a growing number of patients. Increasingly rapid approvals of anticancer drugs force clinicians to quickly assimilate new (and often incomplete) clinical data into routine care delivery.¹³ Many patients are of advanced age or have multiple co-incident illnesses. Relentless cost pressures on clinical facilities and concerns for pervasive workforce shortages exacerbate a challenging clinical environment to deliver these high-risk treatments and support continuing education.¹⁴

Our program was intended to address the knowledge gap of existing oncology nurses and pharmacists to deliver safer cancer care. Informed by our previous work,¹⁵ we addressed topics of high concern and notable variation in quality. Using a multidisciplinary team and in partnership with simulation experts, we designed a state-of-the-science educational offering with the goal of delivering a satisfying, knowledge-generating, and practice-changing activity. To date, data suggest that participants are highly satisfied and report increased knowledge and confidence of managing complex clinical scenarios. Most participants have applied the knowledge learned in the workshop to enact practice change in their workplaces. Despite impressive rates of learners making policy or practice changes in their clinical environment at 3 months, our data do not allow us to understand whether the changes span multiple clinicians or whether the changes reported are sustained over time. Interprofessional training programs may facilitate implementation of evidence-based oncology practice.¹⁶ Program participation and interest remain high.

We acknowledge the extensive marketing efforts that are required to reach potential participants and that enrolled participants are likely more engaged and receptive to educational programs, which may skew evaluation data. Furthermore, although incentives are often provided as part of National Cancer Institute–supported education programs, sustainability of this funding and incentive is uncertain after the funding period. Diverse strategies are needed to reach and engage potential participants.

Despite these strengths, the evaluation data highlight opportunities for improvement. This includes careful attention to recruiting and admitting participants from diverse backgrounds. Although the United States clinical oncology workforce does not resemble the diversity observed in the US population,¹⁷ our demographic data suggest additional improvement opportunities. In subsequent work, we are partnering with healthcare professional organizations that focus on specific races and ethnicities. We are targeting marketing materials to specific geographic regions to increase the reach of our program, and we have noticed more attendees of color over time. A second limitation is the focus on nurses and pharmacists. Other clinicians, including physicians, nurse practitioners, physician assistants, and others, care for patients receiving chemotherapy. We plan to pilot one workshop during the remaining project period to include additional professions. A third limitation is the reach of our program to those who can travel onsite. Formal assessment of the evaluation tool for reliability and validity would strengthen the approach. We are exploring ways to make our instructional materials and simulations available to remote attendees. Remote attendance for professional education has surged during the 2020 severe acute respiratory syndrome coronavirus 2 pandemic, and our team has responded by shifting content to a virtual environment. Our faculty reviews the content biannually to assure that the latest data are integrated into the program.

In conclusion, the Multi-professional Oncology Safety and Simulation Training program is a promising model to deliver educational content to practicing oncology nurses and pharmacists. The program fills an important gap in clinical education by bringing professions together and using simulation activities to practice skills in high-risk situations. Program participants receive instruction, experience, and resources to implement clinical practice change and ultimately improve care delivery to their patients.

AUTHOR CONTRIBUTIONS

Conception and design: Michelle D. Aebersold, Shawna Kraft, MiKaela Olsen, Martha Polovich, Brenda K. Shelton, Guy H. Montgomery, Christopher R. Friese

Financial support: Christopher R. Friese

Administrative support: MiKaela Olsen, Christopher R. Friese

Provision of study materials or patients: Karen B. Farris

Collection and assembly of data: Michelle D. Aebersold, Marylee Scherdt, Martha Polovich, Guy H. Montgomery, Christopher R. Friese

Data analysis and interpretation: Michelle D. Aebersold, Karen B. Farris, Guy H. Montgomery, Christopher R. Friese

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Evaluation of an Interprofessional Training Program to Improve Cancer Drug Therapy Safety

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/op/authors/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).