Effectiveness and Dissemination of the Interprofessional Pediatric Pain PRN Curriculum

Abstract

Introduction:

Cultural transformation is needed for healthcare to be sensitive to children’s pain. The Pediatric Pain PRN Curriculum is the first free, open access, downloadable, interprofessional pain continuing education (IPPCE) curriculum developed to empower healthcare professionals to translate evidence-based pain care into clinical practice. To evaluate the Curriculum and its experiential flipped-learning strategies, we used a mixed methods approach.

Methods:

Interprofessional pediatric teams from 8 healthcare systems evaluated the Curriculum after attending Pediatric Pain PRN Courses in the Fall of 2017 (N=90). Qualitative methods were used to evaluate the acceptability, feasibility, and utility of the Curriculum and IPPCE strategies. Pain knowledge and attitudes were measured pre-/post-IPPCE with the Pediatric Healthcare Provider’s Knowledge & Attitude Survey Regarding Pain (PHPKAS). Web analytics were used to describe dissemination for the first year after the Curriculum’s webpage launch.

Results:

Learners rated their achievement of course objectives as moderate-great. PHPKAS scores significantly increased (P<.0005); but significant differences in physician and nurse PHPKAS question responses pre-/post-IPPCE were found. Within 2 years of the IPPCE, 3 healthcare systems were certified as Childkind Hospitals, 5 Pediatric Pain PRN Programs were established, and various practice changes and improved patient outcomes, such as decreased hospital lengths of stay and emergency department returns for pain, were realized. Curriculum dissemination was global.

Discussions:

Results support the acceptability, effectiveness, feasibility, utility, and global dissemination of the curriculum and IPPCE strategies. More rigorous patient outcome data is needed; however, this study demonstrated the benefit of a free, open-access, downloadable, interprofessional healthcare provider continuing education curriculum.

Pain varies based on individual experiences and personal expression, and pain management is culturally complex.^1–3 Pain care is fragmented, inconsistent, incomplete, and undermined by disparities in access and quality.⁴ Despite advances in our understanding, assessment, and treatment of pain, pain care does not meet societal needs or expectations for wholistic, patient-centered, evidence-based, interprofessional care.^4,5 In 2011, the Institute of Medicine (IOM) confessed that “pain receives insufficient attention in virtually all phases” of healthcare professional education;^4pg191 and concluded that the “enormous burden of pain will require a cultural transformation in the way pain is understood, assessed and treated.”^4pg1

Interprofessional pain education (IPPE) is critical to translate effective pain care knowledge and attitudes into healthcare systems.^6–9 Competencies for collaborative practice are best learned in an interactive IPPE format that enables learners to learn about, from, and with each other.^10–13 The Federal Drug Administration, National Institutes of Health (NIH), and International Association for the Study of Pain, outlined two goals for IPPE: (1) increase understanding of pain mechanisms and related biopsychosocial concepts, and (2) improve communication and collaboration among healthcare professionals.^{4,7,12,14–21} In response, the National Pain Strategy included an IPPE requirement.⁸ The NIH established Centers of Excellence in Pain Education to develop accessible, case-based curricular materials for undergraduate IPPE integration. However, a cultural transformation also requires post-graduate IPPE.²²

We sought to innovate IPPE for pediatric healthcare professionals (PHPs) through the development of the Pediatric Pain PRN Curriculum. To best address the biopsychosocial needs of patients with pain and alleviate suffering, interprofessional collaboration and evidence-based practice must prioritize patients’ experiences and preferences.^1,23,24 More than 20 years of research indicates knowledge and attitudes regarding pain improve with education; however, knowledge improvements are not translated into clinical practice.^3,25–27 PHPs need knowledge, sensitive attitudes, empathy, and clinical leadership skills to provide evidence-based pain care.^{1,4,6,23,24,28} They also need tools, to improve pain management by balancing the advantages of standardized, system-based approaches with sensitivity to individual patient care experiences and treatment choices.^{1,2,23,24,28–30}

The Pediatric Pain PRN Curriculum was designed for interprofessional pain continuing education (IPPCE) and uses a live, flipped-learning model, with an interactive group format to ensure a ‘with, from and about’ learning experience.^{13,20,22,31–36} The Curriculum aligns with recommendations from the IOM, to provide evidence-based guidelines for dismantling barriers to adequate pain care, individualizing pain management strategies, and providing PHPs with better education for understanding pain.⁴ The goal of the Curriculum is to empower PHPs to transform organizational culture by promoting interprofessional collaboration and evidence-based practice. The Curriculum prepares PHPs as front-line leaders to promote change and champion a healthcare culture that is sensitive to children’s pain throughout and beyond organizational boundaries.^30,37,38

The Curriculum can be viewed at luriechildrens.org/PRNpain. The Curriculum was evaluated to: (1) assess acceptability and IPPCE strategies, (2) determine the effectiveness of the content based on change in pain knowledge and attitudes, (3) describe the feasibility and utility of the IPPCE strategies, and (4) illustrate the dissemination of the Curriculum. We begin this report by explaining the guiding principles of the Pediatric Pain PRN Curriculum; and describing key Curriculum features so that others might replicate our work in other subjects for interprofessional continuing education.

Guiding principles

The purpose of the Pediatric Pain PRN Curriculum is to promote a decentralized, cost-effective, evidence-based, and unified approach to implementing high-quality pediatric pain management.

The Pediatric Pain PRN Program acknowledges that direct care providers are in the best position to transform healthcare systems and ensure patients receive the highest quality pain care. Teams working side-by-side in the daily care of patients can change the way pain is managed.

Evidence-based practice (EBP) is defined as the integration of the best available evidence, patient values, and PHPs’ expertise for clinical decision-making in patient care.³⁹ Implementation of EBP has been criticized for allowing research-based knowledge to dominate over patient and clinician experiential know-how.²³ Mechanisms of pain, standardized tests, and objective outcomes are often prioritized over patient experiences, but this research dominant EBP approach conflicts with the definition of pain as a subjective sensory and emotional experience.^23,40

Patient-centered care requires empathetic listening and good communication skills.¹ Care beyond cure is consistent with the moral and ethical obligations taught in undergraduate healthcare professionals’ curriculums.²³ Yet, the subjective nature of pain has marginalized patients with pain as being of little educational value.¹ The absence of care beyond cure experiences in healthcare professionals’ education has promoted a ‘hidden curriculum’ that brands patients with pain as difficult and drug-seeking.¹ Consequently, new PHPs enter a practice culture that lacks sensitivity to pain. Disparities in pain care are reinforced through social learning.^1,23 Thus, IPPCE must deliver more than pain care knowledge. To bridge the gap from knowledge attainment to clinical translation and cultural transformation, PHPs need training in effective communication skills, teaching strategies, leadership competencies, and methods to influence change.

Flipped learning is a pedagogical approach that maximizes individual learning and transforms classes into facilitated dynamic interactions of subject matter application. ^{22,31,33–36} Prior knowledge is coupled with experiential activities, such as case studies, simulations, and group problem solving, to create real-world experiences that facilitate knowledge translation.^34,41,42 Flipped learning accelerates evidence-to-practice translation by increasing learners’ competence and confidence in their knowledge, skills, and ability to influence others.

Pediatric Pain PRN Curriculum Key Features

The Pediatric Pain PRN Curriculum is the first free, open-access, downloadable curriculum designed to engage direct care PHPs to learn and work together to improve pediatric pain care. Healthcare systems have varied needs, schedules, and priorities; therefore, the Curriculum was designed for flexible IPPCE delivery as a 1–2-day educational event or multiple educational sessions held over time, using a live, flipped-learning model with an interactive group format. All curricular materials, including sample applications, schedules, and continuing education materials, are available at luriechildrens.org/PRNpain.

Core Content focuses on essential knowledge of pediatric pain care. Given the lack of uniform undergraduate education and different regional practices, Core Content provides well-referenced foundational knowledge to facilitate shared understanding of evidence-based pediatric pain care. Core Content includes pain assessment and management for acute, chronic, and procedural pain.

PRN Role Implementation content focuses on the PRN Role as a leader, change agent, project manager, educator, and advocate. Content is designed to teach and promote essential knowledge translation skills. Content features leadership development activities and methods to develop, promote, implement, and evaluate pain management improvement efforts.

Specialty Subjects build on Core Content but focus on pain experienced in special pediatric populations. This material adds age (neonatal), type (critical care and palliative care), and disease-specific (cancer, sickle cell disease, abdominal pain, and headache) pain assessment and management content. We recommend use of Specialty Subjects to supplement education at recurring Pediatric Pain PRN meetings.

Standardized course materials include: Participant Guides (pdf), Class Presentations (ppt/pptx), and scripted Facilitator Guides (pdf). Participant Guides provide pre-class reading of key concepts and foundational knowledge. When provided 1–2 weeks prior to classes, these guides prepare learners to engage in active learning.⁴² Learners are responsible for pre-reading so that classroom time focuses on discussion, role-play, or other guided learning activities. Participant Guides also serve as learners’ handouts and references.

Class Presentations are formatted as PowerPoint® slide sets for projection during live classes. Slides are designed to highlight key points from the Participant Guides for interactive activities.⁴² Activities and slide content emphasize the most common challenges faced when translating evidence-based pain knowledge to clinical practice. Some slide modifications may be required by CE providers (e.g., conflict of interest disclosures) or to highlight specific organizational resources (e.g., massage therapy services). Other content, including content that conflicts with current organizational policies, should not be modified. Instead, these differences should prompt discussion of organizational policies, practices, and resources.

Facilitator Guides are class scripts for facilitating in-person flipped learning strategies.⁴² Standardized language is used for quick recognition of prompts, (e.g., READ or ASK). Lecture is limited to reading and reinforcing critical material. Facilitators must be self-aware, respect diversity, be able to analyze group dynamics and their effect on learning and manage power and hierarchies to optimize IPPE.²¹ While Facilitators may have valuable pediatric pain management expertise, the Curriculum is designed for interactive learning and group discovery (Table 1). Facilitators with limited knowledge of pain science and pediatric pain management can successfully facilitate the Pediatric Pain PRN Curriculum. Facilitators are strongly discouraged from going off-script or providing additional content for course delivery other than institutional specifics (e.g., how to access a pain consult service).

Table 1.

Interprofessional Pain Education Flipped Learning Strategies

Strategy	Description	Classes
Case studies	• Presents real-life or hypothetical cases for group analysis, prompting participants to describe the cases, compare them to prior experiences, explore and debate different assessment or treatment options, and discover that there is more than one “right way” to manage pain. • Use of cases from applications is encouraged but standardized cases are also provided.	• Assessment, • Analgesics, • Acute Pain, • Chronic Pain, • Sickle Cell Disease (SCD) Pain, • Neonatal Pain, • Critical Care, • Pediatric Cancer, • Chronic GI Pain, • Headaches, • Palliative Care
Common Ground	• Provide current pain care policies and orientation materials to understand what new staff are taught • Encourage learners to modify content & teaching methods	• Role: Educator
Course Critique	• Evaluate course teaching strategies • Debate flipped-learning activities used during this course and their effectiveness for adult learning	• Role: Educator
Differing Definitions (Words Matter)	• Compare and contrast terms • Language is important and medical terms can often have common and more exact definitions	• Role: Advocate • Opioid REMS • Palliative Care
Evidence Appraisal	• Evaluate scientific literature for best evidence and clinical risk-benefit analysis • Explore organizational analgesic controversies	• Analgesics
Fast FIX The course is designed for quick clinical translation	• Occurs in situ • In 30 minutes, assemble team, go to clinical area, teach something learned in course, and bring back evidence of learning • Critique education and response, then modify plan	• Role: Educator
Goal Setting	• Explore why learners are taking Course and set SMART (Specific, Measurable, Achievable, Relevant, Time-bound) goals to improve pain care • Guide post-Course week1, month1, & year1 work • Regularly encourage learners to reassess progress	• Role: Welcome • Role: Data to Inform • Neonatal Pain
Game of Phones	• Mobile service and phone required • Race to find area services for patient referrals	• Opioid REMS • SCD Pain
Policy R&R	• Review and recommend policy changes based on Pediatric Pain PRN Curriculum knowledge gained	• Pain Assessment
Raise the Bar	• Explore acute pain expectations • Debate risks and benefit of pain and pain treatments as barriers to complete pain relief	• Acute Pain
Ready-Set-Go	• Recall of analgesic mechanism race	• Analgesics
Measure Twice, Change Once	• Create, revise, and finalize pain improvement project plan with SMART goal, team members, leaders, nay-sayers, process, & outcome measures	• Role: Data to Inform • Role: Advocate
Pain Rounds	• Learners accompany pain care specialists on patient care rounds to demonstrate more than one ‘right way’ to treat pain • Patient-centered rounding enhances a team-based collaborative model, interprofessional relations, and facilitates pain care improvement discussions	• All Role classes
Prescribed or Diverted?	• Use urine drug test results to reinforce drug metabolism pathways and opioid monitoring • Interpret results and address testing limitations	• Opioid REMS
Procedural Plans	• Collaborate as care teams to develop a plan for a procedure commonly done in clinical area • Explore patients’ developmental needs, interventions, and roles for optimizing comfort	• Procedural Pain
See 1, Do 1, Teach 1	• Hands on demonstration, return demonstration, and simulated teaching of biobehavioral strategies	• Biobehavioral Strategies
Symptom Science	• Use symptom patterns to differentiate types of headaches	•  Headaches
Roles and Responsibilities	• Role play and simulation • Identify team members, leaders, and nay-sayers for a quick-win PRN project; brand your project	• Role: Change • Role: Advocate • Procedural Pain
Practical Peer Advice	• Evidence-based guidance does not exist for all clinical situations Encourage candid discussion of practical advice • Develops camaraderie & shares experiences	• Analgesics
Self-Assessment	• Explore personal beliefs and biases • Develop plan to influence pain care: (1) What is known, (2) What team needs, (3) Next steps	• Assessment • Opioid REMS • Pain with SCD
Self-Test	• Online Pediatric Healthcare Provider Knowledge and Attitude Survey regarding Pain (PHPKAS) • Pre-test, post-test, and self-assessment method	• Analgesics • Acute Pain • Neonatal Pain
Videos	• Videos entertain while reinforcing learning • Egg Headache • Power of Breastfeeding	• Headaches • Neonatal Pain

Method

After obtaining Institutional Review Board approvals (#2016–481) the acceptability, feasibility, utility, and effectiveness of the Curriculum was evaluated by PHPs attending Pediatric Pain PRN Train-the-Trainer Courses held at Ann & Robert H. Lurie Children’s Hospital of Chicago in August and September of 2017 (N=90). The Curriculum was tested with interprofessional teams from 8 different healthcare systems and nurses from 10 states. Nurses who were accepted into the Course were asked to bring a physician and one other type of PHP from their clinical team to the Course. Child-life specialists (n=6), pharmacists (n=2), physical therapists (n=2), social workers (n=2), and a psychologist attended the Course as members of these clinical teams.

Evaluation strategies

IOM recommends that interprofessional education be evaluated at four levels: reaction, learning, behavior change, and outcomes.⁴³ We elicited learners’ feedback immediately after each Pediatric Pain PRN Course to assess the acceptability of the overall Curriculum and IPPE strategies. Standardized continuing education (CE) evaluation questions were used to assess learners’ IPPCE experience and satisfaction (reaction).³³

To determine content effectiveness, pre-/post-IPPCE change in pain knowledge and attitudes were measured using the Pediatric Healthcare Provider’s Knowledge & Attitude Survey Regarding Pain (PHPKAS ©Renee Manworren, 2014, 2019).^31,44,45 The Pediatric Pain PRN Curriculum provides free access to the PHPKAS via REDCap™.⁴⁵

We mined the Pediatric Pain PRN Train-the-Trainer Course applications to identify barriers to effective interprofessional collaboration for pain care.²⁷ Feasibility and utility of the IPPCE strategies (Table 1) used to address these barriers are described. We did not use standardized measures of interprofessional education perceptions,^46,47 or interprofessional core competencies.^10,48–50 Instead, we used qualitative methods to evaluate impact and process through separate focus groups with facilitators and learners.^50,51

Patient outcomes and evidence of practice changes provide the highest level of learning evaluation.⁴³ For 2 years after the IPPE, we evaluated changes in clinical practice and patient outcomes that learners had identified in their applications; these included, for example, changes in hospital readmission rates for poorly controlled pain or implementation of new pain treatment strategies.^27,50

Finally, we used Webanalytics, to illustrate online interaction and dissemination with the Pediatric Pain PRN Curriculum. CrazyEgg was used to collect and compare data during 2 intervals: 44 consecutive days after webpage launch in June-July 2019 and 57 consecutive days in April-May 2020. Google Analytics collected data for the first year of the Curriculum’s webpage existence.

Instruments

The Pediatric Healthcare Provider’s Knowledge & Attitude Survey Regarding Pain (PHPKAS) is a 41-item survey that takes <20 minutes to complete and has been validated for use with physicians, pharmacists, pediatric nurses, advanced practice registered nurses (APRN) who can and cannot prescribe, and child-life specialists.^44,45 Branching logic is used to customize questions by role and scope of practice. For example, the same analgesic question would read “you prescribed” for a prescribing physician or APRN but would read “… was prescribed” for a nurse or pharmacist. Item difficulty index ranged from 21–96%; and Cronbach’s alpha was 0.80. Construct validity was established by comparing responses before and after education, and by comparing responses from PHPs and students P<.0005. Test-retest reliability was established (r=.89) by repeat testing of PHPs 20–30 days apart before an IPPCE program. Since the first version of this survey was released in 2001, it has been used in hundreds of organizations in 8 English-speaking countries and translated into 13 languages for global use.

CrazyEgg provides a visual heat map of online activity metrics, including visits, impressions, clicks (indicated by blue dots), and popularity of specific areas of the website (represented by a color gradient, with red indicting higher and blue indicating lower popularity; Figure 1). Visits refers to a user accessing the webpage. Impressions are where users stop scrolling on a particular section of the webpage, presumably to read or download webpage material. Visits provide an overall view of the interaction, whereas clicks and impressions provide more specific details of locations users engaged on the webpage. Popularity is the combination of these metrics to show which aspects of the webpage had the most interaction and views. CrazyEgg categorizes data by desktop and mobile users, allowing for comparison of these two ways of accessing the webpage.

Figure 1.

CrazyEgg Heat Map Visual of user interaction with Pediatric Pain PRN Curriculum webpage.

Popularity is indicated by the color gradient on the left, where red is higher popularity and blue is lower popularity. The dots represent clicks on the different content section download buttons.

Google Analytics provided metrics, including total users, webpage visits, traffic sources, users’ locations, average session duration, and percentage of users that visited by desktop or mobile device. One type of traffic source is organic, which means users found the site from a search engine. Session duration is the total time a user spent on the webpage. Unlike CrazyEgg, Google Analytics combines desktop and mobile users’ data.

Results

The majority of those who attended the Pediatric Pain PRN Train-the-Trainer Courses were Caucasian (n=63, 70%), female (n=81, 90%) with a mean age of 40 years. Six learners were >60 and four were <25 years of age. Their work environments included pediatric hematology/oncology, medical/surgical inpatient units, emergency departments, outpatient rehabilitation, ambulatory care clinics, and intensive care units.

Acceptability and satisfaction

A clear majority of learners rated their achievement of course objectives as moderate to great (Table 2). Themes emerged from qualitative analysis of learner comments describing their IPPCE experiences. Quotes from the comments section of the CE evaluation were selected to illustrate themes of IPPCE experiences.

Table 2.

Pediatric Pain PRN Program Continuing Education Evaluation

Objective	Moderate to Great Extent
I have gained new knowledge of strategies for pediatric pain assessment and management	90%
I can demonstrate a newly learned pain and PRN role skill to better manage pediatric pain.	98%
I have gained leadership skills that will enable me to promote change, communicate, and educate patients, parents, families, other healthcare professionals, and the public to improve pediatric pain care.	87%
I have drafted an interprofessional pain management project proposal to implement in my clinical practice environment within a month and a year of attending this Pediatric Pain PRN Course	74%
Quotes	IPPE Themes
• “Exercise of reviewing the evidence was good.” • “Content was well-referenced. I learned from others how to implement ‘one voice’ and ‘comfort plans’ into our workflow.”	Evidence-based Guidelines
• “Loved attending pain rounds. Was really eye opening. Helped learning come to life, made making a change in my hospital feasible.” • “This conference gave us the tools to actually implement change. It doesn’t just stop with us learning here for 2 days.” • “I like that it was interactive, and we got to talk to other people from different institutions about things they did for pain.” • “Creating a change hospital-wide is hard, but my goal is to change one thing, and if I can, I call it a great start.”	Breakdown Institutional Barriers to Pain Care
• “I now have more biobehavioral interventions I can use with patients and teach my coworkers.” • “Will consider what’s the most appropriate assessment for this patient? Rather than just age and numbers scale.”	Individualize Pain Management Strategies
• “Loved the ability to openly give thoughts and hear others’ perspectives.” • “So nice to feel supported amongst the team as a whole.” • “We will implement interprofessional pain rounds.”	Promote Interprofessional Collaboration
• “Very helpful to learn about how to use data, language, and other techniques to get buy-in.” • “I networked with a lot of great people I can use as resources.”	Leadership Development

Change in pain knowledge and attitudes

There was a statistically significant difference in pre-IPPCE PHPKAS scores (Mean 73.7% [11.8]) compared to post-IPPCE PHPKAS scores (Mean 82.2% [8.9], P<.0005; Mean increase 8.4%; 95% CI: 5.7–11.2). Of paired respondents, 62% (n=30) were nurses, 21% (n=10) physicians, and 17% (n=8) were other healthcare professionals. There was no significant difference in mean pre- or post-IPPCE PHPKAS scores by profession.

Using Chi-square test for independence (with Yates continuity correction), there was a significant association between profession and correct answer on one pre-IPPCE and two post-IPPCE questions (Table 3). Knowledge of time to peak analgesia, recommended analgesic route, and duration of analgesia improved; however, in post-IPPCE >50% of learners did not indicate that sedation always precedes opioid-related respiratory depression. In addition, 50% of pre-IPPCE and 36% of post-IPPCE respondents incorrectly indicated 20–70% of pediatric patients over-report pain, even though >90% of learners indicated the child/adolescent was the best judge of their pain and children >8 years of age could reliability self-report pain intensity.

Table 3.

Significant Differences in Physician and Nurse Knowledge and Attitudes

Question	Answer	Professiona	P-value
Pre-Test: The usual time to peak effect for traditional analgesics (acetaminophen, NSAIDs, and opioids) given orally is __________.	60-minutes	Nurses	.042
Post-Test: Anxiolytics, sedatives, and barbiturates are appropriate medications for the relief of pain during painful procedures.	FALSE	Physicians	.034
Post-Test: Andrew is 15 years old and this is his first day following abdominal surgery. As you enter his room, he smiles at you and continues talking and joking with his visitor. Your assessment reveals the following information: BP=120/80; HR=80; R=18; and on a scale of 0 to10 (0 = no pain/discomfort, 10 = worst pain/discomfort), he rates his pain as 8. On the patient’s record you must mark his pain on the scale below. Choose the number that represents your assessment of Andrew’s pain.	8	NURSES All provided the correct answer 50% of the physicians choose a lower score	.001

^aProfession associated with providing the correct answer.

Feasibility and utility of IPPE strategies

To facilitate learning, 22 different IPPE strategies were used across the 20 classes in the Pediatric Pain PRN Curriculum (Table 1). These strategies were designed to promote interprofessional analysis, synthesis, problem solving, and reflection.^{21,34,35,41,42,52,53} Setting and sharing SMART (Specific, Measurable, Achievable, Relevant, Time-bound) goals⁴⁸ was the first flipped-learning strategy used in the Course to explore learners’ motivations and prioirities.^42,52,53 This strategy was repeated in three classes to assist learners in modifying their goals to reflect newly attained knowledge and skills and to guide their work the first week, month, and year after the Course.

Case studies, which assess higher-level learning,^21,53 were the most common IPPCE strategy (11 classes). Case studies were taken from learners’ Course applications. Learners were prompted to compare these cases to their clinical experiences, explore and debate different options, and discover more than one ‘right way’ to manage pain. Learners recommended use of cases from applications in future courses, but standardized cases are also provided in the Curriculum.

“Pair and share” dialogues were the second most used IPPCE strategy (7 classes). Learners paired up and shared their reflections of sensitive subjects such as disparity in pain care, personal attitudes regarding pain and pain treatments, prescribing power, healthcare hierarchies, and how to address care conflicts. Using probing and open-ended questions, learners’ shared reflections and used critical thinking to develop interprofessional strategies to promote collaboration despite power asymmetry and conflict.^20,21,41,42 While this strategy was evaluated as successful by learners; the persistent response by over a third of learners that 20–70% of children overreport their pain suggests the need for strategies that include patient and families’ perspectives.

“See 1, Do 1, Teach 1” provided an opportunity for quick clinical translation. This strategy allowed learners to try, practice, demonstrate, and teach use of virtual reality goggles, yoga, and eight other biobehavioral interventions with evidence of effectiveness for reducing pain and pain associated anxiety and suffering. Learners were highly satisfied with this class. Learners who gave lower ratings explained it was because they were only allowed to attend 4 of the 10 concurrent 20–30-minute sessions (2 hours total with breaks).

These 22 IPPCE flipped-learning strategies successfully inspired more than 50 pediatric pain care project proposals. Less than 2 years after the course, a valid and reliable pain assessment tool was tested and implemented in a NICU. Topical anesthetics were introduced for needle procedures in several inpatient and ambulatory care areas. Clinical care guidelines were developed with interprofessional teams, which lowered opioid use, hospital lengths of stay, and returns to the emergency department due to uncontrolled pain. Biobehavioral strategies were promoted with “comfort kits” and “comfort & coping” care plans.⁵⁴ Of the 8 different healthcare systems participating in the Pediatric Pain PRN Train-the-Trainer Courses, 3 systems are now certified as Childkind (childkindinternational.org), certifying these institutions for excellence in pediatric pain care; and 5 pediatric PRN programs have been started in 3 states.

Curriculum dissemination

From webpage launch on June 1, 2019 to May 27, 2020, 380 different users visited the webpage 452 times. In the first 44 days, the webpage was visited 88 times. Eleven months later, the webpage received 33 visits over 57 days. The average time a user spent on the site was 2 minutes and 9 seconds. Of the 412 page views that came from users with known locations, 41% came from Chicago and surrounding suburbs, 42% from other United States and Canadian locations, and 18% from other countries (Table 4). The majority (66.8%) of pageviews came about organically, and from a desktop device (67.9%). There were 191 total clicks on content download buttons for June-July 2019 and 54 for April-May 2020. In both time intervals, the highest interaction was at the top, and a smaller number of users scrolled to the bottom to view or download those content sections.

Table 4.

User Locations and Number of Desktop User Impressions on Webpage by Time Interval

Content of Webpage	Number of Impressions
	June-July 2019	April-May 2020
Title and Introduction	116	36
Pre-Test/Post-Test	107	36
The Curriculum Facilitator Guide	88	24
Implementing PRN Program Participant Guide	76	20
Welcome: Download the Course Slides	72	20
Welcome: Download the Facilitator Script	68	20
Change and Overcoming Barriers: Download the Course Slides	61	17
Change and Overcoming Barriers: Download the Facilitator Script	61	16
Data to Inform Change: Download the Course Slides	50	12
Data to Inform Change: Download the Facilitator Script	42	10
Education Strategies: Download the Course Slides	36	13
Education Strategies: Download the Facilitator Script	36	11
Communicate, Advocate and Lead: Download the Course Slides	37	8
Communicate, Advocate and Lead: Download the Facilitator Script	37	7
Pain Theories: Download All Materials	26	4
Assessment: Download All Materials	23	2
Analgesics: Download All Materials	25	6
Opioid REMS: Download All Materials	31	4
Biobehavioral Strategies: Download All Materials	32	4
Procedural Pain: Download All Materials	29	11
Acute Pain: Download All Materials	28	12
Chronic Pain: Download All Materials	26	10
Pain from Sickle Cell Disease: Download All Materials	22	6
Pain in Neonates: Download All Materials	21	7
Pain in Critical Care: Download All Materials	15	4
Cancer Pain: Download All Materials	20	2
Abdominal Pain: Download All Materials	20	4
Headache: Download All Materials	22	5

User Locations
Chicago-area Locations	Chicago, Barrington, Bartlett, Crown Point IN, Evanston, Gurnee, New Lenox, Tinley Park, Burr Ridge
United States and Canada	Baton Rouge, Chamblee, Dallas, Iowa City, Kirkland, Lexington, Miami, Montevideo, Nashville, Parkland, Saint Louis, Wauwatosa, Seattle
International Locations	Angola, Mexico, Uruguay, Norway, India

Sources of Traffic
Google	56.5%	Yahoo	3.8%
Direct Link	28.2%	Bing	3.8%
Portal2	5.7%	Other	3%

Discussion

Results support the acceptability, satisfaction, effectiveness, feasibility, utility, and dissemination of the Pediatric Pain PRN Curriculum and its IPPCE strategies. Knowledge and pain-sensitive attitudes were attained. Because the PHPKAS was validated across healthcare professions,⁴⁵ we know these were shared knowledge and attitudes.

Shared pain management knowledge and pain sensitive attitudes have received little research attention; but are critical for implementing effective pain care plans.^{1,23,24,29,55} Like previous studies, we found some differences in knowledge and attitudes related to pain by profession before and after the Pediatric Pain PRN Course.^{2,3,26,29,30,37} Unfortunately, difficulty believing patients and lack of understanding of progression of opioid-related respiratory depression remained prevalent after our IPPCE Courses.

Gains in interprofessional collaboration are just as important as shared knowledge and attitudes. Ideally, IPPCE takes place in situ and includes four features: experiential learning, timely learner feedback, activities based on evidence-based principles of teaching and learning, and diversity of educational methods.²¹ Using prior knowledge coupled with facilitated activities, like Fast FIX and Procedural Plans, the Pediatric Pain PRN Curriculum created real-world experiences of learning ‘with, from and about’ PHPs to rapidly transition knowledge into clinical practice.^{13,22,31–36,}

Learners’ comments provided evidence of the effectiveness of the Curriculum and IPPCE strategies for advancing sensitive attitudes of children’s pain and interprofessional collaboration. Post-CE survey comments include language of ownership, such as “my hospital,” “my goals,” and “my coworkers;” and collaboration, such as “learned from others,” “with us learning,” and “the team as a whole.”

Experiential learning through hands on-demonstration, return demonstration, and simulated teaching of biobehavioral pain interventions like massage and guided imagery, increased knowledge and skill acquisition by involving cognitive, affective, and psychomotor learning domains.⁵⁶ In several classes, learners taught newly acquired skills to help develop learners’ competence and self-confidence for rapid translation into patient care.⁵⁷ Learning also took place in situ, so that learners immediately took on the Pain PRN role in an interactive manner with clinical staff. This helped learners begin to dismantle barriers, problem solve, and encourage their clinical teams to identify pain management as a priority.

The purpose of evaluating the Course is to continuously improve participants’ satisfaction and evaluate the effectiveness of the Curriculum and its IPPCE strategies.⁵⁷ The highest level of learning evaluation is provided by evidence of practice changes and positive patient outcomes.^12,50,53 While we were able to provide evidence of pain care project proposals, practice changes, and organizational certification for pain care, our ability to report improved patient outcomes was limited. Previous studies have shown knowledge to clinical translation gaps by using labor-intensive patient satisfaction surveys, chart audits, and observations.^2,3 Although data acquisition, analysis, and advocacy skills were taught, our learners explained that they needed more ongoing support to obtain and report specific patient outcomes given their direct patient care roles and focus.

Curriculum dissemination

Traffic to the Curriculum’s dissemination webpage remained stable for the first year after initial webpage launch. Given that the Courses were taught in Chicago, Illinois, it is logical that a large percentage of users came from Chicago and surrounding areas. Diverse user location data may reflect our marketing and distribution of 500 flyers at the International Symposium for Pediatric Pain (ISPP), in Basel, Switzerland, June 16–19, 2019, and the American Society for Pain Management Nurses (ASPMN) meeting in Portland, Oregon, September 18–21, 2019. However, traffic source data indicates the majority of pageviews came organically from search engines with only 133 page visits from directly entering the Curriculum URL printed on marketing fliers. The stability of traffic over time, despite few marketing efforts, mirrors other studies in which open access through organic search engines was the most prevalent source of webpage traffic.^58,59

Online curriculum access allows for education dissemination that benefits healthcare globally.⁵⁸ The short, 2-minute average webpage session demonstrates the ease and efficiency of the streamlined download features. Other studies report similar average session durations.^59,60

Curriculum Limitations

There are several limitations to the Pediatric Pain PRN Curriculum. The Curriculum has been tested with PHPs who provide care to pediatric patients. It requires clinical knowledge and experience. While a few students did participate in the Train-the-Trainer Courses, the Curriculum was not designed for undergraduate healthcare students. Moreover, the Curriculum was not designed for pediatric pain management specialists.

The Pediatric Pain PRN Curriculum is accessed from a webpage, but it is not an online course. It was developed and tested for face-to-face classroom delivery. This design led to class cancellations during the COVID-19 pandemic. In addition, learners from the Train-the-Trainer courses expressed concerns that web-based video chats would not provide the intensity of interactions they found effective in promoting interprofessional collaboration.

A final important limitation is that the Curriculum introduces and supports Pediatric Pain PRN Programs. Materials to propose and develop PRN Programs are included in the Curriculum appendix. One or two days of IPPCE will not transform a healthcare culture to be sensitive to children’s pain, but with interprofessional support and organizational commitment, Pediatric Pain PRN Programs have been shown to improve patient outcomes.³⁸

Webpage analytics limitations

There are several limitations for interpretation of the web analytic data. First, various factors could have influenced session duration, such as a user leaving a browser open⁵⁹ or the inability of the analytics software to determine time spent on other linked pages.⁶⁰ Other studies measure bounce rate, which refers to webpage views and exits without linked page interactions.⁶¹ Average bounce rates range from 48% to 56%.^59,60 Instead of bounce rate, we compared the number of impressions on various webpage locations.

A second analytic limitation is the inability to report when specific users accessed the webpage. Whether international users accessed the webpage immediately after the ISPP meeting cannot be determined.⁶¹ Third, the Curriculum dissemination plan is dependent on the internet, and therefore limits users to those with internet access, roughly about 50% of the world.^59,62 On the other hand, the internet provided global access to the Curriculum. Links to the Curriculum from the ISPP, ASPMN, or Childkind International websites may provide more rigorous methods to track Curriculum dissemination.

Finally, while number of views and clicks to download the Curriculum were measured, whether users who viewed the webpage are PHPs or educators who could use this Curriculum is unknown. The PHPKAS provides a measure of Curriculum knowledge, and this was a high click area; but there were only 13 PHPKAS data entries from locations “other” than those represented at the Pediatric Pain PRN Train-the-Trainer Courses from webpage launch until May 27, 2020. Of course, the PHPKAS could have been downloaded and printed, which would prevent data reporting.

Conclusion

The Pediatric Pain PRN Curriculum is designed for quick knowledge to clinical translation using evidence-based content and experiential flipped-learning IPPCE strategies. Our evaluation provides evidence of the acceptability, satisfaction, effectiveness, feasibility, utility, and dissemination of the Curriculum. More rigorous patient outcome data is needed to ensure the highest level of IPPCE evaluation. The global online reach of this Curriculum demonstrates the benefit of a free, open access, downloadable, IPPCE curriculum. Using Google and CrazyEgg analytic software, we were able to determine that this Curriculum spread beyond its target audience and continues to be viewed and downloaded. Additional studies are needed that evaluate the use and effectiveness of the Curriculum outside the oversight of Curriculum developers, as well as the Curriculum’s long-term influence on interprofessional collaboration and patient outcomes.

Lessons for Practice:

• The Pediatric Pain PRN Curriculum is a free, open access, downloadable, interprofessional pain continuing education curriculum that increases local pain care knowledge and pain sensitive attitudes through global dissemination.

• Cases studies drawn from course applications and flipped-learning strategies facilitate shared knowledge, ownership, understanding, and interprofessional collaboration.

• Return demonstrations, teach back, and creation of shared SMART goals and performance improvement projects prompt rapid knowledge translation and improve clinical practice and patient outcomes.