Digital Delivery of a Global Critical Care Education Program

Abstract

Background

CERTAIN (Checklist for Early Recognition and Treatment of Acute Illness and iNjury) education program was developed to accelerate the global dissemination of a standardized, systemic, structured approach to critical care delivery. The coronavirus disease (COVID-19) pandemic prompted the evolution of this program from a live in-person course to a blended synchronous and asynchronous learning experience, including virtual simulation.

Objectives

We describe our experience and insights gained through this digital program transformation and highlight areas in need of further research to advance the delivery of high-quality online education offerings to global interprofessional audiences.

Methods

The CERTAIN education program was delivered to a broad international audience first in person (2016–2019) and then virtually during the COVID-19 global pandemic (2020–present). During this transition, we adopted a flipped classroom model to deliver the core content asynchronously using an online learning management system, supplemented by a novel synchronous online experience to provide learners with the opportunity to apply these concepts using a series of simulated clinical cases.

Results

A total of 400 participants attended 11 CERTAIN courses. We transitioned our 10-hour live course to a 3-hour virtual workshop. The duration of simulation activities (admission, rounding, and shared decision-making) remained constant. Didactic lectures were eliminated from the synchronous online course and presented as recorded videos in precourse materials. We collected 306 postcourse surveys (response rate, 76.5%). The majority of the overall course ratings were excellent (147 [49.5%]) and very good (97 [32.7%]), and learner responses were similar to live and online courses. Simulation activities were consistently the most popular elements of our program. Access to digital learning platforms and language barriers during simulation activities proved to be the greatest challenges during our transition. Delivering mobile-friendly online content and close coordination between dedicated bilingual faculty and local champions helped overcome these challenges.

Conclusion

Critical care education and case-based simulation workshops can be delivered to international interprofessional audiences with similar, high degrees of learner satisfaction to in-person offerings.

Critical care is a new discipline that receives a lower priority in many low- and middle-income settings because of barriers to healthcare access and limitations in human, technology, and infrastructure resources (1, 2). Although early recognition, resuscitation, and timely intervention in acute illness have been shown to improve patient outcomes, available data suggests healthcare disparities may be related to the limited availability of trained staff to recognize common critical care syndromes and deliver appropriate treatment (3, 4). The World Health Organization and international subspecialty societies have advocated for increased global educational efforts using simulation, telemedicine, and internet-based courses to address this problem (5, 6).

CERTAIN (Checklist for Early Recognition and Treatment of Acute Illness and iNjury) was designed and developed to standardize the approach to the evaluation and treatment of acute illness and injury. The CERTAIN approach improved critical care processes and patient outcomes in a large trial in 34 intensive care units in 15 countries with variable resources (7, 8). We developed a CERTAIN education program to speed further global dissemination and implementation of these concepts.

In this report, we describe our transition during the coronavirus disease (COVID-19) pandemic from a live learning experience to a blended asynchronous and synchronous virtual educational program, the lessons learned during this experience, and opportunities for future development and innovation.

Methods

Course Design

Using the ADDIE (Analysis, Design, Development, Implementation, and Evaluation) instructional design framework (9, 10), we developed our CERTAIN education program to disseminate the CERTAIN approach used during our prior quality improvement international study. The CERTAIN program is primarily focused on teaching a standardized and structured approach to initial assessment (including point-of-care ultrasound), ongoing management, and patient-centered decision-making in acutely ill patients. We included didactic presentations and a series of simulation-based activities (i.e., admission, rounding, shared decision-making, and case-based discussions) with blended faculty panels that included recognized local experts in our curriculum design (Figure 1). This project was approved by the Mayo Clinic Institutional Review Board (12-007998).

Figure 1.

Development of CERTAIN online course modules. CERTAIN = Checklist for Early Recognition and Treatment of Acute Illness and iNjury.

Course Implementation and Transition to Online Format

Our 2-day CERTAIN live course was initially offered annually in Rochester, MN, between 2016 and 2019. On the basis of this successful pilot experience, we delivered six 1-day courses in five countries in 2019. Our faculty were selected from participating CERTAIN study sites who had implemented this approach. The course curriculum included the following: 1) the CERTAIN approach to reduce diagnostic error and therapeutic harm, including communication skills and an approach to patient engagement and shared decision-making; and 2) tools and technology to facilitate practice change, including core principles of simulation-based training, clinical informatics and research, patient safety, quality improvement, and change management (11, 12).

The COVID-19 pandemic prompted the cancellation of all planned in-person CERTAIN courses in 2020. On the basis of our previous online training experience, the CERTAIN team pivoted to develop an online, interactive virtual course experience to better reach learners across the globe (13–15). Using a flipped classroom model, we produced 6 hours of on-demand online CERTAIN course modules for learners to complete asynchronously via Blackboard Learning Management System (Blackboard LMS). This was followed by a 3-hour novel, interactive virtual simulation experience on Zoom (Zoom Video Communication) in which learners practiced the core CERTAIN skills of admission, rounding, and shared decision-making during case-based scenarios presented using Articulate (Articulate) to reinforce core online content and practice teamwork and decision-making skills. We created a technology playbook with instructions for setting up Zoom, adjusting video and audio quality, and lighting in advance. We also used a Viber (Rakuten, Inc.) group chat for asynchronous communication, questions, and longitudinal reinforcement of course content. We found the variety of online features (video meetings, share screen, breakout rooms, online polls, and reactions using emojis) offered by these platforms, combined with the incorporation of bilingual faculty, helped significantly with learner engagement.

Course Evaluation

Course material and surveys were delivered in both English and translated into the local language to facilitate learner engagement. At the end of each course, all participants received an anonymous online survey that collected demographic characteristics and feedback on course content and faculty using a five-point Likert scale. The survey was distributed to participants via a web link or QR code for easy access using personal computers or mobile devices. Descriptive statistics were used to summarize participant characteristics and course evaluations using JMP 14 (SAS Institute Inc.) software.

Results

A total of 400 interprofessional participants attended the CERTAIN courses, including 350 U.S. and international clinicians who attended 10 CERTAIN live courses (4 in Rochester, MN, and 6 in international cities in Asia and Europe) over 4 years. All international participants were from the region surrounding each course site (Guangzhou, China; Beijing, China; Hanoi, Vietnam; Ljubljana, Slovenia; Zagreb, Croatia; and New Delhi, India). Fifty participants from Rijeka, Croatia, attended our 2020 online course. There were 306 postcourse surveys collected (76.5% response rate) from various locations. Most of the participants and survey respondents were physicians and nurses (Table 1).

Table 1.

Demographic characteristics of course participants and survey response rates

Course Location		Total Participants (n)	Response Rate [n (%)]	Number of Responses by Profession [n (%)]
Country	City			Physician	Nurse	Health Educator	Medical Student	Other
China	Guangzhou	74	68 (91.9)	62 (91.9)	5 (7.4)	1 (1.5)	0 (0)	0 (0)
China	Beijing	40	32 (80)	27 (84.4)	2 (6.3)	2 (6.3)	0 (0)	1 (3.1)
Croatia	Zagreb	41	41 (100)	22 (53.7)	10 (24.4)	0 (0)	8 (19.5)	1 (2.4)
India	New Delhi	40	35 (87.5)	34 (97.1)	0 (0)	1 (2.9)	0 (0)	0 (0)
Slovenia	Ljubljana	61	47 (77)	47 (100)	0 (0)	0 (0)	0 (0)	0 (0)
United States	Rochester	53	17 (32.1)	16 (94.1)	0 (0)	0 (0)	0 (0)	1 (5.9)
Vietnam	Hanoi	41	41 (100)	22 (53.7)	10 (24.4)	0 (0)	8 (19.5)	1 (2.4)
Croatia*	Rijeka	50	25 (50)	50 (100)	0 (0)	0 (0)	0 (0)	0 (0)
Total		400	306 (76.5)	255 (83.3)	27 (8.8)	4 (1.3)	16 (5.2)	4 (1.3)

^*Virtual online course.

Overall course ratings were either excellent (live 50% vs. online 44%) or very good (live 30.9% vs. online 53%) for both live and online courses (Table 2). Our simulation sessions and case-based panel discussions were particularly well received. The CERTAIN program and both live and virtual delivery methods were consistently well received regardless of the role or nationality of our participants (Tables E1 and E2 in the data supplement). Two hundred and eighty-five participants (93.1%) responded that they would recommend this activity to others, and all survey participants agreed that the content matched their current or potential scope of practice.

Table 2.

Overall course rating and learner satisfaction with simulation

	Response [n (%)]
	Poor	Fair	Good	Very Good	Excellent
Rate the course overall
Live course (n = 272)	0 (0)	12 (4.4)	40 (14.7)	84 (30.9)	136 (50)
Online course (n = 25)	0 (0)	0 (0)	1 (4)	12 (53)	11 (44)
Learner satisfaction with simulation
Rounding simulation practical value
Live course	1 (0.6)	5 (2.8)	23 (12.7)	34 (18.8)	118 (65.2)
Online course	0 (0)	0 (0)	1 (4)	8 (32)	16 (64)
Total	1 (0.5)	5 (2.4)	24 (11.7)	42 (20.4)	134 (65)
Admission simulation practical value
Live course	0 (0)	0 (0)	9 (5.8)	29 (18.8)	116 (75.3)
Online course	0 (0)	0 (0)	0 (0)	7 (28)	18 (72)
Total	0 (0)	0 (0)	9 (5.0)	36 (20.1)	134 (74.9)
Case studies and panel discussions
Live course	1 (0.4)	2 (0.9)	18 (8.0)	54 (24.0)	150 (66.7)
Online	0 (0)	0 (0)	0 (0)	7 (28.0)	18 (72)
Total	1 (0.4)	2 (0.8)	18 (7.2)	61 (24.4)	168 (67.2)

Discussion

Our CERTAIN program experience suggests that virtual blended synchronous and asynchronous delivery of education is a viable alternative to live course offerings and can efficiently expand the global reach of these programs. This transition has taught us several important lessons essential to the successful planning and preparation of both faculty and learning tools and several key barriers that require future research.

First, the transition from live to virtual course delivery was significantly facilitated by the clear and concise instructional design of our course curriculum using the ADDIE framework. Second, we recruited bilingual and local faculty to ensure that our case-based simulation experiences and discussions were placed in the appropriate cultural context to meet the needs of the local learner community. Bilingual faculty presence also created a more inclusive learning environment to support learners less fluent in English and help with learner engagement. Third, the development of a detailed faculty playbook and training sessions both before and immediately before our virtual simulation workshops was critically important to providing a high-quality online learning experience. This included the assignment of cohosts in each simulation breakout room during online sessions to ensure that both the technical delivery and educational objectives were accomplished. We also heavily scripted the case scenarios and key learning points during debriefing to allow a consistent experience in each breakout room. It is important to emphasize that although these changes required significant time and effort, once complete, the transition to virtual, blended synchronous, and asynchronous content delivery significantly reduced faculty time requirements and program delivery costs.

Healthcare professionals have long recognized the need for longitudinal learning throughout their careers to deliver the best possible care to the patients they serve. Continuing medical education is still the primary method healthcare professionals use to learn and apply new skills and technology in practice. Because of the rapid pace and evolution of science, medicine, and technology, conventional lectures and journals alone are proving insufficient to support the effective implementation of evolving best practices at the bedside. Our learners reported that our virtual platform offered them greater flexibility to engage in our program on demand from their office or home and provided greater flexibility to navigate clinical schedule conflicts and time zone differences.

The CERTAIN course is a critical component to disseminating our systematic approach to acute and critical care globally and facilitating organizational change to deliver the best care possible to patients regardless of the setting or resources. Virtual delivery of our CERTAIN program offers an attractive vehicle to support knowledge translation into practice for critically ill patients and offers both flexibility and a number of attractive efficiencies for global audiences. It is especially beneficial for healthcare teams in mid- and low-income countries with fewer resources to travel long distances to attend a live, in-person CERTAIN program (16). We have found that the variety of virtual options we offer learners to interact directly with their instructors and other students using chats, polls, and breakout rooms has also served as an important catalyst for communication and feedback.

Opportunities for Improvement

There are a number of opportunities for improvement that remain as we continue to refine our virtual CERTAIN program. International participants encountered significant difficulty connecting to and navigating our Blackboard using their mobile devices, requiring us at times to share separate links to this content, which limited our ability to track learner engagement. We are currently in the process of switching to a different learning management system platform that offers learners more mobile-friendly options to access course contents, interact with other learners, and build relationships before, during, and after our virtual workshop sessions. Although we invested significant time and effort to partner with local champions to understand the cultural context, patient–clinician communication preferences, and local values and beliefs in advance of our training sessions, the content of our collaborative conversations varied and, at times, did not capture important subtleties that were raised during subsequent virtual discussions. Language barriers were perhaps the greatest challenge that we encountered during our virtual simulation sessions. Although we deliberately placed learners from the same region in a breakout room with a bilingual cohost, our group still encountered a steep learning curve to provide consistency and real-time quality control that has only improved with time and experience. Our course assessments pose methodological concerns, as it only includes a limited number of virtual program participants and only measure learner reaction (level I of the Kirkpatrick model). We found it difficult to use a standardized approach to measure individual learning and team behaviors during virtual simulation scenarios, given the tasks and language barriers our faculty must currently navigate, which represents an opportunity for improvement. Although the CERTAIN approach and longitudinal virtual coaching have been shown to impact clinical behaviors and patient outcomes (Kirkpatrick level III–IV) in our original quality improvement trial and subsequent pilot efforts, our ability to measure similar outcomes over the past several years has been hampered by the clinical impact of the COVID-19 pandemic that has limited our ability to perform extensive data collection at local participating sites.

Conclusions

We have demonstrated high international participant satisfaction with both our CERTAIN live in-person and virtual education programs. Virtual delivery of the CERTAIN approach offers an innovative and flexible approach to facilitate the broad dissemination of best practices to a global audience to advance critical care delivery, and we continue to explore the best approach to address the opportunities for improvement that we have encountered to this point.