Feasibility of an International Remote Simulation Training Program in Critical Care Delivery: A Pilot Study

Abstract

Objective

To determine the feasibility and effectiveness of a video-enabled remote simulation training program to teach a systematic, standardized approach to the evaluation and management of the critically ill patients as part of an international quality improvement intervention.

Patients and Methods

In this pilot “train-the-trainer” prospective cohort study, we provided a remote simulation-based educational program for practicing clinicians from intensive care units involved in an international quality improvement project (www.icertain.org). Between February 21, 2014, and August 6, 2015, participants completed a self-guided online curriculum and participated in structured simulation training using web conference software with recording capabilities. The performance was assessed using a matched pair analysis at baseline and using standardized scenarios and a validated assessment tool postintervention. Participants rated their satisfaction with the training experience and confidence in implementing these skills in clinical practice.

Results

Eighteen local champions from 8 hospitals in 7 countries in Asia, Europe, and South and Central America completed the educational program. Learners exhibited significant improvements in cumulative critical task performance during simulated critical care scenarios with training (60.3%-81.8%; P=.002). Most clinicians (94%) reported that they felt well prepared to manage the common critical care scenarios after training. These local champions have subsequently delivered this educational program to more than 800 international clinicians over a 4-year period.

Conclusion

Insufficient training is a major barrier to the delivery of cost-effective critical care in many areas of the world. Video-enabled remote simulation training is a low-cost, feasible, and effective method to disseminate clinical skills to critical care practitioners in diverse international settings.

The World Health Organization (WHO) estimates suggest that up to 20 million patients require mechanical ventilation and 19 million present with sepsis in low- and middle-income countries annually.¹ Critical care services in these locations are frequently unable to meet this high demand because of limited equipment and inadequate training, prompting the WHO and international subspecialty societies to advocate for increased educational efforts using simulation, telemedicine, and Internet-based courses for indigenous health care workers.2, 3, 4

On the basis of a survey of clinicians from diverse practice settings in resource-limited locations,⁵ an international team of critical care practitioners has conducted a large global quality improvement project aimed at standardizing the approach to the evaluation and treatment of critically ill patients through the use of a checklist with embedded decision support (Checklist for Early Recognition and Treatment of Acute Illness and Injury [CERTAIN]). Recognizing the prohibitive barriers of distance and cost for these practitioners to travel and train at simulation centers, we harnessed the opportunity of readily available web-based video conferencing software to conduct a remote simulation training program.

The purpose of this pilot study was to evaluate the feasibility and effectiveness of remote simulation training to teach a systematic, standardized approach to critical illness and injury to a group of international practitioners and local champions implementing CERTAIN in diverse resource-limited settings.

Patients and Methods

The protocol for this trial was reviewed and approved by the Mayo Clinic Institutional Review Board, and the trial was conducted in adherence to Mayo Clinic conflict of interest policies. Participants were recruited between February 21, 2014, and August 6, 2015, through the CERTAIN Research Network (www.icertain.org).5, 6 After providing informed consent, volunteers participated in an educational intervention offering a systematic, standardized approach to the evaluation and management of the critically ill patients. This educational program included (1) an online curriculum that provided the rationale and orientation to the systematic checklist for decision support using a slide presentation, a video demonstration, and published reference papers establishing its effectiveness and (2) a 2-hour hands-on simulation-based training session for the participants and members of their care team at each hospital facilitated remotely by the investigators (Y.D., R.K., M.S., L.G.-A.) in English through a video conferencing service (Google Hangouts, Skype, or Zoom) (Supplemental Figure 1, available online at http://www.mcpiqojournal.org).

Participants completed a simulation-based assessment at baseline and postintervention in 2 independent sessions, conducted 2 weeks apart to minimize short-term recall bias. We used 3 standardized clinical scenarios—acute dyspnea, sepsis, and acute coronary syndrome—presented via a “screen share” from an office computer in the coordinating center (Rochester, Minnesota), including a simulated monitor display of vital signs, radiographic images, and laboratory results. One participant served as a team leader to direct the resuscitation, with 2 to 3 other team members serving as a physician, nurse, respiratory therapist, or pharmacist. Each participant rotated to serve as a team leader during 3 scenarios. Each participant’s performance was evaluated independently by using 2 raters and a previously validated assessment instrument.⁷ Participants were also asked to complete a knowledge assessment and survey assessing their satisfaction and confidence after training completion.

Statistical analyses were performed using JMP software version 10.0 (SAS Institute Inc.). A 2-sided P value of less than .05 was considered statistically significant. Continuous variables were reported as means ± SD, and categorical variables were reported as frequency and percentage. A matched pair analysis was performed. The task time was reported as median with interquartile range. Each participant served as his or her own control, so paired comparisons were performed using the McNemar and Wilcoxon signed-rank tests as appropriate. Satisfaction survey results were measured using a 5-point Likert scale (1 = strongly disagree; 5 = strongly agree).

Results

Eighteen local champions from 8 hospitals on 3 continents (Asia, Europe, North America) participated in the Internet-based curriculum and remote simulation training in 16 sessions (8 pre and 8 post). Study participant demographic characteristics are listed in Supplemental Table 1 (available online at http://www.mcpiqojournal.org). All participants completed the online curriculum and participated in a remote simulation-based training session to receive a facilitated practical experience in effective checklist utilization, with debriefing and reinforcement from one of the participating investigators. Each training session lasted 2 to 3 hours, with language and comprehension challenges being the major factors contributing to longer sessions. After completing the educational intervention, participants exhibited a significant improvement in total critical task completion rate (60.3%-81.8%; P=.002) (Table 1). The areas of greatest performance improvement were assessment of code status, disability, medical history, home medication, and allergies. Participants also exhibited increased efficiency, completing a considerably greater number of tasks in the standard 10-minute simulation session after training. Also, some of those tasks (code status discussion; airway, breathing, cardiac, disability, and exposure assessment; start oxygen supplementation, etc) were completed much faster in posttraining sessions (Table 2).

Table 1.

Proportion of Individual Tasks Completed

Item	Pretraining group (n=18)	Posttraining group (n=18)	P value
Code status discussion	4 (22.2)	18 (100)	<.001
Airway assessment	14 (77.8)	16 (88.9)	.37
Breathing assessment	14 (77.8)	16 (88.9)	.37
Cardiac assessment	16 (88.9)	17 (94.4)	.55
Disability assessment	8 (44.4)	16 (88.9)	.01
Exposure assessment	9 (50)	11 (61.1)	.50
Evaluation of vital signs	17 (94.4)	17 (94.4)	>.99
Evaluation of temperature	9 (50)	10 (55.6)	.74
Review of medical history	10 (55.6)	15 (83.3)	.07
Review of home medication	7 (38.9)	15 (83.3)	.01
Review of allergies	2 (11.1)	13 (72.2)	<.001
Order initial basic laboratory tests	18 (100)	17 (94.4)	.31
Start oxygen supplementation	16 (88.9)	13 (72.2)	.21
Review of differential diagnosis	8 (44.4)	12 (66.7)	.18
No. of completed checklist elements	8.4±2.5	11.4±2.5	<.001
Completion rate (%)	60.3±17.7	81.8±17.7	.002

Data are presented as mean ± SD or as No. (percentage).

Table 2.

Time to Task Completion for Those Cases That Have Task Completion in Both Pre- and Postintervention Tests

Item	No. of cases available for the assessment	Pretraining time (s)	Posttraining time (s)	Time change (s)
Code status discussion	4	445 (362-532)	74 (65-189)	−371
Airway assessment	12	217 (128-321)	133 (81-183)	−84
Breathing assessment	12	217 (127-310)	104 (80-183)	−113
Cardiac assessment	15	155 (59-290)	97 (51-195)	−58
Disability assessment	8	298 (181-418)	258 (117-326)	−40
Exposure assessment	7	312 (124-514)	217 (130-404)	−95
Evaluation of vital signs	16	119 (51-219)	96 (55-156)	−23
Evaluation of temperature	7	164 (74-171)	161 (60-355)	−3
Review of medical history	10	96 (66-256)	124 (94-250)	28
Review of home medication	7	121 (99-354)	194 (113-257)	73
Review of allergies	2	311 (298-324)	372 (200-545)	61
Order initial basic laboratory tests	14	205(159-346)	210 (129-322)	23
Start oxygen supplementation	14	150 (70-356)	93 (69-203)	−81
Review of differential diagnosis	6	491 (349-561)	322 (233-297)	−169

Data are presented as median (interquartile range) unless otherwise indicated.

Sixteen participants (89%) completed the survey assessing their satisfaction and educational experience. All but 1 clinician (94%) reported that they felt well prepared to manage the common critical care scenarios after training. Most clinicians (75%) felt that the training scenarios were realistic, and 81% recommended that this training should be taken by all critical care team members (Supplemental Table 2, available online at http://www.mcpiqojournal.org).

Through subsequent efforts of local champions, this educational program has been disseminated to more than 800 international clinicians over a 4-year period.

Discussion

Our study finds that remote training using an online curriculum and simulation-based training facilitated through remote video conferencing is both feasible and effective. Our diverse international participants were both satisfied with remote training and expressed confidence in their ability to incorporate this systematic approach into their clinical practice. Our findings suggest that remote simulation sessions can provide an effective educational environment without the cost and resources required for face-to-face training and/or a high-fidelity simulation laboratory.

Critical care is often not prioritized in low- and middle-income countries because of the misconception that its relative cost is too high in the setting of limited resources.⁸ Early recognition, resuscitation, and timely basic interventions (ie, antibiotics, fluid, vasopressors, and respiratory support) in the setting of acute critical illness can make a substantial effect on survival and disability, however, and can be improved using a standardized approach and checklist⁶ (Supplemental Figures 2 and 3, available online at http://www.mcpiqojournal.org). Although data in this area are limited, previous publications have suggested that the incremental cost-effectiveness for medical, emergency obstetric, general surgery, and trauma intensive care patients fall into the “very cost effective” category according to WHO criteria.3, 9, 10, 11, 12, 13

Leveraging technology to facilitate dissemination of educational programs to remote resource-limited settings is an appealing concept, given the vast need in these locations. Studies have shown that online teaching using information technology can achieve results similar to those achieved using traditional instructional methods.¹⁴ Technology-enhanced simulation is now an accepted and effective educational tool and patient safety strategy,15, 16 and its learning effect has been shown to persist using a remote Internet-based learning model.17, 18 Based on these principles, our educational intervention was designed to address common barriers to progress—limited staffing, access to medical education, and resources—by providing the instructors an online curriculum and video-based coaching using only a desktop computer and free online services. This research is innovative because the mobile Internet now offers an unprecedented opportunity to connect people around the world with flexible audio/video communication on a large scale with minimal cost.17, 19, 20, 21

This pilot project has a number of limitations. Without a control arm, the incremental effects of the individual components of our curriculum and the effect of observation alone on performance cannot be evaluated. We assessed only short-term performance changes using standardized simulated scenarios after our educational intervention, and whether these improvements were sustained or resulted in improved patient outcomes is currently under study.⁶ We were forced to integrate different online services and tools to provide training and assessment during this study, limiting the generalizability of this platform at the present time. Our training was conducted in English, and language barriers and other cultural considerations may also have affected clinician performance. There might be some investigator and participant bias in this study, as most participants were motivated to use a standard checklist and decision aid to help their patients; however, other than making the tool available at no cost, there was no monetary compensation from the main academic center. No industry funding was used for the study.

Conclusion

This pilot study found that low-cost remote training including an online curriculum and simulation-based training using video conferencing is both logistically feasible and effective in improving simulated performance and confidence in the systematic evaluation and treatment of critically ill patients. Further studies are needed to examine the long-term effect and frequency of reinforcement needed to ensure that improvements in clinician performance are maintained and result in improved clinical outcomes. Additional work is also needed to examine whether these results can be reproduced and sustained in a larger and even more diverse group of learners and settings and whether progressive training including common diagnostic and therapeutic critical care procedures (ie, bedside ultrasound) can be effectively delivered using these educational methods.

Supplemental Online Material

Supplemental material can be found online at: http://www.mcpiqojournal.org. Supplemental material attached to journal articles has not been edited, and the authors take responsibility for the accuracy of all data.