Comparison of Web-Based and On-Site Lung Simulators for Education in Mechanical Ventilation

Abstract

BACKGROUND:

Training in mechanical ventilation is a key goal in critical care fellowship education. Web-based simulators offer a cost-effective and readily available alternative to traditional on-site simulators. However, it is unclear how effective they are as teaching tools. In this study, we evaluated the test scores of fellows who underwent mechanical ventilation training by using a web-based simulator compared with fellows who used an on-site simulator during a mechanical ventilation course.

METHODS:

This was a nonrandomized controlled trial conducted as part of a mechanical ventilation course that involved 70 first-year critical care fellows. The course was identical except for the simulation technology used. One group of instructors used a traditional on-site simulator, the ASL 5000 Lung Solution (n = 39). The second group was instructed in using a web-based simulator, VentSim (n = 31). Each fellow completed a pre-course test and a post-course test by using a validated, case-based ventilator waveform examination that consisted of 5 questions with a total possible score of 100. The primary outcome was a comparison of the mean scores on the posttest between the 2 groups. The study was designed as a non-inferiority trial with a predetermined margin of 10 points.

RESULTS:

There was no significant difference in the mean ± SD pretest scores between the web-based and the on-site groups (21.1 ± 12.6 and 26.9 ± 13.6 respectively; P = .11). The mean ± SD posttest scores were 45.6 ± 25.0 for the web-based simulator and 43.4 ± 16.5 for on-site simulator (mean difference 2.2; one-sided 95% CI –7.0 to ∞; P_{non-inferiority} = .02 [non-inferiority confirmed]). Changes in mean ± SD scores (posttest – pretest) were 25.9 ± 20.9 for the web-based simulator and 16.5 ± 15.9 for the on-site simulator (mean difference 9.4, one-sided 95% CI 0.9 to ∞; P_{non-inferiority} < .001 [non-inferiority confirmed]).

CONCLUSIONS:

In the education of first-year critical care fellows on mechanical ventilation waveform analysis, a web-based mechanical ventilation simulator was non-inferior to a traditional on-site mechanical ventilation simulator.

Introduction

Mechanical ventilation is recognized as a curricular milestone for both pulmonary critical care medicine and critical care medicine.¹ However, a recent review on the topic noted a paucity of evidence in the best practices to teach mechanical ventilation or how to assess competence.² Medical simulation is a well-suited tool to teach mechanical ventilation.³ Competency in mechanical ventilation requires critical thinking and clinical reasoning skills. Simulations that aim to develop these skills prioritize cognitive engagement over physical realism.⁴ These simulations often blend conceptual and physical fidelity to create scenarios that require trainees to diagnose, make decisions, and apply treatment plans. Indeed, previous studies on mechanical ventilation education have used medical simulation as a best practice to replicate clinical scenarios by providing effective hands-on education in a safe learning environment.^2,3,5 Simulation fidelity frequently serves as a benchmark when evaluating different simulators. Fidelity is defined as the degree of similarity between the training situation and the operational situation that is simulated.⁶ It measures how closely a simulation mirrors an actual situation. However, high fidelity in medical simulations may not result in superior educational outcomes.⁷

A variety of traditional on-site simulators have been developed to teach mechanical ventilation at the bedside and in the classroom. One popular mechanical ventilation simulator is the ASL 5000 Lung Solution (IngMar Medical, Pittsburgh, Pennsylvania),^8,9 which integrates the simulation device with a manikin such as the SimMan 3G, SimJunior, or SimBaby (Laerdal, Stavanger, Norway) to provide high-fidelity simulation of clinical situations. Although useful for providing high-fidelity mechanical ventilation training, the traditional on-site simulators, for example, the ASL 5000 Lung Solution, have limitations. They are expensive, which limits the number of educators who can incorporate them into mechanical ventilation training. Furthermore, considerable training and experience are needed to set one up, which further limits accessibility for instructors. Web-based learning is a type of educational intervention that utilizes the internet to deliver instructional materials and facilitate learning.^10,11 Web-based learning has experienced notable growth in recent years, exemplified by the rise of massive open online courses.^12,13 Web-based learning can enhance public health literacy, provide continuing professional education, and supply innovative teaching models.¹³ In many educational examples, web-based learning serves as a practical and effective alternative to traditional classroom settings.¹⁴

Compared with traditional on-site simulators, web-based simulators are easily accessible to both instructors and learners, and are inexpensive or free to use, and they can be run from a laptop computer. Although web-based simulators may provide less fidelity compared with traditional simulators because they do not interface with actual ventilators or manikins, they may be equally effective in teaching detection and management of ventilator waveform asynchronies. Although it makes sense that a web-based simulator can be used by educators to effectively teach about mechanical ventilation asynchronies, there is a lack of evidence on the effectiveness of web-based simulators for teaching mechanical ventilation, and it is unclear whether a higher-fidelity learning environment provided through traditional on-site simulation is a more effective mechanical ventilation teaching tool than that provided by web-based simulators. In this study, educational outcomes among fellows at 3 sites who were taught with an web-based simulator (VentSim) were compared with fellows at 2 other sites where a traditional on-site simulator was used (ASL 5000 Lung Solution).

Quick Look.

Current Knowledge

Mechanical ventilation training is a vital component of education for critical care fellows. Traditionally, this training has been conducted by using on-site simulators. With advancements in technology, web-based simulators have become available, offering a more cost-effective and convenient alternative to traditional on-site methods. Despite this, there is a lack of clarity about the effectiveness of web-based tools for mechanical ventilation training.

What This Paper Contributes to Our Knowledge

This study demonstrated non-inferiority of a web-based mechanical ventilation simulation tool compared with a traditional on-site mechanical ventilation simulator in the education of critical care fellows on mechanical ventilation waveform analysis. Pre- and posttest scores from a mechanical ventilation course were compared between groups: one trained with an web-based simulator and the other with a traditional on-site simulator. The difference in mean posttest scores between the 2 groups was not significant, confirming non-inferiority of the web-based simulator in mechanical ventilation training. The study provides evidence that web-based simulation tools offer an effective approach to mechanical ventilation training for critical care fellows.

Methods

Design

This was a multi-center, prospective cohort study that included 84 fellows in their first year of critical care fellowship training. These fellows were trained in 25 different critical care fellowship programs. The course was delivered at 5 different sites in July and August 2022. First-year fellows from varied residency training backgrounds (internal medicine, pediatrics, neurology, surgery, anesthesia, and emergency medicine) participated in the course (Table 1). An institutional review board exemption for this study was obtained from all participating institutions.

Table 1.

Learner Demographics

The Web-Based Simulator

VentSim (https://ventsim.cc/) is an open-access web-based simulator developed by one of the authors (SS).^15,16 It allows teachers to create clinical scenarios and modify them in real time based on the learner’s clinical decision making. The software uses a mathematical model to emulate the respiratory system. In addition, it has the capability to simulate respiratory muscle pressure (P_mus) enabling it to replicate the patient’s respiratory effort and thus demonstrate asynchronies. After pretesting with mechanical ventilation experts and pilot testing with trainees, the initial version of VentSim was modified several times. The latest version of VentSim was used as part of a mechanical ventilation course for critical care fellows provided at 5 sites across the United States This course was originally part of the DC-Baltimore Critical Care Education Consortium^17,18 and is now being taught at centers across the country. Traditionally, the simulation portion of the course has been taught by using the ASL 5000 Lung Solution. However, because some centers lacked access to this traditional on-site simulator, VentSim was used to run identical simulations.

Curriculum

This multi-institutional mechanical ventilation course for critical care fellows spans 5 days and is now delivered in a hybrid manner: the first 2 days consist of virtual lectures and the final 3 days involve interactive lectures, small group praxis, and simulation sessions in person (course syllabus [see the supplementary materials at http://www.rcjournal.com]). The curriculum was standardized across all the sites and identical in teaching content and formats. The simulation cases were identical at every site, but the mechanism of delivery varied such that instructors in one group (Washington, DC, and Baltimore sites) used the traditional on-site simulator (ASL 5000 Lung Solution), whereas instructors in the second group (Atlanta and Pittsburgh sites) used VentSim (Fig. 1).

Fig. 1.

Flow chart.

The simulation scenarios used for all learners have been used over several years as a core part of the mechanical ventilation curriculum at centers that use the ASL 5000 Lung Solution. These scenarios relate to recognizing and managing auto-PEEP or asynchronies of triggering (autotriggering or ineffective triggering), flow (flow starvation or excess), and cycle (premature or delayed cycling). The content of the ASL 5000 Lung Solution–based simulation scenarios were translated to the VentSim platform. The newly created VentSim version of the scenarios were reviewed and edited by several experts in mechanical ventilation (BWL, NS, EK, SIM, MA, SS) to ensure consistency with the content taught on the traditional on-site simulator. The objectives of the simulation scenarios are provided in the supplement (see the supplementary materials at http://www.rcjournal.com). Instructors who facilitated the simulations underwent comprehensive training to proficiently execute the scenarios on the designated platforms for their respective groups. Before the commencement of the course, multiple practice sessions were conducted, ensuring that all instructors were well versed in the intended learning objectives.

Assessment

All course participants were given 5 ventilator waveforms to interpret and manage before the start of the course (pretest) and at the conclusion of the course (posttest). The waveforms consisted of pressure-time, flow-time, and volume-time scalar images of real patients who exhibit one or more clinically important issues, such as auto-PEEP or asynchronies of triggering, flow, or cycling. This assessment has been described and validated in previous studies^19,20 The test was scored from 0 to 100, with each case worth 20 points. A version of this test has been published as part of a previous mechanical ventilation education manuscript.¹⁸ All study participants completed a Likert scale survey that rated the educational effectiveness of the simulators (ASL 5000 Lung Solution or VentSim) and their desire to use that platform again to learn similar content (Supplementary Table 1, see the supplementary materials at http://www.rcjournal.com). We created the survey and modified the content via consensus. The survey was piloted among critical care fellows who did not participate in the course and was revised based on learner feedback before being used in the current study.²¹

Statistical Analysis

The primary goal of the study was to determine whether simulation taught by using the web-based simulator was non-inferior to using the on-site simulator based on post-course learner assessment. The primary outcome was the comparison of the posttest scores between the fellows taught in the 2 groups. A pre-specified non-inferiority margin of 10 points was chosen by consensus of four of the authors (SS, JS, BWL, and NS). This was felt to be a reasonable margin because 10 points indicated a difference in test performance of half of a question. Sensitivity analysis that compared the changes in pre- and posttest scores for the 2 groups was also performed. Two-sample t tests were used to compare the 2 groups. Paired t tests were used to compare post- versus pretest scores within each group. One-sided P values were used to assess non-inferiority, whereas 2-sided P values were reported for all other analysis. P values were considered significant if ≤ .05. The statistical analysis was conducted by using R version 4.2.1 (R Foundation for Statistical Computing, Vienna, Austria) and for SAS version 9.4 (SAS Institute, Cary, NC).

Results

Study Subjects

Of 84 first-year critical care fellows who took the mechanical ventilation course, 70 completed the pre- and posttests and were included in the final analysis (Fig. 1). The demographics of training background, type of critical care fellowship, and site where they took the course are listed in Table 1.

Test Scores

The mean ± SD pretest scores were low but similar between the web-based simulator and the on-site simulator groups (21.1 ± 12.6 vs 26.9 ± 13.6, respectively; P = .11). The mean ± SD posttest score increased in both groups (45.6 ± 25.0 for the web-based simulator and 43.4 ± 16.5 for the on-site simulator; P < .001 for both) (Fig. 2). When comparing the pre-specified primary outcome of posttest scores, the web-based simulator was non-inferior when compared with the on-site simulator, with one-sided P = .02 and one-sided 95% CI –7.0 to ∞, confirming non-inferiority. For the pre-specified sensitivity analysis, the participants in the web-based simulator group had a mean ± SD improvement of 25.9 ± 20.9 compared with 16.5 ± 15.9 for the learners in the on-site simulator group, also consistent with non-inferiority of the web-based simulator (one-sided P < .001 and one-sided 95% CI 0.9 to ∞) (Fig. 3).

Fig. 2.

Boxplots comparing pre-test (A) and post-test scores (B) by study group. Pre-test scores were similar in both the web-based and on-site groups. Post-test scores were significantly higher than pre-test scores in both groups. Post-test scores were similar between the two groups.

Fig. 3.

Difference in post-test score compared to pre-test scores was similar between study groups. P for non-inferiority is < .002, non-inferiority confirmed.

Survey Results

Learners rated their experience positively and similarly between the web-based simulator and the on-site simulator groups in most surveyed domains (Table 2). However, learners who went through simulation sessions with the on-site simulator rated their sessions higher than learners who used the web-based simulator in a few domains: the recognition of auto-PEEP, effectiveness of the technology to learn mechanical ventilation, and interest in using the simulator again. Although formal qualitative analysis was not performed, all learner free-text comments related to the web-based simulator and the on-site simulator are provided (see the supplementary materials at http://www.rcjournal.com). A screenshot of the web-based simulator is provided in Fig. 4.

Table 2.

Learner Perception of Simulation Experience

Fig. 4.

A screenshot of VentSim demonstrating flow starvation asynchrony in volume control ventilation.

Discussion

Given the common use of simulation-based education in mechanical ventilation education, there is a clear need to understand the effectiveness of various simulators in supporting mechanical ventilation education. In the current study, learners who participated in an mechanical ventilation course by using the web-based simulator for the simulation scenarios had non-inferior posttest performance compared with those taught with a traditional on-site mechanical ventilation simulator. In addition, the participants in both groups found the simulation scenarios similarly beneficial across multiple domains. These findings are highly relevant because much of the previous published work in teaching mechanical ventilation used traditional high-fidelity simulations, which implied that they are more effective than web-based simulation.^3-5 However, this assumption that high-fidelity simulators are more effective than web-based simulators for mechanical ventilation education overlooks the fact that skill acquisition is effectively promoted through fundamental educational practices such as consistent practice, constructive feedback, and integration into learning curricula mapped to specific learning objectives rather than fidelity alone.

Previous work has shown that practicing intensivists and respiratory therapists struggle to recognize patient-ventilator asynchronies,^21-23 which suggests that current approaches to training in mechanical ventilation may be inadequate. In addition, the COVID-19 pandemic highlighted the dire need for better clinician training in managing mechanical ventilation.^24-26 The web-based simulator lowers the barrier for high-quality mechanical ventilation simulation–based training, which allows instructors to teach learners without the need for expensive equipment that requires expertise to set up and troubleshoot. An effective open-access web-based mechanical ventilation simulator allows more learners to benefit from high-quality simulation training, regardless of local resources or access to traditional on-site simulators. This could potentially allow for better training of physicians and respiratory therapists in understanding the nuances of mechanical ventilation waveform interpretation, managing mechanical ventilation, and having improved standardization of mechanical ventilation management globally.^1,2,18

There are several advantages of web-based learning that enhance the educational experience.²⁷ Medical students generally prefer web-based tutorials over traditional lecture–based education for accessibility, ease of use, freedom of navigation, high-quality medical images, and the ability for repeated practice.²⁸ Web-based learning enhances self-directed learning, which leads to improved retention of knowledge,¹⁴ and web-based learning has proven effective in teaching skills that require practical application.¹⁴ A specific benefit of web-based learning relevant to this multi-center mechanical ventilation course is the ability to provide learners in multiple locations access to the same instructional content. Because these benefits of web-based learning align with the learning objectives of the mechanical ventilation course, the finding that web-based learning is an effective platform for mechanical ventilation simulation makes sense. One of the main challenges of web-based learning is the expense to develop and maintain the simulator. However, because the web-based simulator is an open-access web-based simulator, financial considerations are not an impediment to implementation. However, web-based learning is susceptible to disruptions caused by technical problems such as server downtimes, software bugs, or connectivity issues, which can impede the learning process, which influences learners’ satisfaction and participation rates.²⁴

This study has several strengths. The comparison of the web-based simulator to the traditional on-site simulator was the only difference in the material and platforms within the mechanical ventilation course. Because the course has been provided for many years, all aspects of the learning environment, other than the simulators, were similar between the groups. The simulation scenarios have been used for many years, with the traditional on-site simulators as part of the mechanical ventilation course and were iteratively improved over several years by the educators leading the course to reflect real-world clinical scenarios and meet prespecified learning objectives. The simulation scenarios for the web-based simulator were developed to mirror those from the traditional on-site simulator with the same learning objectives. Five of the authors (Sami Safadi, Megan Acho, Stephanie I Maximous, Eric Kriner, and Burton W Lee) reviewed versions of the web-based simulator scenarios and once they were deemed ready for implementation, pilot testing was done with learners. The diversity of the fellows participating in the study is another strength. These fellows represented a mix of trainees from various geographic areas with different residency backgrounds and were enrolled in both critical care medicine and pulmonary critical care medicine fellowship training programs, which increases the generalizability of the study findings.

There are limitations of this study that should be acknowledged. First, the study was nonblinded for both instructors and learners, which could potentially introduce biases in learners’ responses for the secondary outcomes gathered via the surveys. However, lack of blinding is less likely to have affected the posttest scores, the primary outcome for this study. Another limitation is that, whereas the same simulation cases were used, the instructors themselves were not the same between institutions and/or groups, which could affect test scores. However, this limitation was mitigated by the fact that instructors taught the course with the same simulation platform for several years.

Conclusions

We found that an web-based simulator, VentSim, was non-inferior to a traditional on-site simulator, ASL 5000 Lung Solution, for teaching mechanical ventilation to critical care fellows. These findings suggest that a web-based mechanical ventilation simulator can be used in lieu of a traditional on-site mechanical ventilation simulator, offering an alternative that provides similar educational outcomes while allowing educators to scale up mechanical ventilation teaching without geographic or physical space limitations.