Just-in-Time In Situ Simulation Training as a Preparedness Measure for the Perioperative Care of COVID-19 Patients

Introduction

Routine workflows were redesigned during the first surge of the COVID-19 pandemic to standardize perioperative management of patients and minimize the risk of viral exposure and transmission to staff members. Just-in-time (JIT), in situ simulation training was adopted to implement urgent change, the value of which in a public health crisis has not previously been explored.

Methods

Implementation of workflow changes in the setting of the COVID-19 pandemic was accomplished through JIT, in situ simulation training, delivered over a period of 3 weeks to participants from anesthesia, nursing, and surgery, within our healthcare network. The perceived value of this training method was assessed using a postsimulation training survey, composed of Likert scale assessments and free-text responses. The impact on change in practice was assessed by measuring compliance with new COVID-19 workflows for cases of confirmed or suspected COVID-19 managed in the operating room, between March and August 2020.

Results

Postsimulation survey responses collected from 110 of 428 participants (25.7%) demonstrated significant positive shifts along the Likert scale on perceived knowledge of new workflow processes, comfort in adopting them in practice and probability that training would have an impact on future practice (all Ps < 0.001). Free-text responses reflected appreciation for the training being timely, hands-on, and interprofessional. Compliance with new COVID workflows protocols in practice was 95% (121 of 127 cases) and was associated with lower than expected healthcare worker test positive rates (<1%) within the network during this same period.

Conclusions

These findings support JIT, in situ simulation training as a preparedness measure for the perioperative care of COVID-19 patients and demonstrate the value of this approach during public health crises.

The COVID-19 pandemic, caused by severe acute respiratory syndrome coronavirus 2, required health systems globally to plan for an imminent surge of COVID-19 patients.^1–3 Our organization, Harvard Medical Faculty Physicians and Associated Physician Harvard Medical Faculty Physicians, provides care for a large urban, metropolitan area of nearly 5 million people and offers more than 47,000 elective and emergency surgical procedures per year, delivered across 12 network hospitals in the greater Boston area of Massachusetts. Given the risks posed to healthcare workers (HCWs) involved in direct patient-facing roles and in particular those exposed to aerosol-generating procedures,^4,5 we rapidly redesigned routine workflow processes to prepare our perioperative staff to safely care for COVID-19 patients in the operating room (OR), reduce the risk of COVID-19 transmission among HCWs, reduce anxiety, and promote a shared mental model.

As part of our organizational strategy, urgent just-in-time (JIT) training delivered through in situ simulation facilitated the rapid implementation of these new workflows to perioperative staff within our department. Just-in-time training is an educational strategy that aims to refresh knowledge and skills for infrequently performed high-risk activities.⁶ Previously, in the healthcare setting, JIT training has been used to address a decline in procedural skills over time and to correct gaps in knowledge.^7,8 It has also been proven to promote provider confidence in performing specific tasks and facilitate urgent training when required and is usually delivered in close temporal and spatial proximity to the clinical encounter for which the skill is required.^9–12 Simulation is an effective tool for interprofessional training and has also been used as a tool for implementation with great success in other settings.^13,14 In situ simulation is delivered within providers' actual work environment and can facilitate interprofessional training. It provides a means to implement change, error proof a process, reveal site-specific latent hazards, and identify opportunities for improvement.^15–17 We therefore combined JIT training with in situ simulation to implement the newly designed workflows for the management of confirmed or suspected COVID-19 patients within the perioperative setting.

Throughout the COVID-19 pandemic, this training model has developed in popularity^18–22; however, the impact of this training method on behavior change in clinical practice is not fully understood in the context of a public health crisis. In this study, we therefore report first, the implementation of a JIT in situ training program for the perioperative management of a COVID-19 patient; second, a comprehensive evaluation of its perceived value using both quantitative and qualitative survey results; and third, the impact on change in practice.

METHODS

This study was approved by our local institutional review board at the Beth Israel Deaconess Medical Center (BIDMC) with a waiver of documentation of informed consent. This study follows the guidelines published by the Consolidated Standards of Reporting Trials group by using the STrengthening the Reporting of OBservational studies in Epidemiology checklist.²³

Part 1: In Situ Simulation Training Program Development and Implementation

New standard operating procedures (SOPs) and perioperative workflows were developed based on available evidence and published recommendations at the time^24,25 and converted into single-page checklists to be used in clinical practice as cognitive aids. New SOPs reflected the changes to routine workflows required when managing a patient with confirmed or suspected COVID-19 in the OR, to ensure the safety of HCWs and promote standardization of care. Implementation and training of new workflows was accomplished through in situ simulation training.

Simulation drills focused on aspects of the perioperative workflow, which either posed a high risk of viral exposure and transmission to HCWs, represented a significant deviation to the current routine workflow, or addressed locally recognized gaps in care. As outlined in Table 1, they included a preoperative huddle and OR setup for a COVID-19 case, donning and doffing of personal protective equipment (PPE), transferring patients from the intensive care unit (ICU) to the OR and airway management using enhanced infection control measures.^26–28 Simulations were designed to be brief (<30 minutes), to take place in situ in the actual ORs native to providers and to be interprofessional. A detailed description of each simulation scenario can be found in the Supplementary Digital Content 1 (Table, http://links.lww.com/SIH/A784, scenarios for interprofessional simulation training in the OR).

TABLE 1.

Simulation Scenarios

Simulation Scenario/Drill	Purpose, Goals, and Tasks
Sim 1: The preoperative huddle for a suspected or confirmed COVID-19 patient (including OR set up)	Purpose: Enable participants to become familiar with the new SOP and perioperative workflow checklist in place for the management of suspected or confirmed COVID-19 cases in the OR. Observe the recommended OR set up for a COVID-19 case. Tasks: Perform a preoperative team huddle and complete the preoperative checklist before the patient is collected or transferred into the OR, to enforce a shared mental model among the team.
Sim 2: Donning and doffing of PPE	Purpose: To review the recommended sequence of donning and doffing PPE Tasks: To perform the recommended sequence of donning and doffing PPE
Sim 3: Transfer of suspected or confirmed COVID-19 patient from the ICU to the OR	Purpose: Enable participants to become familiar with the new SOP for the transfer of a suspected or confirmed COVID-19 patient from the ICU in the OR. Enforce how the use of the shared mental model helps to safely transport a high risk, unstable patient and minimizes the opportunity for spread of contagion. Tasks: Perform an actual transfer of an intubated COVID-19 patient from the ICU to the OR, using the SOP guidance and perioperative workflow checklist as a cognitive aid.
Sim 4: Airway management using enhanced infection control measures	Purpose: Enable participants to become familiar with the new SOP for airway management of all cases, including COVID-19 cases, using enhanced infection control measures. Tasks: Perform and practice routine airway skills (intubation, extubation), while adopting new recommended techniques to minimize exposure/contamination to pathogens.

Description of each simulation scenario and their intended purpose. Sim 1, scenario 1; Sim 2, scenario 2; Sim 3, scenario 3; Sim 4, scenario 4; SOP, standard operating procedure.

A core simulation implementation team, consisting of members of the BIDMC's department of anesthesia's quality, safety and innovation team established the content and purpose of each activity. Within 24 hours, simulation materials were developed based on new SOPs, equipment was sourced, and faculty were designated as trainers. A faculty training group consisting of 10 anesthesiologists and 2 OR nursing educators was established and trained within 1 day, to facilitate ongoing delivery of in situ simulations (Fig. 1). This “train the trainer” session took place on March 17, 2020, 3 days after the first patient with suspected COVID-19 presented to our OR. The session was conducted at the BIDMC Hospital in the same vacant ORs that further training sessions would later be carried out in and where future real cases would be managed. Following an on-site walk through all trainers were required to perform and then lead each practical skill and simulated scenario. The session concluded with an in-person group debrief to suggest improvements to training where required. Within 72 hours, the simulation training program was scaled up and rolled out to the entire perioperative department, offering up to 12 simulation sessions per day, for groups no larger than 10, over the course of 3 weeks. Each session was facilitated by 2 simulation trainers. Although not mandatory, participation was strongly advocated by senior leadership within anesthesia, surgery, and nursing.

FIGURE 1.

Schematic representation of the simulation implementation team and framework. The core development team (dark gray), faculty trainers (gray), network-wide interprofessional staff members (light gray).

In keeping with the JIT framework, to provide training to learners as the need arises or in anticipation of the need, simulation drills were conducted within vacant COVID-19 designated ORs or those immediately adjacent. Drills began within days of the first COVID-19 patient presenting to our department and often within days of participants themselves performing the required skills in real time. Participants included staff from anesthesia, surgery, nursing, OR attendants, and technicians, including anesthesia providers from across the network. All participants were encouraged to participate in each simulation drill to promote awareness of workflow changes and a shared mental model. Daily feedback from training faculty was obtained through end-of-day debriefings, using a plus/delta format, and communicated back to the core implementation team.

Part 2: Evaluation of the Perceived Value of Our Training Program

Survey—Design

Participants were invited to complete anonymous postsimulation training surveys immediately after each session. A 34-question survey was designed in an iterative fashion by the study team (M.J.C. and J.D.M.) after review of existing literature. It was then reviewed and piloted among members of the education department with experience in design and implementation of survey instruments,^29,30 to confirm whether survey items were appropriate and in line with study objectives. The survey was intentionally designed to assess Kirkpatrick level 1³¹ through Likert scale questions and open text feedback assessing respondents' self-perceived knowledge and comfort levels before and after simulation and whether participants believed that their participation in simulation will impact their practice. Surveys were created using Google Forms (Google LLC, CA),³² optimized for mobile devices, and allowed respondents to skip nonapplicable questions or questions they did not wish to answer. Participants were invited to complete surveys immediately after simulations, via a Quick Response code (QR code) and online link, with reminders implemented to encourage survey participation through announcements during our department-wide virtual meetings and 2 postcourse e-mails. Surveys were collected over a period of 51 days (April 6, 2020, to May 26, 2020). The complete set of survey questions is available in Supplementary Digital Content 2 (survey, http://links.lww.com/SIH/A785, which replicates the questions listed in the survey distributed to participants).

Survey—Quantitative Analysis

Statistical analyses on quantitative data obtained from our surveys were performed on respondents' perceived knowledge in new protocols and comfort levels in applying them in practice before versus after simulation. As we expected, unidirectional responses after simulation to either improve overall or be relatively unchanged, 1-tailed sign tests were applied. Analyses were performed using statistical calculators from Social Science Statistics (Social Science Statistics, 2018). An α value threshold of less than 0.05 was used to determine statistical significance; after applying Bonferroni corrections, this resulted in Bonferroni adjusted P value thresholds for significance of less than 0.006.³³

Survey—Subgroup Analyses

Exploratory subgroup analyses were performed to assess whether provider discipline or level of training, specifically in anesthesia, may have led to any meaningful differences in responses. Disciplines were defined as either anesthesia, surgery, or nursing. Participants' levels of training were classified into groups defined as attending anesthesiologists, trainees that included anesthesia residents and fellows, or midlevel providers that included certified registered nurse anesthetists, nurse practitioners, physician assistants, and registered nurses. Kruskal-Wallis tests were performed to measure differences between groups in both subgroup analyses.

Survey—Qualitative Analysis

Free-text responses from our survey were analyzed using the method of content analysis previously described by Bengtsson.³⁴ for qualitative studies, which does not interpret words but instead stays very close to the actual text, and describes visible and obvious terms within each response. Written free-text responses were reviewed by the authors L.Z., N.L., J.R.K., and S.K.R., and using a group deliberation approach, responses were grouped into categories of recurring themes.

Part 3: Evaluation of the Impact of Our Training Program

Changes to Organizational Policy

Daily feedback from participants and simulation faculty was obtained either through group debriefing or direct communication via e-mail, to identify location specific hazards or gaps in workflow and enable continuous iterative improvement to the simulation program, as well as the organizational SOP.

Changes Observed in Practice

Between March and August 2020, a total of 127 patients with suspected or confirmed COVID-19 were managed in the OR. Compliance with new perioperative workflow processes was evaluated for each case, by members of the core team or faculty trainers (L.Z., N.L., J.R.K., and S.K.R.). Compliance was assessed either through direct observation, noting which elements of the workflow checklist were completed, or through the review of the electronic medical record. A modified anesthesia information management system helped capture and record if specific elements of the workflow were performed as per COVID checklist protocol: including patient transport, intraoperative management, and early postoperative recovery. This process measure facilitated the assessment of Kirkpatrick level 3.³¹

Healthcare Worker Infection Rate

To provide supportive evidence of Kirkpatrick level 4 beyond reported adherence with the protocols, anonymized data from employee health records were reviewed between March 18, 2020, and June 10, 2020. The number of confirmed COVID-19 test–positive cases was recorded for each perioperative discipline; anesthesia, surgery, and nursing. Results were compared with publicly reported raw and weighted test positivity rates in Massachusetts during the same period.

RESULTS

Part 1: Simulation Program Development and Implementation

In situ simulation training sessions were delivered to perioperative staff members, over a period of 3 weeks, from March 18, 2020, through to April 3, 2020, to a total number of 428 staff members. Our perioperative staff's roster fluctuates approximately 750 active staff from nursing, anesthesia, and surgery combined at any given point; therefore, this reflects approximately 57% of staff members within the organization. Participants included staff from anesthesia (n = 183, 42.8%), surgery (n = 29, 6.8%), nursing (n = 166, 38.8%), and other perioperative staff (n = 50, 11.7%) including clinical advisors, OR attendants, technicians, and unit coordinators. Most staff who participated were local to BIDMC (n = 372, 87.0%), whereas 13.1% (n = 56) were from hospitals across the Harvard Medical Faculty Physicians/Associated Physician Harvard Medical Faculty Physicians network.

Part 2: Evaluation of the Perceived Value of Our Training Program

Survey Results—Demographics

Our survey generated 110 responses from 428 participants (25.7%). Responses were collected from 11 of 12 hospitals within our network. Respondents showed diversity in discipline, level of training, and experience, suggesting that these survey results are representative to perioperative health care workforces (Table 2—survey demographics).

TABLE 2.

Survey Results—Demographics

Survey Responses	Total	110 (25.6%)
Provider discipline	Anesthesia	73 (66.3%)
	Nursing	20 (18.3%)
	Surgery	12 (10.9%)
	Other, including OR technician, unit coordinator	5 (4.5%)
Level of training	Attending	45 (40.9%)
	Resident	19 (17.3%)
	CRNA	14 (12.7%)
	OR nursing	21 (19.1%)
	Other (fellow, technicians, NP, PA)	11 (10%)
Years of work experience	>15 y	56 (51.8%)
	4–9 y	27 (24.6%)
	<3 y	29 (22%)
Prior simulation experience	>10 occurrences	29 (26.6%)
	4–9 occurrences	24 (22%)
	1–3 occurrences	43 (39.4%)
	No prior experience	13 (11.9%)
Primary location of work	BIDMC	74 (67.2%)
	Other hospital within BIDMC network, in Boston	11 (10%)
	Other hospital within BIDMC network, outside of Boston	25 (22.7%)
Experience in managing a COVID-19 case in the OR	Yes, before simulation training	37 (33.6%)
	Yes, after simulation training	41 (37.2%)
	No	32 (29.1%)

CRNA, certified registered nurse anesthetist; NP, nurse practitioner; PA, physician assistance.

Survey Results—Quantitative Analysis

For all 4 simulations, mean values for perceived knowledge in new workflows and comfort levels in applying them in practice improved by at least 1.2 points on the Likert scale (Figs. 2–5). These differences were statistically significant for all 4 simulations (P < 0.001). Shifts from normal distributions (all presimulation skewness values ≤ |0.5|) to left-skewed distributions (all postsimulation skewness values < −0.8) were noted in all 4 simulation scenarios, with >85% of respondents reporting ratings of 4 (agree) or 5 (strongly agree) for knowledge and comfort levels. Similarly, more than 90% of respondents agreed (Likert scale 4) or strongly agreed (Likert scale 5) that participation in the simulation will impact their future clinical practice, for each simulation scenario (Fig. 6).

FIGURE 2.

Preoperative huddle simulation's knowledge and comfort assessment, before and after simulation. Results of knowledge of protocols (A) and comfort in adapting protocols (B) are expressed as percentage of responses. Presimulation responses are noted in dark gray; postsimulation responses are in light gray. X-axis represents 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). Presimulation mean scores: knowledge = 3.0 (SD = 1.3); comfort = 2.8 (SD = 1.2). Postsimulation mean scores: knowledge 4.4 (SD = 0.7); comfort 4.3 (SD = 0.8).

FIGURE 5.

Airway management (including intubation and extubation) using enhanced infection control measures simulation's knowledge and comfort assessment, before and after simulation. Results of knowledge of protocols (A) and comfort in adapting protocols (B) are expressed as percentage of responses. Presimulation responses are noted in dark gray; postsimulation responses are in light gray. X-axis represents a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). Presimulation mean scores: knowledge = 3.3 (SD = 1.3); comfort = 3.4 (SD = 1.3). Postsimulation mean scores: knowledge = 4.7 (SD = 0.6); comfort = 4.7 (SD 0 = 5).

FIGURE 6.

Survey results for perceived impact of JIT simulation training on clinical practice. Results are expressed as percentage of responses for each simulation drill. Simulations are numbered to aid interpretation as follows; sim1: preoperative huddle, sim2: donning and doffing PPE, sim3: ICU transfer, and sim4: airway management using enhanced infection control measures. X-axis represents a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). Mean scores for sim 1: 4.6 (SD = 0.6), sim 2: 4.5 (SD = 0.7), sim 3: 4.5 (SD = 0.6), and sim 4: 4.6 (SD = 0.5).

FIGURE 3.

Donning and doffing PPE simulation's knowledge and comfort assessment simulation's knowledge and comfort assessment, before simulation and after simulation. Results of knowledge of protocols (A) and comfort in adapting protocols (B) are expressed as percentage of responses. Presimulation responses are noted in dark gray; postsimulation responses are in light gray. X-axis represents 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). Presimulation mean scores: knowledge 3.0 (SD = 1.3); comfort 3.4 (SD = 1.3). Postsimulation mean scores: knowledge = 4.4 (SD = 0.7); comfort = 4.6 (SD = 0.6).

FIGURE 4.

Intensive care unit transfer of COVID-19 patient simulation's knowledge and comfort assessment, before and after simulation. Results of knowledge of protocols (A) and comfort in adapting protocols (B) are expressed as percentage of responses. Presimulation responses are noted in dark gray; postsimulation responses are in light gray. X-axis represents 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). Presimulation mean scores: knowledge = 2.8 (SD = 1.2); comfort = 3.0 (SD = 1.3). Postsimulation mean scores: knowledge = 4.5 (SD = 0.6); comfort = 4.5 (SD = 0.6).

Survey Results—Quantitative Analysis (Exploratory Subgroup Analyses)

Subgroup analysis by primary discipline did not reveal meaningful differences between groups (P > 0.18) apart from a difference in knowledge and comfort levels in the airway management drill. Operating room technicians, a single unit-coordinator, and one individual who did not report their discipline were omitted from the subgroup analysis, given their small sample sizes. Mean changes in knowledge for the airway management simulation were 2.25 for nursing, 1.35 for anesthesia, and 1.29 for surgery (P = 0.028); mean changes in comfort were 2.38 for nursing, 1.34 for anesthesia, and 1.29 for surgery (P = 0.017). Before simulation, nursing respondents who participated in this simulation (n = 8) had mean scores of 1.875 on both knowledge and comfort, with none noting scores of 4 or 5. After simulation, there was only a single 3 noted for postsimulation knowledge, with all other knowledge and comfort scores being 4 or 5.

Subgroup analysis of anesthesia providers by level of training did not reveal a meaningful difference between groups when comparing changes in presimulation and postsimulation knowledge and comfort levels or perceived impact of training on clinical practice (P > 0.27). Results remained robust for all comparisons, apart from the perceived impact level scores for the preoperative huddle training session, with mean impact levels of 4.2 for trainees, 4.6 for midlevel workers, and 4.8 for attendings (P = 0.020). However, we did not consider this to be a meaningful difference.

Survey Results—Qualitative Analysis (Free-Text Responses)

Responses from the qualitative content analysis of free text were grouped into the following categories: simulation training content and materials, relevance to practical skills, relevance to clinical experience, perceived benefits and failures of training, interprofessionalism, organizational issues including site-specific issues, and communication. Results are outlined in Table 3. Overall themes noted from free-text responses reflected appreciation for the training being timely, hands-on, and interprofessional. Perceived benefits included the provision of a space to review and run through workflows in detail and ability to provide feedback to facilitate change if needed. Limitations and failures were reflected in responses pertaining to logistical and organizational issues, that is, a lack of consistency in the number of participants in each simulation.

TABLE 3.

Survey Results—Free-Text Responses

Categories	Responses
Simulation training content/materials	Summary: Preference for greater number of clinical scenarios to expand content, use of visual materials to aid in learning and clarification of minor points within our protocols Quotes: - Scenarios chosen were very clear, practical and common situations –Clarity is required around some of the minor details in the new protocols: (eg, allocation of roles during the simulation, correct sequence of a new techniques) –Content could have maybe included emergency situations (stat calls, traumas, combative patients, extreme respiratory depression) –Would have been useful to practice how to use alternative equipment, eg, PAPR –Only a simulation of the transfer of an intubated patient was performed, I would have preferred management of an unintubated COVID patient –May I suggest that we have videos and pictures available –Would be helpful to have larger posters of PPE donning and doffing sequence –Clarification needed in the transfer protocol, over where the runners go in relation to the patient on the transfer
Relevance to practical skills	Summary: Appreciation of the ability to practice a clinical skill not often performed and be “hands-on” in the process Quotes: –Simulations allowed me to practice my skills at using a McGrath VL, donning and doffing I learn better hands-on; it helps me to perform the task –Donning and doffing was important to practice, as its not done regularly enough –The hands-on nature of the ICU transfer was useful to me –I was able to practice the new intubation and extubation protocol –Gave me the opportunity to test my abilities and be familiar with the process –Useful practicing donning and doffing of PPE, since we don't do that often –PPE practice was well done –The hands-on nature of the ICU transfer was useful to me
Relevance to clinical experience/workflow process	Summary: Recognition of training as an opportunity to discuss upcoming or unanticipated hazards/safety issues Quotes: –Simulation was a great way to get used to the workflow and bring it to life –The preop huddle is a great tool, it would be wonderful to extend this to our general practice in the OR –Simulation got you thinking about the issues in dealing with a COVID-19 patient, and helped you learn from others' trial and errors –It was useful to discuss issues likely to come up during a real case, that I hadn't thought of
Perceived benefits of training	Summary: An increase in awareness of the new protocol, comfort in adopting protocols in practice and perceived impact that new protocols will have on clinical practice and safety of the HCW. The training method provided a forum to review the unique steps within a complex protocol, engage in teamwork, hear from others' experiences and offer feedback or suggestions to those who can enable change. Contribution in alleviating anxieties about personal HCW safety in the workplace. Quotes: –Allowed me to become better acquainted with problems pertaining to COVID-19 cases in the OR –Gave me a better understanding of how (the protocol) should work –Leading the sims helped solidify the protocols for me –Was a great way to learn how to stay safe in the OR (both patient and provider) –Confirmed things we already knew and added additional knowledge –The physical training is so much more helpful than reading it on paper or watching someone else! –Excellent practice to do a dry run and improve upon our techniques/flow for real-life patient care –Perceived teamwork –Practice is helpful in situations that are uncommon –Important to practice in a nonstressed, safe environment –Simulation provides an opportunity to critically review each step of the scenarios –It helped me review how to manage a potentially highly infectious patient, which I have not done for a long time –Provided an opportunity to hear about the most up-to-date protocol/policy changes –Provided an opportunity to hear about complaints –Helped prepare me to manage a COVID-19 case, I felt much more confident and comfortable following these simulations –The training made me much more comfortable working with COVID-19 case, it is essential for all clinicians –Mistakes are made without training; your life depends on it
Perceived failures of training	Summary: Challenge for participants to extrapolate lessons learned from simulation into real life. Difficulty in keeping up to date due with training materials due to frequent changes in hospital policy/guidelines. Quotes: –The knowledge that the content might be changing daily, impacted my learning –Not the same as in real-life, if you can't apply the real PPE because of a lack of supplies –Inconsistency between some of the instructors led to confusion –I think there is a big difference on what is in the simulation and how it happens in real practice –Sense of uncertainty with the instructor
Interprofessionalism/collaboration	Summary: The rapid organization of JIT simulation training sessions occasionally resulted in groups where all disciplines were not equally represented, this was viewed as a limitation when specific disciplines were missing, but otherwise as a benefit when appropriately mixed. This reflects the perceived usefulness of collaboration but needs to be balanced with group size (see organizational issues below). Quotes: –I found it most useful when there was nursing and anesthesia collaborating in the sim. There was a great discussion between the two disciplines on different ways to troubleshoot issues that were uncovered –Going over the protocols together as a team was useful –Collaboration, sharing of knowledge deficits, exchanging information with multidisciplinary colleagues, brain storming ideas for improved care –Preference for more of a diverse group of staff during drills –Having nurses there would be better –Not all disciplines are participating due to confusion and scheduling
Site-specific issues	Summary: Site-specific issues that reflect a lack of generalizability to other settings. Quotes –Working in a different location, with different resources and protocols, means I will have to adjust all of this and train my local teams –I'm grateful to have had the training since none has been done at my facility. I had to lead a team for a suspected COVID-19 case today and I'm not sure how I could have done this without having had the training –Resources of alternate sites are different, resulting in the need to modify what we've learned working within the BIDMC –We need administration at the community hospital to align with best practices
Organizational issues	Summary: Organizational planning was timely but felt rushed. JIT simulation training sessions occasionally resulted in groups where there were either too many participants or not enough from each discipline. Quotes: –Small groups together were more helpful –If the groups were too large, we would be watching and waiting a long time –The experience of the faculty instructing the course was too junior and this impacted their ability to be “experts” in this field –Assigned groups could have been better organized, fewer anesthesia providers, etc –I felt there were too many cooks in the kitchen –Too many people were present in the drill, I was unable to perform the skill –Would have preferred using actual PPE, rather than mock-up –Timely, thoughtful
Communication	Summary: JIT training was useful in accomplishing its desired task (implement and training of staff in new protocol) but demonstrated limitations in posttraining communication Quotes: –It really helps the nursing staff in preparing to care for these patients and increases communication between the disciplines –This was a better way of updating staff on changing protocols, it was better than e-mails (we were getting overloaded with emails!) –The protocol changed after I did the simulation, but I wasn't aware of this change
Leadership	Summary: Perception of leadership support within the organization Quotes: –Getting a feel for how thoughtful leadership was taking the situation –Amazing that it was put together so quickly –An overall great idea and I look forward to future simulations that can be rolled out and customized to the other sites

McGrath VL, McGRATH MAC video laryngoscope (Medtronic plc, Ireland); PAPR, powered air-purifying respirators.

Part 3: Evaluation of the Impact of Our Training Program

Change to Organizational Policy

Simulation drills and organizational SOP's underwent iterative updates based on feedback collected through debriefing and direct encounters, as well as evolving organizational polices and guidelines. As detailed in Table 4, over 3 weeks of simulations, we modified our simulation materials 7 times and made more than 20 amendments to organizational perioperative workflow SOPs and checklists.

TABLE 4.

Iterative Changes From Continuous Daily Feedback and Observed Gaps in Care*

Organizational SOPs	Simulation Training
Timeline: –January 26: SOP for airway management in COVID-19 patients –March 6: PPE recommendations (anesthesia) and SOP for in-hospital transfer of COVID-19 patients –March 13: Perioperative workflow checklists developed using hospital approved SOP –March 14: 1st COVID-19 suspected case to the OR –April 3: to date, 16 cases brought to the OR with either suspected or confirmed COVID-19.	Timeline: –March 14–16: Simulation materials developed based on newly designed perioperative workflows and hospital approved SOPs –March 17: 1st simulation scenario delivered to faculty to “train the trainers.” –March 18: 1st simulation scenarios delivered to perioperative staff members –April 3: final day of simulation drills/training scenarios
Feedback directly from real-life cases (staff): –SOPs/workflow checklists were lacking in detail –If already intubated, should the patient be transferred using an Ambu-ventilator or an ICU ventilator? –If not intubated, should they be intubated before arriving into the OR? –If not intubated, how many people should be in the OR during intubation/extubation? –Difficulty communicating during the case from inside the room to outside	Feedback directly from simulation training (faculty/participants): –Goals of each drill not specific enough –Concern of aerosol generation during extubation and lack of physical barriers –Detail in SOP lacking for contamination risk of miscellaneous items (eg, the patient's bed, linens, patients' notes) –Noted absence of postoperative staff in preoperative huddle (eg, PACU nurses)
Hazards/gap in care observed during real-life cases: –Clarity required for the variation in case mix, eg, intubation vs. nonintubated patient or patients receiving sedation only –Risk of contamination of surfaces inside the OR; eg, Omnicell, ventilator, operative supplies which are normally stocked inside –Pagers, mobile phones contaminated during cases	Hazards/gap in care observed during simulation training: –Accidental disconnection of ETT and circuit during transfer of intubated patient (risk of viral exposure during transfer) –Contamination of elevator buttons, doors by transfer team –Route for transfer requires clarity, should have a designated elevator, etc.
Changes made to organizational SOPs: –Workflows amended to reflect greater detail in participants in preoperative huddle, transfer team members and roles, mode of ventilation on transfer, number of people in the room during intubation and extubation, confirmation of verbal consent from patients –Amended the requirement of an RT and ICU ventilator, only if indicated by ICU.	Changes made to simulation materials: –Simulation materials amended to reflect specific goals of each drill, for each participant –Specific equipment, location and routes for transfer adjusted based on logistical availabilities on a daily basis –Online materials developed –Printed materials no longer used, mobile computer with access to materials online uploaded and used live during simulations
Changes to simulation materials: –Participants were shown a mock-up of a COVID-19 OR; decluttered, minimal equipment, machines/nonmobile items covered in plastic, etc. –A “pretransfer huddle” in the ICU was added to ensure safety precautions were met; sedation and paralysis, Kelly clamp, HME filter in place –A “planned circuit disconnection” sequence was added to the transfer simulation drill	Changes made to organizational SOP: –Additional transfer equipment for COVID-19 cases to include a Kelly clamp, HME filter and PEEP valve. –Team leader on transfer designated as sole member of team able to press elevator buttons/open doors, to minimize contamination of surfaces –Barrier (in the form of a blue towel) added to extubation sequence –Virtual presence permitted during the preoperative huddle of members who would otherwise result in prolonged delays

*Table concerns iterative changes made to organizational SOPs and training materials.

ETT, endotracheal tube; HME, heat and moisture exchange; PACU, postanesthesia care unit; PEEP, positive end-expiratory pressure; SOP, standard operating procedures.

Change in Practice

After a chart review or direct observation of cases managed in the OR, with suspected or confirmed COVID-19 between March and August 2020, compliance with COVID workflow protocols in practice was found to be 95% (121 of 127 cases).

Healthcare Worker Infection Rate

The number of confirmed COVID-19 test–positive cases from within our perioperative department over the period of March 18, 2020, to June 10, 2020, totaled 7. By discipline, test-positive cases were within anesthesia (n = 1), surgery (n = 4), and perioperative nursing (n = 2). Given the average number of perioperative staff members in rotation at any given point, this reflects an estimated infection rate of cumulative infection rate of 0.9% over the entire observation period. Although daily test positivity rate for perioperative staff was not provided by the Employee Occupational Health Service because of Health Insurance Portability and Accountability Act concerns, it is expected to have been at 0% during the observation period, except for the specific days when the 7 cases were reported. For reference, the daily community test positivity rates during this observation period were consistently higher, ranging from 4% to 34%, whereas the 7-day weighted average test positivity rate ranged from 4.2% to 16.6%.³⁵

DISCUSSION

The perceived value of our JIT in situ simulation training program, for the implementation of new COVID-19 perioperative workflows, improved provider perceptions of knowledge and comfort in adapting frequently changing workflows and the belief that training would impact future clinical practice. Participants appreciated the fact that training was timely and focused on reducing risk of viral exposure and transmission to HCWs. Immediate adoption and high compliance with new workflow processes in practice demonstrated behavioral change of a significant scale and was associated with a low HCW COVID-19 test–positive infection rate.

Improvements in knowledge, skill, and comfort in performing clinical skills have been demonstrated previously through simulation training.³⁶ However, the rightward shifts observed in our study, although modest in size, were observed across all disciplines within the perioperative team, and in particular from OR nursing staff. This may be considered a significant finding in the context of both the pandemic and the brief time available to achieve team training goals.³⁷ The concept of JIT training focuses on the delivery of a specific task, at a required time. It differs from traditional teaching methods in that it does not aim to deliver training on entirely new topics; rather, it acts to complement preexisting knowledge. Previous studies report on the use of JIT and simulation training¹⁰; however, to our knowledge, JIT in situ simulation training has not been described in the context of a public health crisis, in particular for the implementation of changes in workflow processes. In light of the urgency of the pandemic and reported local risk of community viral exposure, we chose rapid implementation of a JIT in situ simulation training program to address the gaps in knowledge of our perioperative staff. The design, content, organization, and delivery of this program were tailored specifically to facilitate training of a vast number of perioperative staff members in a timely fashion, to perform a specific set of tasks in the context of COVID-19 patients.

In keeping with JIT training framework, simulation content was centered on newly produced guidelines that perioperative staff would be expected to perform. Simulations were site-specific, relevant, and took place within days of the first suspected perioperative COVID-19 case. In the context of COVID-19, the personal risk to the provider outlined in the intubate COVID registry study reported 10.7% of HCWs meeting the primary end point of new laboratory-confirmed COVID-19 (3.1%), new COVID-19 symptoms requiring self-isolation (8.4%), or hospitalization admission with more than 1 symptoms (0.1%).⁵ We speculate that there were clear, unifying drivers for change among perioperative staff, including a “hunger for information” and increased anxiety about a lack of knowledge on risk of viral exposure and transmission. This may have resulted in a bias toward lower scores before simulation.

In situ simulation serves as both an effective educational tool and a medium for implementing changes and identifying hazards. Our delivery of training through in situ simulation provided local context for our learners as well as our faculty. They were able to practice simulations in their own place of work, among their regular teams, and observe how protocols would take place during a real-life case. This enabled the detection of issues within our protocols and encouraged providers to express feedback and reaffirm their involvement in the change process.

Although our entire organization was responsible for the care of nearly 900 COVID-19 patients, our internal COVID-19 test–positive rates among perioperative staff remained lower compared with national and international reported averages, that is, 3% reported by an international registry of HCWs after tracheal intubation.^5,35 Although we appreciate that our findings do not prove causality, our training may have aided in reducing our transmission rates.

Limitations

Communication to staff postsimulation training was challenging. Given the rapidly evolving understanding of the disease, protocols and workflow changes were common during this time and therefore reflected in the following training sessions. We created online resources to consistently communicate changes with those who had already received training. These resources are freely accessible for providers wishing to implement this format of training in their own organization.³⁸

We did not track how long it had been since each respondent had participated in the simulation training exercise. Therefore, we cannot assess for recall bias risk on an individual level. Survey deployment was delayed by a few days after implementation of the training program, which may have hampered the response rate. Although our response rate was 25%, we believe that the overall response from 110 participants from multiple disciplines, training levels, and experience levels provided meaningful results. Furthermore, given the volume of COVID-19 cases that our staff were treating and the reported risk of COVID fatigue,^39–41 the study group elected to be conservative in using e-mail reminders.

Future Directions and Lessons Learned

Given the perceived value and demonstrated impact of this training method, materials have been adapted to enable ongoing training as part of a hospital network-wide simulation training program and approved by Controlled Risk Insurance Company, Ltd (CRICO), which focuses on perioperative risk management. Furthermore, revised concise JIT in situ simulation training areas were developed to help “refresh” skills for all perioperative staff members, in anticipation of the future surges of COVID-19.

Conclusions

These results demonstrate the value and impact of JIT in situ simulation training as a preparedness measure for the perioperative care of COVID-19 patients and as a valuable training method during a time of a public health crisis.

Supplementary Material

sih-18-090-s001.docx ^{(191.9KB, docx)}

sih-18-090-s002.docx ^{(26.5KB, docx)}

sih-18-090-s003.doc ^{(32.5KB, doc)}