In Situ Simulation to Mitigate Threats to Participation in a Multicenter Clinical Trial in High Acuity, Low Frequency Setting

Abstract

Introduction

Multicenter clinical trials of high acuity, low frequency emergencies are expensive and resource-intensive. Current standards for trial preparation have significant limitations. Our objective is to describe our use of in situ simulation (ISS) to identify and mitigate threats to enrollment, protocol adherence, and patient safety in preparation for a multicenter clinical trial of anti-epileptics for status epilepticus in the emergency department (ED).

Methods

This is a descriptive study of ISS conducted in the ED at a free-standing, academic pediatric institution. We designed two scenarios, one for an eligible and ineligible patient, to allow care teams to complete all study procedures. All study training was completed prior to the first ISS. Participants included physicians, nurses, patient care assistants, paramedics, research coordinators, and pharmacists. Ten-minute simulations were followed by 10-minute debriefings, led by trained facilitators using a standard template. Data on threats to enrollment, protocol adherence, and patient safety were recorded. Mitigation strategies were developed by the study team and local experts in ISS.

Results

Ten of 18 planned simulations were conducted. Seven of 10 completed simulations were study eligible patients, with 73 total participants. Nine threats to enrollment and 5 to protocol adherence were identified. Five of 14 were also threats to patient safety. Mitigation strategies included creation of decision aid tools, targeted education during debriefings, adapting study material for use and revision of status epilepticus treatment algorithm.

Conclusion

The addition of ISS to standard preparation for a multicenter clinical trial facilitated the identification and mitigation of threats to study participation and patient safety.

INTRODUCTION

Clinical trials are expensive and resource intensive, requiring significant preparation to optimize study validity and limit risks to patient care.¹ In pediatrics, many critically needed trials are not performed due to concerns about feasibility and lack of funding for investigational drugs and medical devices. Trials of diagnostic and therapeutic interventions for critically ill and injured children are especially problematic, as pediatric emergencies are infrequent, irregular, and unpredictable.² Multiple centers are often required to achieve an adequate sample size, increasing the likelihood of inconsistencies in study conduct and protocol deviations. Participation in a clinical trial in an emergency setting may involve novel interventions and procedures for a given center, increasing the risk of suboptimal care in the emergent setting. When a clinical trial is funded to investigate an intervention for a pediatric emergency, extraordinary and effective preparation are needed to optimize enrollment, protocol adherence, and patient safety.

High acuity, low frequency emergencies present unique and formidable challenges to reliable, high quality care.^3–6 There is an analogous problem for clinical trials of high acuity, low frequency emergencies. The infrequent presentation of eligible patients, the narrow timeframe to conduct study procedures, and the emergent nature of the situation may increase the likelihood of missing eligible patients and important protocol deviations that might also lead to patient harm. Aspects of standard preparatory and monitoring processes for clinical trials, e.g., institutional review board review and data safety monitoring, are intended to optimize enrollment, protocol adherence, and patient safety. However, when an eligible patient presents, study training may have been so remote that, even if it was of the highest quality, providers responsible for enrollment and protocol adherence may struggle to recall the study itself, relevant eligibility criteria, and critical aspects of the study protocol. Moreover, again because of the relative infrequency of both pediatric emergencies and relevant clinical trial experience, many of the most important threats to trial participation may be latent, with important nuances in a given center participating in a multi-center trial.

In situ simulation has been demonstrated to be an important and effective tool in medical education, systems integration, and clinical research.^3,7,8 In situ simulation has specific advantages over other approaches to education, clinical research, and quality improvement, in particular increasing the frequency of high acuity, low frequency emergencies.^4–6,9,10 As an adjunct to the standard educational and preparatory activities, in situ simulation may be a uniquely powerful approach to enhance enrollment and protocol adherence and improve patient safety by both accurately characterizing latent threats in a given center and acting as a vehicle to directly mitigate the same threats. In situ simulation could also be used after enrollment begins to maintain pre-trial levels of knowledge and awareness. Figure 1 is a flow diagram that illustrates this framework of using in situ simulation to enhance the preparatory process and systems integration for a clinical trial.

FIGURE 1.

Simulation to Enhance Preparation and System Integreation for a Clinical Trial

(1) Primary goal of simulation is systems integration of the clinical trial

(2) Allows the site to respond to unexpected or unanticipated threats

Note: If there is not an established simulation program at the study site, this framework will require more work

The objective of the current report is to describe our use of in situ simulation to augment the standard pre-trial preparatory activities before our site’s participation in a multicenter clinical trial. To our knowledge, there are no published reports of the use of in situ simulation to enhance preparation in a clinical trial. We specifically sought to use simulation to allow for identification of latent threats – threats that were not identified or were poorly characterized following completion of the standard study training. We hypothesized that in situ simulation would uniquely enhance the identification and mitigation of threats to enrollment, protocol adherence, and patient safety that were insufficiently addressed through the standard educational efforts.

METHODS

Setting and Simulation Participants

The setting for both the clinical trial and the in situ simulations was the resuscitation area of a pediatric ED at a free-standing, academic, pediatric institution. Of the 90,000 combined annual visits, 4,100 (4.6%) patients are evaluated for medical (2,000) and traumatic (2,100) emergencies in the resuscitation area. Designated care teams evaluate and manage all patients with medical emergencies, led by a physician board-certified in Pediatric Emergency Medicine (PEM) or a PEM fellow. Team members also include nursing team leaders, bedside and medication nurses, a respiratory therapist, a pediatric resident, and a paramedic or patient care assistant. The participants in the study-related simulations were the providers staffing the pediatric ED and designated to respond to the resuscitation area. The clinical research coordinators (CRCs) also participated in these simulations. Before commencement of the study-related simulations, all PEM attendings, fellows and CRCs received formal study training. The study protocol for Established Status Epilepticus Treatment Trial (ESETT) was approved by our Institutional Review Board (IRB). The protocol for the current study was submitted to the IRB as an amendment to the ESETT study and was approved.

The Medical Resuscitation Committee

The medical resuscitation committee (MRC) at our institution is responsible for peer review, quality assurance/improvement, and clinical research for patients evaluated for medical emergencies in the resuscitation area of our institution’s pediatric ED. A central aspect of the MRC’s work is oversight of an in situ simulation program for the ED resuscitation area, which was developed to identify latent safety threats and enhance quality assurance and educational activities for low frequency, high acuity medical emergencies. All MRC simulation facilitators have been formally trained through the simulation center. The MRC schedules an average of 7 simulations per month; in 2017, 38 of 83 simulations were completed (46%). Standard cancellation criteria include high ED volume or patient acuity, a patient being actively evaluated in the resuscitation area, and inadequate ED staffing.

Clinical Trial

The Established Status Epilepticus Treatment Trial (ESETT) is a multicenter, randomized, blinded, comparative effectiveness study of fosphenytoin, valproic acid, or levetiracetam in the emergency department (ED) treatment of pediatric and adult patients with benzodiazepine-refractory status epilepticus (SE; sponsored by National Institute of Neurological Disorders and Stroke 1 U01 NS088034 01, ClinialTrials.gov ID NCT01960075).^11–13 The goals of the study, are to determine the effectiveness and safety of these drugs. There are 56 sites in total that are enrolling for ESETT – 43 adult centers from the Neurological Emergencies Treatment Trials Network (NETT) and 15 pediatric centers from the Pediatric Emergency Care Applied Research Network (PECARN). ESETT is an example of a clinical trial that involves prospective enrollment of a patient with a high acuity, low frequency medical emergency, as the annual incidence of SE has been estimated to be 41 to 61 per 100,000 patients in all age groups.¹⁴ Thus, at any one center, a precious few eligible patients can be expected to present each year. For example, in our pediatric ED during 2015 (prior to study commencement), only 20 patients potentially eligible for ESETT were evaluated.

The primary objective of ESETT is to determine the most effective and/or the least effective treatment of benzodiazepine-refractory SE.^11–13 There are three active treatment arms: fosphenytoin, levetiracetam, and valproic acid. The primary outcome is clinical cessation of SE, determined by the absence of clinically apparent seizures and improving responsiveness, at 60 minutes after the start of study medication infusion, without the use of additional antiepileptic drugs. See Table 1 for inclusion criteria. ESETT enrollment occurs under exception from informed consent rules (EFIC), due to the emergent and life-threatening nature of SE.

Table 1.

Inclusion Criteria for Enrollment into ESETT

Patients greater than or equal to 2 years of age

Witnessed to have a seizure within the past 5–30 minutes

Received an adequate dose of benzodiazepine within the past 5–30 minutes: Patients < 40 kg: lorazepam IV 0.1 mg/kg, midazolam IM 0.3 mg/kg or IV 0.2 mg/kg, diazepam IV or PR 0.3 mg/kg Patients > 40 kg: lorazepam IV 4 mg, midazolam IV or IM 10 mg, diazepam IV or PR 10 mg

Continues to have clinically apparent seizures in the ED

IV = intravenous; IM = intramuscular, PR = per rectum

Trial Preparation

All site investigators received formal, in-person training at an investigator meeting prior to study commencement. Templates of study educational materials were supplied by the lead investigative team, but dissemination of these materials and the site’s approach to pre-study education and training was left to the discretion of the site investigator. All CRCs responsible for ED patient enrollment processes were trained extensively on study procedures, both in small group and one-on-one settings. Mandatory education was assigned to all ED nursing staff, and successful completion of a post-test was required prior to the first patient enrollment. Additionally, a separate presentation was given to nurse team leaders by the site investigator, since this group exclusively fills the nurse team leader role in the resuscitation area and functions as co-team leaders with the PEM attending. Study training for the PEM attendings and fellows included: (1) a presentation at our divisional staff meeting, (2) review of the same presentation via an all-staff email, and (3) individual, in-person training with the site investigator. All PEM attendings and fellows completed at least one of these activities. Our pediatric ED pharmacist participated in the design of the study-related educational efforts, including the ordering and administration of the study drug.

Prior to the in situ simulations, we developed a written document to assist providers in determining patient eligibility and in adhering to the study protocol. We adapted standard study materials and placed the two-page document in a binder located in each trauma bay computer cart. The two-page document (see Text Document, Supplemental Digital Content 1, which shows instructions for enrollment) included: (1) study inclusion and exclusion criteria, (2) a chart for adequate dosing of benzodiazepines, (3) a chart for dosing of the study medication, (4) the definition of complications related to the study medication, i.e., hypotension, (5) contact information for the site investigators, (6) instructions on how to administer the study medication, and (7) a diagram for ordering the study medication in the electronic medical record (EMR). We also loaded a dosing calculator for determing medication-related inclusion crtieria on all trauma bay computers. Both the binder and dosing calculator were included/tested during the in situ simulations, with refinements made based on findings from the simulations.

Integration of the Trial into Clinical Care

Treatment algorithms are used in our resuscitation area to help guide physicians in the management of certain high risk, low frequency conditions. New publicized guidelines about the acute management of the study-related condition, SE, and findings from the simulations were used to inform the revision of our department’s treatment algorithm for SE. A highlighted box was embedded in the algorithm at the point where enrollment in the trial should be considered if the patient met eligibility criteria (e.g. if the seizure was refractory to initial benzodiazepines).

Study-related Simulation Scenarios

The MRC and site investigators designed two study-related scenarios (see Text Document, Supplemental Digital Content 2, which shows the simulation scenario), one for an eligible patient and one for an ineligible patient. To ensure the most critical study steps would be attempted by providers and observed by study staff, the enrollment-eligible scenario was designed to encourage ED care providers to complete all study-related procedures through study drug administration (Table 2). The scenarios were developed using the MRC’s standard process, including being edited and vetted by the full committee. The scenarios were not pilot tested, however, as the scenarios were adapted from a SE scenario used previously by the committee.

Table 2.

ESETT Study Procedures to be Completed During In Situ Simulation

Study Procedures

Clinical research coordinator determines eligibility with assistance from physician team leader Physician team leader/resident physician orders ESETT study medication in the electronic medical record Nurse obtains the study medication from the Pyxis™ (Becton, Dickinson and Company, Franklin Lakes, NJ) Nurse calculates dose, performs an independent double-check, spikes and primes medication bottle Bedside nurse infuses the study medication over 10 minutes at an appropriate weight-based rate Bedside nurse obtains frequent vital signs (at least every 5 minutes) Care team address potential adverse effects, especially hypotension, by decreasing the rate of infusion

The study-related simulations were scheduled for either 9:00am to 12:30pm or at 4:30pm, to target both first- and second-shift providers, respectively, as these were the shifts where CRCs would be present to enroll. Each simulation session was limited to 20 minutes- 10 minutes for the simulation and 10 minutes for debriefing. All participants received a standard pre-brief before the start of each simulation which included orientation to the environment and the simulator as well as a scripted introduction to the simulation scenario. A site investigator, lead CRC, MRC physician and nursing facilitator led the team debriefing. The debriefings were focused on discussing ESETT study procedures, emphasizing threats identified in the current and any previous simulations. Specific efforts to mitigate those threats, e.g., modifications to the 2-page document in the study binder, were emphasized. Medical management of SE was discussed when relevant to both the ESETT study procedures and our site’s treatment protocol for SE (see below).

Data Collection

We created a standard debriefing template (see Text Document, Supplemental Digital Content 3, which is an example of the debriefing template) for the study-related simulations, based on the template used by the MRC for its in situ program. The debriefing template highlighted threats identified during previous simulations, ongoing mitigation efforts, and key points relevant to our site’s SE algorithm. The two MRC facilitators (physician and nurse) present at each study-related simulation were responsible for collecting relevant data using the template, based on direct observation only. The MRC nurse was the primary documenter, recording potential threats identified in the simulation or debriefing. The MRC physician facilitator then reviewed the debriefing record for completeness. These records were disseminated to the MRC and site investigators, who met regularly to review findings from the simulations and plan subsequent simulations/mitigation efforts.

Outcomes

The main outcomes of the current report were: (1) the threats to enrollment, protocol adherence, and patient safety identified during the simulations and (2) our efforts to mitigate each threat. Threats were defined as latent, system-based threats that might be actualized for a given patient at any time and contribute to errors in study enrollment, protocol adherence, or patient safety.¹⁵ Mitigation strategies were developed, implemented, and refined iteratively following each simulation. A given threat could be identified during multiple simulations, but the intent was to eliminate the threat entirely before the next simulation (latent threats) or decrease its risk through dissemination to ED care providers (knowledge deficit).

Following completion of the simulations, the MRC and site investigators reviewed all simulation notes to determine study outcomes by consensus. A secondary goal of our work was to revise our existing algorithm for the treatment of SE before study initiation, leveraging our findings from the study-related simulations and recently published guidelines on SE treatment. The new guidelines and the ESETT protocol deviated significantly from our existing guideline so we anticipated that this would be a threat to enrollment. We revised our existing algorithm to optimize study participation and patient safety, as use of the algorithm was standard practice prior to study commencement.

Analysis

We tabulated both outcomes and report each descriptively.

RESULTS

Simulations

From October to December 2015 (study commencement in late December 2015), we completed 10 of 18 scheduled study-related simulations (56%). Seven of the completed simulations were for eligible patients and 3 for ineligible. Reasons for cancellation were ED staffing issues, high ED volume, and the presence of an actual patient in the resuscitation area. A total of 73 individual providers (43 nurses, 8 physicians, 3 patient care assistants, 3 paramedics and 16 CRCs) participated in the 7 study-related simulations, representing approximately 25% of all ED providers at our site.

Main Outcomes

The definitions of each threat to enrollment, protocol adherence, and patient safety as well as our mitigation strategies in response to each threat are presented in Tables 3 and 4, respectively. Mitigation strategies were implemented immediately after a threat was identified and those strategies were reinforced in subsequent simulations. Table 5 lists the specific threats identified in each simulation as well as the total number of threats identified.

Table 3.

Threats to Enrollment Identified During 7 In Situ Simulations in a Pediatric Emergency Department Prior to Study Commencement

Name of Threat & Explanation of Threat	Root Cause: The core problem with the standard study educational efforts/protocol	Mitigation: Points emphasized in subsequent simulations, other efforts made to re-educate providers, or new/modifications to study tools
Eligibility: Who was ultimately responsible for determining study eligibility	Lack of clarity in the manual of procedures and inadequacy of standard study education and training	Subsequent simulation/debriefing: We clarified that the CRC was ultimately responsible for determining eligibility, with assistance from the physician team leader. All CRCs attended at least one subsequent in situ simulation. Eligibility were reinforced to all simulation participants.
		Outside simulation/debriefing: The lead CRC reviewed threats identified in the simulations with all CRCs during biweekly meetings
EFIC: Confusion about EFIC	ESETT was the first study with EFIC in our pediatric ED and inadequacy of standard study education and training	Subsequent simulation/debriefing: EFIC explicitly discussed.
		Outside simulation/debriefing: We placed in the trauma bay a template for ED providers to explain EFIC to the family. We also included a list of patients who had opted out before study commencement. The availability of both was also mentioned in subsequent debriefings. The lead CRC reviewed threats identified in the simulations with all CRCs during biweekly meetings.
“Study”: Commonly used phrases “study medication” or “ESETT study” - The use of the term “study” could cause parental concern about enrollment and their child receiving experimental treatment.	Lack of comfort and unfamiliarity with clinical trials in the resuscitation area during a medical emergency.	Subsequent simulation/debriefing: In subsequent simulations, we encouraged use of the term “ESETT protocol” and avoidance of “study.”
Intubation: How tracheal intubation affects eligibility for enrollment, before or after study medication infusion	Sedatives used for RSI have anti-epileptic properties, affecting eligibility, and the neuromuscular blockers affect determination of primary outcome.	Subsequent simulation/debriefing: In subsequent simulation debriefings, we clarified that intubation prior to study medication infusion excludes the patient from enrollment but that that potential need for intubation should not preclude enrollment. All CRCs attended at least one subsequent in situ simulation and practiced determining eligibility.
		Outside simulation/debriefing: The lead CRC reviewed threats identified in the simulations with all CRCs during biweekly meetings.
Benzo Types: Which benzodiazepines could be administered before enrollment	Many benzodiazepines are used to treat seizures and inadequacy of standard study education and training	Subsequent simulation/debriefing: We emphasized that the eligible benzodiazepines are midazolam, lorazepam, and diazepam. All CRCs attended at least one subsequent in situ simulation and practiced determining eligibility.
		Outside simulation/debriefing: We included eligibility criteria in the study folder. The lead CRC reviewed threats identified in the simulations with all CRCs during biweekly meetings.
Benzo Timing: Timing of benzodiazepine administration before enrollment	Eligibility criteria for the timing of benzodiazepine administration were inherently confusing (a specific time range rather than a single time point) and inadequacy of standard study education and training.	Subsequent simulation/debriefing: We emphasized that the last dose of benzodiazepine had to have been administered at least 5 minutes but not longer than 30 minutes before study enrollment. All CRCs attended at least one subsequent in situ simulation and practiced determining eligibility.
		Outside simulation/debriefing: We included eligibility criteria in the study folder. The lead CRC reviewed threats identified in the simulations with all CRCs during biweekly meetings.
Adequate Dose: Adequacy of benzodiazepine dosing before enrollment	Eligibility criteria inherently confusing (different doses for each of three eligible benzodiazepines, with additional differences based on route) and inadequacy of standard study education and training.	Subsequent simulation/debriefing: We created and installed a benzodiazepine dose calculator on the computers in our resuscitation area and had participating providers practice using the dose calculator during simulation debriefings. All CRCs attended at least one subsequent in situ simulation and practiced determining eligibility.
		Outside simulation/debriefing: We included eligibility information related to benzodiazepine dosing available in the study folder, which contained a chart to help providers determine adequate dosing. The lead CRC reviewed threats identified in the simulations with all CRCs during biweekly meetings.
Prehospital Benzos: Who could administer benzodiazepines before enrollment	Eligibility criteria inherently confusing (benzodiazepines could be given by caregivers at home or by pre-hospital providers prior to arrival at our ED) and inadequacy of standard study education and training.	Subsequent simulation/debriefing: In subsequent simulation debriefings, we clarified that all benzodiazepines administered within the past 30 minutes can be added up in the cumulative dosing but one of them has to be given by a medical provider (i.e. pre-hospital provider or in the ED). Benzodiazepines given by parents can count towards adequate dosing.
SE Algorithm: Existing site-specific algorithm for treatment of status epilepticus recommended only fosphenytoin as the second-line agent of choice*	Fosphenytoin is the historical standard second-line medication to treat status epilepticus in our ED, and providers frequently use the status epilepticus algorithm as a decision aid. The algorithm did not reflect the clinical equipoise that existed amongst the second-line medications.	Subsequent simulation/debriefing: We encouraged providers to use a revised version of the status epilepticus algorithm during the simulation and highlighted the revisions during the debriefing.
		Outside simulation/debriefing: We revised our status epilepticus algorithm to incorporate the ESETT study as well as the most recent treatment guidelines. The revised guideline was presented at division staff meeting.

^*threat to patient safety

ESETT = Established Status Epilepticus Treatment Trial, CRC = clinical research coordinator, EFIC = informed consent, ED = emergency department, RSI = rapid sequence intubation

Table 4.

Threats to Protocol Adherence Identified During 7 In Situ Simulations in a Pediatric Emergency Department Prior to Study Commencement

Threat	Root Cause	Mitigation
EMR: Issues related to ordering of the study drug in the EMR	No existing order for the study drug.	Subsequent simulation/debriefing: We reviewed order entry with the nurse team leader. For several debriefings, the MRC nurse facilitator practiced ordering the study drug with the nurse and physician team leaders.
		Outside simulation/debriefing: We worked with our investigational pharmacy to create an order for the study medication in our EMR. We also created a laminated instruction sheet with pictures to help providers order the study drug and placed it in the study folder. We presented the order entry instructions at a meeting of the nurse team leaders.
Med Preparation: Method of medication preparation different than standard practice*	Standard practice at our institution for medication administration is to draw up the dose into a syringe and administer via a pump. The study protocol requires spiking the medication bottle directly.	Subsequent simulation/debriefing: We reviewed the approach to study drug preparation. For several debriefings, the MRC nurse facilitator practiced medication preparation with the participating nurses.
		Outside simulation/debriefing: In collaboration with our ED pharmacist and nurse team leaders, we standardized the medication administration to be run directly from the bottle in which it was originally packaged, as opposed to being drawn up into a syringe. We presented the approach to study drug preparation at a meeting of the nurse team leaders.
Med Administration: Method of medication administration different than standard practice*	Standard practice in our ED is to flush tubing or administer chase fluid after infusion of a medication is complete, to ensure the entire volume is administered.	Outside simulation/debriefing: Through mock infusions, we determined that it was not necessary to flush the tubing after the infusion of the study drug was completed because extra medication was built into the calculated dose.
Co-administration: Safety of co-administration of study drug with normal saline *	Lack of clarity in the ESETT Manual of Procedures	Outside simulation/debriefing: In collaboration with our ED pharmacist, we determined that it was safe to infuse the study drug with normal saline. We emphasized this point in subsequent simulation debriefings.
Hypotension: Risk of not detecting life-threatening hypotension and cardiac arrhythmia (key secondary outcomes)*	There are many definitions of hypotension that vary by patient age and size. For patients in status epilepticus, blood pressure monitoring may be infrequent, and awareness of heart rhythm may be low.	Subsequent simulation/debriefing: We reinforced the potential adverse effects of study drug infusion and the importance of frequent vital signs for data collection. We highlighted that there was a reference for hypotension in the study folder.

^*threat to patient safety

EMR = electronic medical record, ESETT = Established Status Epilepticus Treatment Trial, MRC = medical resuscitation committee, EFIC = informed consent, ED = emergency department

Table 5.

Specific Threats and Number of Threats Identified in Each In Situ Simulation Prior to Study Commencement

Simulation	Threats to Enrollment	Threats to Protocol Adherence	Number of Threats Identified
1st	Eligibility	EMR	6
	Benzo Types
	Benzo Timing
	Adequate Dose
	Prehospital Benzos
2nd	Eligibility	EMR	12
	EFIC	Med Preparation*
	Benzo Types	Med Administration*
	Benzo Timing	Co-administration*
	Adequate Dose	Hypotension*
	Prehospital Benzos
	SE Algorithm*
3rd	SE Algorithm*	Med Preparation*	4
		Med Administration*
		Co-administration*
4th	Benzo Types	Hypotension*	6
	Benzo Timing
	Adequate Dose
	Prehospital Benzos
	SE Algorithm*
5th	“Study”	Hypotension*	3
	Intubation
6th	Benzo Types	Med Preparation*	7
	Benzo Timing	Med Administration*
	Adequate Dose	Hypotension*
	Prehospital Benzos
7th	None	EMR	3
		Med Administration*
		Hypotension*

^*threat to patient safety

The most common threat to enrollment was provider confusion about the eligibility criteria, particularly the dosing and timing of benzodiazepines. This threat was identified in the first in situ simulation and in 4 of the 7 total simulations conducted. The protocol lists three potential benzodiazepines in the eligibility criteria, each with a different adequate dose. To mitigate this risk, we created a benzodiazepine dosing chart to aid providers in determining whether the patient received an adequate dose. This chart was placed in the resuscitation area where enrollments would occur. We also installed a benzodiazepine dosing calculator, provided by the study team, on all the computers in our resuscitation area used by the documenting nurse. In subsequent debriefings, we emphasized that at least one benzodiazepine must be administered by a medical provider, which includes pre-hospital providers. Finally, we stressed that the timing of benzodiazepine administration is important, in that it has to be more than 5 minutes but less than 30 minutes from enrollment.

During the simulations, care providers were concerned about the consent process which is a threat to enrollment. This threat was identified in the second simulation conducted. This was the first time our department participated in a study using EFIC that required a bedside intervention. During simulations, providers were educated about EFIC and the necessity of EFIC for a pediatric emergency trial. Providers and research coordinators expressed apprehensions about approaching families whose child was enrolled in a study with EFIC. This led to the creation of a short written template to help providers introduce the study to the family prior to the CRC going in to speak to the family and obtain informed consent. Also, we encouraged providers to use phrases like “ESETT protocol” as opposed to using words like “study”, “enrollment” or “investigational” in front of families until the study has been introduced.

The most common threat to protocol adherence is the detection of and response to hypotension. This threat was identified in the second simulation conducted and in 5 of the 7 total simulations. Definitions for hypotension differ by age which lead to provider confusion. Also, providers were unsure of how to treat hypotension in the setting of an unknown study medication. We created a chart with age-specific thresholds of hypotension and placed it in the study folder in the resuscitation area. We also provided targeted education and clarification during simulation debriefings that providers should slow the rate of infusion of the study medication by half if hypotension is present.

Common threats to protocol adherence were confusion about the preparation and administration of the study medication. These threats were identified in the second simulation conducted and in 4 of the 7 total simulations. The ESETT Manual of Operations specifies to directly administer the study medication from the study vial and not to flush the intravenous (IV) tubing, because extra medication in the line was built into the calculated dose. This deviates from our standard approach in the resuscitation area, which involves first drawing the exact dose from the vial and then administering it using an infusion pump. We also identified threats related to calibrating the infusion pump (to administer the correct volume over 10 minutes) and to flushing the IV tubing after completion of medication (standard practice in our ED). During subsequent debriefings, these issues were emphasized and explicitly discussed with the participating ED nurses. The lead CRC and MRC nurse facilitator frequently spent additional time practicing medication preparation and administration with the participating nurses.

DISCUSSION

Participation in pre-study education and training activities is a ubiquitous and common experience in academic medicine. The typical approaches to trial preparation include didactic, slide-based presentations, email of the presentation slides for self-review, online learning modules, and/or one-to-one training with the principal investigator. Training may also include periodic reminders via email and discussions at staff meetings or fliers posted in the study setting, both highlighting key issues with patient enrollments. There is published evidence demonstrating that inadequate approaches to RCT design and execution may increase bias in treatment effects¹⁶, but the relevant literature is sparse on the effectiveness and specific limitations of the common approaches to study preparation.¹⁷ The United States Food and Drug Administration (FDA) formed the Clinical Trials Transformation Initiative (CTTI) to develop and drive adoption of practices that will increase the quality and efficiency of clinical trials.¹⁸ CTTI provide tools to investigators for improvement in recruitment, informed consent process, and study design, but there is no mention of the use of in situ simulation as a tool for improving clinical trials. Our experience in an academic, pediatric setting is that study preparation efforts are often inadequate. Compounding the issues is the rarity of many pediatric illnesses and injuries, especially medical emergencies, which further limits opportunities for providers to practice enrollment and study procedures, especially in high-volume settings where the exposure of individual providers may be rare.

In the current study, we were able to practice study enrollment, identifying specific threats ourselves, often across multiple simulations. We were then able to design mitigation strategies for specific threats and test them before study commencement. Subsequent simulations were also iterative and cumulative - we not only focused on threats that were identified during that particular simulation but also reiterated threats identified in previous simulations. Second, analogous to systems integration, we were able to integrate the study protocol with our existing treatment algorithm, possibly improving both our care of these patients and our participation in the clinical trial. Third, ED providers and research staff had opportunities to practice enrollment, potentially increasing study knowledge, comfort, and awareness.

The addition of in situ simulation to site preparation for participation in a multicenter clinical trial enhanced the identification and mitigation of threats to enrollment, protocol adherence, and patient safety that were inadequately addressed by the standard approach. In situ simulation specifically allowed the efficient and iterative “testing” of providers and environment, permitting the identification and characterization of various threats while acting as both a mechanism to educate participants and refine study tools designed to improve enrollment and protocol adherence. We completed 10 simulations over a three-month period, whereas over the first 25 months of trial participation, our site saw approximately one eligible patient every two months. Moreover, rather than waiting until errors were made for actual patient enrollments, those same errors could be made during simulations and efforts made to mitigate the risk for actual patients. Given their infrequency and the challenges of data collection, relying on actual patient encounters to identify latent threats is likely inefficient, inaccurate, and unsafe. Dissemination of the results of the simulations to all ED providers, emphasizing the specific threats identified and efforts to mitigate them, was a more effective approach than general updates that lacked specificity to the actual system and the threats most likely to cause errors.

Limitations

This was a descriptive report of our initial efforts to incorporate simulation into the preparatory education and training before participation in a multicenter, therapeutic trial of a relatively uncommon pediatric emergency. As such, we cannot determine the independent effects of in situ simulation on our preparatory process, in particular in comparison to a standard approach alone. It is possible that other forms of simulation or non-simulation-based activities might work as well and with lower costs. However, in situ simulation has significant advantages over other forms of simulation for clinical research, education, and systems integration, and it is reasonable to assume that in situ simulation has analogous advantages for clinical trial preparation. Second, as this was our first use of in situ simulation for trial preparation and our primary objective was optimizing preparation rather than studying the effects of simulation, our approach lacked rigor, especially in data collection. Future studies should focus on the specific contributions of in situ simulation compared with the standard approach. Consideration should be given to the costs associated, especially in centers without existing simulation programs.

Future Directions

Following study commencement, we planned to include regular study-related simulations in our in situ simulation program, both because improving the treatment of SE was a primary goal of this program and to maintain staff awareness of and familiarity with study procedures to facilitate ongoing mitigation of threats. Between January 2016 and June 2017, we completed 15 of thirty planned simulations (50%). We continued to perform structured debriefs and collect data on threats and key protocol deviations. The following are the threats identified in these subsequent simulations with tabulated frequencies in parentheses: med preparation (5), adequate dosing (4), “study” (3), EFIC (2), EMR (2), med administration (2), hypotension (2) and intubation (1). To further address these threats, most recently we have utilized in situ simulation to train the physician team leader to determine eligibility and complete study procedures and data collection, as CRCs are not available on 3^rd (overnight) shifts. In addition, we continue to update and re-educate the staff with presentations every 6 months.

One of the most significant limitations of our initial study is the lack of data supporting the independent benefit of in situ simulation. To partially address this limitation, which is also an obvious gap in the relevant literature, we plan to complete a second study comparing eligible patient capture and protocol deviations between our site and all other centers participating in ESETT, stratifying by pediatric versus adult/general ED. In discussions with the lead investigators, only one other site used simulation to prepare for the trial, and this effort was neither structured nor ongoing.

In addition, our use of simulation for trial preparation demonstrates that a rigorous simulation program can be an asset to principal investigators designing complex clinical trials. We envision simulation playing a role in trial preparation in future RCTs that enroll in our ED, especially those with protocols involving conditions that are time-sensitive, high-acuity and low-frequency.

Conclusion

The addition of in situ simulation to the standard approach to education and training before participation in a multicenter clinical trial facilitated the identification and mitigation of threats to study participation and patient safety. In situ simulation allowed the mitigation of these latent threats over a shortened time period that otherwise would have required errors during actual patient encounters to identify.