Abstract
Background
In situ simulation (ISS) has been reported as an innovative method to identify and mitigate latent safety threats (LSTs) in healthcare. Little is known about the current utilisation of ISS across academic simulation programmes.
Objective
This study aims to describe the use of ISS to identify LST across paediatric academic simulation programmes.
Methods
A 25-question cross-sectional survey was conducted at two simulation meetings in January 2014 to recruit leaders from paediatric simulation programmes. The total eligible sample was 82 individuals representing 48 distinct academic medical centres. The 25 survey questions were created to describe the constructs of: (1) utilisation of ISS (location, participants, cancellations and coordination) and (2) the outcomes of ISS (detection of and response to safety threats). Descriptive statistics were carried out using SPSS V.21.0 (IBM Corp released 2012).
Results
The response rate was 68% (56/82), representing 79% (38/48) of the eligible academic medical centres. The majority of respondents (52/56) reported that their programmes utilised ISS. ISS was most commonly conducted in acute care settings. Almost all respondents (48/52) detected an LST during ISS. More than half of the respondents (28/52) utilised a formal reporting process after ISS sessions to feedback the LST to other individuals within their institution. 23% (12/52) of respondents reported the detection of a serious LST in ISS that was not resolved and subsequently led to a safety event during real patient care.
Conclusions
The use of ISS to identify and mitigate LST is common in this cross-sectional survey of paediatric simulation programmes. Diverse processes and organisational structures exist for reporting and mitigating LSTs identified in ISS. A more integrated and systematic approach to ISS and LST could help ensure the mitigation of LSTs before they impact on patients.
Keywords: simulation, patient safety, safety management, quality improvement, emergency department
Introduction
In situ simulations (ISS) are conducted in actual patient care units using ‘real’ equipment, resources, policies, procedures and staff.1 The physical integration of simulation into the clinical environment adjacent to real patient care improves the contextual fidelity of these events. Inter-professional ISS allows teams that work together to practice together within their clinical space. ISS can drive improved performance and safety in clinical environments during real patient care in ways that centre-based simulation does not. ISS has demonstrated potential as a method of prospective risk reduction via the evaluation of system competence and the identification of the latent conditions that may predispose patients to harm.1 2
Wachter has defined latent safety threats (LSTs) as “less apparent failures of organization or design that contributed to the occurrence of errors or allowed them to cause harm to patients—latent errors are quite literally accidents waiting to happen.”3 LSTs have also been defined as system-based threats to patient safety that can materialise at any time and are previously unrecognised by healthcare providers, unit directors or hospital administration.4 These errors in design, organisation, training or maintenance may have a significant impact on patient safety and, if not mitigated, could potentially delay management in an emergency situation.5 A common example of an LST in paediatrics is a scale that weighs patients in pounds or kilograms when all medication dosing is calculated based on the weight in kilograms. A scale that is locked into kilograms would prevent the risk of a patient receiving an overdose of a medication.
There are a number of recent publications that describe the identification of LSTs using ISS in multiple hospital settings including emergency departments, operating rooms, labour and delivery suites, and various inpatient units.2 6–12 Paediatrics is an area that is suitable for the use of ISS to detect LSTs as many high stakes paediatric conditions are extremely low frequency events in a single institution and therefore there are limited opportunities to assess the safety of care that is provided (eg, paediatric cardiac arrest). A recent review of published ISS research noted that 87% of the programmes described using ISS in some way to identify potential hazards. However, only 21% of the programmes identified some type of feedback mechanism to address the hazards identified during the ISS and debriefing.13
By definition, LSTs have not yet harmed patients, however, in uncovering the LST the ISS demonstrates the potential for patient harm. There are no published descriptions of best practices to address LSTs identified in ISS. Based on the authors’ personal experiences there is substantial variation in the ways that various healthcare organisations address LSTs identified in ISS. This is in contrast to threats to patient safety identified in clinical care where there are standardised approaches to reporting these events (eg, hospital event reporting systems).
The goal of this project is to describe the current use of ISS to improve patient safety. The objectives of this survey were to describe children's hospital simulation programmes: (1) utilisation of ISS and (2) outcomes from ISS (detection and mitigation of LSTs).
Methodology
The survey questions were created to describe the constructs of: (1) utilisation of ISS (location, participants, cancellations and coordination) and (2) the outcomes of ISS (detection and response to safety threats). The questions were developed over a series of four iterations with conference calls to discuss question content, format, wording and order by the three study authors. The authors have a combined 21 years of experience conducting ISS. Next, the final survey was administered to five additional physicians with over 40 years of combined experience in leading paediatric simulation programmes as a pilot test for face and content validity. The survey went through three iterations during this process. The final survey included 25 questions in the following domains (figure 1): 3 demographic questions, 9 questions describing utilisation of ISS and 13 questions describing the outcomes of the ISS programme (the detection and mitigation of LSTs).
Figure 1.
Survey tool. The 25-question survey administered to volunteer participants at two international meetings in January 2014.
This survey was administered to volunteer participants sampled in person during the 2014 INSPIRE meeting (International Network for Simulation–Based Pediatric Innovation, Research and Education)14 on 25 January 2014 in San Francisco, California, USA (77 attendees from 43 academic medical centres) and at the Society for Simulation in Healthcare (SSH) Pediatric Special Interest Group meeting on 27 January 2014 in San Francisco, California, USA (55 attendees from 40 academic medical centres). Many individuals attended both meetings and the total eligible sample included 82 individuals representing 48 academic medical centres. Participants were eligible for inclusion if they were faculty-level paediatric simulation practitioners from academic medical centres attending these meetings. All types of professionals were eligible to complete the survey (RN, MD, PhD). Simulation practitioners from non-academic centres, community hospitals, trainees and industry were excluded from recruitment. We classified academic medical centres as a medical centre that had an affiliation with a university. When multiple divisions or programmes were represented from distinct ISS programmes within a single institution, multiple participants were asked to complete a survey. Of note, on enrolment some institutions had multiple responses (eg, one from an emergency department (ED) ISS programme and another from a neonatal intensive care unit (NICU) ISS programme) while other institutions considered themselves as having a single simulation centre ISS programme that served the entire enterprise. Those who did not conduct ISS completed demographic data only. This study was reviewed by the Akron Children's Hospital Institutional Review Board and deemed exempt. Responses were anonymous and data were presented in aggregate. All data were descriptive or qualitative in nature and descriptive statistics were performed using SPSS V.21.0.
Results
Fifty-six unique surveys described programmes at 38 academic medical centres represented on the map in figure 2. The response rate was 68% (56/82) at the individual attendee level and surveys were completed from 79% (38/48) of the academic medical centres represented at the meetings.
Figure 2.
Participating sites. Academic medical centres represented by the respondents of the survey, with majority located in USA.
Utilisation of ISS in paediatrics
Use of ISS: 93% (52/56) of the respondents reported conducting ISS at their institution and completed the full survey. Four individuals were not conducting ISS and did not complete the remainder of the survey.
Frequency: 48% (25/52) of the respondents reported conducting monthly ISS sessions and 19% (10/52) conducted more than two ISS sessions per week.
Duration of ISS: 52% (27/52) of the respondents reported that the duration of ISS sessions was <20 min.
Participants: 58% (30/52) of ISS involved participants working on the unit (on-shift). Thirty-three per cent (17/52) involved both, participants working on the unit and coming in from home (off-shift). The remaining 10% (5/52) involved only participants coming from home.
Cancellations: 69% (36/52) of respondents had an explicit ‘no go’ or cancellation policy related to ISS. All programmes reported cancelling ISS over the past year with a broad range of cancellation frequency (10–50%) with the majority of programmes (33/52) cancelling 20% of the time or less. The most common factors leading to cancellations of ISS related to patient census, patient acuity and staffing numbers.
Location: At many institutions ISS was conducted in multiple clinical environments. The most common environments included: NICU 73% (38/52), ED 69% (36/52), paediatric intensive care unit 58% (30/52) and/or inpatient wards 58% (30/52). Additionally in some institutions ISS were conducted in operating rooms, post-anesthesia/recovery units, labour and delivery suites, clinics (primary care, specialty care, urgent care and dental), dialysis/infusion centres, helicopter pad, radiology suites, and public areas.
Coordination of ISS: The ISS activities were coordinated by various stakeholders including the simulation programme 33% (17/52), the respective clinical division 23% (12/52), the hospital patient safety programme 17% (9/52), the residency training programme 17% (9/52) and the nursing education programme 10% (5/52). Half of programmes (26/52) reported that other ISS efforts existed within their institution in addition to the programme that they were involved with.
Identification and mitigation of LST in paediatric ISS
Detection of LSTs: 92% (48/52) of programmes reported detecting LSTs via ISS. Table 1 reports examples of responses to the question ‘name the most serious LST identified during the past year’ by themes for the 38/48 individuals who completed this free text question.
Table 1.
Most serious LST identified during ISS at your institution
| Medication |
|
| Missing equipment |
|
| Malfunctioning equipment |
|
| Staff |
|
| Communication infrastructure |
|
| Facility design |
|
| New facilities |
|
ED, emergency department; ICU, intensive care unit; ISS, in situ simulation; LST, latent safety threat.
Number of LSTs identified: Programmes reported identifying up to five LSTs per ISS with 46% (24/52) identifying one LST per ISS and the other programmes identifying more than one per ISS.
Severity of LST: 40% (21/52) reported that an LST identified in ISS could lead to a serious safety event and/or serious harm or death of a patient if not resolved.
Who addresses LSTs: LSTs identified in ISS were addressed by the following stakeholders (in order from most frequent to least frequent): unit level nursing/physician leadership (14/28), hospital quality and safety programmes (5/28), code committees (6/28), and clinical engineering (3/28).
Time to resolution of LST: The mean time that participants reported to mitigation of these serious LSTs was 50 days with an SD of 150 days and a range of 0–700 days.
Repetition of ISS cases to re-evaluate LST: 62% (32/52) of respondents repeated simulation cases to evaluate mitigation of LSTs after dissemination of the ISS reports.
Response to detected LSTs: 54% (28/52) of programmes reported utilising a formal report document and/or process after ISS sessions to communicate the LSTs to other individuals within the institution. For the programmes utilising formal reporting processes these reports were distributed to diverse stakeholders as described in table 2. In the first column each respondent reported all of the stakeholders to whom they send their report. The second column lists the stakeholders whom the respondent identified as primarily responsible to follow-up on the simulation report out document. Note that these numbers exceed the total number of respondents with formal report out processes (n=28) as many respondents reported sending the report out to multiple stakeholders and checked multiple stakeholders as having primary responsibility (item was check all that apply).
Table 2.
ISS report out distribution and follow-up
| Stakeholder | Receives report out on ISS | Responsible to follow-up |
|---|---|---|
| Unit-based leadership MD±RN | 15 | 19 |
| Front-line providers/participants in the simulation | 5 | 0 |
| Simulation centre/team | 19 | 14 |
| Division level committee/leadership: trauma or medical | 19 | 2 |
| Institution level committee/leadership: quality/safety/risk | 18 | 12 |
| Clinical engineering | 5 | 0 |
| Chief medical officer | 6 | 0 |
| Residency director | 2 | 0 |
| Legal management | 8 | 0 |
| Respiratory | 1 | 0 |
| Pharmacy | 3 | 0 |
ISS, in situ simulation.
Unresolved LSTs identified in ISS: of note 23% (12/52) of respondents reported the detection of a serious LST in ISS that was not resolved and subsequently led to a safety event during real patient care.
Discussion
This is the first study to describe the use of ISS in a cross-section of academic institutions. ISS has been described in a variety of fields and is most frequently applied for high stakes and low-frequency conditions.6 8 15–17 Our results describe a high rate of utilisation of ISS across a spectrum of paediatric clinical environments in academic medical centres. These results demonstrate the ‘knowledge dissemination’ of early work in this field described in a series of papers from Cincinnati Children's Hospital and Johns Hopkins.1 2 6 7 13 Recent work has described this application of ISS for safety work in obstetrics and neonatology departments.8 10–12
These data provide evidence for the potential impact of this approach across a diverse set of institutions. Many significant safety threats were identified through ISS at different programmes. Most of the respondents reported actions taken to mitigate LSTs, however, the timeline for solutions was often protracted. This inefficiency in the mitigation process is represented in our data in that one-fifth of programmes reported LSTs that subsequently had a negative impact on actual patients prior to their mitigation. The lack of a consistent process to address LSTs identified through ISS may be a contributing factor to these delays. We propose using techniques such as those described by Patterson et al1 to collate LSTs and develop action plans.
In this survey some institutions reported that they are entering LSTs into their existing quality and safety event reporting system in order to communicate their findings back to the organisation. This is the best example of a systems integration approach to simulation-based quality improvement. ISS programmes should work towards integrating their efforts into the ongoing quality and safety initiatives of their organisation, ideally creating a bidirectional flow of information between organisational safety structure and the ISS programme.
The identification of an LST through ISS provides the opportunity to mitigate risk before any harm occurs to a patient and should be considered a ‘great catch’. Reflecting on our results it is notable that the diversity of LSTs (table 1) were similar to those found in a single-centre study.1 These data support the use of ISS to identify a broad range of safety threats across the complex healthcare system. Most of the LSTs listed in this table have the potential to lead to serious safety events with real and permanent harm. While it is difficult to extrapolate the direct savings resulting from each of the LSTs identified, it is likely that there is potential to utilise this type of data to demonstrate the value on investment of simulation to hospital leadership. One method used by some respondents to document system improvement was the repetition of ISS to evaluate if changes implemented in their system resulted in mitigation of LSTs or created unanticipated consequences. Repeating ISS and tracking data on actual patient harm are important methods for measurement that should be explored in future research.
Future directions
The integration of safety science principles into ongoing simulation experiences has the potential to improve quality and safety. The first step in this process is codifying the LST events and prioritising risk. Carayon's Systems Engineering In Patient Safety framework is one method that is being explored as a method to codify events into domains including: tasks, providers, communication, tools, technology, organisation, environment and culture.18 19 The Healthcare Failure Modes Effects Analysis is an alternative method that can be used to triage LSTs based on their severity and probability.20 These approaches can help individuals determine the factors that are contributing to safety threats and to develop a timeline for improvement.
Systems Integration became an area of accreditation for simulation centres through the SSH in 2010 (http://www.ssih.org/Accreditation) and is defined as “a consistent, planned, collaborative, integrated and iterative application of simulation-based assessment and teaching activities with systems engineering and risk management principles to achieve excellent bedside clinical care, enhanced patient safety, and improved metrics across the healthcare system.” In the authors’ collective experiences implementing ISS into an institution's culture is challenging and requires a persistent effort to engage diverse stakeholders across the system. Existing efforts and experts in quality, patient safety, safety sciences and risk management should be integrated into ISS programmes in the design of simulations, evaluations of performance, and follow-up of LSTs. This work involves engaging stakeholders from the top-down and bottom-up throughout the organisation so that ISS becomes an expected part of the healthcare organisations’ daily work to provide safe care. When LSTs are identified and mitigated that information should be celebrated and communicated to both leadership and front-line providers as a ‘return on investment’ for their participation in or support for the ISS programme. The findings from this survey should encourage existing simulation programmes to leverage ISS to improve safety. Additionally these data provide a foundation as the simulation community works to develop a standardised process for categorising and rating the severity of LSTs to guide prioritisation of interventions. Bi-directional communication and collaboration between the simulation programmes and institutional stakeholders before, during and after ISS will improve the impact of these programmes on patient outcomes. Simulation experts can provide input into the feasibility and practicability of ISS as part of the improvement process. Moving forward, healthcare organisations’ culture of safety and quality should involve the application of ISS for testing of systems and training of providers. Safety scientists such as human factors experts, systems engineers and other content experts can provide additional insight into this work.
Limitations
The validity and generalisability of these findings is limited by the decision to complete this survey at national simulation meetings and completion by only one or two individuals per organisation. This group of respondents may have had intrinsic biases as a population that attends these conferences and this could limit the generalisability of their responses. It is likely that many of the institutions had additional ISS programmes that were not represented. This may have led to selection bias and under-represent the use of ISS in specialties that are less likely to attend this meeting (eg, surgery). In advance of completing the survey, eligible participants were informed that the survey related to ISS. This may have led to selection bias for those individuals working in this area. The survey did not distinguish between announced and unannounced ISS. This may have impacted the risk of cancellation and is an area for future inquiry. As with any survey-based study the validity of the responses could be limited by recall bias. These data likely over represent the use of ISS and LST follow-up as recruitment occurred at an academic simulation meeting in the USA (although there were attendees from other parts of the world) and many of the stakeholders at the meeting come from centres with ‘mature simulation programmes.’ For example, our rate of 54% of programmes using formal reports contrasts with Rosen's work reporting that 21% of programmes used formal reports.13
An additional limitation was that the survey did not collect extensive information on the respondents’ professions or demographic characteristics. This could lead to an over-representation or under-representation of specific specialties or professions that are not an accurate representation of the diverse simulation community. Lastly, this survey did not explore participants’ reactions and experiences with ISS as the respondents were all simulation leaders. Work by Wheeler assessing ISS participant reactions to ISS events across hospital settings noted that the simulation had minimal impact on their work day or their clinical units performance.2 However, participants’ assessment of psychological safety and the clinical impact of ISS was not part of this project. This is an important area of future inquiry.
Conclusions
The use of ISS to identify and mitigate LST is common in this cross-sectional survey of paediatric simulation programmes. Diverse processes and organisational structures exist for reporting and mitigating LSTs identified in ISS. A more integrated and systematic approach to ISS and LST could help ensure the mitigation of LSTs before they impact patients.
Acknowledgments
The authors would like to acknowledge the SSH Pediatrics Section and INSPIRE Network without which the projects described could not have been implemented. They also wish to acknowledge Charmin Gohel, Yedidya Ben-Avie and Doug Street for assistance with data entry, organisation of this manuscript and the submission process.
Footnotes
Competing interests: None declared.
Ethics approval: Akron Children's Hospital Institutional Review Board.
Provenance and peer review: Not commissioned; externally peer reviewed.
References
- 1.Patterson MD, Geis GL, Falcone RA, et al. In situ simulation: detection of safety threats and teamwork training in a high risk emergency department. BMJ Qual Saf 2013;22:468–77. 10.1136/bmjqs-2012-000942 [DOI] [PubMed] [Google Scholar]
- 2.Wheeler DS, Geis G, Mack EH, et al. High-reliability emergency response teams in the hospital: improving quality and safety using in situ simulation training. BMJ Qual Saf 2013;22:507–14. 10.1136/bmjqs-2012-000931 [DOI] [PubMed] [Google Scholar]
- 3.Wachter RM. Understanding patient safety. New York: McGraw-Hill Professional Publishing, 2007. [Google Scholar]
- 4.Alfredsdottir H, Bjornsdottir K. Nursing and patient safety in the operating room. J Adv Nurs 2008;61:29–37. 10.1111/j.1365-2648.2007.04462.x [DOI] [PubMed] [Google Scholar]
- 5.Reason J. Human error: models and management. BMJ 2000;320:768–70. 10.1136/bmj.320.7237.768 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Geis GL, Pio B, Pendergrass TL, et al. Simulation to assess the safety of new healthcare teams and new facilities. Simul Healthc 2011;6:125–33. 10.1097/SIH.0b013e31820dff30 [DOI] [PubMed] [Google Scholar]
- 7.Patterson MD, Geis GL, LeMaster T, et al. Impact of multidisciplinary simulation-based training on patient safety in a paediatric emergency department. BMJ Qual Saf 2013;22:383–93. 10.1136/bmjqs-2012-000951 [DOI] [PubMed] [Google Scholar]
- 8.Riley W, Davis S, Miller KM, et al. Detecting breaches in defensive barriers using in situ simulation for obstetric emergencies. Qual Saf Health Care 2010;19(Suppl 3):i53–6. 10.1136/qshc.2010.040311 [DOI] [PubMed] [Google Scholar]
- 9.Johnson K, Geis G, Oehler J, et al. High fidelity simulation to design a novel system of care for pediatric critical airway obstruction. San Diego, CA: The American Society of Pediatric Otolaryngology, 2012. [DOI] [PubMed] [Google Scholar]
- 10.Davis S, Riley W, Gurses AP, et al. Failure modes and effects analysis based on in situ simulations: a methodology to improve understanding of risks and failures. In: Henriksen K, Battles JB, Keyes MA, et al., eds. Advances in patient safety: new directions and alternative approaches (Vol. 3: performance and tools). Rockville, MD: Agency for Healthcare Research and Quality (US); 2008 Aug. Available from: http://www.ncbi.nlm.nih.gov/books/NBK43662/ [PubMed] [Google Scholar]
- 11.Guise JM, Mladenovic J. In situ simulation: identification of systems issues. Semin Perinatol 2013;37:161–5. 10.1053/j.semperi.2013.02.007 [DOI] [PubMed] [Google Scholar]
- 12.Sørensen JL, Lottrup P, van der Vleuten C, et al. Unannounced in situ simulation of obstetric emergencies: staff perceptions and organisational impact. Postgrad Med J 2014;90:622–9. 10.1136/postgradmedj-2013-132280 [DOI] [PubMed] [Google Scholar]
- 13.Rosen MA, Hunt EA, Pronovost PJ, et al. In situ simulation in continuing education for the health care professions: a systematic review. J Contin Educ Health Prof 2012;32:243–54. 10.1002/chp.21152 [DOI] [PubMed] [Google Scholar]
- 14.International Network for Simulation-Based Pediatric Innovation, Research and Education (INSPIRE).
- 15.Kobayashi L, Shapiro MJ, Sucov A, et al. Portable advanced medical simulation for new emergency department testing and orientation. Acad Emerg Med 2006;13:691–5. 10.1197/j.aem.2006.01.023 [DOI] [PubMed] [Google Scholar]
- 16.Kobayashi L, Overly FL, Fairbanks RJ, et al. Advanced medical simulation applications for emergency medicine microsystems evaluation and training. Acad Emerg Med 2008;15:1058–70. 10.1111/j.1553-2712.2008.00247.x [DOI] [PubMed] [Google Scholar]
- 17.Lighthall GK, Poon T, Harrison TK. Using in situ simulation to improve in-hospital cardiopulmonary resuscitation. Jt Comm J Qual Patient Saf 2010;36:209–16. [DOI] [PubMed] [Google Scholar]
- 18.Holden RJ, Carayon P, Gurses AP, et al. SEIPS 2.0: a human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics 2013;56:1669–86. 10.1080/00140139.2013.838643 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Carayon P, Schoofs Hundt A, Karsh BT, et al. Work system design for patient safety: the SEIPS model. Qual Saf Health Care 2006;15(Suppl 1):i50–8. 10.1136/qshc.2005.015842 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Nielsen DS, Dieckmann P, Mohr M, et al. Augmenting health care failure modes and effects analysis with simulation. Simul Healthc 2014;9:48–55. 10.1097/SIH.0b013e3182a3defd [DOI] [PubMed] [Google Scholar]