Multiprofessional perspectives on the identification of latent safety threats via in situ simulation: a prospective cohort pilot study

Abstract

Objectives

To describe the association between participant profession and the number and type of latent safety threats (LSTs) identified during in situ simulation (ISS). Secondary objectives were to describe the association between both (a) participants’ years of experience and LST identification and (b) type of scenario and number of identified LSTs.

Methods

Emergency staff physicians (MDs), registered nurses (RNs) and respiratory therapists (RTs) participated in ISS sessions in the emergency department (ED) of a tertiary care teaching hospital. Adult and paediatric scenarios were designed to be high-acuity, low-occurrence resuscitation cases. Simulations were 10 min in duration. A written survey was administered to participants immediately postsimulation, collecting demographic data and perceived LSTs. Survey data was collated and LSTs were grouped using a previously described framework.

Results

Thirteen simulation sessions were completed from July to November 2018, with 59 participants (12 MDs, 41 RNs, 6 RTs). Twenty-four unique LSTs were identified from survey data. RNs identified a median of 2 (IQR 1, 2.5) LSTs, significantly more than RTs (0.5 (IQR 0, 1.25), p=0.04). Within respective professions, MDs and RTs most commonly identified equipment issues, and RNs most commonly identified medication issues. Participants with ≤10 years of experience identified a median of 2 (IQR 1, 3) LSTs versus 1 (IQR 1, 2) LST in those with >10 years of experience (p=0.06). Adult and paediatric patient scenarios were associated with the identification of a median of 4 (IQR 3.0, 4.0) and 5 LSTs (IQR 3.5, 6.5), respectively (p=0.15).

Conclusions

Inclusion of a multidisciplinary team is important during ISS in order to gain a breadth of perspectives for the identification of LSTs. In our study, participants with ≤10 years of experience and simulations with paediatric scenarios were associated with a higher number of identified LSTs; however, the difference was not statistically significant.

INTRODUCTION

Patient safety is a core principle of any healthcare system and an outcome of interest to all stakeholders from patients and families, to healthcare providers and administrators. ^{1 2} To improve overall patient safety, a focus on system-level solutions rather than on individual medical error is required. ³ One method of improving patient safety is through the identification (and subsequent mitigation) of latent safety threats (LSTs). LSTs are system-based threats to patient safety that have gone unrecognised by the healthcare team or hospital administrators. ^4–7 Examples of LSTs include an inefficient layout of equipment in the trauma bay, lack of knowledge as a result of inadequate staff training and lack of regular equipment maintenance. ^{3 4 8} A structured process promoting the identification and triage of LSTs, with administrative support to allow subsequent mitigation, has been recommended. ⁹

In situ simulation (ISS) is an effective method of identifying and mitigating LSTs. ¹⁰ ISS takes place in the real workplace environment and allows the authentic multiprofessional patient care team to participate in simulated scenarios. Prior work has shown ISS to be superior in identifying LSTs compared to laboratory-based simulation ¹⁰ due to improved environmental fidelity, ⁵ interaction with systems that are difficult to simulate in a laboratory setting ⁵ and the ability to implement feedback immediately into clinical practice. ¹⁰ For example, Lighthall et al performed simulated resuscitation scenarios in a palliative care ward and discovered that the wall-mounted code buttons were not functional and that a bag valve mask was not available in every patient room, resulting in a delay to appropriate airway support. Corrective measures were made, and the appropriate equipment became mandatory. ¹¹

As the practice of medicine is a multiprofessional endeavour, all members of the multiprofessional team should participate in ISS to maximise the authenticity and effectiveness of the simulation. There is a dearth of studies investigating the influence of team composition on the identification of LSTs during ISS. Previous research has shown that members of different professional groups identify LSTs that may not have been apparent to others. ¹⁰ However, precisely how different professions influence the type and number of LSTs identified is not known.

The primary objective of this study was to describe the association between the participant’s profession and the number and type of LSTs identified during ISS in the emergency department (ED). Secondary objectives were to evaluate the association between the participants’ years of experience and LST identification and the association between the type of scenario and the number of identified LSTs. This information will assist simulation educators with the implementation of ISS for LST identification and correction through departmental patient safety programmes. Specifically, scenario content and the professions involved can be strategically planned to maximise the efficiency of LST identification.

METHODS

Setting and participants

This prospective cohort pilot study was conducted in the ED at the Kingston Health Sciences Centre, a tertiary care teaching hospital with approximately 65 000 annual ED visits. Scenarios were mainly conducted in one of three resuscitation bays; however, other areas of the department were used when the resuscitation bays were required for clinical care. Participants included attending emergency room physicians (MDs), registered nurses (RNs) and respiratory therapists (RTs) who were working in the acute care area of the ED at the time of the ISS session. This multiprofessional group reflects the actual team that would be responsible to provide immediate resuscitative care upon the arrival of a patient. Healthcare learners (residents, medical students, nursing students, etc) were not included as their participation was variable. As part of a departmental continuing education programme, participation in ISS scenarios was strongly encouraged from both physician and nursing leadership; however, informed consent for the use of survey data for research purposes was sought before each ISS session. Healthcare providers were not tracked to determine if there was repeat participation in simulation sessions. Participants were excluded if they did not provide consent for the collection and use of their data for research purposes. This study was approved by the Queen’s University Health Sciences Research Ethics Board (#6024229).

In situ simulation scenarios

ISS sessions took place between July and November 2018. Six resuscitation-based scenarios were developed and piloted in the simulation laboratory by study authors (TC, SH). Scenarios were designed as high-acuity ED presentations, and all had been peer-reviewed by a group of simulation educators with fellowship training in simulation education. LSTs were not purposefully integrated into any of the scenarios. Depending on availability, either the SimMan 3 G or SimMan Essential (Laerdal, Toronto, Canada) interactive patient simulator was used for adult scenarios. The Sim Junior (Laerdal, Toronto, Canada) was used for paediatric cases and the baby HAL (Gaumard, Miami, Florida USA) was used for infant cases.

Each ISS session was facilitated by an emergency medicine physician with an interest in simulation-based teaching. Three of four facilitators completed fellowships in simulation education. Before each ISS, a facilitator led a discussion of 2–3 min in the resuscitation bay with the multiprofessional team. The purpose of this discussion was to review the capabilities of the patient simulator and provide a brief introduction to the scenario. Participants were instructed to interact with the patient simulator as they would with a real patient. For example, they were to order and obtain medications, use equipment and chart as they normally would during the care of a real patient. Participants were informed that the purpose of the ISS was to practice and improve team-based response to high-acuity presentations. They were not aware that they would be asked to identify LSTs following the scenario. Participants had the opportunity to ask questions regarding the patient simulator’s capabilities. Following this, the ISS scenario took place and lasted approximately 10 min.

Survey and data collection

Immediately following each scenario, participants independently completed a brief written survey (supplemental file) and provided consent for their responses to be used in the research study. Surveys were completed and collected prior to group debriefing to mitigate response bias. The survey was designed to take less than 3 min to complete and collected participant demographics and a list of any perceived LSTs relevant to the simulated case. Following survey collection, facilitators led a debriefing based on the PEARLS healthcare Debriefing Tool. ¹² Any additional LSTs identified by the participants during the discussion were recorded by an objective observer who was an ED RN with over 30 years experience. LSTs identified on the survey and from the debriefing were collated separately and categorised using a previously described framework. ¹⁰ Two reviewers (DR and TC) who were blinded to participant profession collated survey responses, removed duplicates and categorised LSTs. Identified LSTs were considered duplicates if they specifically identified the same issue, regardless of the scenario. For example, if the issue of incomplete stocking of endotracheal tubes in the paediatric resuscitation cart was identified, future occurrences of this were tracked as duplicates. Any disagreements in the categorisation of LSTs were discussed and third-party adjudication was used to reach consensus. The notes and surveys were transferred into a secure, password-protected electronic database (Microsoft Excel for Mac 2019). We used the STROBE cohort checklist when writing this manuscript. ¹³

Data analysis

Descriptive statistics were calculated with categorical data described by frequencies and proportions and with continuous data using means and SD and medians and IQRs where appropriate. The primary outcome measure was the type and number of LSTs identified by each profession. The association between participant profession and number of LSTs was analysed using the Kruskal-Wallis test with Dunn’s multiple comparison test. The effect size was measured using Hedges’ g test. Interpretation of effect size was done accordingly: less than 0.2 being a small effect size, 0.5 being a medium effect size and greater than 0.8 representing a large effect size. ¹⁴ Hedges’ g was chosen over Cohen’s d as it corrects for bias in small samples, as found in our study. ¹⁵ Participant survey data were also stratified based on provider years of experience to examine if those variables were associated with the total number or type of identified LST. This association was assessed using the Mann-Whitney test. The proportion of LSTs identified by each profession was assessed using the ᵡ² test. LSTs were also stratified based on the scenario to determine if the identification of LSTs varied by scenario. This association was analysed using the Mann-Whitney test. A p value of less than 0.05 was determined to be statistically significant. All analyses were conducted using Prism 6 (GraphPad).

RESULTS

Demographics

Thirteen ISS sessions were conducted between July and November 2018. Over this period, a total of 59 participant surveys were completed: 12 by MDs, 41 by RNs and 6 by RTs (table 1). All eligible participants took part in the scenarios and consented to be included in the study. A summary of attendees at each simulation session is available in supplemental table 1. One session was led by a resident due to the staff physician being unavailable and, thus, that data is not included here. The mean age of MDs, RNs and RTs was 48.8 (SD 9.9), 37 (SD 12.7) and 38 (SD 5.6) years, respectively. The majority of MDs (75%) and RTs (66%) involved in the ISS sessions had more than 10 years of experience in their current roles while 50% of RNs had less than 5 years of working experience.

Table 1.

Participant demographics

Profession	MD	RN	RT
n	12	41	6
Mean age in years (SD)	48.8 (9.9)	37 (12.7)	38 (5.6)
Mean years of experience (SD)	15.5 (4.3)	9 (6.8)	13.2 (6.8)
Male%/female%	83/17	24/76	0/100

MD, staff physician; RN, registered nurse; RT, respiratory therapist.

LST identification

A total of 51 LSTs were identified from both the survey and debriefing data across all 13 ISS sessions. A 25.6% of the LSTs were identified solely from surveys and 32.5% were identified solely from the debriefing. The remaining 41.9% were identified in both the survey and debriefing. Following the removal of duplicates, 24 unique LSTs were identified from survey data and categorised as follows: 5 medication, 10 equipment, 4 resource, 4 communication and 1 miscellaneous issue. LSTs were coded into categories as seen in table 2.

Table 2.

LSTs identified from surveys

Medication issues (5)	Resource issues (4)	Equipment issues (10)	Communication issues (4)	Other (1)
Drug familiarity/knowledge (2) Drug delay (2) Paediatric dosing (1)	Inadequate staffing of RNs/RTs (2) Insufficient staff to complete charting (1) PCI lab delay (1)	Room equipment missing (3) Paediatric cart equipment missing (5) Defective equipment (1) Unfamiliar equipment (1)	Unclear role designation (1) Ventilation feedback not given (1) Drug administration not communicated during resuscitation (2)	Inadequate CPR technique (1)

Numbers in parenthesis are the reported frequency of that LST in unique circumstances.

LSTs, latent safety threats; RNs, registered nurses; RTs, respiratory therapists; PCI, percutaneous coronary intervention.

Descriptive statistics of survey LSTs identified by each profession is summarised in table 3a. A Kruskal-Wallis test showed a statistically significant difference between the number of LSTs identified by each profession χ²(2)=6.033, p=0.049, with Dunn’s multiple comparison’s test demonstrating a significant difference between the ranks of RT versus RN (−17.65, p=0.042; table 3b). Effect sizes and 95% CIs are summarised in table 3b. Briefly, large effect sizes were seen between RN and RT groups and MD and RT groups, with a Hedges’ g value greater than 1. Simulations that were attended by RTs had a median of 4.5 LSTs (IQR 2.5, 5.5) identified compared to a median of 4 LSTs (IQR 3, 4) identified when they were not present (p=0.49). The most commonly identified category of LSTs reported by MDs (36.8%) and RTs (75%) was equipment issues. RNs identified significantly more medication issues (36.4%) than other professions (MD 26.3%, RT 0%) (p=0.004) (table 4).

Table 3a.

Descriptive statistics of LSTs identified by each profession

Survey LSTs identified	MD N=12	RN N=41	RT N=6
Mean	1.583	1.878	0.6667
SD	0.7930	1.208	0.8165
Median	2	2	0.5
IQR	1, 2	1, 2.5	0, 1.25

LSTs, latent safety threats; MD, staff physician; RN, registered nurse; RT, respiratory therapist.

Table 3b.

Comparisons of LSTs identified between professions

Comparison groups	RN vs MD	RT vs MD	RT vs RN
Dunn’s mean rank difference	2.986	−14.67	−17.65
Adjusted p value (Dunn’s multiple comparison test)	>0.9999	0.2246	0.0426*
Hedges’ g effect size (CI range)	0.26 (−0.90, 0.39)	1.09 (0.05, 2.13)	1.02 (0.14, 1.90)

LSTs, latent safety threats; MD, staff physician; RN, registered nurse; RT, respiratory therapist.

Table 4.

LST proportions identified by each profession

LST category	MD N=12	RN N=41	RT N=6	P value
Medication issues	5 (26.32%)	28 (36.36%)	0 (0%)	0.004
Equipment issues	7 (36.84%)	21 (27.27%)	3 (75%)	0.902
Resource issues	3 (15.79%)	13 (16.88%)	0 (0%)	0.260
Teamwork/communication issues	4 (17.61%)	15 (17.14%)	1 (25%)	0.628
Other	0	0	0	n/a
Total	19	77	4

LST, latent safety threat; MD, staff physician; RN, registered nurse; RT, respiratory therapist.

Participants with 10 years of experience or less in their profession identified more LSTs in surveys than participants with greater than 10 years of experience (n=31 vs 27; median (IQR): 2 (IQR 1, 3) LSTs vs 1 (IQR 1, 2) LSTs, respectively); however, the difference was not statistically significant (p=0.06). Hedges’ g calculation indicated a moderate effect size in favour of having less than 10 years of experience (Hedges’ g=0.52; 95% CI: −0.002, 1.05).

The simulation frequency and distribution of LSTs identified in each scenario are summarized in table 5. Paediatric scenarios resulted in the identification of more LSTs than adult scenarios (5 (IQR 3.5, 6.5 vs 4 (IQR 3.0, 4.0). respectively) but the difference was not statistically significant (p=0.15). Hedges’ g calculation indicated a large effect sise in favour of paediatric scenarios (Hedges’ g=1.07; 95% CI: −0.16, 2.44).

Table 5.

LST amount identified by scenario

Scenario type	Simulation frequency	Total LSTs ID’d	Median LSTs	IQR
STEMI	3	10	3	3,4
ACEI angioedema	2	9	4.5	4,5
Paediatric DKA	1	1	1	1,1
Burns/CN poisoning	3	11	4	3,4
Paediatric NAT	3	13	5	3,5
Paediatric arrest	1	7	7	7,7

ACEI, ACE inhibitor; CN, cyanide; DKA, diabetic ketoacidosis; LST, latent safety threat; NAT, non-accidental trauma; STEMI, ST-elevation myocardial infarction.

DISCUSSION

Our pilot study corroborates previous evidence that a multiprofessional team affords diverse perspectives and allows for the identification of various LSTs using ISS. ¹⁰ We have further described the type and frequency of LST identification by profession in an ED context. In our study, 36% of LSTs identified by nursing staff were medication-related, compared to 26% and 0% by emergency physicians and therapists, respectively. Had nursing staff not been included, a substantial amount of medication-related LSTs may not have been reported. This finding is not surprising, as nursing staff are actively engaged in every step of administering medication and may, therefore, be better positioned to identify issues along this process. Each profession brings its own ‘lens’ to the conversation of LSTs based on its unique expertise and priorities with respect to patient care. Our results support the inclusion of the multiprofessional team in order to maximise the identification of LSTs using ISS.

According to survey responses, nursing staff reported the highest number of patient safety issues (77%), followed by physicians (19%) and RTs (4%). This study is the first to describe the frequency of LSTs that are identified by different professions and provides support for the inclusion of multiprofessional participants in ISS. Patterson and colleagues suggest that different groups identify different LSTs; however, the objective of their study was not to describe how each profession contributes to the identification of LSTs in ISS. The demographics in their study revealed that over 50% of participants were physicians and that 32% of participants were nursing staff. They found that despite nursing staff being at a smaller number, they still identified a substantial amount of LSTs. ¹⁰ Their study differs in the proportion of nursing staff and physicians to ours and also includes other members of the team that were not present in our simulations.

Years of experience had an inverse relationship on the identification of LSTs in our study, with participants having less than 10 years of experience in their profession identifying more LSTs than those with greater than 10 years, with a moderate effect size although not significantly different. Our results appear to contrast with findings by Ammouri and colleagues, who reported that nursing staff with greater years of experience expressed a better understanding of patient safety culture. ¹⁴ However, measuring one’s understanding of patient safety culture and the identification of LSTs cannot be considered as the same. ¹⁶ Participants with fewer years of experience may be more likely to identify LSTs due to a recent focus on patient safety culture at both the training programme level and the healthcare system level. ¹⁷ Curricula in medical schools and nursing programmes now include teaching on patient safety, medical error and human factors, allowing newer graduates to be more aware of system issues. ^{18 19} While learners were not included in our analysis, the finding above would support the inclusion of healthcare learners in future studies and ISS sessions.

No significant differences were observed in the number of LSTs identified from paediatric versus adult scenarios. Despite there being a difference in median LSTs identified in paediatric scenarios and a strong effect size, the lack of statistical significance is likely due to an insufficient number of paediatric simulations. Lack of provider experience with paediatric resuscitations is a well-known issue in emergency medicine. ²⁰ Practitioners completing a general emergency medicine residency participate in only a fraction of paediatric medical and trauma resuscitations when compared to adult cases. ²¹ Nursing literature reports a similar lack of exposure to paediatric cases. ²² Despite low exposure to paediatric patients in training, general EDs are responsible for paediatric care in over 80% of cases, as most of these patients do not present to paediatric speciality centres. ²³ During the ISS sessions, we uncovered issues regarding familiarity with paediatric dosing, the paediatric resuscitation cart and restocking of paediatric consumables. Lack of familiarity regarding paediatric dosing is a well-described issue in patient safety. ²⁴ Data has shown that simulation training can improve competence and familiarity in paediatric resuscitation cases. ²⁰

In this study, teamwork and communication issues made up a fairly even proportion of LSTs identified by each profession. Villemure and colleagues reported that interprofessional miscommunication in high-acuity situations is an issue that often leads to medical error. ²⁵ ISS can improve communication between healthcare workers and is a robust way to incorporate multiprofessional team training without major schedule disruptions for staff. ^{26 27} Patterson and colleagues believe that this occurs during debriefing due to the exchange of ‘silo’ed’ information that may be apparent to one group, but not the other. ¹⁰ Miller et al demonstrated that while ISS significantly improved communication and teamwork skills compared to didactic lectures, the improvements diminished after cessation of simulation, suggesting that ongoing ISS is required. ²⁷

We opted to use LSTs identified through survey responses as the main data source in this study. We believed that by conducting anonymous surveys, we could mitigate hurdles to ‘speaking-up behaviour’ and obtain an unbiased, holistic response from the simulation team. ²⁸ Notes were taken during debriefing for the purposes of identifying further LSTs that arose during discussion. Our results found that while surveys exclusively yielded nearly a quarter of the total LSTs, debriefing helped identify another third. The remaining amount was discussed in both surveys and debriefing. This finding suggests there may be merit in combining surveys and formal debriefing to maximise LST identification.

Limitations

Our study has limitations. First, this was a single-centre study in an academic centre with a strong medical simulation programme and administrative support, which may not be generalisable to other institutions. Second, there was a low repetition of paediatric scenarios that had a high amount of LSTs identified. Due to lack of repetition, it is difficult to conclude whether this truly has many safety issues associated with it or if it was a statistical outlier. Third, we did not track repeat participants in our simulations. This may have led to an inflated reporting of patient safety issues from repeat participants, simply due to the familiarity of the process. Fourth, ‘simple’ LSTs were corrected following each scenario (eg, missing or expired equipment was replaced), potentially resulting in a decreased number of LSTs in future sessions. However, as the ED is used for patient care in between simulation sessions, there was no control over whether these changes were maintained. We did not see a consistent decrease in LSTs upon repeating a scenario (supplemental table 1). An additional potential limitation of this study is the small number of RT participants. As a consequence of convenience sampling, in 13 scenarios, there were only 6 total RTs. Our analysis indicated that there was no difference in LSTs identified between scenarios where RTs attended and those they did not; however, given the small number of simulations and low RT participation rate, further research is needed. While the low participation rate may appear to be a limitation, it may actually be a result of inadequate RT staffing in the ED, an LST in and of itself. However, the reasons for poor RT attendance in our ISS was not established.

CONCLUSIONS

This study suggests that using a multidisciplinary team in ISS leads to the identification of diverse LSTs. We posit that this results in a more valid and holistic representation of systemic threats. Each member of the patient care team can provide a unique viewpoint based on individual training and experience. Future studies can further investigate the role of multidisciplinary teams on LST identification in ISS by including other members of the team (healthcare learners, pharmacists, paramedics, physicians from consulting services, etc) as well as the differences in scenario design (eg, adult vs paediatric) for identification of LSTs. In addition to having academic value, conducting ISS has benefits for mitigating LSTs in the ED and improving patient care. Although resource-intensive, ISS in the ED for LST identification should include a multidisciplinary team.

What is already known on this subject.

• It is well established that conducting in situ simulation results in the identification of more latent safety threats than in a traditional simulation laboratory environment.

• There is scant evidence examining how the composition of the simulation team or the type of scenario influences the number of latent safety threats identified.

What this study adds.

• According to our results, nursing staff identify significantly more latent safety threats than respiratory therapists.

• There was a trend towards a higher number of latent safety threats identified in participants with less than 10 years of experience.

• There was no significant difference in the median number of latent safety threats identified in adult or paediatric patient scenarios.