The Safety Action Feedback and Engagement (SAFE) Loop: Initial Testing and Refinement of a Novel Intervention to Enhance Hospital Incident Reporting and Patient Safety

Abstract

Background:

Voluntary incident reporting has improved safety in many high-risk industries, but several barriers have limited its effectiveness in hospitals. Designed to overcome these barriers, the novel Safety Action Feedback and Engagement (SAFE) Loop: (1) obtains input from nurses about patient safety problems; (2) invites nursing units to select a Target Event to focus on; (3) teaches nurses to write more informative incident reports; (4) prompts nurses to report Target Events for a designated period; (5) standardizes investigative procedures to support mitigation plans; and (6) provides feedback to nurses about contributing factors and mitigation plans. We aimed to refine the SAFE Loop through iterative testing.

Methods:

This study reflected Stage I of behavioral intervention development. At a large U.S. academic hospital, we first conducted proof-of-concept testing on two nursing units. Leveraging these experiences and stakeholder input, we made iterative refinements to intervention design, implementation, and evaluation plans. Finally, we conducted structured pilot testing of the refined intervention on one nursing unit. While SAFE Loop could be applied to any patient safety problem, we focused on medication safety.

Results:

In initial proof-of-concept testing, implementation was feasible and nurses voiced substantial enthusiasm. Following the refinements, the structured pilot test reaffirmed SAFE Loop feasibility, including selecting a Target Event, training nurses in enhanced reporting methods, conducting investigative interviews, developing mitigation plans with nursing unit leaders, and distributing mitigation plans on the unit. The pilot also demonstrated the feasibility of extracting contributing factors from incident reports and surveying nurses about reporting and safety. Qualitative interviews after the pilot reaffirmed intervention acceptability.

Conclusions:

Nurses found the SAFE Loop to be a promising strategy for improving patient safety. Testing and refinements to the SAFE Loop laid the groundwork for efficacy testing via an ongoing pragmatic randomized controlled trial.

Trial Registration:

ClinicalTrials.gov Number NCT05381441

Introduction

Medical errors continue to harm hundreds of thousands of hospitalized patients in the U.S. annually, and voluntary incident reporting is a widely discussed strategy for improving patient safety. Decades of effort by hospitals and policymakers have improved patient safety in major areas such as healthcare-associated infections [1]. Nonetheless, preventable harms remain common [2, 3]. A 2018 study at 11 hospitals in Massachusetts found that preventable adverse events occurred during 7% of hospitalizations, and 1% of hospitalizations involved serious to fatal events [4]. Earlier studies reached similar conclusions [1–3, 5–8].

Voluntary incident reporting is a technique through which frontline personnel describe events that pose risks, particularly near misses. Incident reporting has two foundational premises. First, a major safety event that causes harm is usually preceded by numerous near misses [9, 10], and near misses and harmful incidents share the same causes [11]. Second, first-hand witnesses can uniquely describe the specific conditions leading up to and potentially causing each incident—insights that might otherwise be rapidly forgotten [12]. By analyzing detailed incident report narratives and conducting follow-up investigations, organizations in high-risk industries can learn how specific work-system factors (e.g., breakdowns associated with technology, the environment, teamwork, task factors or the organization’s culture or resource management) contributed to incidents, directly or by influencing human performance. These insights enable organizations to identify and modify work systems to reduce the chance of events that cause harm. Incident reporting systems with these characteristics have a long history of successful use in the aviation, maritime, chemical, nuclear, and railway industries [13, 14].

Since 1965, incident reporting systems have also been widely used in U.S. hospitals to improve safety and alert risk managers of events that could lead to litigation [15]. Nurses see reporting as a professional responsibility and file most incident reports [16], but they experience barriers to reporting, particularly a lack of training, engagement, and feedback [17, 18]. Nurses are often uncertain about which events to report and what information to include [19]. They fear triggering punitive responses and doubt whether actions will be taken to improve systems of care [19–23]. Although hospitals typically receive thousands of reports annually [24, 25], many reports describe similar events and address lower-risk problems [16]. Higher-risk incidents are underreported [16, 26]. Additionally, only 32–65% of reports reveal actionable insights into the work-system factors that may constitute safety threats [27–30].

The optimal procedures for following up on hospital incident reports have not been described, and how hospitals handle reported incidents varies. Reports may be transmitted to managers in nursing units, hospital safety leaders, and/or risk managers, and they can trigger investigations locally or systemwide. Sometimes reporting leads to discipline of the individuals involved [20, 31–33], and feedback to reporters is rare [18, 22, 34–36]. Hospitals have emphasized the passive collection and classification of large numbers of reports, unlike in other industries, where systematic investigation, prompt action, and feedback to front-line personnel are goals [37–41].

To address these challenges, we designed the Safety Action Feedback and Engagement (SAFE) Loop, a novel intervention designed to transform a hospital’s existing incident reporting and follow-up systems into more effective tools for improving patient safety (as described below). The present work sought to refine the nascent SAFE Loop intervention through preliminary testing. While the SAFE Loop could apply to any patient safety problem, we focused on medication errors because medication administration is a core daily responsibility for nurses, medication errors are common on diverse types of nursing units [42–45], and nurses can detect problems in all stages of medication therapy (ordering, dispensing, administering, and monitoring) [46].

Materials and Methods

Overview:

Experts recognize several stages in development, testing, and dissemination for interventions designed to shape human behavior in real-world clinical settings. Stage I involves generating and refining a novel intervention via an iterative approach that allows the intervention to remain fluid, involves initial uncontrolled tests with small numbers of selected participants, and permits ongoing refinements in response to evolving findings. Later stages in intervention development involve fully powered experimental testing to establish internal and external validity [47–50].

To generate a novel intervention, developers often start by considering scientific principles, engaging stakeholders, and weighing possible components, and then bringing these together to define key attributes, as we did for the SAFE Loop (below). Next, experts recognize three initial steps in iterative testing and refinement for a nascent intervention:

1. Proof-of-concept testing occurs in small select samples, using a mixture of qualitative and quantitative data, to shed initial light on the intervention’s feasibility and acceptability to stakeholders;

2. Iterative refinement of intervention design and implementation plans enables developers to specify details of intervention components and prepare for full implementation; and

3. Structured pilot testing of the refined intervention yields additional data about feasibility, acceptability, and potential effectiveness [47, 48].

This paper reports steps (A) through (C) for the SAFE Loop intervention. We conducted these steps in preparation for a subsequent pragmatic trial that randomized 20 acute care nursing units to the SAFE Loop intervention versus usual care [51, 52].

Setting:

The work was conducted at a 915-bed major academic hospital in Los Angeles that is staffed by 1,980 nurses and consistently receives Magnet award recognition for excellence in nursing. The hospital employs several systems for improving patient safety. Relevant to medication safety, the hospital has a fully electronic medical record system with clinical decision support, intravenous infusion pumps with clinical decision support, Pyxis^™ automated medication dispensing machines (“Pyxis^™”), and barcoding systems, among others.

The hospital uses RL6: Risk (RL Solutions, Cambridge, Massachusetts), a risk management and incident reporting software package, for the reporting, classification, root cause analysis, tracking and trending patient safety events, and provision of feedback to leadership with expectations of dissemination to reporters. At baseline, when an incident report is filed from a nursing unit, the nursing unit leadership conducts a brief follow-up review that may include speaking with the reporter and other involved parties. Additionally, the medication safety program team reviews all incidents, has a group of clinicians review each incident to discuss risks and assign levels of harm, conducts more detailed investigations and plans specific actions around common or high-risk events, and summarizes events and mitigation activities annually.

Participants:

Study participants included nursing unit leaders and front-line nurses on three nursing units engaged in SAFE Loop development and testing.

A multidisciplinary research team oversaw development and testing. A physician researcher and a PhD-trained human factors engineer had experience studying patient safety and incident reporting (TN, TC) [16, 27–29, 32]. The team included the Chief Patient Safety Officer (ES), the Director of Nursing Research and Quality (BC), and the Pharmacist Manager for Medication Safety (DL). An external Advisory Board of national leaders in patient safety, medication errors, incident reporting, and nursing practice (see Acknowledgements) provided feedback on refinements in Step (B). Several research methodologists (NQ, KK, XZ) contributed to evaluation activities.

Human Subjects/Ethics Disclosure:

The Cedars-Sinai Institutional Review Board (IRB) reviewed and approved the current work in two stages. For the first stage, proof-of-concept testing (Pro00055431), the IRB determined that this was institutional quality improvement and not regulated as research. Individually identifiable data were not involved and consent was not obtained. For the second stage, iterative refinement and pilot testing (STUDY00001025), the IRB determining that the SAFE Loop intervention met criteria for a quality improvement activity whereas formal data collection activities constituted research. Written informed consent was obtained for nurse surveys and interviews. Consent was waived for reuse of existing data.

Key Attributes of the SAFE Loop:

To generate ideas about improving hospital incident reporting systems, the study team considered published literature on the systems’ current limitations, including our own prior research [16, 27–29, 32, 53]. In considering strategies for overcoming these limitations, the research team drew from literature on effective strategies for improving the clinical quality of care [54–59], literature on how stakeholder input and engagement can promote the adoption and use of novel interventions [60–63], and the team’s professional experience and knowledge of the institution’s systems and practices.

On this basis, we proposed the SAFE Loop intervention. The SAFE Loop activities are organized around a “Target Event,” meaning a specific medication safety event that nursing unit leaders choose after conferring with front-line nurses. The purpose of the Target Event is to bring a nursing unit together in deciding what medication safety events are particularly problematic, which may focus and promote greater engagement in reporting and subsequent follow-up and mitigation activities. Accordingly, the SAFE Loop has six key attributes (Table 1): (1) obtaining input from nurses on the unit about which safety problems to prioritize; (2) focusing on one Target Event safety problem that nursing units select; (3) teaching the nurses to write more informative incident reports; (4) prompting the nurses to report incidents relevant to the Target Event for a designated period; (5) establishing standardized investigative procedures to support development of a mitigation plan; and (6) providing feedback to the nurses about factors contributing to the Target Event and mitigation plan. The attributes occur during three phases: Engagement, Fact-finding, Action and Feedback (see Figure 1). These attributes build on, not replace, the existing reporting system.

Table 1:

Limitations of Current Hospital Incident Reporting Practices, Conceptual Rationale, and Attributes of the Safety Action Feedback and Engagement (SAFE) Loop

Limitations of Current Reporting Practices	Conceptual Rationale	SAFE Loop Key Attributes and Definitions
Lack of engagement by nurses and doubt that reports are used to improve safety deters reporting [20–22, 34].	Nurses are more likely to report events and adopt changes for problems that they find to be particularly common, severe, or preventable and/or impactful to the quality and efficiency of nursing practice.[60–62] Engagement is needed between nurses and safety leaders throughout the process of reporting, investigation, and improvement.	1. Input from Nurses: The SAFE Loop team solicits input from unit leaders and front-line nurses on which patient safety problem to prioritize for reporting, investigation, and mitigation.
Hospitals passively gather thousands of reports on low-priority events each year, making it hard to identify important threats that warrant follow-up and action [16, 24, 25].	Emphasizing one specific problem with a narrow scope may enable safety leaders to make the best use of investigative resources and obtain enough reports to understand the problem. Common, preventable, and potentially severe events should be a high priority.	2. Emphasis on a Selected "Target Event": The SAFE Loop focuses on each nursing unit’s unique Target Medication Event. These are errors and adverse events related to a selected, defined problem involving a particular drug class (e.g., opioids), population (e.g., kidney disease), type of error (e.g., duplicate therapy), and/or work-system factor (e.g., conflicting protocols).
Lack of guidance about what and how to report deters reporting and limits reports' usefulness [20–22, 34]. Gathering rich qualitative insights into contributing factors is central to successful incident reporting but hospital reports provide little detail on contributing factors [9–11, 15, 20–22, 24, 25, 29, 31, 33, 34], When reporting events involving actual harm, nurses are often concerned about being blamed or disciplined. Reports that address harm usually contain little information about contributing work-system factors [28].	If nurses receive guidance on what and how to report, they would be more likely to submit reports. Report narratives that include more details about contributing work-system factors may be more informative and actionable for efforts to redesign systems and improve safety. A mnemonic may help nurses to remember which types of details to include in incident reports. If reporting emphasizes near misses, this may reduce the risk that reporters might be blamed for the event.	3. Training in Enhanced Reporting: The SAFE Loop involves teaching nurses to write more informative incident reports that include details about contributing work-system factors. The SAFE Loop teaches nurses to organize the narrative using the Situation, Background, Assessment, Recommendations (SBAR) format.
Hospital incident reporting systems typically function as passive receptacles for incident reports. Nurses are often concerned about the time burden of reporting [20–22, 34].	Obtaining multiple related reports may provide a more thorough understanding of work-system factors. Asking each nurse to submit one report is a modest request. Having a deadline facilitates progress on a project that involves distinct phases.	4. Stimulated Time-limited Reporting: The SAFE Loop prompts nurses to report at least one Target Medication Event, if possible, during a designated period (in addition to any reports that nurses would usually submit).
There are no established procedures for conducting follow-up investigations of reported incidents [37, 38]. Few safety problems are unique to a particular nursing unit or hospital. Certain AHRQ Patient Safety Organizations systematically examine thousands of incident reports and generate syntheses of their findings.	When interviewed, front-line nurses and other relevant parties may share additional insights about work-system factors. Leveraging publicly available information on the causes of similar events at other hospitals and solutions that have been tested at other hospitals would reduce the work involved and enhance the success of investigative activities and mitigation plans.	5. Investigation and Integration of Information: The SAFE Loop integrates information from incident reports, followup investigations, interviews with front-line nurses and other relevant parties, and internal and external sources (including published literature and AHRQ Patient Safety Organizations). Standard procedures, including mini-Root Cause Analyses and Ishikawa diagrams, can facilitate investigation and integration and help design a mitigation plan.
Nurses seldom receive feedback after filing a report, and a lack of feedback is a major barrier to reporting [20–22, 34].	If nurses receive feedback and observe improvements in systems of care in response to incident reports, they may have more favorable perceptions of reporting, creating a positive feedback loop that promotes safer care [34].	6. Feedback to Nurses: The SAFE Loop Team provides feedback about the causes of safety problems and mitigation plans.

Figure 1: Safety Action Feedback and Engagement (SAFE) Loop with Six Key Attributes Designed to Maximize the Effectiveness of Reporting at Improving Patient Safety.].

Step (A) Proof-of-concept Testing:

Incorporating these key attributes of the nascent SAFE Loop intervention, the Chief Patient Safety Officer oversaw a preliminary test on two nursing units from January to March 2019. One was a neurosurgery/spine floor unit with 78 nurses, and the other was an adult medicine unit with 103 nurses. We selected these units based on having medical-surgical patient populations, engaged nursing staff, and an interested nurse leader who would serve as the intervention champion. Proof-of-concept testing occurred as part of normal operational quality improvement.

Intervention:

First, we met with unit leaders and nursing staff in each nursing unit to prioritize a patient safety Target Event based on staff nurses’ perceptions of the potential threat and their ability to effect change. Because nurses have limited time to add new responsibilities, we incorporated SAFE Loop implementation into routine daily nursing “huddles” where nurses hear announcements and briefly discuss key issues on the unit; “huddles” occur at the 7 am and 7 pm shift changes. Next, the safety leadership team encouraged nurses to submit examples of the Target Event and emphasized the reporting of near misses. We then collected incident reports about the Target Events occurring on the pilot nursing units. We met weekly with unit leaders and staff during the “huddles” to review the incident reports from each unit. We analyzed the reports and conducted mini-root cause analyses (i.e., root-cause analyses of abbreviated scope) [64]. We met weekly with unit managers and nursing staff to discuss findings from the incident reports and mini-root-cause analyses. Finally, we worked with unit managers to consider potential system changes, obtain feedback, implement tests of change to rectify work-system issues, and rapidly disseminate information.

Evaluation:

From February 5 through March 25, 2019, we examined the number of incident reports submitted per week on the pilot nursing units as well as the proportions of reports that were near misses (National Coordinating Council for Medication Error Reporting [NCC MERP] categories A and B) or involved a potential for or actual harm (NCC MERP categories C-I) [65, 66]. Additionally, before and after implementation, we surveyed nurses about the culture of safety in the nursing units. The surveys included three items (scale 1–5, 4 = agree, 5 = strongly agree): (1) My unit is assessing and refining processes to improve patient safety before events occur, (2) Staff in my unit do not feel like their mistakes are held against them, (3) Staff in my unit will freely speak up if they see something that may negatively affect patient care (the latter two items were adapted from the AHRQ Hospital Survey on Patient Safety) [67–69]. The culture of safety composite reflected the percentage of positive responses (agree or strongly agree) across the three items. Finally, we asked nursing unit leaders and front-line nurses open-ended questions to obtain feedback about the intervention.

Step (B) Iterative Refinement of Intervention Design and Implementation:

Building on the proof-of-concept testing, we refined and operationalized the SAFE Loop and specified implementation procedures—both during the development of the funding application and after funding. We broke the key attributes down into component steps, proposed a timeline for each step, defined roles in the SAFE Loop and associated tasks (Table 2), obtained ongoing input and feedback from safety experts and nursing stakeholders, and established how the team would engage unit directors and personnel on the nursing unit.

Table 2:

Roles of Personnel in the Existing Reporting System and the SAFE Loop Intervention

	Position	Roles in Existing Reporting System	ADDITIONAL Roles in SAFE Loop Intervention
	Nurses on Study Nursing Units	Spontaneously submit incident reports.	Provide input on the Target Events, undergo training in reporting, submit reports, receive feedback, and implement mitigation strategies.
	Nursing Unit Leaders	Administratively responsible for overseeing efforts to improve the safety of care on each nursing unit. Participate in analyzing reports, conduct any investigations, design and implement any solutions.	Select the Target Events and work with the SAFE Loop Team throughout implementation.
SAFE Loop Team	Incident Report Clinical Reviewer	Classifies events, assigns harm severity, identifies source nursing unit, and triages high-risk events to relevant departmental and unit leaders.
	Patient Safety Leaders (Hospital, Pharmacy, Nursing)	Assist nursing units with investigations, obtain resources, engage leaders of relevant departments, and overcome barriers. Oversee root-cause analyses of harmful events.	Part of the SAFE Loop team. Facilitate investigation of Target Events.
	Target Event Champion	Not applicable.	Obtains input from and trains nurses, identifies and extracts information from reports on Target Events, assists with investigations, integrates information from reports and other sources, drafts feedback to nurses.
	AHRQ Patient Safety Organization	Not applicable.	Serves as a resource to the SAFE Loop Team, providing information on events at other hospitals.

Further iterative refinements were continuously made throughout the process of preparing for and conducting the structured pilot testing; we report refinements and implementation under Step (B) and evaluation methods under Step (C), even though in reality, these steps were intertwined.

Refinement and structured pilot testing occurred from March 15, 2022 to November 21, 2023, in partnership with a nursing unit where registered nurses cared for adult patients with stroke and other neurological conditions. We selected this nursing unit because the leaders were enthusiastic about participating and because excluding the unit from the subsequent trial would not affect randomization. During the pilot testing, a reorganization of the unit increased the number of full-time nurses from 26 to 59.

Obtaining Input from Nurses and Selecting a Target Event:

For the pilot nursing unit, we selected a SAFE Loop “champion,” meaning an individual (generally a clinician) who promoted nurses’ engagement with the SAFE Loop and supported investigative follow-up procedures. The research team and champion met with the nursing unit leadership and then subsequently visited nursing unit huddles to invite nurses to introduce the SAFE Loop and suggest ideas for potential Target Events.

Choosing the Target Event was an iterative process that involved discussions between the research team, champion, and nursing unit leaders to balance multiple considerations. These included nursing leaders’ and front-line nurses’ perceptions of frequency, risk, and persistence despite prior efforts to address the issues; the ability to clearly and precisely delineate the event in a manner that nurses would recognize; and the likelihood that a few specific actions could be taken to mitigate the event, among others. Once the Target Event was selected, we revisited the huddles to inform the nurses and encourage them to submit incident reports about the Target Event over three months.

Training in Enhanced Reporting:

Our team’s nurse researcher and human factors expert created an online training program to introduce nurses to the SAFE Loop and demonstrate how to write more informative incident report narratives. As reported previously [70], the online training emphasized four key messages for nurses to: (1) participate in the selection and reporting of a high-priority Target Medication Event for their nursing unit; (2) emphasize the reporting of near misses over events involving harm; (3) describe in detail any contributing work-system factors in the narrative portion of the reports on Target Events, and (4) use the Situation, Background, Assessment, Recommendation (SBAR) format to organize their incident report narratives [71]. The training explained how work-system factors contribute to safety events, and defined five broad categories of factors (people, tools/technology, task, environment, and organization), drawn from the Systems Engineering Initiative for Patient Safety (SEIPS) model.[72] The training program incorporated the SBAR format because it is widely used in quality improvement as a framework for communication among healthcare team members. SBAR is an easy-to-remember, concrete mechanism that can frame critical communications about issues that require urgent attention and action, facilitating teamwork in patient safety situations [71]. We considered work-system factors to be an aspect of the Background. To engage nurses, the training program included quizzes about work-system factors. It also provided two in-depth hypothetical examples that compared “typical” versus “enhanced” incident port narratives that used the SBAR format and described work-system factors.

After input and feedback from the core research team, we conducted cognitive tests with four front-line nurses and refined our training. We programmed the novel training using EasyGenerator (Rotterdam, The Netherlands). We also created “badge buddies” (plastic cards worn under the ID badge for easy access) that summarized key messages from the training (Table 3).

Table 3:

SAFE Loop Training in Incident Reporting: Key Messages

Work-System Factors		Communication Using SBAR
Consider the factors that may have contributed to an event • People: knowledge, poor communication or teamwork • Tools/Technology: usable, broken equipment • Task: nonstandard task, too many tasks • Environment: noisy, too hot/cold, faulty room design • Organization: resources or training, faulty process, poor leadership/culture		• Situation: What is the event you are reporting? Is it an error, a good catch, or something else? • Background with Work-system Factors: Provide as much detail about the event and reasons it occurred. Include a description of why this event occurred (e.g., people, teamwork, communication, tools, technology, workload impact, environmental conditions, organizational policies, and/or problems with leadership) • Assessment: What happened as a result of this event? Did a patient get harmed? • Recommendation: What would you recommend to ensure that similar events can be avoided?

To deploy the training program on the pilot unit, we loaded it into an online platform, obtained a list of nursing staff, and enrolled the nurses in the mandatory training. Nursing unit leaders monitored the nurses' completion and provided encouragement where necessary.

Prompting Incident Reporting:

During the pilot testing period, we periodically revisited the nursing huddles to remind nurses to report the Target Events, monitored the medication safety incidents being reported, and let nurses know that we received and appreciated the reports they submitted. We also checked in often with the nursing unit leaders about what nurses were saying about the training, Target Event, and other aspects of the SAFE Loop.

Investigative Procedures:

We designed Target Event follow-up procedures to be brief but thorough. First, the champion reviewed the medication-related incident reports submitted during the six months preceding and during the pilot test. They considered whether each incident might be relevant to the Target Event, identified contributing work-system factors in the narrative, and used the Human Factors Analysis and Classification System for Healthcare (HFACS-Healthcare) to categorize them [73].

Concurrently, the champion and two other research team members interviewed nursing staff on the unit over two 90-minute sessions. During each session, nursing unit leaders directed one to three nurses at a time to a conference room on the unit, where we explored their thoughts about the chosen Target Event and contributing factors. Each nurse participated in the interviews for about 10–15 minutes, without being away from patient care or disrupting work on the unit for too long. As information about the Target Event and contributing work-system factors came to light, the champion and research team members identified additional types of clinicians with the potential to observe or contribute to the Target Event (e.g., pharmacists or physicians), and administrative leaders who oversaw departments that contributed to the work-system environment for unit nurses. We then conducted brief interviews with these individuals about the Target Events, starting with open-ended questions and gradually seeking feedback on what we had heard about contributing work-system factors.

At the same time, we obtained information about the history of the Target Event at the institution from the Pharmacist Manager for Medication Safety and the Chief Patient Safety Officer on our research team. They shared the results of prior investigative activities, shed light on contributing work-system factors, and provided insight into potential strategies to mitigate them and barriers.

Since a medication safety event at our hospital was likely to occur at other institutions, too, we considered external information, to leverage any previous lessons learned. The champion conducted a brief, directed (non-systematic) literature review to identify prior publications. The champion contacted the study’s partners at two AHRQ-designated Patient Safety Organizations (PSOs) to see what prior investigative activities and organizational activities or reports might have been relevant to the Target Event. The PSOs included the Hospital Quality Institute (engaged with patient safety and incident reporting at hospitals in California), and the Institute for Safe Medication Practices (engaged with medication safety and incident reporting at hospitals nationally).

To conclude the follow-up activities, the champion created an Ishikawa Diagram for the Target Event, listing the specific contributing factors identified via incident reports and interviews with nurses or other hospital stakeholders.

Producing a Mitigation Plan:

Next, we met with the nursing unit leaders, the Pharmacist Manager for Medication Safety, the Chief Patient Safety Officer, hospital nursing leaders, and leaders of other relevant departments as needed. We discussed the Target Event and the contributing factors, including those that were reported most often, appeared most preventable, and/or posed the most significant risk. We also discussed relevant institutional history and any information from external sources.

Nursing leaders ultimately decided what work-system factors to prioritize for mitigation and implemented the mitigation plan, based on their understanding of the issues and what changes may be feasible. Once the strategy was selected, the research team drafted a two-page mitigation plan, reiterating the Target Event, describing the contributing factors (including the Ishikawa Diagram), steps we had taken in the work, any outside information we identified, and the nursing unit’s final mitigation plan. After the nursing unit director approved the mitigation plan, we emailed it to all nurses on the unit.

Step (C) Structured Pilot Testing:

As a pilot study, the focus was on maximizing feasibility and acceptability to nursing unit leaders and nursing staff, and on refining the intervention to enhance effectiveness. Analyses were not powered to test for statistical significance.

Incident Reporting Practices:

We measured the number of weekly incident reports submitted on the units during six months of the pilot test (June 2, 2022, to December 12, 2022) and a prior six-month period (December 1, 2021, to June 1, 2022). Dual reviewers examined and reached a consensus on whether each report included work-system factors in the narrative and whether it pertained to the unit’s selected Target Event; the team adjudicated any ties. We also used severity scores assigned by the reporter to identify near misses (NCC MERP categories A to C) [65, 66, 74].

Surveys of Nurses:

Before and after implementation, we surveyed the front-line nurses by email, with 10 items described below (Appendix). To promote the survey, we visited the nursing units with snacks and provided encouragement. Additionally, nurses who completed each survey were entered into a lottery for Apple AirPods^™.

From the AHRQ Hospital Survey on Patient Safety (SOPS) Culture^™, Version 2.0 (English) [75], we derived two composite measures, Communication about Error (3 items) and Reporting Patient Safety Events (2 items), and two individual items, Blame-free Culture and Incident Report Submissions. We made minor changes in item wording to enhance specificity (e.g., changing “we” to “nurses” and “event reports” to “incident reports”). We also created a novel composite measure, Confidence in Reporting (2 items), and a novel individual measure, Time per Report.

Most individual items were scored on a scale from 1 (strongly disagree or similar) to 5 (strongly agree or similar). Composite scores are calculated as the percent of positive responses (agree or strongly agree), ranging from 0 to 100% (higher is better). Incident Report Submissions included five categories (0 to 4+ reports). Time per Report included 5-minute increments up to 90 minutes.

Interviews with Nurses:

After implementation of the SAFE Loop, we conducted one-on-one in-person interviews with three nursing unit leaders and three frontline nurses. We used a semi-structured interview guide with open-ended questions and follow-up probes, guided by the Consolidated Framework for Implementation Research and key publications on information to report about quality improvement interventions [76–79]. Questions were designed to elicit respondents’ views regarding contextual factors, implementation details, and other information recommended for the reporting of interventions related to quality and safety. Pairs of researchers trained in qualitative analysis analyzed interview transcripts in Dedoose using a combination of content analysis and qualitative inquiry, allowing them to identify key themes related to the nurses’ experiences and perceptions.

Results

Step (A) Proof-of-concept Testing:

Feasibility of Implementation:

Nursing unit leaders and front-line nurses were willing and able to select a Target Event. The nurses submitted several incident reports relevant to the Target Events. The unit leaders and nursing staff were receptive to discussing findings from the incident reports and mini-root-cause analyses. Finally, it was feasible to work with the nursing units to implement tests of change and disseminate mitigation plans.

Reporting Practices:

Before the pilot test on these two nursing units, an average of four incident reports were submitted per week and 6.25% represented near misses. During the pilot test, an average of 10 reports were submitted per week and 47% of reports addressed near misses.

Surveys of Nurses:

Before implementation, the Culture of Safety composite measure was 91% on one unit and 75% on the other. These percentages were 94% and 79%, respectively, after implementation.

Interviews with Nurses:

Feedback from unit leaders was strongly favorable: “SAFE Loop was an effective test of change as it engaged front-line staff to voice input and concerns to prevent near misses. It improved their workflow processes in terms of helping eliminate workarounds or delays. Staff had an increased feeling of empowerment.”

One front-line nurse commented, “We saw the value of these rounds. It made our job easier and more efficient, and we could see the long-term efforts of putting in the near misses now.” Another said, “[The advantages include] more reporting of ‘near misses’ and ensuring all staff receive follow-up reports.”

Step (B) Refined Intervention Design and Implementation:

Obtaining Input from Nurses and Selecting a Target Event:

During SAFE Loop implementation in the structured pilot study, unit directors initially proposed the Target Event as “challenges using the Pyxis^™ safely.” Discussions with nursing unit directors and nurses revealed that this encompassed a wide variety of concerns. For the SAFE Loop, the Target Event needed to occur fairly often (e.g., at least once per month), be preventable, and be clearly understood by front-line nurses. Consequently, the study team and nursing unit leaders narrowed down a list of more specific issues related to automated dispensing machines and reframed the Target Event as “wrong medication or wrong patient errors involving the Pyxis^™.” Further clarifications of the event definition were necessary during the investigation and mitigation planning activities to produce a focused and feasible set of actions in the nursing unit.

Training in Enhanced Reporting:

During the cognitive testing of the draft training program, nurses reported that they value information on how to submit more informative incident reports, since they receive limited training as part of nursing education. Testers found the training to be easily understandable and useful, and reported that they could complete the training in 15–20 minutes.

At the beginning of implementation on the pilot unit, 20 of 26 full-time nurses (77%) completed the online training program.

Prompting Incident Reporting:

During the investigative interviews with nurses (below), we learned that nurses might collectively recognize a high-priority safety issue on the unit but not spontaneously submit incident reports about it. In this case, the Target Event—the interruptions while using the automated dispensing machine—occurred so often as to seem a routine part of care rather than a reportable incident. As such, the interviews reinforced the submission of reports on the Target Events.

Investigative Procedures:

During the implementation process, we learned that nurses were generally eager to share their thoughts about the Target Event during the brief interviews. While speaking to individual nurses was helpful, nurses became more engaged when we talked with to two to three at a time, and discussions among the nurses appeared to stimulate more and richer insights. We also learned that identifying and understanding contributing factors were much easier to accomplish via interviews, rather than from incident reports.

The investigation yielded four incident reports directly relevant to the Target Event, 10 interviews (six with nurses on the unit, and four with other informants), 12 relevant peer-reviewed publications, and eight resources shared by the Patient Safety Organizations.

The incident reports and interviews yielded 21 unique contributing factors for this Target Event (Figure 2, Ishikawa Diagram). Of these 21 factors, 17 were derived from nursing interviews alone, one from an incident report alone, and three from both nursing interviews and incident reports. The contributing factor discussed most often was distractions or interruptions while using the Pyxis^™ machine, which was located in an open area by the nurse station.

Figure 2: Results of Structured Pilot Test of SAFE Loop Intervention: Contributing Work-system Factors Identified by Investigation (Ishikawa Diagram).

Black text signifies contributing factors identified only during interviews, Red text signifies contributing factors identified only from incident reports, and orange text signifies contributing factors identified and/or characterized by both data sources.]

Producing a Mitigation Plan:

The selection of the Target Event and subsequent investigation illustrate that nurses recognized numerous threats to patient safety on the unit and that a threat can have many diverse contributing work-system factors. As such, nursing leaders in the unit had to make pragmatic decisions about prioritizing these factors. They chose to focus on distractions while using the automated dispensing machine.

To address this, the nursing directors selected two mitigation strategies: (1) a new “quiet zone” sign on the automated dispensing machine to alert people not to interrupt nurses while they are retrieving medications, and (2) a new policy to defer all non-urgent calls (i.e., from patients’ families) during morning medication administration periods to reduce interruptions while using the automated dispensing machine. The nursing unit directors immediately implemented these changes.

Step (C) Structured Pilot Testing:

Reporting Practices:

In the six months before the pilot test, nurses submitted 48 medication-related incident reports, including 18 (37.5%) related to the Pyxis^™. Of 28 events for which severity was reported, 22 (79%) were near misses (7 unsafe conditions, three did not reach the patient, 12 did not harm). Across the 48 reports, nurses described 53 contributing work-system factors (Table 4).

Table 4:

Results of Structured Pilot Test of SAFE Loop Intervention: Contributing Work-system Factors Obtained from Incident Report Narratives

HFACS-Healthcare Tier	Contributing Factor Category	Baseline Reports, N (%)	Pilot Test Reports, N (%)
Organizational Influences	Organizational Climate	0	0
	Operational Process	0	0
	Resource Management	0	0
	Subtotals	0	0
Supervisory Factors	Inadequate Supervision	0	1
	Planned Inappropriate Operations	0	0
	Failed to Correct Problem	0	1
	Supervisory Violations	0	0
	Subtotals	0	2
Preconditions for Unsafe Acts	Physical Environment	0	0
	Tools/Technology	2	0
	Task	1	3
	Mental State	1	0
	Physiological State	0	0
	Fitness for Duty	0	0
	Communication	7	3
	Coordination	3	3
	Leadership	0	0
	Subtotals	14	9
Unsafe Acts	Decision errors	6	7
	Skill-Based Errors	26	17
	Perceptual Errors	0	0
	Routine Violations	0	3
	Exceptional Violations	1	0
	Subtotals	33	27
	Total Incident Reports	48	41
	Total Contributing Factors	53	39

During the 6-month pilot test, nurses submitted 41 medication-related incidents, including 8 (19.5%) related to the Pyxis^™. Of 30 events for which severity was reported, 25 (83%) were near misses (4 unsafe conditions, four did not reach the patient, 17 no harm). Across the 41 reports, nurses described 39 contributing work-system factors.

Surveys of Nurses:

Survey responses were generally similar before and after SAFE Loop implementation (Table 5). The Time per Report rose from 10 minutes to 15.5 minutes.

Table 5:

Results of Investigation Structured Pilot Test on One Nursing Unit: Attitudes of Front-line Nurses Toward Reporting

		Baseline Survey	Survey after Pilot Test
Response Rate		22 of 54 (40.7%)	29 of 59 (49.2%)
Composite Measures
Confidence in Reporting, % Positive Responses		70.5%	82.7%
Communication About Error, % Positive Responses		74.2%	70.1%
Reporting Patient Safety Events, % Positive Responses		70.5%	69.0%
Individual Items
Blame-free Culture, % Positive Responses		86.4%	82.8%
Incident Report Submissions, 0–4+
	None	15 (68.2%)	19 (65.5%)
	1	5 (22.7%)	5 (17.2%)
	2	1 (4.5%)	2 (6.9%)
	3	1 (4.5%)	1 (3.4%)
	4 or more	0 (0%)	2 (6.9%)
Time per Report, Mean Minutes (SD)*		10(5.8)	15.5 (10.0)

^*Excludes missing responses

Interviews with Nurses:

Nurses’ responses revealed that, before SAFE Loop implementation, attitudes towards incident reporting were generally favorable. However, nurses were uncertain about what changes were made to improve safety in response to incident reporting.

After SAFE Loop implementation, nurses voiced positive opinions of the potential for an increase in incident reporting to improve medication safety, and a nursing leader reported a high level of receptivity and investment from nurses in reporting the Target Event. See Box 1.

Box 1: Qualitative Interviews with Nurses in Structured Pilot Test: Themes and Excerpts from Interviews.

Comments Related to Incident Reporting on the Nursing Unit before SAFE Loop Implementation

• All nurses reported it was their responsibility to submit incident reports because:

  • “[Nurses are] the first in line to see [a] reportable event.”
• The culture of safety was generally positive:

  • “If we see something then we say something. We can tell the charge nurse or they really encourage us to just report anything, whether it’s medication related or [something else].”
• However, nurses’ opinions were mixed about whether they received follow-up information after submitting a report.

  • “I mean it really depends. So far, every time we’ve submitted forms, I’ve gotten the email [confirming that the incident report was received by the reporting system] for sure. And then just depending on the situation, I’ll have management come and say, ‘Hey, just so you know, we got the form and we’re looking into it’ kind of thing.”
  • “Really, once the file has been reported, we don’t really hear about what happens.”
• Nurses were less certain about whether submitting an incident report leads to changes in medication safety on the unit:

  • “I’d like to think that leadership sees it and does things about it. And sometimes I feel like they do, sometimes they talk about what had happened in our huddle, but it’s not for everything. So, I really don’t know how effective the incident reporting is.”
  • “I’m not sure because I don’t know if people, or who is reporting medication errors and what gets done about them. So maybe it’s effective, but I don’t see how it’s affected me before.”
  • “And have you found that that med report incident reporting is effective at improving medication safety? Yes. For pharmaceuticals.”

Comments Related to Incident Reporting on the Nursing Unit after SAFE Loop Implementation

• Nurses voiced a potential for an increase in incident reporting to improve medication safety, particularly the combination of reporting via the electronic interface and in-person to charge nurses.

  • “[I think] people learn from their mistakes, people learn through mistakes… So, if the errors are recorded properly, then I believe they’ll learn about it. And then, we can make plans on how we can further prevent the same incident from happening, and then also study how this kind of incidents happen, if they have this multiple times.”
• One nursing leader reported a high level of receptivity and investment from nurses in reporting the Target Event:

  • “I think they have it down. They have it down. Okay. Yeah, they’re very invested and making sure that when the whole system started, the whole CS-Safe entry process started, they saw it as a way of maybe potentially falling on someone’s desk who was removed from the clinical aspects. We may do nothing with it. So, we weren’t seeing the real meaty, detailed CS-Safe events, but now we’re seeing more and a lot more of the entries being made and capturing. We try to tell them, remove the emotion, just get to the facts. And we’re seeing a lot more of that. So, it’s slowly getting less personal and getting to the core of what the issue was. So, we’re seeing they are more receptive.”

Discussion

This analysis involved preliminary testing and iterative refinement of the Safety Action Feedback and Engagement (SAFE) Loop, a novel multifaceted intervention designed to overcome several limitations of hospital incident reporting systems and transform them into more valuable tools for improving patient safety. We quickly and easily implemented the intervention on two nursing units in a small initial proof-of-concept test, suggesting good feasibility. Nurses voiced substantial enthusiasm for the SAFE Loop approach, suggesting good acceptability. After specifying the SAFE Loop intervention and implementation activities in detail, incorporating ongoing feedback from nurses, and making many refinements, we conducted a structured pilot test of the full intervention using comprehensive mixed evaluation methods. The structured pilot test reaffirmed feasibility, including for specific activities like selecting a Target Event together with nursing units, training nurses in the SAFE Loop enhanced reporting methods, engaging nurses via visits to twice-daily “huddles” at shift change, conducting investigative interviews with front-line nurses about the selected Target Event, integrating information about the Target Event from diverse sources, developing a mitigation plan with the nursing unit leaders, and closing the feedback loop by distributing the mitigation plan to nurses on the unit. The analysis also demonstrated the feasibility of our evaluation methods, including evaluating the information incident reports capture about contributing work-system factors, and surveying nurses’ attitudes toward reporting. Collaborative interactions with nursing unit leaders and front-line nurses throughout the project reaffirmed acceptability, as did qualitative interviews performed after the pilot. As a whole, the testing and refinements laid the groundwork for initial efficacy testing via a randomized controlled trial.

The present analysis did not involve statistical testing because sample sizes lacked sufficient power and underpowered pilot tests to assess efficacy have high rates of false positives and false negatives relative to fully powered trials [80]. Our results, nonetheless, offer a few initial surprises. First, during the Target Event selection process, we learned that nurses might collectively recognize a high-priority safety issue on the unit, but may not spontaneously submit many incident reports about it. In this case, the Target Event—the interruptions during use of the automated dispensing machine—occurred so often as to seem a routine part of care, rather than a reportable incident.

Second, of 21 different work-system factors that nurses identified as contributing to the selected Target Event, we detected only one from incident reporting alone. In contrast, brief investigative interviews with 10 nurses discovered 20 more work-system factors. Three of the 20 factors discovered via interviews were also mentioned in incident reports, but we obtained much richer information during interviews due to the ability to ask follow-up questions, invite detailed descriptions, and explore nuances and diverse opinions. We suspect that many nurses felt more comfortable disclosing sensitive information during unrecorded face-to-face discussions with neutral third parties, than they did about putting this information in a written report that their supervisors would read.

A third unanticipated finding was that, while nurses chose a problem involving the Pyxis^™ machine as their Target Event, they submitted less than half as many Pyxis^™ incidents during the structured pilot test than in the preceding six months. Our team and the unit champion appeared frequently on the nursing unit to prompt nurses to report instances of the Target Event, but this may have backfired by offering an alternative way of sharing insights, i.e., via the interviews.

We will examine these issues in greater depth in our pragmatic randomized controlled trial testing the efficacy of the SAFE Loop intervention (1R01HS027455, clinical trial registration number) [51, 81]. To our knowledge, no prior randomized controlled trials have examined interventions designed to improve hospital incident reporting systems. Moreover, the SAFE Loop is unique in that it is designed to simultaneously overcome multiple limitations of these reporting systems. Most prior reporting interventions have focused on a single detail, such as educating nurses and physicians about reporting, or changing from paper to electronic incident reports [81]. The SAFE Loop’s emphasis on clinician engagement is crucial because incident reporting is voluntary. After all, safety emerges from the summary of countless small actions by clinicians, and nurses can provide rich insights into threats to patient safety. Critically, the SAFE Loop emphasizes the fundamental purpose of incident reports as qualitative tools for gaining insight into the causes of critical events, based on much more successful applications of incident reporting in other high-risk industries. We are unaware of prior efforts to train nurses to recognize and report work-system factors. Also, the SAFE Loop proposes standardizing procedures for leveraging the information in incident reports, closing what experts call the “safety action feedback loop” [34].

Despite these advantages, our present work has several potential limitations. We tested the SAFE Loop at one large urban academic center with a strong commitment to safety, so adaptation to and evaluation in other contexts will be needed. Implementation of the SAFE Loop will likely differ across nursing units and over time because the intervention involves tailoring to local needs, and the engagement of nursing unit leaders and nurses will undoubtedly vary. The types of safety problems that SAFE Loop could solve will be influenced by its current focus on reporting by nurses. Front-line nurses may have a skewed perspective relative to an objective process for prioritizing patient safety events (e.g., based on frequency, severity, and preventability) or patient preferences. Nonetheless, many safety problems are inherently local, such as poor communication and coordination of care, and nursing units would benefit from new strategies for solving their local problems. The intervention could be extended to physicians and other clinicians in the future. The SAFE Loop is resource intensive, requiring the efforts of nursing unit leaders, a safety champion, front-line nurses, and many others. As such, it may be best used to solve only a few safety problems in a given setting per year. As with many multifaceted interventions designed to improve the quality of care, it will be hard to isolate which components of the SAFE Loop are most important to any future successes.

In conclusion, this work demonstrated the feasibility and acceptability of a novel intervention, the SAFE Loop, for hospital nurses. An ongoing trial is testing its effects on reporting practices, nurses’ attitudes toward reporting, medication event rates and exploring nurses’ perspectives on implementation.