Implementation of an Innovative Technology Called the OR Black Box: A Feasibility Study

Abstract

Introduction. The operating room (OR) Black Box is an innovative technology that captures and compiles extensive real-time data from the OR, allowing identification and analysis of factors that influence intraoperative procedures and performances – ultimately improving patient safety. Implementation of this kind of technology is still an emerging research area and prone to face challenges. Methods. Observational study running from May 2017 to May 2021 conducted at Copenhagen University Hospital – Rigshospitalet, Denmark, involving 152 OR staff and 306 patients. Feasibility of the OR Black Box was assessed in accordance with Bowen’s framework with 8 focus areas. Results. The OR Black Box had a high level of acceptability among stakeholders with 100% participation from management, 93% from OR staff, and 98% from patients. The implementation process improved over time, and an average of 80% of the surgeries conducted were captured. The practical aspects such as numerous formal and informal meetings, ethical and legal approval, recruitment of patients were acceptable, albeit time-consuming. The OR Black Box was adopted without any changes in scheduled surgery program, but capturing hours were adjusted to match the surgery program and relocation of OR staff declining to provide consent was possible. Conclusions. Implementation of the OR Black Box was feasible yet challenging. Management, nearly all staff, and patients embraced the initiative; however, ongoing evaluation, information meetings, and commitment from stakeholders are required and crucial to sustain momentum, continue implementation and expansion. Ideas from this study can be useful in the implementation of similar initiatives.

Introduction

The operating room (OR) represents a high-risk environment, and despite a range of initiatives, the rate of surgical adverse events remains high, with 15% of all surgical patients experiencing an adverse event, with more than a third likely preventable.^1,2 The OR is a dynamic, complex, interprofessional, and multidisciplinary environment, where many potential factors (eg, distractions, technical skills, non-technical skills, communication, instrument failure, and patient-related issues) can interfere with and affect the intraoperative procedure and performance, and thus patient safety. With a new innovative and comprehensive data capture platform called the OR Black Box, it is possible to obtain real-time and transparent insights into the OR. The OR Black Box captures and synchronizes anesthesiologic, surgical, and environmental data from intraoperative procedures, allowing identification of multiple factors that independently or in combination may influence the procedure. Ultimately, the OR Black Box can improve surgical quality, staff performance, clinical outcomes, and thus patient safety.^3–5

Implementation of this kind of technology in the OR is still an emerging research area, which means that experiences, discussion, and guidance on how to implement are limited.

Implementation of technology, including artificial-intelligence (AI) driven technology, in the OR designed to improve patient safety and quality have increased significantly over the years. ⁶ One of the advantages of AI-driven technology in health care and in surgery are in generating new knowledge based on an extensive amount of data and making it possible to conduct comprehensive assessments far too expansive for human reviewers.^7–9 However, implementing new technology can be a challenging undertaking and requires knowledge and understanding of the factors, barriers, and mechanisms influencing the implementation process.^10–14 Commitment and engagement from stakeholders, usability of the technology, and the degree of workflow changes have been identified as important variables affecting the success of implementation of new health technologies.^12,13 Furthermore, acceptance of and willingness to use health information technology are major determinants of the success of an intervention. To develop a successful implementation plan, understanding and identifying these factors is critical. Several theories, models, and frameworks can be used to describe, understand, explain, and evaluate implementation. ¹⁵ Applying one of these relevant approaches in the implementation makes it possible to generalize and produce experiences, knowledge, and a shared understanding of the implementation process. ¹⁶ Feasibility studies, which are one way to evaluate whether an intervention is appropriate for further implementation, are indicated when few previously published studies exist or existing data using a specific intervention is sparse, ¹⁷ such as a comprehensive data capture platform like the OR Black Box.

The aim of this study is to evaluate the implementation and feasibility of the OR Black Box in a gynecological setting at a large Danish tertiary hospital. We also wish to highlight facilitators and barriers in the implementation process.

Materials and Methods

Design

This feasibility study was an observational study running from May 2017 to May 2021. Bowen et al.’s ¹⁷ 8 focus areas were applied to describe and evaluate the implementation and feasibility of the OR Black Box.

In this study, the word “implementation” is used in two ways: the broader meaning where implementation refers to the overall process, and in the narrow meaning where it is defined in the applied theoretical framework as one focus area, eg the execution of the intervention.

Setting

The Department of Gynaecology and Department of Anaesthesiology, Juliane Marie Centre, Copenhagen University Hospital – Rigshospitalet, Denmark. Rigshospitalet is a tertiary, highly specialized hospital and one of Denmark’s largest hospitals. Its Department of Gynaecology, which is a referral center for hospitals in eastern Denmark, specializes in endometriosis and gynecological cancer and performs about 2500 surgical procedures annually.

Technology and Data Collection

The OR Black Box captures and synchronizes patient-related, environmental, and intraoperative audiovisual data through the laparoscope, anesthesia equipment, four cameras and two microphones in the ceiling,.^3,18 All data is encrypted, synchronized, and stored securely according to Danish regulations. Data is transferred securely to Surgical Safety Technologies (SST) in Toronto, Canada, for further analysis and de-identification. Data is analyzed at SST using a combination of AI and human experts. This is a consent-based study, meaning that data was only to be captured if all OR staff and the patient had given informed consent. If not obtained, the OR Black Box could be turned off by staff in OR.

Study Subjects

All OR staff members, eg, OR nurses, nurse anesthetists, gynecologists, anesthesiologists, and others working at the surgical ward at both departments were eligible for participation. Individual consent valid for the 5-year project period was obtained from OR staff on an ongoing basis due to new hires and new patients.

Furthermore, porters and cleaning staff received written and oral information material about the project. These two groups of employees included more than 100 people who worked throughout the entire hospital. Their management gave overall consent due to exceedingly high staff turnover.

All patients undergoing gynecological surgery (endometriosis, cancer, and benign) at the gynecological department were eligible for participation, and were contacted prior to the surgery by a member of the research group, who provided oral and written information.

The inclusion criterion for data capture was written consent from both OR staff and patient age >18. The exclusion criterion was missing informed consent from OR staff and/or patients. Acute surgery and children surgery were excluded.

Information Materials

Information materials included informational posters and pamphlets about the OR Black Box initiative distributed by the research group. Furthermore, a website (www.rigshospitalet.dk/orblackbox) with details and information about the research project was developed in collaboration with the hospital’s communication department. Information material was available in both Danish and English.

Feasibility Framework

To structure, ensure transparency, and clarify whether or not the intervention was appropriate for further implementation, we assessed the feasibility of the OR Black Box in accordance with Bowen et al.’s framework ¹⁷ (Table 1). This approach addresses important aspects to be considered in the implementation of a new intervention. Bowen et al. describe 8 general focus areas that can be addressed depending on the outcome of interest.

Table 1.

Overview of the 8 General Focus Areas, Definitions, and Outcomes Applied in the Feasibility Study Adapted from Bowen et al. ¹⁷

Focus Area	Definitions	Outcomes
1. Acceptability	Extent to which targeted individuals and those involved in implementing programs are willing and interested in participating in study	Intent to use
		Individual satisfaction
2. Demand	Extent to which the intervention is likely to be used	Fit within organizational culture
		Actual use
3. Implementation	Extent to which the intervention is implemented as planned	Degree of execution
		Success or failure of execution
4. Practicality	Extent to which the intervention can be carried out with existing resources and commitment	Factors affecting ease or difficulty of implementation
		Amount, type of resources needed to implement
		Ability of participants to carry out intervention activities
		Positive/negative effects on target participants
5. Adaptation	Extent to which the intervention will be appropriate in a new situation	Degree to which similar outcomes are obtained in a new format
6. Integration	Extent to which the intervention can be integrated in the existing system	Perceived fit with infrastructure
7. Expansion	Extent to which the intervention can be expanded	Fit with organizational goals and culture
		Positive and negative effects on organization
8. Limited-efficacy testing	Extent to which the feasibility study is designed to test an intervention in a limited way	Maintenance of changes from initial change
		Intended effects of intervention on key intermediate variables

Ethics

The Danish Data Protection Agency approved this project in 2019 (registration number VD-2019-275). The Danish National Committee on Health Research Ethics reviewed the project in 2018 and found no ethical approval was needed in accordance with Danish legislation since biological material was not used and the project did not influence patient treatment (registration number H-18018801).

Results

A total of 152 OR staff, ie, OR nurses (n = 63), nurse anesthetists (n = 31), gynecologists (n = 42), and anesthesiologists (n = 16) were eligible to participate in the feasibility study.

Of the 306 patients eligible to participate, 289 underwent surgery in the OR with the OR Black Box. Due to the organizational challenges in the anesthesia department and the OR, 17 patients were relocated to other ORs on the day of surgery. The patients had a mean age of 51 ± 17. Gynecological cancer surgery was performed most frequently (Table 2).

Table 2.

Data on Patient Age and Type of Gynecological Procedure.

Patient Characteristics	n = 289
Age, years, mean (SD)	51 ± 17
Type of surgical procedure
Gynecological cancer, n (%)	176 (61%)
Endometriosis, n (%)	80 (28%)
Benign gynecology, n (%)	29 (10%)
Fertility, n (%)	4 (1%)

SD standard deviation.

Feasibility of the OR Black Box

Feasibility was assessed from May 2017 to May 2021 in accordance with Bowen et al.’s ¹⁷ eight focus areas (Table 3).

Table 3.

Definition of Feasibility Outcomes According to Bowen et al. ¹⁷ and the OR Black Box Project.

Focus Areas	Outcomes (Bowen et al. 17 )	Outcomes (OR Black Box Project)
1. Acceptability	Intent to use	Participation rates among stakeholders
	Individual satisfaction
2. Demand	Expressed interest	Local strategic goal
	Fit within organizational culture	Actual use
3. Implementation	Degree of execution	Percentage of surgeries captured and the use of the OR Black Box
	Success or failure of execution	Factors that influence data capture, eg:  Technical issues  OR staff not giving consent  Patient not giving consent  Unscheduled/acute surgery
4. Practicality	Factors affecting ease or difficulty of implementation	Key activities from May 2017–May 2021, such as preparatory meetings, technical support, and commitment from stakeholders
	Amount, type of resources needed to implement
	Ability of participants to carry out intervention activities
	Positive/negative effects on target participants
5. Adaptation	Degree to which similar outcomes are obtained in a new format	Modifications to normal workflow to accommodate the OR Black Box
		Adjustment of capturing hours to reflect elective operating schedule
6. Integration	Perceived fit with infrastructure	Impact on existing operating schedule, workflow, and surgeries
		Technical aspect of integrating intervention into our system
7. Expansion	Fit with organizational goals and culture	Implementation of OR Black Box at other departments, specialties, and hospitals
	Positive and negative effects on organization
8. Limited-efficacy testing	Maintenance of changes from initial change	OR Black Box implementation strategy
	Intended effects of intervention on key intermediate variables

Results in the 8 Focus Areas

• 1. Acceptability

We identified 3 stakeholder groups vital to acceptance of the OR Black Box:

• - Management teams: Departments of Gynaecology and Anaesthesiology (heads of department and head nurses); Juliane Marie Centre (director and head nurse); and Rigshospitalet Board of Management (CEO and deputy chief executives).

• - OR staff: OR nurses, nurse anesthetists, gynecologists, and anesthesiologists. Individual consent obtained from each individual, though overall group consent obtained for porters and cleaning staff.

• - Patients: gynecological patients scheduled for surgery in the OR with the OR Black Box.

There was a broad willingness and interest to participate in the intervention among all stakeholders.

Inclusion and obtainment of informed consent from OR staff and patients was an ongoing process due to staff turnover and daily change of patients. The overall participation rates among stakeholders were:

• - Management: 5 out of 5 levels (100%)

• - OR staff, individual consent: 141 out of 152 staff (93%)

  • • OR nurses: 56 out of 63 (89%)
  • • Nurse anesthetists: 27 out of 31 (87%)
  • • Gynecologists: 42 out of 42 (100%)
  • • Anesthesiologists: 16 out of 16 (100%)
• - OR staff, group consent: 2 out of 2 groups (100%):

  • • Porters (100%)
  • • Cleaning staff (100%)

• - Patients: 300 out of 306 patients (98%)

• 2. Demand

The context for this feasibility study fits well in terms of the organizational structure of the Copenhagen University Hospital – Rigshospitalet strategy, especially the Department of Gynecology, which includes a strong research environment due to the nature of a university hospital with global excellence in oncological gynecological surgery. The OR Black Box project was solidly committed by all levels of management which made the actual use of the technology possible.

• 3. Implementation

From September 2020 to May 2021, there were 229 out of 289 (80%) surgeries captured. The percentage of surgeries captured changed to 85 out of 126 (67%) from September 2020 to December 2020, to a 144 out of 162 (89%) from January 2021 to May 2021. Sixty out of 289 (20%) surgeries were not captured, either because the OR Black Box was turned off or there were technical issues.

• More specifically, the system was turned off:

• • 35 times due to OR staff (OR nurses and nurse anesthetists) declining to provide consent (58%)

• • 2 times due to unscheduled/acute surgeries (20%)

• • 6 times due to patients declining to provide consent (10%)

• • 7 times due to technical issues (12%).

The overall commitment of management, nearly all staff, and patients means that the OR Black Box was implemented as planned. Challenges (eg, addressing staff that declined to give consent, resistance from staff, and funding) and strengths (eg, management and technical support) that occurred during the implementation process were identified, resulting in learning points to apply in future implementation processes.

• 4. Practicality

Timeline and key activities are outlined in Figure 1.

Figure 1.

Timeline and key activities of feasibility study.

In late 2016, initial deliberations about the OR Black Box began. In 2017, the research group conducted four introductory information meetings with management at all levels to discuss the OR Black Box and what its implementation would require in terms of resources (eg, financial and technical support), in addition its suitability in a Danish setting. These meetings emphasized the overall aim of implementing the OR Black Box and also creating an understanding of what the project consisted of and what using the OR Black Box entailed. Once all levels of management approved its implementation, the research group held about 5 information meetings with approximately 200 OR staff, including outpatient nurses, to ensure their involvement and foster a supportive climate.

We applied to the Danish Data Protection Agency and the Danish National Committee on Health Research Ethics for approval of the project, which the former gave after 18 months, while the latter found that ethical approval not required according to Danish law. In addition, funding applications were submitted to cover technical costs and research protocols were drawn up.

In 2018, an implementation plan for the OR Black Box was developed, which included meetings with the SST engineering team in Canada and with the IT department at the Centre for IT and Medical Technology, Capital Region of Denmark to determine the technical aspects of installation. Prior to implementation, SST engineers visited Denmark to ensure that the hardware and software were properly installed and configured to achieve optimal user friendliness. Installation did not require cancelling any surgeries because it occurred on a weekend in April 2020. From April 2020 to July 2020 technical adjustments were conducted.

In August 2020, data capture was optimized over one month to make certain that both the research group and the OR staff were sufficiently familiar with the system and setup. Further to ensure that all technical aspects ran smoothly. In September 2020, the OR Black Box began capturing data, supported by SST and the local IT department.

Concurrently, the research group began planning its patient recruitment strategy. The inclusion process was time-consuming (about 30 minutes per patient) and involved providing information and including patients on a daily, individual basis.

From 2018 to 2021, the research group had approximately 30 formal meetings and numerous informal meetings with all OR staff and management. The primary goal was to provide a progress update and, in particular, to receive feedback from OR staff. The research group also held information meetings with porters, cleaning staff, and their management. From May 2021 and onwards, in close collaboration with hospital management and the management of other surgical specialties, the research group conducted strategic planning on how to expand to other ORs at other specialties.

• 5. Adaptation

After a meeting with management in December 2020, staff who declined to provide consent were relocated. This resulted in an 11% increase in the OR Black Box use, and thus increased data capture.

• 6. Integration

The OR Black Box was integrated into the OR without any workflow changes or adjustments in scheduled surgeries. In collaboration with SST and the local IT department, the integration and maintenance of hardware and software were conducted outside daytime working hours.

• 7. Expansion

The OR Black Box, which is currently being used in 16 hospitals in North America and Europe, fits well with the local strategic goals of the Juliane Marie Centre and Rigshospitalet’s overall strategic plan and goals. This involves improving research and patient treatment through innovation and digitalization eg AI, and scaling health initiatives up worldwide. The overall vision is to accelerate future health care solutions, eg OR Black Box, to benefit patients globally, and to openly and actively involve relevant partners and also the users who will ultimately benefit.^19,20 The research group is part of the Surgical Safety Network, helping to further this vision.

• 8. Limited efficacy

Implementation did not involve limiting the project. Few changes were made in capture hours and not many staff were relocated because they declined to provide consent. When planning expansion to other ORs and specialties, the structure of the consent-based approach needs to be discussed and possibly altered.

Discussion

This study evaluates the feasibility of the implementation of an innovative and comprehensive data capture platform called the OR Black Box in a single gynecological OR at a highly specialized tertiary hospital in Denmark. We evaluated the feasibility and implementation of the OR Black Box over a span of four years, from the initial idea until 8 months of data capture. Overall, this study shows that the OR Black Box project had a high level of acceptability among stakeholders, ie, management, most of the OR staff, and patients. This is an important factor because acceptance is imperative when introducing innovative technology into the health care system, and successful stakeholder engagement is known to be an integral and impactful component of technology implementation in health care.^13,21–23 Despite high acceptability, a discrepancy in consent rate was found across staff professions. Overall, the nurse group (OR nurses and nurse anesthetists) declined consent at a higher rate than the physician group (surgeons and anesthesiologists). Some of the primary concerns that participating staff had, especially among those who declined to provide consent, were data security, deidentification, and legal issues. These concerns, mainly verbally expressed during formal information meetings held by the research group but also informal encounters with staff and management, are in line with findings at other OR Black Box sites in North America and Europe,^24–26 as well as from other initiatives with audio- and video recordings.^27,28 To expand knowledge on these perceptions, qualitative studies should be designed and carried out. A plausible explanation of the difference in accept could be that the project is driven and initiated by physicians. It will be a priority for future implementation to ensure a balanced mix of professions by identifying champions across staff professions who can help build support within their professions,^29,30 troubleshoot problems that arise during implementation, and provide information to colleagues. However, an important point to emerge is that implementing new initiatives and incorporating changes in healthcare depends on local condition, is time-consuming and a complex process, particularly where a large number of staff professions and change processes are involved. ³⁰ Furthermore, in order to adopt a new initiative there will always be individuals that are more apt to adopt the initiative than others and some will first adopt and willing to embrace the initiative after a certain amount of time. ³¹

Despite a high level of demand from management and acceptability from stakeholders, the implementation and practical execution of the OR Black Box varied over time. This mainly reflected in data capturing. Data from 33% of the operations was not captured in the first four months, mainly because some of the OR staff declined to provide consent (11 out of 152 people). Involvement of local clinical management at a formal meeting at this stage raised the awareness of the problem and increased the efforts to resolve it. The OR staff that did not provide consent were relocated to other ORs without an OR Black Box, resulting in an 11% increase in captured operations.

Essentially, implementation of the OR Black Box is bound by existing cultures and beliefs, which is why clearly delineating who has access to the data and how, why and when data from a capturing device will be shared and used, is crucial. To ensure transparency, implementation requires clearly and continuously informing everyone involved about what will happen to their data. An important, common reason why technologies are abandoned is that end-users do not see any clear benefits or the technology represents a threat to their professional identity. ¹⁴ However, if data is handled professionally, privacy is respected, and data is protected, fear of medical litigation or loss of professional status can be assuaged. ³²

Based on our experience, implementing and maintaining the sustainability of an OR Black Box will require important actions such as involving key stakeholders (management and OR staff) at an early state, continuously underlining the potential for positive changes in surgical safety, and addressing ethical and legal issues.

Furthermore, it is important to ensure transparency in the use of data and, finally, creating a strategy to give staff information and inform them about results. A major priority for the group in charge is to develop, in close collaboration with the various management levels, a strategy for presenting the OR Black Box data and provide valuable feedback that meets the needs of the various staff groups. In situations with adverse events or complications, another priority is to create an open and non-punitive culture of learning for all health care professionals. Complications have consequences not only for the patient but can also have a significant impact on the OR staff. ³³ A safe space for speaking up and fostering psychological safety should be created that allows all OR staff to receive feedback without any fear of repercussions or blame. ³⁴

The use of human resources (eg, scheduling and preparation of meetings, technical support and obtaining consent from OR staff and patients) and commitment from stakeholders, especially management at all levels, were crucial factors for the implementation and ability of carrying the OR Black Box project out. Overall the practical aspects of the OR Black Box were acceptable, albeit time-consuming. In general, the practical aspects of this study involved numerous formal and informal information meetings with both management and OR staff held by the research group to ensure mutual feedback on the implementation process, re-examining whether commitment and support still existed for the project, and repeating and emphasizing the overall purpose of implementing the technology. Moreover, involving management at all levels is a key factor since various staff groups refer to different levels of management.

When expanding to other surgical specialties, it should be discussed whether the structure of the consent-based approach should be altered. Even though the group that did not provide consent was relatively small in this study, numerous formal and informal meetings were held with the clinical management to address concerns among this small group and their actual influence on the data capture. If the OR Black Box is described as a way to improve quality, and participation is made mandatory, the staff who are reluctant to participate will be left with no choice. Ideally, a strategic plan should be in place to respond if staff are averse to participating. Likewise, to achieve the goal of improving intraoperative procedures and thus surgical safety, 24-hours data capturing, including data from elective and acute surgeries, should be a priority.

Strengths

This study was designed as a feasibility study that applied Bowen et al.’s framework ¹⁷ helping to structure, describe, and evaluate the process and critical feasibility steps surrounding the OR Black Box. Using this framework increases the possibility for reproducibility, transparency, and producing generalizable knowledge. This approach may be useful for others aiming to implement new and challenging technologies like a comprehensive data capture platform in the OR. We believe that these experiences provide the foundation for further implementation of the OR Black Box (or similarly innovative technologies) in terms of structural, legal, and ethical considerations, namely consent, data management, and the involvement of stakeholders. In an era characterized by exponential growth in technologies that use AI and machine learning, focusing on acceptance, barriers, and user friendliness for end-users is essential.

Limitations

Implementation of a comprehensive data capture platform in the OR is still an emerging research area. Before any conclusions are drawn on how it can improve quality and safety, it should be more widely implemented, and there is a need to continuously describe, discuss, and evaluate its implications. This observational study does not examine why some staff groups decline to provide consent, though this topic would benefit from further exploration in qualitative studies. Furthermore, the study lacks external validation of the results; however, our results are in line with similar studies looking at OR Black Box implementation in a clinical setting.^22,24

Perspectives

The major advantages of implementing the OR Black Box include the opportunity to identify intraoperative factors that, independently or in combination, influence intraoperative performance and/or procedures, as well as factors that the OR staff does not necessarily notice or report.^35,36 Video-recordings, which are a part of the OR Black Box set up, are a powerful tool for documenting in a transparent, real time, manner, what happens in the clinical setting.^37,38 Recordings from the OR can be replayed as often as needed, and assessed by multiple reviewers, giving OR staff the ability to look back at their performance, in addition to advancing learning and the sharing of experiences. Finally, the OR Black Box has the potential to promote a culture of surgical safety and to be a powerful tool for quality improvement initiatives.^37,39 Sharing learnings from surgical incidents has been shown to improve patient care, ³³ and a positive safety culture has been linked to a reduction in adverse events.^40,41

Conclusion

In conclusion, implementation and use of an innovative and comprehensive data capture platform like the OR Black Box in a gynecological setting was feasible. Stakeholders, ie, management, nearly all OR staff, and patients embraced the initiative. Continuous evaluation, information meetings, and adjustments were nonetheless required to sustain momentum. The practical aspects of an OR Black Box are time-consuming and complex, requiring attention from all stakeholder levels, especially commitment from management early on in the process. Results and ideas from this study can be useful in the implementation of similar technological initiatives in the health care system.

ORCID iD

Kjestine Emilie Møller https://orcid.org/0000-0002-2623-7562