Skip to main content
Sage Choice logoLink to Sage Choice
. 2023 Dec 5;31(1):111–122. doi: 10.1177/15533506231218962

A Review of Cognitive Support Systems in the Operating Room

Zhong Shi Zhang 1, Yun Wu 1, Bin Zheng 1,
PMCID: PMC10773165  PMID: 38050944

Abstract

Background

In recent years, numerous innovative yet challenging surgeries, such as minimally invasive procedures, have introduced an overwhelming amount of new technologies, increasing the cognitive load for surgeons and potentially diluting their attention. Cognitive support technologies (CSTs) have been in development to reduce surgeons’ cognitive load and minimize errors. Despite its huge demands, it still lacks a systematic review.

Methods

Literature was searched up until May 21st, 2021. Pubmed, Web of Science, and IEEExplore. Studies that aimed at reducing the cognitive load of surgeons were included. Additionally, studies that contained an experimental trial with real patients and real surgeons were prioritized, although phantom and animal studies were also included. Major outcomes that were assessed included surgical error, anatomical localization accuracy, total procedural time, and patient outcome.

Results

A total of 37 studies were included. Overall, the implementation of CSTs had better surgical performance than the traditional methods. Most studies reported decreased error rate and increased efficiency. In terms of accuracy, most CSTs had over 90% accuracy in identifying anatomical markers with an error margin below 5 mm. Most studies reported a decrease in surgical time, although some were statistically insignificant.

Discussion

CSTs have been shown to reduce the mental workload of surgeons. However, the limited ergonomic design of current CSTs has hindered their widespread use in the clinical setting. Overall, more clinical data on actual patients is needed to provide concrete evidence before the ubiquitous implementation of CSTs.

Keywords: cognitive support, surgery performance, mental workload, anatomical accuracy, patient outcomes

Introduction

Modern surgery has come a long way. From the first successful open surgery to the laparoscopic and robotic-assisted procedures standard in today’s operating room, the success of modern surgery is no anomaly; it is deeply entwined with the development of surgical technology. Ranging from aseptic techniques to intra-operative imaging, technology has always been an effective support for surgeons. 1 Novel surgical procedures such as minimally invasive surgery (MIS) also benefit the patients, as it is often associated with lower patient morbidity, infection, and a shorter hospital stay.2-4 However, it is important to highlight that surgical techniques today have increased in complexity compared to previous decades. Procedures like endoscopic or laparoscopic surgery have restricted the surgeon’s visual field to 2-D instead of the traditional 3-D view in open surgeries. The disparity in the visual field introduces problems such as image-reality incompatibility, loss of depth perception, and abnormal haptic feedback; additionally, novel surgical techniques also require new machines to be introduced to the operating room (OR), which may lead to extra staff to monitor the devices. The mental strain caused by MIS and excessive signals present in the working environment of the surgeon may lead to cognitive overload, which causes surgeon performance to decrease 5 ; this induces surgical errors while also extending operational time.6,7 Another potential cause to surgical errors involves communication. Irrelevant conversations within the OR are associated with poorer team collaboration and performance,8,9 while also extending operational time; equipment-related distractions is associated with higher stress and cognitive load. 8 Compared to open surgery, communication in MIS is more unclear and more frequently directed at equipment instead of the procedure, likely due to the increased equipment and staff used to support the MIS. 10 . Parker et al 11 reported a longer duration of open transforaminal lumbar interbody fusion surgery when comparing MIS to standard open surgery. Experiments that mimicked both MIS and open surgery environments reported a decreased efficiency in task completion and an increased mental workload. 12 Accompanying the increased cognitive load are increased probability of adverse events. In the top 559 hospitals located in the US, close to 81.7% of intraoperative errors are related to surgeons’ cognition being overwhelmed, each offense could cause up to 125,000$ in penalty. Thus, fatigue-related events, such as retained foreign objects or wrong surgical sites would be significantly reduced if surgeons received cognitive support. 13 To counteract the increased cognitive load of surgeons and to reduce surgical errors, cognitive support technology (CST) has developed since the early 2000s and has seen significant growth in recent years to aid the decision-making of physicians and surgeons. However, despite its importance, technology in this field lacks a systematic review and assessment of the impacts on surgical performance. We aim to review the current status and trends of CST in surgery. CSTs covered in this review pertains to information presentation for diagnostic support and visualization for enhancing performance. Specifically, we include the software and hardware employed for data presentation, analysis in surgery, and impact on surgeon performance and patient outcome.

Method

Search Criteria

The inclusion criteria include a list of terms that can be categorized into three major groups: cognitive support, surgery, and digital reality. A figure of an example search in PubMed is provided. Related terms are connected with Boolean operators; terms in a column are connected with the “OR” operator, while terms in a row are connected with the “AND” operator (Figure 1). A CST is a technology that reaches a high level of innovation and technological complexity, often involving programming or robotic design aimed at reducing cognitive load of surgeons. Exclusion criteria were set to be checklist articles because an overwhelming number of checklists was included in the literature search. In this review, a checklist defined as a digital or non-digital element used to verify aspects of the surgery in the OR. Surgical checklists were often specific to a large variety of surgeries which required a large amount of background information. There were also additional checklists assessing other checklists which were too complicated for the purpose of this review. Literature included in this review was gathered through a literature search in medical databases, including the Web of Science, PubMed, and IEEExplore, articles published until May 21, 2021 were included. An effort was made to search for clinical trials or randomized controlled trials, but a few exceptional model and architectural articles are included to provide a framework understanding of the topic. The article search and selection process was completed by a single reviewer (ZSZ). Mendeley reference manager (Relx, New York, USA) was used to organize the references. Studies were included in this review if it targeted the peri-operative phase in surgery, contained the development software or device that aimed to reduce surgeons’ cognitive load, and experimental trials that assessed the efficacy of the technology. Studies that only included a phantom trial, an animal trial, or used synthetic patients were included. Articles were first screened by title and abstract assessments. The remaining articles then underwent full-text analysis to further evaluate relevant articles.

Figure 1.

Figure 1.

Advance search criteria for PubMed.

Data Analysis

A single reviewer (ZSZ) examined key characteristics of each article including data source, development status, experiment type, sample size, phase of surgery, and the type of procedure. Full-text articles selected were organized with Microsoft Excel (Microsoft Corporation, WA, USA). Papers were further stratified into different subgroups by the parameters they measured; clinical outcomes, patient outcomes, and OR time variations were all taken into consideration. Discussion of the advantages and disadvantages of CSTs were gathered from the discussion and conclusion of the articles. Interpretation of the study results was based on the result section of each paper. A secondary reviewer (YW) verified the accuracy of the content. In events where the article cannot be classified into a set group, or parameters on the article was unclear, then a discussion was to occur with two other reviewers (BZ, WL) until an agreement was made.

Results

Overall, 481 citations were found throughout the literature search; after electronically removing 22 duplicates using Endnote (Clarivate, London), 459 remaining articles underwent further screening. Two independent reviewers (ZSZ, LW) screened the citations based on title and abstract, 369 articles were removed, leaving 35 for full-text review. During full-text review, 14 articles were removed: seven lacked quantitative data; six were overview articles; one was a checklist. An additional 19 articles were found through relevant searching, summing the total number of articles included in this review to be 40 articles. Each article had at least one relevant data describing cognitive load reduction. The complete article evaluation process is outlined as Figure 2.

Figure 2.

Figure 2.

Flow chat of the article selection process.

Data Source

CSTs use input data from a few different varieties to generate 3-D images for enhanced visualization or to assist patient diagnoses. Medical imaging databases such as XNAT are developed for imaging upload and information extraction. 14 For all the papers in this review, an overwhelming 23 (58%) CSTs used X-ray-based images such as computed tomography (CT) as their primary data source. 6 (15%) CSTs adopted MRI as their data source while another 6(15%) CSTs used ultrasound; however, they are not as common as the CT scans due to a variety of different reasons relating to costs and accuracy. 2(6%) CST uses other types of input such as live videos with manual markers with its application in nursing, and physiological parameters when analyzing cognitive load. 15 7(18%) CST did not provide specific sources of their input data, which may be due to their employment of previously established data from other sources (Table 1).

Table 1.

Characteristics of Included Studies.

Study Data Source Type of Development Type of Experiment Sample Size (n) (Staff; Patient) Phase of Surgery Procedure
M¨arz et al (2015) Non-specific Software development Development (NS; 184) Pre-operative Hepatectomy
Schoch et al (2016) MRI Software development Development (NS; 624) Pre-operative Cardiac surgery
Thienphrapa et al (2019) CT Software development Development (NS; NS) Intra-operative Video-assisted thoracoscopic surgery
Willekes et al (2008) CT Hardware development User tests (NS; 2) Intra-operative Video-assisted thoracic surgery
Hansen et al (2008) Ultrasound Software+hardware development Development (NS; NS) Intra-operative Hepatectomy
Lang et al (2005) MRI or CT Software development Clinical tests (2; 25) Pre-operative Hepatectomy
Hagenah et al (2018) Ultrasound Software development Development (NS; 10) Pre-operative Cardiac surgery
Chi et al (2018) CT Software+hardware development User tests (1; 6) Intra-operative Cardiac surgery
Abhari et al (2015) MRI Software+hardware development User tests (21; NS) Pre-operative Neurosurgery
Hagenah et al (2019) Ultrasound Software development Development (NS; 24) Pre-operative Cardiac surgery
Simoes et al (2013) CT Software+hardware development Development (NS; NS) Pre-operative Robot-assisted laparoscopic surgery
Hayashi et al (2017) CT Software development Clinical tests (3; 26) Pre-operative Laparoscopic gastrectomy
Azagury et al (2012) CT Software+hardware development Clinical tests (3; 3) Intra-operative Natural orifice transluminal
Endoscopic surgery
Baumhauer et al (2008) CT Software development User test (1; 3) Intra-operative Laparoscopic partial nephrectomy
Birkfellner et al (2003) CT Software+hardware development Development (NS; NS) Intra-operative Skull base surgery
Collins et al (2014) MRI Software development User tests (NS; 1) Intra-operative Laparoscopic myomectomy
Conrad et al (2016) CT Software+hardware development User tests (NS; 1) Intra-operative Laparoscopic rescue of failed portal vein embolization
Debarba et al (2010) CT Software development Development (NS; 4) Pre-operative Hepatectomy
Haouchine et al (2013) CT Software development Development (NS; 2) Intra-operative Liver minimally invasive surgery
Marzano et al (2013) CT Software+hardware development User tests (NS; 1) Intra-operative Pancreatico-duodenectomy
Osorio et al (2010) CT Software development Clinical tests (NS; 24) Intra-operative Laparoscopic surgery
a Linxweiler et al (2020) CT Software development Clinical tests (2; 100) Intra-operative Sinus-related surgery
Ntourakis et al (2015) CT Software+hardware development Clinical tests (NS; 3) Intra-operative Hepatectomy
Ogawa et al (2019) CT Software+hardware development Clinical tests (1; 41) Intra-operative Total hip arthroplasty
Buck et al (2005) MRI or CT Software development Development (NS; 39) Pre-operative + Intra-operative Cardiac surgery
Dias et al (2020) Live video Hardware development User tests (45; NS) Intra-operative Intubation
Dilley et al (2019) Non-specific Software development User tests (36; NS) Intra-operative Laparoscopic cholecystectomy
Dixon et al (2013) CT Software+hardware development Clinical tests (33; 1) Intra-operative Endoscopic transnasal skull base navigation
a Grasso et al (2015) CT Software+hardware development Clinical tests (3; 180) Intra-operative Lung biopsie
Invernizzi et al (2020) Non-specific Software development Clinical tests (3; 30) Pre-operative Breast cancer related lymphedema diagnosis
Jeon et al (2014) Ultrasound Software+hardware development User tests (20; NS) Intra-operative Cardiac surgery
Marcus et al (2015) CT Software+hardware development User tests (50; 1) Intra-operative Laparoscopic neurosurgery
Diana et al (2017) MRI Software development Clinical tests (10; 58) Pre-operative + Intra-operative Gallbladder surgery
Rosenthal et al (2005) Ultrasound Software+hardware development User tests (1; NS) Intra-operative Breast biopsy
Ru¨ger et al (2020) Ultrasound Software+hardware development User tests (20; 1) Intra-operative Ultrasound-guided needle placements
Wei et al (2016) CT Software development Clinical tests (1; 40) Intra-operative Percutaneous kyphoplasty
Wilson et al (2013) Non-specific Software+hardware development Clinical tests (34; 16) Intra-operative Tension pneumothorax decompression
Patel et al (2023) Non-specific Hardware development User tests (15; NS) Intra-operative Robotic suturing
Barragan et al (2022) Physio Signal+ Live video Software development User tests (8; NS) Intra-operative Robotic surgery
a Rompianesi et al (2023) Non-specific Hardware development Clinical test (61; NS) Pre-operative+intra-operative Hepatectomy

Development indicates the lack of human participants; user test indicates the incorporation of surgeons but no patient participants; clinical test indicates the participation of both surgeons and patients are involved in the study.

aLarge sample size studies.

Phase of Surgery

Different CSTs act on three main phases of surgery: preoperative, intraoperative, and post-operative. Preoperative includes all planning activities before the onset of the surgery; intraoperative refers to surgical performance during an operation. CSTs categorized to the different phases of surgery act as interventions or visual assistance to the surgeon. 12 (32%) CSTs are responsible for preoperative interventions and 27 (73%) CSTs are responsible for intraoperative interventions. No CST was found to assist in patient rehabilitation after the surgery had been completed.

Types of Surgery

CSTs are present in a wide variety of procedures. General surgery, thoracic surgery, neurosurgery, urology surgery, otolaryngology surgery (ENT), and orthopedic surgery are all applicable for CST use. While it is difficult to categorize them into specific groups, it is certainly clear that there is an emphasis on MIS (n = 11) (Table 1).

Surgical Performance Outcome

A total of 26 (65%) literature reported their findings numerically. Out of the 26, 7 studies reported their findings in a percentage or an error distance, 16 studies reported their results as a success rate in comparison with a control group, and 3 studies included both methods in their quantitative approach. Of the 10 studies with a numerical value for their error rate (Table 2). 10 studies measured errors related to the software output (labeling error, registration error, or staining error), while 2 studies measured the error in surgeon’s performance. While most studies reported accuracy above 90% while keeping range of error are smaller than 5 mm, Rüger et al reported an error exceeding 5 mm, and Thienphrapa et al reported as low as an 82% labelling accuracy for anatomical structures.16,17 19 articles reported their results as a comparison with a control group (Table 3). 17 (89%) studies reported at least one positive outcome associated with CST uses, including reduced error, increased successful trials, increased efficiency, decreased redirection frequency of surgical tools, increased recognition of pathology, and improved patient outcomes (Table 3). 2 studies reported the difference between CST group and controls were statistically insignificant.18,19 2 studies reported at least 1 negative impact associated with CST use, which were decreased pathology recognition and image quality.20,21 Interestingly, contradictive reports were observed between 2 studies; Dixon reported a decrease in recognition while Marcus reported an increase in recognition.20,22 Additionally, 2 studies involving a surgical predictive tool reported numerical values assessing the efficacy of their algorithm14,23 (Table 4).

Table 2.

Characteristics of Studies Reporting CSTs Performance as Error Rates.

Study Parameters Measured Results Anatomical Location Testing Frequency (per Participant)
Rosenthal et al (2002) Operational error 1.62 mm Breast 50
Birkfellner et al (2003) Average fiducial registration error 0.9 mm (max≤ 1.2 mm) Skull 3
Buck et al (2005) Registration error 1.04±.45 mm (max≤2 mm) Heart NS
Baumhauer et al (2008) Target visualization error ≥.49 mm Kidney 3
Endoscope registration accuracy (±1px) ≥97% Kidney 3
Debarba et al (2010) Volume estimate error 1.96±.96% (max≤3.49%) Liver NS
Haouchine et al (2013) Registration error 3.91 mm (max≤4.4 mm) Liver NS
Marzano et al (2013) Registration error 2.8 mm Pancreas/Duodenum 1
Conrad et al (2016) Registration accuracy ≤5 mm Liver 1
Thienphrapa et al (2019) Anatomical labeling accuracy (≤1 mm) 92% Lung NS
Anatomical registration accuracy 83% to 99% Lung NS
Ruger et al (2020) Operational error 5.0 mm±2.8 mm General 4
Barragan et al (2022) Cognitive load detection accuracy 78% General 1
*Rompianesi et al (2023) Staining success rate 57.1% Liver 1

Table 3.

Characteristics of Studies Reporting Surgeons’ Performances With the Employment of CSTs.

Study Parameters Measured Results Sample size P value
Birkfellner et al (2003) Target localization rate ⇓ error 1 N/A
Ogawa et al (2019) Angle registration accuracy N.S* 41 N/A
Chi et al (2018) Root mean square error ⇓ error 6 P < .05
Abhari et al (2015) Rotational and translational error ⇓ error 21 P < .005
Azagury et al (2010) Failure rate ⇓ error 3 P = 0·002
a Linxweiler et al (2020) Registration error ⇓ error (resident) 100 P < .0001
Registration error ⇓ error (senior physician) 100 P < .019
Dias et al (2020) Error rate ⇓ error 45 P < .001
Success rate ⇑ successful trial 45 P < .001
Dilley et al (2019) Perforation error ⇓ error 36 P = .02
Efficiency ⇑ efficiency 36 P = .0004
Dixon et al (2013) Recognition of unexpected findings ⇓ recognition 31 P < .041
Target accuracy ⇓ error 33 P < .001
Grasso et al (2012) Complications N.S 180 N.S
Jeon et al (2012) # Of redirections ⇓ redirections 20 P < .001
Marcus et al (2015) Recognition of unexpected findings ⇑ recognition 50 P = .025
Depth perception ⇑ perception 50 P = .015
Diana et al (2017) Image quality ⇓ quality 10 P < .0001
Rosenthal et al (2002) Operational error ⇓ error 50 P < .02
Ruger et al (2020) Operational error ⇓ error 20 P < .022
Wilson et al (2013) Success rate ⇑ success 34 P < .008
Wei et al (2019) Operational success ⇑ patient outcome 50 P < .05
Patel et al (2022) Knot tensile strength ⇑ knot tensile strength 15 P < .001
Barragan et al (2022) Tool changing event ⇓ tool change P < .01
Clutching time ⇓ clutching time P < .01

N.S indicates no statistically significant change. ⇑ indicates increases/improvement in a given category compared to control. ⇓ indicates decreases in a given category compared to control.

aLarge sample size studies.

Table 4.

Characteristics of Studies Involving Surgical Predictions.

Study Contribution Results Significance
Marz et al (2015) Patient data driven diagnostic and treatment aid 70 matching patients;
38±15 assertions per patient
84475 patient-related observations
Assists physician in diagnosis, individualizing treatment and provide follow-up for patients that require liver resections
Schoch et al (2016) Patient data driven cardiac prosthetic selection aid 55% elimination of unsuitable annuloplasty ring Increased automation and range of selections for annuloplasty ring

Patient Outcome

9 studies reported patient outcomes after the use of CST in their corresponding surgery (Table 5). 5 (56%) studies reported a minor or no post-operative complications. 2 (22%) studies observed slight bleeding in CST groups.24,25 3 studies reported serious either intraoperative or post-complication.21,25 Ntourakis reported a patient with a urinary tract infection and another patient with additional metastasis in CST groups. Diana reported intraoperative conversion from a MIS to an open surgery in the CST group.21,25

Table 5.

Characteristics of Studies Reporting Patients’ Outcomes With the Employment of CSTs.

Study Patient Outcomes Measured Conclusion Patient Sample Size
Willekes et al (2008) Post operative complications No complications 2
Lang et al (2005) % of patients with complete tumor removal 94.1% patient with complete tumor removal 25
a Linxweiler et al (2020) Post operative bleeding Slight bleeding in experimental group 100
Pain intensity Reduced pain experimental group 100
Ntourakis et al (2015) Operative blood loss 0 - 50 mL 3
Negative surgical margin .7 - 17 mm 3
Post operative outcome 1 patient had a urinary tract infection
1 patient had additional metastasis
3
Ogawa et al (2019) Post operative complications No complications 41
Pain or numbness No complications 41
Scarring Minor complications 41
a Grasso et al (2015) Procedural complications No significant complications 180
Diana et al (2017) Surgical complications 2 cases were converted to open surgery;
1 case was converted to laparoscopic surgery
58
Wei et al (2016) Post operative complication No complications 40
a Rompianesi et al (2023) Post operative complication 19 cases of complications (25%) 61
Length of hospital stay 5
Operative blood loss 150 mL

aLarge sample size studies.

OR Time Variation

18 studies reported an OR time difference (Table 6). 14 (78%) studies reported a trend in decreasing OR time, while 7 (38%) studies are statistically significant (P < .05). 4 (22%) studies reported an increase in OR time, 3 (18%) are statistically significant. Due to the inherent differences between the included procedures, the operational time between each procedure dramatically varies. To standardize the change in operational time, the differences are converted into a percentage change. Overall, there has been a 27.5% decrease in surgical time with the use of CST in comparison to traditional methods (mean: 27.5%; std: 41.8%) (Figure 3). Although the general pattern of CST use reduces procedural time, it is worth highlighting that robotics procedures tends to have longer operational time than their open surgery counterpart.

Table 6.

Characteristics of Studies Reporting Operation Time Changes With the Employment of CSTs.

Study Results Sample size P value
Chi et al (2018) 6 P < .05
Abhari et al (2015) 21 P < .05
Azagury et al (2010) 3 P < .027
Marzano et al (2010) 1 N.S
a Linxweiler et al (2020) ⇑ (senior physicians) 100 P = .3350
⇑ (resident) 100 P = .1934
Ogawa et al (2020) 41 P = .80
Dias et al (2020) 45 P < .001
Dilley et al (2019) 36 P ≤ .01
Dixon et al (2013) 33 P = .153
Grasso et al (2012) 180 P < .001
Jeon et al (2012) 20 P = .0002
Marcus et al (2015) 50 P = .007
Diana et al (2017) ⇓ (NIR-C vs control) 10 P = .00000003
⇓ (AR vs control) 10 P = .000013
Ruger et al (2020) 20 P = .211
Wei et al (2019) 40 P < .05
Wilson et al (2013) 34 N.S
Patel et al (2022) 15 NS
Barragan et al (2022) 8 P < .01

N.S indicates no statistically significant change. ⇑ indicates increases/improvement in each category compared to control. ⇓ indicates decreases in each category compared to control.

aLarge sample size studies.

Figure 3.

Figure 3.

Procedure duration change with the implementation of CSTs.

Discussion

Medical imaging such as X-rays, CT scans, MRIs, and ultrasound data are widely used in surgical planning and intraoperative guidance. Augmented or mixed reality technologies utilize these images to create visual 3-D models for visual enhancement. However, tissue deformation during surgery can cause inaccuracies in these models. In actual surgery, target registrations are primarily done during the end-exhalation plateau phase of the patient’s respiratory cycle to ensure accurate markers for anatomical structures. 26 The margin of error for surgery registration ranges from 1 mm to 15 mm.27,28

Most CSTs are developed for specific types of surgeries, such as hepatic (ablation) cardiac surgery, vascular surgery with cannulation and catheterization technologies, laparoscopic surgery, lung surgery, and arthroplasty. One of the most important functions of CSTs for surgeons is visual assistance. Clinically, one of the procedures that considerably benefit from visual aid is liver resection, because portal blood vessels may be difficult to visualize under the liver, residual liver volume is hard to estimate, and there may be hidden tumours that are not detected initially.25,26,29,30 Patient-specific virtual reality anatomical models may be built through preoperative or intra-operative images depending on data availability.26,30,31 During planning, surgeons may freely rotate the model to detect unexpected tumours and adjust the resection plane accordingly; based on the resection plane, the software returns the residual liver volume as well as security margins around the liver for surgeons to analyze the validity of the plan.26,30 Patient-specific digital models can be generated to support valve replacement in heart surgeries.32,33 Trajectory optimization done by machine learning can reduce the risks in cannulation and catheterization procedures. 34 In laparoscopic gastrectomy, CSTs serve to recommend ideal port locations for increased tool maneuverability. 35 Current CSTs developed for lung biopsy assist the operator by providing a real-time augmented reality video displaying the position of the needle and surrounding tissue. 19 Regardless of the type of operation, CSTs aim to reduce surgeons' cognitive load and decrease surgical complications. Preoperative and intra-operative cognitive support technologies (CSTs) have been developed to reduce the cognitive load of surgeons during surgical procedures. Preoperative CSTs simplify anatomical complexities, aid in prosthetic selection, and provide visualization; the major intended role of preoperative CSTs is to aid the surgeon accurately and efficiently in patient diagnosis and surgical planning. For example, in mitral valve replacement surgery, an algorithm was created to reduce commercial mitral valve options based on patient-specific heart dimensions and medical history. 14 In neurosurgery planning, a CST was developed to create a dynamic 3D model of a patient's brain and pathology, providing recommendations for surgical procedures.22,36 CSTs should progress from cognitive support for individual surgeons to support for the entire surgical team, utilizing virtual and augmented reality to improve communication and decision-making efficiency. During the intraoperative phase, the consequences of overloading the surgeon’s cognition are often more serious, leading to poorer adverse patient outcomes. Intraoperative cognitive overload primarily comes from visual disturbances, unexpected events, and variable environmental factors.24,37-39 Thus, the primary role of intraoperative CSTs is to alleviate these problems and to stabilize the surgeon’s cognitive workload. , For example, to address the problem of intraoperative visual disturbances in endoscopic-related surgeries, CSTs that provide a 3D visualization of an endoscope tip and software that allows for navigation aids were often prioritized; this type of assistance is especially beneficial for surgeries that contain a trans-nasal component because navigation is difficult in this subset of surgeries.24,37,38 In variable and intense environments like the battlefield, Auditory guidance headsets with micro-speaker have been developed to treat needle decompression of tension pneumothoraxes for combat medics. 39 Robotic innovations like the Da Vinci robot and the Xi robot are being implemented in an increasing amount of surgical specialties. Innovations in surgical robotics have shown to improve motion stability and decrease tissue destruction.40,41 Simple CSTs, such as tattooing with methylene markers, have also been developed to aid in target localization. 42 Robotic surgery can also be combined with intraoperative fluorescent markers to provide surgeons with improved visualization of the anatomical structures around the resection plane and tumour location. 43 To assess the cognition levels of surgeons during robotic surgery, machine learning-based CSTs are able to analyze physiological parameters such as electroencephalogram and eye behaviour to detect moments of high cognitive load; in response to cognitive overload, the semi-autonomous program may take control of simple tasks such as blood suction and irrigation. 44 More invasive neurostimulation techniques were also tested; transcranial direct current stimulation on surgeons completing a knot typing task in robotic surgery was reported to have increased knot tensile strength. 45 Cognitive surgical technologies (CSTs) incorporating augmented reality (AR) and image guidance have shown promising results in improving precision and reducing errors in various surgical procedures. Similar to other visual aid CSTs, AR-based CSTs also rely on radiology images to create a patient-specific 3-D model. Intraoperatively, anatomical landmarks around the surgical site and security margin are labelled to the surgeons to aid their incisions; clarifications are given to surgeons around complex anatomical regions, and some CSTs provide procedural instruction during the operation. Studies have reported increased accuracy rates in AR or voice-guided surgery, with decreased error rates of up to 10%.15,16,24,27,34,36,37,46,47 AR-based CSTs have been effective in treating hepatocellular carcinoma, neonatal intubation, laparoscopic nephrectomy, and needle biopsies, resulting in increased success rates and reduced error rates.15,19,29,48 However, in some procedures like total hip arthroplasty and breast cancer-related lymphedema diagnoses, the implementation of CSTs did not result in significant differences.18,49 The presence of CST may also influence procedure time, with increased distraction or unfamiliarity with the device potentially reducing efficiency.24,34,39,46 However, there is insufficient evidence available on intraoperative CSTs to support the conclusion that it significantly lowers cognitive load and boosts performance, future research should focus on this area. With more development, CSTs may be crucial in reducing the cognitive load of surgeons during surgical procedures, improving communication and decision-making efficiency, and ensuring patient safety. The development of CSTs should continue to focus on reducing procedure complexity, reducing error rates, and aiding intraoperative decision-making to improve surgical outcomes. Current CSTs have limitations that hinder their widespread use in the OR. One major problem is the inadequacy of existing VR and AR technology to handle tissue deformation and organ fluctuation, requiring the intervention of expert computer scientists during surgery.25,28,38 Additionally, there is a lack of randomized controlled trials and sufficient clinical data due to the disconnection between medical staff and computer scientists, leading to outsourcing clinical trials to actual hospitals after CST development. Furthermore, CSTs often lack ergonomic design for effective surgical implementation, affecting navigation, reducing the field of view, and other ergonomics-related problems.16,36 Surgeons also have a preference for traditional techniques, and CSTs are often expensive, limiting their use in routine procedures. Learning curve problems and the potential for attentional blindness also pose challenges to CST use. CSTs may be more suitable for complex procedures requiring cognitive assistance, and future development should prioritize ergonomic design to improve usability and maneuverability. Overall, CSTs require further research and development to improve their effectiveness and gain wider acceptance in the ORs.

Conclusion

This review suggests that CSTs have shown promising results in improving surgical performance and reducing errors by increasing surgeons' visual fields. CSTs are currently in development for a wide variety of surgery specialties, and its purpose are often multi-dimensional, aiding surgeons in different areas before and during the procedure. Some advantages of CST, such as accuracy-based parameters, are supported with more evidence than other. However, several challenges still need to be addressed, including inaccuracies in accounting for tissue deformation, a lack of clinical data, particularly patient data, and ergonomic issues with bulky equipment. When developing these CSTs, adequate consideration must also be given to the possibility that the addition of CSTs would fatigue the surgical operator and cause them to regress in performance. Despite these limitations, the potential benefits of CSTs make them a promising area for further development and research, in today’s innovative age, improvements in hardware, software and design could overcome these technological challenges, furthering improving the medical field with interdisciplinary efforts.

Footnotes

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Zhong Shi Zhang has been funded by Alberta Innovate Summer Studentship for this research.

ORCID iDs

Zhong Shi Zhang https://orcid.org/0000-0002-5836-7257

Bin Zheng https://orcid.org/0000-0003-3476-5936

References

  • 1.Owens CD, Stoessel K. Surgical site infections: epidemiology, microbiology and prevention. J Hosp Infect. 2008;70(Suppl 2):3-10. doi: 10.1016/S0195-6701(08)60017-1. [DOI] [PubMed] [Google Scholar]
  • 2.Coccolini F, Catena F, Pisano M, et al. Open versus laparoscopic cholecystectomy in acute cholecystitis. Systematic review and meta-analysis. Int J Surg. 2015;18:196-204. doi: 10.1016/j.ijsu.2015.04.083. [DOI] [PubMed] [Google Scholar]
  • 3.Martínez-Pérez A, de’Angelis N, Brunetti F, et al. Laparoscopic vs. open surgery for the treatment of iatrogenic colonoscopic perforations: a systematic review and meta-analysis. World J Emerg Surg. 2017;12:8. doi: 10.1186/s13017-017-0121-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Nicolás M, Czerwonko M, Ardiles V, et al. Laparoscopic vs open liver resection for metastatic colorectal cancer: analysis of surgical margin status and survival. Langenbecks Arch Surg. 2022;407(3):1113-1119. doi: 10.1007/s00423-021-02396-2. [DOI] [PubMed] [Google Scholar]
  • 5.Mazur LM, Mosaly PR, Jackson M, et al. Quantitative assessment of workload and stressors in clinical radiation oncology. Int J Radiat Oncol. 2012;83(5):e571-e576. doi: 10.1016/j.ijrobp.2012.01.063. [DOI] [PubMed] [Google Scholar]
  • 6.Lacy AM, García-Valdecasas JC, Delgado S, et al. Laparoscopy-assisted colectomy versus open colectomy for treatment of non-metastatic colon cancer: a randomised trial. Lancet Lond Engl. 2002;359(9325):2224-2229. doi: 10.1016/S0140-6736(02)09290-5. [DOI] [PubMed] [Google Scholar]
  • 7.van Bergen P, Kunert W, Bessell J, Buess GF. Comparative study of two-dimensional and three-dimensional vision systems for minimally invasive surgery. Surg Endosc. 1998;12(7):948-954. doi: 10.1007/s004649900754. [DOI] [PubMed] [Google Scholar]
  • 8.Wheelock A, Suliman A, Wharton R, et al. The impact of operating room distractions on stress, workload, and teamwork. Ann Surg. 2015;261(6):1079-1084. doi: 10.1097/SLA.0000000000001051. [DOI] [PubMed] [Google Scholar]
  • 9.Mcmullan RD, Urwin R, Gates P, Sunderland N, Westbrook JI. Are operating room distractions, interruptions and disruptions associated with performance and patient safety? A systematic review and meta-analysis. Int J Qual Health Care. 2021;33(2):mzab068. doi: 10.1093/intqhc/mzab068. [DOI] [PubMed] [Google Scholar]
  • 10.Sevdalis N, Wong HWL, Arora S, et al. Quantitative analysis of intraoperative communication in open and laparoscopic surgery. Surg Endosc. 2012;26(10):2931-2938. doi: 10.1007/s00464-012-2287-3. [DOI] [PubMed] [Google Scholar]
  • 11.Parker SL, Mendenhall SK, Shau DN, et al. Minimally invasive versus open transforaminal lumbar interbody fusion for degenerative spondylolisthesis: comparative effectiveness and cost-utility analysis. World Neurosurg. 2014;82(1-2):230-238. doi: 10.1016/j.wneu.2013.01.041. [DOI] [PubMed] [Google Scholar]
  • 12.Berguer R, Smith WD, Chung YH. Performing laparoscopic surgery is significantly more stressful for the surgeon than open surgery. Surg Endosc. 2001;15(10):1204-1207. doi: 10.1007/s004640080030. [DOI] [PubMed] [Google Scholar]
  • 13.Cohen AJ, Lui H, Zheng M, et al. Rates of serious surgical errors in California and plans to prevent recurrence. JAMA Netw Open. 2021;4(5):e217058. doi: 10.1001/jamanetworkopen.2021.7058. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Schoch N, Philipp P, Weller T, et al. In: Webster RJ, Yaniv ZR, eds. Cognitive tools pipeline for assistance of mitral valve surgery. Washington, DC: NASA; 2016 [Google Scholar]
  • 15.Dias PL, Greenberg RG, Goldberg RN, Fisher K, Tanaka DT. Augmented reality–assisted video laryngoscopy and simulated neonatal intubations: a pilot study. Pediatrics. 2021;147(3):e2020005009. doi: 10.1542/peds.2020-005009. [DOI] [PubMed] [Google Scholar]
  • 16.Rüger C, Feufel MA, Moosburner S, Özbek C, Pratschke J, Sauer IM. Ultrasound in augmented reality: a mixed-methods evaluation of head-mounted displays in image-guided interventions. Int J Comput Assist Radiol Surg. 2020;15(11):1895-1905. doi: 10.1007/s11548-020-02236-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Thienphrapa P, Bydlon T, Chen A, et al. Interactive endoscopy: a next-generation, streamlined user interface for lung surgery navigation. In: Shen D, Liu T, Peters TM, eds. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. Vol 11768. Lecture Notes in Computer Science. Berlin, Germany: Springer International Publishing; 2019:83-91. doi: 10.1007/978-3-030-32254-0_10. [DOI] [Google Scholar]
  • 18.Ogawa H, Kurosaka K, Sato A, Hirasawa N, Matsubara M, Tsukada S. Does an augmented reality-based portable navigation system improve the accuracy of acetabular component orientation during tha? A randomized controlled trial. Clin Orthop. 2020;478(5):935-943. doi: 10.1097/CORR.0000000000001083. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Grasso RF, Luppi G, Cazzato RL, et al. Percutaneous computed tomography-guided lung biopsies: preliminary results using an augmented reality navigation system. Tumori. 2012;98(6):775-782. doi: 10.1177/030089161209800616. [DOI] [PubMed] [Google Scholar]
  • 20.Dixon BJ, Daly MJ, Chan H, Vescan AD, Witterick IJ, Irish JC. Surgeons blinded by enhanced navigation: the effect of augmented reality on attention. Surg Endosc. 2013;27(2):454-461. doi: 10.1007/s00464-012-2457-3. [DOI] [PubMed] [Google Scholar]
  • 21.Diana M, Soler L, Agnus V, et al. Prospective evaluation of precision multimodal gallbladder surgery navigation: virtual reality, near-infrared fluorescence, and X-ray-based intraoperative cholangiography. Ann Surg. 2017;266(5):890-897. doi: 10.1097/SLA.0000000000002400. [DOI] [PubMed] [Google Scholar]
  • 22.Marcus HJ, Pratt P, Hughes-Hallett A, et al. Comparative effectiveness and safety of image guidance systems in neurosurgery: a preclinical randomized study. J Neurosurg. 2015;123(2):307-313. doi: 10.3171/2014.10.JNS141662. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.März K, Hafezi M, Weller T, et al. Toward knowledge-based liver surgery: holistic information processing for surgical decision support. Int J Comput Assist Radiol Surg. 2015;10(6):749-759. doi: 10.1007/s11548-015-1187-0. [DOI] [PubMed] [Google Scholar]
  • 24.Linxweiler M, Pillong L, Kopanja D, et al. Augmented reality‐enhanced navigation in endoscopic sinus surgery: a prospective, randomized, controlled clinical trial. Laryngoscope Investig Otolaryngol. 2020;5(4):621-629. doi: 10.1002/lio2.436. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Ntourakis D, Memeo R, Soler L, Marescaux J, Mutter D, Pessaux P. Augmented reality guidance for the resection of missing colorectal liver metastases: an initial experience. World J Surg. 2016;40(2):419-426. doi: 10.1007/s00268-015-3229-8. [DOI] [PubMed] [Google Scholar]
  • 26.Hansen C, Köhn A, Schlichting S, et al. Intraoperative modification of resection plans for liver surgery. Int J Comput Assist Radiol Surg. 2008;3(3-4):291-297. doi: 10.1007/s11548-008-0161-5. [DOI] [Google Scholar]
  • 27.Birkfellner W, Figl M, Matula C, et al. Computer-enhanced stereoscopic vision in a head-mounted operating binocular. Phys Med Biol. 2003;48(3):N49-57. doi: 10.1088/0031-9155/48/3/402. [DOI] [PubMed] [Google Scholar]
  • 28.Haouchine N, Dequidt J, Peterlik I, Kerrien E, Berger MO, Cotin S. Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery. In: 2013 IEEE international symposium on mixed and augmented reality (ISMAR), Adelaide, Australia, 1-4 October 2013 [Google Scholar]
  • 29.Lang H, Radtke A, Hindennach M, et al. Impact of virtual tumor resection and computer-assisted risk analysis on operation planning and intraoperative strategy in major hepatic resection. Arch Surg. 2005;140(7):629-638. doi: 10.1001/archsurg.140.7.629 [DOI] [PubMed] [Google Scholar]
  • 30.Debarba HG, Zanchet DJ, Fracaro D, Maciel A, Kalil AN. Efficient liver surgery planning in 3D based on functional segment classification and volumetric information. Annu Int Conf IEEE Eng Med Biol Soc. 2010;2010:4797-4800. doi: 10.1109/IEMBS.2010.5628026. [DOI] [PubMed] [Google Scholar]
  • 31.De Buck S, Maes F, Ector J, et al. An augmented reality system for patient-specific guidance of cardiac catheter ablation procedures. IEEE Trans Med Imaging. 2005;24(11):1512-1524. doi: 10.1109/TMI.2005.857661. [DOI] [PubMed] [Google Scholar]
  • 32.Hagenah J, Evers T, Scharfschwerdt M, Schweikard A, Ernst F. A Support Vector Regression-Based Data-Driven Leaflet Modeling Approach for Personalized Aortic Valve Prosthesis Development. Lübeck, Germany: University of Lubeck; 2018 [Google Scholar]
  • 33.Hagenah J, Mehdi M, Ernst F. Generating healthy aortic root geometries from ultrasound images of the individual pathological morphology using deep convolutional autoencoders. In: 2019 Computing in Cardiology (CinC), Singapore, 08-11 September 2019 [Google Scholar]
  • 34.Chi W, Liu J, Abdelaziz MEMK, et al. Trajectory optimization of robot-assisted endovascular catheterization with reinforcement learning. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), Madrid, Spain, 01-05 October 2018 [Google Scholar]
  • 35.Hayashi Y, Misawa K, Mori K. Optimal port placement planning method for laparoscopic gastrectomy. Int J Comput Assist Radiol Surg. 2017;12(10):1677-1684. doi: 10.1007/s11548-017-1548-y. [DOI] [PubMed] [Google Scholar]
  • 36.Abhari K, Baxter JSH, Chen ECS, et al. Training for planning tumour resection: augmented reality and human factors. IEEE Trans Biomed Eng. 2015;62(6):1466-1477. doi: 10.1109/TBME.2014.2385874. [DOI] [PubMed] [Google Scholar]
  • 37.Azagury DE, Ryou M, Shaikh SN, et al. Real-time computed tomography-based augmented reality for natural orifice transluminal endoscopic surgery navigation. Br J Surg. 2012;99(9):1246-1253. doi: 10.1002/bjs.8838. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Marzano E, Piardi T, Soler L, et al. Augmented reality-guided artery-first pancreatico-duodenectomy. J Gastrointest Surg. 2013;17(11):1980-1983. doi: 10.1007/s11605-013-2307-1. [DOI] [PubMed] [Google Scholar]
  • 39.Wilson KL, Doswell JT, Fashola OS, et al. Using augmented reality as a clinical support tool to assist combat medics in the treatment of tension pneumothoraces. Mil Med. 2013;178(9):981-985. doi: 10.7205/MILMED-D-13-00074. [DOI] [PubMed] [Google Scholar]
  • 40.Trastulli S, Desiderio J, Lin JX, et al. Open vs robotic gastrectomy with D2 lymphadenectomy: a propensity score-matched analysis on 1469 patients from the IMIGASTRIC prospective database. Langenbecks Arch Surg. 2023;408(1):302. doi: 10.1007/s00423-023-03032-x. [DOI] [PubMed] [Google Scholar]
  • 41.Toesca A, Sangalli C, Maisonneuve P, et al. A randomized trial of robotic mastectomy versus open surgery in women with breast cancer or BrCA mutation. Ann Surg. 2022;276(1):11-19. doi: 10.1097/SLA.0000000000004969. [DOI] [PubMed] [Google Scholar]
  • 42.Willekes L, Boutros C, Goldfarb MA. VATS intraoperative tattooing to facilitate solitary pulmonary nodule resection. J Cardiothorac Surg. 2008;3(1):13. doi: 10.1186/1749-8090-3-13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Rompianesi G, Pegoraro F, Ramaci L, Ceresa CD, Montalti R, Troisi RI. Preoperative planning and intraoperative real-time navigation with indocyanine green fluorescence in robotic liver surgery. Langenbecks Arch Surg. 2023;408(1):292. doi: 10.1007/s00423-023-03024-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Barragan JA, Yang J, Yu D, Wachs JP. A neurotechnological aid for semi-autonomous suction in robotic-assisted surgery. Sci Rep. 2022;12(1):4504. doi: 10.1038/s41598-022-08063-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Patel R, Suwa Y, Kinross J, et al. Neuroenhancement of surgeons during robotic suturing. Surg Endosc. 2022;36(7):4803-4814. doi: 10.1007/s00464-021-08823-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Dilley JWR, Hughes-Hallett A, Pratt PJ, et al. Perfect registration leads to imperfect performance: a randomized trial of multimodal intraoperative image guidance. Ann Surg. 2019;269(2):236-242. doi: 10.1097/SLA.0000000000002793. [DOI] [PubMed] [Google Scholar]
  • 47.Rosenthal M, State A, Lee J, et al. Augmented reality guidance for needle biopsies: an initial randomized, controlled trial in phantoms. Med Image Anal. 2002;6(3):313-320. doi: 10.1016/S1361-8415(02)00088-9. [DOI] [PubMed] [Google Scholar]
  • 48.Baumhauer M, Simpfendörfer T, Müller-Stich BP, et al. Soft tissue navigation for laparoscopic partial nephrectomy. Int J Comput Assist Radiol Surg. 2008;3(3-4):307-314. doi: 10.1007/s11548-008-0216-7. [DOI] [Google Scholar]
  • 49.Invernizzi M, Runza L, De Sire A, et al. Integrating augmented reality tools in breast cancer related lymphedema prognostication and diagnosis. J Vis Exp. 2020:156:60093. doi: 10.3791/60093. [DOI] [PubMed] [Google Scholar]

Articles from Surgical Innovation are provided here courtesy of SAGE Publications

RESOURCES