Abstract
Background
In recent years, numerous innovative yet challenging surgeries, such as minimally invasive procedures, have introduced an overwhelming amount of new technologies, increasing the cognitive load for surgeons and potentially diluting their attention. Cognitive support technologies (CSTs) have been in development to reduce surgeons’ cognitive load and minimize errors. Despite its huge demands, it still lacks a systematic review.
Methods
Literature was searched up until May 21st, 2021. Pubmed, Web of Science, and IEEExplore. Studies that aimed at reducing the cognitive load of surgeons were included. Additionally, studies that contained an experimental trial with real patients and real surgeons were prioritized, although phantom and animal studies were also included. Major outcomes that were assessed included surgical error, anatomical localization accuracy, total procedural time, and patient outcome.
Results
A total of 37 studies were included. Overall, the implementation of CSTs had better surgical performance than the traditional methods. Most studies reported decreased error rate and increased efficiency. In terms of accuracy, most CSTs had over 90% accuracy in identifying anatomical markers with an error margin below 5 mm. Most studies reported a decrease in surgical time, although some were statistically insignificant.
Discussion
CSTs have been shown to reduce the mental workload of surgeons. However, the limited ergonomic design of current CSTs has hindered their widespread use in the clinical setting. Overall, more clinical data on actual patients is needed to provide concrete evidence before the ubiquitous implementation of CSTs.
Keywords: cognitive support, surgery performance, mental workload, anatomical accuracy, patient outcomes
Introduction
Modern surgery has come a long way. From the first successful open surgery to the laparoscopic and robotic-assisted procedures standard in today’s operating room, the success of modern surgery is no anomaly; it is deeply entwined with the development of surgical technology. Ranging from aseptic techniques to intra-operative imaging, technology has always been an effective support for surgeons. 1 Novel surgical procedures such as minimally invasive surgery (MIS) also benefit the patients, as it is often associated with lower patient morbidity, infection, and a shorter hospital stay.2-4 However, it is important to highlight that surgical techniques today have increased in complexity compared to previous decades. Procedures like endoscopic or laparoscopic surgery have restricted the surgeon’s visual field to 2-D instead of the traditional 3-D view in open surgeries. The disparity in the visual field introduces problems such as image-reality incompatibility, loss of depth perception, and abnormal haptic feedback; additionally, novel surgical techniques also require new machines to be introduced to the operating room (OR), which may lead to extra staff to monitor the devices. The mental strain caused by MIS and excessive signals present in the working environment of the surgeon may lead to cognitive overload, which causes surgeon performance to decrease 5 ; this induces surgical errors while also extending operational time.6,7 Another potential cause to surgical errors involves communication. Irrelevant conversations within the OR are associated with poorer team collaboration and performance,8,9 while also extending operational time; equipment-related distractions is associated with higher stress and cognitive load. 8 Compared to open surgery, communication in MIS is more unclear and more frequently directed at equipment instead of the procedure, likely due to the increased equipment and staff used to support the MIS. 10 . Parker et al 11 reported a longer duration of open transforaminal lumbar interbody fusion surgery when comparing MIS to standard open surgery. Experiments that mimicked both MIS and open surgery environments reported a decreased efficiency in task completion and an increased mental workload. 12 Accompanying the increased cognitive load are increased probability of adverse events. In the top 559 hospitals located in the US, close to 81.7% of intraoperative errors are related to surgeons’ cognition being overwhelmed, each offense could cause up to 125,000$ in penalty. Thus, fatigue-related events, such as retained foreign objects or wrong surgical sites would be significantly reduced if surgeons received cognitive support. 13 To counteract the increased cognitive load of surgeons and to reduce surgical errors, cognitive support technology (CST) has developed since the early 2000s and has seen significant growth in recent years to aid the decision-making of physicians and surgeons. However, despite its importance, technology in this field lacks a systematic review and assessment of the impacts on surgical performance. We aim to review the current status and trends of CST in surgery. CSTs covered in this review pertains to information presentation for diagnostic support and visualization for enhancing performance. Specifically, we include the software and hardware employed for data presentation, analysis in surgery, and impact on surgeon performance and patient outcome.
Method
Search Criteria
The inclusion criteria include a list of terms that can be categorized into three major groups: cognitive support, surgery, and digital reality. A figure of an example search in PubMed is provided. Related terms are connected with Boolean operators; terms in a column are connected with the “OR” operator, while terms in a row are connected with the “AND” operator (Figure 1). A CST is a technology that reaches a high level of innovation and technological complexity, often involving programming or robotic design aimed at reducing cognitive load of surgeons. Exclusion criteria were set to be checklist articles because an overwhelming number of checklists was included in the literature search. In this review, a checklist defined as a digital or non-digital element used to verify aspects of the surgery in the OR. Surgical checklists were often specific to a large variety of surgeries which required a large amount of background information. There were also additional checklists assessing other checklists which were too complicated for the purpose of this review. Literature included in this review was gathered through a literature search in medical databases, including the Web of Science, PubMed, and IEEExplore, articles published until May 21, 2021 were included. An effort was made to search for clinical trials or randomized controlled trials, but a few exceptional model and architectural articles are included to provide a framework understanding of the topic. The article search and selection process was completed by a single reviewer (ZSZ). Mendeley reference manager (Relx, New York, USA) was used to organize the references. Studies were included in this review if it targeted the peri-operative phase in surgery, contained the development software or device that aimed to reduce surgeons’ cognitive load, and experimental trials that assessed the efficacy of the technology. Studies that only included a phantom trial, an animal trial, or used synthetic patients were included. Articles were first screened by title and abstract assessments. The remaining articles then underwent full-text analysis to further evaluate relevant articles.
Figure 1.
Advance search criteria for PubMed.
Data Analysis
A single reviewer (ZSZ) examined key characteristics of each article including data source, development status, experiment type, sample size, phase of surgery, and the type of procedure. Full-text articles selected were organized with Microsoft Excel (Microsoft Corporation, WA, USA). Papers were further stratified into different subgroups by the parameters they measured; clinical outcomes, patient outcomes, and OR time variations were all taken into consideration. Discussion of the advantages and disadvantages of CSTs were gathered from the discussion and conclusion of the articles. Interpretation of the study results was based on the result section of each paper. A secondary reviewer (YW) verified the accuracy of the content. In events where the article cannot be classified into a set group, or parameters on the article was unclear, then a discussion was to occur with two other reviewers (BZ, WL) until an agreement was made.
Results
Overall, 481 citations were found throughout the literature search; after electronically removing 22 duplicates using Endnote (Clarivate, London), 459 remaining articles underwent further screening. Two independent reviewers (ZSZ, LW) screened the citations based on title and abstract, 369 articles were removed, leaving 35 for full-text review. During full-text review, 14 articles were removed: seven lacked quantitative data; six were overview articles; one was a checklist. An additional 19 articles were found through relevant searching, summing the total number of articles included in this review to be 40 articles. Each article had at least one relevant data describing cognitive load reduction. The complete article evaluation process is outlined as Figure 2.
Figure 2.
Flow chat of the article selection process.
Data Source
CSTs use input data from a few different varieties to generate 3-D images for enhanced visualization or to assist patient diagnoses. Medical imaging databases such as XNAT are developed for imaging upload and information extraction. 14 For all the papers in this review, an overwhelming 23 (58%) CSTs used X-ray-based images such as computed tomography (CT) as their primary data source. 6 (15%) CSTs adopted MRI as their data source while another 6(15%) CSTs used ultrasound; however, they are not as common as the CT scans due to a variety of different reasons relating to costs and accuracy. 2(6%) CST uses other types of input such as live videos with manual markers with its application in nursing, and physiological parameters when analyzing cognitive load. 15 7(18%) CST did not provide specific sources of their input data, which may be due to their employment of previously established data from other sources (Table 1).
Table 1.
Characteristics of Included Studies.
Study | Data Source | Type of Development | Type of Experiment | Sample Size (n) (Staff; Patient) | Phase of Surgery | Procedure |
---|---|---|---|---|---|---|
M¨arz et al (2015) | Non-specific | Software development | Development | (NS; 184) | Pre-operative | Hepatectomy |
Schoch et al (2016) | MRI | Software development | Development | (NS; 624) | Pre-operative | Cardiac surgery |
Thienphrapa et al (2019) | CT | Software development | Development | (NS; NS) | Intra-operative | Video-assisted thoracoscopic surgery |
Willekes et al (2008) | CT | Hardware development | User tests | (NS; 2) | Intra-operative | Video-assisted thoracic surgery |
Hansen et al (2008) | Ultrasound | Software+hardware development | Development | (NS; NS) | Intra-operative | Hepatectomy |
Lang et al (2005) | MRI or CT | Software development | Clinical tests | (2; 25) | Pre-operative | Hepatectomy |
Hagenah et al (2018) | Ultrasound | Software development | Development | (NS; 10) | Pre-operative | Cardiac surgery |
Chi et al (2018) | CT | Software+hardware development | User tests | (1; 6) | Intra-operative | Cardiac surgery |
Abhari et al (2015) | MRI | Software+hardware development | User tests | (21; NS) | Pre-operative | Neurosurgery |
Hagenah et al (2019) | Ultrasound | Software development | Development | (NS; 24) | Pre-operative | Cardiac surgery |
Simoes et al (2013) | CT | Software+hardware development | Development | (NS; NS) | Pre-operative | Robot-assisted laparoscopic surgery |
Hayashi et al (2017) | CT | Software development | Clinical tests | (3; 26) | Pre-operative | Laparoscopic gastrectomy |
Azagury et al (2012) | CT | Software+hardware development | Clinical tests | (3; 3) | Intra-operative | Natural orifice transluminal |
Endoscopic surgery | ||||||
Baumhauer et al (2008) | CT | Software development | User test | (1; 3) | Intra-operative | Laparoscopic partial nephrectomy |
Birkfellner et al (2003) | CT | Software+hardware development | Development | (NS; NS) | Intra-operative | Skull base surgery |
Collins et al (2014) | MRI | Software development | User tests | (NS; 1) | Intra-operative | Laparoscopic myomectomy |
Conrad et al (2016) | CT | Software+hardware development | User tests | (NS; 1) | Intra-operative | Laparoscopic rescue of failed portal vein embolization |
Debarba et al (2010) | CT | Software development | Development | (NS; 4) | Pre-operative | Hepatectomy |
Haouchine et al (2013) | CT | Software development | Development | (NS; 2) | Intra-operative | Liver minimally invasive surgery |
Marzano et al (2013) | CT | Software+hardware development | User tests | (NS; 1) | Intra-operative | Pancreatico-duodenectomy |
Osorio et al (2010) | CT | Software development | Clinical tests | (NS; 24) | Intra-operative | Laparoscopic surgery |
a Linxweiler et al (2020) | CT | Software development | Clinical tests | (2; 100) | Intra-operative | Sinus-related surgery |
Ntourakis et al (2015) | CT | Software+hardware development | Clinical tests | (NS; 3) | Intra-operative | Hepatectomy |
Ogawa et al (2019) | CT | Software+hardware development | Clinical tests | (1; 41) | Intra-operative | Total hip arthroplasty |
Buck et al (2005) | MRI or CT | Software development | Development | (NS; 39) | Pre-operative + Intra-operative | Cardiac surgery |
Dias et al (2020) | Live video | Hardware development | User tests | (45; NS) | Intra-operative | Intubation |
Dilley et al (2019) | Non-specific | Software development | User tests | (36; NS) | Intra-operative | Laparoscopic cholecystectomy |
Dixon et al (2013) | CT | Software+hardware development | Clinical tests | (33; 1) | Intra-operative | Endoscopic transnasal skull base navigation |
a Grasso et al (2015) | CT | Software+hardware development | Clinical tests | (3; 180) | Intra-operative | Lung biopsie |
Invernizzi et al (2020) | Non-specific | Software development | Clinical tests | (3; 30) | Pre-operative | Breast cancer related lymphedema diagnosis |
Jeon et al (2014) | Ultrasound | Software+hardware development | User tests | (20; NS) | Intra-operative | Cardiac surgery |
Marcus et al (2015) | CT | Software+hardware development | User tests | (50; 1) | Intra-operative | Laparoscopic neurosurgery |
Diana et al (2017) | MRI | Software development | Clinical tests | (10; 58) | Pre-operative + Intra-operative | Gallbladder surgery |
Rosenthal et al (2005) | Ultrasound | Software+hardware development | User tests | (1; NS) | Intra-operative | Breast biopsy |
Ru¨ger et al (2020) | Ultrasound | Software+hardware development | User tests | (20; 1) | Intra-operative | Ultrasound-guided needle placements |
Wei et al (2016) | CT | Software development | Clinical tests | (1; 40) | Intra-operative | Percutaneous kyphoplasty |
Wilson et al (2013) | Non-specific | Software+hardware development | Clinical tests | (34; 16) | Intra-operative | Tension pneumothorax decompression |
Patel et al (2023) | Non-specific | Hardware development | User tests | (15; NS) | Intra-operative | Robotic suturing |
Barragan et al (2022) | Physio Signal+ Live video | Software development | User tests | (8; NS) | Intra-operative | Robotic surgery |
a Rompianesi et al (2023) | Non-specific | Hardware development | Clinical test | (61; NS) | Pre-operative+intra-operative | Hepatectomy |
Development indicates the lack of human participants; user test indicates the incorporation of surgeons but no patient participants; clinical test indicates the participation of both surgeons and patients are involved in the study.
aLarge sample size studies.
Phase of Surgery
Different CSTs act on three main phases of surgery: preoperative, intraoperative, and post-operative. Preoperative includes all planning activities before the onset of the surgery; intraoperative refers to surgical performance during an operation. CSTs categorized to the different phases of surgery act as interventions or visual assistance to the surgeon. 12 (32%) CSTs are responsible for preoperative interventions and 27 (73%) CSTs are responsible for intraoperative interventions. No CST was found to assist in patient rehabilitation after the surgery had been completed.
Types of Surgery
CSTs are present in a wide variety of procedures. General surgery, thoracic surgery, neurosurgery, urology surgery, otolaryngology surgery (ENT), and orthopedic surgery are all applicable for CST use. While it is difficult to categorize them into specific groups, it is certainly clear that there is an emphasis on MIS (n = 11) (Table 1).
Surgical Performance Outcome
A total of 26 (65%) literature reported their findings numerically. Out of the 26, 7 studies reported their findings in a percentage or an error distance, 16 studies reported their results as a success rate in comparison with a control group, and 3 studies included both methods in their quantitative approach. Of the 10 studies with a numerical value for their error rate (Table 2). 10 studies measured errors related to the software output (labeling error, registration error, or staining error), while 2 studies measured the error in surgeon’s performance. While most studies reported accuracy above 90% while keeping range of error are smaller than 5 mm, Rüger et al reported an error exceeding 5 mm, and Thienphrapa et al reported as low as an 82% labelling accuracy for anatomical structures.16,17 19 articles reported their results as a comparison with a control group (Table 3). 17 (89%) studies reported at least one positive outcome associated with CST uses, including reduced error, increased successful trials, increased efficiency, decreased redirection frequency of surgical tools, increased recognition of pathology, and improved patient outcomes (Table 3). 2 studies reported the difference between CST group and controls were statistically insignificant.18,19 2 studies reported at least 1 negative impact associated with CST use, which were decreased pathology recognition and image quality.20,21 Interestingly, contradictive reports were observed between 2 studies; Dixon reported a decrease in recognition while Marcus reported an increase in recognition.20,22 Additionally, 2 studies involving a surgical predictive tool reported numerical values assessing the efficacy of their algorithm14,23 (Table 4).
Table 2.
Characteristics of Studies Reporting CSTs Performance as Error Rates.
Study | Parameters Measured | Results | Anatomical Location | Testing Frequency (per Participant) |
---|---|---|---|---|
Rosenthal et al (2002) | Operational error | 1.62 mm | Breast | 50 |
Birkfellner et al (2003) | Average fiducial registration error | 0.9 mm (max≤ 1.2 mm) | Skull | 3 |
Buck et al (2005) | Registration error | 1.04±.45 mm (max≤2 mm) | Heart | NS |
Baumhauer et al (2008) | Target visualization error | ≥.49 mm | Kidney | 3 |
Endoscope registration accuracy (±1px) | ≥97% | Kidney | 3 | |
Debarba et al (2010) | Volume estimate error | 1.96±.96% (max≤3.49%) | Liver | NS |
Haouchine et al (2013) | Registration error | 3.91 mm (max≤4.4 mm) | Liver | NS |
Marzano et al (2013) | Registration error | 2.8 mm | Pancreas/Duodenum | 1 |
Conrad et al (2016) | Registration accuracy | ≤5 mm | Liver | 1 |
Thienphrapa et al (2019) | Anatomical labeling accuracy (≤1 mm) | 92% | Lung | NS |
Anatomical registration accuracy | 83% to 99% | Lung | NS | |
Ruger et al (2020) | Operational error | 5.0 mm±2.8 mm | General | 4 |
Barragan et al (2022) | Cognitive load detection accuracy | 78% | General | 1 |
*Rompianesi et al (2023) | Staining success rate | 57.1% | Liver | 1 |
Table 3.
Characteristics of Studies Reporting Surgeons’ Performances With the Employment of CSTs.
Study | Parameters Measured | Results | Sample size | P value |
---|---|---|---|---|
Birkfellner et al (2003) | Target localization rate | ⇓ error | 1 | N/A |
Ogawa et al (2019) | Angle registration accuracy | N.S* | 41 | N/A |
Chi et al (2018) | Root mean square error | ⇓ error | 6 | P < .05 |
Abhari et al (2015) | Rotational and translational error | ⇓ error | 21 | P < .005 |
Azagury et al (2010) | Failure rate | ⇓ error | 3 | P = 0·002 |
a Linxweiler et al (2020) | Registration error | ⇓ error (resident) | 100 | P < .0001 |
Registration error | ⇓ error (senior physician) | 100 | P < .019 | |
Dias et al (2020) | Error rate | ⇓ error | 45 | P < .001 |
Success rate | ⇑ successful trial | 45 | P < .001 | |
Dilley et al (2019) | Perforation error | ⇓ error | 36 | P = .02 |
Efficiency | ⇑ efficiency | 36 | P = .0004 | |
Dixon et al (2013) | Recognition of unexpected findings | ⇓ recognition | 31 | P < .041 |
Target accuracy | ⇓ error | 33 | P < .001 | |
Grasso et al (2012) | Complications | N.S | 180 | N.S |
Jeon et al (2012) | # Of redirections | ⇓ redirections | 20 | P < .001 |
Marcus et al (2015) | Recognition of unexpected findings | ⇑ recognition | 50 | P = .025 |
Depth perception | ⇑ perception | 50 | P = .015 | |
Diana et al (2017) | Image quality | ⇓ quality | 10 | P < .0001 |
Rosenthal et al (2002) | Operational error | ⇓ error | 50 | P < .02 |
Ruger et al (2020) | Operational error | ⇓ error | 20 | P < .022 |
Wilson et al (2013) | Success rate | ⇑ success | 34 | P < .008 |
Wei et al (2019) | Operational success | ⇑ patient outcome | 50 | P < .05 |
Patel et al (2022) | Knot tensile strength | ⇑ knot tensile strength | 15 | P < .001 |
Barragan et al (2022) | Tool changing event | ⇓ tool change | P < .01 | |
Clutching time | ⇓ clutching time | P < .01 |
N.S indicates no statistically significant change. ⇑ indicates increases/improvement in a given category compared to control. ⇓ indicates decreases in a given category compared to control.
aLarge sample size studies.
Table 4.
Characteristics of Studies Involving Surgical Predictions.
Study | Contribution | Results | Significance |
---|---|---|---|
Marz et al (2015) | Patient data driven diagnostic and treatment aid | 70 matching patients; 38±15 assertions per patient 84475 patient-related observations |
Assists physician in diagnosis, individualizing treatment and provide follow-up for patients that require liver resections |
Schoch et al (2016) | Patient data driven cardiac prosthetic selection aid | 55% elimination of unsuitable annuloplasty ring | Increased automation and range of selections for annuloplasty ring |
Patient Outcome
9 studies reported patient outcomes after the use of CST in their corresponding surgery (Table 5). 5 (56%) studies reported a minor or no post-operative complications. 2 (22%) studies observed slight bleeding in CST groups.24,25 3 studies reported serious either intraoperative or post-complication.21,25 Ntourakis reported a patient with a urinary tract infection and another patient with additional metastasis in CST groups. Diana reported intraoperative conversion from a MIS to an open surgery in the CST group.21,25
Table 5.
Characteristics of Studies Reporting Patients’ Outcomes With the Employment of CSTs.
Study | Patient Outcomes Measured | Conclusion | Patient Sample Size |
---|---|---|---|
Willekes et al (2008) | Post operative complications | No complications | 2 |
Lang et al (2005) | % of patients with complete tumor removal | 94.1% patient with complete tumor removal | 25 |
a Linxweiler et al (2020) | Post operative bleeding | Slight bleeding in experimental group | 100 |
Pain intensity | Reduced pain experimental group | 100 | |
Ntourakis et al (2015) | Operative blood loss | 0 - 50 mL | 3 |
Negative surgical margin | .7 - 17 mm | 3 | |
Post operative outcome | 1 patient had a urinary tract infection 1 patient had additional metastasis |
3 | |
Ogawa et al (2019) | Post operative complications | No complications | 41 |
Pain or numbness | No complications | 41 | |
Scarring | Minor complications | 41 | |
a Grasso et al (2015) | Procedural complications | No significant complications | 180 |
Diana et al (2017) | Surgical complications | 2 cases were converted to open surgery; 1 case was converted to laparoscopic surgery |
58 |
Wei et al (2016) | Post operative complication | No complications | 40 |
a Rompianesi et al (2023) | Post operative complication | 19 cases of complications (25%) | 61 |
Length of hospital stay | 5 | ||
Operative blood loss | 150 mL |
aLarge sample size studies.
OR Time Variation
18 studies reported an OR time difference (Table 6). 14 (78%) studies reported a trend in decreasing OR time, while 7 (38%) studies are statistically significant (P < .05). 4 (22%) studies reported an increase in OR time, 3 (18%) are statistically significant. Due to the inherent differences between the included procedures, the operational time between each procedure dramatically varies. To standardize the change in operational time, the differences are converted into a percentage change. Overall, there has been a 27.5% decrease in surgical time with the use of CST in comparison to traditional methods (mean: 27.5%; std: 41.8%) (Figure 3). Although the general pattern of CST use reduces procedural time, it is worth highlighting that robotics procedures tends to have longer operational time than their open surgery counterpart.
Table 6.
Characteristics of Studies Reporting Operation Time Changes With the Employment of CSTs.
Study | Results | Sample size | P value |
---|---|---|---|
Chi et al (2018) | ⇑ | 6 | P < .05 |
Abhari et al (2015) | ⇓ | 21 | P < .05 |
Azagury et al (2010) | ⇓ | 3 | P < .027 |
Marzano et al (2010) | ⇓ | 1 | N.S |
a Linxweiler et al (2020) | ⇑ (senior physicians) | 100 | P = .3350 |
⇑ (resident) | 100 | P = .1934 | |
Ogawa et al (2020) | ⇓ | 41 | P = .80 |
Dias et al (2020) | ⇓ | 45 | P < .001 |
Dilley et al (2019) | ⇑ | 36 | P ≤ .01 |
Dixon et al (2013) | ⇓ | 33 | P = .153 |
Grasso et al (2012) | ⇓ | 180 | P < .001 |
Jeon et al (2012) | ⇓ | 20 | P = .0002 |
Marcus et al (2015) | ⇓ | 50 | P = .007 |
Diana et al (2017) | ⇓ (NIR-C vs control) | 10 | P = .00000003 |
⇓ (AR vs control) | 10 | P = .000013 | |
Ruger et al (2020) | ⇓ | 20 | P = .211 |
Wei et al (2019) | ⇓ | 40 | P < .05 |
Wilson et al (2013) | ⇑ | 34 | N.S |
Patel et al (2022) | ⇓ | 15 | NS |
Barragan et al (2022) | ⇓ | 8 | P < .01 |
N.S indicates no statistically significant change. ⇑ indicates increases/improvement in each category compared to control. ⇓ indicates decreases in each category compared to control.
aLarge sample size studies.
Figure 3.
Procedure duration change with the implementation of CSTs.
Discussion
Medical imaging such as X-rays, CT scans, MRIs, and ultrasound data are widely used in surgical planning and intraoperative guidance. Augmented or mixed reality technologies utilize these images to create visual 3-D models for visual enhancement. However, tissue deformation during surgery can cause inaccuracies in these models. In actual surgery, target registrations are primarily done during the end-exhalation plateau phase of the patient’s respiratory cycle to ensure accurate markers for anatomical structures. 26 The margin of error for surgery registration ranges from 1 mm to 15 mm.27,28
Most CSTs are developed for specific types of surgeries, such as hepatic (ablation) cardiac surgery, vascular surgery with cannulation and catheterization technologies, laparoscopic surgery, lung surgery, and arthroplasty. One of the most important functions of CSTs for surgeons is visual assistance. Clinically, one of the procedures that considerably benefit from visual aid is liver resection, because portal blood vessels may be difficult to visualize under the liver, residual liver volume is hard to estimate, and there may be hidden tumours that are not detected initially.25,26,29,30 Patient-specific virtual reality anatomical models may be built through preoperative or intra-operative images depending on data availability.26,30,31 During planning, surgeons may freely rotate the model to detect unexpected tumours and adjust the resection plane accordingly; based on the resection plane, the software returns the residual liver volume as well as security margins around the liver for surgeons to analyze the validity of the plan.26,30 Patient-specific digital models can be generated to support valve replacement in heart surgeries.32,33 Trajectory optimization done by machine learning can reduce the risks in cannulation and catheterization procedures. 34 In laparoscopic gastrectomy, CSTs serve to recommend ideal port locations for increased tool maneuverability. 35 Current CSTs developed for lung biopsy assist the operator by providing a real-time augmented reality video displaying the position of the needle and surrounding tissue. 19 Regardless of the type of operation, CSTs aim to reduce surgeons' cognitive load and decrease surgical complications. Preoperative and intra-operative cognitive support technologies (CSTs) have been developed to reduce the cognitive load of surgeons during surgical procedures. Preoperative CSTs simplify anatomical complexities, aid in prosthetic selection, and provide visualization; the major intended role of preoperative CSTs is to aid the surgeon accurately and efficiently in patient diagnosis and surgical planning. For example, in mitral valve replacement surgery, an algorithm was created to reduce commercial mitral valve options based on patient-specific heart dimensions and medical history. 14 In neurosurgery planning, a CST was developed to create a dynamic 3D model of a patient's brain and pathology, providing recommendations for surgical procedures.22,36 CSTs should progress from cognitive support for individual surgeons to support for the entire surgical team, utilizing virtual and augmented reality to improve communication and decision-making efficiency. During the intraoperative phase, the consequences of overloading the surgeon’s cognition are often more serious, leading to poorer adverse patient outcomes. Intraoperative cognitive overload primarily comes from visual disturbances, unexpected events, and variable environmental factors.24,37-39 Thus, the primary role of intraoperative CSTs is to alleviate these problems and to stabilize the surgeon’s cognitive workload. , For example, to address the problem of intraoperative visual disturbances in endoscopic-related surgeries, CSTs that provide a 3D visualization of an endoscope tip and software that allows for navigation aids were often prioritized; this type of assistance is especially beneficial for surgeries that contain a trans-nasal component because navigation is difficult in this subset of surgeries.24,37,38 In variable and intense environments like the battlefield, Auditory guidance headsets with micro-speaker have been developed to treat needle decompression of tension pneumothoraxes for combat medics. 39 Robotic innovations like the Da Vinci robot and the Xi robot are being implemented in an increasing amount of surgical specialties. Innovations in surgical robotics have shown to improve motion stability and decrease tissue destruction.40,41 Simple CSTs, such as tattooing with methylene markers, have also been developed to aid in target localization. 42 Robotic surgery can also be combined with intraoperative fluorescent markers to provide surgeons with improved visualization of the anatomical structures around the resection plane and tumour location. 43 To assess the cognition levels of surgeons during robotic surgery, machine learning-based CSTs are able to analyze physiological parameters such as electroencephalogram and eye behaviour to detect moments of high cognitive load; in response to cognitive overload, the semi-autonomous program may take control of simple tasks such as blood suction and irrigation. 44 More invasive neurostimulation techniques were also tested; transcranial direct current stimulation on surgeons completing a knot typing task in robotic surgery was reported to have increased knot tensile strength. 45 Cognitive surgical technologies (CSTs) incorporating augmented reality (AR) and image guidance have shown promising results in improving precision and reducing errors in various surgical procedures. Similar to other visual aid CSTs, AR-based CSTs also rely on radiology images to create a patient-specific 3-D model. Intraoperatively, anatomical landmarks around the surgical site and security margin are labelled to the surgeons to aid their incisions; clarifications are given to surgeons around complex anatomical regions, and some CSTs provide procedural instruction during the operation. Studies have reported increased accuracy rates in AR or voice-guided surgery, with decreased error rates of up to 10%.15,16,24,27,34,36,37,46,47 AR-based CSTs have been effective in treating hepatocellular carcinoma, neonatal intubation, laparoscopic nephrectomy, and needle biopsies, resulting in increased success rates and reduced error rates.15,19,29,48 However, in some procedures like total hip arthroplasty and breast cancer-related lymphedema diagnoses, the implementation of CSTs did not result in significant differences.18,49 The presence of CST may also influence procedure time, with increased distraction or unfamiliarity with the device potentially reducing efficiency.24,34,39,46 However, there is insufficient evidence available on intraoperative CSTs to support the conclusion that it significantly lowers cognitive load and boosts performance, future research should focus on this area. With more development, CSTs may be crucial in reducing the cognitive load of surgeons during surgical procedures, improving communication and decision-making efficiency, and ensuring patient safety. The development of CSTs should continue to focus on reducing procedure complexity, reducing error rates, and aiding intraoperative decision-making to improve surgical outcomes. Current CSTs have limitations that hinder their widespread use in the OR. One major problem is the inadequacy of existing VR and AR technology to handle tissue deformation and organ fluctuation, requiring the intervention of expert computer scientists during surgery.25,28,38 Additionally, there is a lack of randomized controlled trials and sufficient clinical data due to the disconnection between medical staff and computer scientists, leading to outsourcing clinical trials to actual hospitals after CST development. Furthermore, CSTs often lack ergonomic design for effective surgical implementation, affecting navigation, reducing the field of view, and other ergonomics-related problems.16,36 Surgeons also have a preference for traditional techniques, and CSTs are often expensive, limiting their use in routine procedures. Learning curve problems and the potential for attentional blindness also pose challenges to CST use. CSTs may be more suitable for complex procedures requiring cognitive assistance, and future development should prioritize ergonomic design to improve usability and maneuverability. Overall, CSTs require further research and development to improve their effectiveness and gain wider acceptance in the ORs.
Conclusion
This review suggests that CSTs have shown promising results in improving surgical performance and reducing errors by increasing surgeons' visual fields. CSTs are currently in development for a wide variety of surgery specialties, and its purpose are often multi-dimensional, aiding surgeons in different areas before and during the procedure. Some advantages of CST, such as accuracy-based parameters, are supported with more evidence than other. However, several challenges still need to be addressed, including inaccuracies in accounting for tissue deformation, a lack of clinical data, particularly patient data, and ergonomic issues with bulky equipment. When developing these CSTs, adequate consideration must also be given to the possibility that the addition of CSTs would fatigue the surgical operator and cause them to regress in performance. Despite these limitations, the potential benefits of CSTs make them a promising area for further development and research, in today’s innovative age, improvements in hardware, software and design could overcome these technological challenges, furthering improving the medical field with interdisciplinary efforts.
Footnotes
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Zhong Shi Zhang has been funded by Alberta Innovate Summer Studentship for this research.
ORCID iDs
Zhong Shi Zhang https://orcid.org/0000-0002-5836-7257
Bin Zheng https://orcid.org/0000-0003-3476-5936
References
- 1.Owens CD, Stoessel K. Surgical site infections: epidemiology, microbiology and prevention. J Hosp Infect. 2008;70(Suppl 2):3-10. doi: 10.1016/S0195-6701(08)60017-1. [DOI] [PubMed] [Google Scholar]
- 2.Coccolini F, Catena F, Pisano M, et al. Open versus laparoscopic cholecystectomy in acute cholecystitis. Systematic review and meta-analysis. Int J Surg. 2015;18:196-204. doi: 10.1016/j.ijsu.2015.04.083. [DOI] [PubMed] [Google Scholar]
- 3.Martínez-Pérez A, de’Angelis N, Brunetti F, et al. Laparoscopic vs. open surgery for the treatment of iatrogenic colonoscopic perforations: a systematic review and meta-analysis. World J Emerg Surg. 2017;12:8. doi: 10.1186/s13017-017-0121-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Nicolás M, Czerwonko M, Ardiles V, et al. Laparoscopic vs open liver resection for metastatic colorectal cancer: analysis of surgical margin status and survival. Langenbecks Arch Surg. 2022;407(3):1113-1119. doi: 10.1007/s00423-021-02396-2. [DOI] [PubMed] [Google Scholar]
- 5.Mazur LM, Mosaly PR, Jackson M, et al. Quantitative assessment of workload and stressors in clinical radiation oncology. Int J Radiat Oncol. 2012;83(5):e571-e576. doi: 10.1016/j.ijrobp.2012.01.063. [DOI] [PubMed] [Google Scholar]
- 6.Lacy AM, García-Valdecasas JC, Delgado S, et al. Laparoscopy-assisted colectomy versus open colectomy for treatment of non-metastatic colon cancer: a randomised trial. Lancet Lond Engl. 2002;359(9325):2224-2229. doi: 10.1016/S0140-6736(02)09290-5. [DOI] [PubMed] [Google Scholar]
- 7.van Bergen P, Kunert W, Bessell J, Buess GF. Comparative study of two-dimensional and three-dimensional vision systems for minimally invasive surgery. Surg Endosc. 1998;12(7):948-954. doi: 10.1007/s004649900754. [DOI] [PubMed] [Google Scholar]
- 8.Wheelock A, Suliman A, Wharton R, et al. The impact of operating room distractions on stress, workload, and teamwork. Ann Surg. 2015;261(6):1079-1084. doi: 10.1097/SLA.0000000000001051. [DOI] [PubMed] [Google Scholar]
- 9.Mcmullan RD, Urwin R, Gates P, Sunderland N, Westbrook JI. Are operating room distractions, interruptions and disruptions associated with performance and patient safety? A systematic review and meta-analysis. Int J Qual Health Care. 2021;33(2):mzab068. doi: 10.1093/intqhc/mzab068. [DOI] [PubMed] [Google Scholar]
- 10.Sevdalis N, Wong HWL, Arora S, et al. Quantitative analysis of intraoperative communication in open and laparoscopic surgery. Surg Endosc. 2012;26(10):2931-2938. doi: 10.1007/s00464-012-2287-3. [DOI] [PubMed] [Google Scholar]
- 11.Parker SL, Mendenhall SK, Shau DN, et al. Minimally invasive versus open transforaminal lumbar interbody fusion for degenerative spondylolisthesis: comparative effectiveness and cost-utility analysis. World Neurosurg. 2014;82(1-2):230-238. doi: 10.1016/j.wneu.2013.01.041. [DOI] [PubMed] [Google Scholar]
- 12.Berguer R, Smith WD, Chung YH. Performing laparoscopic surgery is significantly more stressful for the surgeon than open surgery. Surg Endosc. 2001;15(10):1204-1207. doi: 10.1007/s004640080030. [DOI] [PubMed] [Google Scholar]
- 13.Cohen AJ, Lui H, Zheng M, et al. Rates of serious surgical errors in California and plans to prevent recurrence. JAMA Netw Open. 2021;4(5):e217058. doi: 10.1001/jamanetworkopen.2021.7058. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Schoch N, Philipp P, Weller T, et al. In: Webster RJ, Yaniv ZR, eds. Cognitive tools pipeline for assistance of mitral valve surgery. Washington, DC: NASA; 2016 [Google Scholar]
- 15.Dias PL, Greenberg RG, Goldberg RN, Fisher K, Tanaka DT. Augmented reality–assisted video laryngoscopy and simulated neonatal intubations: a pilot study. Pediatrics. 2021;147(3):e2020005009. doi: 10.1542/peds.2020-005009. [DOI] [PubMed] [Google Scholar]
- 16.Rüger C, Feufel MA, Moosburner S, Özbek C, Pratschke J, Sauer IM. Ultrasound in augmented reality: a mixed-methods evaluation of head-mounted displays in image-guided interventions. Int J Comput Assist Radiol Surg. 2020;15(11):1895-1905. doi: 10.1007/s11548-020-02236-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Thienphrapa P, Bydlon T, Chen A, et al. Interactive endoscopy: a next-generation, streamlined user interface for lung surgery navigation. In: Shen D, Liu T, Peters TM, eds. Medical Image Computing and Computer Assisted Intervention – MICCAI 2019. Vol 11768. Lecture Notes in Computer Science. Berlin, Germany: Springer International Publishing; 2019:83-91. doi: 10.1007/978-3-030-32254-0_10. [DOI] [Google Scholar]
- 18.Ogawa H, Kurosaka K, Sato A, Hirasawa N, Matsubara M, Tsukada S. Does an augmented reality-based portable navigation system improve the accuracy of acetabular component orientation during tha? A randomized controlled trial. Clin Orthop. 2020;478(5):935-943. doi: 10.1097/CORR.0000000000001083. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Grasso RF, Luppi G, Cazzato RL, et al. Percutaneous computed tomography-guided lung biopsies: preliminary results using an augmented reality navigation system. Tumori. 2012;98(6):775-782. doi: 10.1177/030089161209800616. [DOI] [PubMed] [Google Scholar]
- 20.Dixon BJ, Daly MJ, Chan H, Vescan AD, Witterick IJ, Irish JC. Surgeons blinded by enhanced navigation: the effect of augmented reality on attention. Surg Endosc. 2013;27(2):454-461. doi: 10.1007/s00464-012-2457-3. [DOI] [PubMed] [Google Scholar]
- 21.Diana M, Soler L, Agnus V, et al. Prospective evaluation of precision multimodal gallbladder surgery navigation: virtual reality, near-infrared fluorescence, and X-ray-based intraoperative cholangiography. Ann Surg. 2017;266(5):890-897. doi: 10.1097/SLA.0000000000002400. [DOI] [PubMed] [Google Scholar]
- 22.Marcus HJ, Pratt P, Hughes-Hallett A, et al. Comparative effectiveness and safety of image guidance systems in neurosurgery: a preclinical randomized study. J Neurosurg. 2015;123(2):307-313. doi: 10.3171/2014.10.JNS141662. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.März K, Hafezi M, Weller T, et al. Toward knowledge-based liver surgery: holistic information processing for surgical decision support. Int J Comput Assist Radiol Surg. 2015;10(6):749-759. doi: 10.1007/s11548-015-1187-0. [DOI] [PubMed] [Google Scholar]
- 24.Linxweiler M, Pillong L, Kopanja D, et al. Augmented reality‐enhanced navigation in endoscopic sinus surgery: a prospective, randomized, controlled clinical trial. Laryngoscope Investig Otolaryngol. 2020;5(4):621-629. doi: 10.1002/lio2.436. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Ntourakis D, Memeo R, Soler L, Marescaux J, Mutter D, Pessaux P. Augmented reality guidance for the resection of missing colorectal liver metastases: an initial experience. World J Surg. 2016;40(2):419-426. doi: 10.1007/s00268-015-3229-8. [DOI] [PubMed] [Google Scholar]
- 26.Hansen C, Köhn A, Schlichting S, et al. Intraoperative modification of resection plans for liver surgery. Int J Comput Assist Radiol Surg. 2008;3(3-4):291-297. doi: 10.1007/s11548-008-0161-5. [DOI] [Google Scholar]
- 27.Birkfellner W, Figl M, Matula C, et al. Computer-enhanced stereoscopic vision in a head-mounted operating binocular. Phys Med Biol. 2003;48(3):N49-57. doi: 10.1088/0031-9155/48/3/402. [DOI] [PubMed] [Google Scholar]
- 28.Haouchine N, Dequidt J, Peterlik I, Kerrien E, Berger MO, Cotin S. Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery. In: 2013 IEEE international symposium on mixed and augmented reality (ISMAR), Adelaide, Australia, 1-4 October 2013 [Google Scholar]
- 29.Lang H, Radtke A, Hindennach M, et al. Impact of virtual tumor resection and computer-assisted risk analysis on operation planning and intraoperative strategy in major hepatic resection. Arch Surg. 2005;140(7):629-638. doi: 10.1001/archsurg.140.7.629 [DOI] [PubMed] [Google Scholar]
- 30.Debarba HG, Zanchet DJ, Fracaro D, Maciel A, Kalil AN. Efficient liver surgery planning in 3D based on functional segment classification and volumetric information. Annu Int Conf IEEE Eng Med Biol Soc. 2010;2010:4797-4800. doi: 10.1109/IEMBS.2010.5628026. [DOI] [PubMed] [Google Scholar]
- 31.De Buck S, Maes F, Ector J, et al. An augmented reality system for patient-specific guidance of cardiac catheter ablation procedures. IEEE Trans Med Imaging. 2005;24(11):1512-1524. doi: 10.1109/TMI.2005.857661. [DOI] [PubMed] [Google Scholar]
- 32.Hagenah J, Evers T, Scharfschwerdt M, Schweikard A, Ernst F. A Support Vector Regression-Based Data-Driven Leaflet Modeling Approach for Personalized Aortic Valve Prosthesis Development. Lübeck, Germany: University of Lubeck; 2018 [Google Scholar]
- 33.Hagenah J, Mehdi M, Ernst F. Generating healthy aortic root geometries from ultrasound images of the individual pathological morphology using deep convolutional autoencoders. In: 2019 Computing in Cardiology (CinC), Singapore, 08-11 September 2019 [Google Scholar]
- 34.Chi W, Liu J, Abdelaziz MEMK, et al. Trajectory optimization of robot-assisted endovascular catheterization with reinforcement learning. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), Madrid, Spain, 01-05 October 2018 [Google Scholar]
- 35.Hayashi Y, Misawa K, Mori K. Optimal port placement planning method for laparoscopic gastrectomy. Int J Comput Assist Radiol Surg. 2017;12(10):1677-1684. doi: 10.1007/s11548-017-1548-y. [DOI] [PubMed] [Google Scholar]
- 36.Abhari K, Baxter JSH, Chen ECS, et al. Training for planning tumour resection: augmented reality and human factors. IEEE Trans Biomed Eng. 2015;62(6):1466-1477. doi: 10.1109/TBME.2014.2385874. [DOI] [PubMed] [Google Scholar]
- 37.Azagury DE, Ryou M, Shaikh SN, et al. Real-time computed tomography-based augmented reality for natural orifice transluminal endoscopic surgery navigation. Br J Surg. 2012;99(9):1246-1253. doi: 10.1002/bjs.8838. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Marzano E, Piardi T, Soler L, et al. Augmented reality-guided artery-first pancreatico-duodenectomy. J Gastrointest Surg. 2013;17(11):1980-1983. doi: 10.1007/s11605-013-2307-1. [DOI] [PubMed] [Google Scholar]
- 39.Wilson KL, Doswell JT, Fashola OS, et al. Using augmented reality as a clinical support tool to assist combat medics in the treatment of tension pneumothoraces. Mil Med. 2013;178(9):981-985. doi: 10.7205/MILMED-D-13-00074. [DOI] [PubMed] [Google Scholar]
- 40.Trastulli S, Desiderio J, Lin JX, et al. Open vs robotic gastrectomy with D2 lymphadenectomy: a propensity score-matched analysis on 1469 patients from the IMIGASTRIC prospective database. Langenbecks Arch Surg. 2023;408(1):302. doi: 10.1007/s00423-023-03032-x. [DOI] [PubMed] [Google Scholar]
- 41.Toesca A, Sangalli C, Maisonneuve P, et al. A randomized trial of robotic mastectomy versus open surgery in women with breast cancer or BrCA mutation. Ann Surg. 2022;276(1):11-19. doi: 10.1097/SLA.0000000000004969. [DOI] [PubMed] [Google Scholar]
- 42.Willekes L, Boutros C, Goldfarb MA. VATS intraoperative tattooing to facilitate solitary pulmonary nodule resection. J Cardiothorac Surg. 2008;3(1):13. doi: 10.1186/1749-8090-3-13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Rompianesi G, Pegoraro F, Ramaci L, Ceresa CD, Montalti R, Troisi RI. Preoperative planning and intraoperative real-time navigation with indocyanine green fluorescence in robotic liver surgery. Langenbecks Arch Surg. 2023;408(1):292. doi: 10.1007/s00423-023-03024-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Barragan JA, Yang J, Yu D, Wachs JP. A neurotechnological aid for semi-autonomous suction in robotic-assisted surgery. Sci Rep. 2022;12(1):4504. doi: 10.1038/s41598-022-08063-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Patel R, Suwa Y, Kinross J, et al. Neuroenhancement of surgeons during robotic suturing. Surg Endosc. 2022;36(7):4803-4814. doi: 10.1007/s00464-021-08823-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Dilley JWR, Hughes-Hallett A, Pratt PJ, et al. Perfect registration leads to imperfect performance: a randomized trial of multimodal intraoperative image guidance. Ann Surg. 2019;269(2):236-242. doi: 10.1097/SLA.0000000000002793. [DOI] [PubMed] [Google Scholar]
- 47.Rosenthal M, State A, Lee J, et al. Augmented reality guidance for needle biopsies: an initial randomized, controlled trial in phantoms. Med Image Anal. 2002;6(3):313-320. doi: 10.1016/S1361-8415(02)00088-9. [DOI] [PubMed] [Google Scholar]
- 48.Baumhauer M, Simpfendörfer T, Müller-Stich BP, et al. Soft tissue navigation for laparoscopic partial nephrectomy. Int J Comput Assist Radiol Surg. 2008;3(3-4):307-314. doi: 10.1007/s11548-008-0216-7. [DOI] [Google Scholar]
- 49.Invernizzi M, Runza L, De Sire A, et al. Integrating augmented reality tools in breast cancer related lymphedema prognostication and diagnosis. J Vis Exp. 2020:156:60093. doi: 10.3791/60093. [DOI] [PubMed] [Google Scholar]