The role of artificial intelligence in surgical simulation

Abstract

Artificial Intelligence (AI) plays an integral role in enhancing the quality of surgical simulation, which is increasingly becoming a popular tool for enriching the training experience of a surgeon. This spans the spectrum from facilitating preoperative planning, to intraoperative visualisation and guidance, ultimately with the aim of improving patient safety. Although arguably still in its early stages of widespread clinical application, AI technology enables personal evaluation and provides personalised feedback in surgical training simulations. Several forms of surgical visualisation technologies currently in use for anatomical education and presurgical assessment rely on different AI algorithms. However, while it is promising to see clinical examples and technological reports attesting to the efficacy of AI-supported surgical simulators, barriers to wide-spread commercialisation of such devices and software remain complex and multifactorial. High implementation and production costs, scarcity of reports evidencing the superiority of such technology, and intrinsic technological limitations remain at the forefront. As AI technology is key to driving the future of surgical simulation, this paper will review the literature delineating its current state, challenges, and prospects. In addition, a consolidated list of FDA/CE approved AI-powered medical devices for surgical simulation is presented, in order to shed light on the existing gap between academic achievements and the universal commercialisation of AI-enabled simulators. We call for further clinical assessment of AI-supported surgical simulators to support novel regulatory body approved devices and usher surgery into a new era of surgical education.

Introduction

In recent years, artificial intelligence (AI) technology has fuelled the advancement of surgical simulators, enhancing their accuracy and capabilities. Surgical simulators are increasingly used in modern medical practice to support preoperative planning and enhance the hands-on experience of training a surgeon. The application of modern AI technology can support simulators in providing personalised feedback to the user, while also automating an immersive surgical experience for visualisation of patient anatomy (1–5). In this review, we aim to discuss the current state of AI technology in surgical simulation. By reviewing the available literature and through our consolidated list of FDA/CE approved AI-enabled medical devices, we attempt to highlight recent advancements in the field as well as discuss some gaps between the research and industry communities. The key terms relevant to this article are summarised in Table 1.

Table 1.

Definitions.

Terminology	Definition
Artificial intelligence (AI)	Mimicking of human intelligence through ability to predict, classify, predict, learn, plan, reason and/or perceive to make decisions
Machine learning (ML)	Sub-branch of AI technology that processes calculations and statistics to learn from data without human supervision
Deep learning (DL)	Subset of ML that uses neural networks to solve more complex challenges such as image, audio, and video classifications.
Artificial neural network (ANN)	A collection of simple interconnected algorithms that process information in response to external input
Convolutional neural network (CNN)	A deep learning neural network designed for processing structured arrays of data such as images.
Augmented reality (AR)	A class of ANN; deep learning neural network designed for processing structured arrays of data such as images.
Virtual reality (VR)	A computer-generated environment that replaces the real environment with a digital environment where users interact with virtual objects
Mixed reality (MR)	A hybrid environment that merges both real and virtual worlds with digital and physical objects that can interact in real-time
Extended reality (XR)	A complete immersive environment which integrates AR, VR, and MR technologies to provide an entire spectrum of reality to virtuality.
Head-mounted display (HMD)	A display device, worn on the head or as part of a helmet, that has a small display optic in front of one or each eye.

The application of AI in surgical simulation assessment

AI can improve surgical training simulators by evaluating performance and providing personalised feedback to the end user (6). Several AI-based algorithms have been described in recent years. For instance, a neurosurgical group at McGill University, Montreal, developed a machine learning (ML) algorithm that classifies participants’ skill levels while performing a VR-based hemilaminectomy or brain tumour resection task (7, 8). More recently, the same group developed a “Virtual operative assistant (VOA)”—an open-sourced AI-based software that, in addition to determining the skill level, also provides personalised feedback in relation to expert proficiency performance benchmarks.

However, like any form of emerging technology, AI-driven performance evaluation and feedback generation remains imperfect with numerous limitations. Lam et al.'s systematic review explored several different ML techniques assessing surgical performance and noted the promising accuracy of ML-based assessment software, while delineating several major challenges and limitations of this technology (9). One limitation is that the surgical skill assessment of a simulated benchtop task might not accurately correlate with the trainee's performance in the operating room, as the real-life surgical environment is highly complex. Furthermore, there is a lack of validated assessment criteria against which participants’ performance should be assessed by the AI-algorithm. Another major barrier is the exorbitant volume of surgical data required to effectively train the ML-algorithms. Potential solutions to some of these challenges include developing a surgeon-led consensus statement outlining the core elements of a surgical technique to be evaluated by ML-algorithms and facilitating surgical data exchange through cross-institutional open-source initiatives (10).

Interestingly, we note that our review of the FDA/CE list did not identify any approved AI-powered software for appraising and providing feedback in surgical simulation. However, it is conceivable that such software may be commercially available but have not been evaluated through the regulatory process as they do not directly interact with patients (11).

The application of AI in surgical visualisation

3D visualisation has been a major development in surgical simulation across all surgical specialties ranging from neurosurgery and orthopaedics to maxillofacial, plastics, and general surgery (12, 13). Such universal demand for 3D presurgical planning and its steadfast advancement since the 1980s testifies to the substantial benefits in reducing operation duration, blood loss, and hospital stay (14–18) whilst improving patient survival. Our review of the FDA/CE list identified 11 AI-enabled visualisation devices (Table 2). It is important to note that this list does not include visualisation technology without obvious AI application.

Table 2.

List of FDA/CE approved AI-powered devices.

Device name	Approval date	FDA submission number	Type of approval	Other approval	Approval date	Description	Company	Country	Software function	Source
UNiD Spine Analyzer	5/24/2017	K170172	510 (k) premarket notification			Device that allows surgeons to view, measure images, and to perform generic spine related presurgical simulations for placement of surgical implants	Medicrea International (Device acquired by Medtronic)	USA	2D simulation	1,3
D2P	8/29/2019	K183489	510 (k) premarket notification	CE	10/2017	Pre-operative image segmentation and DICOM visualisation software for surgical planning	3D Systems, Inc.	USA	3D printing, VR simulation	1,2,3,4
HealthJOINT	12/04/2020	K202487	510 (k) premarket notification	NA	NA	Software for 3D reconstruction from 2D images for preoperative planning of knee orthopaedic surgical procedures	Zebra Medical Vision Ltd (Acquired by Nanox.ai)	USA	3D reconstruction	1,3
Innersight3D	NA	NA	NA	CE	06/2019	Modelling technology for surgery planning using models from MRI and CT scans	InnersightLabs Ltd.	UK	3D reconstruction	4
MEDIP PRO	11/07/2019	K191026	510 (k) premarket notification	CE	12/2019	Software interface and image segmentation system for transfer of DICOM imaging information from medical scanner to output file; pre-operative software for treatment planning	Medical IP Co., Ltd.	South Korea	3D reconstruction, 3D Printing, MR/XR Simulation	1,3,4,5
NemoScan	12/11/2019	K192571	510 (k) premarket notification	NA	NA	Diagnostic and treatment support for dental implantology	Software Nemotec S.L.	Spain	3D reconstruction, 3D Printing	1,4
Profuse CAD	11/21/2018	K173744	510 (k) premarket notification	NA	NA	Standalone Computer Aided Detection (CAD) software used to visualise, analyse, plan and interpret CT, MRI, PET images	Eigen	USA	3D reconstruction	2
SurgicalAR	5/13/2019	K190764	510 (k) premarket notification	NA	NA	Display of medical images and other healthcare data, including functions for image review, basic measurements and 3D visualization	Medivis, Inc.	USA	MR/XR Simulation	1,4
Synapse 3D, Synapse 3D Base Tools v6.1	02/09/2021	K203103	510 (k) premarket notification	NA	NA	Software for advanced processing and analysis of imaging aiding in surgical decision support and preoperative simulation	Fujifilm Corporation	USA	3D Reconstruction	1
Syngo Application Software	06/09/2017	K170747	510 (k) premarket notification	CE	03/2017	Provides real-time viewing, image manipulation, 3D-visualisation, communication, and storage of medical images	Siemens Medical Solutions, USA, Inc.	USA	3D reconstruction	1,2,4
VSI Holomedicine	NA	NA	NA	CE	12/2019	Surgery support through mixed reality	apoQlar GmbH	Germany	VR simulation, MR/XR Simulation	4

1. Artificial Intelligence and Machine Learning (AI/ML) Medical Devices [Internet]. U.S. Food and Drug Administration. 2021. Available from: https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices#resources.

2. Wu E, Wu K, Daneshjou R, Ouyang D, Ho D, Zou J. How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals. Nature Medicine. 2021;27(4):582-584.

3. Dreyer K, Wald C, Allen Jr. B, Agarwal S, Bizzo B, Gichoya J et al. AI Central [Internet]. AI Central. 2022. Available from: https://aicentral.acrdsi.org/.

4. Muehlematter U, Daniore P, Vokinger K. Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015–20): a comparative analysis. The Lancet Digital Health. 2021;3(3):e195-e203.

5. STAT Database: AI Tools [Internet]. Statnews.com. 2021. Available from: https://www.statnews.com/wp-content/uploads/2021/02/STAT_FDA_cleared-_AI_tools.xlsx.

The most widely accepted technique for creating 3D simulations from 2D images involves convoluted neural network (CNN) algorithms. AI can complement this method by facilitating 3D segmentation and anatomical labeling (19–21). In addition, AI plays a crucial role in facilitating advanced visualisation modalities such as 3D printing, virtual reality (VR) simulation, and extended reality (XR) environment (Figure 1).

Figure 1.

Summary of AI function in visualisation technology.

3D-printed simulators have been shown to facilitate preoperative planning, improve surgical outcomes and decrease operation duration (22); however, several challenges have hindered its daily application (23). AI has removed the largest barrier of implementing 3D printing—the need for expert knowledge in the 3D-printing process and computer-based design. Manual processes have been automated through AI algorithms that can efficiently process large amounts of data needed to prefabricate a model, convert through slicer, detect, and rectify any errors, and print. Moreover, AI-driven fabrication that adopts real-time adaptation poses a solution to 3D-printing's challenge of reproducing in-vivo tissue (24). AI technology is essential to support a VR environment and automating the workflow. Software such as DICOM to Print (D2P) and MEDIP PRO produces instantaneous 3D Printing and VR rendering options. Moreover, AI is effectively used for segmentation and anatomical annotation of the VR environment and, more importantly, processing user input needed to facilitate user interaction (25, 26). Furthermore, AI is required to support the XR-environment—the most practical form and the holy-grail of surgical simulation. In conjunction with computer vision technology, deep learning (DL) based object detection allows dynamic image identification (27). Such complex feedback processing from DL models bridges the mismatch between virtual and the real world in 3D holographic projections (28).

The simulation techniques are also integral in surgical training. VSI Holomedicine, for example, provides a platform for both preoperative planning and anatomical education. Furthermore, hyperrealism is an AR concept used in training where artificial objects, such as a silicon replica, are visually enhanced by DL algorithms to simulate a realistic scene of surgery (29).

Discussion

Technological challenges

Current AI frameworks used in surgical simulation have some technological limitations. To begin with, AI-powered feedback platforms could address the challenges experienced with the VOA (3). For example, VOA is restricted to categorising two expertise groups, whilst machine learning techniques such as artificial neural networks can be utilised to classify multiple groups of expertise (9, 30). VOA also highlighted the downfalls of using linear machine algorithms; there were instances of misclassification due to high positive scoring metrics overcompensating for other negative scoring metrics (3). Moreover, recent clinical experience with AI models demonstrated difficulties in measuring subjective combinations of actions such as “instrument handling (31).” In addition to overcoming these fundamental obstacles, a leap to translation will require succeeding platforms to be able to recognise and capture different frames from an entire simulation, in order to account for users’ variety of approaches and techniques that can adequately fulfil a single metric (3).

As seen in Figure 1, segmentation and anatomical labelling seems to be the major application of AI in surgical simulation; however, generalisability of these algorithms remains a challenge (32). Although segmentation has proven its efficacy in small clinical studies, its accuracy is yet to reach perfection (33). Studies reported that their algorithms failed to identify certain structures such as complex vasculature or certain types of tumours and that their accuracy can be highly dependent on the fidelity of different imaging modalities (32–34). Furthermore, to train AI algorithms to an absolute precision without bias and to be generalisable, experts are needed to construct comprehensive datasets which require numerous annotations and segmentation of anatomical structures. However, this manual process is tedious and resource intensive, hence, this approach cannot be translated into clinical practice (35). Such technical impediments are closely related to the fact that AI segmentation and anatomical labelling technology has varying progress across different specialties, pathologies, and procedures. Technologies currently in practice such as Syngo Application Software allow automated segmentation across several different specialties. Nevertheless, there is a lack of an overarching AI framework that can learn, train, and segment or label any anatomical structures in a human body. Some intraoperative AI models have been applying deep learning architectures to predict objects that were previously unseen by the AI (36). Albeit its infancy, this seems to be the future in AI-powered segmentation and anatomical labelling.

Additional technical challenges in the visualisation field are attributed to the intrinsic limitations of technological devices, [i.e., VR-sickness, head-mounted-displays (HMD), computing power of graphic processors] which is beyond the scope of this review. However, few AI-based solutions are proposed to overcome these challenges. VR-sickness is commonly experienced due to the delay between sensory input and VR. To reduce such latency, CNN models are employed to improve gaze-tracking by pre-rendering subsequent scenes and predicting future frames (37, 38). Another ground breaking area of development is in producing 3D displays for human eyes. Current AR/VR displays display 2D images to each of users’ eyes instead of 3D which is how individuals perceive the real world. Choi et al. recently proposed an AI algorithm and calibration technique for producing 3D images, namely the “Neural holography,” that allows a more realistic fidelity of VR rendering, comparable to that of an LCD display (39).

Affordable cost

At the moment, the use of AI-based simulation in healthcare comes at a significant financial cost. As traditional practice for both surgical training and pre-operative planning rely on largely inexpensive methods such as near-peer teaching and group discussions, the introduction of AI-based simulators incurs additional costs. The price of such simulators is rarely publicly disclosed and varies greatly depending on quality, capabilities, and use cases. For instance, none of the companies we identified in our FDA/CE database search provided public information on the prices of their product. However, it is well known that the cost of simulator technology is steep. According to some estimates, the price to equip a simulation lab can range anywhere between $100,000 to several millions (40). The use of AI-algorithms adds an additional layer of costs, especially when considering that such algorithms need to be regularly updated to ensure optimal performance and reliability (41).

It is, therefore, important to understand if the clinical benefit from AI-driven surgical simulators would justify the significant costs associated with their purchase and maintenance. Interestingly, the majority of published research failed to yet report a significant superiority of surgical simulators as compared to traditional methods in the realm of surgical education (42). For instance, Madan et al. reported that substituting virtual reality trainers with an inanimate box does not decrease the rate and level of laparoscopic skill acquisitions. Similarly, a systematic review by Higgins et al. failed to identify any studies directly comparing the surgical simulation with traditional surgical skill development in the operating room (43). However, in the area of surgical visualisation for pre-operative planning, there is some, albeit still limited, evidence that supports the superiority of AI-driven simulators compared to traditional surgical techniques (14, 15, 18).

At the same time, it is important to explore if the cost of AI-driven surgical simulators can be reduced while maintaining the value they offer in patient care. The concept of “fidelity” refers to how well a surgical simulator represents reality (44). A range of studies found that higher fidelity simulators were rarely superior to their lower fidelity counterparts in the area of surgical education (45, 46). However, perceptual fidelity might prove to be important in simulators aiming to support surgical visualisation and preoperative planning process (47), although there is currently limited literature on the topic. Therefore, the choice to pursue low-cost, low-fidelity simulator variants needs to be carefully reviewed and balanced with the extent of benefits they can provide.

In addition to pursuing lower-fidelity simulators, it is reasonable to expect the price of simulators will be kept within a reasonable range with growing market competition and widespread adoption of this technology. Furthermore, national healthcare services and large healthcare institutions may have the negotiating power to further reduce the prices when supplying their hospitals with AI-driven surgical simulators and related equipment.

Regulatory challenges

Presently, a clear overview of all approved AI-based medical devices and algorithms does not exist (48). Few efforts are underway to curate an inclusive database of these devices, in which FDA is at the forefront (49–51). Additionally, multiple other regulatory bodies such as the UK's Medicines and Healthcare products Regulatory Agency (MHRA), are making headway in developing regulatory guidelines to ensure safety and effectiveness of AI-powered devices and softwares (52). As valuable as these national endeavours are, it is crucial for internationally governing bodies to bring forth a guideline for implementing AI-powered surgical simulators universally (41). International Medical Device Regulators Forum (IMDRF) is a prime example of a group that can strive towards this goal (53).

To our knowledge, there are only a handful of AI-powered surgical simulators that are regulatory approved. Nevertheless, we emphasise the need for strict scrutiny of these products to ensure patient safety, transparency, and cyber security (41, 54). Although feedback softwares do not directly interact with patients, quality of training indirectly impacts patient care. For AI-based surgical simulation assessments to become the norm, like flight simulators, products should be driven to adhere to rigid requirements that uplift the quality of surgical training. From our experience with reviewing approved preoperative planning simulators, we highlight the significance of regulatory approval for several reasons. Products are often exaggerated or falsely marketed to be AI-powered, to boost their sales (48). Additionally, even for approved devices, the extent of use or the function of AI in their devices are only vaguely communicated to the public and the users. Ultimately, as preoperative simulators hold and analyse important patient information, they need to strictly abide by established data security guidelines for medical devices. Some preoperative simulators aim to automate the decision-making process which makes regulation even more pivotal. Despite the lack of data protection guidelines for AI specifically, the General Data Protection Regulation (GDPR) for example, contains elements that are relevant to such AI-based surgical planning simulators, including use of personal data, profiling, and automation of decision making (55). FDA's most recent endeavours share similar notions (50, 51).

Moving forward, global organisations like IMDRF need to corroborate on a consensus on AI regulation since the aforementioned issues are ubiquitous around the globe. As we aim towards universalisation of AI-based surgical simulators, legal and ethical issues regarding accountability and performance transparency of AI-based surgical simulators are crucial components that should be overseen by such governing bodies (54, 56).

Conclusion

Application of AI has the potential to enhance the quality of surgical simulators and expand their capabilities. In this review article, we have discussed the elements of surgical simulation which are enhanced using AI-driven technologies and reviewed the available literature and FDA/CE-approved products. We summarised the current landscape of AI in surgical simulations and provided suggestions for further clinical implementation in Figure 2. It is promising to see clinical evidence and technological reports attesting to the efficacy of AI-supported surgical simulators. However, the barriers to wide-spread commercialisation of these devices and software are complex and multifactorial. High implementation and production cost, scarcity of reports evidencing the superiority of such technology, and intrinsic technological limitations remain at the forefront. With this in mind, we call for further clinical assessment of AI-supported surgical simulators to support novel regulatory body approved devices and usher surgery into a new era of surgical education.

Figure 2.

Recommendation and suggestions for AI-powered surgical simulators.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Author contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.