Intraoperative use of mixed reality (MR) in humans across surgical disciplines: a major review

Abstract

The objective of the present work is to assess the intraoperative utility, operative benefits, and limitations of mixed reality (MR) by reviewing the entire literature across surgical disciplines. A focused literature search was conducted on December 12, 2024, across major databases, including PubMed, EMBASE, and Cochrane. The review included original studies that examined intraoperative MR applications in human surgical procedures with a technology readiness level (TRL) of 6 or higher. Exclusion criteria ruled out studies involving augmented reality (AR) or virtual reality (VR) alone, as well as non-English literature and conference proceedings. Selected studies were categorized based on their application in surgical navigation, image guidance, and image reference. A total of 41 studies met the inclusion criteria, covering 760 patients across multiple surgical disciplines. Three primary MR applications were identified: (1) Surgical navigation (MR-SN): used for real-time instrument tracking and trajectory definition in procedures such as spinal surgery, neurosurgery, and maxillofacial surgery; (2) Image guidance (MR-IG): employed for visual overlay of 3D models onto anatomical landmarks without instrument tracking, primarily in visceral, plastic, and reconstructive surgery; (3) Image reference (MR-IR): utilized as a static 3D reference model adjacent to the surgical field in various disciplines, including ophthalmology and interventional radiology. The Microsoft HoloLens and HoloLens 2 were the most used head-mounted displays (HMDs), with additional applications of Magic Leap 2 and xvision Spine System. MR demonstrated improvements in surgical precision, efficiency, and visualization. However, limitations such as ergonomic discomfort, latency, narrow field of view, and challenges in anatomical superimposition were noted. Mixed reality is an emerging technology with the potential to enhance intraoperative visualization, guidance, and navigation across multiple surgical disciplines. Several challenges limit its widespread adoption. Further clinical trials and regulatory approvals are required to establish MR as a standard tool in surgical practice.

Introduction

Mixed reality (MR) is garnering significant interest within the medical community due to its potential to enhance the surgical experience of several medical procedures and interventions across multiple surgical disciplines. Defined initially by Milgram et al. in 1994 and later revised by others, the definition and technology of mixed reality is still evolving [1, 2]. Currently, MR represents a real-world environment enriched with virtual data that can be interacted with through head-mounted displays (HMDs). It is important to differentiate MR from other platforms like virtual reality (VR) and augmented reality (AR). Virtual reality immerses the users in a completely virtual environment, whereas AR overlays virtual objects onto the real world without interactions. In contrast, MR anchors virtual data within the real world, allowing for real-time interaction regardless of user movement. This integration of patient-specific data with real-time observations can create interactive interfaces that facilitate procedural planning, intra-procedural navigation, training, and education [3]. Extended reality (XR) is an umbrella term encompassing augmented reality, virtual reality, and mixed reality, thus provides a comprehensive framework for understanding these technologies [3]. It has to be mentioned that MR can be obtained in 2 different ways: direct process (direct mixed reality, D-MR) and indirect process (indirect mixed reality, ID-MR), depending on the type of HMD used (Fig. 1) [4].

Fig. 1.

The Spectrum of Extended Reality (XR)

In the literature, there have been multiple reviews discussing the surgical utilization of VR in medicine, as well as papers addressing the use of AR and MR in medical applications (Fig. 2A) [3, 5–8]. Vervoorn et al. published a comprehensive paper on the application of mixed reality in surgical and interventional practices. However, it encompassed a broad spectrum of interventional applications, including surgical planning, intraoperative use, and educational purposes [3]. Our paper aims to provide a qualitative overview of MR, exclusively in clinical intraoperative practice performed on humans, revealing the current state of practical readiness of this technology and highlighting its advantages, limitations, and potential for the future.

Fig. 2.

Panel A: Surgical planning in VR: image of computed tomography-dacryocystography (CT-DCG) for planning of endoscopic dacryocystorhinostomy (EnDCR) in a case of posttraumatic lacrimal drainage obstruction in a setting of Le Fort fracture (index finger points at the lacrimal sac, blue arrow – indicates a metal plate in the vicinity of the lacrimal sac; image obtained with real-time volumetric rendering in Medical Imaging XR™ software, Medicalholodeck Inc., Zurich, Switzerland). Panel B: Flowchart of data collection

Materials and method

Because this study is a literature review, the following declarations are not applicable: “Human Ethics and Consent to Participate”, “Clinical trial number”, and “Consent to participate”. The research was conducted in accordance with the Tenets of the Declaration of Helsinki’. On December 12, 2024, a focused literature search was conducted across major databases, including PubMed, EMBASE and Cochrane to explore the application of mixed reality in intraoperative practices. The review was narrative in nature like Vervoorn et al. [3] because of the lack of uniform taxonomy in the area of extended reality. The search utilized the following keywords from titles and abstracts: “mixed reality”, “virtual reality”, “extended reality”, and “augmented reality”. The results were then added to form the initial database.

Three independent researchers (RN, MN, and MJA) screened the papers for relevance and originality. The inclusion criteria required original studies that focused exclusively on the intraoperative use of mixed reality (MR) in humans, with a technology readiness level (TRL) of 6 or higher, indicating that the technology had been demonstrated in a relevant clinical environment. The TRL scale assesses technological maturity across nine levels, with TRL 9 representing the highest level [9, 10]. Only articles describing the use of MR within the operating theatre during the course of surgery were included. Studies addressing surgical planning were eligible only if the planning was directly followed by a surgical procedure performed in the same operating session.

Exclusion criteria ruled out papers exclusively involving augmented or virtual reality. The distinction was based on the user’s ability to interact with the 3D object overlayed to the “real environment”, which is unique to MR. Additionally, non-English literature, abstracts, and conference proceedings were excluded. Reference lists of all selected papers were reviewed to identify further relevant studies. The search flowchart is depicted in Fig. 2B.

Results

Study designs

Of the 41 original articles included, 40 featured surgical procedures performed in operating rooms on 760 patients. One paper did not specify the number of surgeries. Study designs included: case reports [11–20], case series [21–38], case-control studies [39, 40], prospective nonrandomized studies [41, 42], prospective nonrandomized controlled studies [43–46] and prospective randomized controlled studies [47–50].

Types of intraoperative MR application

In the analyzed articles, three types of mixed reality applications were used during the surgical procedures: intraoperative navigation, image guidance, and image reference.

Surgical navigation

Surgical navigation (MR-SN) allows surgeons to precisely track the position of instruments and project this position onto preoperative imaging data. The initial step in any procedure involving surgical navigation is the process of “registration,” which aligns corresponding points in the patient’s anatomy with preoperative digital 3D model generated from computed tomography (CT) or magnetic resonance imaging (MRI) data. Surgical instruments are equipped with special markers (optical or electromagnetic) that enable the navigation system to determine their position and orientation in space. Optical tracking uses infrared cameras to detect marker positions, while electromagnetic tracking involves sensors that monitor instrument movements within an electromagnetic field [51, 52].

In conventional systems described above, a surgeon, while performing the medical procedure, must track his instruments on flat computer displays of the navigation unit. In articles describing the use of MR in surgical navigation, a surgeon, wearing a head-mounted display (HMD), can view the 3D reconstruction superimposed onto the surgical field and track surgical instruments directly there – in the space of surgical field. However, it is challenging for an MR navigation system to possess all the features of conventional intraoperative navigation systems. Therefore, we included in MR-SN group not only systems where the surgeon could track the position and movement of instruments (recognized and tracked by the system) but also where the system only defined the trajectory for the surgical instrument.

Integration of MR HMDs with established surgical navigation systems

Although most MR HMDs are used as stand-alone devices, technical integration with conventional surgical navigation (optical or electromagnetic) is both feasible and increasingly reported. In such integrated setups, the HMD functions as an in-situ visualization layer, while instrument pose is provided by an external tracker (or an embedded, FDA-cleared tracking stack) and fused with preoperative or intraoperative imaging.

Two principal integration patterns are observed:

A. Fully integrated MR–SN platforms.

Examples include the xvision Spine System™ (Augmedics, Arlington Heights, IL, U.S.A) and SurgicalAR™ (Medivis, New York, NY, U.S.A.). xvision couples an optical tracking stack with an HMD and has demonstrated high pedicle screw accuracy in clinical series (Gertzbein-Robbins grades A/B ≈ 97–100%) and prospective reports. SurgicalAR’s positional tracking was benchmarked using the FDA-recognized ASTM F2554-18 standard, yielding mean positional errors of 0.75 ± 0.37 mm in multicenter bench testing—indicating sub-millimetric tracker performance when the full integrated pipeline (HMD + software + localization instrument) is employed [14, 53].

B. Navigation-linked “heads-up” overlays.

In this configuration, a conventional neuronavigation station (e.g., StealthStation^™, Brainlab AG, Munich, Germany) provides instrument and patient pose, while the HMD renders the volumetric overlay. Early neurosurgical experiences have demonstrated feasibility in microscope-assisted and intracranial procedures [38].

By contrast, stand-alone HMD neuronavigation—where the headset performs room-scale SLAM (Simultaneous Localization and Mapping) and manual or fiducial registration without an external tracker - still shows variable accuracy. In a clinical pilot comparing HoloLens-based holographic navigation with conventional neuronavigation, the mean fluicidal registration error (FRE) was 4.4 ± 2.5 mm in patients (plastic head model 7.2 ± 1.8 mm), compared to 3.6 ± 0.5 mm and 1.9 ± 0.45 mm, respectively, for conventional systems. Minor hologram drift (≈ 1–2 mm) was also observed with movement around the patient [54]. Taken together, integrated MR–SN systems can achieve accuracy comparable to established navigation when external tracking is used, while purely stand-alone HMD pipelines continue to exhibit higher registration error and occasional spatial drift. Contemporary systematic and narrative reviews of standard neuronavigation report typical target registration error (TRE) in the 1–2 mm range, depending on modality, registration method, and workflow [55–57].

Image guidance

Image guidance (MR-IG) refers to a scenario like the one described above, because the registration is performed, but the instruments are neither being tracked by the system nor the trajectory for the instrument is mapped out. In this approach, a 3D reconstruction is overlaid onto the patient’s body and aligned with corresponding anatomical landmarks. The surgeon can only monitor the movement of instruments in the surgical field visually, without the aid of tracking markers or a pre-designed virtual trajectory for the instrument.

Image reference

Image reference (MR-IR) describes a context where the surgeon places the 3D reconstructed virtual model within the space above or adjacent to the surgical field, serving as a point of reference. Like in MR-IG, the surgical instruments are not equipped with markers. The reconstruction acts as a visual aid, and the surgeon relies solely on visual feedback to monitor instrument movements. Thus, MR-IR represents the simplest application of MR during a surgical procedure, while surgical navigation using MR constitutes the most advanced form of its implementation (Fig. 3A). Figure 3B illustrates the classification of systems for intraoperative use of MR during surgical procedures (Mixed Reality Intraoperative Systems; MR-IOS) based on analysis of the included articles.

Fig. 3.

Panel A: Example of use of MR Image Reference System (MR-IR): surgeon wearing Magic Leap 2 HMD (Magic Leap Inc., Plantation, FL, U.S.A), behind him a large screen displaying image streamed in real-time from HMD. Panel B: Mixed Reality Intraoperative Systems (MR-IOS)

Out of forty one papers 11 presented experience of the use of MR-IR systems [17–19, 21, 23, 26, 32, 35, 36, 41, 45], 12 referred to MR-IG systems [12, 13, 15, 16, 22, 28, 29, 40, 42, 43, 46, 50], whereas 17 described utilization of MR-SN systems [11, 14, 20, 24, 25, 27, 30, 31, 34, 37, 38, 43, 44, 47–49, 58]. One paper discussed both MR-IG and MR-SN [33]. We tried to extract the information about the type of image registration performed in MR-IG and MR-SN, but many articles did not specify the way that registration was performed. However, several types of this process could be distinguished, including the three main methods: manual registration [11, 13, 25, 29, 39, 40, 47], manual marker-based registration [11], and marker-based automatic registration [24, 27]. A surgeon using manual surface registration must manually overlay the 3D reconstruction from the surrounding space onto the patient’s body utilizing the natural anatomical landmarks like the nose or ear. In marker-based manual registration, the same markers are in virtual 3D reconstructions and on the patient’s body. The surgeon matches them together. During the automatic process in MR-SN this procedure is conducted by the software. To make the procedure more accurate MR-SN are often supported by intraoperative imaging systems like O-arm [14, 30, 34] or C-arm [34].

Hardware

The epidemiological context of the coronavirus disease 2019 pandemic has had wide- reaching impacts on all segments and sectors of society. This scenario represented a unique chance to speed up the significant investments by technology companies. Although the most commercially available XR (AR, MR, VR) goggles offer both mixed reality and virtual reality modes, only a few have been adopted for use in operating rooms. In practice, these are limited to HMDs based on direct mixed reality technology (D-MR), known as the “see-through” HMDs. In these devices, the user perceives the real world through transparent lenses on which virtual holographic 3D objects are overlaid, creating a blended mixed reality experience.

Devices equipped with “pass-through” technology (indirect mixed reality; ID-MR), on the other hand, are primarily designed as virtual reality goggles with an added mixed reality feature. In this case, the user views the surrounding real world along with superimposed 3D virtual objects on an opaque screen inside the HMD. However, the real-world environment is displayed via external cameras that capture the surroundings and project them onto the internal display. While the 3D objects are typically well rendered, the quality of the perceived environment depends heavily on the capabilities of the external cameras, which are, unsurprisingly, less effective than the human eye. In the operating room, lighting conditions often present significant challenges for cameras due to high contrasts, which can impact the accurate perception of the environment. This limitation makes the “see-through” HMDs more practical for surgical applications.

In the majority of the analyzed articles, specifically in 26 out of 41 cases, HoloLens HMD (Microsoft, Redmond, WA, U.S.A) was used to perform surgical procedures supported by mixed reality [11–13, 15, 16, 18, 21–27, 29, 31, 33, 36, 37, 39, 41, 43–45, 47, 50, 58]. In nine studies, the newer version, HoloLens 2 (Microsoft, Redmond, WA, U.S.A), was utilized [17, 19, 20, 28, 32, 40, 42, 46, 48]. One study from 2024 reported the use of Magic Leap 2 (Magic Leap Inc., Plantation, FL, U.S.A), and another three papers described the application of an HMD integrated into a surgical navigation system (xvision Spine System^™, Augmedics, Arlington Heights, IL, U.S.A ) [14, 30, 34]. Two studies did not provide enough data about to identify the HMDs [38, 49].

The predominant use of HoloLens in most studies is likely attributable to its longer market presence, as it has now been succeeded by the next generation—HoloLens 2. Table 1 presents a basic overview of the relevant currently commercially available in the market D-MR HMDs and two examples of ID-MR HMDs (Tables 1 and 2; Fig. 4).

Table 1.

Basic technical specifications for mixed reality HMDs

	HoloLens 2	Magic Leap 2	Apple Vision Pro	Meta Quest 3
Specifications
Type of MR	D-MR	D-MR	ID-MR	ID-MR
Release date	2019	2022	2024	2023
Price	app. 3500$	starting at 3299$	starting at 3499$	starting at 499$
Resolution (Pixels per eye)	2000 × 1500	1440 x1760	3660 × 3200	2064 × 2208
Field of view	52 deg. diagonal	70 deg. diagonal	100 deg. horizontal	110 deg. horizontal, 96 deg. vertical
Interaction	hand tracking, voice commands	hand tracking, controller, voice commands	hand tracking, voice commands	hand tracking, controllers
Eye tracking	yes	yes	yes	no
Computing	built-in	separate computer pack	built-in	built-in
Weight	566 g	260 g (headset) + 420 g (compute pack)	600–650 g	515 g
Battery life	up to 3 h of active use	up to 3.5 h of active use	up to 2 h with external battery	up to 2.2 h of general use

Abbreviations: HMD – head-mounted display, MR – mixed reality, D-MR – direct mixed reality, ID-MR – indirect mixed reality

Table 2.

MR hardware and software data and their FDA approval status

Hardware and software data
	Name of hardware/software (in alphabetical order)	Manufacturer’s details (company’s name, headquarters, country)	Description	FDA approved
1	Blender™	Blender Foundation, Amsterdam, the Netherlands	open-source 3D modeling software	-
2	Brainlab Curve® Image Guided Surgery system, Brainlab Elements™	Brainlab AG, Munich, Germany	Brainlab Curve® Image Guided Surgery is a navigation hardware platform (large dual display, optical/EM tracking, integrated workstation) that runs different applications; Brainlab Elements™ a software suite for image processing, segmentation, planning	yes
3	CarnaLife Holo™	MedApp, Krakow, Poland	software that enables 3D visualization of medical imaging data to support the planning and execution of medical procedures	yes
4	C-arm	Zheim Imaging GmbH, Nuremberg, Germany	medical imaging device used primarily in surgical procedures to capture real-time, high-resolution X-ray images; its name derives from the C-shaped arm that connects the X-ray source to the detector, allowing it to move flexibly around a patient	yes
5	Disior Bonelogic™	Disior Ltd., Helsinki, Finland	software developed to automate segmentation and 3D image analytics / anatomical modelling from medical imaging (CT, CBCT, MRI)	yes
6	Holoeyes MD™, Holoeyes XR™	Holoeyes Inc., Tokyo, Japan	software that enables 3D visualization of medical imaging data to support the planning and execution of medical procedures	-
7	HoloLens, HoloLens 2	Microsoft, Redmond, WA, U.S.A	direct mixed reality head mounted display	-
8	Horos™	Horos Project, Annapolis, MD, U.S.A	open-source DICOM viewer	-
9	Houdini™	SideFX, Toronto, ON, Canada	3D animation and visual effects software	-
10	InVesalius™	Centro de Pesquisas Renato Archer, Campanias (CTI), Brasil	open-source software designed for medical imaging and 3D reconstruction	-
11	iPlan CMF™	BrainLAB, Feldkirchen, Germany	specialized software solution designed for craniomaxillofacial surgery; used for preoperative planning and surgical simulation	yes
12	Lumi™	Augmedit B.V., Naarden, The Netherlands	software designed for use in preparing 3D models and visualizing medical images (CT, MRI), especially for pre-surgical planning and augmented reality visualization	yes
13	ITK-SNAP™	PICSL, University of Pennsylvania, Philadelphia, PA, U.S.A	open-source software application used for medical image processing, specifically for segmenting and visualizing anatomical structures in 3D	-
14	Magic Leap 2	Magic Leap Inc., Plantation, FL, U.S.A	direct mixed reality head mounted display	-
15	Maya™	Autodesk, San Francisco, CA, U.S.A	professional 3D computer graphics software widely used in industries such as film, television, video games, and visual effects for creating 3D models, animations, simulations, and rendering	-
16	Medical Imaging XR™	Medicalholodeck Inc., Zurich, Switzerland	software that utilizes extended reality technologies to enhance the visualization, analysis, and interpretation of medical imaging data	-
17	Medical Modeling and Design System	manufacturer not specified by the authors	N/A	-
18	MeshLab™	Visual Computing Lab, ISTI-CNR, Italy	open-source software tool designed for processing and editing 3D triangular meshes	-
19	MiDIVI™	Changzhou Jinse Medical Information Technology Co., Ltd., Changzhou, China	software platform designed for AI-Powered Surgical Planning System, XR Precision Surgical Navigation System; based on HoloLens 2 HMD	-
20	Mimics™	Materialise, Leuven, Belgium	software platform used for processing and analyzing medical imaging data to create accurate 3D models in healthcare; widely used in surgical planning, custom implant design, medical device development, and education	yes
21	MR Neuron	custom-made, China	N/A	-
22	Mixed Reality Toolkit™ (MRTK)	Microsoft, Redmond, WA, U.S.A	an open-source development framework designed for creating applications for mixed reality devices, such as HoloLens	-
23	O-arm	Medtronic, Minneapolis, MN, U.S.A.	advanced intraoperative imaging device designed to provide both 2D and 3D images during surgical procedures; designed to work with Medtronic’s StealthStation™ Navigation System	yes
24	OpenSight™	Novarad, Provo, UT, U.S.A.	augmented reality medical imaging software that uses the Microsoft HoloLens headset to overlay 2D, 3D, and 4D medical images (CT, MRI, PET) as holograms directly onto the patient	yes
25	ParaView™	Kitware Inc., Clifton Park, NY, U.S.A	open-source, multi-platform data analysis and visualization application widely used for handling and visualizing large-scale scientific datasets, particularly popular in fields like computational fluid dynamics, climate modeling, engineering simulations, and geospatial analysis	-
26	ProPlan CMF™	Materialise, Leuven, Belgium	virtual surgical planning software developed to assist surgeons in preoperative planning for cranio-maxillofacial procedures	yes
27	Slicer 3D™	Slicer Community, international/open source	open-source software platform designed for the visualization and analysis of medical images and the processing of 3D datasets; widely used in medical research, clinical studies, and educational applications	-
28	Spectator View™	Microsoft, Redmond, WA, U.S.A	software designed to enhance mixed reality experiences by allowing external observers to see and interact with holograms displayed through devices like HoloLens	-
29	StarAtlas™ Mixed Reality Holographic Imaging System	Visual MedTech Co., Ltd., Beijing, China (formerly Visual3d Medical Technology Development Co., Ltd., see line 30)	advanced medical imaging platform designed to provide precise, 3D visualizations of patient anatomy, enhancing the capabilities of healthcare professionals in surgical planning and surgical navigation	-
30	StealthStation™ Navigation System	Medtronic, Minneapolis, MN, U.S.A.	conventional surgical navigation system that integrates pre- or intraoperative imaging (CT, MRI, O-arm) with optical or electromagnetic tracking to provide real-time, image-guided localization of instruments and anatomy during neurosurgical and spinal procedures	yes
31	SurgicalAR™	Medivis, New York, NY, U.S.A.	advanced mixed reality platform designed for use in surgical planning and navigation; based on HMDs like HoloLens	yes
32	SYNAPSE VINCENT™	Fuji Film Medical Co Ltd, Tokyo, Japan	medical imaging and surgical planning software designed to assist healthcare professionals in visualizing, analyzing, and planning medical procedures using advanced 3D and 4D imaging technologies	yes
33	syngo.via Frontier™	Siemens Healthineers, Erlangen, Germany	advanced imaging software platform designed to enhance the efficiency and accuracy of medical image interpretation across multiple modalities; offers comprehensive tools for 2D, 3D, and 4D image reading and advanced visualization, integrating innovative, AI-powered applications to streamline diagnostic workflows	yes
34	Unity™	Unity Technologies, San Francisco, CA, U.S.A	cross-platform game engine and development platform used to create interactive experiences, including video games, simulations, extended reality, and other 3D and 2D applications	-
35	VECTRA H1 3D Imaging System™	Canfield Scientific Inc., Parsippany, NJ, U.S.A	handheld imaging device designed for capturing high-resolution, three-dimensional photographs of the human body. It is primarily used in aesthetic, dermatological, and reconstructive practices to analyze and document skin conditions, facial features, and body contours	yes
36	Visual Studio™	Microsoft, Redmond, WA, U.S.A	integrated development environment used by developers to create a wide range of applications, including web, desktop, mobile, cloud, and gaming applications	-
37	Visual3D™	Visual3d Medical Technology Development Co., Ltd, Beijing, China (currently under the name Visual MedTech Co., Ltd., see line 25)	advanced medical imaging platform designed to provide precise, 3D visualizations of patient anatomy, enhancing the capabilities of healthcare professionals in surgical planning and surgical navigation	-
38	Vitrea™	Vital Images, Minnetonka, MN, U.S.A	medical imaging software platform designed to provide advanced visualization, analysis, and workflow solutions for healthcare professionals; integrates data from various imaging modalities and enables 2D, 3D, and 4D image interpretation to support diagnostic and treatment planning processes	yes
39	XR90™	MediView XR Inc., Cleveland, OH, U.S.A	Advanced mixed reality visualization and navigation system developed to assist healthcare professionals during minimally invasive, needle-based procedures by providing a 3D view of the patient’s internal anatomy	yes
40	xvision Spine System™	Augmedics, Arlington Heights, IL, U.S.A	innovative mixed reality surgical guidance system designed to enhance the precision and efficiency of spine surgeries; used in surgical planning and navigation	yes
41	Ziostation 2™	Ziosoft, Tokyo, Japan	medical imaging software platform designed to provide healthcare professionals with sophisticated tools for 3D and 4D visualization, analysis, and quantification of medical images obtained from various imaging modalities	yes

Fig. 4.

Mixed reality HMDs: A- HoloLens 2 (Microsoft, Redmond, WA, U.S.A), B- Magic Leap 2 (Magic Leap Inc., Plantation, FL, U.S.A), C- Apple Vision Pro (Apple Inc., Cupertino, CA, U.S.A.), D- Meta Quest 3 (Meta Platforms Inc., Melno Park, CA, U.S.A.)

Software for mixed reality

The functions and applications of software solutions for HMDs used intraoperatively differ significantly. In general, any of the commercially available MR and VR HMDs is able to display a 3D object such as an STL file with the built-in software. Among the 41 studies in the present review, many different software solutions were utilized to display 3D reconstructions in the MR environment of an operating theatre, and include HoloLens built-in software, Ziostation^™, Mimics^™, Visual3D^™, StarAtlas^™ Mixed Reality Holographic Medical Image System, Holoeyes^™, Lumi^™, OpenSight^™. All but one worked with commercially available D-MR HMDs [14, 30, 34]. One study reported the use of Medical Imaging XR^™ software on Magic Leap 2 HMD [35]. A detailed list of types of software extracted from the analyzed studies can be found in Table 2. It should be noted that the publications often describe the details of the software used in an imprecise manner. However, before the intraoperative use of 3D reconstructions in MR, appropriate image processing must be conducted to produce 3D objects of relevant features for MR.

DICOM data and image processing before intraoperative use in XR

The studies utilized various input data and imaging modalities to generate 3D reconstructions for MR-IOS, with most relying on CT DICOM data as the primary input material [11, 14, 15, 17, 18, 20–28, 30–34, 36, 37, 39–41, 43, 44, 47–50, 58], followed by MRI [16, 17, 19, 26, 29, 33, 38, 42, 44–47, 50], while PET was used in two studies [19, 26] and 3D photographs in one [13]. The workflow, illustrated in Fig. 5A, involved windowing to optimize input data for surface-rendered models by enhancing relevant structures for rendering algorithms, ensuring well-defined and high-quality 3D visualizations. However, windowing alone does not enable the extraction of specific radiological regions of interest (ROIs), such as organs, which require segmentation to isolate these structures. Among the 41 studies reviewed, various techniques and software solutions were used to create 3D models for MR-IOS, including open-source programs such as 3D Slicer^™, Horos^™, MeshLab^™, and Blender^™ (Table 2). Once reconstructed, these models were utilized for preoperative planning in virtual reality (VR) and as intraoperative mixed reality (MR) aids during surgery [35].

Fig. 5.

Panel A: Workflow illustrating the steps from obtaining DICOM imaging data to their utilization in extended reality (VR and MR). Panel B: Surgeon’s view from the Magic Leap 2 HMD used in the MR-IR application during powered endoscopic dacryocystorhinostomy (X- real-time rendered anatomical 3D model obtained with Medical Imaging XR^™ software, Y- light-weight prearranged 3D model in GLB format)

Some of the MR-IOS described in the 41 analyzed papers utilize AI for automatic image segmentation, these include SurgicalAR^™ (Medivis, New York, NY, U.S.A.) [20], xvision Spine System^™ (Augmedics, Arlington Heights, IL, U.S.A) [14, 30, 34], Medical Imaging XR^™ (Medicalholodeck Inc., Zurich, Switzerland) [35], Mimics^™ (Materialise, Leuven, Belgium) [25, 31, 40, 41], Holoeyes^™ (Holoeyes Inc., Tokyo, Japan) [17, 18, 36], MiDIVI^™ (Changzhou Jinse Medical Information Technology Co., Ltd., Changzhou, China) [40, 47, 48], syngo.via Frontier^™ (Siemens Healthineers, Erlangen, Germany) [41].

Clinical application by discipline

The articles encompassed a wide range of surgical specialties, including spinal surgery [14, 30, 31, 34, 40, 47–49], neurosurgery [20, 24, 38, 39, 42, 44–46, 58], orthopedic surgery [33, 41], maxillofacial surgery [27, 28], otolaryngology [17, 18, 29], ophthalmic surgery [35], interventional radiology [11, 23, 37], thoracic surgery [25], visceral surgery [12, 15, 19, 21, 26, 32, 36], breast cancer surgery [16, 50], urology [43], and plastic and reconstructive surgery [13, 22]. It should be noted that this classification is not entirely precise, as many procedures could fall into more than one category. For instance, if a distinct category for surgical oncology were established, a significant number of studies could be grouped under it. The same issue applies to the differentiation between spinal surgery, neurosurgery, and orthopedic surgery.

Table 3 provides a summary of publications reporting the intraoperative application of mixed reality (MR).

Table 3.

Summary of publications presenting intraoperative use of mixed reality (MR) in humans

	Reference	Year	Area of MR application	Study design	Discipline	Type of MR intraoperative application	Type of image registration	HMD Hardware	Software (DICOM data processing and holographic visualization)	Number of patients
1	Sauer et al. [21]	2017	Liver surgery	Case series	Visceral surgery	Image reference	N/A	HoloLens	Ziostation 2™, Blender™, Unity™	Not specified
2	Pratt et al. [22]	2018	Lower extremity reconstruction surgeries	Case series	Plastic and reconstructive surgery	Image guidance	Registration performed but not specified	HoloLens	Vitrea™, ITK-SNAP™, MeshLab™, Unity™	6
3	Mascitelli et al. [38]	2018	Intracranial tumor and cerebrovascular lesion resection	Case series	Neurosurgery	Navigation	Registration performed but not specified	N/A	Brainlab Elements™	79
4	Witowski et al. [23]	2019	Percutaneous interventions on pulmonary arteries	Case series	Interventional radiology	Image reference	N/A	HoloLens	Slicer™, Blender™, CarnaLife Holo™	2
5	Zhu et al. [11]	2019	Inferior vena cava filter implantation	Case report	Interventional radiology	Navigation	Marker-based manual registration	HoloLens	Visual3D™	1
6	Li Y. et al. [39]	2019	External ventricular drain insertion	Case-control study	Neurosurgery	Navigation	Manual registration	HoloLens	Slicer™, ParaView™, Blender™	15
7	Zhang et al. [24]	2019	Intracranial tumor resection	Case series	Neurosurgery	Navigation	Marker-based automatic registration	HoloLens	StarAtlas™ Mixed Reality Holographic Medical Image System	3
8	Wei et al. [47]	2019	Percutaneous kyphoplasty	Prospective randomized controlled study	Spinal surgery	Navigation	Manual registration	HoloLens	Midivi™	20
9	Huber et al. [12]	2019	Robotic-assisted transanal total mesorectal excision	Case report	Visceral surgery	Image guidance	N/A	HoloLens	N/A	1
10	Saito et al. [36]	2019	Liver surgery	Case series	Visceral surgery	Image reference	N/A	HoloLens	SYNAPSE VINCENT™, Holoeyes XR™	2
11	Nuri et al. [13]	2020	Microotia surgery	Case report	Plastic and reconstructive surgery	Image guidance	Manual registration	HoloLens	VECTRA H1 3D Imaging System™, Blender™	1
12	Molina et al. [14]	2020	Pedicle screw placement	Case report	Spinal surgery	Navigation	Marker-based registration using O-arm	HMD being integral part of xvision Spine System	xvision Spine System™	1
13	Gu et al. [48]	2020	Lumbar pedicle screw implantation	Prospective, randomized controlled study	Spinal surgery	Navigation	N/A	HoloLens 2	Midivi™	25
14	Guo et al. [49]	2020	Intravertebral foramen puncture location	Prospective, randomized controlled study	Spinal surgery	Navigation	N/A	N/A	N/A	30
15	Fu et al. [66]	2020	Pulmonary nodules puncture for staining with methylene blue	Case series	Thoracic surgery	Navigation	Manual registration	HoloLens	Mimics™	16
16	Li G. et al. [43]	2020	Laparoscopic nephrectomy	Prospective nonrandomized controlled study	Urology	Image guidance	Registration performed but not specified	HoloLens	Visual3D™	50
17	Galati et al. [26]	2020	Resection of liver segments, metastatic neoplasm of adrenal gland, gastrectomies, pancreaticoduodenectomies	Case series	Visceral surgery	Image reference	N/A	HoloLens	InVesalius™, Unity™, Mixed Reality Toolkit™ (MRTK), Spectator View™	10
18	Aoki et al. [15]	2020	Percutaneous puncture for selective laparoscopic liver resection	Case report	Visceral surgery	Image guidance	N/A	HoloLens	Ziostation 2™, Holoeyes XR™	1
19	Gouveia et al. [16]	2021	Breast cancer surgery	Case report	Breast cancer surgery	Image guidance	Marker-based registration	HoloLens	Horos™	1
20	Tang et al. [27]	2021	Maxillary and mandibular tumors resection	Case series	Maxillofacial surgery	Navigation	Automatic registration	HoloLens	iPlan CMF™, ProPlan CMF™, Visual3D™	7
21	Ivanov et al. [28]	2021	Median neck and branchial cyst excision	Case series	Maxillofacial surgery	Image guidance	Marker-based and semiautomatic registration	HoloLens 2	Slicer™, Houdini™	3
22	Qi et al. [44]	2021	Intracranial lesions resection	Prospective nonrandomized controlled study	Neurosurgery	Navigation	Semiautomatic marker-based registration	HoloLens	Slicer™, Unity™, MR Neuron	37
23	Dennler et al. [41]	2021	Different orthopedic procedures	Prospective nonrandomized study	Orthopedic surgery	Image reference	N/A	HoloLens	syngo.via Frontier™, Mimics™, Unity™, Visual Studio™	25
24	Mitani et al. [17]	2021	Maxillary carcinoma resection	Case report	Otolaryngology	Image reference	N/A	HoloLens 2	Ziostation™, Holoeyes XR™	1
25	Yamazaki et al. [18]	2021	Temporal bone surgery	Case report	Otolaryngology	Image reference	N/A	HoloLens	Slicer™, MeshLab™, Holoeyes XR™	1
26	Scherl et al. [29]	2021	Parotid tumor surgery	Case series	Otolaryngology	Image guidance	Manual registration	HoloLens	Slicer™, Unity™	6
27	Yahanda et al. [30]	2021	Pedicle screws placement	Case series	Spinal surgery	Navigation	Marker-based registration using O-arm	HMD being integral part of xvision Spine System	xvision Spine System™	9
28	Liu X. et al. [40]	2021	Transforaminal percutaneous endoscopic lumbar disscectomy	Case-control study	Spinal surgery	Image guidance	Manual registration	HoloLens 2	Mimics™, Midivi™	44
29	Li J. et al. [31]	2021	Pedicle screw placement	Case series	Spinal surgery	Navigation	Marker-based registration	HoloLens	Mimics™	7
30	Kitagawa et al. [32]	2021	Laparoscopic cholecystectomy	Case series	Visceral surgery	Image reference	N/A	HoloLens 2	Holoeyes MD™	9
31	Ivan et al. [45]	2021	Intracranial tumor resection	Prospective nonrandomized controlled study	Neurosurgery	Image reference	Manual registration	HoloLens	OpenSight™	11
32	Peng et al. [58]	2022	Puncture treatment of hypertensive intracerebral hemorrhage	Case series	Neurosurgery	Navigation	Manual registration using “twin markers”	HoloLens	Medical Modeling and Design System, Maya™	8
33	Lu et al. [33]	2022	Various orthopedic cases	Case series	Orthopedic surgery	Navigation and image guidance	Marker-based registration and manual registration	HoloLens	Visual3D™	6
34	Liu A. et al. [34]	2022	Pedicle screw placement	Case series	Spinal surgery	Navigation	Marker-based registration using O-arm, C-arm	HMD being integral part of xvision Spine System	xvision Spine System™	28
35	Gadodia et al. [37]	2022	Percutaneous tumor ablation	Case series	Interventional radiology	Navigation	Marker-based registration	HoloLens	XR90™ (early model)	12
36	Tao et al. [50]	2023	Sentinel lymph node biopsy in breast cancer	Prospective randomized controlled study	Breast cancer surgery	Image guidance	Registration performed but not specified	HoloLens	N/A	150
37	Hayashi et al. [19]	2023	Robotic surgery for rectal cancer	Case report	Visceral surgery	Image reference	N/A	HoloLens 2	N/A	1
38	Berger et al. [20]	2023	Percutaneous rhizotomy of the trigeminal nerve	Case report	Neurosurgery	Navigation	Registration performed but not specified	HoloLens 2	SurgicalAR™	1
39	Van Gestel et al. [46]	2023	Brain tumor resection	Prospective nonrandomized controlled study	Neurosurgery	Image guidance	Semiautomatic	HoloLens 2	Brainlab Elements™, Slicer™	11
40	Nowak et al. [35]	2024	Endoscopic dacryocystorhinostomy in complex lacrimal obstructions	Case series	Ophthalmic surgery	Image reference	N/A	Magic Leap 2	Horos™, MeshLab™, Blender™, Medical Imaging XR™	2
41	Colombo et al. [42]	2024	Intracranial tumor and cerebrovascular lesion resection	Prospective nonrandomized study	Neurosurgery	Image guidance	Manual	HoloLens 2	Disior™, Lumi™	107

Abbreviations: MR – mixed reality, HMD – head-mounted display

Spinal surgery

In spinal surgery several authors evaluated mixed reality (MR) technology in pedicle screw placement. Gu et al. reported that safety of spinal surgery and implantation accuracy of pedicle screw fixation was significantly increased by MR technology in 25 patients. The authors compared mixed reality surgical navigation system (MR-SN)-assisted lumbar pedicle screw placement with traditional lumbar pedicle screw placement. MR-SN group had less bleeding, shorter operation time, higher success rate of first penetration by tap, and fewer times using C-arm fluoroscopy. The implantation accuracy in MR group was higher and the postoperative recovery rate of low back pain was faster. The safety of spinal surgery and implantation accuracy of pedicle screw fixation was significantly increased by MR technology [48]. Similarly, Liu Ann et al. reported the accuracy of 205 pedicle screws consecutively placed in 28 patients using MR-SN with an accuracy of 98% [34]. Yahanda et al. performed percutaneous pedicle screw placement guided by a MR-SN in 9 patients [30]. The accuracy for these screws was 100% overall; all screws were Gertzbein-Robbins grade A or B (96.8% grade A, 3.2% grade B) [30]. Molina et al. reported a case of six pedicle screws inserted with a support of an approved MR-SN (xvision Spine System™, Augmedics, Arlington Heights, IL, U.S.A) [14]. Intraoperative computed tomography was used for navigation registration as well as implant accuracy and precision assessment. Clinical accuracy (per the GS grading scale) was 100% [14]. Li et al. investigated application of MR to lumbar fracture treatment. Posterior vertebrectomy has been operated on in seven patients. A MR-SN was used to assist the implantation of 57 pedicle screws. No screw was found outside planned location, and it was not necessary for the X-ray to provide extra locative information during the surgery. The application of MR-SN was feasible, safe, and accurate while the lumbar fracture surgery was processing, providing satisfactory assistance for spine surgeons [31].

The feasibility and clinical application value of MR-SN in guiding intervertebral foramen microscopic puncture was analyzed by Guo et al. The publication proves that MR can accurately guide the establishment of intervertebral foramen microscopic cannula, promote puncture success rate, reduce repeated puncture times, avoid by-injury, shorten puncture time, and reduce X-ray radiation quantity of operators and patients [49].

Liu Xiaoyang et al. evaluated a mixed reality image guidance system (MR-IG) during transforaminal percutaneous endoscopic discectomy (TPED). MR technology was used to navigate procedures of marking, needle insertion, foraminoplasty, and positioning of the working sheath. The operation time and radiation exposure significantly decreased in the MR-assisted TPED group compared to those in the conventional TPED group. Unfortunately, the incidence of eye fatigue increased owing to the use of head mounted displays (HMDs) in the MR-assisted TPED group. However, the authors noted that the assistance of conventional fluoroscopy was still required [40].

Wei et al. assessed the clinical outcome of percutaneous kyphoplasty (PKP) assisted with MR-SN in treatment of osteoporotic vertebral compression fracture (OVCF) with intravertebral vacuum cleft (IVC). Forty cases of OVCF with IVC undergoing PKP were randomized into a MR technology-assisted group and a traditional C-arm fluoroscopy group. The authors proved that MR-SN can accurately orientate the position of IVC area, leading to more satisfied vertebral height improvement, cement diffusion, and pain relief [47].

Neurosurgery

During neurosurgery, transforming two-dimensional (2D) sectional images into three-dimensional (3D) structures is cognitively demanding due to the complexity of intracranial anatomy, increasing the difficulty and risk of procedures. Mixed reality surgical navigation systems have been evaluated for their potential to enhance visualization and reduce these challenges.

Li Ye et al. demonstrated the feasibility and accuracy of using MR-SN for external ventricular drain (EVD) insertion [39]. Preoperative CT-generated holograms guided the neurosurgeon in aligning the catheter with the planned trajectory. Compared to the traditional freehand method, hologram-guided EVD placement required fewer passes (1.07 ± 0.258 vs. 2.33 ± 0.98, p < .01) without adverse events. The mean additional preparation time was 40.20 ± 10.74 min [39].

Peng et al. evaluated the feasibility of MR-SN in minimally invasive puncture and drainage of hematomas in patients with hypertensive intracerebral hemorrhage [58]. Data collected included hematoma evacuation rate, operation time, drainage tube target deviation, and postoperative complications. No cases of delayed bleeding, acute ischemic stroke, intracranial infection, or epilepsy were observed during the follow-up period [58].

Qi et al. evaluated the feasibility of MR-SN in 37 intracranial lesion surgeries. Multimodal imaging-based holograms of lesions, markers, and surrounding structures were imported into a head-mounted display (HMD) and aligned using point-based registration [44]. The holograms were projected onto the patient’s head, and their contours were compared with standard neuronavigation. The median deviation between MR-SN and standard neuronavigation system was 4.1 mm (IQR 3.0–4.7 mm), with 81.1% of lesions showing high consistency (< 5.0 mm deviation). The mean additional preparation time was 36.3 ± 6.3 min, including a registration time of 2.6 ± 0.9 min, which decreased as neurosurgeons gained experience. The study confirmed the technical feasibility, accuracy, and clinical applicability of MR-SN [44].

Zhang et al. described experience from 3 intracranial tumor surgeries supported by a MR-SN [24]. The results showed that the MR imaging is in a stable state in the operating room with no significant flutter and blur. And the neurosurgeon’s feedback on the comfort of the equipment and the practicality of the technology was satisfactory [24]. Berger et al. documented a case of successful MR-assisted percutaneous glycerol rhizotomy for trigeminal neuralgia performed on 50-year-old female [20]. MR-SN allowed rapid and correct placement of a spinal needle through the foramen ovale. The patient was discharged home after a few hours of observation with no complications and reported pain relief. It is important to highlight that the authors utilized an FDA-approved MR-SN (SurgicalAR™, Medivis, New York, NY, U.S.A.) [20].

Colombo et al. reported a single-center experience of 107 consecutive holograms (oncologic, cerebrovascular, carotid cases), finding that MR did not prolong surgical preparation (48.0 ± 17.3 min vs. 52 ± 17 min in a matched non-MR cohort) and prompted approach modification in 3 cases; the authors highlight usability and training benefits in cranial workflows [42]. Mascitelli et al. described a navigation-linked heads-up display for intracranial surgery, demonstrating feasibility of microscope-linked overlays that reduce attention shifts to side screens (an integration pattern akin to MR-SN) [38]. Van Gestel et al. validated an AR/MR planning pipeline on HoloLens 2 for neuro-oncology, reporting faster and more intuitive tumor-resection planning than conventional neuronavigation; the authors emphasize care in deep-seated tumors [46]. Ivan et al. prospectively studied MR HMD-based incision intraoperative planning in elective craniotomies, demonstrating feasibility of mapping tumor borders and correspondence with microscope-based navigation [45].

Orthopedic surgery

Dennler et al. investigated the feasibility of MR-IR during various orthopedic procedures in 25 patients [41]. Surgeons were generally satisfied with image quality and accuracy of the virtual objects. Wearing the MR HMD was rated as comfortable. Functionality of voice commands and gestures provided less favorable results. The greatest potential in the use of the MR device was found for surgical correction of deformities. While surgeons where generally satisfied with image quality of the MRIR, some technical and ergonomic shortcomings were pointed out [41].

Lu et al. evaluated MR-SN and MR-IG in 6 different orthopedic surgeries [33]. The authors conclude that MR systems enhance orthopedic surgery by providing 3D visualizations of bone fractures for better understanding and treatment planning, facilitating intuitive communication among doctors and patients. During surgery, dynamic holograms blend with virtual 3D models to offer surgeons superior visual guidance. MR-based navigation improves the precision and safety of screw placement, while 5G-enabled MR cloud platforms support telesurgery collaboration. Overall, MR technology reduces reliance on surgical experience and enables personalized 3D models for accurate diagnosis and treatment of orthopedic conditions [33].

Maxillofacial surgery

Ivanov et al. investigated MR-IG approaches in two median neck and one branchial cyst excisions [28]. The main challenge was to develop a scalable mixed reality approach that minimized preparation time and avoided patient-specific custom solutions. The authors employed a novel method of hologram positioning using mixed reality markers, identifying both benefits and challenges of MR surgery. A key issue was the competition between operating room lighting and hologram brightness, making simultaneous use problematic. Additionally, “focal rivalry” complicated direct use of MR during surgical manipulations, highlighting the need for further technological advancements [28].

Tang et al. evaluated the feasibility and accuracy of MR-SN in seven cases of oral and maxillofacial tumor resection [27]. Virtual surgical planning was based on preoperative CT datasets. Mean deviation between the planned osteotomy plane and the actual osteotomy plane was 1.68 ± 0.92 mm; the maximum deviation was 3.46 mm. Chromatographic analysis showed error of ≤ 3 mm for 80.16% of the points. Mean deviations of maxillary and mandibular osteotomy lines were approximate (1.60 ± 0.93 mm vs. 1.86 ± 0.93 mm). While five patients had benign tumors, two had malignant tumors. Mean deviations of osteotomy lines were comparable between patients with benign and malignant tumors (1.48 ± 0.74 mm vs. 2.18 ± 0.77 mm). Intraoperative frozen pathology confirmed negative resection margins in all cases. No tumor recurrence or complications occurred during mean follow-up. The authors concluded that the technology is feasible, safe, and effective for tumor resection in the oral and maxillofacial region [27].

Otolaryngology

Mitani et al. demonstrated the successful use of MRIR for tumor resection in otolaryngology, highlighting its potential for preoperative planning, intraoperative image reference in complex cases, and surgical education [17].

Scherl et al. found a MR-IG feasible in parotid gland surgery. Gestures were recognized correctly [29]. Mean accuracy of superimposition of the holographic model and patient’s anatomy was 1.3 cm. The intraoperative workflow remained unaffected, while providing additional information. Reality-fused 3D holograms enhanced surgical safety and ergonomics without compromising sterility [29].

Yamazaki et al. explored the use of patient-specific VR and MR temporal bone models of a patient with cholesteatoma in anatomical teaching, pre-operative surgical planning and intraoperative surgical referencing [18]. Most participants were favorable about the VR model and considered HMD as superior to a flat computer screen. 91% of the participants agreed or somewhat agreed that VR through HMD is cost effective. In addition, the VR pathological model was used for planning and sharing the surgical approach during a preoperative surgical conference. The MRIR used intraoperatively helped to clarify the relationship between the pathological lesion and vital anatomical structures [18].

Ophthalmic surgery

Lacrimal surgery is a part of ophthalmic surgery, and lacrimal duct obstruction is an indication for surgical treatment. Nowak et al. reported the techniques and outcomes of virtual reality planning and MR-assisted powered endoscopic dacryocystorhinostomy in extremely complex lacrimal drainage obstructions [35]. Their series were patients with traumatic secondary acquired lacrimal duct obstruction (SALDO) in the setting of Le Fort fractures and one was a patient with Aperts syndrome with a complex syndromic congenital nasolacrimal duct obstruction. All the patients demonstrated gross malposition of the lacrimal sac with gross lacrimal and nasal anatomical deformities. Surgery was supported by a MR-IR with the use of two modes of anatomical reconstructions: prearranged lightweight 3D models in the GLB format and real-time-rendered digital models. In contrast to many other authors, Nowak et al. utilized GLB models, as this format is not only small-sized but also retains color information, unlike the STL format. The VR models helped the surgeons to assess the details of the altered anatomy and to preoperatively plan an appropriate approach. Intraoperatively, MR models were present in the surgeon’s view without disturbing the endoscopic procedure. Intermittently, the surgeon could pull any of the models virtually present in the operating room, slice them, rotate them, and intricately study the alterations in a stepwise manner, as the surgery proceeded (Fig. 5B) [35].

The authors presented a complete workflow for preoperative manual segmentation to create small-sized 3D models that require minimal computing power for intraoperative use (Fig. 6). Because of the extreme complexity of cases, they also aided the surgery with 3D models rendered in real-time from CT-Dacryocystography (CT-DCG) data obtained with the Medical Imaging XR™ software. However, this path required much computing power. The experimental setup of the MR-IR operating theatre that allowed remote online consultation is shown in Fig. 7 [35].

Fig. 6.

Workflow for preoperative VR planning and MR supported surgery (black route- production of small-sized GLB 3D model for easy intraoperative use in MR-IR, blue route- real-time volumetric rendering for surgical planning and intraoperative use in MR-IR, red route- production of high-quality 3D reconstructions for medical documentation)

Fig. 7.

MR operation theatre setup for endoscopic dacryocystorhinostomy in complex cases of lacrimal drainage obstruction

Interventional cardiology

Witowski et al. demonstrated feasibility of MR-IR in transcatheter pulmonary interventions: balloon pulmonary angioplasty for treatment of chronic thromboembolic pulmonary hypertension and stent implantation for pulmonary artery stenosis [23]. Personalized life-sized models of the same structures were additionally 3D-printed for preoperative planning. The operative team was able to manipulate the hologram and multiple users of the MR system could share the same image in real time. Clinicians expressed their satisfaction with the quality of imaging and potential clinical benefits. The utilized MR-IR is FDA-approved (CarnaLife Holo™, MedApp, Krakow, Poland) [23].

Zhu at al. presented a case report that describes the first successful use of a MR-SN for the implantation of an inferior vena cava filter (IVCF) in a patient with nephrotic syndrome and recurrent pulmonary embolism [11]. Traditional IVCF implantation under X-ray guidance with contrast agents poses a significant risk of contrast-induced nephropathy (CIN) due to the patient’s renal insufficiency. The MR technology provided a novel, radiation-free approach that required minimal contrast agent during preoperative imaging, significantly reducing the risk of CIN and associated complications. The operation was completed successfully, with no complications or adverse effects reported [11].

Gadodia et al. demonstrated the clinical feasibility and potential benefits of using an MR-SN for percutaneous tumor ablation, highlighting its capability for accurate intraprocedural registration and electromagnetic tracking to aid in guidance and navigation [37]. Although technical issues initially posed a challenge, these have been addressed through increased familiarity with the system and technological advancements. The authors used an initial version of an FDA-approved MR-SN in their study (XR90™, MediView XR Inc., Cleveland, OH, U.S.A) [37].

Thoracic surgery

Localizing non-palpable pulmonary nodules presents a challenge for thoracic surgeons. Fu et al. evaluated the accuracy of combining 3D printing technology with MR in MR-SN for localizing ground glass opacity-dominant pulmonary nodules [25]. Among 16 patients with 17 nodules, 16 were successfully localized (94.1%), with negative pathological resection margins. The study demonstrated that MR is a feasible approach for localizing pulmonary nodules [25].

Breast cancer surgery

Gouveia et al. evaluated MRIG in breast cancer conservative surgery [16]. During the procedure, the MR localization method was compared with the standard pre-operative localization with carbon tattooing (institutional protocol). A successful overlap of the previous standard preoperative carbon tattooing marks and MR tumor visualization within the patient’s breast was successfully obtained [16].

Tao et al. explored the value of MR-IG in sentinel lymph node biopsy (SLNB) in patients with breast cancer [50]. A total of 300 patients with breast cancer who underwent SLNB were enrolled and randomly divided into two groups. In group A, only dye (an injection of methylene blue) was used to detect sentinel lymph nodes, while in group B MR was used for positioning in addition to the dye. Before the surgery, a 1:1 3D reconstruction model based on the patient’s CT or MRI original data was built. Next, the patient was injected with dye, and the authors completed MR localization by overlapping the pre-marked image with the model. The application of MR to SLNB significantly reduced the detection time and the occurrence of complications and improved patient satisfaction [50].

Visceral surgery

Aoki et al. reported a case that introduces a novel holography-guided percutaneous puncture technique for selective near-infrared fluorescence-guided laparoscopic anatomical liver resection using a MR-IG for a suspected malignancy [15]. The holographic display allowed precise visualization and real-time guidance for targeting the Glissonean pedicle, enabling accurate puncture and injection of indocyanine green for fluorescence-guided resection. The combination of MR visualization and near-infrared fluorescence provided enhanced anatomical clarity, enabling precise surgical demarcation and minimally invasive resection [15].

Galati et al. investigated the use of mixed reality technology to enhance surgical procedures in ten open abdomen surgeries [26]. The MR-IR setup allowed multitasking and efficiency. However, despite benefits in surgical performance and education, challenges included physical discomfort and battery limitations of the HMD used [26].

Hayashi et al. successfully performed robotic surgery for rectal cancer with MR-IR [19]. Huber et al. also describe adopting MR-IG for robotic-assisted transanal total mesorectal excision [12]. All participants handled the device intuitively and reported a high level of comfort during the surgery. The task load was easily manageable although the surgeon and assistant both noted a short delay in the HMD wireless connectivity [12]. Kitagawa et al. proved safety and efficacy of MR-IR utilization in laparoscopic cholecystectomy. The authors noted that efficacy of laparoscopic MR-supported may depend on the surgeon’s experience, as indicated by the different ratings provided by the surgeons [32].

Saito et al. investigated the potential of MR-IR in liver surgery. The intraoperative MR utilization contributed to better imagination of tumor locations, and for determining the parenchymal dissection line in the hepatectomy for patients with more than 20 multiple colorectal liver metastases [36]. In another case, the hologram enabled a safe Gliisonean pedicle approach for hepato-cellular carcinoma with a hilar anatomical anomaly. This initial experience suggested that the MR-IR contributed to ‘‘last-minute simulation,’’ but not for ‘‘navigation.’’ The authors concluded that intraoperative hologram might be a new next- generation operation-supportive tool in terms of spatial awareness, sharing, and simplicity [36].

Sauer et al. evaluated the application MR-IR in complex open hepatic surgery [21]. A suitable workflow for segmenting image masks and texture mapping of tumors, hepatic artery, portal vein, and the hepatic veins were developed. The 3D model was positioned above the surgical site. Anatomical reassurance was possible simply by looking it up. The positioning in the room was stable without drifting and minimal jittering. Users reported satisfactory comfort wearing the HMD without significant impairment of movement. The authors concluded that MR-IR has significant potential to enhance surgeons’ actions and perception in open visceral surgery by displaying 3D anatomical models near the surgical site. However, accurately superimposing anatomical structures onto the organs remains challenging due to deformations caused by manipulation, respiratory motion, and interactions with surgical instruments during the procedure [21].

Urology

Li Guan et al. evaluated the clinical value of MR-IG in laparoscopic nephrectomy in 100 patients divided into MR-assisted (MRALN, n = 50) and non-MR-assisted (non-MRALN, n = 50) groups [43]. The MRALN group showed higher Likert scores for operative planning, intraoperative navigation, remote consultation, teaching, and doctor-patient communication. Laparoscopic nephrectomy was successfully implemented in significantly more MRALN patients (82% vs. 46%, P < .001), with shorter operative and warm ischemia times and less blood loss (all P < .001) [43].

Plastic and reconstructive surgery

Pratt et al. demonstrated that MR-IG facilitates accurate identification, dissection, and execution of vascular pedunculated flaps in reconstructive surgery [22]. Preoperative CT-angiography scans were used to delineate osseous, vascular, skin, soft tissue structures, and vascular perforators, and in generating 3D images with two segmentation software packages. Intraoperatively, models were manually registered by the surgeon using tracked hand gestures and voice commands, with MR aiding navigation and precise dissection. MR overlay identified the subsurface location of vascular perforators, which was validated against audible Doppler ultrasound. The surgeon achieved precise and efficient localization of perforating vessels with MR-IG [22].

Nuri et al. reported the successful use of MR-IG in ear reconstruction for a 10-year-old patient with right microtia [13]. This is the only publication to date that demonstrates the use of 3D photography as a source for creating anatomical models used in MR-IG. Preoperative 3D photographs of the unaffected ear were captured using the VECTRA H1 3D Imaging System, horizontally inverted, and superimposed on the affected side based on facial anatomical landmarks. These images were projected onto the patient in the operating room via an HMD, guiding the design and positioning of the auricle. The accuracy of the MR technique was validated by comparing it to the conventional transparent film method, with a deviation of less than 2 mm between the two approaches [13].

Discussion

Much of the discussion and insights are already presented after the results in individual subspecialty sections. What remains are in-depth analysis of the limitations of mixed reality, the increasing integration of artificial intelligence in MR and potential future directions of the technology in surgical fields.

Limitations of mixed reality

Mixed Reality Intraoperative Systems (MR-IOS) face several limitations that affect their widespread adoption. The use of head-mounted displays (HMDs) can lead to increased eye fatigue and other physical discomfort like cybersickness [26, 40]. Technical and ergonomic challenges include less reliable voice commands and gestures, spatial resolution and luminescence issues, narrow field of view, image latency, rendering artifacts, and battery limitations [5, 26, 41]. Competition between operating room lighting and hologram brightness, as well as “focal rivalry,” complicates simultaneous use during surgery [28]. Accurate superimposition of anatomical structures remains difficult due to organ deformations caused by manipulation, respiratory motion, and interactions with surgical instruments [21].

Some authors report a holographic (spatial) drift phenomenon. In optical see-through HMDs that rely on inside-out tracking, the spatial registration of the hologram relative to the patient can slowly drift due to cumulative SLAM error, lighting or occlusion changes, or operating-room reconfiguration. Clinical pilot work observed ≈ 1–2 mm drift during user movement around the patient. Algorithmic drift-compensation and re-anchoring methods (e.g., vision-based relocalization to fiducials/markers or environmental features) have been proposed and validated on phantoms and early clinical tasks, improving stability toward clinically acceptable thresholds. Practical mitigations include robust fiducial geometry, periodic re-registration checkpoints, tracker-to-HMD fusion (integrated MR-SN), and minimizing major scene changes after registration [54, 59, 60].

Additional drawbacks include short wireless connectivity delays, a learning curve for users, extra time and costs for preoperative planning and intraoperative setup, and the need for updates to operating room software and equipment [12, 35]. Despite these challenges, the setup is portable, less costly than conventional surgical navigation systems, and can also be used remotely for preoperative surgical planning. Preoperative 3D reconstruction typically takes 2–3 h, and the additional intraoperative time required ranges from 20 to 30 min [35].

Most of the examined MR-IOS were experimental, relying on commercially available head-mounted displays HMDs, and only a few had FDA-approval as integrated systems [14, 20, 30, 34, 37]. Furthermore, only a few HMDs based on direct mixed reality (D-MR) technology are available on the market, and only these are suitable for operating room environments due to technical limitations of the indirect mixed reality (ID-MR) systems. Another significant issue is the lack of unified terminology, as the distinction between augmented reality and mixed reality is less clear compared to the well-defined concept of virtual reality, complicating the analysis of scientific literature in this field. In fact, the distinction between augmented reality (AR) and mixed reality (MR) may be unnecessary, as merging these concepts could streamline future analyses and foster broader adoption of extended reality (XR) technologies in healthcare.

AI utilization

Artificial intelligence (AI) plays a transformative role in medical image segmentation by automating and enhancing the accuracy of this traditionally time-intensive process [61–65]. Advanced techniques, such as convolutional neural networks (CNNs) and deep learning models, enable precise delineation of anatomical structures in medical images, facilitating the creation of 3D models for diagnostic and therapeutic purposes [66, 67]. AI-driven segmentation reduces manual effort, accelerates workflow, and minimizes variability among practitioners. Open-source AI frameworks and domain-specific tools are further accelerating innovation in 3D medical imaging, supporting application of XR in surgical planning and intraoperative use [6]. By improving efficiency and accuracy, AI is revolutionizing how medical images are analyzed and utilized in clinical and research settings.

Summary and future directions

Mixed reality (MR) is a rapidly evolving technology with significant potential across various surgical disciplines. It offers a compact, cost-effective alternative to traditional navigation systems, requiring minimal infrastructure while providing enhanced visualization and guidance. This review highlights its diverse applications in intraoperative settings, including navigation, guidance, and reference, demonstrating its capability to improve surgical precision and outcomes. However, it is important to note that, to date, relatively few cases of intraoperative MR use have been published, underscoring the need for further research and validation.

Despite its promise, MR faces challenges such as ergonomic limitations, technical constraints, and the need for broader adoption and optimization. The development of artificial intelligence (AI) is likely to accelerate the wider adoption of MR in intraoperative practice by enhancing automation, segmentation accuracy, and real-time data integration. Continued advancements in hardware, software, and integration with AI are essential to overcome these limitations and unlock MR’s full potential in intraoperative practice. As MR technology matures, it has the potential to revolutionize surgical procedures, improving both efficiency and patient outcomes.

Author contributions

R.N. and M.J.A. conceptualized the study, R.N., M.N., M.R., and M.J.A. collected the data. All authors drafted and reviewed the manuscript.

Funding

Hyderabad Eye Research Foundation and SERB, India (MJA).

Data availability

This is a review article and does not have any datasets.

Declarations

Ethical approval

Not applicable.

Human ethics and consent to participate

Not applicable.

Consent to publish

This being a review article, does not have any patients or animals subjects. The authors of this paper provide their necessary consents to publish.

Competing interests

The authors declare no competing interests.