Mixed Reality Interfaces for Achieving Desired Views with Robotic X-ray Systems

Abstract

Robotic X-ray C-arm imaging systems can precisely achieve any position and orientation relative to the patient. Informing the system, however, what pose exactly corresponds to a desired view is challenging. Currently these systems are operated by the surgeon using joysticks, but this interaction paradigm is not necessarily effective because users may be unable to efficiently actuate more than a single axis of the system simultaneously. Moreover, novel robotic imaging systems, such as the Brainlab Loop-X, allow for independent source and detector movements, adding even more complexity. To address this challenge, we consider complementary interfaces for the surgeon to command robotic X-ray systems effectively. Specifically, we consider three interaction paradigms: (1) the use of a pointer to specify the principal ray of the desired view relative to the anatomy, (2) the same pointer, but combined with a mixed reality environment to synchronously render digitally reconstructed radiographs from the tool’s pose, and (3) the same mixed reality environment but with a virtual X-ray source instead of the pointer. Initial human-in-the-loop evaluation with an attending trauma surgeon indicates that mixed reality interfaces for robotic X-ray system control are promising and may contribute to substantially reducing the number of X-ray images acquired solely during “fluoro hunting” for the desired view or standard plane.

1. Introduction

Fully robotic X-ray systems can precisely orient and reposition themselves to align with any viewing direction relative to the patient. However, this does not guarantee that the desired view onto anatomy is indeed achieved easily or quickly, because of challenges in effectively commanding the system. In an interventional radiology (IR) suite, where floor- or ceiling-mounted robotic C-arm systems are prevalent, the interventionalist manipulates the C-arm pose through joysticks, which can be ineffective because adjusting multiple axes in tandem results in complex movements but moving along each axis independently is time consuming. In other scenarios, including those that rely on non- or partially-robotic X-ray imaging systems, the provider interfaces with the X-ray system indirectly through an operator, giving vocal commands such as “more AP” or “roll that way.” The introduction of mobile robotic X-ray systems brings further challenges. Despite being fully robotized, for example, the Brainlab Loop-X requires using a non-sterile control panel to make manual adjustments to the viewing angle (Keil and Trapp 2022). Additionally, the Loop-X is capable of actuating its source and detector independently, allowing for more sophisticated configurations which potentially complicate communication between the surgeon and the operator. Here, we consider alternative interfaces for the surgeon to control robotic X-ray devices more directly, using an optical see-through head-mounted display (OST HMD) that delivers a mixed reality (MR) environment that is spatially calibrated to the X-ray space. This then allows for interactive and on-demand rendering of digitally reconstructed radiographs (DRRs) as a live preview of candidate viewpoints before the X-ray imaging system is moved. In this initial investigation on the usefulness of such approach, we consider the use of a pointer tool to specify the principal ray of the viewing frustum or a virtual “AR Handle” representing the X-ray source, rendered in mixed reality along with the DRR.

Improving the surgeon-C-arm interface has the potential to reduce the number of X-ray acquisitions needed. During surgery, it is common practice to acquire multiple images for the purpose of navigating to a final desired image, a trial-and-error process referred to as fluoro-hunting. In addition to being time-consuming, fluoro-hunting contributes to the radiation dose for patients and clinicians. In a cadaveric study with a non-robotic C-arm, for example, Mandelka et al. (2022) find that an average of 7.1 acquisitions were needed to obtain AP and lateral views of vertebral bodies, exposing the examiners to a median dose of 34.5 μ Gy cm² per level. For the pelvis, De Silva et al. (2018) find that an average 6.4 ± 4.8 acquisitions were needed to obtain the desired view, measured across five radiologists for six views pertinent to pelvic screw placement. Reducing these non-clinical acquisitions is pertinent as per the as-low-as-reasonably-achievable standard (Hansson 2013).

2. Related Work

Mixed Reality (MR) is an emerging technology that integrates computer-generated content with real world objects. Introducing this technology into medical settings has equipped surgeons with a new set of capabilities that promote improved workflow and outcomes of surgical procedures (Fida et al. 2018; Jud et al. 2020; Elmi-Terander et al. 2019). In this context, the introduction of AR Head-Mounted Displays (HMDs) into the surgical suites enables visual guidance and navigation capabilities, facilitates communication, and allows for the visualization of multiple imaging modalities in situ, promoting the understanding of complex anatomical structures using bi-dimension and three-dimensional images (Rahman et al. 2020). These devices have proven particularly valuable in assisting surgeons during the performance of orthopedic procedures (Fotouhi et al. 2020; Deib et al. 2018; Casari et al. 2021; Teatini et al. 2021; Gu et al. 2022). Furthermore, the wide variety of sensors integrated into AR HMDs has enabled the tracking and localization of surgical tools and markers commonly used in surgical settings (Kunz et al. 2020; Gsaxner et al. 2021).

Recent work has explored the use of AR specifically to aid in the reduction of fluoro-hunting. Unberath et al. (2018) use an HMD to show a virtual indicator of the target pose for a manually operated C-arm, assisting the technician, while Andress et al. (2018) demonstrate the utility of an HMD calibrated with a C-arm system for triangulating and visualizing anatomical structures in 3D. Likewise, the utility of DRRs for obtaining standard views for pelvic screw placement has been explored by De Silva et al. (2018), who simulate fluoroscopic images from the current pose of a calibrated C-arm, manipulated by a radiologist. By contrast, we simulate the planned pose of a robotic X-ray device as one aspect of a surgeon-in-the-loop control interface. A marker-free calibration of the C-arm with the patient and the surgeon’s HMD is achieved in Hajek et al. (2018), with applications in spatially aware visualization of fluoroscopic data (Fotouhi et al. 2019). Finally, Gong et al. (2013) propose a human interface to guide DRR generation in order to initialize a 2D/3D registration, relying on (1) a tracked pointer or (2) Microsoft Kinect hand gestures to manipulate patient model pose. We use a similar control scheme, namely (1) a tracked pointer and (2) HoloLens hand gestures, but as an interface with the next robotic pose—rather than an already acquired image—anticipating and adjusting the planned X-ray projection in MR.

3. Methods

Fig. 1 shows a high-level overview of the proposed interfaces for controlling a robotic X-ray system, which we refer to as “Tool + Preview” and “AR Handle + Preview.” In the Tool + Preview interface, the surgeon, positions a pointer tool in line with the principle ray of the desired view. As the surgeon positions the tool in space, they observe a DRR in real time on the HMD. The placement of the DRR is fixed in the surgeon’s field of view, on the side opposite their dominant hand. This ensures that the surgeon can observe the pointer tool and the live preview simultaneously, i.e. without turning their head to look at a physical display or virtual window. When the surgeon is satisfied with the virtual view they have achieved, they note its position for the robotic X-ray system.

Figure 1.

Overview of the proposed interfaces to control robotic gantries. Both solutions use a live preview of the currently specified view, approximating how the real X-ray based on this specification will appear. “Tool + Preview” (left) utilizes the pointer tool to specify this view, whereas “AR Handle + Preview” (right) uses a virtual handle that the surgeon manipulates with hand gestures.

There are several drawbacks to the Tool + Preview approach that the proposed AR Handle, shown in Fig. 3, is equipped to solve. The first is reliance on optical tracking of the pointer tool, which can be obscured by the detector, as well as the need to physically place the tool near the patient. In certain views, the placement of the X-ray arm may obstruct tool placement, requiring repositioning. More importantly, once the surgeon withdraws the tool, there is no persistent visualization of the pose they just specified, which may be useful for further iteration. The AR Handle addresses this by remaining fixed in space, as a virtual object, until manipulated with tracked hand gestures. Since virtual objects can be manipulated from afar, the surgeon can position this tool without extending their arm over the patient for long periods. As in the first interface, once the surgeon is satisfied with the DRR they have acquired, they begin robotic motion.

Figure 3.

An overview of the “AR Handle + Preview” interface. In this interaction, the surgeon manipulates a virtual handle from afar using a pinch and grab interaction, specifying the principle ray of the desired view.

3.1. Experimental Setup

We test each interface by evaluating the ability of an expert user to obtain standard clinical views onto a pelvis phantom. The phantom is fixed in the head-first, supine position and obscured from the surgeon’s view. The robotic X-ray system consists of a Brainlab Loop-X in conjunction with the BrainLab Curve navigation platform. We are interested in the number of real X-ray shots needed to obtain the desired view, namely the anteroposterior (AP), inlet, outlet, (right) obturator oblique, (right) iliac oblique, and teardrop views. After an initial shot using the interface in question, the surgeon evaluated each image and, if needed, used joystick adjustments to obtain the final view. Secondary shots were achieved in this manner for efficiency’s sake, since automated movements first require the Loop-X to return to an upright position before converging on a specified view, rather than moving along the shortest path. Iterative adjustments are more practical in the clinical setting if the first shot is reasonably close. Naturally, a better initial shot will result in fewer adjustments.

To provide a live preview of the given viewpoint, we use a virtual patient model to constantly render digitally reconstructed radiographs (DRRs). Fig. 2 provides an overview of the experimental setup for the Tool + Preview interface. For this study, the patient model consists of a CT scan of the pelvis phantom, and we rely on optical tracking to obtain the pose of the pelvis T_H,P and tool T_H,T with respect to the HMD frame H. The anatomical pose could likewise be obtained via automatic 2D-3D registration, as in De Silva et al. (2018), and tool tracking is unnecessary when using a fully virtual gantry interface, such as our AR Handle. In our case, the pelvis frame P is registered to the virtual patient model by including the fiducial markers in the CT field of view, thus establishing a point correspondence for T_P,Mo. For the Tool + Preview interface, this provides the pose of the pointer tool T_M,T with respect to the patient model

$$T_{\text{M,H}}=T_{\text{P,M}}^{-1}{T}_{\text{H,P}}^{-1}.$$ (1)

Following two pivot calibrations, we obtain the tool tip a_T and back b_T in frame T, which are used to specify the projection matrix of a DRR along the principle ray given by the pointer tool:

$$P=K[R|t]=K\left[Ro{t}_{\theta }({\widehat{\mathbf{r}}}_{\text{M}}\times \widehat{\mathbf{z}})|{\mathbf{a}}_{\text{M}}-d{\widehat{\mathbf{r}}}_{\text{M}}\right]$$ (2)

where a_M = T_M,H T_H,Ta_T, b_M = T_M,H T_H,Tb_T, ${\widehat{\mathbf{r}}}_{\text{M}}=\frac{{\mathbf{b}}_{\text{M}}-{\mathbf{a}}_{\text{M}}}{||{\mathbf{b}}_{\text{M}}-{\mathbf{a}}_{\text{M}}||}$ is the principle ray direction, $\cos \theta ={\widehat{\mathbf{r}}}_{\text{M}}\cdot \widehat{\mathbf{z}}$ and K is the camera intrinsic matrix. d is the distance from the virtual camera center to the tool tip, which we set at 650 mm. Note that we do not constrain rotation in the detector plane, since most digital X-ray devices, including the Loop-X, allow for image rotation after acquisition. The virtual camera approximates an X-ray device with a source-to-detector distance of 1020 mm, 0.194 mm pixel size, and a 1536 × 1536 detector. For the sake of performance, the DRR is rendered with 4 × 4 binning.

Figure 2.

Our experimental setup, with the “Tool + Preview” interface shown. The patient model M is registered to a radiopaque pelvic phantom P (visually obscured). In our experiments, optical tracking by the HMD provides the kinematic chain to obtain T_M,T, while the BrainLab Curve tracker L has line of sight to both the tool T and gantry G, enabling positioning of the Loop-X. A DRR (green) is shown in the surgeon’s line of sight, corresponding to a real X-ray along the same principle ray direction, between the X-ray source (below table) and detector (top). DRRs are flipped left-to-right for viewing due to the source being below the patient.

For the AR Handle + Preview interface, the AR handle consists of a grabbable virtual object with pose T_H,ARH with respect to the HMD. The handle indicates a center point c_ARH analagous to the physical tooltip a_T as well as the principle ray direction ${\widehat{\mathbf{r}}}_{\text{ARH}}$ in the handle frame. The projection matrix is then calculated as above, substituting a_M ← T_M,H T_H,ARHc_ARH and ${\widehat{\mathbf{r}}}_{\text{M}}\leftarrow T_{\text{M,H}}{T}_{\text{H,ARH}}{\widehat{\mathbf{r}}}_{\text{ARH}}$. This AR Handle can be manipulated in 6 DOF by pinching the virtual object itself or from afar.

Our prototype uses a Microsoft HoloLens 2 in communication with a server. For all optical tracking from the HoloLens, we adapt the algorithm proposed by Martin- Gomez et al. (2022) to stream infrared and short depth image sensing from the HoloLens to the server, which computes the pointer tool and pelvis poses.¹ At the same time, the server renders DRRs using a Titan 2080 Ti graphics card and open source software tools modified to support dynamic updates to the viewpoint Unberath et al. (2018, 2019). The workstation communicates the tool poses and updated DRRs back to the HoloLens at a rate of 1 per second, as seen in the surgeon’s field of view. The live preview frame rate and resolution are constrained primarily by the wireless connection to the HMD, rather than DRR render time. The HoloLens also displays holographic reconstructions of the optical markers, confirming tracking is up-to-date. This is necessary because the Loop-X gantry occaisonally obscures the line of sight between the HoloLens and the pointer tool. Additionally, the short depth image sensing of the HoloLens, which was intended to facilitate accurate hand tracking, is limited to 1 m.

4. Experiment

For a baseline, we compare our proposed interfaces to the existing Loop-X feature that enables a clinician to specify the viewing direction with a tracked tool. The Loop-X device repositions itself to obtain this viewing direction is closely as possible, adjusting the source angle, detector angle, lateral movement, longitudinal movement, traction yaw (rotation on the floor), and gantry tilt. For the Tool + Preview interface, the same tracked tool is used to specify the viewing direction of both the DRR and the Loop-X, ensuring the two poses align up to the tracking error of both systems. For AR Handle + Preview, we direct the Loop-X to the indicated pose, after the handle has been specified, by holding the pointer tool in the same position in space. This procedural step overcomes the current implementation challenges of controlling the Loop-X and is not considered to be part of the AR Handle interface.

4.1. Results

Our results show the potential for the proposed gantry interfaces to reduce the number of real X-rays needed. “Tool + Preview” facilitated single-shot acquisition of the desired views, compared to 2–4 shots using the baseline “Tool” interface with no mixed reality component. “AR Handle + Preview” performed similarly, with 4/6 standard views acquired on the first try. In the AP and Inlet views, only minor lateral movements were needed to correct the initial shot and bring the relevant anatomy fully in view. The baseline “Tool” interface, by comparison, required up to 3 manual adjustments to the viewing angle, where the surgeon directed a non-expert operator with instructions such as “roll it up just a little more,” and “tilt back,” in addition to lateral and longitudinal movements.

The final images, shown in Fig. 4, show nearly identical views onto the anatomy were achieved, with the primary differences arising from rotation about the principle ray. For completeness, Table 1 indicates the corresponding pose acquired by the Loop-X for each image, relative to an initial “home” position in line with the patient table. Different poses are not necessarily an indication of significantly different viewing directions, due to the fact that only the viewing direction is specified by the tool, not the rotation thereabout. The Loop-X attains this viewing direction as closely as possible while avoiding collisions with the table.

Figure 4.

The final X-ray images obtained for each view under the three gantry interfaces. 8 out of 10 images using a live preview were evaluated as “perfect” on the first shot, whereas the existing “Tool” interface required at least 2 and up to 4 shots per view. Image artifacts are a result of CLAHE histogram equalization, applied by default.

Table 1.

The final Loop-X pose for each view, using the Brainlab pointer tool interface (white), our “Tool + Preview” interface (red), and our “AR Handle + Preview” interface (blue). Note the number of shots required to obtain each view.

Tool	Source (°)	Detector (°)	Lateral (cm)	Longitudinal (cm)	Traction Yaw (°)	Gantry Tilt (°)	Number of Shots
Tool + Preview
AR Handle + Preview
	179.4	0.47	−0.02	9.3	0.99	−1.92	4
	178.91	2.93	−0.02	13.27	−0.67	−0.48	1
AP	173.31	−2.86	−0.32	14.98	4.64	4.5	2
	175.6	−3.29	−0.07	25.62	3.62	29.39	2
	177.98	3.46	−0.18	22.98	−0.23	21.34	1
Inlet	177.24	6.67	−2.99	23.79	−1.02	24.35	2
	178.63	4.59	2.59	−14.31	9.13	−28.89	4
	177.1	−11.42	−0.01	−11.01	−5.55	−25.01	1
Outlet	181.11	18.61	−2.29	−16.37	7.54	−24.52	1
	222.25	42.94	0.05	9.72	0.93	−4.01	2
	223.85	53.5	−0.11	5.82	4.24	−5.68	1
Obturator Oblique	214.74	45.19	0.04	15.19	−6.23	−0.47	1
	138.95	−22.59	−3.22	11.69	6.04	0.18	2
	127.03	−41.12	−5.42	19.12	9.97	10.26	1
Iliac Oblique	132.93	−25.49	−4.63	11.08	4.14	−1.35	1
	226.41	41.14	8.57	−15.47	9.96	−24.46	4
	198.49	30.71	−0.17	−14.89	9.86	−22.41	1
Teardrop	206.23	37.4	−0.06	−15.99	9.95	−24.26	1

5. Discussion

A mixed reality interface for controlling robotic X-ray devices presents numerous opportunities for future work. For instance, our prototype implementation relies on optical tracking of the anatomy, but the choice of anatomical model for rendering DRRs could likewise consist of a preoperative CT or statistical shape model registered to a tracked C-arm via automatic 2D/3D registration (Grupp et al. 2020; Gao et al. 2020; De Silva et al. 2018). Additional work may focus on questions of human-centered design of the mixed reality interface, which has been shown to affect surgical accuracy (Gu et al. 2022), by leveraging the depth sensing of the HoloLens to enrich the information available when planning the next robotic movement. The motion of the AR Handle might be constrained to reflect physically achievable views with the robotic X-ray system, avoiding collisions with the patient and surgical instruments. Moreover, fine control over the rotational degree of freedom in mixed reality may be integrated with the post-acquisition rotation of the image by the X-ray device. Finally, future work will provide higher resolution DRRs at a faster frame rate by leveraging more efficient data transfer protocols.

6. Conclusion

We have considered the question of how best to interface with robotic X-ray devices, in order to investigate possibilities for more effective interfaces that would reduce the time needed for repositioning of the gantry and reducing the number of images that are acquired solely for the purpose of view finding. With the introduction of mobile robotic arms with independent source and detector movements, this question becomes even more pertinent because the added complexity may result in miscommunication between the surgeon and the radiological technician. We studied the use of mixed reality to show a live preview of the next X-ray, increasing the likelihood that the first shot is as close as possible to the desired viewing plane. Separately, the use of an AR Handle allows the surgeon to specify the next view even when the arm or detector would physically obstruct placement of a pointer tool, as well as to maintain visualization of previously acquired views without holding a tool in place. Furthermore, based on our evaluation, either of these paradigms can enable single-shot acquisition of the desired view, reducing the number of overall acquisitions and consequently radiation exposure for the patient and clinicians.