A Fast and Interactive Augmented Reality System for PET/CT-guided Intervention of Neuroblastoma

Abstract

Neuroblastoma is the most common type of extracranial solid tumor in children and can often result in death if not treated. High-intensity focused ultrasound (HIFU) is a non-invasive technique for treating tissue that is deep within the body. It avoids the use of ionizing radiation, avoiding long-term side-effects of these treatments. The goal of this project was to develop the rendering component of an augmented reality (AR) system with potential applications for image-guided HIFU treatment of neuroblastoma. Our project focuses on taking 3D models of neuroblastoma lesions obtained from PET/CT and displaying them in our AR system in near real-time for use by physicians. We used volume ray casting with raster graphics as our preferred rendering method, as it allows for the real-time editing of our 3D radiologic data. Some unique features of our AR system include intuitive hand gestures and virtual user interfaces that allow the user to interact with the rendered data and process PET/CT images for optimal visualization. We implemented the feature to set a custom transfer function, set custom intensity cutoff points, and region-of-interest extraction via cutting planes. In the future, we hope to incorporate this work as part of a complete system for focused ultrasound treatment by adding ultrasound simulation, visualization, and deformable registration.

1. INTRODUCTION

Neuroblastoma is a form of embryonal tumor that is found in the sympathetic division of the autonomic nervous system.¹ It is one of the most common forms of cancer in children. Typically, since these tumors arise in the sympathetic nervous system, they present as mass lesions on the neck, chest, abdomen, and pelvis.^1,2 If the presence of the disease is discovered early enough, it can be treated with a fairly low risk. The likelihood of death or serious side-effects increases the longer it takes to uncover and diagnose it.¹ In many cases, however, even early diagnosis and treatment is unable to help and more than 50% of patients do not survive despite receiving intensive treatment.³

Conventional treatments for metastatic high-risk neuroblastoma often involve aggressive chemotherapies with long-term side effects.¹ Recently, high-intensity focused ultrasound (HIFU) has been investigated as a non-invasive technique that is able to sensitize previously unresponsive neuroblastomas to checkpoint inhibitor therapy in rodents.¹ HIFU can penetrate deep inside the human body and focus energy on millimeter-sized targets.⁴ HIFU can be used to kill neuroblastoma using either thermal ablation or mechanical fractionation.^3,4,5 Thermal ablation occurs as the tissue absorbs ultrasound energy, causing a rise in temperature as high as 60°C, resulting in immediate cell death.⁴ Mechanical fractionation is a more effective method of treating neuroblastoma and occurs when high-intensity acoustic pulses cause cell cavitation and the self-destruction of cancer cell nuclei.^3,4 To ensure the safety of the treatment, a second imaging modality must be used as a guide. Typically, magnetic resonance imaging or sonography are used, especially for pelvic and abdominal tumors.^2,4,5 Positron emission tomography and computed tomography (PET/CT), however, have been shown to be an effective diagnostic and prognostic tool for pediatric neuroblastoma, especially in meta-iodo benzyl guanidine (mIBG)-avid cases.^2,6 Current PET scanning methods use ¹⁸F-FDG to detect skeletal neuroblastoma, but it is recommended for use primarily in older children rather than infants due to the large radiation dose involved.²

Some intraoperative imaging methods, such as intraoperative three-dimensional cone beam CT imaging, have been used in orthopedic surgery and other applications.⁷ Their guidance of neuroblastoma treatment in children, however, can be challenging. Augmented reality (AR) has been investigated for potential use in interventions. One such case combined the HoloLens 2 head-mounted display with intra-operative CT for use in needle tissue acquisition. This system, however, had to be calibrated using a standard CT grid, rather than using tracking with deformable registration.⁸ Another system helped surgeons place screws during spinal surgeries, but did not use an external head-mounted display, focusing instead on 2D display techniques.⁹ To the best of our knowledge, there is no reported AR system that can register and display PET/CT images in real time while providing an editable transfer function during holographic display.

As PET/CT data is very large, processing speed is a challenging but critical necessity for real-time data manipulation. HoloLens can be used to render and visualize radiologic data,¹⁰ but the ability to dynamically edit what is rendered is limited. Another important feature for AR navigation is the ability to track real-world objects. Ding et al. used QR codes to aid in registration for surgical guidance.¹¹ In order to accurately match the patient’s real-time conditions, such as motion and deformation, the virtual model needs to be processed efficiently. During AR navigation, hand motion tracking and misregistration can cause user input to not be appropriately executed. A common example of this is a grab not working, an open palm being read as a grab motion, or accidentally moving an object in an unexpected manner. Our paper contributes a novel technique for data representation and processing speed gains in AR navigation that can be applied as part of a larger system to guide focused ultrasound therapy for the treatment of neuroblastoma.

2. METHODS

2.1. Data Rendering

We implemented our AR platform in the Unity game engine using C# and High-Level Shader Language (HLSL) and streamed the data to HoloLens 2, a head-mounted display developed by Microsoft. Initially, we tested the Marching Cubes (MC) algorithm for rendering. This was a very effective algorithm as it was relatively easy to implement and understand. MC divides the point cloud of the data into a 3D grid and uses a divide-and-conquer approach to determine how the surface of the data intersects with each cube in the grid.¹² Since there are a limited number of ways that the surface can intersect with the cube, specifically 2⁸ or 256 states a number which is reducible to only 14 due to symmetry, MC can be implemented using a look up table.¹² We implemented an extra feature that allowed us to not render pixels that fell above or below a specific user-defined threshold. Unfortunately, we discovered major limitations in this algorithm early on. The model was too slow to be useable in any clinical setting, as it had to re-render the entire model after any user input.

Since only a limited amount of data needed to be displayed on the screen at once, we opted to use volume ray casting with raster graphics. Raster graphics is a technique where data is represented as a set of discrete, 2D pixels on a screen. Realizing that whatever we display is merely a 2D representation of 3D data, we decided to use volume ray casting for rendering. Volume ray casting is a commonly used technique for converting 3D volumetric data to a 2D image that can be rendered on a screen in near real time. Volume ray casting is a simple process with very few steps. First, a ray is cast that pierces the model at a fixed point and direction. It then takes intensity samples at regular intervals culminating in an overall intensity for that pixel in the viewport.¹³ It can be describe using the following equation:

$$I_t=\sum _1^n{I}_n$$

where I_t is the total intensity rendered, and I_n is the intensity at step n. We use the screen resolution, in our case HoloLens 2’s resolution of 2560×1440 pixels, as the upper limit for the number of rays, as we should only have one ray for every pixel on the screen. Each pixel rendered corresponds to the composite of the data along the line where the ray was cast. Since the rays can be cast in no particular order with a nearly identical outcome, this process can be efficiently parallelized.¹³ Figure 1 gives a graphical explanation of volume ray casting. The implementation of our volume ray casting in Unity was based on the work done by Lavik.¹⁴ Lavik’s implementation used a compute shader to process continuously sampled 3D voxel data, eventually generating the final model view. We altered the provided shader so that the data could be properly scaled to the real-life dimensions of the model. Not only does this improve the code’s interoperability with AR input, but it allows the model to be used in a medically relevant fashion.

Figure 1.

A visual representation of how ray casting works. Rays are cast from behind the volumetric data and sample the data every n steps before displaying a 2D composite on the headset.

We decided to use volume ray casting over the marching cube algorithm because the marching cube algorithm updates and reloads the entire model after every change. In contrast, volume ray casting allows for the continuous updating of the model after every frame. This allows changes in custom intensity cutoff points, transfer functions, and cutting planes to be shown to the user in real time. The only downside is that the result of the volume ray casting isn’t as precise as the marching cube algorithm. The marching cube algorithm has very well-defined cutoff points and positions which depend on the input model. Meanwhile, in volume ray casting, small changes sampling points and starting locations can change the final sample for a particular ray slightly. This change, however, is not noticeable to the human eye. At no point throughout testing was any offset seen without immense magnification of the model.

2.2. User Interface and Tools

Another important component for our AR solution was a graphical user interface (GUI) to allow the user to control what is displayed on HoloLens 2. Our UI contains sliders that allow the user to adjust the minimum and maximum intensity thresholds, as shown in Figure 2. Virtual menu buttons are also included so that the user can import additional DICOM files, switch between different mesh renderings, change the transfer function for a mesh, and crop the data with cutting planes or cubes (in Figure 3). These controls are useful for the locating and visualizing of neuroblastoma and other clinically relevant features, giving surgeons greater control over what they view in the patient.

Figure 2.

An example of real-time rendering while an intensity threshold has been applied. a) CT scan with no intensity thresholding. b) CT scan with intensity threshold applied, leaving only bone. c) PET scan with no thresholding applied. D) MRI scan with thresholding applied, filtering out the air and noise. Virtual menus are also shown in a), b) and c).

Figure 3.

An image taken within HoloLens 2, demonstrating the use of a virtual cube to crop the region of interest so that it focuses on relevant biological features. The rendering transfer function has also been customized to increase the contrast between the lungs and large intestines compared to bones and other soft tissue.

Intensity thresholding was implemented in a very simple way. Any pixel above the maximum intensity and below the minimum intensity was dropped from the render and made transparent. This is a useful feature for surgeons who are trying to localize and isolate a specific tissue type with a well-known, distinct intensity.

Cutoff boxes and planes are useful for physicians who want to isolate a specific region in a scan and ignore the rest of it. We implemented both of these features in such a way that they only apply to the data that they were spawned for. As a result, we can view data like we do in Figure 4, where the CT and MRI data are shown simultaneously, but the CT scan has been partially cut.

Figure 4.

A combination of a transfer function and intensity thresholding applied to a CT scan, highlighting the bone and hiding the tissue.

The last feature that we added to our system was the ability to dynamically load a custom transfer function. We implemented two different kinds of transfer functions. The first method was a simple intensity lookup table. If the final intensity generated by volume ray casting was at intensity I, it was made a specific color. The other method we implemented was a differential transfer function. In this case, the final color is based on the difference in intensity between the minimum and maximum intensity values encountered by the ray being cast. This feature allows the physician to highlight a specific tissue or a lesion, making it easier to see while being rendered, as seen in Figure 3 where the lungs and intestines have been highlighted, and in Figure 4 where the bones have been highlighted.

The model registration works using HoloLens 2 to communicate the expected location of the object to Unity. It approximates the user’s current position using internal time of flight (ToF) sensors which emit photons and record the return time to determine distance. These sensors are broken down into two groups: long throw and short throw. Long throw sensors can be thought of as less precise, long-range sensors used for spatial mapping and understanding. Short throw sensors are more precise, short-range sensors used to measure the hand and determine the user’s inputs.¹⁶ The spatial map generated by HoloLens 2 is used to locate the user and where the object should be placed.

3. RESULTS

We tested the performance, quality, and usability of the AR system on a desktop computer with an Nvidia GeForce GTX 1080 Ti GPU, an Intel i9-7900X CPU, and 64 GB of RAM. This computer was used to process the data and stream it to HoloLens 2. Our performance measurements focus on user experience after initial loading, excluding the loading of the initial dataset. The system performed well, rendering at a rate of consistently above 60 frames per second. Users were given the ability to freely import additional DICOM images from the connected PC for rendering. Qualitatively, the generated meshes appear accurate and high-quality. Unfortunately, since what is represented is derived from the source data rather than being a completely accurate portrayal, there is no objective, quantitative way of measuring accuracy. Instead, we compared it to what was shown by other commonly used pieces of software, specifically 3D Slicer. We found that the resulting isosurfaces from both pieces of software appeared equally high-quality. Based on this assessment, the produced models are as high-quality as the original DICOM images given. Our accuracy level allows the recognition of distinct features, such as the face, from high-quality DICOM images. Another important feature of our system can be seen in Figure 5. Our system was able to consistently render a full PET/CT dataset with different transfer and cropping functions applied to the individual PET and CT images.

Figure 5.

PET/CT images showing the extracted skeleton from CT and the tracer uptake from PET. A cutting plane allows the user to define the view area of the skeleton and the field of view of the tracer uptake.

Our system provides intuitive user interface elements. Despite this, the headset does not have precise enough tracking mechanism to allow for consistent control, therefore, the manipulation of data and GUI elements can be further improved. Long, slow movements are essential for guiding it as simply and smoothly as possible. Future AR headsets will need to improve their tracking capabilities for use in a medical context if this technology is to be used. The current menu, however, is functional for demonstrating the capabilities of the software.

In addition, we incorporated a system utilizing custom 2D and 3D transfer functions whose role is to allow the user to drastically change how hologram rendering takes place. The final visualization can change dramatically, as seen in Figure 6. This allows us to create rendering presets which can be used to aid the user in focusing on certain organs or regions present within the scan in real time for planning and visualization purposes, while eliminating the need for manual segmentation or cropping of the full rendering.

Figure 6.

A demonstration of the versatility of the editable transfer function with the same data resulting in three different visualizations. A) A transfer function configuration resulting in an abdominal view with the stomach, liver, intestines, and kidneys visible. B) A closeup of the abdominal region with the kidneys indicated by arrows. C) A configuration showing soft tissue and skeletal muscle throughout the body.

4. DISCUSSION

In this work, we present a system that allows for the fast visualization of PET/CT data using augmented reality. Our AR system has several unique technical features including intuitive hand gestures and a virtual graphical user interface that allows the user to interact with PET/CT data and perform real time image processing. The AR system also uses volume ray casting to render PET/CT images for holograph visualization, which can accurately represent the model to match the real patient situation. It is well suited for viewing 3D models from multimodal imaging data, such as PET/CT, and editing it in real-time. Users can seamlessly view PET/CT data while still being given significant flexibility as to what aspects of the model are visible. It can be utilized both pre- and intra-operatively to provide the physician with valuable 3D data, enhancing their spatial knowledge of the location of the lesion from PET.

In the future, our proposed rendering system might be used as a major component in a neuroblastoma treatment system. High-intensity focused ultrasound is an important, developing technique for treating neuroblastoma, but unfortunately current methods do not provide real-time visualization and guidance. Using AR could greatly increase the accuracy of HIFU probe placement by allowing physicians to visually locate the tumor on the patient. Ultrasound simulations might also be developed and fed into a headset to display the trajectory of HIFU beams and ensure that they reach the correct target. Work has already been done to increase the accuracy of HoloLens 2’s positioning using infrared optical trackers and highly accurate motion-capture cameras, as recommended by Hübner et al.¹⁶ Coupled with techniques such as deformable registration between intraoperative imaging data and preoperative CT data such as that demonstrated by El-Hariri et al.¹⁷, a safe, effective HIFU system could be created. We have already done preliminary work in these fields and have created more realistic phantoms for both CT and US.¹⁸ As these phantoms exhibit similar mechanical properties to soft tissues, they should aid us in evaluating multimodal image fusion and deformable registration techniques. We will apply the fast-rendering methods to our existing AR systems.^19–21

5. CONCLUSION

We developed a fast and interactive augmented reality system that displays 3D PET/CT data with editable transfer functions and cutting planes. Our system’s rendering is optimized and can display large volumes of data in near real time. The system can register multimodal images and allow the user to interact with the data using intuitive hand gestures and a virtual user interface. Rapid 3D rendering methods such as the one we developed are an essential part of augmented reality systems for medical treatment. By identifying and visualizing the lesion from PET images and registering the lesion with the physical patient, the AR system may be able to assist the clinicians. Combining this with visual ultrasound simulations, the whole system would be useful in the delivery of focused ultrasound for the treatment of not only neuroblastoma, but other tumors as well.