Intraoperative 3-dimensional Projection of Blood Vessels on Body Surface Using an Augmented Reality System

Abstract

Summary: Preoperative understanding of the running pattern of blood vessels is an important factor to approach surgical fields safely. In 2 cases where the vascular abnormalities were estimated, we projected the blood vessels onto the surgical field using an augmented reality device HoloLens. A splint was made to allow the patient to be fixed while undergoing computed tomographic angiography. Three-dimensional (3D) data on the blood vessels, skin surfaces, bones, and the 3 chosen points for alignment were segmented and then projected onto the body surfaces as holograms using the HoloLens. Two types of projection for holograms were used: projection type 1—where the body contours were projected as a line, and projection type 2—where the body surface was projected as meshed skin type. By projecting projection type 2 rather than projection type 1, we gained a better understanding of the 3D anatomic findings and deformation characteristics, including the anatomic blood vessel variation and positional relationships between the organs and body surfaces. To some extent, we could make sure that the depth perception can be obtained by recognizing the bone, vessels, or tumor inside the meshed skin surface. Our new method allows the 3D visualization of blood vessels from the body surface, and helps understand the 3D anatomic variation of the blood vessels to be applied as long as the blood vessels can be visualized.

INTRODUCTION

Preoperative understanding of the running pattern of blood vessels is an important factor to approach surgical fields safely. In particular, anatomic positional abnormalities of the blood vessels may accompany cases with congenital anomalies or those in which multiple operations have been performed. By identifying these abnormalities, complications such as bleeding by injury to the vessels could be prevented.

Computed tomographic angiography (CTA)^1,2 has enabled peripheral blood vessels to be visualized precisely, and 3D images of blood vessels, body surfaces, and bones may be evaluated. Augmented reality (AR) is a technology that combines computer-generated images on a screen with a real object or scene.³ Because of the lack of a shifting viewpoint through the use of a goggle-type device, the AR technology enables comparison between a visible organ and its corresponding simulated image. However, it is difficult to judge the actual distance of organs from the body surface because internal organs are projected onto body surfaces. Problems such as those related to depth perception remain in medical AR.^4–10

In 2 cases where the vascular abnormalities were estimated, we projected the blood vessels onto the surgical field using an AR device (HoloLens; Microsoft Corp., Redmond, Wash.). We described these in Results and Discussion sections of this report.

MATERIALS AND METHODS

We previously reported on the usefulness of intraoperative body surface evaluation using AR devices.¹¹ In addition, we developed a 3-point registration application for alignment between the entity and the holograms.¹² Preoperatively, 3 points were chosen to serve as references for alignment and marked on body surfaces near the surgical field (Fig. 1). A splint was made to allow the patient to be fixed, ensuring that the surgical field would remain in the same position as that during surgery. Moreover, the same splint was fitted to the patient undergoing CTA. For body areas where creation of a splint is not applicable, we can cope with by deciding the positions of the 3 points for alignment.⁴ Digital Imaging and Communication in Medicine data on the blood vessels, skin surfaces, and bones, and those from the 3 chosen points, were obtained from the CTA. The data were segmented using free 3D image analysis software (3D Slicer) and were processed using free 3D computer graphics software (Blender) (Fig. 2). The data were then projected onto the body surfaces as holograms, using the HoloLens combined with our application.⁴ Two different types of holograms were projected: projection type 1 (body contour type)—in which the application was set to project the body contours as a yellow line (Fig. 3), and projection type 2 (meshed skin type) —in which the body surface was processed to be composed of almost uniform tetrahedral polygons, using free software (Instant Meshes; Interactive Geometry Lab, Zurich, Switzerland) (Fig. 4).

Fig. 1.

Workflow of this study. 3D Slicer; free 3D image analysis software (The Slicer Community) Blender; free 3D computer graphics software (Blender Foundation, Amsterdam, The Netherlands) OBJ file; simple data format representing only 3D geometry Stereolithographic (STL); one of the file formats for storing data representing 3-dimensional shapes Unity; and free game engine (Unity Technologies, Copenhagen, Denmark). DICOM indicates Digital Imaging and Communication in Medicine.

Fig. 2.

A 3D image as a base of hologram. The 3-dimensional data of blood vessel, bone, tumor, body surface, and selected 3 points were processed using Blender. The body surface in this figure is indicated by projection type 2 (meshed skin type).

Fig. 3.

Projection type 1 (body contour type). The skin surfaces were projected onto the body surfaces along with the 3-dimensional data of the blood vessels and bones. The figure on the left is the controller as holograms. UI, user interface.

Fig. 4.

Projection type 2 (meshed skin type). Holograms of the meshed skin type, blood vessels, bones, and tumor were displayed on the body surfaces. The hologram of the meshed skin type helped estimate the depth of the tumor from the body surface. Moreover, the 3-dimensional position of the tumor, which was difficult to understand using only the naked eye, was identified, and the excision range of the tumor could be easily understood.

RESULTS

Regarding type 1 (body contour type), a 13-year-old adolescent girl presented with macrodactyly of the right foot. We planned to reduce the volume of the fourth metatarsal and proximal phalanx and to shorten the distal and proximal phalanx. Holograms were displayed on the body surfaces (Fig. 3). (See Video 1 [online], which displays holograms being projected on the body surfaces in projection type 1 and type 2.) Because we judged that previous approach carried a risk of vascular injury, the surgical plan was changed to make a new incision.

Video 1. Video 1 from “Intraoperative three-dimensional projection of blood vessels on body surface using an augmented reality system”.

Download video file ^{(8.9MB, mp4)}

Regarding type 2 (meshed skin type), a 67-year-old woman presented with malignant peripheral nerve sheath tumor in the right forearm. We planned for tumor resection and reconstruction with a free anterolateral thigh flap. Holograms were displayed on the body surfaces (Fig. 4). (See Video 1 [online], which displays holograms being projected on the body surfaces in projection type 1 and type 2.)

On projecting meshed skin-type holograms, we could easily identify the resection range of the tumor and determine the recipient vessel of the free flap. Compared with type 1, we gained a better understanding of the 3D anatomic findings and deformation characteristics, including the anatomical blood vessel variation and positional relationships between the organs and body surfaces. To some extent, we could make sure the depth perception can be obtained by recognizing the bone, the vessels, or the tumor inside the mesh skin surface.

DISCUSSION

Doppler ultrasonography and computed tomographic angiography are common methods for evaluating blood vessels; it is difficult to understand the overall image of blood vessels and the 3D positional relationship between blood vessels and body surfaces in the actual surgical field. Our new method allows the 3D visualization of the blood vessels to be applied even for complex vascular abnormalities as long as the blood vessels can be visualized by CTA. The use of free software for image processing minimizes costs but can perform complicated processing of the peripheral blood vessels.

The AR technology has recently been used to visualize organs and tumors during surgery.^13–16 Evaluations of the perforating artery of free flaps have also been performed on body surfaces.^17–19 Our method is projecting 3D images obtained by CTA. In principle, our method should enable us to visualize blood vessels as long as we can recognize them by CTA. Recently, it has become possible to visualize very thin perforating branches of 0.3–1.6 mm using ultra-high-resolution CT. However, further analysis must be done.^20–22

Problems such as those related to depth perception remain in medical AR.^12–17 It is difficult to judge the actual distance of organs from the body surface because internal organs are projected onto body surfaces. Sielhorst et al⁷ suggested that one of the best visualization modes for interacting with a 3D model in medical AR is to use a virtual window. A virtual window^14–16 that could be overlaid onto the skin of the patient and partially occlude the organs would enhance the perceptive information regarding depth.

In type 1 projection, only the outline of the body is displayed, whereas in type 2 projection, the entire body surface is displayed in a mesh shape. That is, in the type 2, a 3D body surface form can be recognized by seeing the structure through the mesh. Therefore, it is considered that the 3-dimensional positional relationship with organs, including blood vessels, anatomical findings, and deformation characteristics, is easy to understand. In addition, we could make sure that the depth perception can be made.

CONCLUSION

Our method allows visualization of the running pattern of blood vessels from the body surface, and helps understand the 3D anatomic variation of the blood vessels.