Table 3.

Planning workflow

• Load MRI data sets, pre-segmented models, and structure classification

• Generate textures (Host-PC and voice/HoloLens)

• Activate imaging 2D virtual display and present the first selected image set (voice)

• Activate hologram and superimpose forbidden virtual fixtures, per classification

• Inspect hologram and images on the 2D virtual display to identify the targeted pathologic foci

• Select the target point T on 2D virtual display (gesture); software records its coordinates in the HoloScene coordinate system (xT, yT, zT) and replicates it in the hologram.a

• The software generates and presents as a hologram and in the 2D virtual display a virtual trajectory (initially set along the MRI z-axis) that starts from the point (xT, yT, zT).

• The operator interactively adjusts the virtual trajectory (with its distal tip anchored at point T).b

• On-the-fly constraints are applied to avoid collision with forbidden zones and ensure the constraints imposed by the needle’s length are met.

• Calculate insertion point I.c

^aThis is a workflow suggested for the HAR by the work in this paper; other workflows may be appropriate for other procedures and by other operators.

^bAll calculations and maneuvering are performed in the MRI coordinate system; these measures can be transformed to any coordinate, such as that of the operating room, for example, using the appropriate rigid body transformation as illustrated in Fig. 3a.

^cCalculation of the insertion point on the skin could correspond to identification of a drilling point on the patient’s scalp, for example.