Towards Reducing Visual Workload in Surgical Navigation: Proof-of-concept of an Augmented Reality Haptic Guidance System

Abstract

The integration of navigation capabilities into the operating room has enabled surgeons take on more precise procedures guided by a pre-operative plan. Traditionally, navigation information based on this plan is presented using monitors in the surgical theater. But the monitors force the surgeon to frequently look away from the surgical area. Alternative technologies, such as augmented reality, have enabled surgeons to visualize navigation information in-situ. However, burdening the visual field with additional information can be distracting. In this work, we propose integrating haptic feedback into a surgical tool handle to enable surgical guidance capabilities. This property reduces the amount of visual information, freeing surgeons to maintain visual attention over the patient and the surgical site. To investigate the feasibility of this guidance paradigm we conducted a pilot study with six subjects. Participants traced paths, pinpointed locations and matched alignments with a mock surgical tool featuring a novel haptic handle. We collected quantitative data, tracking user’s accuracy and time to completion as well as subjective cognitive load. Our results show that haptic feedback can guide participants using a tool to sub-millimeter and sub-degree accuracy with only little training. Participants were able to match a location with an average error of $0.82\text{\hspace{0.17em}mm}$, desired pivot alignments with an average error of $0.83$^° and desired rotations to $0.46$^°.

1. Introduction

Minimally-invasive procedures typically require precise placement and alignment of surgical tools in accordance with pre-operational plans. To help surgeons meet these challenges, surgical navigation systems integrate medical imaging and position tracking to create a closed loop system, where the surgeons can compare their current tool placement to the planned positioning. Traditional approaches provide this information via 2D monitors, located off-situ and around the operating room which forces the surgeon to split their attention between the surgical scene and off-situ monitors.

Prior work investigated the possibility to make navigation feedback more accessible by visualizing the data using small displays placed on the tool itself (Herrlich et al. 2017; Brendle et al. 2020; Schütz et al. 2021), or providing navigation feedback through optical-see-through headsets (Rahman et al. 2020). The proposed systems eliminate the need for surgeons to move their head away from the surgical scene when checking the position and alignment of their tools.

However, it has been shown that human visual attention is very limited and focusing on a dynamic event can lead us to miss abnormalities in our field of view (Simons and Chabris 1999). This inattentional blindness has been demonstrated in the use of augmented reality (AR) in surgery, where surgeons have missed visible foreign bodies in-situ dixon2013surgeons, (Dixon et al. 2013, 2014).

A way to avoid burdening this limited visual attention is to shift navigation feedback onto another modality, one possible candidate being our sense of touch. Reinschluessel et al. (2018) found that providing haptic feedback through a sleeve with eight evenly distributed vibrotactors, alongside a monitor display, can assist users with aligning a surgical needle. However, we are interested in studying the effectiveness of haptic feedback as a standalone modality. Indeed, there is precedence for using haptics alone as navigation feedback. Hong et al. (2017) asked users to trace invisible paths and locate invisible points on a touch screen with their fingers while guided by 4 or 8 vibrotactors mounted to their wrist. Instead of manipulating tools, Hong’s study focused on guiding a subject’s fingertips. Additionally, the benefit of using solely haptic guidance for tracing invisible paths with a stylus was discussed by Zarei-nia et al. (2009), yet the stylus tool was connected to a table-grounded haptic device.

Against this background, this work investigate how a tool-mounted haptic device alone can guide users to perform tool-based placement and alignment tasks.

2. Methods

To investigate the effectiveness of haptic feedback as an alternative to visual feedback for surgical tool guidance, we placed an array of vibrotactors on the handle of a custom mock surgical tool (Fig. 1) and designed an experimental setup for evaluating the user’s performance during several alignment tasks. To ensure acquisition of accurate spatial transformations during these tasks, we utilized the Polaris Vicra (NDI, Waterloo, Canada) as our tracking system.

Figure 1.:

The custom mock surgical tool consisted of 6 ERMs on the handle; tool shaft and tip; a set of tracking markers; motor drivers and controller; and a power bank connected to the controller via a power cable.

2.1. Hardware

Our mock surgical tool consisted of a 3D-printed handle, a 22-cm-long square tube as the tool shaft, and a 4-mm-diameter, metallic-coated solid sphere rigidly attached to the front of the tube serving as the tool tip. The device’s electrical system comprised of 6 Eccentric Rotating Mass tactors (ERM) (Adafruit Product #1201); a micro-controller (Adafruit Product #2821); two DC motor drivers (Adafruit Product #2927); and a 5-V power bank connected to the micro-controller via a USB cable for providing local power (Fig. 1).

2.2. Vibration patterns

The mock surgical tool guides the user via the 6 ERMs embedded in the handle (Fig. 1). The ERMs have a fixed frequency of $183Hz$, which makes them well suited for tactile stimulation (Bolanowski Jr et al. 1988). To generate different vibration patterns, the ERMs were driven with a voltage signal defined by:

$$V_{out}(t\mathrm{,}i)=5V\cdot \frac{M_i}{255}\cdot \frac{cos(2\pi \cdot t\cdot F_i)+1}{2}$$ (1)

where $i\in [1-6]$ is the index of each vibrotactor; M $\in [0-255]$ is the magnitude of the ERM; $F\in [0-5]$ is the frequency of the modulating signal; and t is time in seconds.

We specified 4 different vibration patterns: Up, Down, Left, and Right. The magnitudes and frequencies for each pattern were determined by internal testing and are described in Table 1.

Table 1.:

$M_i$ and $F_i$ values for each vibration pattern.

		$M_i$						$F_i$
Vibration pattern	Motor Index	1	2	3	4	5	6	1	2	3	4	5	6
Left		0	0	0	0	90	0	0	0	0	0	2	0
Right		0	0	0	0	0	90	0	0	0	0	0	2
Up		110	0	110	0	0	0	3	0	3	0	0	0
Down		0	110	0	110	0	0	0	3	0	3	0	0

Our first iteration of the mock surgical tool had four vibrotactors (5, 6, 2, and 1) placed on the handle indicating the four directions on a plane (Left, Right, Up, and Down, respectively). Prior work has shown that such vibrotactor placement has advantages in a guidance task involving haptic feedback (Aggravi et al. 2016). However, initial tests of the vibrotactor placement revealed that vibrotactors indicating Up (1) and Down (2) were easily misidentified as the Forward direction orthogonal to the plane. As a result, we added vibrotactors 3 and 4 to the Up and Down directions.

Throughout the course of guiding the user, a series of vibration patterns were provided based on the tool’s pose relative to the desired pose. To reduce the cognitive load imposed on the user, only one vibration pattern was presented at a time (Hong et al. 2017).

2.3. Experimental setup

Fig. 2 shows our experimental setup. We used an 12” by 24” acrylic panel as the tool’s workspace and two rigid markers, each consisting of four retro-reflective passive spheres. We defined the tracking system’s frame as $\left\{\mathbf{P}\right\}$ and established the base frame $\left\{\mathbf{W}\right\}$ from the marker fixed to the panel. The other marker was rigidly fixed to the tool shaft to establish the tool’s frame $\left\{\mathbf{T}\right\}$. Using pivot calibration, we localized the tool tip’s position and orientation with respect to the base frame as ${ }^{\mathbf{P}}{\mathbf{H}}_{\mathbf{T}}$.

Figure 2.:

The experimental setup for our pilot study, consisted of the tracking system and the panel serving as workspace for the tool. This figure also demonstrates the alignment tasks. The proctor was isolated from the participant during the study and not pictured in this figure.

2.4. Communication

We used an open-source Python library (Thompson et al. 2020) to communicate with the tracking system and designed several tool alignment tasks and guidance algorithms using the spatial transformations (Sec. 2.3). During the experiment, our Python script ran on a PC communicating with the tracking system via a USB cable and the micro-controller via WiFi. A human proctor was present during the experiment for explaining each task and controlling the experiment.

3. Pilot study

Our pilot study consisted of a training stage (Sec. 3.2.1) and an evaluation stage (Sec. 3.2.2 onward).

Data on participants’ perceived vibration pattern; alignment accuracy; and the time to completion in each trial were collected. After finishing the training and the evaluation stages, participants were asked to answer an evaluation questionnaire.

3.1. Participants

We recruited six volunteer participants in total, aged between 20 to 32. All gave consent prior to participating in the pilot study (School of Medicine, Johns Hopkins University, IRB00135294). All participants were right-handed and had no prior experience performing surgical tool alignment tasks or with our device.

3.2. Procedure

Prior to the experiment, participants were asked to complete a questionnaire and given the same introduction to our mock surgical tool and the experimental setup. After the introduction, our pilot study consisted of the following steps:

3.2.1. Training

To familiarize participants with the vibration patterns, participants were presented each of the 4 patterns: Up, Down, Left, and Right at 5 magnitudes: M = 80, 90, 100, 110, 120 (Sec. 2.2) for a total of 20 combinations in a sequence. The order was randomized to avoid participants anticipating the next pattern. The sequence would not advance until the participant had reported which vibration pattern they thought they were currently sensing. Subjects were then told the actual pattern and the sequence advanced.

3.2.2. Evaluation of training

After the training, participants were presented the same 20 vibration patterns appearing in a newly randomized order but not told the correct pattern identification. The actual pattern provided, the participant’s perceived pattern, and the time taken to identify each vibration pattern were recorded via participant inputs on a wireless keyboard.

3.2.3. Path-tracing

In this task, participants were guided to use the mock surgical tool tip to trace three paths on the panel, as demonstrated in Fig. 2(a). The paths were only defined virtually and invisible to the user. Each participant was assigned the paths in the same order. The tool tip’s position in the panel’s frame was given by the translation component of the transformation from the panel to the tool tip, ${ }^{\mathbf{W}}{\mathbf{H}}_{\mathbf{T}}$, where ${ }^{\mathbf{W}}{\mathbf{H}}_{\mathbf{T}}{=}^{\mathbf{P}}{\mathbf{H}}_{\mathbf{W}}^{-\mathbf{1}}{ }^{\mathbf{P}}{\mathbf{H}}_{\mathbf{T}}$. Participants were instructed to interpret the Up, Down, Left and Right vibration pattern as moving the tool tip in the positive X-, negative X-, positive Y-, and negative Y- direction of the panel’s frame, respectively. We manually generated in our Python script three different paths on the panel of which the frame was defined as $\left\{\mathbf{V}\right\}$ (shown in Fig. 3), all invisible to the participant. As shown in Fig. 4, each path was composed of connecting horizontal and vertical line segments:

Figure 3.:

Spatial transformations among the tracking system, tool, panel, and virtual positions on the panel. The line and eight dots in red illustrates one path for path-tracing task and desired positions for tool tip position alignment task.

Figure 4.:

The path and margin for each trial during the path-tracing task. The x- and y-axes of the plots in millimeters correspond to those of the panel.

Each path was defined by a number of waypoints, which themselves were linear transformations from the origin of the panel’s frame defined as ${ }^{\mathbf{W}}{\mathbf{H}}_{\mathbf{V}}$. At the start of each trial, participants were instructed to begin at the start location of the path. After the vibration pattern started, the tool tip’s position relative to the path was given by the linear transformation from the path to the tool tip, ${ }^{\mathbf{V}}{\mathbf{H}}_{\mathbf{T}}$, where ${ }^{\mathbf{V}}{\mathbf{H}}_{\mathbf{T}}{=}^{\mathbf{W}}{\mathbf{H}}_{\mathbf{V}}^{-\mathbf{1}}{ }^{\mathbf{W}}{\mathbf{H}}_{\mathbf{T}}$.

Participants were guided to move between waypoints while staying inside the margin shown as the polygon surrounding each path in Fig. 4. Each polygon was in the shape of a tunnel with a width of $6mm$, $3mm$ on each side of the path. While the tool tip position was outside the margin, the vibration patterns would steer the participant back towards the path. Otherwise, the participant was guided to move the tool tip towards the upcoming waypoint. As soon as the tool tip position was within $3mm$ from the upcoming waypoint, a new waypoint was assigned. Once the tool tip position was within $3mm$ from the path’s finish location, vibration would stop, informing the participant that the end of the path had been reached.

During each trial, the path and the time-stamped positions of the tool tip from the trial’s start to the tool tip reaching the end of the path were collected. Data collection rate was $20Hz$.

3.2.4. Tool tip position alignment

In this task, the participants were guided to position the mock surgical tool tip to multiple desired positions on the panel, as demonstrated in Fig. 2(a). The participants’ interpretation of the vibration patterns remained the same during this task as during path-tracing tasks. We manually generated eight different desired positions, all invisible to the participants. The desired positions were defined also in $\left\{\mathbf{V}\right\}$, by specifying the translation component of the transformation from the panel to the position, ${ }^{\mathbf{W}}{\mathbf{H}}_{\mathbf{V}}$. The order of each trial’s appearance was the same for each participant. After the vibration started during a trial, the tool tip’s position relative to the desired position was given by the translation component of ${ }^{\mathbf{V}}{\mathbf{H}}_{\mathbf{T}}$.

Since each vibration pattern could only guide the participant to move the tool tip in one direction, the vibration patterns always guided the participant to first align the tool tip’s position to the desired position’s x-axis. While the tip’s x-position was within a certain alignment threshold, the handle then guided the participant in the y-direction using the same threshold. When the tool tip was within 1mm from the desired position, the vibration patterns stopped, informing the participant that they had reached the desired position. Prior work on guidance algorithms for position alignment tasks has shown that providing an initial, coarse alignment is beneficial for increasing alignment speed while maintaining accuracy (Usevitch et al. 2020). Therefore, the participant was guided by our algorithm in two phases:

• Phase 1: while the Euclidean distance between the tool tip and the desired position was larger than $5mm$, the vibration patterns would guide the participant towards an initial alignment. The alignment threshold during this phase was $\pm 3mm$ in both x- and y-position so that the participant could maintain the x-position alignment easily while rapidly adjusting the y-position alignment.

• Phase 2: while the Euclidean distance between the tool tip and the desired position was less than or equal to $5mm$ the alignment threshold was $\pm \frac{\sqrt{2}}{2}mm$ in both x- and y-position, which guided the participant to within $1mm$ in Euclidean distance from the desired position.

For each trial, participants were asked to start at a set position on the panel. Participants were asked to hold their best alignment attempt for each trial, while the tool tip’s position, the desired position, and the time to completion were recorded.

3.2.5. Pivot alignment

In this alignment task, participants were guided to pivot the mock surgical tool to align for several desired pivots, as demonstrated in Fig. 2(b). The x- and y-axes of the tool’s frame were defined to be orthogonal to the tool shaft. Pivot alignment could be reached by user adjusting the x- and y-rotation components of ${ }^{\mathbf{W}}{\mathbf{H}}_{\mathbf{T}}$. The vibration pattern Up, Down, Left, and Right would instruct the user to pivot the tool in the positive x-, negative x-, positive y-, and negative y-direction of the panel’s frame, respectively. A fixture (see Figure 4.) restricting tool translation on the panel was attached onto the panel. We manually generated six different desired pivots: one trial pivot, which presented first to each participant and then five evaluation pivots, appearing in randomized order as trials.

Our algorithm during this task guided the participants similarly in two phases as described in Sec. 3.2.4. Once a trial started, participants were first guided to reach a coarse pivot alignment with a threshold of $\pm 3$^° in both x- and y-rotation. Next, participants were guided to fine tune the tool’s pivot with a threshold of $\pm 0.5$^°. This threshold would yield an sub-degree angular error in axis-angle representation between the actual and desired pivots. While the angular error was within $1$^°, the vibration patterns would stop, informing the participant that they had reached the desired pivot alignment. For each trial, participants were asked to hold their best alignment attempt for each trial while the actual and desired pivots were collected.

3.2.6. In-plane rotation alignment

Participants were guided to rotate the mock surgical tool to within $0.5$^° for several in-plane rotations, as demonstrated in Fig. 2(c). The fixture used during pivot alignment remained on the panel. As the z-axis of the tool’s frame was parallel to the tool shaft, in-plane rotation alignment could be reached by participants adjusting the z-rotation component of ${ }^{\mathbf{W}}{\mathbf{H}}_{\mathbf{T}}$. We manually generated six different target rotations: one trial rotation, used as training, and then five appearing in randomized order as trials. When participants aligned the tool and target rotation to within the threshold the vibration would stop, informing the participant that the desired rotation alignment had been reached. For each trial, participants were asked to hold their best alignment attempt while the desired and actual in-plane rotation angle values were collected.

3.3. Experimental variables

Data regarding the positions and orientations of the tool as well as trial duration was used to compute the following dependent variables:

• For path-tracing task: a) path efficiency (PE), measuring how well the tool tip stayed on the path, computed by dividing the length of each path by that of the length of the tool tip’s trajectory; b) average velocity (S), computed as the total length of the tool tip trajectory divided by time to completion defined as the time elapsed from the first vibration pattern until the last.

• For tool tip position alignment task: a) positional error as the Euclidean distance between the tool tip and the desired alignment position; b) velocity, computed as the Euclidean distance between the start and target positions divided by time to completion.

• For pivot alignment task: orientation errors between the tool’s and the desired orientations using an axis-angle representation.

• For in-plane rotation alignment task: rotation errors between the tool’s and the desired rotations as the difference between the tool’s and the desired z-rotation.

For evaluating task load and usability, at the end of the study, we asked participants to answer the NASA task load index questionnaire (TLX) using a 21-point rating scale. Additionally, participants were asked to answer ”It was clear when the haptic feedback was guiding me to the [left/right/up/down]” on a Single Ease Question (SEQ), 7-point Likert scale ranging from 1-(Agree) to 7-(Disagree).

4. Results

Evaluation of training:

Table 2 shows how users identified the different vibration patterns in an evaluation after the training stage. A total of 12 trials had to be dropped from the dataset due to a technical glitch, wherein a participant’s keyboard press was logged for two different trials. Participants correctly identified all remaining Left’s (27 out of 27) and all Down’s (27 out of 27). Participants were able to correctly identify the majority of Right’s (25 out of 28) but 2 out of 5 users misidentified at least one Right as Up. The same two users also misidentified one Up as Down. Mean time participants took to identify each pattern was $2.35$ seconds (SD: $0.89$ seconds).

Table 2.:

Confusion matrix of rendered and perceived patterns from the evaluation of the training stage.

		User Feedback
		Left	Right	Up	Down
	Left	27	0	0	0
Rendered	Right	0	25	3	0
Pattern	Up	0	0	24	2
	Down	0	0	0	27

Path-tracing:

We visualized the tool tip trajectory of each trial from each user by plotting the collected x- and y-positions of the tool tip (Fig. 5). In addition, Table 3 shows the path efficiency and average velocity of each participant during each trial. Overall, participant’s average PE increased with each trial: $44.63\%$, $68.16\%$ and $71.73\%$ for trial 1, 2 and 3 respectively. Participants’ average velocities were $12.66mm/s$, $8.31mm/s$ and $9.98mm/s$ for trial 1, 2 and 3.

Figure 5.:

Participants’ tool tip trajectories for each trial plotted comprehensively with respect to trials. While the tool tip was outside the margin, the tool tip position was plotted as a red dot; while inside, the tool tip position was plotted as a green dot.

Table 3.:

Participant’s Path Efficiency (PE) in % and their average velocity (S) in mm/s for each path-tracing trial

Trial:	1		2		3
	PE	S	PE	S	PE	S
User 1	23.83	8.98	54.60	6.86	46.73	8.40
User 2	33.81	10.58	69.66	8.29	82.00	11.26
User 3	70.41	12.42	60.70	13.27	46.70	16.16
User 4	5.60	24.07	77.40	4.83	81.23	5.57
User 5	51.55	13.17	85.83	9.76	81.36	11.48
User 6	82.57	6.73	60.74	6.86	92.36	7.04

Tool tip position alignment:

The positional errors and velocity for each trial from all participants are shown in Fig. 6. The only alignment attempt in which the desired positional alignment (positional error $\le 1mm$) was not reached came from User 2’s trial 7. The average positional error across all 39 trials was $0.82mm$ (SD: $0.58mm$).

Figure 6.:

Each participant’s positional error and velocity during each trial plotted. Also plotted, the $1mm$ margin within which the desired position alignment was reached. 3 trials from User 2 were missing due to a technical issue during the study.

Pivot and in-plane rotation alignment:

The median error of the pivot alignment task was $0.83$^° (SD: $0.18$^°) across 29 trials. The median error in the in-plane rotation alignment task was $0.46$^° (SD: $0.31$^°) across 25 trials. User 5 was unable to complete the last pivot alignment trial due to a technical issue, and user 6 was unable to attempt in all rotation trials due to time constraints. Apart from these, all participants were able to complete all pivot and in-plane rotation alignment trials.

Task load and SEQ:

In the NASA Task Load Index, the average score for mental load was 10.4 with the scale ranging from 1 to 21. On the SEQ, participants, on average, rated the different directions Left: 2.4, Right: 3, Up: 4.4 and Down: 5.

5. Discussion

In this work, we investigated the effectiveness of haptic feedback as the sole modality for surgical tool alignment task guidance.

Participants were generally able to correctly identify all vibration patterns after the training stage (Table 2). However, a majority of participants reported that Up and Down required a lot of mental effort to identify. One participant reasoned that Up was difficult because the vibrotactors for the Up, Left and Right vibration patterns were all making contact with the same finger, making them hard to distinguish. Users also reported that they had difficulty feeling vibrotactors 2 and 4 (Fig. 1), which were used for the Down vibration pattern, due to their locations.

When examining the tool tip trajectories for each trial of the path-tracing task (Fig. 5), we found that participants experienced the greatest difficulty in staying within the margin during trial 1. We believe this is caused by insufficient introduction to the path-tracing task, even after participants received training in identifying the four vibration patterns. The high amplitude oscillating motions from trial 1 were caused by participants moving too fast and repeatedly moving past the trajectory. Once they better understood the guidance algorithms during trial 2 and 3 they slowed down and such movements stopped. Additionally, participants sometimes took longer to realize that the vibration pattern had changed. We believe that adding a short pause between two different vibration patterns may more clearly signify the changes in pattern and allow users to react quicker.

During tool tip position, pivot, and in-plane rotation alignment tasks, some participants tended to get progressively faster in completing each trial at the start of the task yet slow down by the end. We believe this can be related to a learning effect at the beginning; and fatigue in the end from maintaining the same posture or over-stimulation on the tactile sensors in the skin, as participants were not given breaks between trials.

At last, participants generally ranked the mental demand of the experiment as medium-to-high. We believe this is correlated with the feedback that the Up and Down vibration patterns were difficult to interpret, which was confirmed by the generally higher values these patterns received in the SEQ (Section 6). Multiple participants mentioned that the placement of the vibrotactors was not optimal, being generally located away from sensitive areas, such as fingertips or the base of the fingers. However, we believe that improvements to tactor placement and the feedback patterns could address these issues as we will discuss in section 6.

6. Conclusions and future work

This paper presents a pilot study to investigate the feasibility of tool based haptics as the only feedback modality for surgical tool guidance tasks. After a training session on the vibration patterns, participants could trace invisible paths and achieve average errors of $0.82\text{\hspace{0.17em}mm}$, $0.83$^° and $0.46$^° in position, orientation and rotation alignment tasks using our mock surgical tool. However, users reported that the vibration patterns were hard to interpret and that the system was cognitively demanding.

Based on the results and user feedback from our pilot study, we plan to iterate on the vibrotactor placement to improve clarity of the vibration patterns, followed by an investigation of the feedback for users wearing surgical gloves. Furthermore, we want to explore how we can potentially guide surgeons in more than just four directions through additional types of vibration patterns. Lastly, we will investigate if proximity modulated feedback can help users perform tasks faster.

We believe that several clinical applications can benefit from using the navigation paradigm we proposed in this work. For surgical spine procedures, surgeons need to precisely place and align pedicle screws (Schütz et al. 2021). As another example, core decompression of the hip for the treatment of osteonecrosis as well as femoroplasty and hip fracture reduction applications may requires the correct placement, pivot, and rotation of the surgical drill (Ma et al. 2021). Therefore, further studies incorporating clinicians and specific surgical procedures may contribute to better understanding the benefit of our proposed navigation paradigm in clinical practice.