Drill-mounted video guidance for orthopaedic trauma surgery

Abstract.

Purpose: Percutaneous fracture fixation is a challenging procedure that requires accurate interpretation of fluoroscopic images to insert guidewires through narrow bone corridors. We present a guidance system with a video camera mounted onboard the surgical drill to achieve real-time augmentation of the drill trajectory in fluoroscopy and/or CT.

Approach: The camera was mounted on the drill and calibrated with respect to the drill axis. Markers identifiable in both video and fluoroscopy are placed about the surgical field and co-registered by feature correspondences. If available, a preoperative CT can also be co-registered by 3D–2D image registration. Real-time guidance is achieved by virtual overlay of the registered drill axis on fluoroscopy or in CT. Performance was evaluated in terms of target registration error (TRE), conformance within clinically relevant pelvic bone corridors, and runtime.

Results: Registration of the drill axis to fluoroscopy demonstrated median TRE of 0.9 mm and 2.0 deg when solved with two views (e.g., anteroposterior and lateral) and five markers visible in both video and fluoroscopy—more than sufficient to provide Kirschner wire (K-wire) conformance within common pelvic bone corridors. Registration accuracy was reduced when solved with a single fluoroscopic view ($\mathrm{TRE}=3.4\mathrm{mm}$ and 2.7 deg) but was also sufficient for K-wire conformance within pelvic bone corridors. Registration was robust with as few as four markers visible within the field of view. Runtime of the initial implementation allowed fluoroscopy overlay and/or 3D CT navigation with freehand manipulation of the drill up to $10\text{frames}/\mathrm{s}$.

Conclusions: A drill-mounted video guidance system was developed to assist with K-wire placement. Overall workflow is compatible with fluoroscopically guided orthopaedic trauma surgery and does not require markers to be placed in preoperative CT. The initial prototype demonstrates accuracy and runtime that could improve the accuracy of K-wire placement, motivating future work for translation to clinical studies.

1. Introduction

Orthopaedic trauma is a prominent healthcare burden in terms of cost and quality of life.¹ Fractures of the pelvic ring present a particular challenge, affecting 37 out of 100,000 individuals/year (3% to 7% of skeletal fractures).^2–4 Pelvic fracture surgery involves (1) reduction of bone fragments to their natural position and (2) fixation of the reduced fracture by insertion of a guidewire [e.g., a Kirschner wire (K-wire)] along bone corridors—e.g., the pubis, ischium, and/or ilium. A cannulated screw is then delivered over the wire for fixation of the fracture, and the wire is removed.⁵

Surgical treatment increasingly involves percutaneous approach under fluoroscopy,^6–8 with intermittent exposures acquired during placement of the guidewire to assess position relative to surrounding anatomy—namely conformance within bone corridors.⁹ Surgeons qualitatively estimate the 3D position of the K-wire from multiple projection views (e.g., inlet, outlet, PA, and lateral views). However, 3D reckoning within the complex morphology of the pelvis is challenging even to experienced surgeons, and accurate K-wire placement often requires “fluoro hunting” and trial and error.^5,10 It is not uncommon for the guidewire to be completely withdrawn if the K-wire appears in danger of breaching the bone corridor and reinserted along a new trajectory, leading to extended procedure time and fluoroscopic exposure, often exceeding 120 s of fluoroscopy time and hundreds of radiographic views.⁵

Surgical navigation based on optical or electromagnetic (EM) trackers is fairly common in brain and spine surgery.^11,12 However, orthopaedic trauma surgery has not seen widespread adoption of these systems, primarily due to factors of cost, additional equipment, line of sight, and workflow bottlenecks in setup and registration. Given the steep requirements in cost and workflow of orthopaedic trauma surgery, fluoroscopic guidance has remained the mainstay for decades.

Navigation solutions that are potentially better suited to the demanding workflow of orthopaedic trauma surgery are an active area of research. For example, mounting a camera on the C-arm provides a basis for augmented fluoroscopic guidance and tool tracking.^13–15 Mounting a stereoscopic camera on the C-arm demonstrated improved accuracy and better line-of-sight compared to a conventional infrared tracking arrangement and provided stereoscopic video for augmented views of the surgical field and/or fluoroscopy.¹⁶ EM trackers (despite sensitivity to metal objects and workflow limitations similar to infrared trackers) have been similarly investigated—e.g., incorporating an open frame EM field generator into the operating table ¹⁷ for real-time tracking while compatibility with fluoroscopy through the open frame. Others have mounted tracking equipment on the instrument itself.^18–20 For example, Magaraggia et al.¹⁸ described a video guidance system involving an implant-specific drill sleeve for distal radial fracture surgery and a video camera mounted on the surgical drill. While the approach relies on the use of a custom drill sleeve, the study reported improvements in screw positioning accuracy compared to conventional fluoroscopic guidance.

In this work, we present a video-based surgical drill guidance system that is potentially suitable to the workflow of fluoroscopy-guided orthopaedic trauma procedures. The system uses a miniature camera mounted on a surgical drill for real-time visualization of the drill trajectory in fluoroscopy and/or CT. The relationship between camera images and intraoperative fluoroscopy is established via multimodal (optical and radio-opaque) fiducial markers placed about the surgical field at the time of surgery. Notably (and likely essential to realistic workflow), the markers are not required in preoperative CT. The solution couples 3D–2D registration with vision-based tracking to register the drill axis without additional equipment in the operating room (OR). The proposed solution also has the potential to reduce “fluoro hunting,” and registration can be performed with commonly acquired (e.g., inlet and outlet) views. In principle, the system could reduce the number of views required for K-wire placement from hundreds⁵ to as few as two—one for initial registration and one for visual confirmation of device placement.

The proposed system is distinct from previously reported systems in a variety of ways. Compared to the drill-mounted video system described by Magaraggia et al.,¹⁸ the system uses multimodal markers (rather than custom drill sleeves) to register video images to fluoroscopy. Furthermore, the system can register the drill axis to a preoperative CT [or intraoperative cone-beam CT (CBCT)] for 3D navigation. Other video-based systems require that markers be placed on the patient in preoperative CT.^21–25 The need to place markers in preoperative imaging presents a workflow barrier that is not compatible with emergent trauma, where the preoperative CT is acquired quickly for many diagnostic purposes (e.g., to rule-out hemorrhage, detect and characterize the fracture, etc.). Such systems also require the markers to remain unperturbed throughout the case. The proposed system allows the markers to be placed during the procedure (e.g., after the patient is draped, immediately prior to K-wire insertion), and perturbations of the markers are accommodated by updating the registration with as little as one fluoroscopic view.

The following sections detail the video-guided drill system, the video-to-fluoroscopy and fluoroscopy-to-CT registration methods, and integration with two modes of guidance—“video-fluoro” and “video-CT.” Experiments quantify the geometric accuracy and runtime in placing K-wires within bone corridors in a phantom study. We conclude with a discussion of improvements in future design iterations that will support translation to clinical studies.

2. Materials and Methods

2.1. System for Surgical Drill Guidance Using Computer Vision and 3D–2D Registration

Figure 1 summarizes the high-level workflow of the drill guidance system. As detailed below: if a preoperative CT is not available, the system enables video-fluoro navigation; and if a preoperative CT is available, the system enables both video-fluoro and/or video-CT navigation. The system assumes two calibrations [Fig. 1(b)]: camera calibration (a one-time step to correct lens distortion) and drill axis calibration (a relatively quick step that could be performed offline or in the OR if the camera mount is swapped between drills).

Fig. 1.

Workflow and algorithms for surgical guidance with a drill-mounted video camera. (a) If available, preoperative CT is registered to the patient (dashed lines) via 3D–2D registration to intraoperative fluoroscopy, allowing video-CT guidance. (b) Offline calibration of the camera and drill axis. (c), (d) Intraoperative registration: (c) 3D–2D registration to localize marker poses in C-arm fluoroscopy and register preoperative CT to the patient; and (d) a continuous, real-time loop that registers the drill axis to fluoroscopy and/or CT via the video scene.

Intraoperative workflow includes two main steps. The first [Fig. 1(c)] is to place multimodal markers about the surgical field such that at least three are in the fluoroscopy field of view (FOV). It is worthy to note that the markers need not be rigidly affixed with respect to each other or the patient, since the registration is updated with each fluoroscopic view. The drill itself does not need to be visible in the fluoroscopic view; rather, the drill-mounted camera only need have clear view of the markers as present in the fluoroscopic image. The second step [Fig. 1(d)] involves registration of the drill trajectory to the fluoroscopy image (and/or optionally to CT) via the video scene. This registration can be computed continuously in real time, noting that the overlaid trajectory shows the drill axis (not the K-wire tip). Such guidance may therefore be most useful in the challenging initial step of finding an acceptable entry point and trajectory orientation.

Table 1 summarizes relevant notation, with 3D vectors denoted in uppercase and 2D vectors denoted in lowercase. Throughout this work, the hat symbol denotes an estimate (subject to measurement error) of the given quantity. Figure 2(a) illustrates the coordinate systems and transforms.

Table 1.

Summary of notation for video-based surgical drill guidance. Coordinate frames: $D$ (for drill camera), $C$ (for the C-arm), and $V$ (for preoperative CT volume). Multimodal markers ($m=1,\dots ,M$) are co-registered between the video frame, one or more fluoroscopic images at view angles ($\theta$), and the CT volume via 3D–2D registration.

System parameters
$\theta$	Projection view angle for a C-arm fluoroscopy frame
$m$	Index for a particular multimodal marker ($m$: $1\dots M$)
$\kappa$	3D mesh representation of radio-opaque marker features
$\mu$	Preoperative CT volume
Drill camera coordinate frame ($D$)
$K^D$	Drill camera calibration matrix
${\hat{X}}_D^{(m)}$	3D feature point estimate for marker $m$ in drill camera 3D coordinate frame
$x_D^{(m)}$	2D feature point for marker $m$ in drill camera 2D image plane
${\hat{\mathrm{P}}}_D^{(m)}$	Pose estimate of marker $m$ in the drill camera 3D coordinate frame
${\stackrel{\rightharpoonup }{L}}_D$	Drill axis in drill camera 3D coordinate frame
C-arm coordinate frame ($C$)
${\mathcal{P}}^C(\theta )$	C-arm projection matrix for fluoroscopy frame at view angle
${\widehat{X}}_C^{(m)}$	3D feature point estimate for marker $m$ in the C-arm 3D coordinate frame
$x_C^{(m,\theta )}$	2D feature point for marker $m$ in the C-arm 2D image plane for a fluoroscopy frame at view angle $\theta$
${\widehat{P}}_C^{(m)}$	Pose estimate of marker $m$ in the C-arm 3D coordinate frame
${\widehat{T}}_C^D$	Estimated transformation from the drill camera to C-arm 3D coordinate frames
${\stackrel{\rightharpoonup }{L}}_C$	Drill axis in the C-arm 3D coordinate frame
${\stackrel{\rightharpoonup }{l}}_C$	Drill axis projected to the C-arm 2D image plane (i.e., 2D fluoroscopy frame)
Preoperative CT coordinate frame ($V$)
${\widehat{T}}_C^V$	Estimated transformation from the preoperative CT to C-arm 3D coordinate frames
${\stackrel{\rightharpoonup }{L}}_V$	Drill axis in the preoperative CT coordinate frame

Fig. 2.

Coordinate frames and multimodal marker design. (a) Coordinate frames for the video-guided drill setup, including the drill camera ($D$) and C-arm ($C$). The zoomed-in view shows the drill camera (and frame $D$) in relation to the drill axis (${\stackrel{\rightharpoonup }{L}}_D$). (b) Schematic of multimodal marker features visible in both video (center of the ArUco tag) and fluoroscopy (central BB). Additional BBs and wires encode individual markers and provide a basis for 3D–2D registration between the marker design ($\kappa$) and fluoroscopy.

2.2. Multimodal Markers

2.2.1. Marker design

Previous work used “multimodal” markers visible optically and radiographically to relate surgical trackers to fluoroscopic images—for example, Hamming et al.²⁶ and Dang et al.²⁷ Figure 2(b) shows an exploded view of multimodal markers to relate the drill camera and C-arm frames via optical and radio-opaque features. Optical features were presented by ArUco marker tags²⁸ consisting of a $6\times 6$ inner matrix of bits to uniquely identify each marker. Each marker contained a central ball bearing (BB) (tungsten, $Ø2.0\mathrm{mm}$) and a unique arrangement of outer BBs differing in size ($Ø2.0$ or 3.5 mm) and location. To aid feature extraction in fluoroscopy, BBs were constrained to be collinear with at least two other BBs. Each marker was encircled by a tungsten wire ($Ø0.8\mathrm{mm}$) to create a 48-mm periphery that assisted in marker detection (by Hough transform) and pose estimation (circle-to-ellipse perspective relationship).

The marker base was 3D-printed (Vero PureWhite, Connex-3 Objet 260, Stratasys, Eden Prairie Minnesota) with pockets to hold the BBs, a peripheral groove to hold the wire, and a $30\times 30{\mathrm{mm}}^2$ square recess (2-mm deep) in which the ArUco tag was placed such that its center coincides with the central BB. The depth of the recess introduces a small offset between the ArUco tag center and central BB along the axis normal to the marker. Since this offset is known, it can be corrected prior to registration (discussed in Sec. 2.4). The initial design has $Ø50\mathrm{mm}$ and can generate up to 48 unique markers, with future work to investigate more compact designs.

2.2.2. Marker detection

The detection of ArUco tags in video was based on open-source tools available in OpenCV.^29,30 The algorithm first performs adaptive thresholding and contour detection of grayscale images to isolate candidate regions for multiple tags. The inner area of each candidate is analyzed by correcting the tag perspective to a square region and then binarizing the resulting region to a regularly spaced grid upon which marker identification can be performed.

Marker detection in the fluoroscopic scene was performed first by ellipse detection (using the peripheral wire) to coarsely identify individual marker positions. A Canny edge filter was followed by morphological closing to yield binarized elliptical contours and filter out smaller objects (e.g., BBs). A Hough-based ellipse detector³¹ returned elliptical fits ordered by accumulator score, which was cutoff according to the known number of markers ($M$). Within each ellipse, a morphological top-hat filter isolated marker features from surrounding anatomy, and the ellipse was removed by morphological opening to isolate the BB features. Hough-based circle detection was used to identify the position and radius of BBs in each marker, and candidate BBs were filtered based on the known range of BB radii. Detections within a certain proximity to each other were also filtered based on the known marker designs, and collinearity was enforced to eliminate any remaining false positives. The resulting BB detections were then hierarchically clustered in two groups according to size, and markers were uniquely identified according to a lookup table.

2.3. Camera and Drill Axis Calibration

Camera calibration determined the intrinsic parameters of the camera (pinhole model) and distortion coefficients of the lens. Calibration was performed as in the work by Zhang³² using multiple images of a planar checkboard to obtain a closed-form solution for the camera matrix ($K^D$). Lens distortion was corrected using the Brown–Conrady even-order polynomial model,³² describing both radial and tangential distortion.

The orientation of the drill axis in the camera coordinate frame (${\stackrel{\rightharpoonup }{L}}_D$) was solved using a calibration jig described in previous work.³³ The jig consisted of a drill sleeve centered on an ArUco board. As the jig (drill sleeve) rotates about the drill axis, the pose of the ArUco board in the 3D frame of the camera was estimated in multiple images yielding a 3D cylindrical point cloud. A RANSAC-based cylindrical fit (MLESAC³⁴) was computed to obtain the central axis of the point cloud—i.e., the drill axis (${\stackrel{\rightharpoonup }{L}}_D$).

2.4. Video-to-Fluoroscopy Registration

Video images were registered to the fluoroscopic scene through localization and registration of the point-based feature correspondences of markers discussed in Sec. 2.2. For the ArUco tags, the 3D pose of each marker in the drill camera coordinate frame [${\hat{P}}_D^{(m)}$] was estimated with the perspective-N-points algorithm for $m=1,\dots ,M$ markers. The translational component of the resulting pose estimate was extracted and represents the center of the ArUco tag. To correct for the offset between the tag center and central BB mentioned in Sec. 2.2.1, the translational component of the pose estimate was adjusted along the axis normal to the surface of the marker [third column of the rotational component of ${\widehat{P}}_D^{(m)}$] by the known offset (2 mm). The resulting estimate represents the central marker feature point in the drill camera frame $[{\widehat{X}}_D^{(m)}]$ corresponding to the location of the central BB.

A mobile C-arm (Cios Spin, Siemens Healthineers, Forchheim, Germany) was used in all studies described below. The projective relationship (${\mathcal{P}}^C$) relating 3D points in the C-arm frame ($X_C$) to 2D fluoroscopic image points on the C-arm detector plane ($x_C$) (in homogenous coordinates) was determined by standard C-arm geometry calibration,³⁵ with the projection matrix defined by

$${\mathcal{P}}^C=K^C[\begin{array}{cc}{\mathcal{R}}_{\theta }& {\mathcal{T}}_{\theta }\end{array}].$$ (1)

The intrinsic matrix $K^C$ describes the geometric relationship between the C-arm source and detector. The extrinsic parameters ($[\begin{array}{cc}{\mathcal{R}}_{\theta }& {\mathcal{T}}_{\theta }\end{array}]$) describe the pose of the C-arm source–detector assembly for a fluoroscopy frame at view angle $\theta$ in a common coordinate frame, referred to as the C-arm coordinate frame $C$.

To extract marker pose from fluoroscopic images, 3D–2D “known-component” registration (KC-Reg) was performed³⁶ to register markers according to the known 3D design ($\kappa$) of each marker. The general framework for KC-Reg is shown in Fig. 3(a). A generic mesh model ($\kappa$) representing the radio-opaque features of the marker is iteratively transformed to optimize the image similarity between digitally reconstructed radiographs (${\mathrm{DRR}}_{\kappa }$) and one or more intraoperative fluoroscopic images ($I_{\theta }$). Fluoroscopic views were selected such that markers were not overlapping in projection data. 3D–2D registration of the fluoroscopy image $(I_{\theta })$ and the corresponding DRR was computed by optimization of gradient information (GI), which has been shown to be robust in a variety of realistic fluoroscopy scenarios.^37,38 To solve for the pose of marker $m$, the cumulative GI across $\theta =1,\dots ,N_{\text{view}}$ fluoroscopic views was maximized:

$${\widehat{P}}_C^{(m)}=\arg \underset{P_C^{(m)}}{\max }\sum _{\theta }\mathrm{GI}(I_{\theta },{\int }_{{\stackrel{\rightharpoonup }{r}}_{\theta }}{P}_C^{(m)}(\kappa )\mathrm{d}{\stackrel{\rightharpoonup }{r}}_{\theta }),$$ (2)

where the integral represents the DRR at view $\theta$. Optimization was performed using the covariance matrix adaptive evolution strategy (CMA-ES) algorithm³⁹ initialized using features extracted during marker detection (Sec. 2.2.2). For each multimodal marker $m$, an initial estimate of the central 3D feature point in the C-arm frame $[{\hat{X}}_C^{(m)}]$ was obtained by first backprojecting a ray from the corresponding 2D image feature point in homogenous coordinates [denoted ${\mathit{x}}_C^{(m,\theta )}$] toward the x-ray source. The backprojected ray was estimated for each fluoroscopic view $\theta$, from which an initial estimate of the 3D feature point could be reconstructed by

$${\widehat{X}}_C^{(m)}=\frac{1}{N_{\text{view}}}\sum _{\theta =1}^{N_{\text{view}}}\frac{{\lambda }^{(m,\theta )}{(K^C{\mathcal{R}}_{\theta })}^{-1}\begin{array}{c}{\mathit{x}}_C^{(m,\theta )}\end{array}}{\Vert {(K^C{\mathcal{R}}_{\theta })}^{-1}\begin{array}{c}{\mathit{x}}_C^{(m,\theta )}\end{array}\Vert }+(-{\mathcal{R}}_{\theta }^{-1}{\mathcal{T}}_{\theta }),$$ (3a)

where

$${\lambda }^{(m,\theta )}=\frac{\mathrm{SDD}}{{\mathcal{M}}^{(m,\theta )}}$$ (3b)

describes the estimated distance along the backprojected source–detector ray. The magnification $\mathcal{M}$ was estimated for each marker $m$ in each fluoroscopic view using the perspective relationship between the diameter of a circle and the major axis length of its elliptical projection. Once the 3D position of each marker was initialized, a rotational initialization was performed with a planar fit of the global marker arrangement. The computed plane normal was used as an initial estimate of the out-of-plane axis for each marker.

Fig. 3.

3D–2D registration of intraoperative fluoroscopy to (a) multimodal markers and (b) preoperative CT.

With corresponding point estimates obtained in both the drill camera ($D$) and C-arm ($C$) coordinate frames, a transformation between the two was estimated using point-based registration.⁴⁰ The resulting video-to-fluoroscopy transform (${\hat{T}}_C^D$) was used to represent the surgical drill axis in the C-arm coordinate frame:

$${\stackrel{\rightharpoonup }{L}}_C={\widehat{T}}_C^D{\stackrel{\rightharpoonup }{L}}_D$$ (4a)

and its projection on the C-arm detector plane:

$${\stackrel{\rightharpoonup }{l}}_C={\mathcal{P}}^C{\stackrel{\rightharpoonup }{L}}_C.$$ (4b)

Augmentation of fluoroscopic views with ${\stackrel{\rightharpoonup }{l}}_C$ realizes the video-fluoro navigation mode. For drill guidance, marker localization in the camera image is continuously updated to allow freehand motion of the drill. In the event that the markers are perturbed relative to anatomy (or if the surgeon desires overlay in different fluoroscopic view), then the video-to-fluoroscopy registration is updated with the acquisition of one or more fluoroscopy views.

2.5. Fluoroscopy-to-CT Registration

Intraoperative fluoroscopy was used to register the patient to a preoperative CT (or intraoperative CBCT) volume using 3D–2D registration.^38,41 The workflow for patient registration is shown in Fig. 3(b). The CT volume ($\mu$) is iteratively transformed to optimize similarity between fluoroscopic images ($I_{\theta }$) and forward projections of the rigidly transformed CT (${\mathrm{DRR}}_{\mu }$). An objective function similar to Eq. (2) was solved using CMA-ES with gradient correlation (GC) as the similarity metric. GC was used for CT registration because it is independent of absolute gradient magnitudes and is robust for images in which corresponding anatomical gradients may differ due to differences in imaging technique or mismatch in image content (e.g., tools in the fluoroscopic scene that are not in the CT).⁴¹

The resulting CT-to-fluoroscopy transform (${\widehat{T}}_C^V$) was used to transform the surgical drill axis into the preoperative CT coordinate frame as

$${\stackrel{\rightharpoonup }{L}}_V={({\widehat{T}}_C^V)}^{-1}{\widehat{T}}_C^D{\stackrel{\rightharpoonup }{L}}_D.$$ (5)

Augmentation of the CT image (e.g., orthogonal CT slices, MIPs, or volume renderings) realizes the video-CT navigation mode.

2.6. Experimental Evaluation

2.6.1. Experimental setup

The performance of the video-guided drill system was evaluated in terms of the accuracy of drill axis registration and guidance along K-wire trajectories common in pelvic trauma surgery. The imaging setup is illustrated in Fig. 4(a), consisting of a mobile C-arm, the video-guided drill, and an anthropomorphic pelvis phantom. The mobile C-arm was used to acquire fluoroscopic images and CBCT (for truth definition). CBCT images were acquired with 400 projections over a 195-deg semi-circular orbit and reconstructed on a $0.512\times 0.512\times 0.512{\mathrm{mm}}^3$ voxel grid with a standard bone kernel. Figure 4(b) shows the initial prototype for the drill guidance system, consisting of a Stryker System 6 (Kalamazoo, Michigan) surgical drill with a pin collet for 3–mm-diameter K-wires and a Logitech C900 USB webcam. The camera was rigidly attached to the drill body using a custom 3D-printed chassis that positioned the camera with clear view of the drill axis and surgical scene. Experiments were performed in an OR laboratory with lighting conditions that approximated realistic ambient lighting with or without an overhead OR light. To assist in experimental workflow and provide a stable platform for experiments, a UR3e robotic arm (Universal Robotics, Odense, Denmark) was used as a drill holder throughout the experiments. The robot was for convenience in experiments, is not required for the envisioned freehand system, and does not affect the accuracy of the registration and guidance methods.

Fig. 4.

Experimental setup for video-based drill guidance. (a) Imaging setup showing the C-arm, phantom A, and video-drill held by a robotic positioner. A similar setup was created for phantom B (not shown). (b) Initial prototype of the drill camera and markers. The markers are placed about the entry for the AIIS-to-PSIS trajectory (shown in green).

The system was evaluated in two phantom experiments. The first assessed the accuracy of individual registration steps and the end-to-end accuracy in guiding the initial entry point and orientation of the drill. The effect of relevant experimental parameters (viz., the number of markers, $M$, and the number of fluoroscopic views, $N_{\text{view}}$) on drill registration accuracy was also investigated. The second experiment assessed accuracy during K-wire insertion along multiple pelvic trajectories, including potential mechanical sources of error (e.g., bending of the K-wire during drilling).

For the first set of experiments, an anthropomorphic pelvis phantom composed of a natural adult skeleton in tissue-equivalent plastic (Rando^®, The Phantom Lab, Greenwich, New York) was used (referred to as “phantom A”). The drill camera was aligned with respect to the anterior inferior iliac spine (AIIS) to posterior superior iliac spine (PSIS) trajectory in the left hip. Five markers were placed about the planned entry site. The robotic arm was used to position the drill camera at a distance of $\sim 20\mathrm{cm}$ from the surface of phantom A to emulate realistic surgical drill positioning. The drill camera was also placed at nine different positions about the planned trajectory to measure the sensitivity of registration accuracy to various camera perspectives. At each camera position, a single camera image was acquired from which the position of each marker was determined. All other experimental parameters (and specifically, the number of markers) were held constant, varying only the position of the camera. An initial CBCT scan (110 kV, 350 mAs) was acquired with only the phantom A and markers in the FOV. Projections in which markers were not overlapping (drawn from the scan arc from $\theta =-20\mathrm{deg}$ to 70 deg) were selected as views for solving 3D–2D registration. The CBCT defined the true position of the BBs. Fluoroscopic views (100 kV, $\sim 0.9\mathrm{mAs}$) were collected at common clinical orientations for augmentation in video-fluoro mode, including anteroposterior view ($\theta =0\mathrm{deg}$, $\varphi =0\mathrm{deg}$), lateral view ($\theta =90\mathrm{deg}$, $\varphi =0\mathrm{deg}$), inlet view ($\theta =0\mathrm{deg}$, $\varphi =-25\mathrm{deg}$), and outlet view ($\theta =0\mathrm{deg}$, $\varphi =30\mathrm{deg}$). The drill camera was then positioned with a 3-mm K-wire extending from the drill tip to the surface of phantom A, and video images of the marker arrangement were collected. A final CBCT scan (110 kV, 380 mAs) was acquired with the K-wire in the FOV for truth definition of the drill axis (determined by threshold-based segmentation of the K-wire followed by principal component analysis of the segmented voxels to extract the drill axis). This process was robust to metal artifacts about the K-wire. In total, two CBCT scans and four radiographs were acquired for experiments with phantom A.

For evaluation of the video-CT navigation mode, a preoperative CT ($0.82\times 0.82\times 0.5{\mathrm{mm}}^3\text{voxel}$ grid) of phantom A was acquired (SOMATOM Definition, Siemens, Erlangen Germany). Pelvic K-wire trajectories were planned in preoperative CT using the atlas-based planning method in Goerres et al.⁴² and Han et al.⁴³ Acceptance volumes interior to bone corridors were created for three common K-wire trajectories: AIIS-to-PSIS, superior ramus (SR), and iliac crest to posterior column (PC). Acceptance volumes were used to visualize and evaluate drill axis conformance within pelvic bone corridors.

For the second set of experiments, an anthropomorphic (left) hemi-pelvis phantom composed of radiopaque material (Sawbones^®, Pacific Research Laboratories Inc., Vashon, Washington) was used (referred to as “phantom B”) since phantom A did not permit drilling. Due to the simple nature of phantom B (lack of surrounding bulk anatomy), an additional 3.2-mm Cu filtration ($\sim 15\text{-}\mathrm{cm}$ water equivalent) was added at the x-ray source to provide realistic exposure levels and quantum noise in the fluoroscopic images. System performance during K-wire insertion was evaluated for the same three pelvic trajectories (AIIS-to-PIIS, PC, and SR). For each trajectory, five markers were placed about the planned entry site. The video-drill system was then used to insert a K-wire (3-mm diameter) along each trajectory. Data were collected at three positions along each path, referred to as: (1) “entry,” acquired at $\sim 0$ to 5 mm along the trajectory; (2) “midpoint,” acquired at $\sim 40$ to 60 mm along the trajectory; and (3) “endpoint,” acquired at $\sim 80$ to 120 mm along the trajectory. At each position, video images of the marker arrangement were collected, and a CBCT scan (110 kV, 55 mAs) was acquired. Projections from the scan were selected (differing in view angle by up to 30 deg) for solving 3D–2D registration, and CBCT images defined ground truth of the K-wire drill axis as described above. In total, nine CBCT scans (three scans for each trajectory) were acquired for experiments with phantom B.

2.6.2. Evaluation of system accuracy

Table 2 summarizes the figures of merit for assessing the accuracy of individual registration steps and the overall system. The performance of video-to-fluoroscopy registration was quantified in terms of errors related to 3D marker localization in the drill camera ($D$) and C-arm ($C$) coordinate frames. The positional estimate for each marker in the camera frame $[{\hat{X}}_D^{(m)}]$ was evaluated with respect to the truth definition (points defined in CBCT) and quantified in terms of fiducial registration error (FRE):

$$\mathrm{FRE}=\Vert {\widehat{T}}_C^D{\widehat{X}}_D^{(m)}-X_C^{(m,\text{true})}\Vert ,$$ (6a)

where $X_C^{(m,\text{true})}$ represents the true location of marker $m$ in the C-arm frame and ${\hat{T}}_C^D$ is an estimate of the true camera-to-C-arm transform derived from point-based registration with all markers. The term ${\widehat{T}}_C^D{\widehat{X}}_D^{(m)}$ is therefore the estimated location of marker $m$ in the C-arm frame. Fiducial errors were further decomposed into in-plane and depth errors with respect to the drill camera coordinate frame (${\mathrm{\Delta }}_D$) as

$${\mathrm{\Delta }}_D^{(m)}={\mathcal{R}}_D^C({\widehat{T}}_C^D{\widehat{X}}_D^{(m)}-X_C^{(m,\text{true})}).$$ (6b)

Estimation of ${\mathcal{R}}_D^C$ (the rotation matrix from the C-arm to the drill camera coordinate frame) was performed independently of the fiducials by solving the rotation between the calibrated drill axis (${\stackrel{\rightharpoonup }{L}}_D$) and the true drill axis segmented from CBCT [denoted as ${\stackrel{\rightharpoonup }{L}}_C^{(\text{true})}$].

Table 2.

Figures of merit for evaluating registration accuracy for individual registration methods and end-to-end system performance.

Video-to-fluoroscopy registration
${\mathrm{FRE}}^{(m)}$	Norm of error in 3D localization of marker $m$ from video images
${\mathrm{\Delta }}_D^{(m)}$	In-plane and out-of-plane translational error components in 3D localization of marker $m$ from video images
${\delta }_C^{(m)}$	Norm of error in 3D localization of marker $m$ from fluoroscopic images
${\mathrm{\Delta }}_C^{(m)}$	In-plane and out-of-plane translational error components in 3D localization of marker $m$ from fluoroscopic images
Fluoroscopy-to-CT registration
${\mathrm{\Gamma }}_V$	Fluoroscopy-to-CT registration difference transform
${\mathrm{\Gamma }}_V^{\mathrm{\Delta }}$	In-plane and out-of-plane translational error components in fluoroscopy-to-CT registration
${\mathrm{\Gamma }}_V^{\mathrm{\Phi }}$	In-plane and out-of-plane rotational error components in fluoroscopy-to-CT registration
${\delta }_V$	Norm of translational error in fluoroscopy-to-CT registration
Drill axis registration
${\mathrm{TRE}}_x$	Norm of translational error between computed and true drill axes
${\mathrm{TRE}}_{\phi }$	Angular skew between computed and true drill axes

Marker localization errors in the C-arm frame ($C$) were estimated relative to the true marker locations [$X_C^{(m,\text{true})}$]. The 3D–2D registration of each marker was solved using $N_{\text{view}}=1-3$ fluoroscopic views selected to span a total arc of 30 deg with equiangular spacing. Accuracy was assessed in terms of the norm of the translational error (${\delta }_C$):

$${\delta }_C^{(m)}=\Vert {\widehat{X}}_C^{(m)}-X_C^{(m,\text{true})}\Vert .$$ (6c)

Translational errors were further examined with respect to in-plane (parallel to the detector plane) and out-of-plane (depth) components (referred to as ${\mathrm{\Delta }}_C$).

Fluoroscopy-to-CT registration was evaluated over the same set of fluoroscopic views used during marker localization for $N_{\text{view}}=1$ and 2 (over a 30-deg arc). To evaluate accuracy, truth was defined from a large number ($N_{\text{view}}=15$) of fluoroscopic views to solve the true 3D–2D patient registration (referred to as ${\mathbb{T}}_C^V$), selecting distinct views for registration and truth definition to mitigate bias. Performance was calculated in terms of the difference transform (${\mathrm{\Gamma }}_V$) between the registration result and the truth definition by

$${\mathrm{\Gamma }}_V={\hat{T}}_C^V{({\mathbb{T}}_C^V)}^{-1}.$$ (7)

The difference transform was further decomposed into translational (${\mathrm{\Gamma }}_V^{\mathrm{\Delta }}$) and rotational error components (${\mathrm{\Gamma }}_V^{\mathrm{\Phi }}$), from which the norm of the translational error (${\delta }_V$) was also computed.

End-to-end system performance was evaluated by comparison of the computed drill axis (${\stackrel{\rightharpoonup }{L}}_C$) [Eq. (4a)] with the true drill axis from CBCT $[{\stackrel{\rightharpoonup }{L}}_C^{(\text{true})}]$. Taking the true axis as reference, possible mismatch from the automatically planned trajectory was corrected by applying a transformation to the computed and true drill axes. This transformation places the drill within the context of the bone corridor, as would be done in clinical use, while preserving the relative errors between the computed and true drill trajectories. The error between the computed and true drill axes was reported in terms of target registration error (TRE), separated into positional (${\mathrm{TRE}}_x$) and angular (${\mathrm{TRE}}_{\phi }$) errors by decomposing axes into translational (${\tau }_C$) and rotational (${\rho }_C$) components:

$${\mathrm{TRE}}_x=\Vert {\hat{\tau }}_C-{\tau }_C^{(\text{true})}\Vert ,$$ (8a)

$${\mathrm{TRE}}_{\phi }={\cos }^{-1}\frac{{\hat{\rho }}_C·{\rho }_C^{(\text{true})}}{\Vert {\hat{\rho }}_C·{\rho }_C^{(\text{true})}\Vert },$$ (8b)

where the hat symbol here denotes components of the computed drill axis. The rotational components (${\rho }_C$) are given by a vector describing the direction of the drill axis, and the translational components (${\tau }_C$) describe a point along the drill axis. The translational components (${\tau }_C$) were selected to evaluate ${\mathrm{TRE}}_x$ with respect to the bone corridor. In experiments with phantom A, the target location (${\tau }_C$) corresponded to the bone corridor entry point. Since the drill was placed at the surface of the phantom (rather than the surface of the bone corridor), the target locations (${\tau }_C$) were calculated as the intersection of the computed and true drill axes with the surface of the planned acceptance volume. In experiments with phantom B, the target location (${\tau }_C$) corresponded to the K-wire tip position as the K-wire was physically inserted into the bone corridor. Registration errors reported were computed in the CT coordinate frame ($V$). The truth trajectory in the CT coordinate frame was estimated using the fluoroscopy-to-CT truth definition $[{\stackrel{\rightharpoonup }{L}}_V^{(\text{true})}={({\mathbb{T}}_C^V)}^{-1}{\stackrel{\rightharpoonup }{L}}_C^{(\text{true})}]$.

To evaluate the conformance of the drill axis trajectories within pelvic bone corridors, a mesh of the planned acceptance volume was created for each corridor to represent the cortical bone surface. The acceptability of a resulting trajectory was measured by first computing the entry and exit point at which the given trajectory intersects the cortical bone surface. The resulting path from entry to exit point was equidistantly sampled along the trajectory, and the distance from each sample to the nearest cortical bone surface point was calculated.

3. Results

3.1. Accuracy of Video-to-Fluoroscopy Registration

The accuracy of 3D marker localization from video images [FRE in Eq. (6a)] was evaluated over all ($M=5$) markers to examine how localization errors in the video scene translate to errors in fiducial registration. Among the nine different camera positions measured, the registration performance exhibited considerable variation between markers ($\text{interquartile}\text{range}(\mathrm{IQR})\sim 3.5\mathrm{mm}$) with FRE up to 5 mm in some cases due to poor marker corner detection and pose estimation. The median FRE, however, was fairly consistent across all camera poses, with median error of 1.5 mm (IQR = 0.42 mm). Localization errors with respect to the drill camera coordinate frame [${\mathrm{\Delta }}_D$ in Eq. (6b)] were decomposed in terms of in-plane (parallel to the camera image plane) and out-of-plane (perpendicular to the camera plane) translational components for the planned trajectory. In-plane translations exhibited median error of 0.46 mm ($\mathrm{IQR}=0.52\mathrm{mm}$), and median out-of-plane error was 1.6 mm ($\mathrm{IQR}=1.2\mathrm{mm}$). Overall, the results demonstrate consistent FRE across the sampled trajectories, suggesting reasonable robustness in localizing markers from video images from a broad variety of camera poses. Such robustness is important as the surgeon maneuvers the drill about the scene to align with the planned trajectory. The out-of-plane errors reflect challenges in resolving depth with a monocular camera, addressed in future work incorporating a stereoscopic camera system.

Table 3 summarizes the errors associated with 3D marker localization from fluoroscopic images [${\delta }_C$ in Eq. (6c)] pooled over all ($M=5$) markers. The localization error was notably higher for registrations solved with a single fluoroscopic view, with median error of 1.8 mm ($\mathrm{IQR}=1.7\mathrm{mm}$). The main directional component of such errors was out-of-plane (i.e., along the x-ray source to detector direction), again illustrating the challenge to depth resolution from a single perspective.

Table 3.

Accuracy of individual registration methods. Values shown are the median (IQR) in registration error across 60 views.

Video-to-fluoroscopy registration
	$N_{\text{view}}=1$	$N_{\text{view}}=2$	$N_{\text{view}}=3$
${\delta }_C$	1.80 mm ($\mathrm{IQR}=1.7\mathrm{mm}$)	0.67 mm ($\mathrm{IQR}=0.25\mathrm{mm}$)	0.66 mm ($\mathrm{IQR}=0.20\mathrm{mm}$)
${\mathrm{\Delta }}_C({\mathrm{x}}_C)$	0.27 mm (0.30 mm)	0.36 mm (0.33 mm)	0.36 mm (0.33 mm)
${\mathrm{\Delta }}_C({\mathrm{y}}_C)$	0.44 mm (0.51 mm)	0.25 mm (0.28 mm)	0.24 mm (0.25 mm)
${\mathrm{\Delta }}_C({\mathrm{z}}_C)$	1.60 mm (1.8 mm)	0.50 mm (0.53 mm)	0.65 mm (0.20 mm)
Fluoroscopy-to-CT registration
	$N_{\text{view}}=1$		$N_{\text{view}}=2$
${\delta }_V$	0.41 mm ($\mathrm{IQR}=0.30\mathrm{mm}$)		0.42 mm ($\mathrm{IQR}=0.30\mathrm{mm}$)
${\mathrm{\Gamma }}_V^{\mathrm{\Delta }}({\mathrm{x}}_V)$	0.11 mm (0.16 mm)		0.13 mm (0.12 mm)
${\mathrm{\Gamma }}_V^{\mathrm{\Delta }}({\mathrm{y}}_V)$	0.12 mm (0.15 mm)		0.10 mm (0.09 mm)
${\mathrm{\Gamma }}_V^{\mathrm{\Delta }}({\mathrm{z}}_V)$	0.39 mm (0.40 mm)		0.32 mm (0.31 mm)

3.2. Accuracy of Fluoroscopy-to-CT Registration

Fluoroscopy-to-CT registration accuracy was computed in terms of the difference in true and estimated 3D transformations [${\mathrm{\Gamma }}_V$ in Eq. (7)]. As shown in Table 3, the translational error norm (${\delta }_V$) for registrations solved with $N_{\text{view}}=1$ and $N_{\text{view}}=2$ was similar, with median error of 0.4 mm (0.3-mm IQR) in both cases, noting a few outliers (up to $\sim 2\text{-}\mathrm{mm}$ error) when $N_{\text{view}}=1$. The errors were further separated into translational errors and rotational errors (not shown for brevity) relative to the detector plane. For the $N_{\text{view}}=1$ case, median in-plane translational error $(x_V,y_V)$ was 0.12 mm ($\mathrm{IQR}=0.15\mathrm{mm}$), and out-of-plane error $(z_V)$ was slightly higher (median of 0.39 mm, $\mathrm{IQR}=0.40\mathrm{mm}$) and accounted for the outliers observed. In-plane and out-of-plane rotational errors were consistently low (median of 0.07 deg and 0.03 deg, respectively).

3.3. Accuracy of Drill Axis Registration

Figure 5 shows positional and rotational TRE [${\mathrm{TRE}}_x$ and ${\mathrm{TRE}}_{\phi }$ in Eq. (8)] for the AIIS-to-PSIS trajectory in phantom A evaluated using $N_{\text{view}}=1$ or 2 fluoroscopic images with a variable number of markers, $M$. For $M=3$ or 4, each possible subset among the five markers was evaluated and pooled as a single distribution. As expected, overall performance diminished with fewer markers, with end-to-end performance for $M=3$ markers yielding median ${\mathrm{TRE}}_x=4.7$ to 6.9 mm and ${\mathrm{TRE}}_{\phi }=3.7\mathrm{deg}$ to 6.9 deg for $N_{\text{view}}=1$ or 2 with an unacceptably high rate of outliers. For $M=4$ markers, registration from $N_{\text{view}}=1$ was still unacceptably high (median ${\mathrm{TRE}}_x=4.2\mathrm{mm}$ and ${\mathrm{TRE}}_{\phi }=3.8\mathrm{deg}$) and improved considerably with $N_{\text{view}}=2$ (median ${\mathrm{TRE}}_x=2.2\mathrm{mm}$ and ${\mathrm{TRE}}_{\phi }=2.8\mathrm{deg}$ with no outliers). For $M=5$, the end-to-end system performance with a single fluoroscopic view ($N_{\text{view}}=1$) yielded median ${\mathrm{TRE}}_x=3.4\mathrm{mm}$ (1.9-mm IQR) and ${\mathrm{TRE}}_{\phi }=2.7\mathrm{deg}$ (0.79-deg IQR), attributed primarily to out-of-plane error in marker localization in fluoroscopic images. For $M=5$ markers and $N_{\text{view}}=2$, the end-to-end accuracy improved to median ${\mathrm{TRE}}_x=0.88\mathrm{mm}$ (0.16-mm IQR) and ${\mathrm{TRE}}_{\phi }=2.0\mathrm{deg}$ (0.16-deg IQR).

Fig. 5.

Registration performance for end-to-end drill axis localization in the preoperative CT coordinate frame ($V$). Registration error was assessed as a function of the number of markers in terms of (a) translational error (${\mathrm{TRE}}_x$) at the entry point of the planned bone corridor and (b) rotational error (${\mathrm{TRE}}_{\phi }$) between the computed and true drill axes. The outlier rate (fraction of measurements greater than 10 mm or 10 deg) is shown above each violin plot.

The accuracy of drill axis registration was further assessed (for $M=5$ markers) to analyze and visualize how the measured registration errors relate to conformance of the drill trajectory within bone corridors of the pelvis. A set of simulated drill axis registrations were computed by perturbing the planned trajectory according to a uniform distribution given by the median ${\mathrm{TRE}}_x$ and ${\mathrm{TRE}}_{\varphi }$ (Fig. 5), generating a “cone” of perturbed registrations about the plan. Figure 6 shows the conformance analysis for three planned trajectories. In each case, the dispersion of simulated trajectories is shown relative to a volumetric surface rendering of phantom A, with the acceptance corridor (region interior to bone cortex) shown in green and the range of simulated trajectories shown in blue and pink for $N_{\text{view}}=1$ and 2, respectively.

Fig. 6.

Analysis and visualization of drill axis trajectories within the bone corridors of the (a) AIIS-to-PSIS, (b) PC, and (c) SR trajectories for $N_{\text{view}}=1$ and $N_{\text{view}}=2$ fluoroscopic views. The distributions reflect the distance between the computed drill axis and the outer bone cortex as a function of distance along the drill axis for an ensemble of simulated trajectories (with the solid line representing median distance from cortex and the shaded region representing the interquartile range). The horizontal dashed line represents the thickness of the K-wire (1.5-mm radius, 300-mm length), with values below this threshold indicating a breach of the bone cortex. The distance between the planned trajectory and the outer bone cortex is also shown for reference (solid black line labeled as “plan”).

Also shown in Fig. 6 are plots of the distance between the registered drill axis and the bone cortex as a function of position along the trajectory. Distances below the radius of the K-wire (dashed line in Fig. 6) indicate a breach of the bone cortex. In clinical practice, the K-wire is not necessarily inserted along the entire length of the planned trajectory (enough to guide a cannulated screw). For $N_{\text{view}}=1$, the average distance to cortex was 4.4 mm for AIIS-to-PSIS, 5.6 mm for PC, and 2.5 mm for SR—each without breach of the cortex, apart from a minor impingement ($\sim 0.1\mathrm{mm}$) for the lower quartile of the data in the narrowest region of the central SR, which would likely correspond to a glancing incidence of the K-wire. For $N_{\text{view}}=2$, the average distance to cortex was 5.1 mm for AIIS-to-PSIS, 6.4 mm for PC, and 2.9 mm for SR—all well within the acceptance corridor.

The accuracy of drill axis registration was further evaluated (for $M=5$ markers and $N_{\text{view}}=2$) to measure registration errors during K-wire insertion along multiple pelvic bone corridors in phantom B. Positional and rotational TRE [${\mathrm{TRE}}_x$ and ${\mathrm{TRE}}_{\phi }$ in Eq. (8)] were estimated at three K-wire positions (pooled across three trajectories) and are illustrated in Fig. 7. Overall, system performance was found to be relatively consistent, noting a slight increase in error at increased depth along the trajectory. At the entry position, median ${\mathrm{TRE}}_x=1.8\mathrm{mm}$ (1.1-mm IQR) and ${\mathrm{TRE}}_{\phi }=0.4\mathrm{deg}$ (0.6-deg IQR). At the midpoint position, the error slightly increased with ${\mathrm{TRE}}_x=2.1\mathrm{mm}$ (1.8-mm IQR) and ${\mathrm{TRE}}_{\phi }=0.6\mathrm{deg}$ (0.8-deg IQR), primarily attributed to propagation of positional and rotational errors from the entry point. At the endpoint, errors increased further to median ${\mathrm{TRE}}_x=2.5\mathrm{mm}$ (3.2-mm IQR) and ${\mathrm{TRE}}_{\phi }=0.7\mathrm{deg}$ (0.6-deg IQR), primarily attributed to mechanical bending of the K-wire as it was advanced freehand into the bone model. Figure 7(c) illustrates an example visualization of augmented guidance from this experiment (video-fluoro navigation mode) for a K-wire inserted along the AIIS-PSIS trajectory. The figure illustrates the error propagation along the length of the trajectory, and an error boundary is shown, describing an “error cone” (cyan) describing the upper quartile in TRE propagated from the entry position.

Fig. 7.

End-to-end performance in drill axis localization during actual K-wire insertion. Registration error was assessed at the entry, midpoint, and endpoint of each trajectory in terms of (a) translational error (${\mathrm{TRE}}_x$) at the K-wire tip and (b) rotational error (${\mathrm{TRE}}_{\phi }$) between the computed and true drill axes. (c) Visualization of augmented guidance within the AIIS-to-PSIS trajectory, illustrating a con” of trajectories (cyan) about the K-wire bounding the IQR of trajectories based on the TRE at the entry position.

3.4. Video-Fluoro and Video-CT Guidance

Figure 8 illustrates two modes of surgical guidance enabled by the video drill system, taking the AIIS-to-PSIS trajectory from phantom A as an example. Figure 8(a) shows video-fluoro augmentation in which fluoroscopic views are overlaid with the registered drill axis trajectory in real time according to the current pose of the drill—e.g., trajectories overlaid in Fig. 6(a) corresponding to $N_{\text{view}}=1$ or 2 (cyan or magenta, respectively). This mode of navigation allows the surgeon to adjust freehand drill trajectory in real time to align with the planned trajectory, with the background anatomical scene providing visual context. Since only the axis of the drill (cf., the “tip” of the K-wire) is conveyed, the trajectory overlay may be of primary benefit to the surgeon by helping to guide selection of the entry point and initial K-wire angulation, which are essential (and challenging) in establishing a safe approach into the bone. Visualization in multiple augmented fluoroscopic views gives reliable 3D reckoning of the scene (normally done mentally, requiring years of training in understanding the complex morphology of the pelvis in projection views). To the extent that the presence of markers in the fluoroscopy image is visually distracting [e.g., in Fig. 8(a)], they can be digitally masked (e.g., by median filter), since the precise location of the wire and BB features was accurately determined in the marker detection step. Once the K-wire is advanced into the bone, of course, the K-wire tip is directly visible in the fluoroscopy image, and the overlays are confirmatory. This video-fluoro mode could be well suited to procedures for which no preoperative 3D image is available, as it establishes registration using only the fluoroscopic images collected in standard clinical workflow for fluoroscopically guided procedures.

Fig. 8.

Illustration of video-fluoro and video-CT modes of guidance with the drill-mounted video system. (a) Augmented fluoroscopy guidance (video-fluoro mode) with commonly acquired fluoroscopic views. If a preoperative CT is available, the planned trajectory and acceptance corridors may also be overlaid (shown in green). (b) Augmented CT guidance (video-CT mode) illustrated with multiplanar slices of the preoperative CT image. The acceptance corridor and planned trajectory (green) are shown along with real-time rendering of the drill axis solved with $N_{\text{view}}=1$ (cyan) or $N_{\text{view}}=2$ (magenta). The trajectory shown in this example corresponds to K-wire delivery along the AIIS-to-PIIS bone corridor.

For cases in which a preoperative CT (or intraoperative CBCT) is available, the system permits video-CT guidance as shown in Fig. 8(b), providing real-time 3D navigation analogous to common surgical tracking systems. The CT image and fluoroscopic views are overlaid with acceptance corridors and planned trajectories (green), and the drill axis trajectory is rendered in real time according to the pose of the drill to aid the surgeon in determining whether the current trajectory conforms within the bone corridor. Such a 3D-navigated view is common in neurosurgery and spine surgery via surgical trackers. The video-CT guidance provided by the video drill system could help to bring the precision of 3D navigation to orthopaedic trauma surgery without the additional cost and workflow associated with tracking systems.

The computational runtime of the system is summarized in Table 4, divided in two sections: (top) fluoroscopy/CT registration steps that are performed intermittently during the case (e.g., with each new fluoroscopic view and/or in the event that the markers arrangement is perturbed); and (bottom) video navigation computed continuously according to freehand motion of the drill. In the current research implementation, the runtime for fluoroscopy/CT registration was 3.1 min total for $N_{\text{view}}=1$ and 4 min for $N_{\text{view}}=2$. In each case, the most time-consuming step (72% to 74% of total runtime) was fluoroscopy marker localization. It is worthy to note that the fluoroscopy-to-CT registration is performed in parallel. In the current implementation, the 3D–2D localization of each marker was performed sequentially, suggesting a potential $5\times$ speedup (from $\sim 175$ to $\sim 30\mathrm{s}$ via parallelization). Methods for feature extraction described in Sec. 2.2.2 were also fairly rudimentary for initial feasibility testing, and more efficient implementations could reduce the feature extraction step to a few seconds.⁴⁴

Table 4.

Computational runtime for the video-guided drill system. Values are mean ± standard deviation in 60 trials from the AIIS-to-PSIS trajectory experiment.

Fluoroscopy/CT Registration	$N_{\text{view}}=1$	$N_{\text{view}}=2$
Fluoroscopy marker feature extraction	$13.1\pm 0.4\mathrm{s}$	$26.3\pm 0.9\mathrm{s}$
Fluoroscopy marker localization	$139.0\pm 15.0\mathrm{s}$	$175.0\pm 20.0\mathrm{s}$
Fluoroscopy-to-CT registration	$35.5\pm 7.0\mathrm{s}$	$40.3\pm 3.3\mathrm{s}$
Video navigation
Video feature extraction and marker localization	$56.9\pm 9.4\mathrm{ms}$
Video-to-fluoroscopy registration	$1.5\pm 0.6\mathrm{ms}$
Fluoroscopy/CT augmentation	$40.1\pm 6.4\mathrm{ms}$

The lower part of Table 4 summarizes runtime for the continuous video navigation loop, consisting of video feature extraction, marker localization, video-to-fluoroscopy registration, and augmentation of the fluoroscopy image or CT volume with the trajectory. Total runtime for the video guidance loop was $96.2\pm 11.4\mathrm{ms}$, permitting an update rate of $\sim 10$ frames per second. Runtime in the current implementation is limited primarily by the video feature extraction step (59% of runtime in the video navigation loop), with potential speedup through alternative image segmentation schemes for detection of candidate markers and multiresolution, image pyramid strategies³⁰ for marker extraction. Taken together—and recognizing the intermittent nature of the slower (fluoroscopy/CT registration) step—the results are fairly promising with respect to further development and potentially practical clinical implementation.

4. Discussion and Conclusions

A video-based system for surgical drill guidance in pelvic trauma surgery was reported. The system uses a drill-mounted miniature video camera and multimodal markers visible in both video and fluoroscopic images for real-time registration of the drill axis with respect to the patient. The registered drill trajectory is augmented onto fluoroscopic images (or a preoperative CT via 3D–2D registration) in real time to provide navigation and guidance to surgeons, particularly in the important task of initial K-wire entry point placement and orientation.

Feasibility assessment in phantom evaluated system performance in the context of common pelvic bone corridors. Drill axis registration solved with multiple ($N_{\text{view}}>1$) fluoroscopic views demonstrated conformance within multiple pelvic bone corridors (viz., the AIIS-PSIS, PC, and SR), with margins sufficient for common K-wire sizes (median ${\mathrm{TRE}}_x=0.9$ mm and ${\mathrm{TRE}}_{\phi }=2.0\mathrm{deg}$ for $M=5$ markers). Registration solved with a single ($N_{\text{view}}=1$) fluoroscopic view was notably less accurate (median ${\mathrm{TRE}}_x=3.4$ mm and ${\mathrm{TRE}}_{\phi }=2.7\mathrm{deg}$ for $M=5$ markers), but still yielded acceptable conformance within bone corridors. While registration based on multiple fluoroscopic views is more accurate, single-view registration may be adequate in scenarios of complex setups (e.g., external frames of fixators that constrain C-arm rotation), non-isocentric C-arms (which may require gross repositioning of the C-arm to maintain structures within the FOV), and/or rapid workflow requirements. Experiments showed that registration was robust with as few as four markers ($M\ge 4$) within the FOV. Experiments also quantified errors during K-wire insertion, demonstrating relatively consistent registration performance along the trajectory, subject to additional mechanical sources of error that may arise during the drilling process (e.g., K-wire bending).

Some previously reported video-based guidance solutions^21–25 require the placement of fiducial markers during preoperative setup to establish CT-to-patient registration. This requirement presents a major bottleneck in workflow and is likely infeasible in emergent trauma scenarios in which the CT is acquired for a multitude of purposes in diagnosis, rule-out, and planning. Moreover, such methods require the markers to be unperturbed during the procedure.⁴⁵ The proposed system eliminates this bottleneck by establishing image-to-patient registration intraoperatively via 3D–2D registration (i.e., fluoroscopy-to-CT registration) with each fluoroscopic view. The resulting registration accuracy (${\delta }_V<2\mathrm{mm}$) is comparable to that of established surgical tracking systems.⁴⁶ Furthermore, the system may operate independent of (i.e., without) a preoperative CT via video-fluoro mode to allow navigation in cases for which CT is unavailable or obviated due to clear emergent need for surgery—which is often the case in trauma surgery.⁴⁷

The system uses routinely acquired fluoroscopic images for registration and a minimum of additional hardware components—viz., a drill-mounted camera that can be swapped between drills (and drill axis calibration established with a simple jig) and multimodal markers placed about the surgical field (e.g., via sticky-back tabs). The system has a strong degree of robustness against variations in marker positions that may commonly occur during the procedure. Unlike conventional navigation systems which often require time consuming, manual re-registration steps, the proposed system automatically updates the registration with each new fluoroscopic view, thereby restoring navigation with minimal impact on workflow. The system therefore appears to be potentially compatible with standard workflows in mainstream orthopaedic trauma surgery, where the adoption of surgical tracking systems has been limited. The system has the potential to reduce trial-and-error in K-wire placement and thereby improve precision, reduce radiation dose, and reduce procedure time. Furthermore, augmentation of the drill trajectory in fluoroscopy and/or CT is anticipated to improve the accuracy of device placement in a manner analogous to the improvements gained by incorporation of trackers in navigated brain or spine surgery—potentially bringing the benefits of precision guidance to mainstream orthopaedic trauma surgery.

Limitations in the current work include practical considerations in the implementation of hardware components (viz., the multimodal markers and camera). The marker size ($Ø50\mathrm{mm}$) in the initial prototype was somewhat large and limited the number of markers and arrangements that could be conveniently arranged about the surgical field ($M\le 5$). Reducing the form factor of the markers is an important next step for clinical translation, as it will allow: (1) a larger number of markers to be used for registration; (2) an increase in the operating space about the surgical site, and (3) greater flexibility in placing the markers about the patient. Alternative realizations of the multimodal markers include a flexible mask or frame that bears the marker arrangement all at once and eliminates the need to individually place markers about the surgical field. Similarly, the initial camera configuration used a somewhat bulky drill mount, and ongoing work aims to streamline the mount and refine the camera selection for improved resolution and FOV. Future work will identify methods for sterilizing hardware components prior to surgery, including: (1) use of a sterile, disposable camera drape with a clear camera lens cover; and (2) the use of sterile, disposable multimodal markers.

In the current work, fluoroscopy-to-CT registration operated under the assumption of rigidity between the patient and preoperative CT. This assumption is reasonable in scenarios with simple, non-displaced fractures (which account for the majority of fragility fractures⁴⁸) but is challenged with complex, comminuted fractures that exhibit strong anatomical mismatch as they are manipulated during the procedure. Methods to overcome such challenges have been proposed, including image masking (to focus registration on a particular region of interest and masking the effect of comminuted/displaced fracture components on the registration) and recently developed multibody 3D–2D registration methods^49–51 that account for displacement of multiple, fractured bone fragments by computing the pose of multiple rigid components through statistical pose modeling. Integration of such methods with the drill-mounted video system could improve the robustness of fluoroscopy-to-CT registration and extend its applicability beyond non-displaced fractures.

The current work is very specifically focused on the first implementation of the system, development of requisite registration algorithms, and proof of concept with quantitative assessment of registration accuracy and runtime—recognizing that such studies were conducted within the limited context of anthropomorphic phantoms. The current work investigated the feasibility of the video-drill system for fluoroscopically guided procedures, focusing on the particularly challenging step of identifying the initial entry point and orientation. The experiments also provided initial insight on considerations such as K-wire bending during drilling, and future work will investigate additional aspects of practical use, including camera jitter. The current studies were sufficient to provide quantitative insight on registration performance and tradeoffs in accuracy with a variable number of fluoroscopic views ($N_{\text{view}}$) and markers ($M$) while quantifying end-to-end-performance and conformance within pelvic bone corridors. Future work will further assess system performance in pre-clinical cadaver studies in which the system will be operated in real time by a fellowship-trained orthopaedic surgeon. Studies will assess the accuracy of K-wire delivery by both novice and experienced surgeons with both the proposed system for video-drill guidance and a conventional approach with fluoroscopy alone. In addition to accuracy (e.g., conformance of the guidewire in bone corridors), the workflow implications of the system will also be evaluated in terms of fluoroscopy time and procedure time compared to conventional fluoroscopic guidance.

In conclusion, a video-based surgical drill guidance system was described and tested in initial feasibility studies, addressing several issues that may be suitable for the fast-paced workflow of orthopaedic trauma procedures. The accuracy and feasibility of the system were quantified in terms of guidewire placement in phantom, serving as an important precursor for future pre-clinical studies. The current studies focused on the context of pelvic trauma surgery, and the proposed solution may be generalizable to a broader range of image-guided procedures (e.g., hip, femur, knee, and wrist surgery) that conventionally use fluoroscopy to guide instrument placement.

Biography

Biographies of the authors are not available.

Disclosures

No conflicts of interest, financial or otherwise, are declared by the authors.