Utility of the LevelCheck algorithm for decision support in vertebral localization

Abstract

Study Design

An automatic radiographic labelling algorithm called “LevelCheck” was analyzed as a means of decision support for target localization in spine surgery. The potential clinical utility and scenarios in which LevelCheck is likely to be the most beneficial were assessed in a retrospective clinical data set (398 cases) in terms of expert consensus from a multi-reader study (3 spine surgeons).

Objective

To evaluate the potential utility of the LevelCheck algorithm for vertebrae localization.

Summary of Background Data

398 intraoperative radiographs and 178 preoperative CT images for patients undergoing spine surgery in cervical, thoracic, lumbar regions.

Methods

Vertebral labels annotated in preoperative CT image were overlaid on intraoperative radiographs via 3D-2D registration. 3 spine surgeons assessed the radiographs and LevelCheck labeling according to a questionnaire evaluating performance, utility, and suitability to surgical workflow. Geometric accuracy and registration run time were measured for each case.

Results

LevelCheck was judged to be helpful in 42.2% of the cases (168/398), to improve confidence in 30.6% of the cases (122/398), and in no case diminished performance (0/398), supporting its potential as an independent check and assistant to decision support in spine surgery. The clinical contexts for which the method was judged most likely to be beneficial included the following scenarios: images with lack of conspicuous anatomical landmarks; level counting across long spine segments; vertebrae obscured by other anatomy (e.g., shoulders); poor radiographic image quality; and anatomical variations / abnormalities. The method demonstrated 100% geometric accuracy (i.e., overlaid labels within the correct vertebral level in all cases) and did not introduce ambiguity in image interpretation.

Conclusion

LevelCheck is a potentially useful means of decision support in vertebral level localization in spine surgery.

Level of Evidence

N/A

Introduction

Wrong-level spine surgery is reported to occur at a rate of 1 in 3110 procedures¹ and carries substantial patient morbidity and monetary cost². Aside from such adverse never-events, accurate localization of target vertebrae can be a challenging task and requires considerable time and attention in the operating room, often presenting a source of stress and uncertainty. In order to facilitate reliable intraoperative target localization, some centers conduct a separate preoperative procedure to mark target vertebrae with cement injection or pedicle marker insertion under CT fluoroscopy, requiring additional time, expense, radiation, and coordination of hospital resources and personnel. In our institution, we do this specifically for thoracic disc herniations as a standard. Intraoperative decision support and verification of vertebrae localization in the form of an independent check could provide valuable assistance to the surgeon. Within the context of safety science, an independent check constitutes one of the highest levels of preventative measures, and is defined as a mechanism by which two entities separately check each component of the process and compare results³. The availability of a tool for such decision support in spine surgery could provide a useful means to improve confidence in level localization, reduce time and stress, and potentially prevent wrong-level errors.

We have developed a software-based image registration algorithm referred to as “LevelCheck” for automatic vertebral localization. The method utilizes intraoperative radiographs/fluoroscopy images acquired for localization within the current standard-of-care and performs image co-registration with a preoperative CT image. Image registration enables the vertebral labels (and/or other annotation) identified in preoperative CT to be overlaid onto corresponding locations in intraoperative radiographs. Alternative solutions for assistance in localization include surgical tracking and navigation systems^4–9; however, the use of surgical navigation introduces cumbersome steps to clinical workflow and technical barriers that have somewhat limited broad utilization in spine surgery. Technical development of the LevelCheck method was described in ^10–12, including validation in simulation, phantom, and cadaver studies that exhibited robust performance even in the presence of anatomical deformation and the introduction of surgical instrumentation in the radiograph. First clinical application of the method was reported by Lo et al.¹³ in a study of 20 patients that demonstrated basic feasibility, identified potential failure-modes, and motivated further investigation of its utility.

Clinical translation of this method requires an understanding of how it could benefit surgeons during surgery. The work reported below investigated the utility of the LevelCheck algorithm in a much larger clinical dataset (398 intraoperative radiographs) than previous work, covering a broader spectrum of cases and expanding assessment to a multi-reader study. Whereas previous work focused on aspects relating to the technical development of the algorithm itself, this work focuses on the potential clinical utility and it provides the first assessment of the extent to which LevelCheck may be helpful and/or improve confidence in level localization.

Methods

Data acquisition

Clinical data were collected retrospectively from an IRB approved study of patients undergoing surgery in cervical, thoracic, and lumbar regions in the spine. Basic inclusion criteria included the availability of a preoperative CT image acquired within one year prior to the surgery. These included multiple pathologies, including degenerative disease, deformity, infection, congenital disease, metabolic disorder, and cancer. All radiographs acquired during surgery for localization prior to decompression, instrumentation, or post-instrument verification were initially considered. From an initial dataset consisting of 504 radiographs from 198 patients, some were excluded according to the following criteria: (1) limited field-of-view covering 3 or fewer spine levels (13 radiographs); (2) minimal or no overlap with CT (22 radiographs); (3) cervical cases with differences between preoperative and intraoperative patient setup where head/neck positioning was intentionally manipulated to ease access to the surgical site (53 radiographs); and (4) vertebrae boundaries not visible due to poor image quality (21 radiographs). The remaining dataset consisted of 178 patients, each with a preoperative CT, giving 398 intraoperative radiographs which were analyzed in the experiments detailed below. Definition of vertebral levels was provided in each preoperative CT by a fellowship-trained neuroradiologist. Standard designation of vertebral levels was used (C1-C7, T1-T12, L1-L5, and S1) and allowed for normal anatomical variations, such as L6, T13, etc.

3D-2D registration

Registration achieves a mapping of image coordinates between the intraoperative radiograph and the preoperative CT by calculating the digitally-reconstructed-radiograph (DRR) that maximizes similarity. The algorithm was the same as described in Lo et al. 2015 (Figure 1). The registrations were performed following a coarse manual initialization along the longitudinal axis of the spine that provided some initial overlap between the images. Registration was shown to be robust against initialization errors of ~10 vertebrae-levels¹⁴. A simple polygonal mask was defined as in Lo et al.¹³ to exclude collimator edges and surgical instrumentation from radiographs. The registration output consisted of an overlay of vertebral labels on the radiograph as shown in Figure 1.

Figure 1.

Flowchart diagram for the LevelCheck algorithm. The image on the right shows an example LevelCheck overlay of labels on an intraoperative radiograph.

Assessment

Utility assessment

Registrations were performed for the 398 radiographs, and the LevelCheck output was evaluated by 3 spine surgeons completing the questionnaire in Table 1. The questionnaire consisted of two major portions. In the first portion (questions 1-4), the radiograph was shown without the LevelCheck labels, and the surgeons were queried regarding: purpose of the radiograph; image quality; whether localization support would be helpful; and approximate permissible time for decision support. In the second portion (questions 5-8), the LevelCheck labels were overlaid on the radiograph, and the surgeon responded regarding: effect on confidence; accuracy of localization; error assessment (if any); and complexity of mask. At this stage, the neuro-radiologist-defined labels were also displayed to assist the surgeons’ evaluation. The case order was randomized for each surgeon to minimize potential bias associated with reading order or fatigue. Readers were free to exercise basic display controls, and time was unconstrained in evaluation of each case. The time to read all cases was ~8 hours for each reader (~1.2 min per case).

Accuracy assessment

As independent verification of LevelCheck geometric accuracy, the projected level location was quantitatively compared to the neuroradiologist-defined true position. The distance between the LevelCheck label and ground truth position is the Projection Distance Error (PDE) (units mm). Note that the PDE is measured in the plane of the radiographic detector, such that the corresponding distance in the patient should be scaled down by the image magnification (typically ~1.4 – 1.6 for mobile radiographs). For example, a PDE=30 mm corresponds to a distance of ~20 mm in the patient. Registration was computed on a Windows 7 64-bit workstation with an Intel Xeon processor (3.6 GHz). Run time was measured as the time required after initiating the LevelCheck algorithm to overlay labels onto the radiograph.

Results

Assessment of utility

The potential utility of assistance in level localization was assessed via question 3 (Table 1) where surgeons assessed whether assistance may be helpful prior to observing LevelCheck output. As summarized in Table 2(a), localization assistance was judged to be helpful in 42.2% (168/398) of cases on average across the three surgeons, ranging from 31.7% (126/398) to 53.3% (212/398).

The surgeons’ initial assessment of each radiograph was compared via question 5 (Table 1) with the output of the LevelCheck algorithm to determine whether LevelCheck annotations would improve (or degrade) their level of confidence in localization. As shown in Table 2(b), LevelCheck was judged to potentially improve the surgeon’s confidence in 30.6% (122/398) of the cases on average, ranging from 10.1% (40/398) to 46.5% (185/398). In no case did the algorithm degrade surgeon’s confidence. In the current study there were no cases in which the surgeon’s initial assessment was in conflict with the LevelCheck output.

Assessing the results of Table 2(a–b), we see that not all cases in which the surgeons judged LevelCheck to be “helpful” were also associated with an improvement in the surgeons’ “confidence.” In all cases, LevelCheck was found to provide confirmatory evidence that was in concordance with the surgeon’s initial assessment which might not necessarily result in a confidence improvement. In an independent check, such confirmatory evidence is the most common scenario and does not undermine utility. The fairly high variability in confidence ratings measured among the readers (10.1%–46.5%) could be due to subjectivity inherent to surgeons’ assessment of the value of an independent check in comparison to their initial assessment. The retrospective nature of this study could have been an additional limiting factor, and the utility may rate higher in online prospective studies in the operating room, where time is limited, and the stakes are higher.

For what anatomy is LevelCheck most helpful?

As shown in Figure 2, LevelCheck was found to be the most helpful in scenarios involving the thoracic region, where surgeons reported the method to be useful in 77.7% (23/30) of cases. This agrees with the general notion motivating previous work¹⁰ that ambiguity of anatomical landmarks in the midthoracic region is a source of uncertainty in level localization. LevelCheck was judged to be helpful in 46.5% (47/101) of cases in cervical spine, 44.9% (21/69) of cases in thorocolumbar spine and 33.5% (66/198) of cases in lumbar spine. Reasons for the demonstrated utility even in the C-spine and L-spine are discussed below.

Figure 2.

Utility of LevelCheck in localization of cervical, thoracic, thoracolumbar, and lumbar regions of the spine. For each region, the responses are grouped according to readers (#1, 2, and 3), with the average response (over three readers) denoted by ‘A’.

At what stage of the procedure is LevelCheck helpful?

As shown in Figure 3, and as might be expected, LevelCheck was found to be most helpful in radiographs acquired during the localization stage of surgery, where it was judged to be helpful in 59.8% (93/156) of cases. LevelCheck was found to be helpful in 33.7% (31/91) of cases in the later intraoperative verification stage and in 28.9% (44/152) of cases in the final verification stage, where such utility presumably relates to the ability to assist with quick interpretation of the radiograph in quality assurance of the surgical product.

Figure 3.

Utility of LevelCheck at the localization, intraoperative verification, and final verification stages of surgery. At each stage the responses from individual readers are denoted by 1, 2, and 3 with the average response among all three readers denoted by ‘A’.

Consensus case examples for utility of the LevelCheck algorithm

Despite reader variability on which particular cases benefited most from LevelCheck annotation, consensus was observed in a subset of the images. For example, there was unanimous consensus (3/3 readers) in 2.8% of cases (11/398) that LevelCheck would both be helpful and improve confidence and majority consensus (2/3 readers) in 17.6% of cases (70/398). By examining the majority consensus cases, we identified 5 primary clinical scenarios described below.

Lack of clearly visible reference anatomical landmarks

Unambiguous anatomical reference landmarks are often the basis for spine level counting, including the sacrum, odontoid, the twelfth rib, or (possibly) pre-existing hardware. As shown in Figure 4(a), the lack of such landmarks within the radiographic field-of-view can challenge reliable level localization.

Figure 4.

Scenarios in which LevelCheck was judged to be most helpful and improve confidence. (a) Lateral radiograph of thoracic vertebrae with no clearly visible anatomical reference points that could challenge manual level counting. (b) Long spine segment spanning levels T9-S1. Although the sacrum is clearly visible in the radiograph length of the segment and the blurred visibility of vertebrae endplates due to coronal curvature could challenge localization during surgery. (c) Occlusion due to shoulder obstructing vertebrae visibility and localization in the lower cervical spine. (d) Poor image quality observed in the lateral lumbar radiographs challenge the visibility of the sacrum and the lower lumbar vertebral levels limiting surgeon’s ability to use the sacrum as an anatomical reference level. (e) Anatomical abnormalities such as the presence of a lumbarized sacrum could confound level counting primarily based on a radiographic view.

Long spine segments

Even with a clearly visible anatomical reference landmark, manual level counting over long segments of spine could be an error-prone, stressful task especially due to poor visibility of disc spaces and endplates. LevelCheck annotation for additional verification (even if confirmatory) could augment reliable localization and reduce the stress of counting over such long segments as in the example case shown in Figure 4(b).

Vertebrae obscured by other anatomy

The visibility of certain vertebrae can be challenged by the presence of overlapping anatomy in the radiograph. For example, the lower cervical vertebrae are often obscured by the presence of high and broad shoulders as shown in Figure 4(c) and can obstruct clear visualization and manual level counting. In such cases, despite pulling shoulders down to acquire oblique radiographs, level counting is often challenging and could lead to a larger incision, also increasing the risk of traction brachial plexopathy.

Poor radiographic image quality

Poor image quality in radiographs along with other complications can challenge confidence in manual level counting. For example, poor image quality along with coronal curvature in the spine, previous fusion masses, or artifact from any other implants in the body can make it difficult to see the vertebral end plates and thereby confound manual level counting. An example is shown in Figure 4(d), where all three surgeons responded to image quality assessment (question 2, Table 1) with a score <3.

Anatomical complexities/abnormalities

Normal anatomical variations and/or abnormalities – e.g., extra vertebrae, missing vertebrae, or composite vertebrae – can also confound manual level counting. Examples include T13, L6, and/or a lumbarized sacrum. In such instances, LevelCheck could help to overcome uncertainty and also help to standardize approaches to level annotation among radiologists and surgeons (counting from the 12^th rib vs counting from the sacrum) that could be problematic during the presence of abnormalities. An example is shown in Figure 4(e) involving the lumbar sacral transitional vertebrae (seen in 15-25% of the general population^15,16).

LevelCheck performance

Accuracy assessment

As shown in Figure 5, geometric accuracy evaluated as PDE (median ± interquartile range, IQR) was 6.13 ± 4.23 mm over all 398 registrations. All cases were within the nominal range of “successful registration” (PDE <30 mm) for which the label was within close proximity to the true vertebrae. This agrees with the 100% accuracy rate reported by Lo et al¹³. Table 2(c) summarizes the surgeons’ visual assessment of geometry accuracy, combining responses from questions #6 and #7 (Table 1). While Reader 1 determined that all cases (398/398) were registered with acceptable accuracy, Readers 2 and 3 identified 0.3% and 6.0% of cases (1/398 and 24/398, respectively) for which the registration was considered to be questionable due to the label being overlaid near the periphery of the vertebrae (not on the centroid). The reason is suspected to be anatomical deformation (i.e., strong change in spinal curvature between the supine CT and the prone radiograph), and the effect was observed to primarily affect labels at the superior or inferior ends of the radiograph. While this could be acceptable with respect to the task of level localization, a non-rigid variation of the algorithm that allows for a deformable registration¹⁷ could further enhance geometric accuracy and utility.

Figure 5.

Geometric accuracy of LevelCheck registration evaluated in terms PDE (mm) at the detector. All cases were successfully registered (PDE < 30 mm) such that each LevelCheck annotation label was within close proximity to the correct vertebral level, in agreement with the radiologist-defined true level. The vertical dashed line marks the approximate size of the vertebrae projected on the detector.

Registration run-time and workflow

According to responses to question 4 (Table 1), surgeons were willing to wait 20s – 1 min for registration for all cases, regardless of the utility. In cases for which the surgeons judged LevelCheck to be helpful, they were willing to wait more than 1 min – specifically, in 60.6% of cases (305/503). The measured registration run-time for LevelCheck registration in this study was generally consistent with such workflow, demonstrating run-time (median ± IQR) of (44.4 ± 28.4)s.

Discussion

Retrospective assessment of radiographs in 398 cases showed LevelCheck to be helpful in ~42% of the cases and would have improved surgeon’s confidence in ~30% of the cases, demonstrating the potential for LevelCheck to serve as an independent check, verification, and decision support in spine surgery. In no case did LevelCheck diminish decision making. The method was found to be most helpful in radiographs acquired in the thoracic region and during the initial localization stage of surgery.

Achieving robust performance with minimal disruptions to the surgical workflow is imperative for successful clinical translation of this software. This study demonstrated that LevelCheck achieved robust performance over a broad variety of clinical scenarios and demonstrated the potential to improve surgeon’s confidence in some cases; moreover, in no case did the algorithm degrade the surgeon’s confidence. The responses indicated that surgeons were willing to wait approximately 20s – 1 min for the output in all cases, regardless of its judged utility. Therefore, potential use scenarios for LevelCheck include implementation as an independent check in decision support in which the algorithm is confirmatory (and a preventative measure against localization error) immediately following (and within 1 min) of each radiographic image acquisition. The surgeon could then resolve uncertainties at his/her discretion. Broad clinical utilization of this method will benefit from a variety of improvements and ongoing development. Technical factors include current work that aims to automate initialization, eliminate manual masking steps¹⁴, and overcome errors arising in cases that exhibit particularly strong anatomical deformation.¹⁷ In addition, reduction of registration run-time from ~20-40 s to <10 s would enhance integration within streamlined clinical workflow and likely requires the incorporation of high-speed computing hardware (e.g., as now commonly available in high-performance graphics processing units) within the clinical workstation. The system also requires ready access to the patient’s preoperative imaging - e.g., via PACS connection in the OR. Radiology techs who setup the imaging systems and acquire / process the images for display should be suitably trained with respect to LevelCheck capabilities, parameters, and limitations, and all surgical staff should recognize the limitations of any decision support system - as an adjuvant - and never a replacement for their expertise, experience, and senses. Ultimately, evaluation in prospective clinical studies and/or multi-center trials will help to demonstrate the improvements to safety and surgical workflow.

In principle, there could be an instance in which the surgeon makes a localization error, followed by the algorithm making the same error, thereby reinforcing the error. In our study, as validated according to annotations defined by the neuroradiolgoist, we did not encounter this scenario in any case. In a more general context, when implemented as an independent check, the probability of both the surgeon and the algorithm making the same error is an order of magnitude lower than the scenario of a surgeon making the error on his/her own.

Conclusion

LevelCheck offers a potential means of decision support and independent check that could improve surgeons’ confidence in spine level localization, speed workflow, and improve patient safety. Moreover, this could help to reduce or eliminate invasive preoperative procedures (e.g., CT-guided placement of a marker on target vertebrae) to localize spinal levels, especially in the thoracic region, which have become increasingly common. Analysis in this work demonstrated that LevelCheck was successful in registration over a very broad range of cases capturing very realistic conditions with respect to the patient-setup, anatomical deformation, surgical instrumentation, and image quality motivating prospective studies of the method in clinical use.