Neurologist Consistency in interpreting information provided by an Interactive Visualization Software for Deep Brain Stimulation Post-Operative Programming Assistance

Abstract

Introduction

Post-operative programming in deep brain stimulation (DBS) therapy for movement disorders can be challenging and time consuming. Providing the neurologist with tools to visualize the electrode location relative to the patient’s anatomy along with models of tissue activation and statistical data can therefore be very helpful. In this study, we evaluate the consistency between neurologists in interpreting and using such information provided by our DBS programming assistance software.

Methods

Five neurologists experienced in DBS programming were each given a dataset of 29 leads implanted in 17 patients. For each patient, probabilistic maps of stimulation response, anatomical images, models of tissue activation volumes and electrode positions were presented inside a software framework called CRAVE. Consistency between neurologists in optimal contact selection using the software was measured.

Results

With only the efficacy map, the average consistency among the five neurologists with respect to the mode and mean of their selections was 97% and 95% respectively while these numbers were 93% and 89% respectively when both efficacy and an adverse effect map were used simultaneously. Fleiss’ kappa statistic also showed very strong agreement among the neurologists (0.87 when using one map and 0.72 when using two maps).

Conclusion

Our five neurologists demonstrated high consistency in interpreting information provided by the CRAVE interactive visualization software for DBS post-operative programming assistance. Three of our five neurologists had no prior experience with the software which suggests the software has a short learning curve and contact selection is not dependent on familiarity with the program tools.

Introduction

Deep brain stimulation (DBS) is an accepted and beneficial treatment option for movement disorders including moderately advanced Parkinson’s disease (PD) and essential tremor (ET) in situations where medications do not adequately control motor symptoms [1–4]. However, DBS programming can be a challenging and time consuming process [5,6]. Traditionally, a programming neurologist takes a lengthy step by step approach by first testing individual contacts at a standard pulse width and frequency at progressively higher stimulation amplitudes until an adverse effect threshold is reached. The objective of programming is to maximize therapeutic benefit for the patient, i.e. reduce motor symptoms as much as possible while minimizing adverse effects. If this cannot be achieved with single contacts or monopolar settings, a combination of contacts is then used.

Over the last several years, much progress has been made in building functional atlases [7–14] for use in DBS therapy using advanced image registration techniques. Much progress has also been made on models of activation volumes [15,16] and three-dimensional histological atlases [17–19]. Utilization of such data during post-operative programming requires a program that can render and present these data in a way that has a user friendly interface and more importantly leads to consistent interpretations across users. The only known relevant work to date using interactive mapping software for post-operative programming was recently published by Butson et. al. [20]. They demonstrated interactive visualization on mobile computing platforms for clinician selection of DBS programming parameters. The information provided to potential programmers was the volume of tissue activation (VTA) and the relevant anatomical structures overlaid on a DBS lead. They used an anatomical-driven approach whereby five clinicians were asked to select the optimal contact and settings based only on the interaction of the contacts and their corresponding VTAs with the anatomical structures. The study was conducted in four unilateral subthalamic (STN) targeted DBS PD patients who were both good responders to DBS therapy and whose implants were found to be localized inside the STN. Their work was not aimed at investigating consistency in the interpretation of the presented data, yet they did demonstrate a lack of consensus on the anatomical location of the optimal target.

With our software, we use a functionally-driven approach based on probabilistic maps of stimulation response mapped onto individual patients with patient-specific anatomical information. In a previous study [21], we introduced a software suite and a complete processing pipeline that could be used to assist physicians in pre-operative planning, intra-operative placement and post-operative programming of the DBS lead. The system consisted of a central repository (CranialVault) and a suite of software modules called CRAnialVault Explorer (CRAVE) that permit data entry and data visualization at each stage of the therapy. Our goal in this study was to evaluate the consistency among neurologists in interpreting and using information from the CRAVE programming module to validate its further use in a clinical setting as a tool to expedite and improve postoperative DBS programming.

Methods

Five neurologists (FP, CT, TD, JF and PH) experienced in DBS programming participated in this study. This study was approved by the Vanderbilt IRB. The neurologists were each given an identical dataset which included 29 leads implanted in 17 patients (7 bilateral STN-DBS for PD, 2 unilateral STN-DBS for PD, 5 bilateral ventral intermediate (VIM) nucleus of the thalamus-DBS for ET and 3 unilateral Vim-DBS for ET). A pre-operative MRI and a post-operative CT were acquired for each patient. Typical CT images were acquired at kVp = 120 V, exposure = 350 mAs and 512×512 pixels. In-plane resolution and slice thickness were respectively 0.5 mm and 0.75 mm. MRIs (TR 12.2 ms, TE 2.4 ms, 256×256×170 voxels, with typical voxel resolution of 1×1×1 mm³) were acquired using the SENSE parallel imaging technique (T1W/3D/TFE) from Philips on a 3T scanner. To build probability maps for the patients in this study, stimulation response data from a large population of our DBS patients were used. These data were mapped onto an atlas MRI using non-rigid registration [22,23]. Then, for individual patients in the study, data from the atlas were again projected onto the patient using non-rigid registration between the atlas and patient MRIs. For PD patients, maps of rigidity reduction and muscular contraction were built. For ET patients, maps of tremor reduction and paresthesias were built. Briefly, this involves associating each stimulation response observation with a probability density function that captures the likelihood that a region in the vicinity of the measurement point is the responsive region. Rigidity reduction and muscular contraction probability maps were built using 760 data points in 154 STN-DBS implantation and 191 points in 72 STN-DBS cases respectively. Tremor reduction and paresthesia probability maps were built using 663 data points in 85 Vim-DBS cases and 216 points in 57 Vim-DBS cases respectively. Using the post-operative CT, individual contacts in the implanted lead were extracted. By registering the MRI to the CT, the probability maps and the leads were overlaid onto the anatomical images from the MRI. For every patient, this entire process was pre-computed and packaged into a file that the neurologists could simply load into CRAVE. The software suite allowed the neurologists to visualize all this information and interact in 2D as well as in 3D. In 2D, the neurologists had access to tools that allowed visualizing and interacting with the information in the coronal, axial and sagittal views simultaneously. They could navigate through slices as well as zoom into regions of interest. In 3D, they could visualize and interact with the renderings of the statistical maps and the lead. The neurologists were also provided with several pre-computed VTA models that could be visualized in both 2D and 3D to assess the interaction of the singular or multiple active contacts with the efficacy and adverse effect maps as well as the anatomical MRI (Figure 1). Two of the five neurologists had prior experience with the software while the other three neurologists were trained just prior to the study through collective demonstration of a trial case on one large screen and an individual tutorial of the software tools on their personal laptops.

Figure 1.

CRAVE software interface that shows how models of volumes of tissue activation can be overlaid on a DBS electrode along with the patient’s MRI and probability maps. The neurologists can visualize models of volumes of tissue activation for various stimulator parameters, with one or more contacts turned on and with monopolar as well as bipolar settings. A 3D rendering is also shown.

The five neurologists were required to independently choose the single best contact (0, 1, 2 or 3) expected to cause maximum symptom reduction in the patient based on the overlay of only the efficacy map (rigidity reduction map for PD and tremor reduction map for ET) and the extracted lead on the patient’s MRI. Then the neurologists loaded the adverse effect map (muscular contraction for PD and paresthesia for ET) in addition to the efficacy map and chose the single best contact that would maximize symptom reduction while minimizing the chances of the adverse effect. Over the 29 cases, consistency between the 5 neurologists was computed as normalized indices of the total number of contacts by which the neurologists were off with respect to the mean and mode of the selections by the participating neurologists. In order to measure the agreement among the neurologists we also computed Fleiss’ kappa statistic [24,25], which is used as a chance-adjusted measure of agreement in a multi-rater multi-category dataset.

Results

When only the efficacy or symptom reduction map was used to choose the single best contact, all 5 neurologists choose the same contact in 23 out of 29 cases. In the 6 cases where there was a mismatch, the neurologists were off by no more than 1 contact. In 5 of those cases, only one neurologist differed from the other 4 neurologists’ chosen contact. In only 1 of the 29 cases was there a 2–3 split between the neurologists. The average consistency with respect to the mode of the neurologists’ selections was 97% while that with respect to the mean of the neurologists’ selections was 95%. When both the efficacy and adverse effect maps were presented for consideration, all 5 neurologists choose the same contact in 16 out of 29 cases. In the 13 cases where there was a mismatch, the neurologists were off by no more than 1 contact. In 11 of those cases, only one neurologist’s chosen contact differed from those of the other 4 neurologists. In only 2 of 29 cases was there a 2–3 split between the neurologists. The average consistency with respect to the mode of the neurologists’ selections was 93% while that with respect to the mean of the neurologists’ selections was 89%. Using only the efficacy map, Fleiss’ kappa statistic was 0.8707 with 95% CI of [0.8516, 0.8899] indicating a very strong agreement among the neurologists. With the efficacy map and an adverse effect, Fleiss’ kappa statistic was slightly lower at 0.7207 [0.7014, 0.7401].

Discussion

The results indicate that a panel of five neurologists using the CRAVE software suite were very consistent in independently selecting optimum active contacts using information provided by the statistical maps of efficacy and adverse effect. The consistency between the neurologists using the efficacy map alone was 95% or more. Fleiss’ kappa statistic showed very strong agreement among the neurologists when they used only the efficacy map, and substantial agreement when both the adverse effect and efficacy maps were used together. The marginal drop in agreement when an adverse effect map was additionally provided suggests that while such maps bring more information that can be relevant and useful, they can also increase the variability in interpretation. An analysis of the results reveals that the dataset can be divided into three categories. In the first category (14 cases), the efficacy map was closer to a single contact than the adverse effect map. In the second category (8 cases), the efficacy and adverse effect maps were at the same distance to the same single contact. In the third category (7 cases), several contacts were equidistant to both the efficacy and adverse effect maps. The availability of the adverse effect map in addition to the efficacy map changed the inter-rater agreement for two cases in the first category. In one case, all neurologists agreed with only the efficacy map but one of the neurologist changed contact when both maps were available. In another case, one of the neurologists disagreed with the others with only the efficacy map but was in agreement when both maps were shown. Availability of the adverse effect map in addition to the efficacy map for cases in the second category decreased the number of cases for which all neurologists agreed by one. Adding the adverse effect map to cases in the third category decreased the number of cases for which all neurologists agreed by 6. These results suggest that when a single contact can cover the efficacy map alone or when a single contact covers both the regions of efficacy and adverse effect, there is little room for individual preferences. In these cases, the availability of an adverse effect map in addition to the efficacy map does not substantially affect the neurologists’ decision. For cases in the third category, the situation is not as clear and tradeoffs have to be made. It is likely that, in this situation, personal preferences, e.g., relative weight put on efficacy or side effects, decrease inter-neurologist agreement.

One of the limitations of our study was that we only provided at most two maps simultaneously. It is possible that the neurologists may be interested in more than one adverse effect map along with the efficacy map and perhaps some other information as well which could potentially lower consistency. Also, we did not record the time taken by each neurologist to load each dataset (pre-registered images, pre-computed maps and pre-extracted electrodes), interact with the data and choose the optimal contact in the 29 cases. However, offline assessment over several cases showed that this entire process takes under two minutes for a bilateral case. If the registrations are not made available pre-computed, the software has built-in algorithms with complete functionality of performing these registrations on the neurologist’s computer. Computing the rigid registration between the patient MRI and CT, the non-rigid registration between the patient MRI and atlas MRI and validating these on a typical 1.8 GHz Intel Core™ laptop computer with 4GB RAM takes on the order of five minutes.

The results suggest that the CRAVE DBS software may be well suited for clinical use by programming neurologists. The program also has a small learning curve with user friendly interaction given our high consistency despite the fact that 3 of our 5 study participants had no prior experience with the software. These results are promising for potentially widespread use by physicians for post-operative DBS programming. We are now in the process of testing this in a clinical setting.