Validity and Efficacy of Screening Algorithms for Assessing Deep Brain Stimulation Candidacy in Parkinson's Disease

Abstract

DBS for Parkinson's disease (PD) may be underutilized because of limited access to care (most DBS surgeries are performed at specialized centers) or over‐referral of poor candidates, leading to inequitable utilization of limited evaluative resources. There is a pressing need for a widely employable screening algorithm to aid in the evaluation of PD candidates for DBS. The aim of this study was to compare the validity and efficacy of two published screening algorithms, the Florida Surgical Questionnaire for PD and Stimulus, to predict candidacy for DBS. We reviewed the clinical data at our DBS center for 147 consecutive PD DBS referrals between 1 September 2007 and 31 December 2011. Florida Surgical Questionnaire and Stimulus scores were applied retrospectively through a chart review of the movement disorder neurologist's initial clinical evaluation. The validity and accuracy of these two algorithms in predicting candidacy for DBS was compared to the decision to offer DBS surgery by our multidisciplinary DBS team. Of the 130 consecutive PD referrals who presented for initial evaluation, 50 were offered DBS after a standardized multidisciplinary evaluation. The Stimulus scale was a superior screening tool for predicting PD DBS candidacy in these referrals (area under the receiver operating curve [AUROC] = 0.8088), compared to the Florida Surgical Questionnaire for PD (AUROC = 0.6285). In this single‐center study, Stimulus was a more appropriate screening measure than the Florida Surgical Questionnaire for PD to assess DBS candidacy for PD.

DBS is a well‐established surgical therapy for Parkinson's disease (PD) patients with medication‐refractory tremor or bothersome motor fluctuations, despite an optimized medication regimen.1 DBS has traditionally been considered only for advanced PD patients,2 but the results of a recent European trial suggest that PD patients who are earlier in the course of their disease (i.e., within 3 years of developing motor fluctuations) may also have significant benefit from DBS.3

Among Medicare beneficiaries in the United States, the primary responsibility for the longitudinal care of patients with PD is shared relatively evenly between neurologists and primary care physicians.4 Greater than 90% of DBS surgery, however, is performed at urban tertiary care facilities.5 Primary care physicians may not be aware of when a patient would qualify for DBS, raising the possibility of heterogeneity in access to DBS. Successful DBS outcomes depend heavily on appropriate patient selection, with poorly selected candidates accounting for approximately one third of DBS treatment failures.6 Thus, DBS may be underutilized because of limited access to care or over‐referral of poor candidates, leading to inequitable utilization of limited evaluative resources.

Two published algorithms are available to aid non–movement disorder clinicians in deciding whether or not to refer PD patients for DBS evaluation: the Florida Surgical Questionnaire for Parkinson Disease (FLASQ‐PD)7 and the Stimulus tool.8 We explored the validity and efficacy of these two screening algorithms in a cohort of PD patients evaluated for DBS candidacy at the University of Michigan (UM) Surgical Therapies Improving Movement (STIM) Program, using an affirmative decision to offer DBS surgery after the multidisciplinary STIM team evaluation as the comparative reference standard.

Patients and Methods

Subjects

We conducted a retrospective cohort study of 147 consecutive idiopathic PD (iPD) patients newly referred for consideration of DBS at the UM STIM program between 1 September 2007 and 31 December 2011 (Fig. 1). All referrals for DBS were logged at the time of referral, and all patients were subsequently scheduled for an initial clinic evaluation with a movement disorder neurologist (K.L.C.). At their initial clinic visit, patients were considered to be good candidates for further DBS evaluation if they had motor fluctuations not optimally managed by medications or a severe rest tremor despite high doses of dopaminergic treatment. Medication strategies that had to be tried for wearing‐off symptoms included adjusting levodopa frequency, adding or increasing the dosage of dopamine agonists, and addition of a catechol‐O‐methyl transferase (COMT) or monoamine oxidase (MAO)‐B inhibitor. For dyskinesias, medication strategies that had to be tried included addition of amantadine, decreasing dosage of l‐dopa, and discontinuing COMT inhibitors or MAO‐B inhibitors. Tremor was labeled refractory to medications if it still interfered with quality of life despite a daily l‐dopa dose of >1,000 mg and a trial of trihexyphenidyl. If patients had dementia, clear lack of motor response to l‐dopa (other than medication‐refractory tremor), an unstable psychiatric condition, or a desire for improvement in symptoms not expected to improve with DBS (such as gait instability requiring a walker even in the best medication on state), no further presurgical evaluation was performed. Those individuals who passed the initial screening clinic evaluation then completed a standardized multidisciplinary workup, including 3T MRI of the brain, a standardized OFF‐ON evaluation using the International Parkinson and Movement Disorder Society/UPDRS (MDS‐UPDRS) motor exam to assess for response to dopaminergic therapy,9 a comprehensive neuropsychological evaluation (see Appendix 1 in the Supporting Information), neurosurgical consultation, a formal speech evaluation, and social work consultation. After all these evaluations were completed, prospective candidates were discussed at a multidisciplinary STIM team meeting, where a consensus formal decision was made about whether or not to offer the patient STN‐DBS surgery.

Figure 1.

Flowchart of subjects.

In general, the STIM team used the following criteria for deciding whether or not to offer DBS surgery: (1) motor fluctuations and/or severe dyskinesias that interfere with quality of life despite optimal medical management or medically refractory parkinsonian tremor; (2) a dopaminergic medication response of at least 30% in the ON‐medication state, compared to the OFF‐medication state (patients with medication‐refractory tremor did not need to meet this criteria), as measured by the MDS‐UPDRS, part III; (3) no evidence for dementia based on neuropsychological testing; (4) the absence of refractory or severe psychiatric symptoms based on screening questionnaires (see Appendix 1 in the Supporting Information); and (5) no other medical contraindications for surgery.

The STIM team criteria listed above are intentionally broad, given that many clinical scenarios demand considerations that do not lend themselves to inflexible cutoffs. As an example, the degree of dyskinesia or motor fluctuations warranting surgery depends on individual factors, including occupational status and patient goals. Furthermore, results on cognitive tests might be interpreted differently depending on the patient's educational status. Thus, we did not utilize strict cut‐off values for any motor, cognitive, or psychiatric rating scales, other than the 30% OFF‐ON improvement in the MDS‐UPDRS. The severity of any motor, cognitive, or psychiatric deficits were given differential consideration in the context of the patient's background and goals.

Screening Algorithms

For each patient, FLASQ‐PD and Stimulus scores were calculated through a chart review of the movement disorder neurologist's initial clinical evaluation. FLASQ‐PD is a brief, five‐part evaluation7 that assesses (1) criteria for the diagnosis of PD, (2) potential contraindications to DBS surgery, (3) general patient characteristics, such as age, duration of symptoms, presence of and dyskinesias/fluctuations, (4) favorable or unfavorable characteristics with regard to DBS surgery, such as l‐dopa response, and (5) a history of previous medication trials. Individual scores from parts 3 to 5 are summed, yielding a final FLASQ‐PD score ranging from 0 (worst possible) to 34 (best possible). A cut‐off score of 25 or greater has been suggested to identify optimal candidates and scores of 15 or less indicate poor candidates.7

Stimulus is a two‐part online decision tool developed using the RAND/UCLA Appropriateness method8 and can be found online (http://test.stimulus-dbs.org). Potential DBS candidates must first meet five absolute criteria in part 1: (1) a diagnosis of iPD; (2) troublesome motor symptoms despite optimal pharmacological treatment; (3) clear motor improvement with l‐dopa; (4) the absence of significant medical conditions preventing surgery; and (5) the absence of significant medically resistant mental disease, such as depression or dementia. If these criteria are not met, the patient “does not meet the requirements for consideration of DBS” and no score is given. If a prospective DBS candidate meets all five of the absolute criteria in Stimulus part 1, they are scored on part 2 for seven key variables: age; disease duration; severity of symptoms during the OFF medication state; severity of dyskinesias; l‐dopa‐unresponsive axial symptoms; refractory tremor; and intellectual impairment. Once these are entered, the Stimulus program displays a score from 1 to 9. Scores of 7 or greater are considered “appropriate.” Scores of 4 to 6 are “uncertain” and scores of 3 or less are “inappropriate.”

For our study, all patients presenting for initial clinic evaluation were given FLASQ‐PD and Stimulus part 2 scores, even if they would have been excluded based on FLASQ‐PD section 2 (potential contraindications) or Stimulus part 1. This was done to enhance the generalizability of both screening algorithms given that the interpretation of these sections may vary depending on one's expertise with PD DBS patient selection. As an example, “optimal pharmacological management” might be different for a primary care physician, compared to a movement disorders specialist. Moreover, if Stimulus part 1 criteria were strictly followed, some PD patients whose tremor did not respond to high doses of l‐dopa would not be considered DBS candidates.

Human Subjects and Standard Protocol Approval

The study was approved by the institutional review board of UM.

Statistical Analysis

The decision by the STIM team to offer DBS surgery served as the comparative reference standard for FLASQ‐PD and Stimulus scores. Receiver operating curves (ROCs) were calculated for each scale in order to determine the area under the curve (AUC). The AUC for each of the two scales was estimated using Mann‐Whitney's U test. ROC contrast estimation between the two scales and between the FLASQ‐PD scale with chance alone were calculated using Wald's chi‐square testing. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for FLASQ‐PD scores using two different cutoffs: ≥25 or ≥15. A similar analysis was performed for the Stimulus scale using cutoffs of ≥7 or ≥3. The interscale agreement between the FLASQ‐PD and Stimulus scales was calculated using Cohen's kappa.

In order to explore the effects of variable interpretation and utilization of each of the two scales, we also conducted a sensitivity analysis with two subsets of the overall cohort, excluding subjects who had potential contraindications as assessed in part 2 of the FLASQ‐PD scale or who did not meet all five absolute criteria in Stimulus part 1 criteria. All statistical analyses were performed using SAS 9.3 (SAS Institute Inc., Cary, NC).

Results

Of the 147 PD patients who were newly referred for consideration of DBS by the STIM team between 1 September 2007 and 31 December 2011, 50 were approved and 49 underwent DBS for PD (Fig. 1). The cohort who presented for initial clinical evaluation (n = 130) included 94 men and 36 women with a mean age of 65.2 ± 9.3 years, mean duration of motor symptoms of 10.6 ± 5.9 years, and mean time since PD diagnosis of 9.2 ± 5.6 years. Sixty‐five (50%) were referred by movement disorder neurologists, 46 (35%) by general neurologists, and 19 (15%) by non‐neurologists. Mean FLASQ‐PD score for the cohort was 22.51 ± 4.41. Mean Stimulus score was 6.54 ± 2.41 for the entire cohort. Of the 130 who presented for initial evaluation, only 82 were placed on the DBS evaluation pathway (Fig. 1). The most common reason for not being placed on the DBS pathway was that medications were not optimized (21 patients), though many did not even have motor fluctuations or severe tremor (12 patients). Of the 68 patients who completed the multidisciplinary DBS evaluation, 18 were not approved by the STIM team. Fourteen of these were not offered DBS because of dementia, uncontrolled depression, or <30% improvement with OFF‐ON testing. Two patients were not offered DBS for medical reasons: 1 because of severe obesity and an implanted pacemaker and 1 because of the development of daily syncopal episodes resulting in multiple facial and vertebral fractures. The remaining 2 patients decided to retry medication adjustments that were previously documented as being of no benefit and decided that fluctuations or dyskinesias improved to the point where it did not interfere with quality of life.

The AUC for each scale's ROC were estimated in order to determine the comparative utility of each of the scales as screening tools (Table 1; Fig. 2). The Stimulus scale showed an AUC of 0.8088 (95% confidence interval [CI]: 0.7377, 0.8798), which was 0.1803 (95% CI: 0.0745, 0.2860) higher than the AUC for the FLASQ‐PD scale (chi‐square = 11.16; P = 0.0008). The AUC for the FLASQ‐PD scale proved superior to chance alone by 0.1285 (95% CI: 0.0276, 0.2294; chi‐square, 6.23; P = 0.0125).

Table 1.

ROCs and contrast estimation for FLASQ‐PD and stimulus

	Estimate	Standard Error	Lower 95% CI	Upper 95% CI
FLASQ‐PD	0.6285	0.0515	0.5276	0.7294
Stimulus	0.8088	0.0363	0.7377	0.8798
Δ between stimulus and FLASQ‐PD curves	0.1803	0.0540	0.0745	0.2860
Δ between FLASQ curve and chance	0.1285	0.0515	0.0276	0.2294

Figure 2.

ROC curves for FLASQ‐PD and stimulus.

Data on the sensitivity, specificity, PPV, and NPV for each of the two scales are presented in Table 2. The greatest degree of sensitivity was achieved by a Stimulus cut‐off score of ≥3 (sensitivity = 100%; specificity = 12.5%), and the greatest degree of specificity was demonstrated by the FLASQ‐PD scale using a cut‐off score of ≥25 (sensitivity = 50.0%; specificity = 68.8%). Using a cut‐off score of ≥7, the Stimulus scale showed the highest PPV (59.7%) and highest NPV (92.5%). Interscale agreement was poor between the lower (FLASQ‐PD ≥15 and Stimulus ≥3; Cohen's kappa: −0.046; 95% CI: −0.079, −0.012) and higher cut‐off scores (FLASQ‐PD ≥25 and Stimulus ≥7; Cohen's kappa: 0.218; 95% CI: 0.067, 0.369) for both scales.

Table 2.

Measures of validity and efficacy for FLASQ‐PD and stimulus

	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
FLASQ‐PD score of ≥15	96.0	2.5	38.1	50.0
FLASQ‐PD score of ≥25	50.0	68.8	50.0	68.8
Stimulus score of ≥3	100	12.5	41.6	10/0 = infinity
Stimulus score of ≥7	92.0	61.3	59.7	92.5

Sensitivity Analysis

Of the 130 subjects who presented for the initial clinic evaluation, 16 reported at least one potential contraindication on the FLASQ‐PD scale. After excluding these subjects from analysis, AUCs (n = 114) for FLASQ‐PD (AUC = 0.6357; 95% CI: 0.5199, 0.7515) and Stimulus (AUC = 0.7831; 95% CI: 0.7012, 0.8651) remained fairly similar to those of the overall cohort. If those subjects who did not meet all five absolute criteria on Stimulus part 1 were excluded, the AUCs (n = 81) for FLASQ‐PD (AUC = 0.5442; 95% CI: 0.4151, 0.6733) and Stimulus (AUC = 0.7203; 95% CI: 0.6059, 0.8347) were slightly lower than the values seen for the overall cohort.

Discussion

The Stimulus and FLASQ‐PD screening batteries were developed to aid clinicians in assessing DBS candidacy in PD. Though intended for non–movement disorder specialists, both scales contain elements that may be challenging for those without movement disorders expertise to answer, limiting their sensitivity. The FLASQ‐PD has a potential contraindications section (part 2), where if the answer is “yes” to any one of these questions, the patient may not be a good candidate for DBS. However, many of these items may not be routinely assessed by non–movement disorder clinicians. For the Stimulus tool, if the part 1 absolute criteria are not met, then the patient should not be considered for DBS. One criterion is “still has motor improvement with levodopa.” If strictly followed, some tremor‐predominant PD patients would not be referred for DBS. In order to explicitly examine the utility of these scales as screening instruments for non–movement disorder providers, we decided to administer the scales to our entire cohort, even if they would have been excluded based on FLASQ‐PD part 2 or Stimulus part 1. In this particular cohort of patients, when compared to the decision of a single‐center multidisciplinary DBS team, the Stimulus scale at a cut‐off score of ≥7 is a better screening test than FLASQ‐PD, with a larger AUC (0.8088) as well as PPV (59.7%) and NPV (92.5%). This finding is not entirely surprising given that the Stimulus part 1 criteria closely resemble the criteria that our STIM team uses to decide DBS candidacy. However, a sensitivity analysis excluding those subjects that did not pass Stimulus part 1 and excluding those with potential contraindications on the FLASQ‐PD resulted in AUCs that were similar or worse.

The Stimulus scale also has a few other features that may make it more practical for clinical use by non–movement disorder clinicians. Most important, it focuses only on seven key variables, so is less complex and takes less time to complete, compared to FLASQ‐PD. It is also available online and has a color‐coded visual analog scale to aid interpretation of results. FLASQ‐PD has more questions and is of longer duration. It also includes features not likely to be evaluated by non–movement disorder specialists, including primitive reflexes and ideomotor apraxia.

The validity and efficacy of the FLASQ‐PD in determining appropriate DBS candidates has not been reported on previously. We found a high sensitivity (96.0%) using a FLASQ‐PD score of greater than 15 as a cutoff for referral, meaning that it would be unlikely to exclude appropriate referrals. However, the specificity (2.5%) at that cutoff suggests that many inappropriate candidates would also be referred. Based on their initial report presenting the development of the FLASQ‐PD scale, Okun et al. suggested that scores ≥25 reflected the best DBS candidates.7 At this value in our cohort, specificity was adequate (68.8%), but sensitivity was low (50%), suggesting that a number of appropriate candidates would not be referred.

There are several limitations to our study. Patients who were thought to be appropriate DBS candidates at the initial screening clinic evaluation, but discontinued the multidisciplinary pathway evaluation, would not have been discussed by the STIM team. There were 14 such candidates, and it remains unknown whether the STIM team would have offered them DBS. Yet, for the purpose of this study, all of these patients were considered “not‐offered” candidates. It should be noted that not all 130 patients underwent MDS‐UPDRS I, II, and III examinations at the time of their initial clinic assessment. Subsequently, we cannot draw more‐detailed inferences about specific patterns of motor and nonmotor impairments that might alter the likelihood of STIM team approval. Figure 1, however, does provide the reasons given for those candidates who were ultimately not offered surgery. Of the 18 subjects who completed the evaluative pathway, but were not offered DBS, 9 were not offered surgery because of severity of cognitive or psychiatric problems, highlighting the importance of neuropsychological testing in our center's final decisions. Ascertaining positive and negative predictive values depend on an approximation of prevalence. We have no data on those individuals who were referred, but did not present, for initial clinical evaluation or for whom data are missing. Our study explores the validity (i.e., sensitivity and specificity) as well efficacy (NPV and PPV) of FLASQ‐PD and Stimulus, but its design does not allow for an exploration of the reliability of these two scales (i.e., their ability to produce consistent results in repeated applications). Prospective studies in primary care– and community‐based referral settings will be useful to help answer these questions.

These decision tools are designed to be applied prospectively by a non–movement disorder specialist. In our study, FLASQ‐PD and Stimulus were completed retrospectively with the data extracted from a movement disorder specialist's clinical note by an investigator not blinded to the outcome. It is impossible to know whether the referring physicians would have prospectively completed the screening tools in a similar manner. However, the sensitivity (100%) and specificity (12.5%) of Stimulus in our study using a cutoff of 3 or greater was almost identical to the sensitivity (99%) and specificity (12%) found in a European study, where Stimulus was applied in a prospective fashion with general neurologists completing the tool before referral.10 Furthermore, the reference standard in our study was the consensus decision of a single‐center multidisciplinary DBS team. No standardized guidelines exist for the selection of DBS candidates for PD, and as a result, practices may vary between DBS centers. Our team performs mostly STN DBS. Other centers may routinely perform globus pallidus interna DBS in addition to STN DBS and may apply different criteria for DBS candidacy depending on the target. Our study also carries inherent selection bias, in that it includes only patients who have already been referred and does not address the issue of the many patients who may be appropriate, but who were either not offered or declined referral. Given these considerations, the results of our study may not be generalizable.

Although our results will need to be validated at other DBS centers utilizing non–movement disorder specialists in a prospective fashion, we found that the Stimulus decision tool was a better screening measure to assess DBS candidacy in PD patients than the FLASQ‐PD. The implementation of easy‐to‐use algorithms such as the Stimulus scale could be an effective way to improve access of care to DBS centers among patients with PD, and future studies focused on these topics are warranted.

Author Roles

(1) Research Project: A. Conception, B. Organization, C. Execution; (2) Statistical Analysis: A. Design, B. Execution, C. Review and Critique; (3) Manuscript: A. Writing of the First Draft, B. Review and Critique.

R.R.C.: 1B, 1C, 2C, 3A, 3B

V.K.: 1C, 2A, 2B, 3A, 3B

P.G.P.: 1A, 2C, 3B

K.L.C.: 1A, 1B, 1C, 2A, 2C, 3A, 3B

Data Access

Dr. Chou had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Disclosures

Funding Sources and Conflicts of Interest: The authors report no sources of funding and no conflicts of interest.

Financial Disclosures for previous 12 months: V.K. receives research support from a National Institutes of Health (NIH) T32 training grant (NS007222), the American Academy of Neurology Clinical Research Training Fellowship, and the Blue Cross and Blue Shield of Michigan Foundation. P.G.P. receives research support from the Coulter Foundation, the A. Alfred Taubman Medical Institute, and St. Jude Medical Inc.; receives royalties from Demos Health; and serves as a consultant for Medtronic, Inc. and Monteris, Inc. K.L.C. receives research support from the NIH (NS44504‐08 and 5R44NS070438) and the Michael J. Fox Foundation; participates as a site principal investigator in clinical trials sponsored by the Huntington Study Group (2CARE); receives royalties from UpToDate and Demos Health; and serves as a consultant for Medtronic, Inc., and Accordant.

Supporting information

Appendix 1. University of Michigan STIM team neuropsychological evaluation.