Potential reliability and validity of a modified version of the Unified Parkinson’s Disease Rating Scale that could be administered remotely

Abstract

Background

By permitting remote assessments of patients and research participants, telemedicine has the potential to reshape clinical care and clinical trials for Parkinson disease. While the majority of the motor Unified Parkinson’s Disease Rating Scale (UPDRS) items can be conducted visually, rigidity and retropulsion pull testing require hands-on assessment by the rater and are less feasible to perform remotely in patients' homes.

Methods

In a secondary data analysis of the Comparison of the Agonist pramipexole vs. Levodopa on Motor complications in Parkinson’s Disease (CALM-PD) study, a randomized clinical trial, we assessed the cross-sectional (baseline and 2 years) and longitudinal (change from baseline to 2 years) reliability of a modified motor UPDRS (removing rigidity and retropulsion items) compared to the standard motor UPDRS (all items) using intraclass correlation coefficients (ICC), stratified by treatment group. Internal consistency of the modified UPDRS (mUPDRS) was measured using Cronbach’s alpha, and concurrent validity was assessed using Pearson’s correlation coefficient (r) between the standard motor UPDRS and mUPDRS.

Results

The mUPDRS versus standard motor UPDRS is cross-sectionally (ICC ≥ 0.92) and longitudinally (ICC ≥ 0.92) reliable for both treatment groups. High internal consistencies were also observed (α ≥ 0.96). The mUPDRS had high concurrent validity with the standard UPDRS at both time points and longitudinally (r ≥ 0.93, p < 0.0001).

Conclusions

A modified version of the motor UPDRS without rigidity and retropulsion pull testing is reliable and valid and may lay the foundation for its use in remote assessments of patients and research participants.

Introduction

The high burden of clinical trials on participants with a neurological disease and their caregivers is a significant barrier to study participation. The costs of clinical trials and clinical drug development are rising rapidly [1], in part due to slow recruitment and poor retention of study participants and the resultant large sample sizes needed to demonstrate an effect. Home-based assessments via web-based video-conferencing and remote mobility may reduce study costs by minimizing the number of required in-person visits, thereby enhancing enrollment and subsequently decreasing the time needed to meet enrollment goals, and decreasing administrative costs. A recent study suggested that home-based visits may be the most effective means of enhancing enrollment into clinical trials for individuals with neurodegenerative diseases [2]. Additionally, remote monitoring of clinical trial participants may permit the use of a single centralized rater for important clinical trial outcomes. A randomized, controlled trial of web-based video conferencing for the management of Parkinson’s disease (PD) in community-dwelling individuals and nursing home residents suggested that remote assessment of the motor Unified Parkinson’s Disease Rating Scale (UPDRS) is feasible, valid, and reliable (using a trained registered nurse to perform the rigidity assessment and conducting the pull test) compared to the in-person assessment [3,4].

While the majority of the motor UPDRS items can be conducted visually, rigidity and retropulsion pull testing require hands-on assessment by the rater. However, these tests may not add significant diagnostic or predictive benefit; specifically, an abnormal pull test was not associated with future fall risk [5] and rigidity, while responsive to dopaminergic therapy, is of unclear clinical and functional significance [6]. Prior to implementing a modified motor UPDRS (without rigidity or retropulsion pull testing) for remote assessment in clinical trials, it is necessary to establish its reliability and validity in the context of a clinical trial where an experimental therapy has been shown to significantly impact the standard UPDRS. Therefore, we evaluated the reliability and validity of the UPDRS without rigidity and pull testing through a secondary analysis of the Comparison of the Agonist pramipexole vs. Levodopa on Motor complications in Parkinson’s Disease (CALM-PD) study, a RCT of levodopa versus pramipexole.

Methods

The CALM-PD study [7] compared the development of dopaminergic motor complications in 301 individuals with early PD randomized to initial levodopa or initial pramipexole. While the primary analysis demonstrated a lower incidence of motor complications with pramipexole, an early (10 weeks) and persistent benefit at 2 years on the total and motor UPDRS (5.0 and 3.9 point treatment effect, respectively) was observed favoring levodopa. This benefit persisted at the 4-year time point (5.9 for total and 4.9 for motor treatment effect) [8]. This study represents an ideal opportunity to assess whether the use of a modified UPDRS, with rigidity and pull testing removed, affects the results of a clinical trial where a profound effect on the outcome of interest is seen.

In the original CALM-PD study, UPDRS scores were evaluated via in-person clinic visits from baseline through 2 years of follow-up. All 14 items (items 18–31) of the motor UPDRS examination were assessed for each patient, defined as the standard (total) motor UPDRS. The modified motor UPDRS (mUPDRS) is defined as the sum of all motor items except neck rigidity, right upper extremity rigidity, left upper extremity rigidity, right lower extremity rigidity, left lower extremity rigidity (all measured by item 22) and retropulsion pull test for postural stability (item 30).

Reliability and consistency

To establish the cross-sectional reliability and assess the agreement between the standard motor UPDRS and mUPDRS at baseline and 2 years for both the levodopa (n=150) and pramipexole (n=151) treatment groups, the intraclass correlation coefficient (ICC) and 95% confidence intervals (CI) were calculated. Longitudinal reliability was assessed by calculating the ICC of the change in scores from baseline to 2 years, stratified by treatment groups (levodopa: n=131 and pramipexole: n=127 due to loss to follow-up), between the standard motor UPDRS and mUPDRS. The threshold for reliability was set by an ICC greater than or equal to 0.7 (“excellent agreement”) [9]. The MIXED procedure in SAS version 9.2 (SAS Institute, Inc., Cary, NC) was used for ICC calculations. Internal consistency (homogeneity of items in relation to the measure) was measured using Cronbach’s alpha for the mUPDRS at baseline, 2-year follow-up, and longitudinally (change between baseline and 2-year follow-up). A value of 0.80 or higher was considered highly consistent.

Validity

To assess concurrent validity, the motor mUPDRS was correlated with the standard motor UPDRS. Pearson’s correlation coefficients (r) and 95% CIs were calculated using the CORR procedure. The magnitude of correlation coefficients was expected to be high (r > 0.80) to demonstrate strong convergent validity.

Kappa and percent agreement

Agreement on the direction of individual change from baseline to 2-year follow-up was assessed between the standard and mUPDRS. A decline in UPDRS score from baseline to 2-year follow-up meant improved motor function while an increase in UPDRS score meant a worsening of motor function. Analyses were stratified by treatment group and performed using the kappa test in PROC FREQ to obtain kappa coefficients and 95% confidence intervals. Percent agreement was calculated by summing the concordant (diagonal) elements and dividing by the sample size for that treatment. Kappa estimates above 0.8 and percent agreement above 90% were considered “excellent agreement” [10].

Results

The distribution of UPDRS motor item scores by treatment group is shown in Table 1. All motor items, except for speech in the pramipexole treatment group, demonstrate improvement in function from baseline to 2 years. Improvements were seen the most for rigidity, finger taps, hand pronate/ supinate, and body bradykinesia, which was consistent between treatment groups. Total motor UPDRS and mUPDRS scores were similar at baseline between treatment groups but lower in the levodopa arm at follow-up. The ICC (95% CI) for the cross-sectional reliability under the levodopa treatment arm at baseline and 2 years was 0.93 (0.90–0.95) and 0.92 (0.89–0.94), respectively. Similar ICC’s were observed at baseline and 2 years with pramipexole treatment (0.93 (0.91–0.95) and 0.92 (0.89–0.95), respectively). Longitudinal reliability also demonstrated excellent agreement for levodopa (ICC = 0.92, 95% CI: 0.89–0.94) and pramipexole (ICC = 0.93, 95% CI: 0.90–0.95). Internal consistency of the mUPDRS was similar between the levodopa (α_baseline = 0.98, α_{2 years} = 0.98, α_Δ = 0.96) and pramipexole (α_baseline = 0.98, α_{2 years} = 0.98, α_Δ = 0.98) treatment arms. Evidence of concurrent validity was present in both treatment arms, with statistically significant correlations between the UPDRS and mUPDRS at baseline (r_lev = 0.96, 95% CI: 0.94–0.97; r_pram = 0.96, 95% CI: 0.95–0.97), 2 years (r_lev = 0.95, 95% CI: 0.93–0.97; r_pram = 0.96, 95% CI: 0.95–0.97), and longitudinally (r_lev = 0.93, 95% CI: 0.90–0.95; r_pram = 0.96, 95% CI: 0.94–0.97). As variability between UPDRS and mUPDRS may be higher in subjects who are tremor-free, we conducted a stratified analysis on this subsample, which showed only small decreases in Pearson’s correlations (from 0.01 to 0.03), suggesting little evidence to conclude a difference in variability between the subsample of patients without tremor and the total sample. Figures 1a (baseline) and 1b (2 years) illustrate scatter plots of total motor versus mUPDRS scores, which demonstrate tight correlations about the best fit line, similar for both treatment groups. The 95% prediction interval is nearly identical between groups as well as time periods. Figure 1c illustrates the change from baseline to 2 years in the UPDRS versus mUPDRS total motor scores about the identity line. Most individual ΔmUPDRS scores were slightly underestimated compared to ΔUPDRS scores, but appeared non-differential with respect to treatment.

Table 1.

Distribution of UPDRS motor item scores by treatment group at baseline and 2-year follow-up

	Levodopa (Mean ± SD)			Pramipexole (Mean ± SD)
UPDRS Item (Measure)	Baseline (N=150)	2-Year (N=131)	Δ (N=131)	Baseline (N=151)	2-Year (N=127)	Δ (N=127)
18 (Speech)	0.85 ± 0.64	0.60 ± 0.68	−0.24 ± 0.61	0.85 ± 0.67	0.88 ± 0.73	0.03 ± 0.57
19 (Facial expression)	1.29 ± 0.70	0.92 ± 0.78	−0.37 ± 0.82	1.36 ± 0.74	1.24 ± 0.73	−0.11 ± 0.63
20a (Tremor at rest: face, lips, chin)	0.23 ± 0.55	0.10 ± 0.35	−0.13 ± 0.46	0.12 ± 0.37	0.04 ± 0.20	−0.06 ± 0.27
20b (Tremor at rest: R hand)	0.94 ± 1.01	0.55 ± 0.83	−0.32 ± 0.75	0.92 ± 1.04	0.50 ± 0.72	−0.43 ± 0.90
20c (Tremor at rest: L hand)	0.73 ± 0.89	0.40 ± 0.68	−0.37 ± 0.75	0.79 ± 1.00	0.50 ± 0.79	−0.24 ± 0.72
20d (Tremor at rest: R foot)	0.24 ± 0.57	0.11 ± 0.40	−0.10 ± 0.56	0.18 ± 0.49	0.05 ± 0.23	−0.14 ± 0.47
20e (Tremor at rest: L foot)	0.23 ± 0.55	0.13 ± 0.40	−0.13 ± 0.57	0.20 ± 0.48	0.10 ± 0.40	−0.08 ± 0.39
21a (Action tremor: R)	0.57 ± 0.74	0.34 ± 0.59	−0.22 ± 0.64	0.49 ± 0.68	0.30 ± 0.53	−0.20 ± 0.62
21b (Action tremor: L)	0.58 ± 0.65	0.33 ± 0.53	−0.26 ± 0.59	0.52 ± 0.63	0.33 ± 0.56	−0.20 ± 0.49
22a (Rigidity: neck)	1.06 ± 0.80	0.82 ± 0.82	−0.21 ± 0.69	1.09 ± 0.81	0.89 ± 0.80	−0.20 ± 0.78
22b (Rigidity: R upper extremity)	1.25 ± 0.82	0.83 ± 0.78	−0.37 ± 0.74	1.30 ± 0.78	0.99 ± 0.75	−0.30 ± 0.71
22c (Rigidity: L upper extremity)	1.18 ± 0.84	0.76 ± 0.75	−0.42 ± 0.80	1.14 ± 0.82	0.89 ± 0.80	−0.21 ± 0.61
22d (Rigidity: R lower extremity)	0.80 ± 0.83	0.50 ± 0.69	−0.27 ± 0.78	0.78 ± 0.77	0.67 ± 0.79	−0.10 ± 0.72
22e (Rigidity: L lower extremity)	0.74 ± 0.82	0.54 ± 0.71	−0.22 ± 0.82	0.81 ± 0.83	0.68 ± 0.82	−0.08 ± 0.70
23a (Finger taps: R)	1.25 ± 0.97	0.79 ± 0.82	−0.46 ± 0.84	1.26 ± 0.85	0.84 ± 0.74	−0.41 ± 0.75
23b (Finger taps: L)	1.25 ± 0.95	0.87 ± 0.79	−0.41 ± 0.90	1.16 ± 0.90	0.85 ± 0.84	−0.28 ± 0.90
24a (Hand grips: R)	0.90 ± 0.84	0.51 ± 0.74	−0.39 ± 0.79	0.99 ± 0.78	0.77 ± 0.74	−0.23 ± 0.69
24b (Hand grips: L)	1.00 ± 0.88	0.63 ± 0.74	−0.39 ± 0.89	0.98 ± 0.87	0.79 ± 0.72	−0.14 ± 0.73
25a (Hand pronate/supinate: R)	0.93 ± 0.85	0.50 ± 0.75	−0.42 ± 0.73	1.02 ± 0.79	0.71 ± 0.76	−0.30 ± 0.79
25b (Hand pronate/supinate: L)	1.00 ± 0.92	0.58 ± 0.71	−0.42 ± 0.82	1.00 ± 0.91	0.81 ± 0.83	−0.18 ± 0.87
26a (Leg agility: R)	0.71 ± 0.78	0.42 ± 0.64	−0.30 ± 0.79	0.87 ± 0.80	0.56 ± 0.70	−0.29 ± 0.77
26b (Leg agility: L)	0.84 ± 0.78	0.55 ± 0.68	−0.32 ± 0.86	0.83 ± 0.77	0.71 ± 0.83	−0.09 ± 0.70
27 (Arise from chair)	0.22 ± 0.42	0.12 ± 0.31	−0.08 ± 0.40	0.28 ± 0.47	0.23 ± 0.38	0.01 ± 0.42
28 (Posture)	0.69 ± 0.62	0.51 ± 0.60	−0.17 ± 0.61	0.75 ± 0.66	0.67 ± 0.62	−0.01 ± 0.55
29 (Gait)	0.61 ± 0.57	0.31 ± 0.44	−0.29 ± 0.50	0.65 ± 0.58	0.50 ± 0.56	−0.13 ± 0.49
30 (Postural stability)	0.28 ± 0.55	0.13 ± 0.35	−0.09 ± 0.45	0.29 ± 0.52	0.18 ± 0.42	−0.09 ± 0.46
31 (Body bradykinesia)	1.59 ± 0.67	1.00 ± 0.80	−0.59 ± 0.81	1.69 ± 0.72	1.30 ± 0.76	−0.37 ± 0.70
18–31 (Standard Motor UPDRS)	21.97 ± 9.64	13.85 ± 8.98	−8.12 ± 8.47	22.27 ± 9.16	16.97 ± 8.89	−4.73 ± 7.47
18–31, excl. 22 & 30 (Modified Motor UPDRS)	16.66 ± 7.42	10.27 ± 6.91	−6.30 ± 7.40	16.85 ± 7.04	12.69 ± 6.66	−3.74 ± 5.90

UPDRS, Unified Parkinson’s Disease Rating Scale; SD, standard deviation; R, right; L, left.

Figure 1.

Scatterplots for (A) modified motor UPDRS (mUPDRS) versus standard motor UPDRS at baseline, (B) mUPDRS versus UPDRS at 2-year follow-up, and (C) change from baseline to 2-year follow-up for mUPDRS versus UPDRS. Solid lines represent best-fit linear regression line (plots A and B) and line of identity (plot C). For plots A and B, dashed line represents 95% confidence interval and dotted line represents 95% prediction interval about the best-fit line.

The direction of individual change (improved, no change, worsened) from baseline to 2 years between the standard motor UPDRS and mUPDRS for the levodopa treatment group produced a kappa statistic of 0.90 (95% CI: 0.80, 0.99) and percent agreement of 96.18%. Less favorable results were obtained for the pramipexole arm, yielding a kappa statistic of 0.81 (95% CI: 0.70, 0.91) and percent agreement of 89.60%. The kappa for both treatment groups combined was 0.85 (95% CI: 0.77, 0.92) with a percent agreement of 92.97%.

Discussion

Based on the evidence from this study, a modified version of the UPDRS without rigidity and retropulsion pull testing is reliable and valid, both at cross-sectional time points and longitudinally. Items scores by treatment group demonstrated a wide range of clinical values that support the applicability of this data set for comparing the mUPDRS with the UPDRS. A greater 2-year treatment effect with levodopa is also consistent with current clinical trial data [11]. The large ICC’s, measuring reliability of the mUPDRS compared to the UPDRS, were expected since the mUPDRS is a large subset of the standard motor UPDRS. The internal consistency of the mUPDRS was also high when measuring the cross-sectional and longitudinal effects between scales for both treatments. In addition, we have demonstrated statistically significant correlations of the mUPDRS with the UPDRS for both treatment groups. Narrow 95% prediction intervals of approximately ± 5 points suggest only slight deviations between the two scales cross-sectionally. Although the ICC’s and correlation coefficients for ΔmUPDRS versus ΔUPDRS were strong, we expected to see observations lie closer to the identity line, representing equal change in scores from baseline to 2 years. However, a majority of observations fell below this line due to the higher scores, on average, from items 22a–e (rigidity) that were dropped in the mUPDRS. In light of this, kappa statistics assessing the direction of change from baseline to 2 years demonstrated excellent agreement (kappa > 0.8) between both measures and both treatment arms. The 95% CI for the pramipexole and combined levodopa and pramipexole kappa statistic, however, fell below the cut-off for excellent agreement and should thus be interpreted cautiously. Strong percent agreement scores of the two measures were observed for the levodopa arm and combined analysis of levodopa and pramipexole. Pramipexole alone was slightly below the a priori cutoff set. These results suggest that therapeutic treatment regime, either with levodopa or a dopamine agonist such as pramipexole, may be of little importance when considering remote evaluation of PD patients.

Although this study has limitations, it is a critical first step in the utilization of web-based videoconferencing for the assessment of UPDRS outcomes remotely. First, we recognize that with the recent development of the Movement Disorder Society-UPDRS (MDS-UPDRS), the standard UPDRS may eventually become obsolete. However, correlational studies have shown the motor UPDRS to have both excellent correlation (Pearson’s r = 0.97, p < 0.001) [12] and high internal consistency (Cronbach’s alpha = 0.90) [13] with the motor MDS-UPDRS, suggesting that reliability and validity of the mUPDRS in our study may be applicable to the MDS-UPDRS, but should be assessed in a separate trial using this scale. Also, while Cubo et al. [14] has recently demonstrated that in home web-based assessments for assessing PD-related impairments is feasible, our study did not address feasibility. There is the possibility of complications associated with technology, such as glitches in internet connection and sound or visual problems, lack of access to the necessary technology for study subjects particularly in non-westernized regions, and limited knowledge of its use. We are also making the assumption that visually rated items are rated the exact same when rated on a screen versus in-person, representing a best case scenario. Therefore, if there were any variability in the assessment of the other items between remote versus in-person evaluations, due to factors such as positioning or 2D versus 3D perception, the reliability and validity of the mUPDRS would be lower than reported in this study. In addition, little is known about the compliance with telemedicine use in this population and whether PD patients would rather be seen in person for clinical evaluations or in the comfort of their own home. The setting in which patient assessments take place (at home versus in a clinic) has been shown to influence PD symptom severity [14], resulting in motor score variability between environments. As our study only used data from patients with early PD, this remote approach should be tested in a longitudinal study using a wide range of patients of varying severity and with rigorous psychometric analyses, particularly in more advanced PD patients who are burdened with travel constraints due to progressive disability.

The assessment of clinical outcomes remotely via web-based videoconferencing can increase access to clinical trials for individuals who would otherwise find travel to study sites burdensome, and therefore may improve recruitment for future PD trials. Remote web-based assessments will also obviate the need for frequent in-person assessments and hence improve retention. This is particularly relevant in studies requiring frequent monitoring and long-term follow-up, such as NET-PD clinical trials, where evaluation of long-term outcomes is critical. In order to maintain stability and prove feasible, implementing remote assessments into clinical practice may require standardized procedures for participants, including having adequate training and being responsive to technical difficulties. Confirming the reliability and validity of the mUPDRS may lay the foundation for its use in remote assessments of PD patients and research participants and allow for more efficient clinical trials.