Missing Data in the Unified Dysksinesia Rating Scale (UDysRS)

ABSTRACT

Objective

Identify the number of allowable missing values still permitting valid surrogate score calculation for the Unified Dyskinesia Rating Scale (UDysRS).

Background

Missing data frequently occur in Parkinson's disease rating scales, and they compromise data validity, risking data exclusion from final analyses.

Methods

Accessing the International Parkinson and Movement Disorder Society‐sponsored UDysRS translation databases (3313 complete scores). We sequentially removed item scores, consistently or randomly from subjective and objective sections. Lin's Concordance Correlation Coefficient compared prorated scores with complete scores. We considered prorated scores valid when Coefficients exceeded 0.95.

Results

For consistently missing items, three from the subjective section and five from the objective section are allowable. For randomly missing items, seven from the subjective section and four from the objective section are permissible.

Conclusions

We provide guidelines for constructing valid surrogate summary UDysRS scores with clear thresholds for retaining or rejecting scores based on missing values.

Introduction

The Unified Dyskinesia Rating Scale (UDysRS) was developed as a comprehensive rating tool of dyskinesia in Parkinson's Disease (PD).1 The scale was developed in English with a clinimetric program to provide validated non‐English translations.2, 3 The UDysRS is often paired with the Movement Disorder Society Revision of the Unified Parkinson's Disease Rating Scale (MDS‐UPDRS) for a comprehensive evaluation of both parkinsonism and dyskinesia.3 In prior studies, we examined the threshold of permissible missing values for the MDS‐UPDRS.4 We now present the same type of analysis for the UDysRS. In parallel to our earlier MDS‐UPDRS analysis, we approach the problem of missing values with a prorated score, derived by taking the mean of the observed scores and substituting that value for all the missing values in the scale. We test the prorated method using a data‐based approach and a rigorous threshold for handling missing values in this scale.

Methods

The UDysRS Dataset

We accessed the cross‐sectional combined translation dataset (n = 3313) of fully completed UDysRS scores from 13 languages (Traditional Chinese [n = 250], French [n = 250], German [n = 284], Greek [n = 260], Hungarian [n = 256], Italian [n = 252], Japanese [n = 250], Korean [n = 250], Portuguese [n = 256], Russian [n = 251], Slovak [n = 251], Spanish [n = 253], Turkish [n = 250]).3

The UDysRS consists of two sections (i.e., subjective and objective sections). The subjective section has 15 items (Part I: items 1–11 and Part II: items 12–15 reflecting patient perceptions) of ON (medication‐effective state) and OFF (medication‐ineffective state) dyskinesia impact on salient activities. The objective section has 11 items (Part III: items 16–22 and Part IV: items 23–26) reflecting the objective rater‐based assessment of impairment and disability from dyskinesia. Each item has 0 to 4 rating options with the clinical concepts that 0 = normal, 1 = slight, 2 = mild, 3 = moderate, and 4 = severe. To estimate the number of allowable missing values for each section, we first computed the score as the sum of each patient's individual item scores for each section. This score, with complete data, was used as the gold standard to which the prorated scores, based on purposeful deletion of item scores (missing values), could be compared. In this process, the prorated score was calculated as the sum of the available scores multiplied by the number of total items in this UDysRS section, and this result was divided by the number of items with actual scores. Comparison between the missing value‐based prorated score and the complete score was evaluated using Lin's Concordance Correlation Coefficient (CCC).5 The CCC measures the exact matching between values (scores generated with all data vs. prorated scores with missing values). We set our critical CCC level at ≥ 0.95, interpreted as near‐perfect agreement between the missing value‐based and the full data based scores.5 To accommodate different clinical situations, we applied two approaches to calculate the missing values prorated scores. First, we systematically omitted a consistent item score (e.g., time spent with on‐dyskinesia) for all cases from a given section of the UDysRS starting with one missing item and extending up to the maximal number of missing values that still maintained a CCC ≥ 0.95 for all combinations of consistently missing items. This approach modeled the clinical situation of a consistent elimination of items in each Section across all patients, as might occur with the practical constraints of field work or telemedicine where certain questions could be considered too sensitive to apply outside the office setting, where safety concerns (e.g., ambulation) preclude inclusion, or where an item or items are inadvertently excluded in the questionnaire design. We refer to this approach of omitting items as consistently missing.

In the second approach, we omitted a given number of item scores randomly selected across cases (e.g., speech from one patient, handwriting from another), starting with one selected omission and extending up to the maximal number of missing values that retained CCC ≥ 0.95. This analysis mimicked a clinical trial or day‐to‐day clinic encounter where raters mistakenly overlook items so that the final sample has missing values that vary across raters and sites. We refer to this approach of omitting items as randomly missing. For the randomly missing approach, the CCC was calculated with 1000 randomly selected replications of missing items in all combinations for each UDysRS section.

Results

Consistent Deletion of the Same Item Across All Patients

In Table 1, when given items were consistently deleted from the UDysRS subjective section (15 items total), the minimum CCC remained ≥ 0.95 across for three missing item. With four consistently deleted items, the minimum CCC fell below the ≥ 0.95 threshold and did not allow an adequate prorated score to match the actual subjective score generated by complete data. Similarly, for the objective section (11 items total), when any five items were consistently deleted from the data set, the minimum CCC remained ≥ 0.95, but an accurate prorated score could not be obtained if six items were consistently absent.

Table 1.

Number of allowable missing items to calculate UDysRS score when the same item is consistently missing across all patients

Subjective section # of missing items	CCC
	Min	Median	Mean	Max.
1	0.994	0.997	0.997	0.998
2	0.979	0.994	0.993	0.996
3	0.952	0.990	0.989	0.996
4	0.912	0.985	0.984	0.994

Objective section Number of missing items	CCC
	Min.	Median	Mean	Max.
1	0.996	0.997	0.997	0.998
2	0.988	0.994	0.993	0.995
3	0.980	0.989	0.989	0.993
4	0.971	0.984	0.983	0.991
5	0.959	0.976	0.976	0.988
6	0.940	0.966	0.965	0.983

White rows indicate that the minimal CCC falls at or above 0.95 and is acceptable. Grey rows indicate the point when the number of missing items renders a minimal CCC below threshold for valid calculation of the UDysRS section score.

Abbreviations: CCC, Lin's Concordance.

Random Deletion of Items

In Table 2, when item scores were deleted at random within a given section, for the subjective section, a total section score could be validly calculated if up to and including seven items per patient were missing. Likewise, for the objective section, a total section score could still be validly calculated if there were four randomly deleted items.

Table 2.

Number of allowable missing items to calculate UDysRS score when items are randomly missing across all patients

Subjective section Number of missing items	CCC
	Min	Median	Mean	Max.
1	0.996	0.997	0.997	0.997
2	0.992	0.993	0.993	0.994
3	0.988	0.989	0.989	0.990
4	0.982	0.984	0.984	0.985
5	0.976	0.978	0.978	0.980
6	0.968	0.971	0.971	0.973
7	0.959	0.962	0.962	0.965
8	0.946	0.951	0.951	0.954

Objective section Number of missing items	CCC
	Min.	Median	Mean	Max.
1	0.993	0.994	0.994	0.994
2	0.984	0.986	0.986	0.987
3	0.973	0.975	0.975	0.977
4	0.957	0.961	0.961	0.964
5	0.934	0.941	0.941	0.945

White rows indicate that the minimal CCC falls at or above 0.95 and is acceptable. Grey rows indicate the point when the number of missing items renders a minimal CCC below threshold for valid calculation of the UDysRS section score.

Abbreviations: CCC, Lin's Concordance.

Discussion

The results of our analysis show that the UDysRS is robust for randomly deleted items, and a valid surrogate score can still be calculated when up to seven items from the subjective (Parts 1 and 2) and four items from the objective (Parts 3 and 4) are missing. Often, in clinical trials and clinical practice settings, rapid reading and lack of checking lead to errors on a random basis, and our findings document that mistakes of this type can be largely tolerated. Importantly, however, there is no capacity to reconstitute the subjective score if eight or more items are missing and it is equally impossible to use a complete subjective score to predict a fully‐missing objective score or vice versa.

Whereas most clinical errors will be random, there are instances when a consistent missing value could occur. In the case of culturally sensitive questions, a given group might want to purge a question from the subjective section of the UDysRS, or if one of the motor tasks related to the scale (communication, dressing, drinking, walking) were deemed impractical, a group might want to delete that task systematically. The data document the limits of consistent deletions, and only three subjective questions can be deleted without compromising the overall validity of a calculable surrogate score. For the objective component, only five items can be deleted, and more missing values negate any possibility of using the scores to generate an accurate summary rating. The much smaller capacity for accommodating missing values in the consistently missing subjective items, compared to the objective items, likely reflects the high specificity of some tasks to dyskinesia severity in many patients.

For our missing item analysis, we divided the UDysRS into the subjective and objective components, because the UDysRS is organized in this manner and the factor structure of the scale supports a single total summary score. Further, if we are aiming to reduce missing values, the solutions to the patient‐based sections and the rater‐based sections will be necessarily different. The challenge of reducing missing values has been the subject of several reviews and is a focus of current research. Whereas the methodology we utilized here focuses on dealing with missing values in already acquired data sets, moving forward, we wish to focus on strategies to prevent missing values before they occur. To this end, our efforts are focusing on applying intervention mapping techniques6 to identify barriers to data completeness throughout the health care team in clinical trials, to correct those barriers, and to ensure ongoing internal checks that obviate the need for surrogate solutions as described here.

Author Roles

1. Research Project: A. Conception, B. Organization, C. Execution; 2. Statistical Analysis: A. Design, B. Execution, C. Review and Critique; 3. Manuscript Preparation: A. Writing the First Draft, B. Review and Critique.

S.L.: 2B, 2C, 3A, 3B

X.R.: 2B, 2C, 3B

W.H.: 2B, 2C, 3B

C.G.G.: 1A, 1B, 1C, 2A, 2B, 2C, 3A, 3B

G.T.S.: 1A, 1B, 1C, 2A, 2C, 3B

Disclosures

Ethical Compliance Statement: Duke University Medical Center (DUMC) Institutional Review Board (IRB) has approved this study. Informed patient consent was not necessary for this work. We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this work is consistent with those guidelines.

Financial Disclosures and Conflicts of Interest: The authors received compensation from the Movement Disorder Society for the management of this program. The authors declared that there are no conflicts of interest in preparing this manuscript.

Financial Disclosures for the previous 12 months: Sheng Luo: NIH grants (R01NS091307) and salary from Duke University Medical Center. Xuehan Ren: Salary from the University of Texas Health Science Center at Houston (MDS support for project included). Weilu Han: Salary from the University of Texas Health Science Center at Houston (MDS support for project included). Christopher G. Goetz: Consulting or Advisory Board Membership with honoraria from Boston Scientific, research grants from Rush University Medical Center from NIH, and the Michael J. Fox Foundation. He directs the Rush Parkinson's Disease Research Center that receives support from the Parkinson's Disease Foundation and some of these funds support his salary as well as his research efforts. He participated in the directorship of the translation program for the MDS‐UPDRS and UDysRS and received funds directed to Rush University Medical Center from the International Parkinson and Movement Disorder Society (IPMDS) for this effort. He receives honoraria from the Presidential stipend from the International Parkinson and Movement Disorder Society that is paid to Rush University Medical Center (part of his salary), a faculty stipend from the International Parkinson and Movement Disorder Society, legal deposition payment received from Cray‐Huber Attorneys, and a lecture honorarium from the University of Pittsburgh. He also receives royalties from Elsevier Publishers, Oxford University Press, and Wolters Kluwer. Glenn T. Stebbins receives consulting and advisory board membership with honoraria from Acadia Pharmaceuticals, Adamas Pharmaceuticals, Biogen, Ceregene, CHDI Management, Cleveland Clinic Foundation, Ingenix Pharmaceutical Services (i3 Research), Neurocrine Biosciences, Pfizer, Tools‐4‐Patients, and Ultragenyx. Grants and research from the NIH, the Michael J. Fox Foundation for Parkinson's Research, Dystonia Coalition, CHDI, International Parkinson and Movement Disorder Society, and CBD Solutions. He also receives honoraria International Parkinson and Movement Disorder Society, American Academy of Neurology, the Michael J. Fox Foundation for Parkinson's Research, the Food and Drug Administration, and the NIH. He also has salary from Rush University Medical Center.