Using Cognitive Pretesting in Scale Development for Parkinson’s Disease: The Movement Disorder Society Unified Parkinson’s Disease Rating Scale (MDS-UPDRS) Example

Abstract

Background

Cognitive pretesting, a qualitative step in scale development, precedes field testing and assesses the difficulty of instrument completion for examiners and respondents. Cognitive pretesting assesses respondent interest, attention span, discomfort, and comprehension, and highlights problems with the logical structure of questions/response options that can affect understanding. In the past this approach was not consistently used in the development or revision of movement disorders scales.

Methods

We applied qualitative cognitive pretesting using testing guides in development of the Movement Disorder Society-sponsored revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS). The guides were based on qualitative techniques, verbal probing and “think-aloud” interviewing, to identify problems with the scale from the patient and rater perspectives. English-speaking Parkinson’s disease patients and movement disorders specialists (raters) from multiple specialty clinics in the United States, Western Europe and Canada used the MDS-UPDRS and completed the testing guides.

Results

Two rounds of cognitive pretesting were necessary before proceeding to field testing of the revised scale to assess clinimetric properties. Scale revisions based on cognitive pretesting included changes in phrasing, simplification of some questions, and addition of a reassuring statement explaining that not all PD patients experience the symptoms described in the questions.

Conclusions

The strategy of incorporating cognitive pretesting into scale development and revision provides a model for other movement disorders scales. Cognitive pretesting is being used in translating the MDS-UPDRS into multiple languages to improve comprehension and acceptance and in the development of a new Unified Dyskinesia Rating Scale for Parkinson’s disease patients.

INTRODUCTION

In 2001 the Movement Disorder Society (MDS), in response to critiques of the Unified Parkinson’s Disease Rating Scale (UPDRS) [1–3], created a Task Force to revise the scale. Prior to field testing, cognitive pretesting of the revised scale, the MDS-UPDRS, was conducted to ensure that questions and response options effectively extracted accurate information [4]. Since the Advanced Research Seminar on Cognitive Aspects of Survey Methodology occurred in 1984 [5], cognitive pretesting has been increasingly accepted as a qualitative method for investigating the extent to which respondents understand questions in the way they were intended [6]. The current report gives examples from the cognitive pretesting of the MDS-UPDRS and gives insights into this lesser known aspect of scale development relative to the study of movement disorders.

MATERIALS AND METHODS

Qualitative interviewing methodologies for cognitive pretesting

We used cognitive pretesting to assess task difficulty for both examiners (clinicians in movement disorder clinics) and respondents (patients with Parkinson’s disease). We evaluated respondent interest, attention span, and comfort with all items involving patients. Our assessment was designed to detect problems in overall comprehension and problems of logical structure in questions and response options that could affect understanding [7]. Each question from Part I, non-motor experiences of daily living (11 items), Part II, motor experiences of daily living (13 items) and Part IV, motor complications (6 items) was presented to PD patients and examiners. Items from Part III, motor examination (18 items), were presented to examiners only. We gathered their answers to MDS-UPDRS items and then asked focused cognitive test questions of both the respondents and the examiner. Respondents were asked how well they understood the questions and answer options, the concepts being assessed, and particular medical terms or phrases. Examiners were asked to identify problems with the phrasing of a question or response options that made a question difficult to administer. Patients and/or raters were asked about other types of problems such as discomfort with the items, response options or instructions. In addition, examiners were asked to numerically rate the level of difficulty administering the questions, and patients were asked to rate difficulty understanding both questions and response options as well as appropriateness of response options for each item on the scale from 1(very difficult) to 6 (very easy). This resulted in a minimum of 3 ratings for each item. Numeric ratings were not intended for statistical analysis but rather as indicators of items needing revision based on qualitative responses and further cognitive testing.

Based on results, multiple rounds of pretesting could be required.

Two specific qualitative techniques were used in cognitive pretesting:

1. Verbal probing - the examiner asks the respondent a question, the respondent answers, then the examiner probes for the respondent’s understanding of the question and basis for the selected answer.

2. “Think aloud interviewing” - respondents were asked to “think aloud” as they selected an answer to a self-administered question in order for the examiner to understand the respondent’s decision process as it related to interpretation of the question and response options [7] (Table 1 for examples of probes and any supplemental materials for more detail).

Table 1.

Examples of cognitive test probes

Example 1: Examiner Administered Item

Examiner reads the question to the patient.

1. How easy or difficult is it for you as the examiner to use the current wording of the question verbatim? (Circle choice)  Very difficult 1 2 3 4 5 6 Very Easy

Examiner notes difficulties.

2. Using the question below mark each item Y or N:  – Examiner experienced difficulty reading question  – Examiner asked to repeat all or part of question  – Examiner had difficulty explaining question  – Other examiner issue (please specify) _____

Questions for the patient after they have heard the MDS-UPDRS question:

3. How easy or difficult is this question for you to understand? (Circle choice)  Very difficult 1 2 3 4 5 6 Very Easy

4. What parts of the question were difficult to understand? What was the difficulty?

5. What do you understand by the following words?  “emotional and motor consequences of tremor”   Understood correctly? (Y/N) _____  What words would be easier to understand to capture the meaning?

Questions for the patient after they have decided on a rating:

6. How easy or difficult was it for you to rate your answer to that question?  Very difficult 1 2 3 4 5 6 Very Easy  What made it difficult to rate?  Suggested solution?

7. Who answered this item?  ___ Patient primarily ___ Caregiver primarily ___ Both

Example 2: Self-administered Item (Round 2 of testing only)

Examiner’s observations while the patient is reading/responding to the question:

1. What issues with the question did you observe while the patient or caregiver was reading or interpreting the question?

2. What issues with the response options did you observe while the patient or caregiver was reading or interpreting the response options?

Questions for the patient after they have read and responded to the MDS-UPDRS question:

3. How easy or difficult is this question for you to understand? (Circle choice)  Very difficult 1 2 3 4 5 6 Very Easy

4. What parts of the question were difficult to understand? Why were they difficult?

5. How easy or difficult was it for you to select a response choice to that question?  Very difficult 1 2 3 4 5 6 Very Easy  What made it difficult to select a response?  Suggested solution?

6. Who answered this item?  ___ Patient primarily ___ Caregiver primarily ___ Both

Study sample

Prior studies documented that major improvements in questions and quality of response data can be obtained with relatively few cognitive pretesting interviews [8]. However, because the MDS-UPDRS, even in its primary English version, was planned for wide usage nationally and internationally we sought a geographically diverse sample for cognitive pretesting. All examiners were movement disorder specialists and all participants, both patients and raters, were native English speakers. No demographic data or clinical data were collected, other than geographical site, on the examiners or respondents. All sites received human subjects’ approval from their respective institutional review boards prior to beginning the cognitive testing.

Data collection procedures

Given a limited budget, we used clinicians at the movement disorder clinics untrained in the administration of cognitive pretesting interviews, but guided by self-directed cognitive testing manuals (see Supplemental Materials developed by the qualitative researcher (NL)). Cognitive pretesting was facilitated by having both those who developed the revised scale and those who were uninvolved in the development do the testing while acting as examiners. For evaluating survey questions a minimum sample size of 20 interviews has been suggested to capture the diversity of responses [4].

Data analysis

Completed cognitive testing guides for each patient were submitted to the research team and data were entered into a central data base. In the data base identifying numbers for patients, examiners and sites were registered along with responses for each item in the scale. The data base was then exported to an EXCEL file and was sorted by patient ID, examiner ID and site ID within item number. Difficulty ratings were used to indicate where problems occurred. In order to focus on problem areas all responses that had examiner and patient ratings “easy” or “very easy” (5–6) and no qualitative concerns recorded were deleted. Qualitative data for remaining responses were summarized for each item. A qualitative researcher (NL) provided an evaluation of the responses for each MDS-UPDRS item, a report summarizing problems identified, and a summary of solutions suggested by either the examiners or respondents. Those items judged difficult to use (scoring less than 5–6 for questions or response options by multiple patients or examiners) or items where an important qualitative concern was identified were revised as needed. Revised items were retested iteratively in subsequent pretesting until they were no longer rated as difficult to use (with accompanying supporting qualitative data). If an item was rated less than 5–6 but had no supporting qualitative data, we looked carefully across other responses for that item to see if other participants had issues with the item. Because the required sample size was small, if only a single rater raised a concern considered important by a patient or examiner, this single concern also warranted scrutiny and possible revision to the scale assuming that the concern could arise in the future for other patients. The iterative cognitive pretesting process stopped when the Task Force determined that only easily resolvable minor difficulties remained.

RESULTS

First round of cognitive pretesting

Ten study sites participated in the first round of cognitive pretesting, including four sites in Western Europe among native English speaking examiners and patients, one site in Canada and five sites in different parts of the United States. A total of 19 clinicians, up to four per site, played a dual role as cognitive interviewers and examiners conducting the in-depth cognitive pretesting of the MDS-UPDRS. Examiners at each site conducted two to five interviews resulting in a total of 43 cognitive tests, more than double the required sample size.

Problems identified

In Part 1 of the MDS-UPDRS varying types of difficulties were reported by at least one site for all 14 questions. In Part 2, problems with complexity and length of question were reported for 9 of 14 questions by raters from at least one site. In Part 3 problems were noted in 10 of 18 questions. In Part 4 there were specific difficulties with 2 of the 7 questions.

Patients expressed difficulty distinguishing PD-related issues from issues related to other comorbidities or earlier causes. A few examiners noted that the reading level for many MDS-UPDRS items was much higher than 7th grade level. Medical terminology was a problem for many patients. Patients had difficulty with phrases such as “the emotional and motor consequences of tremor” (one patient saying, the only “motor” I know is “motor car”). Many patients had difficulty differentiating between the ON and OFF states. In Part 4, examiners found it difficult to explain and patients found it difficult to grasp the meaning of “fluctuations”, “OFF-state”, and “predictability of OFF function” according to the provided definitions.

Many questions were flagged as too long and complex (Table 2, Example 1). In these cases response options were usually judged too lengthy and complicated for examiners to read easily and for patients to understand or recall (Table 2, Example 2). Questions and response options giving two concepts linked by “and” or “or” were confusing to patients (Table 2, Example 3). Some patients had difficulty differentiating between two response options or difficulty in finding an option appropriate to their situation (Table 2, Example 4). Examiners identified long and complex instructions in Part 3 that might be more easily demonstrated than described (Table 2, Example 5). Examiners noted that some patients had difficulty with the use of percentage estimates in the Part 4 response options (See Table 2, Example 6).

Table 2.

Examples of problems encountered in cognitive pretesting (Round 1^*) and solutions

Original MDS-UPDRS Item	Item as Revised
Example 1: Shorten/simplify question
COGNITIVE IMPAIRMENT - On the average during the past week, have you experienced cognitive or thinking impairment as a result of your PD? By “cognitive impairment”, I mean cognitive or thinking deficits including overall intellectual function, attention, memory, mental flexibility or ability to juggle multiple mental tasks simultaneously and speed of thinking. I want to know if you have any problems and if so the extent of interference with your daily life.	COGNITIVE IMPAIRMENT - Over the past week have you had problems remembering things, following conversations, paying attention, thinking clearly, or finding your way around the house or in town?
Example 2: Shorten/simplify response option
2: Mild. Lightheadedness occurs with changes in posture so that you return to a sitting or lying position to manage symptoms. No falls and no loss of consciousness.	2: Mild: Dizzy or foggy feelings cause me to hold on to something, but I do not need to sit or lie back down.
3: Moderate. Lightheadedness occurs with changes in position and has been associated with at least one fall in the past week, but without loss of consciousness.	3: Moderate: Dizzy or foggy feelings cause me to sit or lie down to avoid fainting or falling.
4: Severe. Lightheadedness occurs with changes in posture and has been associated with at least one episode of loss of consciousness over the past week.	4: Severe: Dizzy or foggy feelings cause me to fall or faint.
Example 3: Eliminate compound/complex concepts
2: Mild. Urinary frequency or urgency sufficient to cause inconvenience and requiring some adaptations in daily function, although no incontinence.	2: Mild: Urine problems cause a few difficulties with my daily activities. However, I do not have urine accidents.
3: Moderate. Urinary frequency, urgency with occasional incontinence; significantly interferes with daily activities such as social functions.	3: Moderate: Urine problems cause a lot of difficulties with my daily activities, including urine accidents.
Example 4: Make response options appropriate for all
1: Slight. Some difficulty with swallowing but no choking or extra time needed to chew food. However, food is not cut or prepared in a special way for you to chew or swallow.	1: Slight: I am aware of slowness in my chewing or increased effort at swallowing, but I do not choke or need to have my food specially prepared.
2: Mild. Chokes but not daily, or expends considerable time and effort to chew food, but food is not cut or prepared in a special way for you to chew or swallow.	2: Mild: I need to have my pills cut or my food specially prepared because of chewing or swallowing problems, but I have not choked over the past week.
3: Moderate. Daily choking or food needs to be cut or prepared in a special manner because of difficulty chewing or swallowing.	3: Moderate. I choke at least once in the past week.
4: Severe. Unable to obtain adequate nutrition without an alternative route for nutritional support (i.e. nasogastric tube or gastrostomy).	4: Severe: Because of chewing and swallowing problems, I need a feeding tube.
Note: Patient who has no choking but needs food specially prepared could not find appropriate response option above	Note: Here same patient could select option 2.
Example 5: Simplify/demonstrate rather than instruct
FINGER TAPPING - Instructions to examiner: Each hand is tested separately. You may demonstrate the task, but do not continue to perform the task while the patient is tested. Once the task is understood so that patient taps as quickly AND as fully as possible, have the patient carry out 10 finger taps. Rate each side separately. Investigator will rate the number of halts or hesitations, the speed and ability to maintain a full open and close motion without fatigue or decrement. Instructions to Patient: Please flex your right (left) elbow with the palm facing me. Spread apart your fingers and thumb on this hand. Tap your thumb with the tip of your index finger in rapid succession using BOTH the largest amplitude possible and the fastest speed possible.	FINGER TAPPING - Instructions to examiner: Each hand is tested separately. Demonstrate the task, but do not continue to perform the task while the patient is being tested. Instructions to the patient: Tap the index finger on the thumb 10 times as quickly AND as big as possible. Rate each side separately, evaluating speed, amplitude, hesitations, halts and decrementing amplitude.
Example 6: Ask for hours rather than percentage estimates
PAINFUL OFF-STATE DYSTONIA {For patients with fluctuations and OFF time} - What proportion of the OFF episodes include painful dystonia?	PAINFUL OFF-STATE DYSTONIA - Instructions to examiner: For patients who have motor fluctuations, determine what proportion of the OFF episodes usually includes painful dystonia? You have already determined the number of hours of “OFF” time (4.3). Of these hours, determine how many are associated with dystonia and calculate the percentage. If there is no OFF time, mark 0. Instructions for patient [and caregiver: In one of the questions I asked earlier, you said you generally have ___ hours of low or “OFF” time when your Parkinson’s disease is under poor control. During these low or “OFF” periods, do you usually have painful cramps or spasms? Out of the total ____ hrs of this low time, if you add up all the time in a day when these painful cramps come, how many hours would this make? Examiner does calculations.
0: Normal. No dystonia	0: Normal: No dystonia OR NO OFF TIME.
1: Slight. 1–25% of OFF episodes	1: Slight: <25% of time in OFF state.
2: Mild 26–50% of OFF episodes	2: Mild: 26–50% of time in OFF state.
3: Moderate 51–75% of OFF episodes	3: Moderate: 51–75% of time in OFF state.
4: Severe 76–100% of OFF episodes	4: Severe: >75% of time in OFF state

^*Cognitive Testing was conducted in two Rounds. Round 1 was the initial cognitive testing of the entire questionnaire.

Additionally, cognitive testing identified questions that upset patients or were awkward for examiners at 3 of 10 sites. Upsetting questions generally related to cognitive impairment, hallucinations/psychosis, and depressed mood. Some patients in early stages of PD became concerned about impairments they had not previously considered.

Given the number and variety of problems identified in the first cognitive pretesting phase, the Task Force members rejected the original version of the MDS-UPDRS and revamped the scale to address each weakness.

Revisions

Major changes made by the Task Force Committee are displayed in Table 2, column 2, and included:

• Focus on Self-rated patient status, not just PD: As a general concept, the revised version no longer attempted to distinguish between problems related to PD and problems related to other conditions. The guidelines of “rate how you feel (patients)” and “rate what you see (clinical raters)” were adopted and inserted into the instructions.

• Reduction of Reading level: The reading level was lowered to at or below the 7th grade using the Flesch-Kincaid Grade Level program in Microsoft Word for verification. Difficult medical terminology including terms such as “apathy”, “abnormal sensory sensations”, “physical disimpaction”, “adaptive changes needed to eat” were replaced by simpler phrases: “indifference”, “uncomfortable feelings”, “physical help to empty my bowels”, and “help handling my food”, respectively.

• Elimination of Calculations by Patients: To address difficulties with percentages, patients were asked in the revised MDS-UPDRS to give estimates in hours and examiners were asked to convert hours to percentages.

• Clarifying ON versus OFF function: To address patient-related problems differentiating between ON and OFF states, new definitions were provided and Parts 1 and 2 were redesigned to cover an overall perspective rather than a separate ON and OFF score for each part.

• Patient reassurance: Text was added as a result of recurring patient and examiner concerns to assure patients that they may never experience all impairments assessed by the instrument.

Second round of cognitive testing

Due to the extent of revisions to the initial MDS-UPDRS version, a second round of cognitive testing was initiated. A new cognitive pretesting manual compatible with the revised assessment instrument was developed. Fourteen examiners from seven sites, most having participated in the first round of cognitive tests, conducted the second phase of cognitive pretesting. Round 2 included 32 patients. Most patients had not been interviewed in Round 1. The seven sites included a new site in Western Europe and six sites that participated previously in Round 1 including five sites in the United States and one site in Canada. Most sites enrolled three to five patients with most clinicians each interviewing two to three patients.

For the second round of cognitive pretesting, MDS-UPDRS parts 1B and 2 were self-administered with patients and caregivers reading and responding to the questions. Examiners took notes on “think-aloud” discussions between patient and caregiver to identify any difficulties in choosing the best option. This “think-aloud” approach supplemented the verbal probing methods used after a response was selected in examiner administered interviews. Again the numeric ratings of difficulty were used only as an indicator as to where scrutiny of the qualitative responses was most important.

Raters and patients remained concerned about the length of the scale, and length of instructions, particularly for examiners. From comments some concerns appeared related to the addition of the cognitive pretesting questions and not to scale itself. Consistency of terminology was also a concern (e.g., terms such as many, most, frequently). Some of the remaining medical terminology also was also deemed to be problematic and needed simplification.

Language changes

In simplifying the language some negative terminology was introduced. Based on Round 2 testing, sensitive and negative terminology was replaced with more neutral wording; for example “clumsy” eating was changed to “slow with my eating and have occasional food spills”; “use a diaper” was changed to “use a protective garment”.

Table 3 shows examples of some of the specific changes made in round two. Examiners noted awkwardness in using the new question related to dysregulation syndrome/sex in Part 1a, and the question directed to the patient was modified so that “sex” was mentioned only optionally at examiner discretion. Items with multiple sequential questions or lists of things patients were asked to consider for an individual MDS-UPDRS item remained confusing and were further revised. Some patients had difficulty distinguishing between response options because of inconsistent use of terminology (e.g., troubles vs. problems) and of quantifiers (such as a few, a lot, many, most frequently) and these response options were revised to be internally consistent within the scale.

Table 3.

Examples of problems encountered in cognitive pretesting (Round 2^*) and solutions

Example 1: Reduce list of questions to patient; mention “sex” optionally
Phase I Revision: Over the past week, have you had unusually strong urges that are hard to control? For example, have you gambled too much? Have you put things together or taken things apart over and over again? Do you think a lot about sex?	Over the past week, have you had unusually strong urges that are hard to control? Do you feel driven to do or think about something and find it hard to stop? [Give patient examples such as gambling, cleaning, using the computer, taking extra medicine, obsessing about food or sex, all depending on the patients.]
Example 2: Eliminate word confusion (“personal” problems vs. trouble sleeping)
Phase I Revision: Over the past week, have you had trouble going to sleep at night or staying asleep through the night? Consider how rested you felt after waking up in the morning.	Over the past week, have you had trouble going to sleep at night or staying asleep through the night? Consider how rested you felt after waking up in the morning.
0: Normal: No problems	0: Normal: No problems.
1: Slight: Problems are present but usually do not cause trouble getting a full night of sleep.	1: Slight: Sleep problems are present but usually do not cause trouble getting a full night of sleep.
2: Mild: Problems usually cause some trouble getting a full night of sleep.	2: Mild: Sleep problems usually cause a few difficulties getting a full night of sleep.
Example 3: Medical Terminology
Phase I Revision:	Phase IA Revision:
0: Normal: Not at all. I have no tremor.	0: Normal: Not at all. I have no shaking or tremor.
1: Slight: Tremor occurs but does not cause problems with any activities.	1: Slight: Shaking or tremor occurs but does not cause problems with any activities.

^*Cognitive Testing was conducted in two rounds. The second round used cognitive pretesting to assess questions that had been revised due to problems identified in the round one, and included cognitive testing of the patient-completed version of the MDS-UPDRS.

In both rounds of cognitive pretesting for Part 3 of the MDS-UPDRS (motor examination), several examiners noted difficulty in estimating and rating movement interruptions, movement decrements, and tremor amplitudes. These problems were considered by the Task Force to be best resolved by the planned MDS-sponsored Teaching Program with video-based examples of each rating option. These questions and answers were not changed. After the second round of testing the modified version of the scale was approved by the Task Force for the final step of large-scale field testing in over 800 patients to assess the psychometric properties of the MDS-UPDRS [9]. The current version of the scale is posted on the MDS website, www.movementdisorders.org.

DISCUSSION

Cognitive pretesting of the MDS-UPDRS led to important changes in the scale components directed to both the rater and to the patient/caregiver team affecting both overall structure and individual questions or response options. Most changes focused on simplification and clarification, and on adding a reassuring message to patients at the end of the scale questionnaire.

There were some limitations of the MDS-UPDRS cognitive pretesting. To minimize the need for busy clinician-examiners to record lengthy open-ended responses, we restricted the types of cognitive pretesting probes asked. In the first round of cognitive pretesting, we simply asked patients what made the MDS-UPDRS question or response options difficult and used a Likert type rating to assess level of difficulty. The “think aloud” method was used in the second round to pre-test self-administered MDS-UPDRS questions and to facilitate documentation of more lengthy open-ended patient responses.

We could not digitally record the interview for later analysis due to costs and the additional time and inconvenience for the busy volunteer examiners and their staff. To address patient fatigue, some examiners gave half of the questions to one patient and a second half to another patient. Despite the limitations, consistent patterns observed in the types of problems identified led to multiple improvements to the final MDS-UPDRS version.

A strength of our approach was the combination of pretesting with subsequent data collection to assess the psychometric properties of the scale [10, 11]. For those creating a completely new scale, other approaches to cognitive pretesting combining qualitative and quantitative data may also be of interest [12–14].

Cognitive pretesting is a step often skipped in both the development of movement disorders scales and in the revision of existing scales. Most scales are tested psychometrically (factor analysis, etc) but not qualitatively. The MDS-UPDRS cognitive testing demonstrates the value of this qualitative approach as a first step before large clinimetric field testing is pursued. The qualitative approach does not provide data for quantitative inference but rather serves as a rich source of information for the scale developer in terms of word refinement, question construction, and clarity. With this technique upsetting or complex words or unclear concepts can be corrected before the large scale effort of full quantitative clinimetric testing. Cognitive pretesting is necessarily followed by a more traditional quantitative psychometric testing in larger groups of patients to address core scale properties such as reliability and validity. This two-step strategy provides a model for developing scales for other movement disorders. It is a core component of the translation program for non-English versions of the Unified Dyskinesia Rating Scale (UDysRS), and is planned to be incorporated into testing of a core definition of Parkinson’s disease sponsored by the Movement Disorder Society (CG Goetz, personal communication).

Cognitive pretesting is currently being used in the translation program of the MDS-UPDRS into other languages to improve culturally-based comprehension and acceptance. Cognitive pretesting is the first step in the psychometric testing process followed by collection of a larger sample of subjects to allow assessment of more traditional psychometric properties. To date (Jan 1, 2014), nine programs have successfully completed cognitive pretesting and large field testing with resultant official MDS-UPDRS translations (Spanish, Italian, French, Estonian, German, Japanese, Russian, Hungarian and Slovak). Five other language versions have successfully passed the cognitive pretesting phase and are currently in large field testing (Traditional Chinese, Korean, Hebrew, Dutch and Greek). Six language teams are currently in the process of developing or executing their cognitive pretesting program and will move into the large field testing phase after successful completion (Thai, Hindi, Portuguese, Serbian, Polish and Simple Chinese). In all instances, the cognitive pretesting phase has been important to the overall scale translation program, allowing the original translation to be modified and strengthened before full field testing. (http://www.movementdisorders.org/publications/rating_scales/).