Development of the Comprehensive Cervical Dystonia Rating Scale: Methodology

Abstract

We present the methodology utilized for development and clinimetric testing of the Comprehensive Cervical Dystonia (CD) Rating scale, or CCDRS. The CCDRS includes a revision of the Toronto Western Spasmodic Torticollis Rating Scale (TWSTRS‐2), a newly developed psychiatric screening tool (TWSTRS‐PSYCH), and the previously validated Cervical Dystonia Impact Profile (CDIP‐58). For the revision of the TWSTRS, the original TWSTRS was examined by a committee of dystonia experts at a dystonia rating scales workshop organized by the Dystonia Medical Research Foundation. During this workshop, deficiencies in the standard TWSTRS were identified and recommendations for revision of the severity and pain subscales were incorporated into the TWSTRS‐2. Given that no scale currently evaluates the psychiatric features of cervical dystonia (CD), we used a modified Delphi methodology and a reiterative process of item selection to develop the TWSTRS‐PSYCH. We also included the CDIP‐58 to capture the impact of CD on quality of life. The three scales (TWSTRS2, TWSTRS‐PSYCH, and CDIP‐58) were combined to construct the CCDRS. Clinimetric testing of reliability and validity of the CCDRS are described. The CCDRS was designed to be used in a modular fashion that can measure the full spectrum of CD. This scale will provide rigorous assessment for studies of natural history as well as novel symptom‐based or disease‐modifying therapies.

Cervical dystonia (CD) is a chronic neurological disorder characterized by abnormal, involuntary posturing of the head, neck, and shoulders. Patients with this disorder also experience pain as well as impaired activities of daily living (ADLs). In addition, psychiatric comorbidities are experienced, especially anxiety and depression.1, 2 Furthermore, CD impacts overall quality of life (QoL).3, 4 Several published rating scales have been designed to evaluate the severity of CD.5, 6, 7 These have been used primarily to measure changes resulting from therapeutic interventions, yet none has been adequately evaluated for clinimetric properties. These scales also lack sufficient assessments of motor, nonmotor, and psychiatric features.8, 9, 10

Most clinical studies rely on the Toronto Western Spasmodic Torticollis rating scale (TWSTRS) to assess severity of CD.11, 12 The TWSTRS is comprised of three subscales that focus on the severity of the movement (10 items), disability (six items), and pain (three items).6 This scale has been used extensively in clinical studies of CD and yet has some deficiencies. Although inter‐rater reliability has been shown to be good to excellent,13 the methodology included only three raters asked to rate only one item at a time and not in the context of the entire examination.13 Furthermore, there are no measures in the scale that evaluate the severity of psychiatric symptoms or the impact of CD on QoL. Recently, a scale evaluating impact of CD on QoL has been developed and validated, the 58‐item Cervical Dystonia Impact Profile (CDIP‐58),14 but this scale does not include measures of motor severity. The lack of a validated comprehensive outcome measure that incorporates all aspects of CD is a major obstacle for longitudinal natural history studies and impedes progress in developing new therapeutic interventions. The experience gained in developing comprehensive outcome measures for Parkinson's disease, the UPDRS,15 provides a model for development of a similar scale for use in CD.

This multicenter study included an interdisciplinary team of experts in dystonia, psychiatry, clinimetrics, and biostatistics. The overall aim of the study was to develop a comprehensive CD rating scale (CCDRS) that encompasses the primary motor and nonmotor features of this disorder. This article describes the process of scale revision and development used to create and evaluate the CCDRS.

Methods

Scale Revision and Development

The overall methods for revision of existing scale items (TWSTRS) and development of new items assessing psychiatric features (TWSTRS‐PSYCH) followed a modified Delphi approach15 with input from experts in CD and patient feedback. For the revision of the three sections of the TWSTRS, the Dystonia Medical Research Foundation (DMRF) and the Dystonia Study Group (DSG) sponsored a CD Rating Scales Workshop in 2006. This workshop reviewed existing CD rating scales, including the Tsui rating scale, the Cervical Dystonia Severity Scale, and the TWSTRS,5, 6, 16 and found the TWSTRS to be the superior scale, but noted deficiencies in the coverage of CD symptoms and the scaling metrics used. From this workshop, an expert panel (C.L.C., G.T.S., K.P.B., S.H.F., and W.M.M.) formed the CD Rating Scale Committee (CD‐RSC), which was charged with suggesting modifications to the original TWSTRS. The CD‐RSC made recommendations to include descriptors of the individual items with clearly defined anchors for severity measures, the addition of a measure of dystonic head tremor, and standardization of the scaling metric for other items. The modified scale, TWSTRS‐2, included assessments for motor severity (12 items), pain (five items), and disability (six items).

Development of the TWSTRS‐PSYCH followed a similar pattern with an expert panel (M.Z., W.M.M., L.M., C.L.C., G.T.S., and H.A.J.) generating items based on semistructured interviews of 30 subjects with CD. The interview included items from established rating scales: the Beck Depression Inventory (BDI) II; the Hospital Anxiety and Depression Scale (HADS); and the Liebowitz Social Anxiety Scale (LSAS). The TWSTRS‐PSYCH panel then forwarded their results to the CD‐RSC, who used this information to develop specific items for the psychiatric scale.

Because the TWSTRS does not include an assessment of QoL specific to CD, we added the CDIP‐58. The CDIP‐58 is a self‐administered scale with eight subscales measuring head and neck symptoms (six items), pain and discomfort (five items), sleep (four items), upper‐limb activities (nine items), walking (nine items), annoyance (eight items), mood (seven items), and psychosocial functioning (10 items). The CDIP‐58 has been evaluated for reliability and validity in CD and shown to be superior to the 36‐item Short Form Health Survey, a widely used, but generic, QoL measure. The CDIP‐58 has demonstrated sensitivity to change after botulinum toxin (BoNT) injections in CD patients.14, 17

The TWSTRS‐2, TWSTRS‐PYSCH, and CDIP‐58 then were combined into the CCDRS and used in the data collection phase of the study along with other demographic and disease‐related measures, as noted below.

Assessment Measures

In addition to the CCDRS, demographic variables (age, gender, ethnic background, and comorbid medical conditions) and disease‐related variables (duration of CD and family history of dystonia current treatment for CD) were collected. Psychiatric diagnoses were assessed using the Structured Clinical Interview for DSM‐IV‐TR (SCID),18 administered by trained raters. Subjects also completed self‐report measures of psychiatric status (HADS, LSAS, and BDI).

Study Sites and Subjects

Study Sites

Ten study sites in the United States and Canada were selected based on the following criteria: (1) an investigator with expertise in CD; (2) adequate facilities to conduct the study, including videotaping capabilities, space for videotaping, and a computer with Internet access for data entry; and (3) personnel trained in administering all necessary rating scales.

Subjects

A total of 208 subjects with isolated CD as diagnosed by the site investigator were recruited from each site's clinical practice. All subjects gave written informed consent for the study before participation. The study procedures were approved by the local ethics committee and were carried out in accord with the Declaration of Helsinki. Subjects currently treated with BoNT had their last treatment at least 3 months before inclusion. Subjects were excluded if they had previous surgery for CD, neurological findings other than dystonia, significant coexisting medical conditions that could confound assessments of CD (e.g., significant cervical arthritis), muscle contractures or other medical conditions that would impair participation, significant dystonia in body regions other than the neck that would interfere with ratings, treatment with dopamine receptor antagonists, or treatment with BoNT less than 3 months preceding study entry. Other concomitant medications were permitted. We specified these exclusions because they are standard exclusions used in clinical trials of CD19, 20 and ensure that the scale is measuring only the severity of CD.

CD subjects were stratified by severity at each site to ensure that each site included an adequate range of severity. The stratification was done through an evaluation of the overall severity of CD assessed using the Global Dystonia Rating Scale (GDS) by the principle investigator (C.L.C.).21 Approximately equal numbers of CD patients with mild (GDS rating for neck from 1 to 3), moderate (GDS rating for neck from 4 to 6), and severe symptoms (GDS rating for neck from 7 to 10) were included.

Study Visits

There was one study visit. Each subject was evaluated using the TWSTRS‐2 pain and disability subscales, TWSTRS‐PSYCH, and CDIP‐58. Subjects were videotaped using standardized protocol developed by the Dystonia Coalition (Table 1). During the videotaping, the movement disorder specialist rated the subject using the TWSTRS‐2 motor severity scale. Subjects were then interviewed by a psychiatrist or psychologist using the SCID and completed the self‐administered psychiatric and QoL measures. All subjects were also enrolled in the Dystonia Coalition DNA repository, providing a DNA sample.

Table 1.

Dystonia coalition video examination protocol

Data Analytic Plan: CCDRS

The clinimetric examination of the CCDRS will follow a classical test theory (CTT) model.22 The first step will be to examine the content validity of the CCDRS. Utilizing the content validity ratio (CVR), each item will be assessed for importance as judged by experts in the field. The formula for determining the CVR is: CVR = (n_e − N/2)/(N/2), where n_e = number of raters indicates “essential” and N = total number of raters. Any item that falls below a CVR of 0.75 will be identified as potentially problematic and subjected to further review during internal consistency testing and construct and external validity testing.

The ratings for the revised TWSTRS‐2 and its subscales and the TWSTRS‐PSYCH will be examined for internal consistency determined by pair‐wise correlations among all items in the scale in all combinations of possible pairs and assessed using Cronbach's alpha, with a minimum alpha of 0.85 as the criterion. Additional analyses will investigate item‐to‐total correlations as well as changes to alpha given the exclusion of items. Items demonstrating a low item‐to‐total correlation (<0.40) and an increase in alpha, when omitted, will be identified as potentially problematic and subjected to further review.

The next step will be an assessment of the construct validity of the CCDRS.23 Construct validity will be assessed using exploratory factor analyses to determine the number and types of constructs measured. We will first use unweighted least squares for ordered categorical measures24 to extract common variance patterns among the rated items. The number of item sets, or factors, within a given scale can be best determined by the amount of shared variance between the scale and the variance accounted for by the eigenvalues. The scree plot shows the total variance accounted for by the different factors and can be used to determine the number of relevant factors. To determine the optimal number of factors, we will interrogate the scree plot for diminishing increases in percent variance explained by increasing factors. Once the factors are identified, each factor can be rotated to enhance the interpretation of the set construct. Two rotational methodologies are typically employed: orthogonal rotation and oblique rotation.25 Orthogonal rotations assume independence among the lower‐dimensional items sets, such that changes in one set do not affect any other sets, and oblique rotation that makes no such assumption. In this study, we will test both orthogonal and oblique rotation methods to see whether the resultant rotated factors are independent.24 Item relevance to each factor and item redundancy will be assessed through examination of item loadings on each factor. A minimum loading of 0.40 will be used as a criterion for factor relevance, and redundancy will be assessed by items loading on multiple factors. If an item loads significantly on more than one item, it may represent a lack of specificity of that item to any one factor. Dual loading criteria will be set at 0.25. If an item is identified as loading on more than one factor, or not loading on any factor, this item will be identified as potentially problematic and subjected to further review. Because the CDIP‐58 has already demonstrated excellent clinimetric properties, we do not plan on conducting CTT reliability and validity assessments on this instrument.

Inter‐ and intrarater reliability for the motor section of the TWSTRS‐2 will be assessed by having each investigator in the study rate two master tapes. The first master tape will have 25 subjects. When ratings are completed, the ratings will be submitted and the second master tape, consisting of 20 additional subjects and 5 repeat subjects from master tape 1, will be rated. Ratings of master tape 2 will occur at least 2 weeks following ratings for master tape 1. These master tapes will be randomly assigned to each investigator based on computer‐generated random numbers stratifying for CD severity (global severity score) and site, such that no investigator rates subjects from their own site. Inter‐ and intrarater agreement will be assessed using a weighted κ statistic for individual items and intraclass correlation coefficients (ICCs) for summary scores. Acceptable levels of κ and ICCs will be defined as ≥0.4026 and ≥0.60,27 respectively.

Items identified as potentially problematic at the CVR, internal consistency, construct validity, or inter‐ or intrarater reliability level of analyses will be reviewed by the Delphi panel and either retained, modified, or omitted from the CCDRS. The decision to retain, modify, or omit will be based on a combination of clinimetric result and clinical utility of the individual offending item. Should an item have multiple clinimetric failures (e.g., poor CVR, poor item‐to‐total correlation, and poor factor loadings) and judged to not have strong clinical utility, it will be omitted from the scale. If an item has only a few clinimetric failures, but strong clinical utility, it will be retained. These decisions will be based on a majority opinion from the Delphi panel.

Sample‐Size Estimation

There are no accepted formulae for calculating required sample sizes for scale validation studies, particularly factor analytic methods, at given levels of power.28 Instead, recommended subject‐to‐item ratios are employed. For the present study, we have a 9.1:1 subject‐to‐item ratio, which exceeds the recommended 8:1 ratio shown to be adequate for this analysis.25, 29

Development of TWSTRS‐2 Training Tape

A videotape will be developed that can be used for training investigators in the use of the new scale. The CCDRS training tape will incorporate theTWSTRS‐2, the TWSTRS‐PSYCH, and the CDIP‐58 (QoL) and provide a tool that can be reviewed by clinicians to ensure that the scale is applied in a uniform fashion. This type of training tool has been routinely utilized in previous multicenter studies of CD and other movement disorders13, 30, 31, 32 and will provide visual and written guidance for the CCDRS. Agreement among raters will be assessed using a weighted κ statistic. The segments with the highest concordance for each item among rating investigators will be included on the training tape for motor severity. The training tape will be packaged with detailed written instructions on the correct application of each of the relevant subdomains.

Discussion

Although numerous rating scales have been developed to assess the severity of CD, only the TWSTRS and the CDIP‐58 were recommended in a recently published evidence‐based critique of dystonia rating scales.33 Although both scales have undergone initial clinimetric testing, neither encompasses all aspects of CD, including specific motor and nonmotor features. This is the impetus for the development of a comprehensive scale, the CCDRS, which comprises measures for motor, pain, disability, psychiatric complications, and the impact of CD on QoL. The CCDRS is a composite scale, with rating modules for each of these features. It includes a revision of the original TWSTRS, the TWSTRS‐2, which provides uniform ratings for all items and incorporates items that were lacking in the initial version, but accepted as important to the severity of CD. In particular, dystonic tremor was not included in the original TWSTRS, yet has been reported to be a common feature of the disorder.1

The development of the TWSTRS‐2 and TWSTRS‐PSYCH uses a modified Delphi method. This method provides a model for the development of stable linear measures from rating scale data. The psychiatric comorbidities34, 35, 36, 37 observed in CD have not been included previously in any rating scale, and, consequently, their impact on overall severity of the disorder has not been investigated. The TWSTRS‐PSYCH is intended to rate several categories of psychiatric disturbances that are associated with CD and provide clinicians with a signal of psychiatric disorders that may require further investigation and treatment.

The inclusion of the CDIP‐58, a previously developed scale with demonstrated reliability and validity, provides a patient‐based rating scale that assesses the impact of CD. Although the scale has not yet been widely used, it has demonstrated reliability and validity and its addition to the CCDRS provides a valuable patient‐reported outcome.

In this study, a reliable and validated scale will be developed that encompasses all aspects of CD. Such a tool will provide a critical outcome measure for rigorous evaluation of new or existing treatment strategies. However, scale validation is often an ongoing process. For the CCDRS, future studies could include assessing rater reliability in international raters, to enhance the generalizability of the scale, testing the responsiveness of the scale to therapeutic interventions, determination of the Minimal Clinical Important Change, and assessing the reproducibility of scores in a test‐retest setting. This scale will be valuable for charting the natural history of the severity of CD as well as associated symptoms of pain, disability, psychiatric symptoms, QoL, and their interdependencies.

Finally, the methods used in this study provide a model for future rating scale development for other focal dystonias, such as blepharospasm, spasmodic dysphonia, craniofacial dystonia, and limb dystonia, for which there currently are no widely acceptable rating scales.

Author Roles

(1) Research Project: A. Conception, B. Organization, C. Execution; (2) Statistical Analysis: A. Design, B. Execution, C. Review and Critique; (3) Manuscript: A. Writing of the First Draft, B. Review and Critique.

C.L.C.: 1A, 1B, 1C, 2A, 2C, 3A, 3B

S.H.F.: 1A, 1B, 1C, 3B

K.P.B.: 1A, 1B, 1C, 3B

J.S.P.: 1A, 1B, 1C, 3B

H.A.J.: 1A, 1B, 1C, 3B

M.Z.: 1A, 1B, 1C, 3B

W.M.M.: 1A, 1B, 1C, 3B

L.M.: 1A, 1B, 1C, 3B

A.R.R.: 1A, 1B, 1C, 3B

T.W.: 1B, 1C

L.J.W.: 1A, 3B

W.R.G.: 1A, 3B

G.T.S.: 1A, 1B, 1C, 2A, 2B, 2C, 3A, 3B

Disclosures

Funding Sources and Conflicts of Interest: This study was funded by National Institutes of Health (NIH) grant NS 065701 from the National Institute of Neurological Disorders and Stroke (NINDS) and the Office of Rare Diseases Research (ORDR) at the National Center for Advancing Translational Sciences (NCATS), NIH. Funding was also received from the DMRF and Allergan Inc. The authors report no conflicts of interest.

Financial Disclosures for previous 12 months: C.L.C. has received grants to institution from NINDS, ORDR NS065701; NINDS, R43NS077652‐01A1; and NINDS, R01NS074343; the Parkinson Disease Foundation, Merz Pharmaceuticals, Allergan Inc., and Ipsen Biopharmaceuticals; has received compensation for services as a consultant or an advisory committee member from Allergan, Inc., Impax Pharmaceuticals, Ipsen Biopharmaceuticals, Inc., Medtronic Inc., Merz Pharmaceuticals, US World Meds, Teva Neurosciences, Merck, and Acadia Pharmaceuticals; and has received royalties from Cambridge, Humana Press, and Wolters Kluwer. S.H.F. has received financial support for clinical studies, as well as consultancy and speaker fees, from AstraZeneca, Avanir, Eisai, Kyowa, Merz, Merck Serono, Novartis, Teva, and Zambon; has received funding for research from the Canadian Institutes of Health Research ,,,CIHR, the Michael J. Fox Foundation for Parkinson Research (MJFF), Parkinson Society Canada, and the NIH; and has received salary support from the University Health Network Department of Medicine Practice Plan. K.P.B. has received funding for travel from GlaxoSmithKline (GSK), Orion Corporation, Ipsen, and Merz Pharmaceuticals, LLC; serves on the editorial boards of Movement Disorders and Therapeutic Advances in Neurological Disorders; receives royalties from Oxford University Press; received speaker honoraria from GSK, Ipsen, Merz Pharmaceuticals, LLC, and Sun Pharmaceutical Industries Ltd.; has received personal compensation for serving on the scientific advisory board for GSK and Boehringer Ingelheim; and received research support from Ipsen and from the Halley Stewart Trust through Dystonia Society UK, and the Wellcome Trust MRC strategic neurodegenerative disease initiative award (Ref. No.: WT089698), a grant from the Dystonia Coalition, and a grant from Parkinson's UK (Ref. No.: G‐1009). H.A.J. has received research grant support from the NIH, the Atlanta Clinical & Translational Science Institute, the Emory University Research Council, the Lesch‐Nyhan Syndrome Children's Research Foundation, the DMRF, the Bachmann‐Strauss Dystonia & Parkinson Foundation, and the Benign Essential Blepharospasm Research Foundation and is principal investigator for the Dystonia Coalition, which receives the majority of its support through NIH grant NS065701 from the ORDR in the NCATS and NINDS. The Dystonia Coalition receives additional material or administrative support from industry sponsors (Allergan Inc., Ipsen Biopharm, Medtronics Inc., and Merz Pharmaceuticals) as well as private foundations (the American Dystonia Society, the Bachmann‐Strauss Dystonia & Parkinson Foundation, BeatDystonia, the Benign Essential Blepharospasm Foundation, Dystonia Europe, Dystonia Ireland, the Dystonia Medical Research Foundation, the Dystonia Society, the Foundation for Dystonia Research, the National Spasmodic Dysphonia Association, and the National Spasmodic Torticollis Association). Dr. Jinnah serves on the scientific advisory boards for Cure Dystonia Now, the DMRF, Tyler's Hope for a Dystonia Cure, the Lesch‐Nyhan Syndrome Children's Research Foundation, and Lesch‐Nyhan Action France. W.M.M. has received grants to the institution: NINDS, ORDR NS065701; HRSA 6UB4HP19215‐01‐01; Cervel Neurotech; and the Stanley Foundation; has received support from Neuronetics (equipment) and Soterix (equipment); and has received royalties from Oxford University Press. L.M. has received grants to the institution: NINDS, ORDR NS065701; has received research support from the Veterans Health Administration Cooperative Studies Program; and has received royalties from Taylor Francis/Informa. G.T.S. has consulting and advisory board membership with honoraria with Acadia Pharmaceuticals, Adamas Pharmaceuticals, Inc., Ceregene, Inc., CHDI Management, Inc., Ingenix Pharmaceutical Services (i3 Research), and Neurocrine Biosciences, Inc.; has received grants and research support to institution from the NIH, the MJFF, the Dystonia Coalition, CHDI, and the International Parkinson and Movement Disorder Society (MDS); and has received honoraria from the MDS, American Academy of Neurology, and the MJFF.