Ranking of Dystonia Severity by Pairwise Video Comparison

Abstract

Background

Reviewers of dystonia rating scales agree on the need to assess symptoms more comprehensively. During the development of a quantitative dystonia assessment by video‐perceptive computing, we devised a video‐based severity ranking as a procedure to create a validation standard without the use of clinical scales.

Methods

Thirty‐four patients with dystonia (17 with dystonic tremor) and 2 controls were assessed with clinical scales and video‐recordings of 24 short movement tasks. Two to 4 raters compared multiple permutations of videos from 22 subjects, including 2 healthy controls, until a complete rank order was achieved. Inter‐rater agreement was expressed as normalized Kendall tau distance. Spearman correlations of video rank order with clinical scales and self‐rating were repeated for tremor/nontremor subgroups.

Results

Normalized Kendall tau distances were <0.3 for 15 items. The video rank order for sitting and head movements correlated with clinical scales for the whole group (rho 0.52–0.87) and in the subgroup without tremor. In the tremor subgroup such correlation was perceived in the 2 items involving sitting. Video rank order correlated with quality of life self‐rating only in 1 item (arms held in front, palm down).

Conclusions

The agreement of video rankings between raters is remarkable. The lack of correlation in the tremor subgroup in several items may be interpreted as tremor being considered in video comparisons but not in clinical scales. This supports video‐based ranking as a more comprehensive rating of dystonia and as a possible validation instrument applicable in situations in which no reference standard is available

Dystonia is a movement disorder characterized by motor symptoms such as sustained or intermittent involuntary muscle contractions.1 The most frequent type of focal dystonia is idiopathic cervical dystonia2 with sustained and frequently painful postures of the head, in many cases accompanied by tremorlike phasic symptoms. Dystonic postures are heterogeneous in their appearances,3 even within 1 patient, which complicates the development of assessment tools with valid and reliable clinimetric properties.4 Nevertheless, the quality of assessment tools is crucial for therapeutic monitoring,5 especially in the case of dystonia, in which treatment effects of current therapeutic lines are to be expected with some delay and often with a gradual onset and offset.6, 7, 8

Clinical rating instruments have received increasing attention for some time: quality criteria for medical outcomes have been defined9 and new rating scales have been developed and validated within the field of movement disorders.10, 11, 12 Dystonia motor scales have been the topic of several reviews4, 5 and, remarkably, none of the widely used scales was found to satisfy all quality criteria.13, 14 The most commonly used scales are the Toronto Western Spasmodic Torticollis Rating Scale (TWSTRS) for cervical dystonia15 and the Burke‐Fahn‐Marsden Dystonia Rating Scale (BFMDRS)16 in cases of generalized dystonia.5, 14 However, lack of distinction between dystonic posturing and dystonic movement has been stated as their major limitation, as both phenomena might respond differently to treatment.12, 17 Further, dystonic tremor is increasingly recognized as a frequent phenotype.18

The quantitative instrumented assessment of motor symptoms has been suggested as more sensitive to change than clinician rating19, 20 with the advantage of rater‐independence. Whereas several devices have been tested in Parkinson's disease19 and some have become commercially available,21, 22, 23 similar attempts in dystonia to date have been confined to pilot studies.24, 25, 26

In the process of developing a quantitative assessment of dystonia by video‐perceptive computing (Microsoft Kinect),27 we are facing the situation that available dystonia rating scales seem of limited value for its evaluation. We hereby present a method for disease severity ranking generated without the use of predefined scoring systems but derived from pairwise video‐comparison of a series of clinical cases that may be used to train output algorithms.

Methods

Participants

We included 34 adult patients with focal cervical, segmental (cervical and arms), or generalized dystonia with or without current symptomatic treatment from the movement disorders clinic at Charité University Medicine Berlin. The control group consisted of 2 subjects without a history or clinical finding of movement disorder. Subjects with motor complaints resulting from other conditions were excluded in both groups. Approval from the institutional review board (Identifier: EA1/156/13) and written informed consent from each participant were obtained.

Clinical Assessments

We here present cross‐sectional data, except for 1 patient measured twice with different settings of deep brain stimulation. The assessment contained clinical rating with BFMDRS16 for generalized dystonia and TWSTRS15 for cervical and segmental dystonia. The presence or absence of additional dystonic tremor defined as head dominance of tremor and/or tremor onset after dystonia onset18 on examination were used to define tremor and nontremor subgroups of the cohort.

Patients’ self‐rating included visual analogue scales (VAS, 0–10) of symptom severity and pain. Dystonic posture, movements, and tremor comprise symptom severity. Health‐related quality of life (hrQoL) was assessed with the Craniocervical Dystonia Questionnaire (CDQ‐24)28 in the 22 subjects included in the video‐ranking procedure described in the following text (2 missing). As no disease‐specific hrQoL instrument is currently available for generalized dystonia, and all patients suffering from the latter condition exhibited craniocervical involvement, the CDQ‐24 was applied for both conditions. It consisted of 24 items, each with a 5‐step rating (0–4).

Video Recording and Ranking

A set of 24 short motor items (Table 1; Table S1) was defined by expert agreement (T.S.‐H., A.L., A.A.K., S.M.‐M., K.O., A.U.B.) especially with regard to its use in video‐perceptive computing by Microsoft Kinect. The items mainly involve the upper body according to the patient target group. Subjects performed the items after standardized operator instructions at a defined distance in front of a Kinect camera. The Microsoft Kinect combines a motion sensing function that uses infrared emitters and sensors (30 Hz frame rate; 512 × 424 pixels resolution) with conventional color video recording (30 Hz frame rate; 1280 × 720 pixels). Only color‐video material was used for the approach described in this article. Depth data from infrared sensors were stored for later analysis.

Table 1.

Fifteen Motor Items Designed for Microsoft Kinect Recording with Sufficient Inter‐Rater Agreement on Video Rank Order Along with Short Outline of Validation Results^a

Item	Description	Results
Posture sitting	Relaxed sitting positions first with back against backrest (10 s), then without using the backrest (20 s), and finally with eyes closed (20 s)	Correlation with clinical scales and self‐rating r = 0.832, P < 0.001
Posture sitting straight	Sitting as straight as possible (60 s) without backrest after a phase of 30 s sitting in a relaxed position with eyes closed	Correlation with clinical scales r = 0.633, P = 0.002
Head rotation static	Moving the head to maximal rotation and holding this position for 20 s. Item is performed separately for left and right	Correlation with clinical scales r = 0.865, P < 0.001
Head rotation dynamic	Sequence of 5 head rotations between right and left maximal excursion at comfortable speed	Correlation with clinical scales r = 0.772, P < 0.001
Head tilt static	Moving the head to maximal tilt toward the shoulder and holding this position for 20 s. Item is performed separately for left and right	Correlation with clinical scales and self‐rating r = 0.617, P = 0.002; r = 0.446, P = 0.033
Head tilt dynamic	Sequence of 5 head movements between right and left maximal tilt positions at comfortable speed	Correlation with clinical scales and self‐rating r = 0.541, P = 0.009; r = 0.446, P = 0.037
Head extension	Moving the head in maximal extension (facing upward) and holding this position for 20 s.	Correlation with clinical scales r = 0.506, P = 0.016
Head extension/flexion dynamic	Sequence of 5 head movements between maximal head extension and flexion at comfortable speed	Correlation with clinical scales r = 0.659, P = 0.001
Arms held in front, palm down	Holding arms 45° elevated to the front and 45° from midline with palms down for 20 s	Correlation with self‐rating r = 0.541, P = 0.024
Arms held flexed	Holding arms 45° elevated to the front with elbows flexed in pronation for 20 s (fingertips opposed in front of the sternum with about 2 cm distance)	Correlation with self‐rating r = 0.492, P = 0.023
Arm flexion dynamic	Sequence of 5 arm movements between maximal elbow flexion and extension starting with arm 45° elevated to the front and 45° from midline, palms up. Item is performed at comfortable speed and separately for left and right	Correlation with clinical scales r = 0.547, P = 0.010
Standing	Standing relaxed with eyes open (30 s), then with eyes closed (15 s)	Correlation with clinical scales r = 0.521, P = 0.019
Walking in place	Walking on the spot in comfortable pace for 30 s	Correlation with self‐rating r = 0.485, P = 0.035
Trunk rotation without hips	Moving the upper body from straight ahead to maximal side rotation while standing with feet and hips in place; maximal excursion is held for 3 s. Item is performed separately for left and right	Correlation with self‐rating r = 0.686, P = 0.001
Trunk rotation with hips	Moving the whole body from straight ahead to maximal side rotation while standing with feet in place; maximal excursion is held for 3 s. Item is performed separately for left and right	No correlations

^aQuotes for whole group only; for details, see Table 3.

For the video‐ranking procedure, 22 video recordings from the 2 healthy controls and a subset of 20 patients representing the complete range of observed symptom severity were drawn from all recordings (Table 2). This number of patients was considered appropriate to establish and train machine‐learning algorithms. Healthy controls in this context serve as anchor reference for ranking. For each item, pairs of videos were simultaneously presented on screen (Fig. 1) to 3 to 4 raters (A.A.K, K.L., P.K., T.E., T.S.‐H.), who performed the ranking independently. Raters determined for each pair whose performance in the specific item was more affected by dystonia. Equality was a possible option. Multiple permutations of video pairs were algorithmically generated until a complete rank order of all 22 cases was achieved.

Table 2.

Summary of Patient Characteristics and Clinical Ratings for all Subjects and Relevant Subgroups

Characteristics	ALL (Patients) n = 34 (21 female) cervical dystonia: 24; segmental dystonia: 2; generalized dystonia: 8			TREMOR Subgroup n = 17 (15 female) cervical dystonia: 13; segmental dystonia: 2; generalized dystonia: 2			Video‐ranked patients n = 20 (11 female) cervical dystonia: 13; segmental dystonia: 1; generalized dystonia: 6			ALL (Healthy controls) n = 2 (1 female)
	Mean (standard deviation)	Median	Range	Mean (standard deviation)	Median	Range	Mean (standard deviation)	Median	Range	Result of female/male control
Age	56.54 (±16.03)	58	27–84	55.9 (±15.8)	58	27–78	55.62 (±17.05)	53	29–84	66/20
VAS Pain	2.77 (±2.76)	2	0–10	2.39 (±2.2)	2	0–7	2.05 (±2.43)	1	0–7.5	0/2.5
VAS symptom severity	4.9 (±2.67)	5	0–10	4.19 (±2.86)	4	0–10	5.05 (±2.55)	5	0–10	0/0
CDQ‐24	39.79 (±20.18)	42	2–78	46.44 (±15.6)	53	12–63	38.5 (±19.94)	41	2–78	0/5
TWSRTS (motor scale)	12.24 (±5.65)	12	1–22	11.0 (5.42)	11	1–22	12.71 (±6.22)	13	1–22	0/0
BFMDRS (motor scale)	31.31 (±19.61)	32.5	5–52	14.5 (±13.44)	24	5–48	25.01 (±18.73)	23.25	5–52	0/0
Relative clinical score	32.15 (±16.5)	32.5	3–63	29 (±16.1)	31	3–63	30.81 (±18.23)	34	3–63	0/0

Relative clinical score as BFMDRS or TWSTRS motor score as % of each scale's maximum (minimum–maximum, 0–100); TWSTRS or BFMDRS are displayed for cervical/segmental or generalized dystonia, respectively; the segmental dystonia case involves arms and cervical dystonia.

CDQ‐24, craniocervical dystonia questionnaire (minimum–maximum score, 0–96); VAS, visual analogue scale (minimum–maximum score, 0–10); TWSTRS, Toronto Western Spasmodic Torticollis Rating Scale (minimum–maximum score, 0–35, n for cervical and segmental dystonia is stated in the headings); BFMDRS, Burke‐Fahn‐Marsden dystonia rating scale (minimum–maximum score, 0–120, n for generalized dystonia is stated in the headings).

Figure 1.

Buttons for severity rating are bottom center (options: left is worse—both equal—right is worse). Buttons stating observations of phasic phenomena within the video clip are bottom right and left corners (options: yes—no per subject).

Importantly, pairwise video comparisons were conducted without previous standardization of rating criteria. Only after completion, raters were asked to outline their individual ranking criteria for each item, eg, movement amplitude, velocity, and smoothness. Two raters were experienced clinicians specializing in movement disorders, 1 was a postdoc with intermediate experience, and 2 were doctoral students. All raters were blinded, as they did not clinically evaluate the subjects before ranking.

Statistics

The Rank Centrality sorting algorithm, an iterative rank aggregation algorithm,29 was used to generate a rank order of the pairwise comparison. Thus an order of the 22 selected cases—based on observed impairment in task performance—was obtained for each item separately and was termed video rank order.

Inter‐rater reliability of ranking results was described as normalized Kendall tau distance with, eg, 0.3, meaning that 30% of pairs differ in rank order between 2 raters.30

For items, in which an acceptable inter‐rater agreement of rankings was identified (Kendall tau <0.3), the rankings of different raters were pooled into 1 order applying the rank centrality algorithm.

Results of the 2 dystonia rating scales (TWSTRS and BFMDRS) were transformed into a percentage of the respective scale's maximum score for comparison, and this was termed the relative clinical score.

The associations of clinical and self‐rating scales, and video‐based rankings where available, were explored by Spearman rank correlations and were repeated for the subgroups with or without tremor. Statistical analyses were performed with SPSS, version 22 (IBM Corp., Armonk, NY).

Results

Clinical Characteristics

In total, 36 subjects (34 dystonia and 2 controls) were assessed with clinical scales and self‐rating. Clinical scores were missing for 1 patient and CDQ‐24 sum scores were missing for 2 subjects. In 17 patients an additional dystonic tremor was noted on clinical examination, including 11 patients with head tremor and 6 patients with combined head and arm tremor. For details on study cohort and subgroups, see Table 2.

Correlations Among Clinical Scales and Patients’ Self‐Evaluation

In the whole group of 34 dystonia subjects, the relative clinical score was not correlated to self‐ratings of symptom severity or pain according to VAS. No correlation of CDQ‐24 could be perceived with symptom severity VAS, but a fair correlation with pain VAS self‐ratings was evident (r = 0.54, P = 0.02; Table 4). Unexpectedly, higher relative clinical scores were associated with lower CDQ‐24, indicating better hrQoL with more severe dystonia (r = −0.51, P = 0.03; Fig. 2).

Table 4.

Correlations Among Clinical Evaluation and Patients’ Self‐Rating Presented as Spearman Rank Correlation Coefficient

Group	Spearman correlation	Relative clinical score	VAS symptom severity	VAS pain
All (n = 34)	Relative clinical score
	VAS symptom severity	0.089
	VAS pain	−0.021	0.165
	CDQ‐24	−0.509*	0.287	0.542*
Non‐tremor(n = 17)	Relative clinical score
	VAS symptom severity	0.182
	VAS pain	0.02	0.263
	CDQ‐24	−0.231	0.225	0.457
Tremor(n = 17)	Relative clinical score
	VAS symptom severity	−0.131
	VAS pain	−0.097	0.035
	CDQ‐24	−0.289	0.395	0.636

*P < 0.05.

CDQ‐24 has only been applied in the subset of probands included in pairwise video comparisons described in Methods (All: n = 18; Nontremor: n = 10; Tremor: n = 8).

CDQ‐24, craniocervical dystonia questionnaire; VAS, visual analogue scale.

Figure 2.

Relative clinical score is BFMRS or TWSTRS motor score as % of each scale's maximum (minimum–maximum, 0–100).

Video Ranking Procedure by Pairwise Comparison

The ranking procedure was well applied according to all raters and took between 15 and 40 minutes per item, depending on the number of videos to be compared (52–88 permutations per item). Items, in which the performance of the left and right side were recorded separately, were pooled for the ranking procedure and required an increased number of permutations and more time for ranking, accordingly.

Algorithmic ranking scores could be retrieved for 20 of 24 items. For the items sitting with headrest, head flexion, holding arms in front palm up, and trunk site inclination, the rank centrality algorithm did not yield a rank order with the given video permutations (algorithm failure). A possible reason for this could be the small number of rankings, which does not fit the requirements of the connectivity of the underlying graph model. This is a random effect for these 4 particular items and is rater‐independent.

In the remaining 20 items for which a video rank order could be retrieved, the inter‐rater agreement described by normalized Kendall tau distance was acceptable (≤0.30) in 15 items (Table 3). The 5 items in which agreement was too low were excluded from rank‐order correlation analysis (object–nose and object–object pointing tasks, shoulder elevation, arm abduction dynamic, and foot tapping).

Table 3.

Inter‐Rater Agreement of Video Ranking and Correlations of Video Rank Order with Different Severity Scales^a

Item	Inter‐rater agreement of video ranking	Relative clinical score vs. video rank order			VAS symptom severity vs. video rank order			QoL (CDQ‐24) vs. video rank order
	Maximal Kendall tau	All	Non‐tremor	Tremor	All	Non‐tremor	Tremor	All	Non‐tremor	Tremor
Posture sitting	0.21	0.832***	0.821**	0.862**	0.413	0.61*	0.201	0.042	0.264	−0.084
Posture sitting straight	0.18	0.633**	0.509	0.742*	0.308	0.533	−0.024	0.046	0.028	0.12
Head rotation static	0.19	0.865***	0.893***	0.577	0.222	0.255	0.335	0.022	0.161	0.359
Head rotation dynamic	0.17	0.772***	0.81**	0.427	0.205	0.213	0.31	−0.083	−0.035	0.611
Head tilt static	0.28	0.617**	0.752**	−0.085	0.446*	0.606*	0.122	0.026	0.182	0.359
Head tilt dynamic	0.19	0.540**	0.719**	0.243	0.446*	0.511	0.476	0.077	0.209	0.252
Head extension	0.25	0.506*	0.692**	0.51	−0.029	0.211	−0.402	0.114	0.165	−0.036
Head extension/flexion dynamic	0.23	0.659**	0.895***	0.377	0.065	0.296	−0.305	−0.096	0.182	−0.12
Arms held in front, palm down	0.26	0.253	0.326	0.233	0.367	0.618*	0.311	0.514*	0.427	0.847*
Arms held flexed	0.29	0.304	0.467	0.05	0.492*	0.423	0.647	0.133	−0.127	0.712
Arm flexion dynamic	0.24	0.547*	0.534	0.262	0.305	0.528	−0.325	0.188	0.294	−0.087
Standing	0.21	0.521*	0.53	0.262	0.119	0.389	−0.59	0.105	0.224	−0.319
Walking in place	0.24	0.259	0.242	0.19	0.485*	0.79**	−0.012	0.21	0.455	0.493
Trunk rotation with hips	0.28	0.032	0.319	−0.429	0.372	0.414	0.181	0.139	0.224	0.029
Trunk rotation without hips	0.3	0.222	0.326	0.095	0.686**	0.712**	0.599	0.269	0.392	0.324

^aSpearman Rank Correlation Coefficients. Only shown are the 15 items with acceptable inter‐rater agreement (maximal Kendall tau ≤0.3). ALL: n = 22; Non‐tremor: n = 12; Tremor: n = 10; (Numbers vary for different items and scales).

*P < 0.05, **P < 0.01, ***P < 0.001.

CDQ‐24, craniocervical dystonia questionnaire; QoL, quality of life; VAS, visual analogue scale.

Correlations of Video Rank Order and Clinical Severity Scales

The following part of analysis refers to the 15 items with sufficient inter‐rater agreement on video rank order (Table 1). It should be noted that independent video rank orders are available per item, whereas clinical scoring and the tremor attribute are available per patient. Thus all correlations are described per item (Table 3).

A correlation of the relative clinical scores with video rank order was identified for 10 items, 8 of these related to posture sitting and head movement in all planes (r = 0.52–0.87, P = 0.02 to <0.001); strongest correlations were seen for posture sitting and head rotation static (r = 0.83 and 0.87, P < 0.001). With the exception of posture sitting straight, similar results were obtained in the subgroup without tremor with a considerable increase of strength of correlation for some items (Table 3). In contrast, only video rank order for the 2 posture sitting items correlated with the relative clinical score in the tremor subgroup (r = 0.86 and 0.74, P < 0.01 and < 0.05). Video rank order for arm flexion dynamic and standing correlated fairly with relative clinical score for the whole group (r = 0.55 and 0.52, P = 0.01 and 0.02) but not in any of the subgroups.

Concerning patient‐based severity ratings, an expected positive correlation of higher VAS symptom severity with higher video rank order was seen in 5 of 15 items in the whole group (head tilt static and dynamic, arms held flexed, walking in place, and trunk rotation without hip; Table 3). In 2 of these items (head tilt static and dynamic) a correlation was also identified with relative clinical score. In the nontremor subgroup, the correlations remained significant for 3 items, whereas in the subgroup with tremor, no correlations were seen between video rank order and VAS symptom severity.

Interestingly, the HrQoL‐scale CDQ‐24 (sum score) did not correlate with video rank order in the analysis of any item except arms held in front, palm down for the whole group and the tremor subgroup (r = 0.514 and 0.847, P = 0.02).

Discussion

We propose a severity rank order obtained by pairwise video‐comparison and independent of predefined scoring systems as a possible validation standard for research purposes in dystonia patients, where biometric limitations of existing rating scales have been reported.

The main result in support of this is the high agreement of rank orders achieved independently by up to 4 video raters with different levels of expertise in 15 of 20 items. Disagreement was higher in finger pointing tasks, trunk side inclination, shoulder elevation, foot tapping, and arm abduction. As one possible explanation, only minimal differences of performance in these items may have led to difficulties for each rater, and consequently between raters, to achieve consistent ratings. In fact, all of these items do not involve the body regions most affected in our cohort of mainly cervical dystonia subjects. As another explanation for lower rater agreement, the implicit rating criteria on which raters based their ranking decisions may differ more in these items, that is, different aspects of performance, such as regularity, amplitude, or velocity of movement, might be weighted differently by different raters. Reevaluating these items after a previous briefing of the raters could test this. However, we consider both explanations as justifications to exclude these items from our assessment battery, as both factors are likely to hamper instrumental movement analysis in the same way.

Failure to achieve a video rank order at all in 4 other items (Table S1) is considered a rater‐independent effect. The existence of only minimal differences of performance among subjects in certain items may also have contributed here. Concerning the practicality of the video‐ranking approach, the time to create video rankings varied per item and rater, because the number of video pairs per item was not determined beforehand, but was procedurally defined until a complete rank order was achieved. However, we consider the estimated average of about 30 minutes for the ranking of 22 video cases feasible in the context of scientific purposes.

Two points support our hypothesis that video ranking may capture clinical motor severity of dystonia more comprehensively than existing clinical scales. First, our raters acknowledged in their comments that the presence of tremulous symptoms influenced their rankings. Second, for all items that involve head position or head movement, correlations of video rank order with the results of the clinical scales were only seen in the whole group and the subgroup without tremor, but not in the tremor group. We interpret this fact as the tremor impacting ranking decisions but not scores of the applied dystonia rating scales. In contrast, for the item posture sitting, an invariably strong correlation between ranking and clinical scales was seen irrespective of the presence or absence of tremor. This may be interpreted as abnormal posturing being the main clinical feature considered for video ranking (as well as clinical scoring) in this item. The lack of correlation of video rank order with clinical rating scales for some items is partly attributable to a subject group that included mainly cervical dystonia patients.

We explored the relationship of both, video rank order (per item) and relative clinical score (by patient), to the patients’ self‐rating of symptom severity, pain, and hrQol. Correlation analysis with CDQ‐24 yielded a moderate correlation with video rank order for only 1 item (arms held in front, palm down) with even stronger correlations in the tremor subgroup. It is conceivable that this item represents a movement of higher functional relevance than the remaining items. With respect to relative clinical ratings, a moderate, but unexpectedly negative, correlation to CDQ‐24 score was noted for the whole group, which seems not explainable by single outliers or a phenotypic subgroup (Fig. 2).

This considerable divergence between hrQoL ratings and ratings based on motor assessment in dystonia is in line with previous data. The original CDQ‐24 publication reported only weak correlations of CDQ‐24 subscales with the Tsui score (r = 0.18–0.29).31 The highest of those correlations was with the activities‐of‐daily‐living (ADL) subscale.28 Similar results were replicated in a large study in >500 de novo cervical dystonia patients.32 Such findings can be attributed to different underlying constructs: perceived quality of life is only partially determined by motor symptom severity as has been shown in numerous studies in different movement disorders.33 Specifically in the case of dystonia, the perceived stigma,34 affective symptoms,35 and pain28, 32, 36 have an impact in addition to the perceived effect of treatment and other sociodemographic factors.35, 37 Consequently, current guidelines recommend consideration of these nonmotor aspects in the setting of treatment goals and the use of hrQoL instruments as secondary outcome measures to assess treatment efficacy.38, 39 Although clearly our study was not designed to make inferences on the determinants of hrQoL in dystonia, our observations fit previously defined factors, given the high ratings in the CDQ‐24 stigma subscale and the association of VAS pain ratings with CDQ‐24 in our cohort. In our tremor subgroup, the average CDQ‐24 score was higher despite lower average clinical ratings (Table 2), and the correlation coefficients for CDQ‐24 versus VAS pain and versus video rank order (arms held in front, palm down) were both higher in tremor subgroups than for the whole group. This fact suggests an impact of dystonic tremor on hrQoL in dystonia patients. To our knowledge, this aspect has not been studied to date.

Correlations of video rank order with VAS self‐ratings of symptom severity do not parallel those for relative clinical score nor for CDQ‐24 sum score. Similarly, VAS symptom severity is neither correlated to relative clinical score nor to CDQ‐24 sum score in the whole cohort. Thus the construct of the VAS symptom severity remains questionable. Although patients were instructed to provide VAS ratings according to their perceived symptom severity at the time of assessment, it is conceivable that such ratings are confounded by perceived functional impairment in daily life.

In conclusion, we propose video rank order as a more comprehensive criterion of clinical severity of dystonia than current clinical rating scales. However, it should be kept in mind that both—video ranking and clinical scales—rely exclusively on the assessment of motor symptoms and are likely to miss other important aspects of the disease. Likewise, any instrumental movement analysis based on such ratings will need the complementary use of other tools, for example questionnaires, to capture nonmotor aspects of therapeutic relevance. The general perspective in developing instrumented movement analyses is, thus, not to replace all other types of severity scorings, but rather to allow therapeutic evidence to be obtained more efficiently. This can only be achieved after the sensitivity and reliability of such instrumental assessments have been demonstrated.

Finally, we point out that the tool described here is a research instrument. It is not applicable for routine clinical assessment, as rankings by construct represent the intragroup relations in a defined (small) group of subjects. Thus, ranking scores cannot be generated for a single subject independently. Although exemplified here in a cohort of dystonia patients, the feasibility and usefulness of the video ranking approach may be generalized to other conditions,40 for example gait disorder, in which clinical scoring is considered incomprehensive and of low sensitivity. The method may further be generalized to different kinds of movement analysis tools provided that videos for ranking are acquired from the same set of subjects on which the algorithms will be trained. The main potential of our method is seen for the training and evaluation of machine‐learning algorithms for automated movement analysis that we aim to apply to the Kinect depth data recordings of our cohort.

Author Roles

1. Research Project: A. Conception, B. Organization, C. Execution; 2. Statistical Analysis: A. Design, B. Execution, C. Review and Critique; 3. Manuscript Preparation: A. Writing the First Draft, B. Review and Critique; 4. Video‐Rating Software: A. Conception, B. Design and Development; 5. Ranking Analysis: A. Design and Development.

A.U.B.: 1A, 1B, 2A, 2C, 3B

T.E.: 1A, 1B, 1C, 2A, 2B, 3A

E.G.: 1C, 2C

F.H.: 1C, 2C, 3B

T.S.‐H.: 1A, 1B, 1C, 2A, 2C, 3B

B.K.: 4B, 5A

P.K.: 1C, 2C, 3B

A.A.K.: 1B, 1C, 2C, 3B

K.L.: 1C, 3B

A.L.: 1A, 2C, 3B

S.M.‐M.: 2C, 3B, 4A, 4B, 5A

K.O.: 2C, 3B, 4A, 4B, 5A

F.P.: 1C, 2C, 3B

A.S.: 1C, 2C, 3B

Disclosures

Funding Sources and Conflict of Interest: Alexander U. Brandt is cofounder and consultant of the technology startup Motognosis, which is a commercial entity with an interest in offering Microsoft Kinect‐based motion‐analysis products. Tanja Schmitz‐Hübsch is supported by NeuroCure FlexFund. Andrea A. Kühn is supported by NeuroCure FlexFund. Friedemann Paul is supported by DFG Exc 257. The study was performed with an institutional grant (NCRC Flex Fund). The remaining authors report no sources of funding and no conflicts of interest.

Financial Disclosures for the previous 12 months: Alexander U. Brandt holds stock options on Motognosis and received research grants and consultancy fees from Novartis, Bayer, and Biogen. Tanja Schmitz‐Hübsch is supported by Bundesministerium für Wirtschaft und Energie, received funding from Deutsche Forschungsgemeinschaft (DFG KFO 247) and Ipsen Pharma, and received travel support from Ipsen Pharma. Bastian Kayser is an employee at Motognosis UG. Patricia Krause is funded by the German Research Foundation (DFG KFO247). Andrea A. Kühn is funded by the German Research Foundation (DFG KFO247), holds a grant from NeuroCure FlexFund for this project, has received honoraria from Medtronic, St Jude Medical, and Boston Scientific, and has received travel support from Ipsen Pharma and Merz. Axel Lipp is funded by the German Research Foundation (DFG LI 1301/4‐1). Sebastian Mansow‐Model is an employee at Motognosis UG and holds stock options in Motognosis UG. Karen Otte is an employee at Motognosis UG. Friedemann Paul is supported by Deutsche Forschungsgemeinschaft (DFG Exc 257), Bundesministerium für Bildung und Forschung (BMBF Competence Network Multiple Sclerosis), the EU FP7 program (combims.eu), the Guthy Jackson Charitable Foundation, and the Arthur Arnstein Foundation Berlin. Friedemann Paul has received research support and personal compensation for activities with Bayer, Biogen, Alexion, Chugai, MedImmune, Teva, MerckSerono, SanofiGenzyme, and Novartis. The remaining authors declare that there are no additional disclosures to report.

Supporting information

Table S1. Motor Items Designed for Microsoft Kinect Recording but Excluded from Video Rank Order Analysis According to Results of Ranking Procedure.