Auto-Gait: Automatic Ataxia Risk Assessment with Computer Vision on Gait Task Videos

Abstract

Many patients with neurological disorders, such as Ataxia, do not have easy access to neurologists, –especially those living in remote localities and developing/underdeveloped countries. Ataxia is a degenerative disease of the nervous system that surfaces as difficulty with motor control, such as walking imbalance. Previous studies have attempted automatic diagnosis of Ataxia with the help of wearable biomarkers, Kinect, and other sensors. These sensors, while accurate, do not scale efficiently well to naturalistic deployment settings. In this study, we propose a method for identifying ataxic symptoms by analyzing videos of participants walking down a hallway, captured with a standard monocular camera. In a collaboration with 11 medical sites located in 8 different states across the United States, we collected a dataset of 155 videos along with their severity rating from 89 participants (24 controls and 65 diagnosed with or are pre-manifest spinocerebellar ataxias). The participants performed the gait task of the Scale for the Assessment and Rating of Ataxia (SARA). We develop a computer vision pipeline to detect, track, and separate the participants from their surroundings and construct several features from their body pose coordinates to capture gait characteristics such as step width, step length, swing, stability, speed, etc. Our system is able to identify and track a patient in complex scenarios. For example, if there are multiple people present in the video or an interruption from a passerby. Our Ataxia risk-prediction model achieves 83.06% accuracy and an 80.23% F1 score. Similarly, our Ataxia severity-assessment model achieves a mean absolute error (MAE) score of 0.6225 and a Pearson’s correlation coefficient score of 0.7268. Our model competitively performed when evaluated on data from medical sites not used during training. Through feature importance analysis, we found that our models associate wider steps, decreased walking speed, and increased instability with greater Ataxia severity, which is consistent with previously established clinical knowledge. Furthermore, we are releasing the models and the body-pose coordinate dataset to the research community – the largest dataset on ataxic gait (to our knowledge). Our models could contribute to improving health access by enabling remote Ataxia assessment in non-clinical settings without requiring any sensors or special cameras. Our dataset will help the computer science community to analyze different characteristics of Ataxia and to develop better algorithms for diagnosing other movement disorders.

1. Introduction

Ataxia is a general term referring to a lack of muscle coordination and control, typically arising due to damage to the cerebellum in the brain (Mayo-Clinic-Stuff, 2021). Common symptoms include difficulties walking, deterioration of fine motor skills, imbalance, and trouble eating and swallowing (Seladi-Schulman, 2020). In this paper, we analyze two forms of a hereditary ataxia named spinocerebellar ataxia (SCA) types 1 and 3 (SCA1 and SCA3) which demonstrate the fastest progression rate and highest prevalence respectively (Jacobi et al., 2012).

Early detection and consistent monitoring can greatly improve symptom management and long-term patient prognosis (Mayo-Clinic-Stuff, 2021). The Scale for the Assessment and Rating of Ataxia (SARA) is a popularly used scale for measuring ataxia progression with high inter-rater reliability, inter-rater consistency, and test-retest reliability (Schmitz-Hübsch et al., 2006; Perez-Lloret et al., 2021; Hartley et al., 2015; Bürk et al., 2009). SARA scale has eight different tasks: gait, stance, sitting, speech disturbance, finger chase, finger-to-nose, fast alternative hand movement, heel-shin slide. These tasks are performed by a participant during a clinical visit in front of a trained neurologist, who observes the patient and assigns them a score for each tasks. Although this overseeing nature of the evaluation ensures the quality, it limits the accessibility and equity of the care. People without the access to a high quality healthcare are deprived of this care, and even people with access to it are limited to the frequency of their visit because of the time commitment of each visits. In this paper, we try to alleviate this problem by developing a computer-vision based video analysis technique for automatically detecting SCA.

In this study, we have analyzed the gait task in SARA evaluation metric as, 1) SCA heavily impacts common gait characteristics, and 2) gait provides objective, clinically applicable measurements (Buckley et al., 2018). We have collected 183 recorded videos of SCA patients performing Gait tasks from 11 different medical sites in the US. The videos were annotated with SARA score by trained neurologists who observe the person during the walk. Thus, we address the following research questions:

1. RQ1: Can we provide an early-stage screening of ataxia from the SARA gait task with computer vision?

2. RQ2: Can we measure early-stage severity of ataxia from the SARA gait task with computer vision?

In order to answer those questions, we apply computer vision to quantify different subtle relevant features of the participant’s walking. Figure 1 demonstrates our pipeline step by step. First, using an object detection and tracking algorithm, we detect every person from a video frame and track them across the video (Fig. 1 B). We develop a robust Subject Separation Algorithm based on the characteristics of Gait task, that separates the participant from several other people in the frame (Fig. 1 C). Furthermore, we measure their movement using a pose estimation model and use this information to build meaningful signals for detecting SCA (Fig. 1 D). For our Risk-prediction task (RQ1), we divided our data into two classes: one with gait task SARA scores of 0 (Not in risk: Class 0) and another with gait task SARA scores > 0 (In risk: Class 1). We train a machine learning classifier that can differentiate between these two classes with an 83.06% binary accuracy from 10-fold cross-validation (Table 3). For early-stage Severity-assessment (RQ2), we divided our dataset into four classes with gait task SARA scores: 0, 1, 2, > 2. For all data with a gait task SARA score > 2, we replace their label with 3 since we have very few data with gait task SARA scores > 3. Our Severity-assessment regression model has a mean absolute error of 0.6225 (STD 0.0132) and a Pearson’s correlation coefficient of 0.7268 (STD 0.0144). For both of these models, we ran SHAP analysis (Lundberg et al., 2018). Figure 7(a)subfigure and Figure 7(b)subfigure, corresponding to the Risk-prediction and Severity-assessment models respectively, show that SHAP can provide intuitive, previously-validated results (Sec 4.3). Furthermore, we ran a “leave-one-site-out” analysis, where we exclude data from one site in the training set and test the model’s performance on that site’s data. This allows us to assess how well the model performs on completely unseen data. While the performance of both models decreases through this method, they still perform significantly better than random chance (Table 3) and zero-correlation (Table 4).

Figure 1.

Ataxia risk/severity prediction pipeline from Gait task.

Table 3:

Ataxia risk prediction (binary classification) model performance. The model was evaluated in 20 iteration of 10-fold cross-validation and leave-one-site-out manner (Section 3.6).

Test	F1 score (%)		Accuracy (%)
	Mean	STD	Mean	STD
10-fold CV	80.23	9.19	83.06	6.79
Site 2	80.5	2.65	80.54	2.64
Site 3	65.34	5.04	65.79	4.85
Site 4	74.65	6.32	81.43	6.55
Site 5	66.94	5.83	72.86	4.29
Site 6	94.55	16.36	98.33	5.0

Figure 7:

Features’ impact on models prediction computed by SHAP (Lundberg et al. (2018)). (a) Risk prediction model (binary) and (b) Severity assessment model (regression)

Table 4.

Ataxia risk prediction (binary classification, excluding all data with label 4,5,6) model performance. The model was evaluated in 20 iterations of 10-fold cross-validation and leave-one-site-out manner (Section 4.6).

Test	F1 score (%)		Accuracy (%)
	Mean	STD	Mean	STD
10-fold CV	80.21	9.89	81.78	9.32
Site 2	73.07	3.31	73.15	3.29
Site 3	70.8	8.69	71.05	8.57
Site 4	77.01	3.33	82.5	3.63
Site 5	69.86	8.42	75.0	6.19
Site 6	100.0	0.0	100.0	0.0

Our contributions are as follows:

1. We collected a large high-quality annotated dataset of patients with Ataxia and control group. The dataset contains 155 annotated video from 89 participants that spans several years representing different geographic locations and demographics (Table 2). Upon acceptance, we will release an anonymized version of our dataset containing body pose coordinates to the research community for developing better algorithms for ataxia assessment. To the best of our knowledge, it will be the largest accessible dataset on Ataxia analysis from gait task.

2. We designed a robust and automated computer vision pipeline that can detect, track, and separate out participants from videos across different site settings that include both doctors and other passersby, and track their body pose movement.

3. We developed machine learning algorithms for Risk-prediction and Severity-assessment of ataxia from recorded videos of the SARA gait task. Our models achieve 83.06% accuracy and an 80.23 F-1 score for the Risk-prediction task (Table 3). For the Severity-assessment task, we achieve mean absolute error (MAE) of 0.6225 and a Pearson’s correlation coefficient of 0.7268 (Table 4).

4. We test our models with completely unseen data using leave-one-site-out method and show that our models generalizes beyond the sites they were trained on (Table 3 and Table 4).

Table 2:

Description of features. Since we initially extract a sequence of features for each Feature group entry (except height reduction), we calculate six sub-features to represent that sequence: mean, std-deviation, max, min, range (max-min), entropy. SL and SW represent step-length and step-width respectively. The values 1 and 2 in Source column represent (Buckley et al., 2018) and (Honda et al., 2020) respectively.

Feature group	Measurement	Estimation for	Source
height_reduction	Total height reduction for the 6s walk (equation 1)	Speed	1
feet_dist	Distance between feet	SL, SW, swing	1, 2
feet_angle	Angle between the x-axis and line joining feet	Swing, SL	1
right_x	X-Coordinate of right feet	SL, SW, swing	1, 2
right_y	Y-Coordinate of right feet	SL, SW, swing	1, 2
left_x	X-Coordinate of left feet	SL, SW, swing	1, 2
left_y	Y-Coordinate of left feet	SL, SW, swing	1, 2
left_knee bent	Angle between the left thigh and knee	Swing	1
right_knee_bent	Angle between the right thigh and knee	Swing	1
imbalance	Horizontal deviation of the mid-shoulders and hip	Stability	2
tilt	Angle of shoulder line with horizontal axis	Stability	2
nose_x	X-coordinate of the nose	Stability	2
nose_y	Y-coordinate of the nose	Stability	2
stance	% of frames where the participant was stationary	Stance phase	1

2. Related Works

Objectively assessing Ataxia features is an active area of interest for many researchers. Although prior efforts mostly focused on using sensors to assess several aspects of Ataxia [28, 37, 55], there is a growing interest in the use of telemedicine-type platforms to increase the accessibility of care for Ataxia patients in non-clinical settings.

2.1. Automated Gait Analysis

Automatically analyzing walking patterns and gait have been studied using various ways. For example, RFID tags [24], wifi signals [58, 60], acoustic signals [59], cellphone [25], accelerometers and sensors[1, 2, 18, 32, 36, 38], Kinect [61], cell-phone [25], etc., all have been shown to be successful in analyzing gait. However, these sensorbased technologies, although accurate in lab settings, sacrifice the accessibility of healthcare. Only a handful of people who own and have the ability to operate these sensors can benefit from such technologies. A more ubiquitous way of sensing gait would be analyzing videos of a participant walking. Previous studies [27, 39] have attempted analysis of gait for clinical application using computer vision. Yet, video-based gait analysis has mostly been explored in a simplistic studio setup – the data is collected from a single site without background noise. Furthermore, the videos are captured from a lateral view requiring a large space, which is unsuitable for homes and even many clinics. In this paper, we introduce a robust video-based gait analysis method trained on videos captured in a more realistic setting. Our method automatically identifies a patient when multiple persons are visible in a video and works across multiple sites. Such generalizability was missing in prior literature.

2.2. Teleneurology for Neurological Disorders

The elderly population is the riskiest group for the most common neurological disorders like Ataxia and Parkinson’s disease. Arranging clinical visits for them can be difficult since a clinical visit requires a long commute unsuitable for the elderly, which has been exacerbated by the COVID-19 pandemic. Amid the ongoing public health crisis, experts have recommended greater use of telemedicine to care for patients with such neurological disorders [34]. A shift to telemedicine would be beneficial on two fronts; it would improve the frequency of assessment (therefore painting a more realistic picture of a patient’s disease experience), and increase equity by decentralizing care from the clinic. The application of telemedicine in the domain of Parkinson’s disease (PD) sets an excellent precedent for this domain [14], especially considering the similarities between PD and Ataxia symptoms. CloudUPDRS [49] is a smartphone application recognized as a Class I medical device by MHRA, UK that captures data from people with PD for future clinical diagnosis. The app employs a deep learning model to ensure the quality of data collected without human supervision [50]. Another smartphone application named mPower [6] can objectively measure PD tremor severity using remotely collected activity data with built-in smartphone sensors. PARK [30] is a web-based framework that collects webcam videos from subjects for detecting Parkinson’s disease. However, compared to PD, remote assessment of Ataxia has received less attention until recently. Lately, Researchers are developing telemedicine care for patients with Cerebellar Ataxia (CA) [34]. Notably, Grobe et al. [17] developed SARAhome, a video-based instrument to measure Ataxia severity from a subject’s home. Independently, a user performs 5 of the 8 SARA items; items were modified so that the tool could be easily self-applied without the presence of a provider. After all the tasks are completed, the recorded videos are securely transferred to be evaluated by an Ataxia expert offline using SARA criteria. Separately, Summa et al. [52] created a pilot study for SaraHome, a tool that interlinks the Microsoft Kinect 2.0 and Leap Motion Controller(LMC) for automatic detection of SARA scores. This platform was intended for use by children with early onset Ataxias (EOAs); with help from a caregiver, users complete six of the eight SARA tasks to receive an objective, automated assessment of Ataxia severity [52]. The tool was well-received; however, the aforementioned pilot study was only intended to gauge participant reception and the feasibility of the tool. To date, it has not been tested on actual video footage to assess its performance compared to traditional measures. The emergence of these things platforms shows a growing need for automatic home diagnosis tools for CA where our automatic video-based analysis tool would be valuable.

2.3. Detecting Ataxia from Gait Task

The gait task is a significant indicator of Ataxia since patients with CA may demonstrate clumsy, staggering movements while performing the gait task [48]. In the literature, detecting CA from gait has mainly focused on sensor-based analysis. Both wearable sensors and the Kinect system have shown promise in analyzing CA from gait task [15, 20, 26, 31, 42, 53]. LeMoyne et al. [31] gathered accelerometry and gyroscopic data by mounting wireless inertial sensors on the ankle joints of participants and trained a machine-learning model to distinguish subjects with Friedreich’s Ataxia from healthy controls. Phan et al. [42] used data from a single gait task to capture features at self-selected slow, preferred, and fast walking speeds for both detecting CA and evaluating its severity. Honda et al. [20] analyzed the gait task by extracting features using a Kinect-based system. Most recently, Dostál et al. [15] obtained accelerometry data using 31 time-synchronized sensors placed on different parts of the body. They obtained an accuracy of 98.0%, and 98.5% in classifying between healthy subjects and subjects with CA, using neural networks extracted features from the shoulders and head/spine respectively. Recently a similar approach obtained 95.8% accuracy in classifying Ataxic and normal gait by using deep learning to analyze frequency components of accelerometric signals simultaneously recorded at multiple body positions. Similarly, Prochazka et al. [43] applied accelerometric signals to train deep learning-based models to achieve 95.8% accuracy in differentiating between ataxic and normal gait. While the above-mentioned studies depend on using external sensors, Ortells et al. [40] explored automated video assessment to analyze Ataxic gait. However, the dataset used in the study was simulated by 10 healthy volunteers, so it did not contain real data from subjects with Ataxia.

2.4. Detecting Ataxia through Video Analysis

While automated video analysis is challenging, it has the potential to increase accessibility and reduce the subjectivity of care. It can offer preliminary screening or diagnosis for subjects that do not have access to clinical care. Automated video analysis can also help doctors by providing objective metrics, eventually reducing the subjectivity in assessments. Although there has been limited work on the video-based analysis of CA, the work of Jaroensri et al. [23] is notable. They introduced an automated method for quantifying motion impairment of Ataxia patients by analyzing video recordings of the Finger to Nose Test. They used a neural network for pose estimation and optical flow techniques to track participant hand motion. Their models performed comparably to traditional clinical assessments of Ataxia severity. However, video-based analysis of CA for the gait task is not well-studied in the literature.

3. Dataset and Clinical Annotation

For this study, we collaborated with 11 different clinical sites across 8 different states in the United States and collected a dataset consisting of 183 recorded videos of participants completing the SARA gait task at their annual visits. During this task, a participant is supposed to walk through a corridor for approximately 10m, take a turn, and then walk back to the starting point. A neurologist observes the patient on the side and then assigns a score. This score works as the annotation for our supervised machine learning approach. After the videos are captured, a separate clinician observes the video and marks the exact timestamp the walking begins and ends.

We used the following exclusionary criteria to determine participant eligibility:

• Known genotype consistent with that of other inherited ataxia (for example, Freidrich’s ataxia)

• Concomitant disorder(s) that affected assessment of ataxia severity during study

• Changes in physical and occupational ataxia therapy in the two months prior to study enrollment

To reduce the possibility of identification, the participant’s faces were blurred using Mediapipe Face Mesh1 before transferring the data to our site. Upon removing noisy data (missing labels, interruption during walking, fully assisted walking, etc.), we were left with 155 videos from 89 participants. We collected 26, 66, 30, and 32 videos in 2018, 2019, 2020, and 2021 respectively. Among them, 24 were healthy controls and the remaining 65 were either already symptomatic with spinocerebellar Ataxia type 1 or type 3 (SCA1 and SCA3 respectively) or were otherwise pre-symptomatic. We will refer to these 65 participants as People-with-Ataxia (PwA). Across the 65 PwA and 24 controls, we obtained 116 and 39 unique videos respectively. The participants were diagnosed as either control or Ataxia patients at the beginning of the study and diagnosis has not changed for any participant. However, in some of the participants, the stage/degree of Ataxia may have changed which is reflected in the corresponding Gait scores for each video data. Table 1 provides demographic information for the participants.

Table 1:

Demographics information of our participants

Status	#Participants (N=89)	Mean age (std dev)	#Female (percentage)	#Videos
PwA	65	47.07(9.08)	36 (55.38)	116
Control	24	41.73(8.72)	15 (62.5)	39

According to the study protocol, both the control and PwA will provide data once every year for the duration of the study. Figure 5 shows that more than 50% of participants provided data more than once. However, since it is a multi-site longitudinal study, all sites did not start the study simultaneously. Besides, some participants dropped out of the study midway, and thus, we don’t have the complete set of data for all participants.

Figure 5:

Change of bounding box height for each identified person in a sample video (first 6s of walk, 180 frames.)

Figure 4 provides an overview of the data count and the gait-specific SARA score distribution across sites. As can be seen, the first three sites provided most of the videos and have a good variety of videos with different SARA scores. Figure 2 provides snapshots of videos collected at the various locations. Figure 3 provides a breakdown of the distribution of gait-specific SARA scores across all 155 collected videos (PwA and control).

Figure 4:

Number of videos recorded at each site and gait-SARA score [0- 6] mix in those videos.

Figure 2:

Gait walk sample from four sites. We analyzed the first six seconds of data for each video. In all cases, the participants was still going away from the camera after six seconds (corresponding to first two images for each site).

Figure 3:

Number of videos for each SARA score for gait.

4. Methods

Developing a supervised Machine Learning framework for predicting the Ataxia risk and severity from videos requires us to capture subtle differences in the participant’s movement during the walk. Although the SARA evaluation metric is well established in the medical community, the task of predicting it from video data poses several challenges, for example,

1. Each site has different lighting conditions, camera resolution, frame rate, and camera position, and uses different lengths of walking.

2. The framework should be generalizable across different heights, ages, gender, weight, and other physical characteristics of the participants.

3. Each video contains two or more people, i.e. the participant, one or more neurologists, and often time passersby. The framework has to separate participants from non-participants automatically.

4. As the participants walk away from the camera, the camera perspective, and the participant’s observed height change (Figure 2). The framework should be independent of these dynamic changes.

4.1. Dataset Preparation

We split the raw video with the start and end timestamp of the Gait task provided by the clinicians. In order to eliminate the variability of different walking distance across sites, we only consider the first 6 seconds of the walking. During those 6 seconds, all participants were moving away from the camera and did not have time to turn around and return back towards the camera. We split these 6s videos into individual frames. With a frame rate of approximately 30 fps, every video is split into a sequence of around 180 images.

4.2. Person Detection and Tracking

The first step of analyzing Gait task from video would be separating the participants from the background. In computer vision, this is a well-studied task known as Object Detection, and there are several open source models for this task. We experimented with OpenCV Object detection (Bradski and Kaehler, 2000), Yolo V4 (Bochkovskiy et al., 2020), Yolo V5, and FasterRCNN (Ren et al., 2015) and found that FasterRCNN provided most consistent and smooth boundaries for a person across frames. The object detection model takes an image frame as input and outputs a bounding box around each person in the frame. However, the object detection model does not provide any information to track the person across frames. In order to do that, we applied the SORT algorithm (Bewley et al., 2016) that takes the bounding boxes produced by FasterRCNN as input and assigns a unique ID to each person. Thus, we could generate a uniquely identifiable series of bounding boxes for all persons detected in the videos. This is presented in Figure 1, transition A to B.

4.3. Gait Identification Algorithm

In the previous step, we assigned a unique ID for each person in a video. However, separating the participant from the non-participant is a non-trivial problem. There are different types of non-participants captured in the videos, for example, doctors who are evaluating the participant or passersby. The movement of non-participants is less predictable. To isolate the participants from non-participants, we develop a gait identification algorithm based on the nature of the gait task. A participant who is performing the gait task is expected to constantly move away from the camera and therefore consistently shrink in size (Figure 6). If we calculate the cumulative reduction of each person’s height, the participant typically has the highest value (of this metric). For each unique person identified by our object-tracking algorithm, we calculate their cumulative height reduction for $N$ frames with Equation 1. Here $Hi$ represents the height of the bounding box for a person for ith frame.

Figure 6:

height reduction score (equation 1) for incremental frames for each identified person. Participant has the highest value–which helps separate the participant.

$$height\_reduction=\sum _{i=0}^{N-1}(H_i-H_{i+1})$$ (1)

We calculated a $height\_reduction$ for each of the persons tracked; the individual with the highest score has walked the farthest distance consistently, hence, they are the participant (Figure 6). As doctors typically show relatively less forward movement than the participant, and passersby usually appear for fewer number of frames, they automatically get a lower $height\_reduction$ score. If the top two $height\_reduction$ scores in a video are within 90% of each other, it indicates an assisted walk, where the patient’s condition is so severe that they need assistance from the doctor to complete the gait task. We discard these videos from our dataset, as our goal is to identify non-trivial cases of SCA. Finally, we manually observe all outputs and make sure that our method can successfully detect the participant in all videos.

4.4. Gait Pose Estimation

In computer vision, pose estimation is the task of using a Machine Learning model to estimate the pose of a person from an image by calculating the spatial locations of key body joints (a.k.a key points). Using pose estimation models we can precisely track different key points of a participant’s body, and use this for classifying the Ataxia risk. We experimented with several open-source pose estimation models, and we found MoveNet2 in the Tensorflow library to provide consistent pose estimation. Once the participant is identified in a video in the previous section, we crop the participant’s bounding box from each frame and upscale or downscale them to uniform 800 × 400 pixel images. This scaling eliminates the variation in the dataset regarding participants’ height, distance from the camera, camera resolution, and perspective. From each one of these images, we use MoveNet to detect 17 key points of the participant’s body–which is later used to calculate meaningful features for our prediction task.

4.5. Feature Extraction from Pose

After separating out the frames containing the participant of interest, we extracted several features based on previously validated gait characteristics of SCA patients [8]. Notably, these features reflect common Ataxia gait features including step width, step length, swing, stability, stance phase, swing phase, and speed. Table 2 provides a short description of our features. Studies found reduced walking speed as an important characteristic of SCA patients [8]. For our case in a fixed time window (6 seconds), walking speed will translate to the furthest distance covered–which is indirectly captured by our height reduction score calculated in Section 4.3 (a person who travels further, would get smaller quickly). We use the height reduction score calculated for each patient as a feature for the classifier. The next 13 entries in Table 2 represent discrete signals, each consisting of nearly 180 timestamps. In order to capture learnable parameters from these signals, we identified six aggregate quantities: mean, std-deviation, max, min, range (max —min), and entropy. The entropy of a signal measures the irregularity in the data. Mathematically, a periodic signal will have zero entropy, and a random signal will have high entropy. As SCA patients have difficulties maintaining a consistent rhythm in their walking, entropy-related features should effectively capture these irregularities in a walking pattern. We used an open-source Python implementation [56] of the spectral entropy method originally designed to capture irregularities in EEG brain signals [21]. Thus, we finally have 79(1 + 13 * 6 = 79) features to build our models from which we select the final features using Recursive Feature Elimination with Cross-Validation (RFECV) algorithm.

4.6. Ataxia Risk-Prediction

Our first goal in this study (RQ1) is to automatically recognize an individual’s risk of having SCA. In order to do that with the help of our annotated dataset, we label the participants with a SARA gait score of 0 as ‘risk-free’ (Class 0) and participants with a score > 0 score as ‘at risk’ (Class 1) and train a Machine Learning classifier to classify between these two. We employed the recursive feature elimination with cross-validation to select the optimal set of features and train a Random Forest model as the classifier. The complete list of features used for training this model is available in A.1. We evaluated the model using 10-fold cross-validation and leave-one-site-out methodology. In the second case, we trained the model on all sites but one and tested the model on the held-out site. Good performance on this task provides a useful metric for evaluating our model’s generalizability in completely unseen situations.

There are 88 videos in Class 1 and 67 videos in Class 0. Therefore, the accuracy of choosing the majority class is 0.568. Hence, an accuracy of more than 0.568 indicates the model is performing better than random chance. We perform all of the experiments 20 times and average the results to minimize randomness.

To evaluate how our model performs on PwA with subtle symptoms, we ran the same experiment by removing all videos with SARA Gait score > 3. Therefore, gait score 0 is in Class 0, and gait score 1, 2, 3 is in Class 1. From Table 4, we can see that except for a 7% decrease in accuracy for the "Site 2" sub-study, the performance remains on par with those reported previously in Table 3. However, since we are working with two different datasets (due to removing videos with gait scores > 3), we should be cautious about making a direct comparison between these two experiments.

4.7. Ataxia Severity-assessment

Our second goal (RQ2) is to predict the SARA score directly from the gait data, which indicates the severity of the disease. We design this task as a regression problem and use a random forest regression model to predict a continuous SARA score value from the video-extracted data. As our primary motivation is to identify early-stage CA, we focus on lower scores of SARA and categorize all SARA scores of 3 or above to simply 3. The model takes the numeric features from participants’ walking as input and predicts a continuous score between 0 and 3. We use the feature importance score of the random forest algorithm to identify the best features for training this model (a complete list is available in A.2. We evaluate the regression model by calculating the mean absolute error (MAE) and Pearson’s correlation coefficient (PCC).

To predict severity for PwA with mild symptoms, we repeated the above experiment by removing videos with gait SARA score > 3 and reported the performance in Table 6. Although the models’ performances degrade slightly for ’10-fold CV’, ’Site 5’, and ’Site 6’, it improves or remains on par with those reported in Table 6.

Table 6:

Ataxia severity prediction (regression between 0~3 excluding all data with score 4 and above) model performance. The model is evaluated with 20 iterations of 10-fold cross-validation and a leave-one-site-out manner. In the second case, a model is tested on a site that was excluded during training (Section 4.7). The mean and standard deviation of Mean Absolute Error (MAE) (lower is better) and Pearson’s Correlation Coefficient (higher is better) is reported.

Test	MAE		Corr Coeff.
	Mean	STD	Mean	STD
10-fold CV	0.7035	0.0135	0.6885	0.0126
Site 2	0.5644	0.0	0.5738	0.0
Site 3	0.72	0.0	0.6192	0.0
Site 4	0.7187	0.0	0.5796	0.0
Site 5	0.7565	0.0	0.3549	0.0
Site 6	0.5197	0.0	0.0608	0.0

5. Results and Interpretation

5.1. Ataxia Risk-prediction Results

The mean and standard deviation of the 20 runs of the Ataxia Risk-prediction are reported in Table 3. We can see from Table 3 that the model achieves a mean F1 score of 80.23% and an accuracy of 83.06% in 10-fold cross-validation. Clinically, the model is able to distinguish a CA patient from a non-CA subject with above 80% correctness. The low standard deviation between results indicates that the results are fairly consistent between different runs. In leave-one-site-out tests, the model performs fairly well, despite never seeing data from a particular site.

5.2. Ataxia Severity-assessment Results

On 10-fold-cross-validation, our model achieves MAE and PCC scores of 0.6225 and 0.7268 respectively. A lower MAE is better, as it indicates the mean difference between the model’s prediction and the actual SARA score. PCCs range from −1 to 1, where higher values are better, implying a correlation between the prediction and ground truth. We can see from Table 5 that in most cases, the model achieves less than 1.0 MAE through the leave-one-site analysis and its predictions are positively correlated with ground truth SARA scores. The low standard deviation indicates that the predictions are consistent across runs. By manually inspecting 6 data points (Figure 2) in Site 6, we find that although the model has high accuracy in Table 3 and the lowest of all MAE in Table 5, a single misclassification results in a relatively lower but still acceptable correlation score. Figures 9 (a) and 9 (b) visualize the positive correlation through a scatter plot and Gaussian kernel density estimation.

Table 5:

Ataxia severity prediction (regression between 0~3) model performance. The model is evaluated with 20 iteration of 10-fold cross-validation and leave-one- site-out manner. In the second case, a model is tested on a site that was excluded during training (Section 3.7). The mean and standard deviation of Mean Absolute Error (MAE) (lower better) and Pearson’s Correlation Coefficient (higher better) are reported.

Test	MAE		Corr Coeff.
	Mean	STD	Mean	STD
10-fold CV	0.6225	0.0132	0.7268	0.0144
Site 2	0.5673	0.0	0.7081	0.0
Site 3	0.7723	0.0	0.4569	0.0
Site 4	1.0886	0.0	0.2146	0.0
Site 5	0.6334	0.0	0.6244	0.0
Site 6	0.4402	0.0	0.4297	0.0

5.3. Model Interpretation

Figure 8 (a) and 8 (b) give us insights into which features have a greater influence on our models’ predictions. We can see that the coordinates of the left and right feet, as well as the distance of steps, play important roles in model prediction. Based on these summaries, we see that wider steps (min_left_x, max_feet_dist, max_right_x, etc.) and variation in steps (std_left_x, range_feet_dist, etc.) both affect Ataxia scores; this is consistent with [8, 45, 57], who also found that step length, step width, step distance, and differences in step length between the left and right feet are indicative of CA. [8, 45] also reported that individuals with CA have lower walking speeds than healthy controls, which is captured by the height_reduction score. We can see that in both graphs, height_reduction negatively affects the CA score. In addition, [45] found that Ataxia participants’ have oscillatory walking patterns, which are effectively captured through the imbalance features in our model; range_imbalance, std_imbalance, min_imbalance, etc.) contribute to higher SARA scores, thus denoting more advanced Ataxia. Position of the nose (Figure 8 (b)) can also be indicative of oscillation in upper body movement. Overall, Figure 8 (a) and 8 (b) demonstrate that the most significant features in our model are consistent with those identified in prior literature.

Figure 8:

Visualizing the predictions for Severity-assessment task. The red line in (a) denotes ideal scatter plot. The light-green in Gaussian kernel density estimation (b) shows an estimation of the prediction density. The lighter the color, the higher the number of predictions in that region.

6. Limitations

6.1. Study Design and Feature-set

In this study, the classification features are computed with clever engineering grounded on the existing literature. However, a more exhaustive study of Ataxia literature and its effects could reveal more discriminative features that can result in better performance from our dataset. In order to normalize different walking distances between sites, we only considered the first 6 seconds of walking forward part of the video-recorded gait. However, in a standard setting, a patient walks forward, turns, and then walks back to the initial position. Taking the full walk into consideration could potentially provide more information.

With our feature analysis techniques, we were able to computationally explain the different features used for classification with findings from the clinical literature. Our dataset can potentially reveal more such findings about the characteristic of Cerebellar Ataxia (SCA) than what we have been able to demonstrate in the paper. By opening the dataset to the research community, we hope that it will develop novel computational models and interpretable results for detecting Cerebellar Ataxia.

Due to dataset size and the desire to have transparent and interpretable metrics, our effort to achieve high accuracy using the more advanced deep learning techniques did not pan out. Incorporating the latest techniques into our algorithmic workflow remains part of our future work.

6.2. Generalizability

Our dataset was collected on people either diagnosed or at risk of Cerebellar Ataxia (SCA). While SARA can be also used to assess other types of Ataxia aside from SCA – such as Ataxia due to brain tumors and Ataxia telangiectasia [41] – the target populations nonetheless differ. Therefore, our results may not translate to other types of Ataxia. In addition, the videos were collected in clinical settings, and all the participants were assisted by a doctor. It would be interesting to explore whether our findings hold for home-recorded videos, potentially without any supervision of clinical personnel.

7. Future Work

7.1. Home Applications

Our broader methodology of detecting, separating, and tracking a clinical participant and predicting their disease diagnosis by tracking body poses has several clinical applications. With minor modification, our methodology is applicable to Tandem Gait – another task in SARA evaluation. Nonetheless, this work opens up possibilities for the development of computer vision frameworks for more tasks on SARA evaluation. Our video-based analysis method is robust to different environments, backgrounds, and the number of people present in a video frame. Our feature set is also based on movement and independent of the participant’s age, gender, height, weight, and other physical characteristics. This makes our model suitable to be used outside controlled clinical settings and possibly at home setup [17]. Deployment of such a diagnosis tool at home setup will make neurological care accessible for a large population and provide earlier interventions to vulnerable people.

7.2. Differentiating between Different Movement Disorders

Gait is a popular diagnosis task for a different types of movement disorders. For example, both SARA for measuring ataxia severity and the Movement Disorder Society Unified Parkinson’s Disease Rating Scale (MDS-UPDRS) for measuring Parkinson’s include a gait task [16, 51]. Different movement disorders often present similar types of symptoms which results in clinical misdiagnosis. In the words of an Ataxia patient, “I know there are people walking around with exactly what I have and they don’t know it…. It is often misdiagnosed because it resembles other things like multiple sclerosis, stroke, and Parkinson’s disease.”[5]. Our pipeline and dataset can further be used to distinguish between different movement disorders and help reduce misdiagnosis.

7.3. Human-centric Applications

In the future, we plan to expand our work in several directions. For example, we can design systems to help doctors annotate relevant parts in the video that contain important information for detecting Ataxia [10]. Similarly, we can integrate multimodal data – gait task, demographic information, stance task, etc – to better help the neurologists [11]. Moreover, if we want to integrate our tool in the Ataxia diagnosis pipeline, then we will need to build a proper human-centric AI assistant [12] and quantify how it helps neurologists better diagnose Ataxia [13].

8. Conclusion

Ataxia is a rare disorder and its treatment is only available in urban hospitals for which an in-person visit is necessary for diagnosis and continuous treatment of the condition. Although sensor-based systems have shown promising results in diagnosing Ataxia in lab settings, equipping people with sensors or installing high-end 3D cameras is not scalable, which is why algorithms that work on the existing and diverse infrastructure are needed. In this paper, we present an end-to-end computer vision pipeline that can detect participants displaying Ataxia-impacted gait characteristics and can measure the severity of Ataxia from the SARA gait task. Our models open up new possibilities for the future development of a user-friendly, automated platform that can remotely assess Ataxia in non-clinical settings. This would have various applications, including independent use by patients and working as an objective clinical outcome measure for pharmaceutical companies, academic institutions, and other key stakeholders. Our technology could aid clinicians in having a second opinion in their existing diagnosis pipeline. Besides our general methodology of person detection, tracking, and separation, it can be further utilized to identify gait anomalies in elderly individuals who are vulnerable to falls [29], which is one of the main reasons for fractures, physical incapacity, and even death [54] in the elderly population. Our dataset will help the computer science community to develop better algorithms to quantify and potentially diagnose movement disorders. Our method is scalable and interpretable, which is helpful for future deployment and possible adoption in healthcare settings.

CCS Concepts:

• Applied computing→Health care information systems; • Computing methodologies→Object detection; Tracking; Interest point and salient region detections.

A SELECTED FEATURES

A.1. Features Used For Risk-prediction Task

[mean_feet_dist, std_left_x, std_imbalance, std_nose_y, max_feet_dist, max_right_x, min_left_x, min_imbalance, range_left_x, range_imbalance, range_nose_x, height _reduction]

A.2. Features Used For Severity-assessment Task:

[ ’mean_feet_dist’, ’height_reduction’, ’max_feet_dist’, ’max_right_x’, ’std_imbalance’, ’min_left_x’, ’range_imbalance’, ’range_nose_x’, ’mean_nose_y’, ’min_nose_x’, ’entropy_nose_y’, ’std_tilt’, ’std_left_x’, ’range_feet_dist’, ’range_left_x’]