Detection of Stereotypical Motor Movements in Autism using a Smartwatch-based System

Abstract

Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder that is often accompanied by stereotypical motor movements. Health professionals typically assess the severity of these behaviors during therapy, which limits observations to a structured clinical setting. Recent advancements in ubiquitous computing and wearable sensors enable an ability to monitor these motor movements objectively and in real-time while children with ASD are in different environments. In this paper, we present a smartwatch-based system designed to detect stereotypical motor movements. To validate the feasibility ofour approach, we collected data from adults imitating example behaviors captured in YouTube videos of children with ASD, and we then evaluated several classification methods for accuracy. The best model can identify stereotypical motor activities of hand flapping, head banging, and repetitive dropping with 92.6% accuracy (precision 88.8% and recall 87.7%) in the presence of confounding play-type activities. We present the trade-offs between accuracy ofthe assessments and power consumption due to sensing from multiple modalities. Cross-participant validation shows that the results ofusing the model on an unknown subject are promising.

Introduction

Advancement in sensing technology holds tremendous promise to improve public health and wellbeing. This technology can create a dramatic shift in the healthcare delivery processes by building a bridge between the provider, making assessments in his or her office, and the patient, living a complex life outside the office. Sensing of health and well being has become more acceptable as smaller wearable form factors better integrate seamlessly with daily life. Wireless connectivity combined with sensing is enabling remote monitoring and telemedicine¹. In our research, we are evaluating how advancements in sensing technology and computational modeling can help caregivers, who have children with autism spectrum disorder (ASD), and therapists to monitor and manage behaviors through just-in-time interventions (JITI).

ASD is a complex developmental disorder that manifests in combinations of atypical communication, social interaction, restricted, repetitive patterns of behavior, interests, and activities². About one in every 68 children is diagnosed with autism according to the Centers for Disease Control and Prevention (CDC)³. Although parents start having concerns about their child at a very young age, the definitive diagnosis of ASD takes significant time⁴, causing stress to the parents and delays in early treatment. Children with ASD can face challenges in their day to day activity because of stereotypical behaviors, such as hand flapping, head banging, repetitive movement, and verbal protest⁵. Accurate detection of these behaviors through sensing technology can allow caregivers, clinicians and researchers to characterize their nature objectively, identify triggers for their occurrence and monitor therapeutic interventions when the behaviors become challenging.

There are several important considerations and challenges for designing a wearable system. Placing too many wearables on the different parts of a subject’s body may cause unnecessary burden on the user. The design thus needs to make a trade-off between the quantity and the quality of the sensing. To increase acceptability, the system should be user-friendly, which in our case means lightweight and acceptable to a child with ASD. The system should take into account the battery life to limit repetitive charging. To facilitate rapid adoption, we have decided upon selecting a commercially available smartwatch that has gone through primary quality control and that the user can purchase at a competitive price. For the JITI to be effective in caregiving, the system should not provide too many false-positives that trigger unnecessary interventions.

For the presented work, we have conducted a feasibility study on adults (who are members of the research team) imitating the stereotypical behaviors commonly observed among children with ASD (hand flapping, head banging, and repetitive dropping) in the presence of other confounding child-type playing activities, such as card matching and drawing. In this paper, we present the results of our ability to detect stereotypical motor activities with different classification approaches. We then evaluate the generalizability of the model by conducting cross-subject validation. We also discuss the trade-offs between the accuracy of the model and the energy consumption.

Related Work

Advanced sensing technology has been widely used to assist in diagnosing children with ASD and improving their quality of life. Actigraphy had been used to monitor commonly reported sleep disorders among children with ASD^6,7. ASD screening tools had been designed by tracking abnormal eye gaze pattern^8,9 during social interaction. Technology had been developed to improve such social communications and emotional well-being by developing touch-based therapies¹⁰’¹¹. Humanoid robots are also being used as a companion to engage the ASD child in learning activities and therapeutic intervention^12-13.

The most relevant research to our work is reported by Westeyn et al.¹⁴ and Albinali et al.¹⁵. Westeyn et al.,¹⁴ used a pattern recognition method to detect stereotypical motor movements commonly observed in children with ASD. A recruited adult imitated seven common autistic self-stimulatory behaviors (drumming, hand flapping, hand striking, pacing, rocking, spinning, and toe walking). Data were recorded from three custom-made accelerometer sensors placed at wrist, waist, and ankle. A hidden Markov Model (HMM) based pattern recognition method was able to detect 69% of those seven activities. The group extended this work by segmenting episodes into higher-level categories of severe behaviors, such as self-injury, disruption, and aggression¹⁶ by keeping the sensing modalities and the placement similar. Albinali et al.,¹⁵ conducted a study on six children with ASD in both laboratory and classroom settings. Three custom-made wireless accelerometers were placed on both wrists and the torso. The proposed models were customized for each child and were able to achieve 89% accuracy on average.

Our research attempts to improve upon the previously reported efforts¹⁴’¹⁵ in several ways. In contrast to custom-made accelerometer sensors, we used a commercially available smartwatch. To reduce user-burden, we placed (smartwatch) sensor on the dominant hand and did not put any additional sensor on any other parts of the body. In addition to the accelerometer, we investigated the feasibility of gyroscope sensors for characterizing challenging behaviors. Finally, to develop a generalizable model, we conducted cross-subject validation of the proposed model.

Experiment Design

Participants: To generate data for validation of the smartwatch based system, six IBM employees wore the smartwatch and mimicked the stereotypical behaviors of autistic children publicly found on example Youtube videos. The participants all were right hand dominant. No personally identifying information was collected on the participants. Involvement was limited to generating examples of motor movement data through the smartwatch.

Wearable Sensor: Our sensing system is based on a direct recording of activity patterns using a commercially available smartwatch (Motorola Moto 360), which is instrumented with a 3-axis accelerometer and a 3-axis gyroscope. Participants wore this smartwatch on their dominant hand. It is connected to an Internet-enabled, Android tablet using Bluetooth for the transmission of measurements. The sampling rates for the sensors were 32 Hz for each accelerometer axis and 16 Hz for each gyroscope axis.

Android Tablet: An Android tablet application was developed and used to conduct the study and document the precise timing of the start and end of an ongoing activity. The app is connected to the smartwatch to receive readings from the smartwatch. A first-in-first-out buffer is used to hold data for approximately 10 seconds. When buffer overflows (roughly every 10 seconds) the app sends data in bulk to a remote server over the internet. This bulk operation reduces the network overload by sending only limited number of packets over the internet. This process lowers battery consumption. Annotation timings are sent in real-time as they are limited in quantity.

Data Collection: The study observer watched the subjects and used the android app to annotate the stereotypical behaviors the subjects were performing. When the observer starts the study, the Android app sends a signal to the watch to begin recording sensor signals. The smartwatch then starts sending those signals to the tablet over the Bluetooth connection. The received signals from the smartwatch and the annotations made by the study observer are sent to the IBM Bluemix IoT platform over the internet which later on relayed to a secure and reliable database server to persist for later analysis.

Experiment Set-up: Parents and caregivers of children with ASD are increasingly using online video sharing platforms, such as YouTube, to share complicated behaviors and stereotypical activities of their children. We followed an approach proposed by Sarker et al.¹⁷ to discover and identify the frequently observed complicated behaviors and other therapy related activities. The complicated behaviors depicted were hand flapping, head banging, and repetitive dropping. The other behaviors were coloring, lego building, and card matching on top of each other, which are the common child playing activities observed in this population.

In order to capture similar data from participants wearing the smartwatch, a mock video was created. The video was a 22.8-minute long compilation of 34 thirty-second clips of videos found on YouTube. Figure 1 shows the order of the activity tasks. Participants were given time to watch the example data video and practice any behavior before beginning the study. They were instructed to imitate the hand movements and body positions on the video as closely as possible using child-like gestures when applicable. During the study, the videos alternate between a complicated autistic behavior and a video of behavior marked as other. Between each clip, there was a ten-second long pause. During that time, the participant was given time to review instructions and prepare for the next behavior. All behaviors are repeated twice in the original order. The mock data video was played on a laptop. The sessions were also video recorded for backup. Paper, a marker, legos, and a box were used as props throughout the video for coloring, lego building, and repetitive dropping.

Figure 1:

Activities performed by the subject. Each activity is 30 second long. There is a 10-second pause in between activity. Each subject repeats these activities n times observing different examples.

Data Collected: In total, 2.28 hours data was recorded from six participants with a usable data yield of 98.5%.

Experiments and Results

Each gyroscope sensor reading is a vector of three components (from three axes). Magnitude is computed as $\sqrt{x^2+y^2+z^2}$. Similarly, we get four components from the three-axis accelerometer. In addition, we derive roll and pitch from ac-celerometer samples¹⁸ to capture the orientation of the hand with respect to gravity. It is also possible to derive roll and pitch from gyroscope readings. However, we need to integrate those readings to compute them. Given that signals are recorded at high frequency (16 Hz), a minuscule error in the reading will magnify over time. In contrary, accelerom-eter readings capture both linear acceleration and the acceleration due to gravity. We used the gravity component to compute roll and pitch, which provides us the orientation of the hand with respect to gravity.

These stream of data are segmented into 5-second windows with a 0.5-second stride. Seven statistics are computed for each of the 10 data stream (70 in total). These statistics include: (1) mean of the signal to capture the orientation of the hand; (2) variance to capture the displacement and oscillation of the hand; (3) skewness to measure the asymmetry of the distribution during a specific activity; (4) kurtosis to quantify the shape of the distribution; (5) root mean square to compute the magnitude of the movement in particular axis; (6) energy calculated from the FFT of the signal; (7) entropy to capture the type of activity.

Figure 2 lists the top 25 features (out of 70) and their relative importance according to mutual information¹⁹. Out of these 25 features, 17 are derived from the accelerometer. Hence, in comparison to the gyroscope, the accelerometer plays a more important role in the recognition of targeted stereotypical motor movements. The system for detection of these movements is intended to run on a resource-constrained battery operated smartwatch. Real-time application of such model is desired to provide JITI. Computing a large number of statistics in real-time will take computation resource. Using too many features will overfit the detection model and limit the generalizability. Hence, we limit the feature count to 25 features listed in Figure 2.

Figure 2:

Top 25 features ordered according to mutual information.

Figure 3 show the distribution of top 12 features. Figure 3b holds the distribution of the mean of the pitch. The metric of the pitch is a degree which is derived from the accelerometer. It measures the orientation of the dominant hand with respect to the gravity. We observe that pitch clearly distinguishes between hand flappingand head banging. During hand flappingpitch mostly ranges between 10° and -80° and during head bangingranges mostly between +40° and +80°. Similarly, root mean square (RMS) of pitch can mostly distinguish othersclass from the remaining (see Figure 3e). Gyroscope mean magnitude (Figure 3c) in collaboration with accelerometer mean X-axis, mean pitch, RMS X-axis, and RMS pitch can mostly capture the repetitive dropping (Figure 3 a, b, d, and e, respectively).

Figure 3:

Distribution of top 12 features across four classes.

Prior works^14,15 on the detection of stereotypical motor movements focused on developing a personalized model. This is based on a well-established assumption that every child is different; therefore, the model should be tailored to each individual. However, in practice, it is impractical to run a study on each subject prior to the therapy phase. Therefore, we consider developing a person-independent generalizable model. During training, we use data from n – 1 subjects and validate on the remaining subject. While the model will not perform as well as the personalized model, we do not have to conduct a study for each individual. The developed model can act as a default one. Based on reinforcement learning, over time, the model can be tailored to each subject based on the feedback from the caregiver.

We develop a person-independent model to recognize four target behaviors. Logistic Regression, SVM, RandomFor-est, and Gaussian Naive Bayes-based models obtained accuracy of 87.3%, 88.2%, 88.6%, and 90.7%, respectively. Gradient boosting based model outperformed the prior approaches and obtained an accuracy of 92.6%, precision 88.8%, recall 87.7%, andF-1 score 0.881.

Our current settings for the system uses both accelerometer and gyroscope sensors. This system is designed so that the subject can wear it in their free-living condition. There is two primary design consideration for such a system. First, frequent charging of the smartwatch will cause additional burden on the caregiver. Using only one sensor will significantly improve the battery life and reduce the need for frequent charging. Second, frequent false alarms will trigger unnecessary intervention, inducing interruption fatigue to the subject^20,21. The system will also provide erroneous statistics of challenging behaviors rendering the system ineffective to the caregiver. With these two design consideration, we avoided the use of the gyroscope readings and only use accelerometer readings to develop the model. The accuracy reduces from 92.6% to 90.8%. Precision, recall, and Fl-score is 86.3%, 85.2%, and 0.856, respectively.

Figure 4 reports the confusion matrix of these two approaches. Using both accelerometer and gyroscope the model is able to detect hand flapping, head banging, repetitive dropping, and others, 0.952,0.922,0.793, and 0.884 of the cases, respectively. Disregarding gyroscope derived features, the accelerometer-only model is able to detect the same hand flapping, head banging, repetitive dropping, and others, 0.936, 0.893, 0.752, and 0.873 of the cases, respectively. The majority of false positives are classified as others. In our proposed use scenario, such false positives will not trigger a response from the caregiver. Another observation from Figure 4a is that the model sometimes confounds between hand flapping and repetitive dropping. It is understandable considering the case that hand flapping and repetitive dropping are subsets of repetitive behaviors and thus have a close resemblance. An accelerometer only model only finds a minor drop in repetitive dropping (from 0.793 to 0.752). Repetitive dropping includes angular hand motions, so the gyroscope helps here by capturing the angular velocity. We also observe a drop in the detection of head banging class (from 0.922 to 0.893). Hence, accelerometer only model will trigger more false positives. These are trade-offs we need to consider before deploying as a real-time application. In a battery-constrained setting, we can use accelerometer only model which triggers more false positives. When triggering false-positive JITIs leads to too many interruptions by our system, we can use both inertia sensors to limit false positives.

Figure 4:

Confusion matrix.

To further assess the results of our model, we applied the developed model to a random participant’s study data. Figure 5 show the result. The top part of the figure is the ground truth labels marked by the study coordinator. The bottom part of the figure shows model inferences computed every 500 milliseconds. Inferences are four probabilities, each signifies the model generated probabilities of belonging to a specific activity type — hand flapping, head banging, repetitive dropping, and others. Higher probability means that the model is confident about the class. The summation of four probabilities is always one. We observe that the model has low accuracy during the first or the last two seconds of a specific activity. In addition, there is confusion observed between repetitive dropping and hand flapping, as they are similar activity.

Figure 5:

Comparison of ground truth activities against the model labels. Every 500 milliseconds, model outputs four probabilities for each of the four classes – hand flapping, head banging, repetitive dropping, and others.

Discussion and Future Works

In this paper, we present the feasibility of using a smartwatch to detect stereotypical motor behaviors commonly observed in children with ASD. With this goal in mind, we had three primary design considerations. First, we considered improving the accuracy and decreasing the false positive rates for assessing those stereotypical motor movements. False positive rates are especially important for such a system that interacts with a human to provide JITI upon detection of a set of relevant behaviors. We developed a model to limit the cognitive burden imposed on the user by repeated false alarms. Second, the system should be capable of running in real time on a ubiquitous computing device in a resource-constrained setting. Computation of too many features in a real-time system will consume many CPU cycles. We limit the number of features to reduce the computational burden on the smartwatch to speed up the prediction and reduce energy consumption. We have shown the performance trade-off for adopting one versus two inertia sensors. Having accelerometer-only system can minimize battery usage significantly with the compromise of an increased false-positive trigger. Third, as an extension of prior work^14-15, we conducted a cross-participant validation of the developed model. Unlike earlier works, this is a generalizable model, readily available to work for any subjects. Over time, the model may be able to be personalized to the user. Real-time feedback from the caregiver can be used to tailor the model for each subject using reinforcement learning.

We believe that the proposed work opens the opportunity for many future directions. In ongoing monitoring, our system can recognize the desired activities in more than 99% of the cases. The false positives mostly occur during the transition of these activities—the first or last two seconds. It is possible that the model may miss the occurrence of short-lived challenging behaviors. Future work on improving the performance during this transitioning state will reduce the false-positive rate even further. The construction of a low-energy, high-recall model can perform a first pass recognition task. Upon detection of a stereotypical motor movement, the system can run the duty-cycled sensors in a high-frequency mode to obtain a more accurate assessment of the events of interest. Finally, challenging behaviors often cause stress among parents and caregivers. Our system can be used to monitor and identify the antecedent biomarkers and the surrounding context, which may be able to establish a causal relationship with behavior types. A real-time method can be developed in the future to anticipate the occurrence of relevant motor behavior and provide JITI, e.g., by notifying the caregivers to take appropriate steps. In addition, this system has enormous potential to quantify motor behaviors objectively and assess the eficacy of treatment methods.

Conclusion

We present results to establish the feasibility of recognizing stereotypical motor movements in ASD using a commercially available smartwatch. We have discussed the design considerations for such a system and their trade-offs during real-time deployment. We have also developed a classiication model using this system that can identify those activities at an accuracy of 92.6%. Our system is intended to allow therapists and caregivers to monitor a child with ASD unobtrusively and to provide an objective report on the frequencies and patterns of behaviors. A major limitation of our validation study was that it used simulated data from adults imitating children with ASD. In our next phase of work, we are conducting a ield study on children with ASD, which we will use to evaluate the system and improve upon based on new evidence.