Computer vision and behavioral phenotyping: an autism case study

Abstract

Despite significant recent advances in molecular genetics and neuroscience, behavioral ratings based on clinical observations are still the gold standard for screening, diagnosing, and assessing outcomes in neurodevelopmental disorders, including autism spectrum disorder. Such behavioral ratings are subjective, require significant clinician expertise and training, typically do not capture data from the children in their natural environments such as homes or schools, and are not scalable for large population screening, low-income communities, or longitudinal monitoring, all of which are critical for outcome evaluation in multisite studies and for understanding and evaluating symptoms in the general population. The development of computational approaches to standardized objective behavioral assessment is, thus, a significant unmet need in autism spectrum disorder in particular and developmental and neurodegenerative disorders in general. Here, we discuss how computer vision, and machine learning, can develop scalable low-cost mobile health methods for automatically and consistently assessing existing biomarkers, from eye tracking to movement patterns and affect, while also providing tools and big data for novel discovery.

The need for computer vision for computational behavioral coding

Despite significant recent advances in molecular genetics and neuroscience, behavioral ratings based on clinical observations are still the gold standard for screening, diagnosing, and assessing outcomes in neurodevelopmental and neurodegenerative disorders, including autism spectrum disorder (ASD). Autism screening tools rely on parent reports [50], which can lead to biased assessments depending on level of parental knowledge about child development and cultural factors [30] (see also [10] for additional discussion on the virtues and challenges of the current standard of care). Diagnostic assessments are on the basis of clinical behavioral observations and ratings. Such behavioral ratings are subjective, require significant clinician expertise and training, typically do not capture data from the children in their natural environments such as homes or schools, and are not scalable for large population screening, low-income communities [13], or longitudinal monitoring, all of which are critical for outcome evaluation in multisite studies and for understanding and evaluating symptoms in the general population.

The development of computational approaches to standardized objective behavioral assessment is, thus, a significant unmet need in ASD in particular and developmental and neurodegenerative disorders in general. Here, we discuss and present some recent results on how computer vision, combined with machine learning and data analysis, can lead to the development of scalable mobile health methods for assessing existing behavioral biomarkers, while also providing tools and big data for novel discovery. The computer vision tools range from gaze and attention monitoring to movement patterns and affect coding.

Developing such scalable digital behavioral measurement tools will be important for addressing the major public health challenge of identifying and treating children with neurodevelopmental disorders. Neurodevelopmental disorders affect ~15% of the population in the United States (US). In the case of ASD, one out of 59 children is affected, with an estimated annual cost to society of $265 billion. Early intervention is critical, resulting in improved cognitive, language, and social functioning, which affects long-term outcomes and results in an estimated per-person yearly cost savings of $19,000 ($1.2 million lifetime) [8]. Early intervention starts with behavioral screening, a major challenge when considering that many US clinics have wait lists of 4–12 months to see experts, which are even less available in low-income countries.

To truly scale in behavioral coding, methods that allow for automated data collection in natural environments, such as the home, are needed. We have recently demonstrated that it is feasible to collect high quality video data in response to stimuli delivered in an iPhone app downloaded from the web and independently administered by parents at home [15]. In this particular study, we collected data from 1756 participants (4441 videos) who behavior was recorded with the camera in the phone. We showed that 87.6% frames provided high quality data and automated coding via computer vision determined that children with typical development versus autism risk could be differentiated based on patterns of facial affective expression and attention [4,15,26], more on this is discussed in the following sections. A scalable integrated approach to active stimulus design, consumer-grade sensing exploitation, and automatic analysis via computer vision and machine learning is an example of the key driving force to improve the assessment of ASD and related disorders. In addition to scalability and access, computational behavioral phenotyping via computer vision and machine learning will result in big data at unprecedented multiscale resolution, opening the door to new understandings in developmental disorders.

Although, in this article, we concentrate on computer vision tools for ASD screening, it is important to stress that this is just one example of the use of computer vision for computational behavioral phenotyping. Similar tools can be used for diagnosis and symptoms and therapy monitoring, as well as other developmental disorders such as attention deficit hyperactive disorder (ADHD); see Refs. [27,29] for further related discussions on the subject of digital phenotyping, of which behavior is a particularly important component and one is ready to be addressed by computer vision. Moreover, the tools go beyond developmental disorders, for example, Parkinson’s disease [1] and pain [45], and, thereby, the description below should be interpreted as one example of how computer vision can complement and augment current standards of care (see also for example https://md2k.org/for a major initiative on mobile sensing, including behavior, and numerous publications and tools in multiple areas and diseases). The works described next clearly illustrate that we are ready for this important step of utilizing and extending computer vision as an integral step in behavioral analysis for mental and developmental health care.

Examples of computer vision in autism spectrum disorder

Now, we present a number of examples on the use of computer vision tools for computing behavioral biomarkers relevant to ASD and other neurodevelopmental and neurodegenerative disorders. As mentioned before, these are illustrative only, showing that the tools are not only ready to replicate known biomarkers but also to discover new ones. Moreover, works as the one reported in Ref. [15] demonstrate that these computer vision tools can be applied to data collected in natural environments.

Gaze and attention

Gaze and attention are well-established behavioral measurements in developmental disorders; this is reflected in the very extensive literature on the subject, e.g., Refs. [7,31,35–37,42,43]. Their connection with genetic factors has also been recently reported [9]. Given that differences in patterns of attention are considered a key behavioral indicator of certain developmental disorders, an important goal is to take advantage of new technologies that will allow us to collect data on attention patterns outside laboratory settings and provide scalable and low-cost use on mobile devices [4,15].

Some of our initial studies [2] provide a proof of concept of how data on gaze and attention can be collected in a manner that is scalable (see also [6,32] for more generic computer vision algorithms for low-cost gaze analysis on consumer devices). A dynamic movie that displayed salient social and nonsocial stimuli (Figure 1, top) was used to investigate attention patterns in toddlers with (N = 22) and without (N = 86) ASD. The capabilities of the sensing and analysis tools, namely an off-the-shelf video camera and computer vision, were taken into consideration for designing the stimuli. In particular, the stimuli are designed to require only a region-based accuracy, right or left attention discrimination in this particular case, to evaluate dynamic attentional preference for social or nonsocial stimuli. The movie showed a social stimulus on the left (singing women) and a nonsocial on the right (toys). Both halves changed during the 60 s of presentation, defining a total of nine temporal blocks of distinct social (left half) and nonsocial (right half) combinations. Computer vision first automatically detects key facial landmarks of the participants (Figure 1, bottom left), and then we use these landmarks to compute the extreme yaw angle values to determine the midrange yaw angle value. We define two thresholds by adding/subtracting 10% of the difference between the midrange value and the extreme values to the midrange value. With this, we determine whether the participant is looking at the left (social) or right (nonsocial) parts of the dynamic stimuli, or if the yaw angle value was not large enough to conclude. In this last case, we further use the landmarks to make a decision. In particular, we use the landmarks at the edges of the eye to estimate the position of the middle of the eye and the distance between this middle and both edges. Then, we check whether or not the pupil is close enough to one of the edges to conclude the attention direction. If not, we assume that the participant is looking somewhere in the middle of the stimuli. We use this method with both eyes. Details of this fully stimuli-algorithm integration for scalable attention tracking are provided in our publications.

Figure 1.

Example of a stimulus with a social and nonsocial component simultaneously presented (see also graphical abstract for another typical stimulus example). Bottom Left: Example of automatic detection of landmarks, head position, and affect while responding to a visual stimulus (see example on the graphical abstract). The participant is the child on the right. Bottom Right: Example showing we can automatically track body features (colored sticks), critical for ASD and developmental disorders. (Faces of participant and caregiver occluded here in the figure for privacy protection. Analysis is performed on the original video without occlusions.) ASD, autism spectrum disorder.

To validate this preliminary approach, we consider three predictions derived from previously published studies that used state-of-the-art high-end (e.g., Tobii) eye tracking technology [2]. First, we showed that the ASD participants were more likely to have reduced attention to the movie overall. We next examined differences between social and nonsocial attention. We found that it was very rare for a child without ASD to focus the majority of their attention on the nonsocial stimuli, whereas this occurred much more often among the children with ASD. Thus, this biomarker has potential sensitivity as a risk marker for ASD. Finally, we took into account the temporal changes in the stimulus to investigate patterns of fixation and shifting of attention. We showed that participants with ASD are more likely to fixate on only one type of stimulus in the movies (e.g., social/nonsocial regions) than the non-ASD children, an additional potential dynamic biomarker.

This example illustrates how when combining computer vision with stimulus design, we can use consumer devices to infer behavioral information that was previously available only with the use of high-end laboratory-style tools.

Motor analysis

Early manifesting impairments in motor abilities have been documented throughout the lifespan in individuals with ASD, from infancy through adults. Examples of early motor differences include head lag when pulled to sit, delays in walking, postural stiffness, slumped poster, and difficulty maintaining midline position of the head, e.g., Refs. [3,16–19,46]. Quantitative, objective methods for assessing atypical motor/movement in ASD are needed. Experts in computer vision will immediately notice that the abovementioned behavioral biomarkers are great candidates for automatic coding, while at the same time challenging state-of-the-art computer vision to work on young children and on very unconstrained scenarios. We illustrate next a few very promising examples demonstrating the value and readiness of computer vision to address these challenges.

Body motion

Toddlers with ASD often presented asymmetric arm positions in early life [16]. Using computer vision, we are able to estimate the 2D body pose of the toddlers and estimate arm angles [24]. This is one of the motor measurements that can be automatically performed with computer vision but is not the only one. One of the main diagnostic criteria for ASD in the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) is restricted, repetitive-patterns of behavior, interests, and/or activities. A primary way in which these behaviors manifest in ASD is stereotypical motor movements (SMMs). Traditional measures of SMMs include rating scales, direct behavioral observation, and video-based methods, all of which can be as mentioned before subjective, inaccurate, time-intensive, and difficult to compare across different individuals with ASD. More reliably, accurately, and efficiently detecting and monitoring SMMs over time could provide important insights for understanding and intervening upon a core ASD symptom [20,21]. Using computer vision to track body parts, e.g. Refs. [5,23], see Figure 1, and modern data analysis tools can be exploited to study this as well, e.g., Ref. [47].

Head movement

While body motion is recognized as an important behavioral measurement for ASD, more subtle motions from the head are critical as well. In Ref. [4], we examined atypical orienting and attention behaviors in toddlers with ASD (see also [34]). One hundred four toddlers, 16e31 months old (mean = 22) participated in this study. Twenty-two of the toddlers had ASD, and 82 had typical development or developmental delay. Toddlers watched video stimuli on a tablet while the built-in camera recorded their head movement. By automatically following facial landmarks (see Figure 1), computer vision analysis measured participants’ attention and orienting in response to calling the child’s name. Differences in orienting behavior were analyzed between the ASD group and the comparison group. Reliability between computer vision analysis and human coding for orienting to name was excellent (intraclass coefficient 0.84, 95% confidence interval 0.67e0.91). Only 8% of toddlers with ASD oriented to name calling on >1 trial, compared with 63% of toddlers in the comparison group (p = 0.002). Mean latency to orient was significantly longer for toddlers with ASD (2.02 vs 1.06 s, p = 0.04). These results already provide very strong automatically computed behavioral biomarkers (see Ref. [4] for additional details).

Attention and head turning is not the only important characteristic we can measure, once we exploit computer vision tools to track the head motion at high frame rate and spatial accuracy. For example, the development of postural control is an index of neuromuscular reactions to the motion of body mass to retain stability [28]. Research on children with ASD has shown that ASD is associated with deficits postural control, which is manifest in postural sway. Such differences become more evident when the child views complex multisensory and social stimuli, e.g., Refs. [18,22]. We have preliminary results on using computational tools to automatically measure postural sway when watching multisensory stimuli, using computer vision and landmark tracking as input to new metrics for postural sway and other subtle motions [10]. The high frame rate, compared with human naked eye observations, is critical, as it was in our work on the latency of orienting behavior mentioned earlier. This opens the door to discovering new high-resolution biomarkers. The preliminary results show not only great statistical power to distinguish ASD from non-ASD but once again help to discover previously unknown characteristics of ASD.

Affect analysis

Differences in affect and emotion is an important biomarker for numerous developmental and neurodegenerative disorders and have been studied extensively in ASD as well. Following works on affective computing and literature on automatic encoding of facial affect, e.g. Refs. [12,14,41,44], we have recently deployed and reported preliminary studies on automatic encoding of facial affect for ASD. As shown in Figure 1, with automatic detection of critical facial landmarks, we can compute emotions in addition to tracking head motion and overall body posture (see previous subsections). These computational computer vision tools are developed taking into account the unique application, from the use of only consumer-grade sensing technology such as mobile phone cameras to the population (children) and deployment environments (e.g., homes); all of them brining unique challenges that need to be addressed to develop truly scalable computer vision—based tools and interventions. We have validated the automatic tools in toddlers with ASD and typical development showing excellent reliability [25,26]; for example, overall agreement between the human raters for coding of engagement, facial expression, and social referencing achieved an intraclass correlation coefficient (ICC) score of 0.84 with 95% confidence intervals of 0.76–0.95. Reliability between the automatic methods and the expert human rater when coding total time of ‘happy’ was excellent, achieving an ICC of 0.90. The reliability for the subgroups of participants with and without ASD was also excellent, achieving ICC scores of 0.90 and 0.89, respectively. Performance of the automatic methods was also validated on a per-frame basis; 136,450 frames (~75 min) were coded for emotion across all the participants. Overall, the automatic method achieved high precision, recall, and F1 scores: 0.89, 0.90, and 0.89, respectively. The agreement between computer vision and expert raters is as strong as among raters themselves, and most of the differences come at the ends of the expression because it is hard to exactly define the frame for the beginning or end of an emotion. See Ref. [26] for details and also [15] for results on correlations between automatically computed affect and ASD risk.

Future challenges

Behavioral coding is still the gold standard for multiple developmental disorders. Recent studies, as described in this note (see also [11]), have already demonstrated both the feasibility and the value of developing computer vision—based scalable computational tools for behavioral phenotyping. The tools here described were deployed both in the clinic [4,24,26] and in the wild [15]. The machine-learning tools that complement the computer vision components are explainable, for example based on decision trees and random forests, meaning the discoveries are not black boxes but provide fundamental insights into the discovery of biomarkers and their potential neurological basis. While validating the reliability of these novel scalable computational tools, we confirmed known behaviors, this time collected with low-cost scalable paradigm, contrary to high-end and highly sophisticated and expensive instrumentation, while at the same time, discovering new ones.

A number of challenges and opportunities remain, including:

• Extension of the computational tools to consider not only the presence of a behavior but its actual strength, thereby considering behaviors as continuous biomarkers (see Ref. [26]);

• Addressing multiple ages, including infants (e.g., Ref. [36]);

• Extension of the tools to take into account cultural differences, see for example [40];

• Extension of the computational tools to measure interactions and not just individual behaviors (see also [38,49] and data at http://www.cbi.gatech.edu/mmdb/);

• Extension of the computational tools to address fully unconstrained scenarios;

• Large epidemiological validation, as currently addressed by a number of researchers as part of the National Institutes of Health Autism Center of Excellence program at Duke University;

• Integration of computer vision with other senses, in particular, audio (see also [40]);

• Integration of automatic behavioral coding with other more standard measurements, including the Modified Checklist for Autism in Toddlers, Revised [39], electronic health records, and diagnostic methods, such as the Autism Diagnostic Observation Schedule [33].

• Exploitation of tools from computer vision beyond ASD, including but not limited to ADHD, eating disorders, PTSD, movement disorders, pain, and Alzheimer’s disease.

Broad and multimodal data will be particularly important for distinguishing ASD from closely related conditions, for example, ADHD; see, for example, [48] for recent work in such integration of multiple measurements and its application to comorbidity understanding.

The challenges do not end at the engineering level. For example, human factors and ethical considerations are important.

To address these challenges, the need for interdisciplinary teams is very clear. The development of the scalable computational tools here reported need domain experts, engineers, statisticians, app developers, ethical and regulatory experts, and clinicians, to name just a few of the must have components of the team.

The need for a revolution in health care in general and developmental disorders in particular is clear. Computational behavioral analysis is a necessary component of this revolution, and the recent results have demonstrated that tools from computer vision are ready to contribute to this challenge.