A Computational Study of Expressive Facial Dynamics in Children with Autism

Abstract

Several studies have established that facial expressions of children with autism are often perceived as atypical, awkward or less engaging by typical adult observers. Despite this clear deficit in the quality of facial expression production, very little is understood about its underlying mechanisms and characteristics. This paper takes a computational approach to studying details of facial expressions of children with high functioning autism (HFA). The objective is to uncover those characteristics of facial expressions, notably distinct from those in typically developing children, and which are otherwise difficult to detect by visual inspection. We use motion capture data obtained from subjects with HFA and typically developing subjects while they produced various facial expressions. This data is analyzed to investigate how the overall and local facial dynamics of children with HFA differ from their typically developing peers. Our major observations include reduced complexity in the dynamic facial behavior of the HFA group arising primarily from the eye region.

1 Introduction

Children with Autism spectrum disorder (ASD) have significantly impaired social communication abilities, even those who have preserved language and cognitive skills and are commonly referred to as having high functioning autism (HFA). These social impairments are characterized by difficulties in using non-verbal cues [1], including difficulties in perceiving and producing facial expressions [2], [3]. It has been noted that individuals with HFA have difficulties processing dynamic facial information related to emotional state (a key component in social interaction) as compared to static stimuli [4]. They also display ambiguous expressions and more neutral/flat affect expressions compared to their TD counterparts [5]. Several studies have established that facial expressions of children with HFA are often perceived as atypical, awkward or less engaging by typical adult observers [5], [6], [7]. In fact, their expressions are perceived as awkward after just one second of interaction [7]. Even naïve observers, without any knowledge of the subjects' diagnosis, perceive children with HFA to be more awkward compared to their typically developing (TD) peers [7]. Despite this clear deficit in the quality of facial expression production, very little is understood about the underlying mechanisms and characteristics of facial expressions.

Atypicality in facial expressions of children with Autism has been studied through observations by humans [6], [7]. However, subtle movements causing differences in the facial dynamics between the HFA and the TD groups may not be easy to capture by visual inspection alone, which underscores the need for objective data analysis methods. The usefulness of computational behavior analysis has been noted in several recent studies, such as in analyzing atypicality in prosody [8], [9], [10], and in asynchronization of speech and gestures of children with autism [11]. Following this line of computational research to better understand behavioral phenotyping in autism, the work in this paper aims at quantifying atypicality of facial expressions in children with HFA, which is otherwise difficult to achieve by plain visual inspection. Our approach relies on direct measurement of facial movements during specific expressions, followed by analyses of those movement patterns.

To objectively understand facial expression-related atypicality in autism, we use direct motion capture (mocap) technique to record subtle facial movements in both children with HFA and TD subjects as they produced specific expressions. Mocap is a powerful technique to obtain detailed, precise description of gesture dynamics. It is widely used for supporting multimodal modeling in a variety of application domains including animation, human-machine interaction and sports [12], [13]. In this work on expressive facial movements, we use a mimicry paradigm to control task performance and variability, where subjects mimic a fixed set of facial expressions performed by actors in stimuli videos [14]. Movements of markers affixed to the face are recorded while the subjects mimicked facial expressions, and subsequently used for computationally examining the movement dynamics of the resulting facial expressions. The usefulness of computational approaches for analyzing such data was established in our previous work [15], [16], [17]. Our preliminary analysis indicated that facial dynamics of children with HFA differ from those of TD children in various objective ways, such as in exhibiting rougher head motion, higher variability within the HFA group, and lower bilateral symmetry [15], [16].

Our present study of facial expression production mechanisms in autism involves understanding the overall dynamics of the entire face, localized dynamics in specific facial regions, as well as the dynamical relationships between the movements across different facial regions.

The notion of complexity is fundamental to any dynamical system - mechanical or physiological, and can be understood as the rate at which new information is generated by the system. There is significant evidence of various physiological (dynamical) systems being associated with atypical and often reduced measures of complexity [18], [19], although the interpretation of complexity varies with the physiological parameters being studied and the developmental condition being investigated. Motivated by these past observations, we study the facial expression production system (a physiological dynamical system) in terms of its dynamic complexity. First, we consider the hypothesis whether expressive facial movement patterns in HFA represent a system of reduced complexity compared to the TD peers. Reduced complexity in the context of facial dynamics may be interpreted as partial loss of subtle movements, repetitive patterns in the dynamics, and highly correlated movements between facial regions. We investigate whether the TD group and the HFA group have similar or differing patterns of complexity in their overall facial dynamics using the multiple scale entropy (MSE) method, which provides a measure of system complexity [20]. Next, we hypothesize that subjects with HFA will exhibit lower complexity than the TD group for some, if not all, emotion conditions. To understand the dynamics of specific facial surface regions, we divide the face into smaller regions (eye, cheek, and mouth) following [21]. MSE-based complexity analysis is performed on each specified region separately. Through this analysis, we expect to identify those particular regions of the face that contribute to differences across HFA and control groups.

In case of reduced complexity in facial dynamics, one would expect higher predictability between facial regions i.e. the movement pattern of one region could be predicted well using the information from another. To understand such dynamical relationship between facial regions, i.e., covariation in their movement patterns, we adopt a predictive modeling approach. Using the Granger causality model [22], we attempt to predict the dynamics of one facial region using the information from the other. In cases where behavior of one region can be predicted significantly well by another region, a causal dependency is established. We study the pairwise causal dependency between facial regions for the various facial expression conditions. Our hypothesis is that in case of reduced complexity in overall dynamics, there will be a stronger presence of causal dependency between facial regions.

The rest of the article is organized as follows: Section 2 provides details on the data collection process, Section 3 describes the computational methodologies we have used to analyze facial dynamics along with observations, followed by a discussion in Section 4 and conclusion in Section 5.

2 Database

In this computational study, mocap technique is used to record facial movements of HFA and TD subjects as they produce various expressions related to emotion. To reduce task-specific variability, we adopt a mimicry paradigm, where subjects mimic a fixed set of facial expressions performed by actors in stimuli videos.

2.1 Participants

Twenty participants (2 females and 18 males) with HFA and nineteen (1 female and 18 males) TD subjects, all aged between nine and fourteen years, were recruited for this experiment at the FACE Lab at Emerson College. Diagnosis of ASD was confirmed via the Autism Diagnostic Observation Schedule, Module 3 [1] by trained administrators and confirmed by clinical impression. All participants in the ASD group demonstrated language and cognitive skills within normal limits, allowing us to describe them as having HFA. We administered the Childhood Autism Rating Scale [23] to all participants and excluded participants who scored above the threshold indicating concern for ASD from the TD cohort. Participants with learning differences (e.g. dyslexia), known genetic disorders, or other relevant diagnoses (e.g. attention deficit hyperactivity disorder) were excluded from both groups to reduce heterogeneity of the cohort. The mean ages of HFA participants (12.90 ± 3.19) and TD participants (12.67 ± 2.34) were not significantly different (t(37) = 0.25, p = 0.80). We assessed the subjects' IQ using the Leiter International Performance Scale [24] and receptive vocabulary with the Peabody Picture Vocabulary Test [25]. The mean IQ of HFA participants (106.35 ± 15.38) and control participants (108.74 ± 11.93) were also not significantly different (t(37) = 0.53, p = 0.59). Parents of all participants gave informed written consent and participants over the age of twelve provided written consent for participation in the experiment. This study was approved by the Institutional Review Board of University of Massachusetts Medical School.

2.2 Data Acquisition and Preparation

Thirty two reflective markers were affixed to the face of each participant (see Fig. 1 for marker positions). The movement of these markers was recorded by six infrared motion-capture cameras at 100 frames per second. The participants were instructed to mimic expressions in video stimuli selected from the Mind Reading corpus [14]. A study staff member was present in the room throughout the duration of the study to answer questions and repeat instructions if necessary.

Fig. 1.

Positions of the 32 facial motion capture markers affixed to each participant's face (left), Division of the 28 markers (excluding the stability markers) into eye, cheek and mouth regions (right)

The stimuli included two predefined, very similar sets of expressions with 18 tasks (smiling, frowning, being tearful, etc.) in each set. Each participant mimicked only one set of expressions. For ease of interpretability, these expressions are grouped into the six basic emotion categories that they are associated with - anger, fear, disgust, joy, sadness and surprise. The emotion labels for the tasks are provided in the Mind Reading corpus, and have been validated in prior research [26], [27]. Since these are not spontaneous emotions, we will refer to these categories as facial expression condition in this paper.

The specific arrangement of markers was based on a ninety marker, high-resolution face template [28] developed through the analysis of basic facial movement patterns using concepts of the Facial Action Coding System (FACS) [29]. We reduced the set to the 32 most critical markers delineating all central facial features, as well as overall head movement, and maintained a minimum of 4mm separation between markers to maximize recording resolution. Out of the 32 facial markers, 4 stability markers (the solid orange markers on the forehead and near the ears as shown in Fig. 1) are used to measure and later correct head motion. The positions of the remaining 28 markers are recomputed with respect to these stability markers to remove head motion. The resulting motion data are aligned, centered, and inspected manually to remove and correct artifacts. Face normalization is performed to remove subject-specific structural variability that may exist due to different facial structures and shapes of the subjects. The processed mocap data thus accounts for pure expression-related motion, and is free from head motion and subject-specific structural variability. For more details on the preprocessing methods, refer to our previous work [16].

Each such preprocessed mocap time series, consisting of horizontal and vertical coordinates of a marker, is next converted to a distance time series $\mathcal{D}=\{d_1,d_2,\dots ,d_N\}$, where d_i is the distance of the marker from its rest position at the i^th time instant. We use this distance time series data for further computational analyses.

3 Data Analysis and Interpretation

Our study of facial expression patterns in autism involves understanding the overall dynamics of the markers of the entire face, localized dynamics in specific facial regions, as well as the covariation of dynamics between facial regions.

3.1 Overall Facial Dynamics

In this section, we investigate whether the TD group and the HFA group have similar or differing patterns of complexity in their overall facial dynamics using the multiple scale entropy (MSE) method [20], [30]. Although the interpretation of complexity varies with the physiological parameters being studied and the developmental condition being investigated, there is significant evidence for various pathological processes being associated with atypical and often reduced measures of physiological complexity [18], [19]. We hypothesize that subjects with HFA will exhibit lower complexity than the TD group for some, if not all, facial expression conditions.

The complexity of a dynamical system can be understood as the rate at which new information is produced. MSE is a measure of dynamic complexity which is often useful in the context of physiological time series. This method estimates the complexity of a time series by computing an information theoretic quantity, called the sample entropy, at multiple time scales. The sample entropy S_e at a given scale is computed as the negative logarithm of the conditional probability of two sequences within the time series being similar (in the sense of a distance metric) in an (m + 1)-dimensional space, given that they are similar in the m-dimensional space.

Consider a time series X = {x₁, x₂, …, x_N} from which we may form an m-dimensional vector $X_i^m=\{x_i,x_{i+1},\dots ,x_{i+m-1}\}$. Let the distance between any such two vectors be denoted as $d(X_i^m,X_j^m)$, where i ≠ j enforces no self-matching. The function d(·) in our work is the Chebyshev distance, although any distance function is applicable. Let C^m(r) be the number of vector pairs for which $d(X_i^m,X_j^m)<r$ where r is a predefined threshold. Similarly, C^m+1(r) denotes the number of cases where $d(X_i^{m+1},X_j^{m+1})<r$. The sample entropy is computed as:

$$S_e=-ln\frac{C^{m+1}\left(r\right)}{C^m\left(r\right)}$$ (1)

For a time scale factor τ, the original time series X is first coarse-grained to obtain Y_τ = {y₁, y₂, …, y_N/τ} where $y_j=\frac{1}{\tau }{\Sigma }_{i=(j-1)\tau +1}^{j\tau }{x}_i$. For multiscale analysis, S_e is computed for multiple values of τ. In this study, we have used m = 2, and r = 0.2× standard deviation of the time series. These parameter values are chosen based on previous studies that show that sample entropy has good statistical validity for these values [31].

To perform MSE analysis on our database, we use the distance time series data $\mathcal{D}$ containing the distances of each facial marker from its rest position (see section 2.2). For each subject performing a mimicry task, we thus have a multichannel time-series, ${\mathbf{D}}_{\mathit{task}}^q\in {\mathbb{R}}^{N\times M}$, where q is the subject id, N is the length of the time series, and M = 28 facial markers. The sample entropy of ${\mathcal{D}}_{\mathit{task}}^q$ is computed for each of the 28 channels at τ = 1, 2, .., 25. An overall measure of complexity is obtained by adding the S_e values across all channels for each ${\mathcal{D}}_{\mathit{task}}^n$ at each scale. If a subject performs multiple tasks under the same expression condition, the MSE values are averaged across tasks at each scale.

The overall complexity of each subject was computed in terms of the sample entropy values at 25 scale factors. To identify group difference, a two-sample t-test was performed at each scale for every expression condition at 5% significance level. In general, a system is considered more complex than the other if it produces higher values of sample entropy relative to the other consistently over increasing values of scale factor. As predicted, the results of MSE analysis shows that the HFA group has significantly reduced sample entropy patterns for three expression conditions: disgust (all 25 scales), joy (all except scales 4 and 7) and sadness (all 25 scales). Plots of group differences are presented in Fig. 2 where red markers indicate the scales at which the group difference is significant i.e. p ≤ 0.05. For anger, significant group difference was observed only at τ = 1 which is not sufficient to infer group difference in complexity pattern. No difference is observed for fear or surprise.

Fig. 2.

Group differences in overall complexity. The red markers on the HFA plot indicate the scale factors at which HFA group has significantly lower complexity compared to the control group (error bars are omitted for clarity).

3.2 Localized Facial Dynamics

It is possible that group differences in overall facial dynamics, when they exist, arise from only certain parts of the face while other facial regions exhibit normal behavior. A well established approach to studying local facial movements involves using the FACS [29], [32]. The FACS is designed to encode movement of individual facial muscles (called action units) from slight changes in facial appearance. Examples of action units are inner brow raise and lip corner pull. The facial action units thus can be considered as the low-level building block of facial expressions.

In this work, however, we do not follow the action units-based approach. Studies have reported important behavioral traits of children with autism related to facial regions - the eye avoidance hypothesis in autism [33], [34], [35] for example. Hence, we intend to analyze facial movements at the level of facial regions. We group the markers into three facial regions, eye, cheek and mouth, pertaining to the core feature areas [36] (see Fig. 1). This kind of partition has also been adopted in [21].

3.2.1 Complexity analysis

To compute the local complexity of a region, we average the MSE values across all the markers present in that region, separately at each scale factor. Two regions are considered different if their complexity is significantly different (p ≤ 0.05) for a majority of the scale factors.

To investigate the group differences in local regions of a face, a two sample t-test was performed on the MSE results for each of the three regions (see Fig. 3). For anger and fear, no facial region was found to be significantly different between the groups. For disgust (eye: p = 0.008, cheek: p = 1.77e − 5, mouth: p = 0.003) and sadness (eye: p = 7.34e − 4, cheek: p = 0.025, mouth: p = 6.61e − 4), reduced complexity is observed for the HFA group in all three regions. For joy, the group difference comes from eye (p = 0.014) and mouth (p = 0.022) regions. Although, the overall complexity pattern of the HFA group does not differ from that of the control group for surprise, the mouth region (p = 0.009) still shows significantly reduced complexity. Overall, the eye region shows the highest between-group difference in complexity, followed by mouth and cheek regions (see Fig. 4).

Fig. 3.

Group difference in complexity for facial regions. Shaded regions indicate where the HFA group exhibit significantly lower complexity (p ≤ 0.05) than the control group.

Fig. 4.

Group difference in complexity for facial regions for all expression conditions

In addition, we report the region of maximum complexity pertaining to each expression condition for the two groups (see Table 1). We observe that apart from fear, the regions of highest complexity between the two groups are always different. While the control group produces the most complex motion in the eye region in cases of anger, joy, sadness and surprise, the highest complexity in the HFA group is observed in the cheek region for these emotion-related expressions.

TABLE 1.

Region of highest complexity

	HFA	TD
Anger	cheek	eye
Disgust	mouth	cheek
Fear	cheek	cheek
Joy	cheek	eye
Sadness	cheek	eye
Surprise	cheek	eye

3.2.2 Similarity analysis

In this section, we investigate the (dis)similarity between the HFA and control groups in terms of their facial expression dynamics in the three facial region. To measure the group difference, we employ the dynamical time warping (DTW) method. DTW measures the similarity (or dissimilarity) between two temporal sequences by finding the best alignment between them in terms of a warping distance. It directly compares the dynamical patterns in the sequences without making any statistical assumptions. DTW has wide applications in diverse domains, such as speech recognition [37] and emotion classification [38].

Given two multidimensional time series X ∈ R^d×N_x, Y ∈ R^d×N_y, where d is the data dimensionality and N_x and N_y are the lengths of X and Y. DTW finds the best warping path by optimizing the distance between X and Y. We construct a distance matrix $\mathcal{D}\in R^{N_x\times N_y}$, where the element d_i,j measures the distance between the i^th point x_i in X and the j-th point y_j in Y. In this work, we use ℓ₂-norm to measure the pointwise distance: d_i,j = ∥x_i − x_j∥₂. A warping path W = w₁,w₂, ⋯,w_K defines a mapping between X and Y, where the k-th element of W is defined as w_k = (i, j), where w_k(1) = i and w_k(2) = j. X can be warped to the same length of Y based on the warping path, i.e., the i^th point of X corresponds to the j^th point of Y. The optimal warping path is the one that minimizes the warping distance d_w(W):

$$d_w\left(W\right)=\sum _{k=1}^K\mathcal{D}\left(w_k\right(1),w_k(2\left)\right)$$ (2)

This path can be found using dynamic programming with space and time complexity of $\mathcal{O}\left(N_x{N}_y\right)$.

We apply DTW to measure the similarity between every pair of HFA and TD subjects using the corresponding facial region-based time series data, and compute the warping distance d_w(W). Fig. 5 presents the average HFA-TD similarity of facial dynamics in each region, where brighter color indicates larger warping distance, i.e., lower similarity between the groups. Consistent with our previous observation, the eye region shows a significantly larger dissimilarity than the cheek region across expression conditions (p < 0.01), and than the mouth region in the expression of anger, fear, joy and surprise. In addition, the cheek region exhibits the smallest dynamic difference (p < 0.01). The eye region-based HFA-TD difference in the expressions of joy and fear is significantly larger compared to those in other expression conditions (p < 0.01).

Fig. 5.

Average dissimilarity in facial region dynamics between HFA and TD. Brighter color indicates larger dissimilarity.

3.3 Relationship in Movement Patterns between Facial Regions

Facial expressions are produced as the results of a complex interplay and coordination between various facial regions. Such dynamic relationships may be highly non-linear, and difficult to discern with simple measures like correlation or coherence. In addition to studying the existence of the cross-region relationship, we are interested in the interaction (directional) effects between movement patterns in different regions. To study the complex dynamic relationship between facial regions, we use the Granger causality model [22].

Granger causality is a popular statistical measure for analyzing the directional influence of one time series on another [22]. It has been widely applied to analyze various physiological and biomedical signals [39] [40] [41]. Compared to other metrics, such as correlation which is linear and symmetric, Granger causality measures a non-linear and directional relationship between variables and is more suitable for the lag-lead system.

Given two time series X and Y, the Granger causality measure ${\mathcal{F}}_{X\to Y}$ defines the influence of the past information of X on improving the prediction of the current value of Y. Let's consider the following two linear regression models:

$$\begin{array}{cc}\hfill Y\left(t\right)=\sum _{i=1}^P{\alpha }_{i}Y(t-i)+{ϵ}_Y\left(t\right)& \hfill \\ \hfill Y\left(t\right)=\sum _{i=1}^P{\beta }_{i}Y(t-i)+\sum _{i=1}^P{\delta }_{i}X(t-i)+{ϵ}_{XY}\left(t\right)& \hfill \end{array}$$ (3)

where P is the maximum number of lags for X and Y in the models; ${\left\{{\alpha }_i\right\}}_{i=1}^P$, ${\left\{{\beta }_i\right\}}_{i=1}^P$ and ${\left\{{\delta }_i\right\}}_{i=1}^P$ are the regression coefficients, and ∊_Y and ∊_XY are the residuals of the regression models. The Granger causality measure is then defined by taking the natural logarithm of the ratio of the total variance of the two residuals ∊_Y and ∊_X:

$${\mathcal{F}}_{X\to Y}=\ln \left(\frac{\mathit{var}\left({ϵ}_Y\right)}{\mathit{var}\left({ϵ}_{XY}\right)}\right)$$ (4)

If ${\mathcal{F}}_{X\to Y}$ is significantly greater than zero, then X is Granger causing Y. This significance is examined using an F-test.

For Granger causality analysis, we first determine the model order P using the Akaike information criterion (AIC). Analysis shows that 35% of the subjects attain a model order of 3, and hence we use P = 3 throughout our analysis. Causal dependency is examined for all possible ordered pairs of facial regions. This requires testing 6 unique ordered pairs for causality. Fig. 6 presents the Granger causality patterns between facial regions for the HFA and TD groups. The arrows indicate the direction of causal dependency originating from the region that predicts pointing to the predicted region. The thickness and the color of the arrows correspond to the percentage of subjects showing statistically significant (p ≤ 0.05) causality between the relevant region pairs. Since the resulting percentage of subjects showing significant causality over all the pairs of facial regions ranges from 10% to 70%, we equally divide the interval [10%, 70%] into three groups to represent different strength levels between regions: low-level [10%, 30%], medium-level [30%, 50%] and high-level [50%, 70%]. As expected, we observe that subjects with HFA exhibit stronger causal dependency between regions compared to their TD peers. The dependency is particularly pronounced for the expression conditions of anger, disgust, joy and sadness.

Fig. 6.

Granger causality pattern between facial regions in the HFA and TD groups. The thickness and color of the arrows represent the percentage of subjects showing statistically significant causality (p ≤ 0.05) for the corresponding region pairs.

We also observe that the causal relationship between adjacent regions is stronger than that between non-adjacent ones for both groups. For example, the coupling between cheek and mouth regions is stronger than that between eye and mouth. A strong eye-to-mouth relationship for HFA subjects is also observed for the expressions of joy.

4 Discussion

The present study found reduced complexity in facial expression dynamics of subjects with HFA relative to their TD peers. Significant difference is observed for expressions related to disgust, joy and sadness. These movement differences may be the reason naïve observers perceive the facial expressions of individuals with ASD as awkward. In the context of facial expression, reduced complexity can be understood as a lack of variability, subtlety, or richness in overall facial dynamics. Our observation supports a previous study that noted more neutral/flat affect expressions in HFA group compared to their TD counterparts [5]. Our findings of reduced complexity are also consistent with past research showing atypical, often reduced, complexity measures of physiological dynamic systems in people with developmental or other conditions [18], [19].

We have also identified the facial regions with atypical complexity in HFA subjects for each expression condition. For expressions related to emotions like disgust and sadness the reduced complexity in HFA subjects is observed across all facial regions, while for other emotions (joy and surprise) only parts of the face exhibit atypical dynamics.

On the other hand, from both local complexity and DTW-based similarity analysis, we find that the major group difference in facial dynamics comes from the eye region. Recall that the eye region exhibits the highest complexity in the TD group (see Table 1), but the HFA group shows reduced complexity in this region. The observed dynamic dissimilarity may result from this difference in complexity.

As shown in Table 1, HFA subjects exhibit highest complexity in the cheek region, which also explains the smallest dynamic difference observed in Fig. 5. These observations indicate that the cheek movements of participants with HFA are more natural and also reinforce that the perceived atypicality and awkwardness likely result from reduced complexity of movements in the eye, rather than the cheek region. One possible explanation for this behavior may come from the eye avoidance hypothesis in autism where children with autism avoid looking at the eye region of the face [33], [34], and may not be able to produce the intricate movements in the eye region, because they lack experience perceiving and processing this complex dynamic information.

The Granger causality analysis shows that HFA subjects, in general, have stronger causal dependency between facial regions compared to TD subjects. Group difference is especially prominent in the emotions of anger, disgust, joy and sadness. Recall that the HFA subjects have reduced complexity for these emotions. The strong dependency between facial regions in the HFA group suggests that subjects with HFA have smaller degrees of freedom in the underlying production mechanism, which in turn, supports our observation of reduced complexity in overall facial movements.

5 Conclusion

Several autism research studies have emphasized the importance of understanding vocal [9], [11] and facial behavioral expressions [7], since they have such significant impact on how children with HFA are perceived by naïve observers. In this study, we analyzed mimicked facial expressions in children with HFA using computational techniques. We observe that facial expressions of children with HFA have reduced complexity when compared to those of TD subjects. This is further emphasized by the observation of higher causal dependency between various facial regions in HFA across various expression conditions, suggesting lower degrees of freedom in the underlying mechanism. The group differences in expression dynamics are more prominent for expressions of anger, disgust, joy and sadness, and mainly arise from the eye region.

Since our observations are made based on analyzing the mocap recordings of only 20 children with autism, we cannot generalize the findings to the entire autism spectrum population, particularly due to the inherent heterogeneity in the condition. Our current experimental setup also can not identify whether the observed differences between the HFA and TD subjects resulted from an underlying group difference in perceiving the visual stimuli vs. a pure production difference. Our findings also do not speak to the spontaneous production of expressions during naturalistic social interactions - since the expressions we analyzed are based on a mimicry paradigm. Nevertheless, within the scope of this study, our analyses suggest that children with HFA lack richness and variability in their facial expression patterns, indicating a production mechanism that allows less degrees of freedom of movement between regions.

Biographies

Tanaya Guha is an Assistant Professor in the department of Electrical Engineering at the Indian Institute of Technology (IIT) Kanpur. Prior to joining IIT, she was a postdoctoral fellow at the Signal Analysis and Interpretation Lab (SAIL), University of Southern California (USC) from 2013 to 2015. She has received her PhD in Electrical and Computer Engineering from the University of British Columbia (UBC), Vancouver in 2013. She was a recipient of Mensa Canada Woodhams memorial scholarship, Google Anita Borg memorial (finalist) scholarship and Amazon Grace Hopper celebration scholarship. Her current research interests include human-centered signal processing including human emotion and behavior analysis, and multimodal signal processing.

Zhaojun Yang is a PhD candidate in Electrical Engineering at the University of Southern California (USC). She received her B.E. Degree in Electrical Engineering from University of Science and Technology of China (USTC) 2009 and M.Phil. Degree in Systems Engineering and Engineering Management from Chinese University of Hong Kong (CUHK) 2011. Her research interests include multimodal emotion recognition and analysis, interaction modeling and spoken dialog system. She was awarded with the USC Annenberg Fellowship (2011–2015). Her work (with S. S. Narayanan) has won the Best Student Paper Award at ICASSP 2016.

Ruth B Grossman is an associate professor at Emerson College where she teaches graduate and undergraduate classes in the department of Communication Sciences and Disorders. Her research is focused on various aspects of face-to-face communication in children with Autism Spectrum Disorders (ASD). She is specifically interested in how children with ASD integrate and produce verbal and nonverbal information, such as facial expressions and prosody. Her work has been published in several refereed journals, including Journal of Child Psychology and Psychiatry, Journal of Speech, Language, and Hearing Research, Journal of Nonverbal Behavior, Autism, and Autism Research. More about her her can be found in facelab.emerson.edu.

Shrikanth (Shri) Narayanan (StM'88 - M'95 - SM'02 - F'09) is Andrew J. Viterbi Professor of Engineering at the University of Southern California (USC), and holds appointments as Professor of Electrical Engineering, Computer Science, Linguistics, Psychology, Neuroscience and Pediatrics and as the founding director of the Ming Hsieh Institute. Prior to USC he was with AT&T Bell Labs and AT&T Research from 1995–2000. At USC he directs the Signal Analysis and Interpretation Laboratory (SAIL). His research focuses on human-centered signal and information processing and systems modeling with an interdisciplinary emphasis on speech, audio, language, multimodal and biomedical problems and applications with direct societal relevance. [http://sail.usc.edu].

Prof. Narayanan is a Fellow of the Acoustical Society of America and the American Association for the Advancement of Science (AAAS) and a member of Tau Beta Pi, Phi Kappa Phi, and Eta Kappa Nu. He is the Editor in Chief for IEEE Journal of Selected Topics in Signal Processing, an Editor for the Computer Speech and Language Journal and an Associate Editor for the IEEE Transactions on Affective Computing, APSIPA Transactions on Signal and Information Processing and the Journal of the Acoustical Society of America. He was also previously an Associate Editor of the IEEE Transactions of Speech and Audio Processing (20002004), IEEE Signal Processing Magazine (20052008), IEEE Transactions on Multimedia (2008–2011) and the IEEE Transactions on Signal and Information Processing over Networks (2014–2015). He is a recipient of a number of honors including Best Transactions Paper awards from the IEEE Signal Processing Society in 2005 (with A. Potamianos) and in 2009 (with C. M. Lee) and selection as an IEEE Signal Processing Society Distinguished Lecturer for 20102011 and ISCA Distinguished Lecturer for 2015–2016. Papers co-authored with his students have won awards including the 2014 Ten-year Technical Impact Award from ACM ICMI and at Interspeech, 2016 ICASSP, 2015 Nativeness Detection Challenge, 2014 Cognitive Load Challenge, 2013 Social Signal Challenge, Interspeech 2012 Speaker Trait Challenge, Interspeech 2011 Speaker State Challenge, InterSpeech 2013 and 2010, InterSpeech 2009-Emotion Challenge, IEEE DCOSS 2009, IEEE MMSP 2007, IEEE MMSP 2006, ICASSP 2005 and ICSLP 2002. He has published over 700 papers and has been granted seventeen U.S. patents.