Computerized facial analysis for understanding constricted/blunted affect: initial feasibility, reliability, and validity data

Abstract

Diminished expression is a diagnostic feature of a range of schizophrenia-spectrum disorders/conditions and is often unresponsive to treatment, is present across premorbid, first episode and various clinical states, and is considered a poor prognostic indicator. Surprisingly, little is known about diminished expression. The present study sought to address this issue by evaluating a commercially-available computerized facial analysis software for understanding diminished expressivity. We analyzed natural facial expression from a series of laboratory interaction tasks in 28 individuals with psychometric schizotypy – defined as the personality organization reflecting a putative genetic schizophrenia liability, and 26 matched controls. We evaluated (a) feasibility – defined in terms of the number of video frames recognized by the software, (b) reliability – defined in terms of correlations between facial expression variables across the three laboratory interactions, and (c) construct validity – defined in terms of relationships to clinical variables. For most subjects (~ 80%), approximately three-quarters of the video frames were analyzable by the software; however, a minority of the videos was essentially unreadable. The facial expression variables showed excellent reliability across interaction conditions. In terms of construct validity, facial expression variables were significantly related to a measure of psychoticism, tapping subjective cognitive concerns and “first-rank” schizophrenia symptoms, but were generally not different between groups. Facial expression variables were generally not significantly related to measures of depression, anxiety, paranoia or, surprisingly, self-reported negative schizotypy. While computerized facial analysis appears to be a reliable and promising method of understanding diminished expressivity across the schizophrenia-spectrum, some work remains. Implications are discussed.

1.0 INTRODUCTION

Many individuals with schizophrenia-spectrum disorders have blunted or constricted affect (American Psychiatric Association [APA], 2000) - expressive deficits that are generally intractable and, despite considerable empirical attention, are a mystery in terms of underlying pathophysiology. One major factor contributing to this dearth of understanding is that measurement of expressive deficits is, for the most part, dependent on symptom rating scales (e.g., Andreasen, 1984; Horan et al., 2011; Kirkpatrick et al., 1989; Kirkpatrick et al., 2011; see also Kring and Sloan, 2007 for behavioral based coding strategies). Data from symptom rating scales often cover wide temporal intervals (e.g., weeks, months, or years), are relatively insensitive to change given their comparatively few response options and ambiguous operational definitions, produce ordinal data that are inappropriate for parametric statistics, and are imprecise for isolating specific behaviors from other negative symptoms (Alpert et al., 2002; Cohen et al., 2008; Mueser et al., 1994; Stahl and Buckley, 2007). Although integral to schizophrenia research, these scales have limited use for providing all but a gross understanding of how expressive deficits vary within individuals, how they differ between individuals, and how they are broadly correlated to other clinical and pathophysiological phenomena. Emerging computer-based technologies have allowed for assessment of natural behavior with near perfect inter-rater reliability, greater sensitivity and specificity than clinical rating scales (Alpert et al., 2002; Cohen et al., 2008), and even greater efficiency than behavior-based coding systems. Several programs are currently in development for measuring expressive deficits in schizophrenia-spectrum disorders (e.g., Alvino et al., 2007; Cohen et al., 2008; Cohen et al., 2010a). The present study sought to compliment this endeavor by evaluating a commercially-available, computerized facial analysis software – an advantageous effort in that this software is easily accessible to researchers, has dedicated customer service and technological support, and has full-time developers interested in expanding its application.

In this study, we employed facial analysis software to understand expression in individuals with schizotypy – defined as the personality organization reflecting the putatively genetic risk for schizophrenia-spectrum pathology. There are advantages for feasibility testing in this population, in that these individuals are generally high functioning and computer-literate, and thus more able to comply with laboratory procedures. Moreover, these individuals are, for the most part, unmedicated, cognitively intact, and show lower levels of comorbid (and thus, potentially complicating) substance abuse, depression, anxiety and other disorders. With these benefits in mind, we sought to evaluate whether output variables from the commercial software package (a) can meaningfully detect facial expressions in individuals with psychometrically-defined schizotypy and controls (i.e., feasibility), (b) show appreciable temporal stability across laboratory assessments (i.e., reliability), and (c) meaningfully relate to demographic and clinical features of schizotypy (i.e., construct validity).

2.0 METHODS

2.1 Participants

Participants were college undergraduates approached by email to participate in an on-line survey and offered a chance to win monetary prizes (N = 10,258). The survey included a consent form, basic demographic questions, the Schizotypal Personality Questionnaire – Brief Revised (SPQ-BR; Cohen et al., 2010b), the Brief Symptom Inventory (Derogatis and Melisaratos, 1983), and the Chapman Infrequency Scale (Chapman and Chapman, 1983). Individuals endorsing more than three of 14 infrequency items were considered ineligible for the present study. The response rate was modest (n = 2300). This study was approved by the Louisiana State University Human Subject Review Board and all participants offered informed consent prior to completing the surveys. Individuals scoring in the 95^th percentile on the gender determined means for the positive (n = 64), disorganization (n = 36), or negative (n = 53) subscales from the SPQ-BR were contacted about participating in a laboratory study. Fifty-three individuals were also recruited who had elevations on multiple SPQ-BR scales. Individuals scoring high on the negative subscale (defined as a sum of “constricted affect” and “no close friends” subscales; Cohen et al., 2010b) were considered for the study only if they (a) also showed elevations (defined as the 95^th percentile or higher) on the positive or disorganization subscales, or (b) had a depression subscale score from the Brief Symptom Inventory (Derogatis and Melisaratos, 1983) below their gender determined mean. In this manner, we sought to reduce the chances of individuals who were depressed (but not schizotypal) being recruited for the study. Subjects were excluded if they endorsed a personal history of schizophrenia diagnosis. Of the 206 subjects eligible for the schizotypal group, 37 were recruited and completed the laboratory testing. Thirty-three control subjects, of a total possible 485, selected from participants scoring below the gender-determined means for each of the positive, disorganization, and negative SPQ-BR factors, were also recruited. The 50th percentile was selected based on our prior research that individuals scoring below this cut-off are highly unlikely to have a history of schizophrenia diagnosis, inpatient hospitalization, or psychiatric or psychological treatment more generally (Cohen and Najolia, 2011). Controls were excluded if they reported a family history or self-diagnosis of schizophrenia.

2.2 Traits and Symptoms

Schizotypal traits were measured using the Brief-Revised version of the Schizotypal Personality Questionnaire (Cohen et al., 2010b). Reported in this manuscript are data for the superordinate cognitive-perceptual, negative, and disorganization factor scores. Clinical symptoms were measured using the BSI (Derogatis and Melisaratos, 1983) which measures a broad range of psychopathology during the past seven days. We were particularly interested in depression (i.e., “feeling no interest in things”), anxiety (i.e., “feeling tense or keyed up”), paranoia (i.e., “feeling that most people cannot be trusted”), and psychoticism (i.e., “the idea that someone else can control your thoughts”, “the idea that something is wrong with your mind”) scales from this instrument, as those scales would be expected to be related to either diminished facial expression or schizotypy more generally. The BSI has well-documented psychometric properties and has been used in hundreds of published, peer-reviewed studies to date.

2.3 Interaction Task

Subjects were seated in front of a computer monitor and asked to discuss three separate 90-second autobiographical memories separately involving neutral, positive, and negative memories from their lives. Instructions, for example, for the positive condition were as follows:

“Tell me some stories about when you were feeling really good. Please get into telling this story as much as you can, and talk for about 90 seconds. Some things to talk about include:

1. Times you were really happy with someone.

2. Times when you accomplished something really special.

3. Times you were feeling at your best.

The research assistant was out of view of the subject while they were speaking in order to minimize effects due to experimenter characteristics (i.e., individual differences in sex, ethnicity). Subjects were asked to talk to the research assistants (while looking at the computer screen), though research assistants were not allowed to speak during the narrative task. The task was digitally recorded using a camera fixed to the top of the computer monitor. Consistent with the recommendations detailed in the software manual, the overhead lights were turned off and two LED light towers, located on either side of the subject’s face, were illuminated. Video was continuously recorded during the various speaking conditions and was spliced by research assistants using video editing tools to ensure each recording was exactly 90 seconds long. Beyond discussing autobiographical memories, subjects were blind to the purpose of this task.

2.4 Computerized Facial Analysis

FaceReader version 4.0 (Noldus Information Technology, 2010), a commercially-available program developed by Noldus Information Technology, was used to measure facial expressions. FaceReader is an automated program that uses algorithms to evaluate, on a video frame-by-frame basis, facial images in terms of seven emotional states – happy, sad, angry, surprised, scared, disgusted, and “neutral”. These variables reflect a measure of the magnitude of that emotion being shown from 0 (not at all) to 100 (perfect match). Additionally, FaceReader can measure movement in eye gazes, mouth movements, and head movements along X, Y, and Z axes. For the present study, each subject yielded approximately 6750 frames for analysis. It is important to note that each frame is not necessarily analyzable, as FaceReader requires recognition of key facial features in order to evaluate emotional valence and head movement. As such, the “percentage of frames recognized” is a key variable for evaluating whether the software can be applied to schizophrenia research using our laboratory methods. For the present study, we report data on neutral, positive, and negative valence (defined as a sum of sad, angry, scared, and disgusted category scores) and on variability in eye, mouth, and head movement (defined as the standard deviation of values for each of these categories). Note that the head movement variable reflected a sum of movements across X, Y, and Z axes. For further scientific applications of the FaceReader software, see Chentsova-Dutton and Tsai (2010) as well as Choliz and Fernandez-Abascal (2012). “Surprise” emotions were excluded from the present study, given their ambiguity with respect to valence. Computer problems led to two subjects’ data being unusable. Data for these subjects were excluded from all analyses.

2.5 Analyses

The analyses were conducted in five steps. First, we examined potential demographic and clinical differences between the schizotypal and control groups that might inform subsequent analyses. Second, we evaluated the number of frames filled for the whole sample, and whether this number varied as a function of demographic or clinical variables. Third, we evaluated the consistency in facial expressions across the three separate interaction conditions using bivariate correlations (for the whole sample). Given that the task was not intended to be a mood manipulation per se but rather as a comprehensive assessment of facial expressions across a range of topics, we expected that the variables would show high consistency across the affective conditions. Given the well-established differences between men and women in terms of emotional expression (Sobin and Alpert, 1999), sex was included as a covariate for this analysis. Fourth, we computed zero-order correlations to determine the extent to which the six facial expression variables (computed across the three conditions) correlated with each other (for the whole sample). High correlation values indicate redundancy (e.g., > .90), whereas low correlation values suggest that the variables are distinct. Fifth, we compared the schizotypal and control groups in facial expression variables. For these analyses, sex was examined as an independent variable. In this manner, we were able to evaluate whether there were group differences, sex differences, and interactions between these variables in expressiveness. Finally, we computed partial correlations (controlling for sex) between facial expression measures and clinical characteristics – cognitive-perceptual, negative, and disorganization scores from the SPQ-BR and depression, anxiety, paranoia, and psychoticism scores from the BSI (within the schizotypal group only). All analyses in this study are two-tailed and all variables are normally distributed (skew < 1.5) unless otherwise stated.

3.0 RESULTS

3.1 Demographics

Descriptive and clinical information is provided in Table 1. The control and schizotypal groups were not statistically different in sex, ethnicity, or age. As per our group definitions, the schizotypal group had more severe schizotypal traits (all p’s < .001). Similarly, the schizotypal group was more severe in depression, anxiety, paranoia, and psychoticism (p’s < .001). Nearly 20% of the schizotypal group reported a family history of schizophrenia.

Table 1.

Descriptive statistics for demographic and clinical variables for the control and schizotypal groups.

	Controls	Schizotypy
	n = 26	n = 28
Sex (% Female)	62%	61%
Ethnicity
% Caucasian	92%	75%
% African American	4%	11%
% other	4%	14%
Age	19.00 ± 2.28	18.54 ± 1.10
Family History of Schizophrenia a	--	19%
Schizotypal Personality Questionnaire – Brief Revised
Cognitive-Perceptual a	6.81 ± 5.18	30.71 ± 9.61
Negativea	3.31 ± 2.17	11.71 ± 6.32
Disorganization a	9.42 ± 4.36	25.54 ± 5.34
Brief Symptom Inventory
Depression a	7.78 ± 1.70	17.27 ± 7.50
Anxiety a	8.17 ± 2.67	15.69 ± 6.87
Paranoia a	6.52 ± 2.54	12.54 ± 5.06
Psychoticism a	6.04 ± 1.26	12.5 ± 4.55
% of Video Frames Filled	.75 ± .24	.71 ± .26

^aGroups statistically different for this variable

3.2 Feasibility

The percentage of frames filled for the entire sample was generally good, however, there was a subset of subjects for whom very few frames were recognized by the emotion recognition software. Seven controls and seven schizotypal individuals had less than 10% of their frames recognized. These individuals were excluded from all analyses in this study. The remaining subjects had, on average, approximately three-quarters of their frames filled (see Table 1). There were no significant differences between these subjects and other subjects (i.e., those with < 10% of frames filled) in terms of ethnicity, age, or sex (p’s > .10). In the remaining 54 subjects, percentage of frames filled was not different between men and women (F[1] = 1.14, p = .29), nor different between the control and schizotypal groups (F[1] = .012, p = .90), nor was there a significant sex by group interaction (F[1] = .16, p = .69). The number of frames filled was not significantly related to age (r[56] = .04, p = .79), or any of the SPQ-BR or BSI or other clinical variables (p’s > .10). In sum, while video from a notable minority of subjects in this study were essentially “unreadable” by the software (approximately 20%), this did not reflect a systematic bias in terms of descriptive or clinical features. Note that these subjects were excluded from all main analyses in this paper, including those presented in Table 1.

3.3 Temporal reliability

Partial correlations (controlling for sex) were computed between facial expression measures across the three speaking conditions for the groups combined. Without exception, each of the expression measures was highly stable across the various conditions (all r’s > .60 but < .89, p’s < .001). It is notable that the valence variables were particularly stable (all r’s > .80 but < .89, p’s < .001). Recomputing these correlations separately by sex (instead of controlling for sex) did not change the findings in any meaningful way. For data reduction purposes and to improve reliability (i.e., the number of observations), measures were combined across conditions for subsequent analyses.

3.4 Zero-order correlations

Zero-order correlations (controlling for sex) were computed between the facial variables (omitted for space concerns). These findings suggest that (a) the valence variables were significantly correlated with each other (r[53]’s −.33 to −.48, p’s < .01), (b) with one exception (see below), the facial movement variables were not correlated with each other (r[53]’s < −.08, p > .10), and (c) the valence and facial movement variables were distinct (r[53]’s < .22, p > .10). Magnitude of neutral expression was associated with less happy (r[53] = −.34, p = .01) and negative (r[53] = −.34, p = .01) emotions, and magnitude of happy and negative emotions were inversely correlated with each other (r[53] = −.43, p < .001). Variability in head movement was associated with variability in eye gaze (r[53] = .43, p < .001). These results suggest that the emotional valence and movement variables were generally distinct. Recomputing these correlations separately by sex (instead of controlling for sex) did not change the findings in any meaningful way.

3.5 Schizotypal versus controls

Figure 1 contains the means and standard errors for the schizotypal and control groups, stratified by sex. There were no significant group effects (p’s > .10). None of the group differences exceeded a small effect size level (d’s < 0.33), suggesting this was not a power issue. Sex effects were observed for the neutral valence (F[1] = 14.77, p < .001, d = 1.01) and, at a trend level, happy valence (F[1] = 3.531, p = .07, d = 0.52) variables such that females were less neutral and happier in their facial expressions. None of the other sex effects were notable (p’s > .10, d’s < .33). One interaction term was at a trend level: neutral valence (F[2,52] = 3.96, p = .05). This interaction reflected female control subjects being lower in neutral valence expressions than male control (t[24] = 4.00, p = .001, d = 1.57) and male schizotypal (t[ 52] = 2.67, p = .01, d = 1.03) subjects. Conversely, female and male schizotypal subjects were not statistically dissimilar (t[26] = 1.35, p = .19, d = .50), and female schizotypal and control subjects were different at a trend level (t[31] = 1.97, p = .06.

Figure 1.

Computerized facial expression variables compared between schizotypal and control subjects, stratified as a function of sex (men = black bar, women = grey bar).

3.6 Clinical correlates

Table 2 contains the correlations between clinical and facial expression variables. There were three notable findings from these analyses. First, severity of negative traits was associated with less head movement but none of the other variables. Second, severity of depression and anxiety was not significantly related with any of the facial expression variables, though greater depression was associated with less positive valance and more variability in mouth movement at a medium effect size level. Finally, severity of psychoticism was associated with less positive valence and less eye and head movement at a trend or better level.

Table 2.

Partial correlations (controlling for sex) between facial expression variables and clinical variables for the schizotypal group only.

	Valence			Gesture
	Neutral	Positive	Negative	Eye	Mouth	Head
SPQ
Cognitive-Perceptual	0.07	−0.36+	0.20	0.04	−0.05	−0.20
Negative	−0.08	0.03	−0.05	−0.14	0.24	−0.40*
Disorganization	0.21	−0.31	0.11	0.13	−0.27	−0.12
General Psychopathology
Depression	0.14	−0.33	0.23	−0.23	0.30	−0.23
Anxiety	0.12	−0.25	0.07	−0.28	0.07	−0.27
Paranoia	−0.17	−0.32	0.33	−0.17	0.23	−0.30
Psychoticism	0.13	−0.41*	0.29	−0.37*	0.28	−0.42*

Note – correlations at or above a medium effect size level are underlined.

^*p < .05,

⁺p < .10

3.7 Post-hoc analysis: Family history

We then compared individuals from the schizotypal sample reporting a family history of schizophrenia versus those without on the facial expression variables. Individuals with a family history of schizophrenia showed a trend for decreased positive (t[27] = 1.90, p = .08) and (t[27] = 1.83, p = .09) increased negative emotions, but there were no statistically significant group differences.

4.0 DISCUSSION

The present study conducted an evaluation of computerized facial analysis for use in schizophrenia research. Feasibility was a qualified success in that approximately 80% of the video recordings were readable by the commercial software. It is noteworthy that approximately 20% of the recordings were unreadable, and this did not appear to be a function of identifiable clinical or demographic characteristics. It seems probable that lighting, camera angle, or other recording conditions contributed to this and these factors would be important to consider in future research. Future studies employing this program would likely benefit from a “calibration” phase to help optimize the recording conditions for each individual participant. In this regard, analysis of archived data (presumably when these idiosyncratic recording factors are not considered) using this software will be difficult. It is notable that similar considerations are standard practice for other methodologies and indeed the rates of usable data found in this study are not dissimilar to those found using other methods, such as skin conductance (Green et al., 1989). In terms of reliability, the software variables showed strong reliability across laboratory interactions suggesting that short term temporal reliability is good. This, taken with the fact that test-retest reliability for this software is near perfect (given the objective and automated nature of its analysis), suggest that reliability is excellent. The present study offers construct validity in that the computerized facial analysis variables were related to important clinical features (see below for further discussion), though abnormalities in facial expressiveness were not observed for the schizotypal group as a whole. In sum, computerized facial analysis appears to be a promising method of understanding expressive deficits across the schizophrenia-spectrum, and worthy of continued empirical investigation.

In the present study, schizotypy, as a group, was not atypical in computer-assessed facial expressions despite these individuals self-reporting high levels of negative schizotypy. Moreover, while negative schizotypal traits were significantly associated with reduced head movement, suggesting that this is an aspect of emotional expressivity that may be relevant to schizotypy, negative schizotypal traits were generally not related to abnormal facial expressions. When interpreting these null findings, it is important to note that emerging evidence suggests that schizotypy is characterized by a number of dysjunctions between self-report and behavior. For example, a dysjunction between self-report and behavioral variables has been observed in cognitive (e.g., subjective cognitive concerns versus actual cognitive performance; Chun et al., 2012), quality of life (i.e., subjective satisfaction versus objective quality of life indicators; Cohen et al., under review) and emotional experience (e.g., subjective experience versus psychophysiological and behavioral indicators; Cohen et al., 2012; Gooding et al., 2002) variables. Thus, this dysjunction may also manifest in self-reported versus objective emotional expression. It is also important to note that negative schizotypy, assessed using the SPQ-BR, taps a range of expressive and social interaction-related variables. One might expect that only a modest portion of the variance in negative traits would be explained by facial expressive deficits.

Of the clinical variables examined in this study, the psychoticism scale from the BSI showed the highest number of significant facial expression correlates. The reasons for this are unclear at the present time, but are interesting for future research. The psychoticism scale measures subjective cognitive and “first-rank” schizophrenia-like concerns. In this regard, it stands to reason that individuals with high scores on this scale may be those, at least in our sample, that were showing the most pronounced clinical schizophrenia-related symptomatology. It is noteworthy that facial expression generally did not significantly vary as a function of anxiety or depression, suggesting that the relation between facial expressiveness and clinical state may be specific to schizophrenia-spectrum pathology. Unfortunately, the present study lacked longitudinal data, so it is unclear whether facial expressive deficits are primarily reflective of clinical state – thus waxing and waning depending on severity of psychotic symptoms, or whether facial expressive deficits reflect a durable individual difference variable predicting elevated risk for psychosis. In the schizophrenia literature, facial expression deficits have been observed across a range of clinical states regardless of medication use (Kring et al., 1993; Kring and Neale, 1996) and have been observed well before onset of clinical symptoms (Walker and Lewine, 1990), suggesting that, at least in patients, facial expressive deficits reflect a trait-like quality. Longitudinal analysis of facial expressions in vulnerable populations would be helpful in this regard.

The present findings should be interpreted in light of the following limitations. First, the sample was composed of college students, which may be a concern from a generalizability perspective. A notable minority of these subjects reported a family history of schizophrenia, thus bolstering the notion that schizotypy is familial to at least some degree in these individuals. Second, our measures of schizotypy were based exclusively on self-report. Third, we did not control for multiple comparisons in our correlational analyses. Given the novelty of our analysis procedures and the relative power of our study, we chose to err on the side of type two versus type one errors. Fourth, this study had relatively weak statistical power, and covariate analysis weakened this power further. Many of the statistical trends may have been significant with increased power. This was particularly true for the analyses examining sex effects. Fifth, the present study employed an autobiographical memory based mood induction procedure, and it is unclear the extent to which alternate procedures, for example, involving standardized stimuli, would change the results. Finally, while the cutoff scores for the control group were defined in terms of average or below in schizotypal trait severity, the trait scores for this group were quite low. The “control” subjects may not be representative of all “non-schizotypal” individuals.

There is considerable work to be done before computerized facial analysis is ready for wholesale use in schizophrenia research, even in relatively healthy schizotypal samples. First, as previously noted, it is important to individually tailor lighting, participant-camera spatial relations, and other recording characteristics to optimize conditions and maximize the data available for analysis. As noted earlier, it seems unlikely that the facial analysis software, at least in its present incarnation, will be applicable to archived data. Second, it is important to consider the role of context – particularly in regards to developing a task that maximizes facial expressions. In the present study, many subjects were expressing neutral or no emotions for the majority of the interaction task (see Figure 1), so it would be important to design a task that promotes a variety of facial expressions from subjects. Third, it is important to consider sex differences. An interesting finding of the present study was that female subjects with schizotypy were equally unexpressive as male subjects with or without schizotypy (at least, in terms of neutral valence), and it is unclear whether there is something unique about how expressive deficits manifest in females with schizotypy. We were unable to evaluate sex differences in the correlational analyses due to low power (i.e., sample sizes of 17 and 11 for men and women in the schizotypal group respectively). Fourth, the trend level findings that individuals acknowledging a family history of schizophrenia showed some abnormalities of expression are worth evaluating further. Finally, it is, as yet, unclear which facial expression variables are most important for understanding expressive deficits. The software employed in this study was relatively comprehensive in scope of variables, and the most recent version of the software boasts even more variables. Practically speaking, it would be important to determine which of these variables are abnormal in schizophrenia-spectrum disorders. This type of endeavor may also hold important insights into the underlying nature of expression across the schizophrenia-spectrum.