Symptom Correlates of Static and Dynamic Facial Affect Processing in Schizophrenia: Evidence of a Double Dissociation?

Abstract

Schizophrenia patients have been shown to be compromised in their ability to recognize facial emotion. This deficit has been shown to be related to negative symptoms severity. However, to date, most studies have used static rather than dynamic depictions of faces. Nineteen patients with schizophrenia were compared with seventeen controls on 2 tasks; the first involving the discrimination of facial identity, emotion, and butterfly wings; the second testing emotion recognition using both static and dynamic stimuli. In the first task, the patients performed more poorly than controls for emotion discrimination only, confirming a specific deficit in facial emotion recognition. In the second task, patients performed more poorly in both static and dynamic facial emotion processing. An interesting pattern of associations suggestive of a possible double dissociation emerged in relation to correlations with symptom ratings: high negative symptom ratings were associated with poorer recognition of static displays of emotion, whereas high positive symptom ratings were associated with poorer recognition of dynamic displays of emotion. However, while the strength of associations between negative symptom ratings and accuracy during static and dynamic facial emotion processing was significantly different, those between positive symptom ratings and task performance were not. The results confirm a facial emotion-processing deficit in schizophrenia using more ecologically valid dynamic expressions of emotion. The pattern of findings may reflect differential patterns of cortical dysfunction associated with negative and positive symptoms of schizophrenia in the context of differential neural mechanisms for the processing of static and dynamic displays of facial emotion.

Introduction

Deficits in social cognitive functioning are now widely recognized as a core feature of schizophrenia¹ and one that relates strongly to functional outcome.^2–4 An aspect of social functioning that has been widely studied in schizophrenia has been people's ability to appropriately recognize faces and facial expressions of emotion. Problems in recognizing facial expressions of emotion are well documented in schizophrenia,^4–8 although there is substantial debate regarding the scope and generality of this deficit.^9–11 Moreover, there is no firm agreement with regards to the abnormal neuronal mechanisms underlying impaired performance in this regard. Some explanations have focused upon limbic system abnormalities,^12–14 whereas others have implicated early visual processes relating to the structural encoding of faces.^15–17 Notwithstanding these theoretical disputes, there is broad agreement that facial emotion-processing deficits in schizophrenia represent both a potential window onto abnormalities in brain function associated with the condition and a potential target for cognitive remediation strategies aimed at improving social functioning and quality of life in patients.

Anent to both these issues, a number of studies have reported associations between facial emotion recognition performance and clinical measures of symptomatology, particularly in relation to negative symptoms.^6,18–21 Negative symptoms are generally believed to be associated with abnormalities in frontal brain function^22–25 and have also been shown to be predictive of global functional outcome measures.^26–28 In relation to associations between facial emotion recognition performance and positive symptoms, we are aware of one study that reports such a finding.²⁹ This study, by Mandal and colleagues, explicitly selected schizophrenia patients on the basis of high Positive and Negative Symptom Scale (PANSS: Kay et al³⁰) ratings for either positive or negative symptoms and grouping the patients accordingly and suggests that patients with predominantly negative symptoms exhibited a generalized facial emotion-processing deficit, performing poorly in their recognition of a range of facial emotions. Patients with predominantly positive symptoms, on the other hand, were particularly impaired in their recognition of sad faces, and exhibited a “positivity bias,” consisting of a tendency to miscategorize stimuli as representing happiness. In summary, while an association between negative symptoms and facial emotion recognition is supported, to date, such an association with positive symptomatology is less clearly established because a number of studies testing for such an association have failed to provide such evidence.^6,18–21

Although facial displays of emotion are dynamic by nature, by and large, the preponderance of studies examining processing of deficits in schizophrenia has relied exclusively upon the use of static stimuli. This is potentially problematic because the ecological validity of such stimuli may be questionable.^31–33 Recent research has shown that dynamic displays of emotion are recognized more accurately than static displays,³⁴ lead to greater levels of arousal than static displays,³²and are more likely to lead to spontaneous mimicry than static displays.³³ Moreover, there is growing evidence of specialized brain systems that are preferentially activated by “biological motion” stimuli including moving faces.^35–37 While neurologically intact humans generally have little difficulty in recognizing emotions or inferring moods from static facial displays, it is possible that these may represent noncanonical stimuli that are processed by different mechanisms from their more ecologically valid dynamic counterparts. Kilts and colleagues³⁸ in a positron emission tomography study have shown that during emotional processing, static stimuli invoke a greater response than do dynamic stimuli in regions of the frontal cortex. This study also suggested that dynamic face stimuli were associated with greater temporal lobe activation than static stimuli. This latter finding is supported by 2 functional magnetic resonance imaging studies^39,40; however, neither of these studies examined the possibility of greater regional brain activation to static rather than dynamic stimuli. There is, therefore, tentative neuroimaging evidence that the processing of static and dynamic depictions of facial emotion may rely upon dissociable brain systems.

To date, we are aware of only one behavioral study³¹ involving patients with schizophrenia in which facial emotion-processing accuracy to dynamic facial emotion stimuli was assessed. This study showed that patients with schizophrenia performed more poorly than both healthy control subjects and patients with depression in their ability to recognize dynamic expressions of emotion but did not differ from either of these groups in recognizing dynamic familiar face stimuli. The authors also suggested that “schizophrenic performance is significantly poorer when dynamic rather than static facial displays are employed”; however, the credibility of this claim is strained for a number of reasons. First, it is based upon a comparison with a previous study⁴¹ using a different patient group and a static stimulus set that was not directly related to the stimuli used in the second report. Second, the claim appears to be based upon a simple observation of the different patterns of results derived from both these studies rather than through any statistical analysis. Another study⁵ has used dynamic stimuli in the context of emotion recognition; however, the stimulus materials involved vignettes in which characters performed prolonged monologues. The face and upper torso were shown, as well as verbal and vocal timbre cues, and the participants were asked to identify the dominant affective theme of the vignettes. The diverse nature of perceptual and contextual cues means that the task cannot be considered simply as a test of facial affect recognition. A more recent neuroimaging study⁴² used a number of sequential increments of morphed images presenting each increment for 500 ms and compared brain activation and behavioral performance across groups of paranoid and nonparanoid schizophrenia patients and healthy control participants. This study reported differing behavioral results and regional brain activations across experimental groups as well as differences in accuracy for emotion perception, such that paranoid patients performed more poorly than controls in their recognition of fear, while nonparanoid patients did not. The perceptual experience of the stimuli may not have sufficiently approximated the experience of observing real-time dynamic displays of emotion to view these results as categorically pertaining to dynamic stimulus displays. Moreover, there was no direct comparison of these measures in relation to differences between static and dynamic stimuli.

Therefore, to our knowledge, there have been, to date, no studies directly comparing behavioral performance in the recognition of static vs dynamic displays of emotion in patients with schizophrenia. The aim of the current study is to examine this issue for the first time and to examine the relationship between task performance and measures of symptom severity. In order to confirm the specificity of facial emotion-processing deficits in our patient sample, we also included a discrimination task comparing groups for accuracy in discriminating facial identity, facial emotion, and complex visual stimuli (butterfly wings), where the relative difficulty across these task subcomponents had been well calibrated in healthy controls. We predicted that patients would perform more poorly than controls only for facial emotion discrimination.

Methods

Participants

Nineteen people with a diagnosis of schizophrenia (n = 13) or schizoaffective disorder (n = 6) and 17 age- and gender-matched controls participated in the study. All patients were medicated (but free of benzodiazepines and anticholingerics for a minimum of 4 weeks) and had been diagnosed according to Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision criteria. All were outpatients living in the community. Demographic and clinical data are presented in table 1. There were no statistically significant differences between schizophrenia patients and schizoaffective patients in terms of clinical symptom measures. This project was approved by the ethics committees of the Alfred and Monash University, and all participants provided informed consent.

Table 1.

Demographic and Clinical Data

	Schizophrenia/SAD (n = 19)	Control (n = 17)	P
Age	41.74 (8.50)	37.00 (10.71)	.149 (F test)
Gender	15 male, 4 female	10 male, 7 female	.191 (χ2)
Years of formal education	15.59 (4.52)	18.00 (3.30)	.085 (Ftest)
PANSS positive	16.37 (4.75)	—
PANSS negative	15.37 (4.55)	—
PANSS psychopathology	28.84 (4.66)	—

Note: PANSS, Positive and Negative Syndrome Scale.

Materials

Following Enticott and colleagues,⁴³ 2 separate tasks were used to assess facial emotion processing, the first in relation to facial identity processing and the subtle discrimination of complex visual stimuli and the second in the context of static and dynamic stimuli.

Visual Discrimination Task

In the visual discrimination task, participants were presented with paired stimuli, and according to given criteria were required to indicate (via keypress) whether the stimuli were the same or different. In the first condition, a series of 32 pairs of faces were presented (obtained from the University of Stirling face database; http://pics.psych.stir.ac.uk/), and participants indicated whether they were the same person or different people (identity discrimination, reflecting general facial processing). The second condition again presented 32 pairs of faces (obtained from Matsumoto and Ekman's⁴⁴ Japanese and Caucasian Facial Expressions of Emotion and Neutral Faces), but here participants indicated whether the 2 faces showed the same or different emotions (emotion discrimination, reflecting facial emotion processing). The emotions displayed were happy, sad, surprised, angry, and neutral. The final condition presented 32 pairs of lepidoptera wings, and participants indicated whether they were from the same or different insect (lepidoptera discrimination, reflecting pattern recognition). Stimuli for all conditions were selected from a larger pool of stimuli that had been extensively pilot tested. The piloting exercise involved a 2-phase process, the first phase of which 40 participants made similarity judgments on stimulus pairings from the larger pool of stimuli. As a result of this process, stimulus subsets of same, similar, and dissimilar pairings were derived. In the second phase, 30 participants were tested for speed and accuracy in making “same vs different” decisions in relation to the stimulus pairings across the 3 tasks. Item analysis of response time and discrimination accuracy for stimuli within each task allowed the selection of subsets of stimuli within each task, such that the average discrimination difficulty is equated. The 3 resultant sets of 32 stimuli for each condition were well matched for mean accuracy (and mean response reaction time) across conditions (ie, mean accuracy reaction time [RT]: identity, 93.7% [1803 ms]; emotion, 91.3% [1940 ms]; lepidoptera, 92.4% [1730 ms]). All stimuli were shown in gray scale against a white background and presented using E-prime v1.1 (Psychology Software Tools Inc., Pittsburgh, PA; http://www.pstnet.com/prime).

Static and Dynamic Emotion Recognition Task

In the static and dynamic emotion recognition task (static/dynamic), participants were presented with a single face (developed from the NimStim⁴⁵ stimulus set) and required to indicate (via keypress) whether the face showed fear or surprise. Fear and surprise were chosen because they use similar muscle linkages and are susceptible to reciprocal miscategorization errors. The aim here was to avoid ceiling effects in healthy controls and ensure that participants did not use a heuristic single-feature approach (ie, happiness may be detectable from mouth in isolation). There were 96 trials (comprising 6 actors). For each actor, there were an equal number of static (ie, still photograph) and dynamic (ie, 1-s video clip demonstrating the morphing from a neutral expression to an emotional expression at 24 frames per second) exemplars depicting 2 real expressions of emotion (fear and surprise) and 2 morphed chimeric expressions each representing a blend of these 2 emotions, one weighted more heavily toward surprise (66% surprise) and the other weighted more heavily toward fear (66% fear). Consistent with Enticott et al,⁴³ chimeric representations of facial emotion were included in order to increase task difficulty and, thereby, avoid ceiling effects in healthy controls. Stimulus presentation was randomized across all 96 (static and dynamic) stimuli. Stimuli were shown in color against a black background and presented using E-prime v2.0 (Psychology Software Tools Inc.; http://www.pstnet.com/prime).

The number of correct responses was calculated for the each of the conditions in the discrimination (ie, identity, emotion, and lepidoptera conditions, maximum of 32 in each) and static/dynamic (ie, static and dynamic conditions, maximum of 48 correct in each) tasks.

Procedure

Participants attended a single experimental session. They first completed the discrimination task, followed immediately by the static/dynamic task.

Data Analysis

One-way ANOVA was used to compare groups for each condition of the visual discrimination task. A 2 (emotion: fear, surprise) × 2 (face: static, dynamic) × 2 (group: schizophrenia, control) mixed-model ANOVA was used to investigate performance on the static/dynamic task. Spearman correlations were used to investigate associations between facial emotion-processing and clinical presentation (ie, positive and negative symptoms). All data were analysed using SPSS v16.0.

Results

Visual Discrimination Task

Performance on the visual discrimination task is presented in figure 1. There were no group differences for either identity discrimination, F_1,34 = 0.74, P = .396, or lepidoptera discrimination, F_1,34 = 2.73, P = .108. Patients with schizophrenia, however, demonstrated poorer performance for emotion discrimination, F_1,34 = 8.14, P = .007.

Fig. 1.

Average ± SE Percentage Correct Responses for the Identity, Emotion, and Lepidoptera Discrimination Tasks for the Schizophrenia and the Control Group.

Static/Dynamic Task

Performance on the static/dynamic task is presented in figure 2. A mixed-model ANOVA revealed a main effect of emotion, F_1,34 = 25.71, P < .001, fear being more accurately recognized than surprise, but no main effect for stimulus motion (static vs dynamic), F_1,34 = 0.83, NS. There was an effect of group for overall emotion recognition performance on this task, F_1,34 = 17.88, P < .001, with patients performing worse than controls. There were no significant interaction effects.

Fig. 2.

Average ± SE Percentage Correct Responses for Static/Dynamic Presentations of Fear, Static/Dynamic Presentations of Surprise for the Schizophrenia Group and the Control Group.

Clinical Symptoms

Results are displayed in table 2. There was a significant negative moderate correlation between negative symptoms and correct responses to static facial images on the static/dynamic task, r = −.547, P = .015, indicating that higher scores on the negative symptom subscale of the PANSS were associated with a reduction in correct responses to static facial images. There was no significant relationship between negative symptoms and correct responses to dynamic facial images, r = −.156, P = .524.

Table 2.

Double Dissociation of Symptom Ratings (Positive and Negative Syndrome Scale) and Static/Dynamic Facial Emotion Identification

	# Correct Responses (Dynamic)	# Correct Responses (Static)
Positive symptoms	−0.510*	−0.360
Negative symptoms	−0.156	−0.547*

*P < .05.

Conversely, there was a significant negative moderate correlation between positive symptoms and correct responses to dynamic facial images on the static/dynamic task, r = −.510, P = .026, indicating that higher scores on the positive symptom subscale of the PANSS were associated with a reduction in the ability to correctly identify dynamic facial emotions. There was no significant relationship between positive symptoms and correct responses to static facial images, r = −.360, P = .130. Visual inspection of the data suggested that the significant correlations were not driven by outliers (tests of skewness of the relevant data distributions all yielded values of less than 1, thus offering statistical confirmation of this). A method for using Fisher r-Z transformations for comparing nonindependent correlations⁴⁶ was used to assess whether the correlations of interest were significantly different to each other. The difference between correlations between negative symptom scores and static faces and negative symptom scores and dynamic faces was significant (Z = −1.66, P < .05); however, the difference between correlations between positive symptom scores and static and dynamic face processing did not reach significance (Z = −0.67, NS).

Discussion

Nineteen patients with schizophrenia or schizoaffective disorder were compared against a control group matched for age and gender on a visual discrimination task involving facial identity, facial emotion, and lepidoptera as well as a static and dynamic emotion recognition task. In terms of performance on the visual discrimination task, patients differed from controls only in terms of emotion discrimination decisions, on which they performed more poorly. There were no differences between patients and controls for identity discrimination or lepidoptera discrimination. This pattern of findings is consistent with a specific deficit in the processing of emotional faces because performance comparable to control participants suggests that general visual discrimination of complex stimuli (lepidoptera discrimination) and general face processing (identity discrimination) were intact. The stimuli used in this task had been extensively piloted and selected from a larger set of stimuli to ensure equivalent difficulty across the 3 subcomponents. Performance of healthy controls did not differ across the subcomponents of the task. It is therefore extremely unlikely that pattern of findings here relates to variations in task difficulty; we therefore suggest that the observed pattern of results represents evidence for a differential deficit in facial emotion processing.^10,47,48

In the static/dynamic task, there was a significant main effect of group (schizophrenia patients having an overall lower level of accuracy than controls) and a significant main effect of emotion (with recognition of fear being more accurate than recognition of surprise). There were no differences in performance to static vs dynamic stimuli, and there were no significant interactions. That fear recognition was superior to surprise recognition across both groups is, perhaps, somewhat surprising given past research, suggesting that fearful faces are among the most difficult to recognize^6,43,49,50 and that patients with schizophrenia may experience the most pronounced difficulties in the recognition of fearful faces.^5,13,20,41 Because there were robust differences in category accuracy across both experimental groups, it seems likely that the effect is real, rather than being attributable to low power. It is therefore probable that this finding is attributable to the characteristics of the task and may reflect systematic differences in the differential discriminability of the 2 stimulus categories^47,48 or to some systematic bias toward fear responding in the face of a dichotomous fear/surprise choice. It may be instructive, in this regard, to repeat the experiment with a broader range of emotion categories.

It was also contrary to our expectations that there were no differences in performance to static vs dynamic stimuli.³⁴ Dynamic facial expressions are thought to hold greater ecological validity; in this respect, they might be expected to be associated with increased performance and, perhaps, even better distinguish performance in clinical and control groups. The current study, however, does not support either of these suggestions. While this may be attributable to low power, it is possible that dynamic facial expressions do not add significantly to neurobehavioral measures of facial emotion processing, although such a claim deserves much further investigation. One possible explanation for this unexpected finding may lie in the choice of stimulus categories used: fear and surprise faces are among the more difficult emotion categories to recognize and disambiguate from each other⁶ and rely on highly similar muscle groupings.⁵⁰ The choice of these (hard to disambiguate) stimulus categories was based upon a desire to avoid ceiling effect in healthy controls. It may be that a further consequence of this choice, however, was to obviate the potential usefulness of dynamic information for aiding in the discrimination of the stimuli. Again, the inclusion of a broader range of emotion categories may help elucidate this issue.

Perhaps the most intriguing findings were the significant associations between performance on the static and dynamic components of the behavioral task and (respectively) negative and positive symptom ratings among patients that are suggestive of the possibility of a double dissociation. That is, negative (but not positive) symptoms ratings were negatively correlated with static emotion-processing accuracy, while positive (but not negative) symptoms were negatively correlated with dynamic emotion-processing performance. The size of the (significant) correlation between negative symptom scores and static faces was significantly greater than the size of the (nonsignificant) correlation between negative symptoms. However, in the case of the correlation strengths relating to positive symptoms, the difference in sizes did not reach significance. One cannot rule out, therefore, with certainty, the possibility that the absence of a significant finding in relation to an association between positive symptoms and static emotion recognition (r = −.36) is not attributable to a lack of power: by the same token, one cannot rule out the possibility that the lack of significant differences in the size of the 2 correlations in relation to positive symptoms might equally reflect insufficient power. Therefore, while these results are not in themselves sufficient to conclude a double dissociation, they do at the very least show evidence of an extremely interesting dissociation and point to the possibility of a double dissociation. In either case, the finding is novel and may have important implications for our understanding of the pathophysiology of schizophrenia in relation to facial emotion-processing deficits and for the interpretative framework with which we approach the existing body of empirical data which focuses solely upon the use of static emotion stimuli. It is not immediately apparent what is causing these novel findings, and in the absence of neuroimaging data to support and extend these behavioral findings, it is not possible to draw firm conclusions regarding the brain mechanisms that may be involved. We propose that the observed pattern of findings may reflect differential patterns of cortical dysfunction associated with negative and positive schizophrenia patients in the context of distinct neural substrates for static and dynamic emotion processing. For example, there is evidence that static face processing is associated with increased activation of frontal cortical regions³⁸ as compared with dynamic emotion processing; frontal dysfunction has also been attributed to negative symptoms including alogia, anhedonia, and avolition.^22–25 By contrast, temporoparietal regions have been implicated in dynamic stimulus processing (ie, biological motion^36,39,40) and with positive symptoms such as auditory hallucinations.⁵²

This study is limited by a relatively small sample and the study of a limited number of emotions (particularly in the static/dynamic task) but has demonstrated, for the first time, that emotion processing of static and dynamic faces may be associated with different aspects of schizophrenia. In relation to the restricted range of emotions used in the current study, it may be noted that previous work suggests that fear and surprise are among the more difficult emotions to recognize/discriminate (for both schizophrenia patients⁶ and healthy controls⁴⁸). Where differences between schizophrenia patients and controls have been reported in relation to particular emotion categories, the effect is likely to be driven largely by the psychometric properties of the stimulus materials rather than due to a specific deficit in the recognition of a particular emotion.^6,10,48 This being the case, the most parsimonious account of facial emotion-processing deficits in schizophrenia is that of a general deficit, the effects of which are more or less apparent as a function of stimulus category difficulty: what is true for fear and surprise is likely to be true (to a greater or lesser extent, depending on psychometric properties of the stimuli) for other emotion categories. However, because previous work has been based exclusively on static stimuli, it is not a priori clear that this should necessarily hold true for dynamic stimuli. It will therefore be important to replicate the current findings in the context of a broader range of emotions. Another potential limitation of the study rests in the use of posed vs naturally occurring facial expressions of emotion. One study has suggested that patients with paranoid schizophrenia may be impaired only in their ability to recognize posed expressions of emotion.⁵³ At present, it is not clear how seriously this finding must be taken because it was based on a small number of participants and to date has not been replicated; however, it suggests that some degree of caution is necessary in the interpretation of data from studies involving posed emotional expression. In relation to the current study, it will also be important to use functional neuroimaging to further investigate these neurobehavioral findings in schizophrenia and to examine the relationship with specific aspects of positive and negative symptomatology (which may provide a better indication of the neurological basis of our correlational findings). This will further advance our understanding of the pathophysiology of schizophrenia, particularly in relation to the neural basis of social cognitive deficits and positive and negative symptoms.

Funding

Dr. Patrick Johnston was supported by an NHMRC Clinical Training Fellowship. Prof. Paul Fitzgerald was supported by an NHMRC Practitioner Fellowship.