Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2010 Jun 1.
Published in final edited form as: Schizophr Res. 2008 Sep 14;105(1-3):30–39. doi: 10.1016/j.schres.2008.05.030

Dynamic evoked facial expressions of emotions in schizophrenia

Christian G Kohler a,*, Elizabeth A Martin a, Marina Milonova a, Peng Wang b, Ragini Verma b, Colleen M Brensinger a, Warren Bilker a, Raquel E Gur a, Ruben C Gur a
PMCID: PMC2879330  NIHMSID: NIHMS200286  PMID: 18793827

Abstract

Objective

Impaired facial expressions of emotions have been described as characteristic symptoms of schizophrenia. Previous investigations of dynamic facial expressions have reported on global assessment of positive and negative emotion expressions. In this study, we examined facial expression differences based on duration and frequencies of emotion expressions.

Methods

12 persons with stable schizophrenia and matched healthy controls underwent a standardized procedure for evoked facial expressions of five universal emotions, including happy, sad, anger, fear, and disgust expressions. Subjects completed self-ratings of their emotion experience. Reliable raters coded evoked facial expressions according to the Facial Expression Coding System. For each emotion, facial expressions were coded as target, non-target or neutral expressions. Logistic regression analyses examined group differences in duration and frequencies of facial expressions.

Results

Comparing overall duration of and frequencies of emotion expressions revealed affective flattening and inappropriate affect in patients, as evidenced by neutral and non-target expressions. Separated by emotion, impaired emotion expression was found in happy, sad and anger expression, but not for fear and disgust in which expressions were not well recognized.

Conclusion

In matched groups of participants, we found evidence for altered expressions in schizophrenia but equal subjective experience. Both affective flattening and inappropriate affect comprise abnormal facial expressions but may differ with respect to interpersonal communication and engagement. Future directions may include automated measurement, remediation of expressions and early detection of schizophrenia.

Keywords: Dynamic emotion expression, Schizophrenia, Facial Expression Coding System, Affective flattening, Inappropriate affect

1. Introduction

Facial expressions are shared in humans and animals, and are central for communication both within and across species (Darwin, 1872). Abnormal and muted expressions of emotions have been described as characteristic symptoms of schizophrenia (Andreasen, 1984a; Bleuler, 1911), may precede the onset of illness by many years (Walker et al., 1993) and have been reported as predictors for developing a psychotic disorder (Mason et al., 2004). Affective flattening and other negative symptoms are present at onset of illness (Gelber et al., 2004; Shtasel et al., 1992) more common in males, increase with illness duration (Shtasel et al., 1992) and appear distinct from depression (Kohler et al., 1998). In contrast to positive symptoms of schizophrenia, negative symptoms may not respond as well to antipsychotics and have been linked to impairment in psychosocial functioning (Edwards et al., 1999; Ho et al., 1998).

Whereas there are widely used and validated instruments that measure and parse aspects of cognitive dysfunction and its neurobiology in schizophrenia, clinical assessments of affective flattening and other negative symptoms have been limited to observer based rating scales. The ability to quantify emotional expression, especially in the face, has been enhanced by work aimed at measuring unique features of universal emotions. Six universal emotions are recognized across cultures in facial expressions — happiness, sadness, anger, fear, disgust and surprise (Eibl-Eibesfeldt, 1970; Ekman and Friesen, 1975; Izard, 1994). Based on facial muscle movement, Ekman and Friesen (1978) developed the Facial Action Coding System (FACS), which identifies discrete facial muscle movements, called Action Units (AUs). FACS has been simplified and adapted for clinical research, producing the Emotion FACS (Friesen, 1986), which identifies AUs associated with the predicted expression of the particular emotion, and the Facial Expression Coding System (FACES: Kring et al., 1993; Kring and Sloan, 2007), which rates overall dynamic facial changes, according to number of expressions, intensity and duration.

Clinical rating scales are usually obtained in nonstandardized settings and these ratings attempt to provide an overall measure of facial expressivity, while FACS and FACES dependent measures focus on local and global indicators of facial expressions within standardized settings. Standardized measurements of emotion expressions in schizophrenia have included FACS derived measures without analysis of specific AUs (Berenbaum, 1992; Berenbaum and Oltmanns, 1992; Gaebel and Wolwer, 2004; Tremeau et al., 2005) and the FACES procedure applied to videos of emotional expressions of participants. In the latter group of studies, emotion expressions were obtained during emotional evocative film clips (Kring et al., 1993; Kring and Neale, 1996) and during simulations of social situations (Aghevli et al., 2003). Analysis of the videos showed persons with schizophrenia to present emotionally less expressive on measures of duration, frequency and intensity of emotion expressions. Ratings of static facial expressions allow for determination regarding differences in individual muscle movements, but require extensive training on synchronizing the different aspects of image acquisition. The acquisition of static expressions is highly dependent on capturing the representative facial expression and images may be obtained that miss the intended emotion expression. Videotaped acquisition offers the advantage of capturing duration and frequency of emotion expressions. Due to the complexity and length of video data sets, dynamic facial expressions ratings have been limited to more global assessment of positive and negative emotion expressions, rather than changes in specific face regions.

Most studies have supported affective flattening in general, rather than inappropriate affect, and more selective impairment in happy (Gottheil et al., 1976), sad (Putnam and Kring, 2007), angry (Gottheil et al., 1970; Schwartz et al., 2006) and disgusted (Schwartz et al., 2006) expressions. While affective flattening is considered characteristic of schizophrenia, comparisons with psychiatric (Berenbaum, 1992; Gaebel and Wolwer, 2004; Schneider et al., 1990; Tremeau et al., 2005) and medical (Martin et al., 1990; Steimer-Krause et al., 1990) control groups have raised questions regarding specificity.

Antipsychotics, particularly first generation, are associated with extrapyramidal symptoms, but their influence on emotion expression remains unclear. Some studies indicated an adverse effect of medications on facial expression (Gaebel and Wolwer, 2004; Schneider et al., 1992), while others (Earnst et al., 1996; Putnam and Kring, 2007; Tremeau et al., 2005) examined patients both on and off antipsychotics and found no clear effect on expressivity.

Previously, we investigated AUs in high intensity evoked expressions of universal emotions expressed by actors and determined which AUs were essential for accurate recognition (Kohler et al., 2004). Based on the experience with actors, matched groups of persons with stable schizophrenia and healthy controls underwent a standardized procedure of eliciting posed and evoked facial expressions of five universal emotions, while static and dynamic images were obtained. For static images, we applied FACS to examine facial changes based on the presence of AUs for each emotion, intensity and condition (Kohler et al., in press). While overall expression patterns were similar, differences in presence of AUs classified as frequent and infrequent for each emotion support clinical observations of flattened and inappropriate affect in schizophrenia. Specific differences in AUs involved the Duchenne smile for happy expressions and furrowed brows in all negative emotions.

In an effort to further characterize impaired facial affect in schizophrenia, we examined dynamic facial expressions of evoked emotions using an adapted FACES procedure. We compared control and patient groups on duration and frequencies of facial expressions, determined to represent target, non-target, and neutral expressions, in response to a verbal recount of self-reported emotional vignettes. We anticipated that persons with schizophrenia display flattened affect, as evidenced by duration of target and neutral facial expressions, and inappropriate affect, as evidenced by frequencies and duration of non-target expressions.

2. Methods

2.1. Subjects

There were 12 persons (mean age=31.50±SD=7.74, range=21–42) with the DSM-IV diagnosis of schizophrenia based on the Diagnostic Interview for Genetic Studies (Nurnberger et al., 1994) and 12 healthy controls, case matched to patients according to gender, ethnicity, age and parental education. No participant in this study displayed any facial hair, such as beard or mustache, which may have obscured facial movements. Participants were recruited from the Schizophrenia Center at the University of Pennsylvania, according to the following general exclusion criteria: History of substance related disorders (DSM-IVand laboratory data including toxicology) within the past 6 months; mental retardation defined as a full scale IQ<70; not proficient in English; history of any neurologic event or disease affecting brain function; medical diseases that may affect brain function or interfere with participation: hypertension (BP>140/90), cardiac disease, diabetes mellitus, endocrine disorders, renal disease, chronic obstructive pulmonary disease; corrected vision<20/30; pregnancy (determined by a serum test). In addition, healthy controls were free of Axes I and II disorders and Axis I disorders in first-degree relatives.

Patients were screened and excluded for self-reported and observed depression, using standard rating scales. All patients lived independently or with family, none had been hospitalized within 6 months prior and six patients –compared to all controls – worked at least 20 h per week or were students. Patients had assessment for positive (Andreasen, 1984b) (mean total score±SD=5.8+10.7) and negative (Andreasen, 1984a) (mean total score+ SD=29.0±18.0) symptoms, including affective flattening (mean score+SD=2.3±1.0) at the time of testing. Symptom assessments were performed by trained raters meeting inter-rater reliability (icc>.80). All patients were treated with standard dosages of second generation-antipsychotics (n=12) and augmenting dosages of first generation-antipsychotics (n=2), specifically long acting haloperidol, without increase in antipsychotic medication for 3 months. Antipsychotic dosages were converted to estimated olanzapine-equivalents (mean dosage ± SD=15.5±2.3) and chlorpromazine-equivalents (mean dosage±SD=312.5±187). None of the participants exhibited clinical evidence of tardive dyskinesia or acute extrapyramidal symptoms. After complete description of the study to the subjects, written informed consent was obtained.

2.2. Facial emotion expression

The complete procedure for expression has been described in more detail previously in actors (Gur et al., 2002), control and patient groups (Kohler et al., in press), and included a standardized procedure for expression of five universal emotions. The order of emotions expressed remained fixed: Happiness, Anger, Fear, Disgust and Sadness. A priori decision was made to not include surprise, since surprise can be conceptualized as the abrupt onset of any other universal emotion. The sequence of emotions was based on the assumption that happiness as the only positive emotion is most easily achieved in isolation, i.e. being first, and sadness was positioned last, since its effect in the evoked condition may last longer. Prior to filming of evoked expressions, participants identified biographical situations for each universal emotion where the target emotion was experienced in mild, moderate and high intensities. Situations were reviewed for suitability regarding evoking the target emotion without interference by another emotion and the situations were summarized as vignettes progressing from mild to medium to high intensity experiences with identification of time points for the three intensities. Collection of the vignettes ranged from 10 to 20 min.

Participants were seated in a brightly lit room and instructed to remain within direct view of the digital video camera. Research personnel, including the instructor and videographer, were located beyond the camera in an unlit area. Autobiographical vignettes were recounted to participants in a narrative manner using exact wording derived from vignettes, and participants were instructed to concentrate on the emotional experience at three intensities and the corresponding facial expression. Participants were instructed to avoid speech, as it would interfere with facial expression of emotions. Following each evoked emotion the subject was prompted to return to a neutral or non-emotional state. Based on observed facial expressions and subject’s report, acquisition progressed to the next target emotion. Time interval between emotion expressions of each valence lasted between 30–60 s. Duration of acquisition of evoked expressions was between 15 and 25 min. Following acquisition of emotion expressions, participants rated their subjective experience of each target emotion at the three intensities on an 11-point self-rating scale (0=no emotion, 10=extreme intensity).

2.3. Video acquisition

Video acquisition was performed via a high performance digital 12-bit CCD camera system (COOKE PixelFly). Each frame in the video was saved as an individual image in a bitmap, or BMP, format, to preserve the original data. The video acquired from each of the 24 participants was divided into 5 separate segments according emotion vignette, transferred to mpeg format, randomized, and burned onto a compact disc. The video segments were viewed in RealOne Player (version 10.5). Video segments ranged from 19 to 254 s (schizophrenia group: mean= 106.9 s±50.7, control group: mean=83.5 s±32.6) and the videos amongst groups differed in length (z=2.68, p =0.009).

2.4. Coding facial expressions

The methodology for coding facial expressions was based on the Facial Expression Coding System (FACES; Kring et al., 1993). In FACES, facial expressions in video segments are coded for frequency, duration, valence (positive or negative), and intensity (low, medium, high, very high). In an adapted version of FACES, two trained raters coded the occurrences and durations of facial expressions in each video segment. Raters were blind to participant status and intended emotion. Expressions were coded if a neutral expression changed to an emotional expression and changed back to a neutral expression (1 expression coded) or to a different emotional expression (2 expressions coded). Facial changes independent of emotion expression (e.g. yawning, licking lips, talking, head nodding, head tilt, diverted eye gaze) were not counted as an emotion expression. For every expression, the perceived emotion (happy, sad, anger, fear, disgust) and duration (in seconds) were coded. For analyses of emotion expressions and based on agreement between the two raters, video segments were coded for total duration and frequencies of target expressions that were consistent with the intended emotion, non-target expressions that were different from the intended emotion, and neutral expressions. Within each emotion, we examined correlations between clinical assessments of affective flattening and inappropriate affect with presence of target and non-target expressions.

2.5. Data analysis

For two trained FACES raters, we calculated Pearson correlation coefficients for the number of target expressions (rho=0.887) and the number of non-target expressions (rho=0.940). In addition, we calculated the mean number in agreement across all seconds as 82% and 89% for the control and patient group respectively. Agreement between the raters for determination of target vs. non-target vs. neutral expressions, collapsing over all seconds from all videos, was almost perfect (kappa=0.823). Based on the high degree of rater agreement, expression duration ratings without agreement, which for all emotions combined were 17% in controls and 10% in patients, are not included in Fig. 2. The number of target and non-target expressions was summarized for participant groups using means, standard deviations, medians, and ranges. Poisson regression analyses, as implemented in SAS Proc Genmod, version 9.1 (SAS Institute, Cary, NC), were used to test for differences in the number of target and non-target expressions per video by patient status. The GEE models used the video length (in seconds) as an offset, which converts the model to examine the rate of target and non-target expressions, and adjusted for video length as well. To examine the time spent in target, non-target, and neutral expressions, each second of each video was classified as being a target expression, non-target expression, or neutral. Logistic regression was used to model the odds of target vs. non-target expressions comparing groups. The logistic regression was fit by using generalized estimating equations (GEE) to account for the non-independence or clustering of the multiple seconds for each video. The comparisons are expressed in terms of odds ratios. To address the relationship between antipsychotic medication and facial expression ratings in the patient group we performed GEE logistic regression models looking at percent time spent in target, non-target, and neutral expressions by current antipsychotic equivalent dose. In addition, we performed Spearman correlation coefficients between current antipsychotic dose vs. negative and positive clinical ratings.

Fig. 2.

Fig. 2

Open in a new tab

Emotion expression ratings.

3. Results

Participants underwent the procedure for evoked expressions of emotions without signs of undue emotional stress or difficulties. The two groups rated similar subjective experience of emotions for each emotion (see Fig. 1).

Fig. 1.

Fig. 1

Open in a new tab

Emotion experience intensity.

The following results are based on comparisons between the control and schizophrenia groups. For all emotions combined, controls spent more time in target (38% vs. 21%) (OR=3.15; z=3.83, p <.001) and less time in neutral expressions (32% vs. 51%) (OR=.40; z=−3.79, p <.001) per video. Groups did not differ in amount of time spent in non-target (13% vs. 18%) expressions. Groups did not differ in mean frequency of target expressions per video (1.47, range: 0–6 vs. 1.92, range: 0–7), but differed in mean frequency of non-target expressions (1.11, range: 0–6 vs. 2.69, range: 0–19) (x-sq=9.31, p=.002). Results for individual emotions are presented based on intended emotion (see Figs. 2 and 3).

Fig. 3.

Fig. 3

Open in a new tab

Emotion expression frequencies.

3.1. Happy

Controls spent more time in target (65% vs. 36%) (OR=3.38; x-sq=3.94, p=.047) and less time in non-target (1% vs. 8%) (OR=.13; x-sq=6.25, p=.012) expressions. Groups did not differ in amount of time spent in neutral (25% vs. 44%) expressions. Groups did not differ in mean frequency of target expressions (2.00, range: 1–6 vs. 3.11, range: 1–7) or mean frequency of non-target (0.73, range: 0–4 vs. 1.67, range: 0–7) expressions.

3.2. Sad

Controls spent more time in target (55% vs. 23%) (OR=4.22; x-sq=6.23, p=.013) and less time in neutral (11% vs. 48%) (OR=.14; x-sq=19.95, p <.0001) expressions. Groups did not differ in the amount of time spent in non-target (6% vs. 19%) expressions. Groups did not differ in mean frequency of target (1.73, range: 0–4 vs. 2.00, range: 0–7), but differed in mean frequency of non-target (0.82, range: 0–6 vs. 3.58, range: 0–19) (OR=.32; x-sq=9.31, p=.002) expressions.

3.3. Anger

Controls spent more time in target (51% vs. 16%) (OR=5.18; x-sq=9.35, p=.002) and less time in neutral (27% vs. 57%) (OR=.28; x-sq=7.84, p=.005) expressions. Groups did not differ in the amount of time spent in non-target (9% vs. 19%) expressions. Groups did not differ in mean frequency of target expressions (1.18, range: 0–3 vs. 1.80, range: 0–6), but differed in mean frequency of non-target (1.09, range: 0–3 vs. 3.60, range: 0–10) (OR=.49; x-sq=4.29, p=.038) expressions.

3.4. Fear

No significant differences were found in duration of target (7% vs. 16%), neutral (57% vs. 59%) and non-target (24% vs. 12%) expressions. Groups did not differ in mean frequency of target (1.36, range: 0–5 vs. 1.90, range: 0–4) or mean frequency of non-target (1.27, range: 0–6 vs. 1.80, range: 0–5) expressions.

3.5. Disgust

No significant differences were found in duration of target (24% vs. 9%), neutral (32% vs. 43%) and non-target (23% vs. 38%) expressions. Groups did not differ in mean frequency of target (1.09, range: 0–2 vs. 0.90, range: 0–4) or mean frequency of non-target (1.64, range: 0–4 vs. 2.50, range: 0–11) expressions.

3.6. Clinical ratings and antipsychotic effects

Within the schizophrenia group, ratings for affective flattening showed negative correlation with overall duration of target expressions (OR=.44, z=2.97, p=.003). Ratings for inappropriate affect did not correlate significantly with overall duration of non-target expressions.

There were no statistically significant associations between current antipsychotic dose and percent time spent in target, non-target, or neutral emotions. In addition, we performed Spearman correlation coefficients between antipsychotic dose vs. negative and positive clinical ratings. Current antipsychotic dose was significantly associated with total negative (rho=.59, p=.043) and positive (rho=.80, p=.002) symptoms, overall alogia (rho=.68, p=.014), and overall delusions (rho=.70, p=.011) but not with any specific measures of flat affect.

4. Discussion

Impaired facial expression of affect is viewed as a characteristic symptom in schizophrenia that predates the onset of psychosis. Observer-based rating scales are widely applied to quantify facial expressions in clinical situations, while recent procedures that investigated regional and global face changes in standardized settings reported face changes, related to emotion and condition, face regions and disorder specificity. Based on extensive work with actors (Gur et al., 2002) we developed a standardized procedure for static and dynamic acquisition of facial expressions. Previously, we applied FACS to static facial expressions and determined regional face differences, as measured by AUs, in expressions of universal emotions in actors (Kohler et al., 2004) in support of previous reports (Ekman and Friesen, 1978; Gosselin et al., 1997). Extending this methodology to stable schizophrenia and control subjects (Kohler et al., in press), differences in presence of frequent and infrequent AUs underscored clinical observations of flattened and inappropriate affect in schizophrenia. More specific differences involved the Duchenne smile for happy expressions and furrowed brows in all negative emotions. Applying FACS to static facial expressions allows for assessment of small and regional face changes at the expense of failing to capture dynamic face changes associated with each emotion expression. While assessment of dynamic expressions yields information on face changes over time and is capable of capturing even brief emotion expressions, this occurs at the expense of examining global rather than regional face changes and assigning valence from a predetermined choice of options.

In the present study, involving young and stable schizophrenia outpatients and case matched healthy controls, we applied a standardized procedure to elicit evoked facial expressions of five universal emotions and measured dynamic facial expressions with a FACES-based methodology. Our study differed from previous investigations in the selection of patients, and acquisition and measurement of facial expressions. Patients were selected for clinical stability and shorter duration of illness, and the control group was case matched according to relevant socio-demographic criteria. Facial expressions were obtained in a standardized setting that focused on evoked emotion expressions and attempted to eliminate interference of verbal expressions. In addition, we adapted the FACES paradigm to include duration and frequency of emotion expressions, accounted for length of the experimental procedure, and included determination of emotional valence beyond positive or negative emotion.

Comparisons in duration and frequencies of expressions underscored evidence from clinical ratings and experimental studies that document altered facial emotion expressions in persons with stable schizophrenia. Examining overall duration of emotion expressions, patients spent less time in target expression and more time in neutral expressions, consistent with affective flattening in schizophrenia. Overall frequencies of target emotion were similar, yet patients displayed more non-target emotions, consistent with inappropriate affect. While total duration of non-target expressions amounted to less than one-fifth of expressions in schizophrenia, increased frequency of these expressions indicated the presence of more frequent but fleeting inappropriate affect. Our findings on subjective emotion experiences are in line with previous studies employing emotion arousing film clips (Berenbaum and Oltmanns, 1992; Earnst et al., 1996; Kring et al., 1999; Kring et al., 1993), emotional experiences of participants (Berenbaum and Oltmanns, 1992; Kring et al., 1993) and social interactions (Aghevlie et al., 2003) that have established the concept that flat affect in schizophrenia does not indicate diminished emotional experience.

When broken down by emotion, duration of target expression in happy, sad, and anger expressions and duration of neutral expressions in sad and anger expressions further confirmed our predictions of affective flattening in individual emotions. In these emotions controls spent the majority of time expressing target emotions, however in patients there was a striking reversal in expression patterns. Duration of non-target expressions differed for happy expressions, where it was almost absent in controls. Frequencies of non-target emotion expressions were increased in sad and anger expressions, where patients displayed more non-target than target expressions.

Of interest is that no specific findings were found for both duration and frequency of expressions in fear and disgust expressions. In these emotions, expression patterns were similar across groups showing brief duration of target expressions. In fear this resulted in longer duration of neutral expressions both groups and while target expressions amounted to less than 20% of total duration, this was more pronounced in controls, who failed to emote fear while expressing relatively more non-target emotions compared to patients. Putnam and Kring (2007) recently reported fear as a difficult emotion to express and reasons for muted fear expressions may have an evolutionary basis, in that fearful expressions signal weakness. Another possible explanation, although not clearly supported by experience ratings, is that the experimental and safe testing setting prevented experience and, consequently, expression of fear. Thus, the general lack of expressiveness for fear in controls may have obscured a finding for schizophrenia, where processing (Gur et al., 2007; Holt et al., 2005; Kohler et al., 2003; Van’t Wout et al., 2007) and experience (Suslow et al., 2003) of fear stimuli may have a more specific role in the pathophysiology and phenomenology of the illness. Lack of significant results for disgust may be based on disgust expressions representing combinations of other emotions, which will lead to increased identification as non-target expressions.

Limitations of the present study relate to sample size and methodology of capturing and analyzing facial expressions. The small sample size is mediated by careful matching of patient and control groups, and complex analysis that accounted for emotion expression at each second. We chose controls as comparisons, since actors or coached controls would magnify the difference found amongst groups. Given the limited power of our sample size and clinical stability of patients, the major concern was that the study would fail to detect differences and the limited statistical power may have affected findings for fear and disgust. Applying FACES to larger groups with wider range of clinical symptomatology is needed to replicate our findings and investigate the effect of illness acuity and symptom clusters on facial expressions. The sequence of expressed emotions remained fixed and we decided against a randomized sequence of emotions where happy expressions could be affected by negative emotions and sad experiences could interfere with other expressions. Participants were given time to return to baseline emotional state, but our approach may have produced an order effect. Previous procedures for evoked expressions were not uniform and included spontaneous expressions within dyadic interactions, expressions associated with emotional film clips or expressions associated with describing autobiographical emotional experiences. Limitations surround each of these methods and our methodology differed in that autobiographical vignettes were recounted to participants with the intention to maximize emotion expressions without verbal interference. Lastly, comparisons between groups on intensity of expressions may have produced interesting findings, but the small sample size precluded meaningful comparisons.

In summary, employing a standardized procedure for expression of evoked or felt emotions, we were able to confirm that persons with schizophrenia, while showing equal subjective experience, display altered facial expressions that extend to individual emotions. While both affective flattening and inappropriate affect comprise abnormal facial expressions, we put forth that consequences differ with respect to interpersonal communication and engagement. More specifically, affective flattening may result in lack of recognition of the expressed emotion and misinterpretation as neutral or no emotion, inappropriate affect will result in misinterpretation of the emotional valence as another emotion.

Future directions involving measurement of facial muscle movements lie in further development of automated measurement of facial expressions and therapeutic application of measuring facial expressions. Over the past 15 years, automated versions of FACS have been developed that extract action units from facial video sequences in healthy persons (Bartlett et al., 1999; Cohn et al., 1999; Essa and Pentland, 1997). Based on MRI morphometry, our group (Verma et al., 2005) has developed a computerized algorithm of facial expression analysis that has been validated in comparison with the FACES procedure in healthy controls (Wang et al., 2008) and in comparison with clinical ratings of flat affect in patients (Alvino et al., 2007). Computerized methods will be able to provide a measure of the subtle changes in facial expression and may replace visual inspection as a more reliable and sensitive measure. Practical applications of more detailed measurement hold promise to extend recent attempts at instruction (Schwartz et al., 2006) and remediation (Frommann et al., 2003) of emotion expression. Lastly, affective flattening can be present well before clinical onset of schizophrenia (Walker et al., 1993) and will increase during the prodromal phase (Malla et al., 2002). In conjunction with other candidates for endophenotypes, facial expressions of emotions in persons with vulnerability to schizophrenia may inform us about risk of illness and assist in future efforts at prevention or postponement of onset of schizophrenia.

Acknowledgments

This work was supported by NIMH MH-01839, MH-60722 and the National Alliance for Research on Schizophrenia and Depression. The funding sources had no role in the design of study, interpretation of data, manuscript preparation and submission.

Footnotes

Contributors

Drs. Kohler, R.E. Gur and R.C. Gur designed the study and Dr. Kohler prepared the manuscript. Drs. Wang and Verma, E. Martin and M. Milonova were responsible for data acquisition, and C. Brensinger and Dr. Bilker were responsible for data analysis.

Conflict of interest

There are no potential conflicts of interest to report for Drs. Kohler, Bilker, Wang, Verma, R.C. Gur and R.E. Gur; C. Brensinger, E. Martin and M. Milonova.

References

  1. Aghevli MA, Blanchard JJ, Horan WP. The expression and experience of emotion in schizophrenia: a study of social interactions. Psychiatry Res. 2003;119 (3):261–270. doi: 10.1016/s0165-1781(03)00133-1. [DOI] [PubMed] [Google Scholar]
  2. Alvino C, Kohler CG, Barrett F, Davatzikos C, Gur RE, Gur RC, Verma R. Classification of facial expressions for measuring affect deficit in schizophrenia. J Neurosci Methods. 2007;163:350–361. doi: 10.1016/j.jneumeth.2007.03.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Andreasen NC. The Scale for the Assessment of Negative Symptoms (SANS), Trans. ed. Vol. The University of Iowa; Iowa City: 1984a. [Google Scholar]
  4. Andreasen NC. The Scale for the Assessment of Positive Symptoms (SAPS), Trans. ed. Vol. The University of Iowa; Iowa City: 1984b. [Google Scholar]
  5. Bartlett MS, Hager JC, Ekman P, Sejnowski TJ. Measuring facial expressions by computer image analysis. Psychophysiology. 1999;36 (2):253–263. doi: 10.1017/s0048577299971664. [DOI] [PubMed] [Google Scholar]
  6. Berenbaum H. Posed facial expressions of emotion in schizophrenia and depression. Psychol Med. 1992;22 (4):929–937. doi: 10.1017/s0033291700038502. [DOI] [PubMed] [Google Scholar]
  7. Berenbaum H, Oltmanns TF. Emotional experience and expression in schizophrenia and depression. J Abnorm Psychol. 1992;101 (1):37–44. doi: 10.1037//0021-843x.101.1.37. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Bleuler E. Dementia Praeox oder die Gruppe der Schizophrenien, Trans. ed. Vol. 1911. [DOI] [PubMed] [Google Scholar]
  9. Cohn JF, Zlochower AJ, Lien J, Kanade T. Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding. Psychophysiology. 1999;36 (1):35–43. doi: 10.1017/s0048577299971184. [DOI] [PubMed] [Google Scholar]
  10. Darwin C. The Expression of Emotions in Man and Animals, Trans. ed. Vol. Philosophical Library; New York: 1872. [Google Scholar]
  11. Earnst KS, Kring AM, Kadar MA, Salem JE, Shepard DA, Loosen PT. Facial expression in schizophrenia. Biol Psychiatry. 1996;40 (6):556–558. doi: 10.1016/0006-3223(96)00171-0. [DOI] [PubMed] [Google Scholar]
  12. Edwards J, McGorry PD, Waddell FM, Harrigan SM. Enduring negative symptoms in first-episode psychosis: comparison of six methods using follow-up data. Schizophr Res. 1999;40 (2):147–158. doi: 10.1016/s0920-9964(99)00043-2. [DOI] [PubMed] [Google Scholar]
  13. Eibl-Eibesfeldt I. Ethology, The Biology of Behavior, Trans. ed. Vol. Holt, Rinehart & Winston; New York: 1970. [Google Scholar]
  14. Ekman P, Friesen WV. Unmasking the Face, Trans. ed. Vol. Prentice-Hall; Englewood Cliffs: 1975. [Google Scholar]
  15. Ekman P, Friesen WV. Manual of the Facial Action Coding System (FACS), Trans. ed. Vol. Consulting Psychologists Press; Palo Alto: 1978. [Google Scholar]
  16. Essa IA, Pentland AP. Coding, analysis, interpretation and recognition of facial expressions. IEEE Trans Pattern Anal Mach Intell. 1997;19 (7):757–763. [Google Scholar]
  17. Friesen WV. Recent developments in FACS-EMFACS. Facial Meas Newsl. 1986;1:1–2. [Google Scholar]
  18. Frommann N, Streit M, Wolwer W. Remediation of facial affect recognition impairments in patients with schizophrenia: a new training program. Psychiatry Res. 2003;117 (3):281–284. doi: 10.1016/s0165-1781(03)00039-8. [DOI] [PubMed] [Google Scholar]
  19. Gaebel W, Wolwer W. Facial expressivity in the course of schizophrenia and depression. Eur Arch Psychiatry Clin Neurosci. 2004;254 (5):335–342. doi: 10.1007/s00406-004-0510-5. [DOI] [PubMed] [Google Scholar]
  20. Gelber EI, Kohler CG, Bilker WB, Gur RC, Brensinger C, Siegel SJ, Gur RE. Symptom and demographic profiles in first-episode schizophrenia. Schizophr Res. 2004;67 (2–3):185–194. doi: 10.1016/S0920-9964(03)00083-5. [DOI] [PubMed] [Google Scholar]
  21. Gosselin P, Kirouac G, Doré FY. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS) Oxford University Press; New York: 1997. Components and recognition of facial expression in the communication of emotion by actors. [Google Scholar]
  22. Gottheil E, Paredes A, Exline RV, Winkelmayer R. Communication of affect in schizophrenia. Arch Gen Psychiatry. 1970;22 (5):439–444. doi: 10.1001/archpsyc.1970.01740290055007. [DOI] [PubMed] [Google Scholar]
  23. Gottheil E, Thornton CC, Exline RV. Appropriate and background affect in facial displays of emotion. Comparison of normal and schizophrenic males. Arch Gen Psychiatry. 1976;33 (5):565–568. doi: 10.1001/archpsyc.1976.01770050033004. [DOI] [PubMed] [Google Scholar]
  24. Gur RC, Sara R, Hagendoorn M, Marom O, Hughett P, Macy L, Turner T, Bajcsy R, Posner A, Gur RE. A method for obtaining 3-dimensional facial expressions and its standardization for use in neurocognitive studies. J Neurosci Methods. 2002;115 (2):137–143. doi: 10.1016/s0165-0270(02)00006-7. [DOI] [PubMed] [Google Scholar]
  25. Gur RE, Nimgaonkar VL, Almasy L, Calkins ME, Ragland JD, Pogue-Geile MF, Kanes S, Blangero J, Gur RC. Neurocognitive endophenotypes in a multiplex multigenerational family study of schizophrenia. Am J Psychiatry. 2007;(5):813–819. doi: 10.1176/ajp.2007.164.5.813. [DOI] [PubMed] [Google Scholar]
  26. Ho BC, Nopoulos P, Flaum M, Arndt S, Andreasen NC. Two-year outcome in first-episode schizophrenia: predictive value of symptoms for quality of life. Am J Psychiatry. 1998;155 (9):1196–1201. doi: 10.1176/ajp.155.9.1196. [DOI] [PubMed] [Google Scholar]
  27. Holt DJ, Weiss AP, Rauch SL, Wright CI, Zalesak M, Goff DC, Ditman T, Welsh RC, Heckers S. Sustained activation of the hippocampus in response to fearful faces in schizophrenia. Biol Psychiatry. 2005;57 (9):1011–1019. doi: 10.1016/j.biopsych.2005.01.033. [DOI] [PubMed] [Google Scholar]
  28. Izard CE. Innate and universal facial expressions: evidence from developmental and cross-cultural research. Psychol Bull. 1994;115 (2):288–299. doi: 10.1037/0033-2909.115.2.288. [DOI] [PubMed] [Google Scholar]
  29. Kohler C, Gur RC, Swanson CL, Petty R, Gur RE. Depression in schizophrenia: I. Association with neuropsychological deficits. Biol Psychiatry. 1998;43 (3):165–172. doi: 10.1016/S0006-3223(97)00033-4. [DOI] [PubMed] [Google Scholar]
  30. Kohler CG, Turner TH, Bilker WB, Brensinger CM, Siegel SJ, Kanes SJ, Gur RE, Gur RC. Facial emotion recognition in schizophrenia: intensity effects and error pattern. Am J Psychiatry. 2003;160 (10):1768–1774. doi: 10.1176/appi.ajp.160.10.1768. [DOI] [PubMed] [Google Scholar]
  31. Kohler CG, Turner T, Stolar NM, Bilker WB, Brensinger CM, Gur RE, Gur RC. Differences in facial expressions of four universal emotions. Psychiatry Res. 2004;128 (3):235–244. doi: 10.1016/j.psychres.2004.07.003. [DOI] [PubMed] [Google Scholar]
  32. Kohler CG, Martin EA, Stolar NM, Barrett FS, Verma R, Brensinger C, Bilker WB, Gur RE, Gur RC. Static posed and evoked facial expressions of emotions in schizophrenia. In press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Kring AM, Neale JM. Do schizophrenic patients show a disjunctive relationship among expressive, experiential, and psychophysiological components of emotion? J Abnorm Psychol. 1996;105 (2):249–257. doi: 10.1037//0021-843x.105.2.249. [DOI] [PubMed] [Google Scholar]
  34. Kring AM, Sloan DM. The Facial Expression Coding System (FACES): development, validation, and utility. Psychol Assess. 2007;19 (2):210–224. doi: 10.1037/1040-3590.19.2.210. [DOI] [PubMed] [Google Scholar]
  35. Kring AM, Kerr SL, Smith DA, Neale JM. Flat affect in schizophrenia does not reflect diminished subjective experience of emotion. J Abnorm Psychology. 1993;102 (4):507–517. doi: 10.1037//0021-843x.102.4.507. [DOI] [PubMed] [Google Scholar]
  36. Kring AM, Kerr SL, Earnst KS. Schizophrenic patients show facial reactions to emotional facial expressions. Psychophysiology. 1999;36 (2):186–192. [PubMed] [Google Scholar]
  37. Malla AK, Takhar JJ, Norman RM, Manchanda R, Cortese L, Haricharan R, Verdi M, Ahmed R. Negative symptoms in first episode non-affective psychosis. Acta Psychiatr Scand. 2002;105 (6):431–439. doi: 10.1034/j.1600-0447.2002.02139.x. [DOI] [PubMed] [Google Scholar]
  38. Martin CC, Borod JC, Alpert M, Brozgold A, Welkowitz J. Spontaneous expression of facial emotion in schizophrenic and right-brain-damaged patients. J Commun Disord. 1990;23 (4–5):287–301. doi: 10.1016/0021-9924(90)90005-j. [DOI] [PubMed] [Google Scholar]
  39. Mason O, Startup M, Halpin S, Schall U, Conrad A, Carr V. Risk factors for transition to first episode psychosis among individuals with ‘at-risk mental states’. Schizophr Res. 2004;71 (2–3):227–237. doi: 10.1016/j.schres.2004.04.006. [DOI] [PubMed] [Google Scholar]
  40. Nurnberger JI, Blehar MC, Kaufmann CA, et al. Diagnostic interview for genetic studies: rationale, unique features, and training. Arch Gen Psychiatry. 1994;51:949. doi: 10.1001/archpsyc.1994.03950110009002. [DOI] [PubMed] [Google Scholar]
  41. Putnam KM, Kring AM. Accuracy and intensity of posed emotional expressions in unmedicated schizophrenia patients: vocal and facial channels. Psychiatry Res. 2007;151 (1–2):67–76. doi: 10.1016/j.psychres.2006.09.010. [DOI] [PubMed] [Google Scholar]
  42. Schneider F, Heimann H, Himer W, Huss D, Mattes R, Adam B. Computer-based analysis of facial action in schizophrenic and depressed patients. Eur Arch Psychiatry Clin Neurosci. 1990;240:67–76. doi: 10.1007/BF02189974. [DOI] [PubMed] [Google Scholar]
  43. Schneider F, Ellgring H, Friedrich J, Fus I, Beyer T, Heimann H, Himer W. The effects of neuroleptics on facial action in schizophrenic patients. Pharmacopsychiatry. 1992;25 (5):233–239. doi: 10.1055/s-2007-1014412. [DOI] [PubMed] [Google Scholar]
  44. Schwartz BL, Mastropaolo J, Rosse RB, Mathis G, Deutsch SI. Imitation of facial expressions in schizophrenia. Psychiatry Res. 2006;145 (2–3):87–94. doi: 10.1016/j.psychres.2005.12.007. [DOI] [PubMed] [Google Scholar]
  45. Shtasel DL, Gur RE, Gallacher F, Heimberg C, Gur RC. Gender differences in the clinical expression of schizophrenia. Schizophr Res. 1992;7 (3):225–231. doi: 10.1016/0920-9964(92)90016-x. [DOI] [PubMed] [Google Scholar]
  46. Steimer-Krause E, Krause R, Wagner G. Interaction regulations used by schizophrenic and psychosomatic patients: studies on facial behavior in dyadic interactions. Psychiatry. 1990;53 (3):209–228. doi: 10.1080/00332747.1990.11024505. [DOI] [PubMed] [Google Scholar]
  47. Suslow T, Roestel C, Ohrmann P, Arolt V. The experience of basic emotions in schizophrenia with and without affective negative symptoms. Compr Psychiatry. 2003;44 (4):303–310. doi: 10.1016/S0010-440X(03)00085-3. [DOI] [PubMed] [Google Scholar]
  48. Tremeau F, Malaspina D, Duval F, Correa H, Hager-Budny M, Coin-Bariou L, Macher JP, Gorman JM. Facial expressiveness in patients with schizophrenia compared to depressed patients and nonpatient comparison subjects. Am J Psychiatry. 2005;162 (1):92–101. doi: 10.1176/appi.ajp.162.1.92. [DOI] [PubMed] [Google Scholar]
  49. Van’t Wout M, van Dijke A, Aleman A, Kessels RP, Pijpers W, Kahn RS. Fearful faces in schizophrenia: the relationship between patient characteristics and facial affect recognition. J Nerv Ment Dis. 2007;195 (9):758–764. doi: 10.1097/NMD.0b013e318142cc31. [DOI] [PubMed] [Google Scholar]
  50. Verma R, Davatzikos C, Loughead J, Indersmitten T, Hu R, Kohler C, Gur RE, Gur RC. Quantification of facial expressions using high-dimensional shape transformations. J Neurosci Methods. 2005;141 (1):61–73. doi: 10.1016/j.jneumeth.2004.05.016. [DOI] [PubMed] [Google Scholar]
  51. Walker EF, Grimes KE, Davis DM, Smith AJ. Childhood precursors of schizophrenia: facial expressions of emotion. Am J Psychiatry. 1993;150 (11):1654–1660. doi: 10.1176/ajp.150.11.1654. [DOI] [PubMed] [Google Scholar]
  52. Wang P, Barrett FS, Martin EA, Milonova M, Gur RE, Gur RC, Kohler C, Verma R. Automated video-based facial expression analysis of neuropsychiatric disorders. J Neurosci Methods. 2008;168 (1):224–238. doi: 10.1016/j.jneumeth.2007.09.030. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES