Visual and cognitive processing of face information in schizophrenia: Detection, discrimination and working memory

Abstract

Face recognition involves several physiological and psychological processes, including those in visual, cognitive and affective domains. Studies have found that schizophrenia patients are deficient at recognizing facial emotions, yet visual and cognitive processing of facial information in this population has not been systematically examined. In this study, we examined visual detection, perceptual discrimination and working memory of faces as well as non-face visual objects in patients. Visual detection was measured by the accuracy when detecting the presence of a briefly displayed face, image which contained only the basic configural information of a face. Perceptual discrimination was measured by discriminability scores for individual facial identity images, in which the degree of similarity between images was systematically varied via morphing. Working memory was measured by the discriminability scores when two comparison face images were separated by 3 or 10 seconds. All measurements were acquired using a psychophysical method (two-alternative forced choice). Relative to controls, patients showed significantly reduced accuracy in visual detection of faces (p=0.003), moderately degraded performance in perceptual discrimination of faces (p=0.065), and significantly impaired performance in working memory of faces (p<0.001 for both 3 and 10 sec conditions). Patients' performance on non-face versions of these tasks, while degraded, was not correlated with performance on face recognition. This pattern of results indicates that greater signal strength is required for visual and cognitive processing of facial information in schizophrenia.

Introduction

Face recognition plays a foundational role in everyday life. Processing of emotion information, one aspect of face recognition, is shown to be impaired in schizophrenia (Gur et al. 2002; Walker et al. 1984). Face recognition depends not only on the processing of emotion information but also on the processing of visual and cognitive information (Bruce and Young 1986). ‘Face’ is sui generis relative to other visual objects in that processing facial information involves distinct visual and cognitive processes. Although abundant research has shown that visual and cognitive processes are deficient in schizophrenia (Chen et al. 1999; O'Donnell et al. 1996; Park and Holzman 1992), visual and cognitive processing of face information has not been systematically explored in this population.

The cortical system for visual and cognitive processing of facial information involves the fusiform gyrus, including fusiform face area (FFA) and several adjacent areas. Several structural MRI studies have shown that the volume of fusiform gyrus matter is reduced in schizophrenia (Lee et al. 2002; Onitsuka et al. 2003). Studies on the functional activity of FFA in schizophrenia, however, have yielded more nuanced results. FMRI studies have found that responses to a face working memory task were either comparable between patients and controls (Yoon et al. 2006) or altered in patients (Yoo et al. 2005). This pattern of results suggests that some neurophysiological processes of face recognition are deficient in schizophrenia whereas the others are preserved. Thus, sophisticated study designs are needed to tease out the complex visual and cognitive processes implicated in face recognition in schizophrenia.

Previous studies have used non-affective visual and/or cognitive categorization tasks as a comparison for emotion recognition tasks (Archer et al. 1992; Martin et al. 2005). Several recent studies have shown that schizophrenia patients are deficient in detecting face images (Butler et al. 2008; Chen et al. 2008; Zivotofsky et al. 2008), and that patients are also deficient in face memory tasks (Calkins et al. 2005; Onitsuka et al. 2003). The paradigms used in these studies were primarily based on matching between familiar or unfamiliar faces, without adjusting the signal strength of face images or altering task difficulty levels. Moreover, due to the nature of the task paradigms used, the involvement of various stages of face information processing was not experimentally differentiated. As a result, the issue of which processing stages, sensory or cognitive (or both), are implicated in facial recognition deficits in schizophrenia has not been resolved.

In this study, we examined visual detection, perceptual discrimination and working memory of faces as a function of facial signal strength. Visual detection involves only the sensory processing of configural information required for identifying a face qua face, without extracting information about identity, intention or emotion. Perceptual discrimination entails distinguishing individual identities (i.e. one individual from another) based on face images varying in degree of facial similarity. Working memory requires that participants discriminate a particular face from similar faces presented in the immediate past (i.e. 3 and 10 seconds previous). A psychophysical approach was employed to probe into sensory and cognitive processing of face information separately, by establishing the relationship between face signal strength and the detection, discrimination and working memory of faces.

Methods

Subjects

Twenty-nine schizophrenia patients and 27 normal controls participated in this study. Eleven patients had a diagnosis of schizophrenia and 18 patients had a diagnosis of schizoaffective disorder. Diagnoses were based on a structured clinical interview conducted by experienced clinicians who were blind to the purposes of this study (Spitzer 1994) and by a review of all available medical records. Of the 29 patients, one had social phobia, one had general anxiety, and the rest had no other Axis I co-morbidities at the time of the study. All patients were treated with antipsychotic drugs (mean daily Chlorpromazine dose equivalent (CPZ) was 602.4 mg; SD = 414.3 mg). The Positive and Negative Symptom Scale was administered (positive subscale: 15.5 ± 7.7; negative subscale: 13.4 ± 6.7; general subscale: 29.5 ± 11.9). Normal controls were assessed for the absence of axis I psychiatric disorders, using a standardized interview based on SCID-I/NP (First et al. 2002). The two groups did not differ in average age or sex. A modified version of The Situational Feature Recognition Test (Corrigan and Green 1993), which assesses one's ability to make appropriate judgments in certain social contexts, as well as the Wechsler Adult Intelligence Scale (Wechsler 1981) (verbal component), were administered to both groups. Table 1 provides demographic information for the participants.

Table 1.

Demographic and social functioning information of the sample Mean (standard deviation)

Group	Age (years)	Sex*	Verbal IQ	Education (years)	Parental education (years)**	Social functioning scale***
Schizophrenia (n=29)****	41.6 (9.5)	F – 15 M – 14	101.8 (13.1)	14.1 (1.8)	4.98 (1.46)	0.86 (0.08)
Normal Control (n=27)*****	41.0 (13.6)	F – 15 M – 12	111.4 (8.9)	15.8 (2.2)	4.90 (1.22)	0.90 (0.04)

The subjects were included based on the following general criteria: (1) no history of any neurological disorders (such as seizure or stroke) or head injuries, (2) IQ > 70, and (3) no substance abuse in the six months prior to participation. The participants had normal or corrected to normal vision, as assessed by the Rosenbaum Pocket Vision Screener.

^*F – female, M – male

^**Hollingshead parent education score

^***based on the Situational Feature Recognition Test (Corrigan and Green 1993)

^****Two being non-Caucasian

^*****Five being non-Caucasian

Stimulus and procedure

1) Detection

The target was a line-drawn face (Figure 1a) embedded in another large scrambled line drawing, and was positioned either on the left side or on the right side of the drawing. The task was to judge whether the face was located on the left or the right side of each drawing. This is an abbreviated version of the task used in a previous study (Chen et al. 2008). The stimulus for non-face object detection was a line-drawn tree. The procedure for the non-face object detection was identical to that for face detection.

Figure 1.

Schematic illustration of stimuli used in (a) face detection and (b) face discrimination and working memory. For face detection, the testing sessions were blocked according to duration for which stimuli were displayed (13, 26, 52 or 104 msec). Face signal strength here is modulated by display time. Note that the face signal is limited to configural face information. In each session, 42 stimuli were presented. Presentation order of the sessions was randomized across subjects. This procedure contains no requirement for judging or memorizing facial identity or expression. For face discrimination and working memory, the difference between the two comparison face images in the second presentation, or the similarity level, could be 5, 12.5, 25, 50, or 100%. A 100% difference would entail a comparison of the two original face photographs (i.e. of two different faces). The larger a difference (i.e. the greater face signal strength for discrimination), the more easily the task can be performed. The similarity levels varied across trials according to the method of constant stimuli. Stimulus presentation and response recording were programmed within a VisionShell and controlled by a Macintosh G3 computer. All task procedures employed a two alternative forced choice method. The order of the tasks was counter-balanced across subjects. Subjects practiced on each task before data collection began. The face and the non-face tests each took about 60 minutes to complete.

2) Discrimination

The targets were photographed face images, either the original images or morphed versions of the original images (Figure 1b). The task was to discriminate between a series of paired face images with neutral expressions. The series of images was created by morphing together the two separate faces (i.e. two different individuals) such that the resultant image contained varying proportions of the two original face photographs. Each trial contained two presentations; the first presentation (600 msec) contained a single face image, while the second (1,200 msec) contained two face images side by side, one of which was identical to that of the first presentation, the other differing to varying degrees. Subjects determined which of the two face images from the second presentation was the same as the face image from the first presentation. The critical measure for discrimination was the just-noticeable-difference (JND) between the two comparison face images at which performance reached the criterion of 80% correct. This JND can be extracted from a psychometric function of the percent correct scores, and is defined as the threshold for face discrimination (Chen et al. 2005). The time interval between the two presentations was brief (500 msec), which minimized the involvement of working memory (Magnussen and Greenlee 1999). The stimulus for non-face object discrimination was a series of morphed images based on two photographed cars. The procedure for car discrimination was identical to that for face discrimination.

3) Working memory

The targets and procedures were the same as that used for perceptual discrimination, with one exception - the time intervals between the first and the second face (or car) image presentations were prolonged to 3 or 10 sec.

The study protocol was approved by the Institutional Research Board (IRB) of McLean Hospital. Written informed consent was obtained after complete description of the study to the subjects.

Results

Face detection

ANOVA on group and stimulus duration yielded a significant group difference (F=9.08, p=0.003). The stimulus duration effect was also significant (F=3.93. p=0.019), whereas the group-stimulus duration interaction was not. To provide an overall measure of face detection, the accuracies were averaged across stimulus durations. Figure 2 shows that the averaged accuracies were significantly lower in patients than in controls (t=2.22, p=0.030; effect size (ES)=0.575). A non-parametric Wilcoxon Signed Rank test yielded a similar result (z=2.10, p=0.036).

Figure 2.

Averaged percent correct score or accuracy in detecting faces. The abscissa denotes subject groups. The ordinate denotes subject accuracy in face detection. Higher accuracy means better performance. Open symbols are for normal controls and filled symbols for schizophrenia patients. The dark bars represent the mean accuracy of each group. When the outliers, (defined as points that were 2 standard deviations below and above the group mean) were removed, the group difference remained similarly significant.

A subgroup of patients (n=22) and controls (n=16) were also tested on tree detection. The averaged accuracies were significantly lower in patients than in controls (t=3.78, p=0.001). A non-parametric Wilcoxon Signed Rank test yielded a similar result (z=2.87, p=0.004).

Face discrimination

ANOVA on group and stimulus similarity level yielded a significant stimulus similarity effect (F=146.08, p<0.001). The group effect approached significance (F=3.44, p=0.065) whereas the interaction effect did not (F=1.54, p=0.191). Figure 3 shows that the discriminability scores, the inverses of thresholds, were lower in patients than in controls, yet the group difference was not significant (t=1.36, p=0.18; ES=0.440). A non-parametric Wilcoxon Signed Rank test yielded a similar result (z=1.33, p=0.18).

Figure 3.

Discriminability scores for perceptual discrimination of similar faces. The ordinate denotes the discriminability score. The higher the score, the better performance is. The abscissa denotes subject groups. The dark bars represent the mean score of each group. See legend of Figure 2 for additional information.

For a subgroup of patients (n=14) and controls (n=16) who were also tested for car discrimination, the discriminability scores were significantly lower in patients than in controls (z=2.37, p=0.018) (non-parametric Wilcoxon Signed Ranks test).

Face memory

For the 3 sec inter-stimulus interval, ANOVA yielded a significant effect of group (F=22.05, p<0.001), stimulus similarity (F=138.41, p<0.001) and interaction between them (F=4.63, p=0.001). For the 10 sec inter-stimulus interval, ANOVA again yielded a significant effect of group (F=12.85, p<0.001) and stimulus similarity (F=110.16, p<0.001). However, the interaction between group and stimulus similarity was not significant. Figure 4 shows that patients had significantly lower discriminability scores than controls (3 sec: t=3.94, p<0.001; ES=1.178; 10 sec: t=2.37, p=0.02; ES=0.698). A non-parametric Wilcoxon Signed Rank Test yielded a similar result (3 sec: z=3.06, p=0.002; 10 sec: z=1.98, p=0.048).

Figure 4.

Discriminability scores for working memory of similar faces. The left panel is for the inter-stimulus interval of 3 sec, and the right panel is for the inter-stimulus interval of 10 sec. See legends of Figures 2 and 3 for additional information.

For a subgroup of patients (n=14) and controls (n=16) who were also tested for working memory of cars, the discriminability scores were significantly lower in patients than in controls for the 3 sec (z=2.1, p=0.036) but not for the 10 sec condition (z=1.96, p=0.050) (non-parametric Wilcoxon Signed Rank test).

Relationships among detection, discrimination and memory

In both groups, the averaged accuracies in detection were not correlated with the thresholds in discrimination or working memory (Table 2). In patients, the discrimination thresholds were significantly correlated with those in the short (3 sec) and intermediate (10 sec) working memory conditions, and correlations between the two working memory conditions were also significant. In controls, however, the correlations between (1) face discrimination and the short working memory condition, and (2) short and intermediate working memory conditions were not significant.

Table 2.

Correlations among detection, discrimination and working memory performance (patient/control)

	detection	discrimination	short Memory	intermediate memory
Detection	1/1	-	-	-
Discrimination	-0.07/-0.1	1/1	-	-
short Memory (3 sec interval)	-0.06/-0.1	0.70*/0.05	1/1	-
intermediate memory (10 sec interval)	-0.06/-0.16	0.62*/0.44*	0.62*/-0.05	1/1

^*Significant correlation

Relationship between the recognition of faces and non-face objects

In patients, the average accuracies in face detection were weakly correlated with those in tree detection (r=0.32). ANOVA with group (patient and control) and task (face and tree detection) yielded an insignificant interaction between group and task (F=0.154, p=0.697). The discriminability scores for discrimination and working memory of faces and cars were not correlated (Table 4). Consistent with the correlation analysis, ANOVA with group and task (face and car discrimination) yielded a significant interaction (F=4.75, p=0.039). In controls, the average accuracies for face detection and for tree detection were not correlated. The discriminability scores for face discrimination and for car discrimination were also not correlated (Table 4).

Table 4.

Correlations between performances in recognition of face and non-face objects

	Patient	Control
Detection of Face vs Tree	0.32	0.41
Discrimination of Face vs Car (0.5 sec)	0.01	0.00
Memory of Face vs Car (3 sec)	0.49	0.46
Memory of Face vs. Car (10 sec)	-0.04	-0.19

Relationship with social functioning and clinical variables

In controls, social functioning scores were significantly correlated with performance in the face detection task (r=0.64). In patients, social functioning scores were not correlated with performance on any task (Table 5). The Z statistic for the patients' and controls' correlations between face detection and social functioning was 3.38 (based upon the Fisher r to z transformation), exceeding the criterion level (1.96) for a significant difference, and indicating that the relationship between face detection and social functioning differs between groups.

Table 5.

Correlations among performance in face tasks and social and clinical variables in patients

	social functioning	PANSS (+)	PANSS (-)	PANSS general	CPZ
Detection	-0.06	-0.22	-0.22	-0.37*	-0.43*
Discrimination	0.29	0.38*	0.36*	0.32	0.31
short memory (3 sec interval)	0.15	0.44*	0.34	0.35	0.51*
intermediate memory (10 sec interval)	0.30	0.15	0.15	0.15	0.42*

^*Significant at p < 0.05

In patients, correlations between face recognition performance and PANSS scores varied from weak to moderate (Table 3). Performance in face detection and working memory tasks and the level of antipsychotic drug consumption were moderately correlated (Table 5).

Table 3.

Performance in recognition of non-face visual objects

Task	Tree Detection*	Car Discrimination** (0.5 sec)	Short memory of Car** (3 sec)	Intermediate memory of car** (10 sec)
Group
Normal Control	94.7 (1.9)***	1.69 (.05)	1.72 (.03)	1.67 (.05)
Schizophrenia	89.5 (1.6)	1.40 (.08)	1.47 (.09)	1.48 (.08)

^*Accuracy

^**Discriminability score

^***Standard error in parentheses.

Patients with schizophrenia (n=14) and schizoaffective disorder (n=15) did not differ in performance on any face perception tasks tested (detection: p=0.97; discrimination: p=0.11; working memory: p=0.71 (3 sec); p=0.76 (10 sec)), suggesting that inclusion of schizoaffective patients did not alter the general finding of this study.

Discussion

This study found that patients' ability to detect faces as such was deficient. Their ability to discriminate individual identities based on similar face images was also deficient but to a lesser degree, as measured by effect size. Their ability to discriminate a current face image from those seen recently (3 sec earlier) was impaired to a greater extent. The impaired performances on these face recognition tasks were not correlated with one another, except between the discrimination and working memory performances (Table 2).

A cascade of face recognition processes

Face recognition includes a cascade of facial information processes. The classification of a visual stimulus as a face (i.e. face detection) may be seen as the first step (Ellis 1981). Next, identity discrimination distinguishes individuals based on visual cues presented in the face. Analysis of facial expressions identifies the types of emotions presented in the face. The processes for recognition of facial identity and facial emotion can be dissociated (Duchaine et al. 2003; Kurucz and Feldmar 1979). Following these processes, working memory encodes and, when needed, retrieves the various aspects of face information processed earlier.

The present study found that non-affective processing of facial information is compromised in schizophrenia. In face detection where only very limited facial signals were presented, patients showed reduced accuracies, suggesting that this basic judgment suffers from deficient sensory processing. In identity discrimination, patients showed elevated thresholds, suggesting that they need an increasing strength of facial signals for differentiating individuals. The magnitude of the threshold elevation was, however, moderate (ES: 0.440). One major difference between face detection and face discrimination is the use of information-rich photographs in the latter, as compared to the simple sketched images used in the former. The additional information embedded in the photographs (such as brightness, contrast and shape) might have helped patients achieve the less degraded performance found in face discrimination (relative to face detection). The working memory task probes maintenance (2 different lengths of delay – 3 and 10 sec) and working load manipulation (5 different stimulus similarity levels). Patients showed much higher thresholds in the 3 sec condition than those in the basic discrimination task (ES: 1.178), suggesting that stronger signals are required in order to maintain and retrieve face information. It is intriguing that patients demonstrated the greatest working memory impairment in the short (3 sec) rather than the intermediate (10 sec) delay condition, suggesting that poor retention is not the sole factor underlying deficient face working memory. Note that in controls performances under 0.5 sec and 3 sec condition were not correlated (r=0.07), whereas in patients this correlation was significant (r=0.55). It is thus possible that the encoding process in schizophrenia is slow and results in incomplete consolidation of face information for working memory until later. The consequence of slow encoding would be poor performance in short but not necessarily in intermediate memory conditions. Together, these data derived from detection, discrimination and working memory demonstrate that visual and cognitive processes differ in their dependence on face signal strength.

Recognition of face and non-face objects

Whether face recognition represents a unique process independent of recognition of non-face objects is still a debated topic (Tovee 1998). Like in face recognition, patients showed poor performance in tree detection and discrimination and working memory of cars, which seems to suggest a general deficit in object recognition. Such a notion is, however, not fully supported by a further analysis of correlations between performances in face and non-face object recognition. The correlations between performance in the two types of tasks varied from moderate to none in patients, suggesting the existence of both face-specific and non face-specific deficits.

Face recognition and social and clinical variables

In controls, the correlation between scores in face detection and social functioning (r=0.64) suggests that the ability to detect a face is associated with social interaction. In patients, however, no such correlation was present, suggesting that their visual processing of face information is no longer associated with this aspect of social functioning. Poor face recognition is unlikely a consequence of social functioning problems as the social functioning scores in this sample of patients were relatively normal. A small portion of face recognition deficits may, however, be attributable to clinical features, as the two sets of variables were modestly correlated in patients.

One limitation of this study was the inclusion of a mixture of patients whose antipsychotic medication treatment varied substantially. The correlations between CPZ equivalents and detection and working memory of faces seem to suggest that a portion of face recognition performance variance in patients may be related to the medication. Yet, CPZ equivalents were also related to psychotic symptoms (positive and general PANSS subscales) - patients who performed poorly in face recognition were those who had high PANSS scores and thus needed to take high doses of antipsychotic drugs. Further studies in patients with low levels of psychotic symptoms and/or little or no antipsychotic medication would clarify this issue.

Face recognition and cortical mechanisms

The behavioral results found in this study have two implications for the search for underlying physiological mechanisms in schizophrenia. First, different degrees of impairment in visual detection, identity discrimination and working memory suggest that in order to assess a cortical mechanism for face recognition, it is necessary to select specific face features and then examine corresponding physiological responses. Second, varying signal strength of specific face features would provide an effective approach to assess the extent to which cortical processes for face recognition are impaired. For example, to examine the functional properties of the FFA, not only a face vs. non-face object strategy, but also comparisons of faces with different levels of signal strength, should be employed.

Face recognition and signal strength

The increased demand for facial signal strength in patients raises a practical question as to whether poor face recognition can be improved by modulating visual and cognitive features of face images. The strength of visual and cognitive signals plays a crucial role in object recognition, including faces (Bokde et al. 2005). It has been shown that sensory representation of remembered stimuli is maintained within the prefrontal cortex (Constantinidis et al. 2001), where perceptual decisions are ultimately formed. It has also been shown that both recognition of degraded images (such as faces) and neural activity in associated cortical regions are enhanced once persons have been exposed to the ‘undegraded’ (i.e. original) versions of the same images (Dolan et al. 1997). This intimate relationship between the strength of visual and cognitive signals and the perceptual and cortical responses to faces suggests a plausible basis for using face images with salient visual and cognitive features in training patients to improve their face recognition abilities.

In summary, this study offers new insights into how recognition of visual and cognitive features of faces is related to the signal strength of facial information in schizophrenia. Future studies should explore how visual and cognitive processes in schizophrenia interact with emotional processes in the context of face recognition. It would be beneficial to examine whether manipulation and learning of visual and cognitive features can help to improve face recognition, and possibly the quality of everyday life, in patients.