Holistic Processing Predicts Face Recognition

Abstract

The concept of holistic processing is a cornerstone of face-recognition research. In the study reported here, we demonstrated that holistic processing predicts face-recognition abilities on the Cambridge Face Memory Test and on a perceptual face-identification task. Our findings validate a large body of work that relies on the assumption that holistic processing is related to face recognition. These findings also reconcile the study of face recognition with the perceptual-expertise work it inspired; such work links holistic processing of objects with people's ability to individuate them. Our results differ from those of a recent study showing no link between holistic processing and face recognition. This discrepancy can be attributed to the use in prior research of a popular but flawed measure of holistic processing. Our findings salvage the central role of holistic processing in face recognition and cast doubt on a subset of the face-perception literature that relies on a problematic measure of holistic processing.

Face recognition challenges perception because similar facial features are arranged in similar configurations on all human faces. As such, subtle differences in facial features and their spatial relations are particularly useful for discriminating faces. To facilitate extraction of configural information, people process faces holistically, as evidenced by the fact that it is more difficult to ignore part of a face than part of an object (Farah, Wilson, Drain, & Tanaka, 1998; Maurer, Le Grand, & Mondloch, 2002; Young, Hellawell, & Hay, 1987). Accordingly, people's ability to discriminate and recognize faces should depend at least in part on holistic processing.

Surprisingly, holistic processing and face-recognition ability have never been linked empirically. Support for the relationship between holistic processing and face-recognition ability is mainly indirect, coming from studies in which perceptual experts with superior object-identification ability also demonstrate holistic processing in their domain of expertise (Bukach, Phillips, & Gauthier, 2010; Gauthier & Tarr, 2002; Wong, Palmeri, & Gauthier, 2009). However, in a recent article, Konar, Bennett, and Sekuler (2010) argued that holistic processing does not predict face-identification ability. It is important to examine this issue further because holistic processing plays a pivotal role in studies of face recognition. Studies have used holistic processing to track the development of face recognition (e.g., Cassia, Picozzi, Kuefner, Bricolo, & Turati, 2009; Mondloch, Pathman, Maurer, Le Grand, & de Schonen, 2007), to study abnormal development of face recognition (e.g., among individuals with developmental prosopagnosia; Le Grand et al., 2006) or populations with face-recognition deficits that are part of more widespread cognitive impairment (e.g., schizophrenia; Schwartz, Marvel, Drapalski, Rosse, & Deutch, 2002), and to evaluate computational models of face recognition (Dailey & Cottrell, 1999). If holistic processing does not relate to performance recognizing faces, such efforts may constitute wild-goose chases.

Konar et al. (2010) suggested that their failure to find a relationship between face identification and holistic processing could be related to the specific nature of the tasks they used. We followed up on this concern by reassessing the relationship between holistic processing and face processing. In particular, we addressed two key issues.

First, we have questioned the validity of the composite design used by Konar et al. (2010 this design was adapted from a naming task with familiar faces devised by Young et al., 1987; see also Hole, 1994) elsewhere because of its susceptibility to response biases (e.g., Cheung, Richler, Palmeri, & Gauthier, 2008; Richler, Mack, Palmeri, & Gauthier, 2011). In the study reported here, we used a measure of the composite task that is arguably more valid than the design used by Konar et al. (2010) and that has been related to expertise for objects (Wong, Palmeri, & Gauthier, 2009).

Second, in everyday face recognition, an encountered face must be compared with many representations stored in memory to determine identity. Measures of face processing in a task in which participants need only to match faces within each trial, as used by Konar et al. (2010), may overestimate the contribution of featural strategies that are less available in real-world situations. Therefore, a task in which multiple target faces are stored in long-term memory may tap into the robustness of stored face representations and better represent individual differences relevant to everyday face recognition. For example, in a recent study, Furl, Garrido, Dolan, Driver, and Duchaine (2010) found that although a face-memory task was associated more strongly with a face-processing factor (as determined by principal component analysis), a perceptual face-matching task was associated more strongly with an object-processing factor. To sample individual differences in face processing better than the previous study by Konar et al. (2010), we used both the face-matching task used by Konar et al. (2010) and the Cambridge Face Memory Test (CFMT), a well-validated measure of individual differences in face recognition (Duchaine & Nakayama, 2006).

Method

Thirty-eight members of the Vanderbilt University community (11 male, 27 female) ranging in age from 18 to 40 years (median age = 20 years) were compensated for participation in the study. Participants completed three tasks in the following order: the CFMT, the composite task, and the face-identification task. The study was approved by the local institutional review board.

CFMT

At the start of the CFMT, participants studied frontal views of six target faces for a total of 20 s. Then they completed an 18-trial introductory learning phase, after which they were presented with 30 forced-choice test displays. Each display contained one target face and two distractor faces. Participants were told to select the face that matched one of the original six target faces. The matching faces varied from their original presentation in terms of lighting condition, pose, or both. Next, participants were again presented with the six target faces to study, followed by 24 test displays presented in Gaussian noise. All trials were combined for each participant to yield a single measure of accuracy.

Composite task

Stimuli in the composite task consisted of 20 female faces. These images were cropped to create 20 face tops and 20 face bottoms. Top and bottom halves were randomly combined on every trial to form composite faces 256 × 256 pixels in size (see Fig. 1b). A white line (3 pixels thick) separated the face halves, resulting in a stimulus 256 × 259 pixels in size. The white line ensured that it was completely unambiguous where the top face half ended and the bottom half began, and this, if anything, was expected to facilitate selective attention.

Figure 1.

Design of the composite task and sample trial structure. In the schematic diagram (a) letters represent facial identities. Task-relevant face halves are shown in white, and task-irrelevant halves are shown in gray. In same trials, task-relevant halves of the study and test faces were the same; in different trials, task-relevant halves were different. Both types of trials featured congruent and incongruent conditions. In congruent same trials, the irrelevant halves of the study and test faces were the same; in incongruent same trials, the irrelevant halves were different. In congruent different trials, the irrelevant halves of the study and test faces were different; in incongruent different trials, the irrelevant halves were the same. Face halves were presented aligned or misaligned. In the partial-design version of this task, only the trial types outlined in the gray boxes were presented; in the complete design, all trial types were presented. The examples in (b) illustrate the stimuli and trial sequence.

On each of 160 trials, participants were instructed to judge whether the top half of the test face was the same as or different from the top half of the study face while ignoring the other, irrelevant bottom half. On 80 trials, the top and bottom halves were aligned, and on the other 80 trials, the halves were misaligned. On misaligned trials, the top half of the test face was moved 35 pixels to the left, and the bottom half was moved 35 pixels to the right; thus, the edge of the top half always fell on the center of the bottom half (see Fig 1b for examples of stimuli and trial sequences).

There were four trial types in the composite task (see Fig. 1a). Two types were same trials, in which the relevant halves of the study and test faces were the same. In the two types of different trials, the relevant halves of the faces were different. Within same trials and different trials, faces could also be congruent or incongruent. In congruent trials, the irrelevant half was associated with the same response as the relevant half. In incongruent trials, the irrelevant face half was associated with a different response than the relevant half. Therefore, in congruent same trials, the irrelevant half of the test face was the same as the irrelevant half of the study face. In incongruent same trials, the irrelevant halves of the test and study faces were different. In congruent different trials, the irrelevant half of the test face was different from the irrelevant half of the study face. In incongruent different trials, the irrelevant halves of the study and test face were the same.

Two versions of the sequential-matching composite task have been used in previous research: the partial design and the complete design (see Fig. 1a). The partial design, used by Konar et al. (2010), consists of only two types of trial: congruent different and incongruent same. In same trials, an alignment effect indexes holistic processing: Accuracy is greater or reaction time (RT) is faster in misaligned trials than in aligned trials (Cassia et al., 2009; de Heering, Houthuys, & Rossion, 2008; Goffaux & Rossion, 2006; Hole, 1994; Le Grand et al., 2006; McKone & Robbins, 2007; Michel, Rossion, Han, Chung, & Caldara, 2006; Mondloch et al., 2007).

The complete design includes the two partial-design trials plus congruent same and incongruent different trials. Participants’ failure to selectively attend to parts of faces is indexed by a congruency effect: Performance is better on congruent trials than on incongruent trials (Cheung et al., 2008; Farah et al., 1998; Gauthier, Curran, Curby, & Collins, 2003; Goffaux, 2009; Richler, Mack, Gauthier, & Palmeri, 2009; Richler, Tanaka, Brown, & Gauthier, 2008). Misalignment reduces the congruency effect (Cheung et al., 2008; Richler et al., 2008), and this interaction between congruency and alignment is particularly sensitive to expertise-driven holistic processing (Richler, Bukach, and Gauthier, 2009; Wong, Palmeri, & Gauthier, 2009).

The partial-design measure was the first index of holistic processing used in the composite task (Hole, 1994; Young et al., 1987), but subsequently, both partial and complete designs have been extensively used (see the Supplemental Material available online). In the study reported here, we used the complete design, which gave us the flexibility to perform partial-design as well as complete-design analyses (see Cheung et al., 2008; Richler et al., 2011).

Face-identification task

Our face-identification task was modeled after the task used by Konar et al. (2010). On each of 120 trials, a target face was presented (200 ms). Participants then viewed a four-face display and had to select the face that matched the target face. This display was shown until a response was made. Target and foil faces were either all male (60 trials) or all female (60 trials) and differed in lighting conditions to prevent image matching.

Results and discussion

Holistic processing measured in the complete design of the composite task was observed in the group-level data, as revealed by a significant interaction between alignment and congruency in the analysis of d′, F(1, 37) = 5.28, p = .027. This interaction was not significant in the analysis of RT, F(1, 37) = 3.36, p = .075.

Partial analyses revealed no significant effect of alignment in the group-level data in the analysis of accuracy, t(37) = 1.31, p = .198, or of RT, t(37) = –1.587, p = .121. The failure to find an alignment effect is not the result of running partial analyses on data collected in the complete design (see the Supplemental Material). Moreover, all other measures suggest that our participants were typical and processed faces holistically according to the complete design: The absence of an alignment effect in partial analyses may reflect the poor reliability of this measure of holistic processing.

Next, we examined correlations between holistic processing and measures of face-recognition ability. For each correlation, 95% confidence intervals (CIs) are reported. Significant correlations were still significant using Spearman correlations and after removing outliers (see the Supplemental Material).

Average accuracy on the CFMT and face-identification task was 76.35% (SD = 14.39%) and 75.13% (SD = 10.55%), respectively. Performance on these tasks was strongly but not perfectly correlated, r(38) = .702, p < .0001, CI = [.448, .815]; this finding perhaps indicates an upper limit between each of these measures and holistic processing.

Using partial analyses within the complete design, we found that the magnitude of the alignment effect in RT did not correlate with face recognition, CFMT: r(38) = .128, p = .445, CI = [–.199, .430]; face-identification task: r(38) = .160, p = .336, CI = [–.168, .456] (see Fig. 2), nor did the alignment effect in accuracy correlate with face recognition, CFMT: r(38) = .190, p = .252, CI = [–.138, .480]; face-identification task: r(38) = .093, p = .579, CI = [–.233, .400]. Furthermore, the alignment effect did not correlate with the alignment effect indexed using d′, which was the measure used by Konar et al., CFMT: r(38) = –.074, p = .660, CI = [–.384, .251]; face-identification task: r(38) =.040, p = .809, CI = [–.283, .335]. In sum, our partial-design analyses replicated Konar et al.'s (2010) findings: We found no evidence that holistic processing is linked to face recognition.

Figure 2.

Scatter plots (with best-fitting regression lines) showing correlations between measures of holistic processing in the partial design of the composite task (x-axes) and face-identification ability (y-axes). Results are shown for the Cambridge Face Memory Test (CFMT; top row) and the face-identification task (bottom row). Holistic processing in the partial design was indexed by the alignment effect (difference in performance between misaligned and aligned incongruent same trials) in accuracy (left column) and reaction time (RT; right column). CI = confidence interval.

In contrast, holistic processing in the complete design predicted individual differences in face recognition (Fig. 3). Performance on the CFMT was significantly correlated with the magnitude of the Congruency × Alignment interaction in analyses of both d′, r(38) = .396, p = .014, CI = [.088, .635], and RT, r(38) = .334, p = .040, CI = [.017, .590]. Performance on the face-identification task was significantly correlated with holistic processing in analyses of RT, r(38) = .482, p = .002, CI = [.192, .694], but not of d′, r(38) = .031, p = .851, CI = [–.291, .347]. At least one prior study found that face matching and face memory differentially correlate with speed and accuracy in face recognition (Wilhelm et al., in press), but it is also possible that the RT measure of holistic processing is sometimes more sensitive than the d′ measure (e.g., in Wong, Palmeri, & Gauthier, 2009, only the RT index correlated with right fusiform gyrus activity). Holistic processing has been traditionally indexed by either or both of these dependent variables, and it is important to note that we found no trade-off between the two.

Figure 3.

Scatter plots (with best-fitting regression lines) showing correlations between measures of holistic processing in the complete design of the composite task (x-axes) and face-identification ability (y-axes). Results are shown for the Cambridge Face Memory Test (CFMT; top row) and the face-identification task (bottom row). Holistic processing in the complete design was indexed by the magnitude of the Congruency × Alignment interaction in d′ (calculated as z scores for hits minus z scores for false alarms; left column) or reaction time (RT; right column).

In addition to holistic processing, featural processing may contribute to face-recognition performance. Two multiple regression analyses were conducted, with performance on the face-identification task and on the CFMT as dependent variables. The four predictors in the model were the Congruency × Alignment interaction in the analysis of d′ and of RT, and performance in the analysis of d′ and of RT in the misaligned-faces conditions (averaging across congruency). Performance on misaligned trials provides an estimate of featural processing because when face parts are misaligned, selective attention to a part is more successful, as evidenced by the smaller congruency effect on misaligned trials. Consistent with our conjectures about the differences between the face-identification task and CFMT, we found independent contributions of the Congruency × Alignment interaction in the analysis of RT and of d′ for misaligned trials on face-identification scores, but only the Congruency × Alignment interaction in d′ was a significant predictor of CFMT scores (Table 1). Performance on both tasks relies on holistic processing, but the CFMT allows little or no contribution from featural processing.

Table 1.

Results of Multiple Regression Analyses

Model and predictor	β	SE	t	p
Face-identification task (R2 adjusted = 27.9%)
Intercept	0.52928	0.09480	5.580	.000
Congruency × Alignment (d′)	0.01139	0.01560	0.731	.470
Congruency × Alignment (RT)	0.00043	0.00010	3.030	.005
Misaligned faces (d′)	0.05529	0.02210	2.500	.018
Misaligned faces (RT)	0.00014	0.00010	1.370	.179
Cambridge Face Memory Test (R2 adjusted = 20.5%)
Intercept	48.02460	17.80000	2.700	.011
Congruency × Alignment (d′)	8.00504	2.92400	2.740	.010
Congruency × Alignment (RT)	0.04713	0.02640	1.790	.083
Misaligned faces (d′)	5.81439	4.14700	1.400	.170
Misaligned faces (RT)	0.00397	0.01930	0.206	.838

Note: The only predictors that were significantly correlated were d′ and reaction time (RT) for misaligned faces (RTs; r = –.37, p = .02).

Holistic processing measured with the complete design in the composite task predicted individual differences in face recognition: The larger the effect of holistic processing (Congruency × Alignment interaction), the better the face-recognition performance. This finding reconciles the idea that holistic processing is important to face processing with studies linking holistic processing and perceptual expertise (Gauthier & Tarr, 2002; Wong, Palmeri, & Gauthier, 2009). In fact, because the face-identification task and the CFMT are similar to measures of expertise in nonface domains (Bukach et al., 2010; Gauthier et al., 2003; Wong & Gauthier, 2010), our results suggest that holistic processing predicts expertise for both faces and nonface objects.

Why do the complete- and partial-design measures of holistic processing lead to different conclusions? One reason is that the partial-design measure does not take into account possible influences of response biases, whereby participants choose to respond “same” more often in some conditions regardless of their ability to discriminate the face halves (Cheung et al., 2008; Richler, Mack, et al., 2009; Richler et al., 2011). Indeed, although d′, as used by Konar et al. (2010), provides a discriminability measure that is independent of response bias, its use does not fully resolve this issue. In the partial design, irrelevant face halves are always different, therefore same trials are always incongruent and different trials are always congruent. In the complete design, congruency often influences response bias, and often differentially on aligned versus misaligned trials (Cheung et al., 2008; Richler, Mack et al., 2009; Richler et al., 2011). A limitation of the partial design is that the alignment effect is confounded with congruency, and there is no way to measure the bias associated with congruency and how this bias is modulated by alignment. In contrast, in the complete design, d′ as a function of both alignment and congruency is robust to manipulations that influence response bias (Cheung et al., 2008; Richler, Mack, et al., 2009; Richler et al., 2011). Indeed, in the study reported here, response bias was correlated with the magnitude of the alignment effect in analyses of accuracy and d′—accuracy: r(38) = –.523, p = .001, CI = [–.721, –.245]; d′: r(38) = –.666, p < .0001, CI = [–.44, –.812]—but not holistic processing measured in the complete design, r(38) = –.280, p = .088, CI = [–.550, .043].

General Discussion

The fact that faces are processed holistically is what makes them special—face perception relies on holistic processing more than object perception to maximize sensitivity to configural information (Farah et al., 1998). A considerable amount of research depends not only on the validity of the way in which holistic processing is measured, but even more fundamentally on the assumption that holistic processing is relevant to understanding face processing (e.g., Cassia et al., 2009; Dailey & Cottrell, 1999; Le Grand et al., 2006; Schwartz et al., 2002). If holistic processing is not predictive of face-identification performance (Konar et al., 2010), this undermines the motivation for this line of research and may require researchers to rethink how face recognition is studied. Fortunately, this is not necessary: Although there are some differences between the two face-processing tasks we used, individual differences in both face matching and face identification were related to holistic processing. However, the choice of holistic-processing measure seems to be critical: Our results were consistent with Konar et al.'s (2010) findings when the partial-design measure of holistic processing was used, but opposite conclusions were reached using a different measure in the same task.

Our confirmation of the relationship between holistic processing and face recognition corroborates a widely held assumption, but, counterintuitively, it is problematic in other ways. For example, we would expect that face-recognition abilities improve over the course of development, yet there are reports of young children exhibiting adult-like holistic processing (e.g., Cassia et al., 2009; Mondloch et al., 2007). Similarly, individuals with developmental prosopagnosia and patients with schizophrenia show deficits in face recognition. If holistic processing were critically related to face recognition, we would expect abnormal holistic processing for these groups. But in both cases, holistic processing is reported to be normal (Le Grand et al., 2006; Schwartz et al., 2002). However, these conclusions are based solely on research using the partial design, and they could reflect artifacts of important (and potentially informative) group differences in response biases. In another debate, holistic processing measured in the complete design is one of the hallmarks of face perception that can be acquired for nonface objects (Gauthier & Tarr, 2002; Wong, Palmeri, & Gauthier, 2009), but partial-design studies have failed to replicate this result (Robbins & McKone, 2007).

Just as abandoning phrenology did not mean rejecting cortical specialization of functions, this is a case in which abandoning a flawed measure increases the construct validity of holistic processing. However, we cannot hope to make theoretical progress in our understanding of the mechanisms underlying face perception if we continue to use the partial design of the composite task. Holistic processing is a valuable construct that provides a link between experience, performance, and brain specialization. For instance, practice individuating objects produces increases in holistic processing that predict activity in the fusiform gyrus (Wong, Palmeri, & Gauthier, 2009; Wong, Palmeri, Rogers, Gore, & Gauthier, 2009). These previous studies offer experimental evidence for the causal influences of holistic processing on individuation ability that can only be inferred from the correlations obtained in the present study of face recognition.