Does Temporal Integration of Face Parts Reflect Holistic Processing?

Abstract

We examined whether temporal integration of face parts reflects holistic processing or response interference. Participants learned to name two faces “Fred” and two “Bob”. At test, top and bottom halves of different faces formed composites and were presented briefly separated in time. Replicating prior findings (Singer & Sheinberg, 2006), naming of the target halves for aligned composites was slowed when the irrelevant halves were from faces with a different name compared to that from the original face. However, no interference was observed when the irrelevant halves had identical names as the target halves but came from different learned faces, arguing against a true holistic effect. Instead, response interference was obtained when the target halves briefly preceded the irrelevant halves. Experiment 2 confirmed a double-dissociation between holistic processing vs. response interference for intact faces vs. temporally separated face halves, suggesting that simultaneous presentation of facial information is critical for holistic processing.

The ability to individuate faces is an important skill. Because faces are composed of features that do not vary much and are organized in a similar configuration, subtle differences in features and in their spacing become critical. Faces are generally thought to be processed more holistically than other objects (Farah et al., 1998; Tanaka & Farah, 1993; Young et al., 1987). Specifically, recognition of a facial feature is better within a whole face than when the feature is shown alone (Tanaka & Farah, 1993). Also, naming one half of a face is more difficult when the task-irrelevant half is from a different face (Young et al., 1987), revealing an inability to selectively attend to parts in the context of a face. This holistic processing is sensitive to changes in configuration and is reduced for inverted faces or misaligned face parts (see Maurer et al., 2002 for a review).

Holistic processing occurs rapidly for upright intact faces (e.g., 50ms after the onset of a face; Richler et al., 2009a). It has been suggested that holistic processing supports integration when face parts are separated briefly in time (Singer & Sheinberg, 2006; Anaki et al., 2007; Anaki & Moscovitch, 2007). In particular, failures of selective attention to parts in the context of a face persist when the face parts are temporally separated by up to approximately 120ms (Singer & Sheinberg, 2006). Recognition is more successful for upright than inverted faces when sequentially presented face parts are shown within a brief time (up to 450ms, Anaki et al., 2007; Anaki & Mosocovitch, 2007). Such temporal integration is consistent with the idea that facial features become diagnostic over time (Vinette et al., 2004). One account suggests that facial features separated in time can be stored and integrated into a holistic percept in a short-term visual buffer; in other words, holistic processing might not require the simultaneous presentation of facial features (Anaki et al., 2007; Anaki & Moscovitch, 2007).

Here, we ask whether integration of temporally separated face parts is indeed of the same nature as integration in an intact face. We seek to distinguish processes that are more specific to faces vs. those that may be more general to any object category. When all parts are shown at once, holistic processing is more important for faces than for other objects (Tanaka & Farah, 1993; Farah et al., 1998). It is not as clear that temporal integration of face parts shows the same advantage, since these effects obey temporal constraints that are strikingly similar to integration of non-face visual stimuli. For instance, the interference effect for faces is strongest when the target face part is presented up to 80ms before the irrelevant face part (Singer & Sheinberg, 2006). Likewise, brief temporal intervals between incompatible target and distractor information in the Stroop task also result in impaired performance (Glaser & Glaser, 1981; Taylor, 1977). The Stroop interference, at least in part due to conflicts at the response stage (MacLeod, 1991), peaks when the target slightly precedes the distractor for up to 100ms under randomized conditions (Schooler et al., 1997). Temporal integration also occurs for visual word recognition: letters presented in alternation are perceived as a whole word when the temporal gap between two frames is no longer than 80ms (Forget et al., 2010). Because similar temporal integration effects arise for other types of visual stimuli (e.g., words, color), we ask whether temporal integration of face parts truly reflects holistic processing.

To differentiate the sources of various types of integration of face parts, we adopted a variation of the composite task. In most composite paradigms (e.g., Young et al., 1987; Singer & Sheinberg, 2006; Richler et al., 2008), a composite face is made from pairing top and bottom halves from different individuals. In the naming version of this task, observers have to name either the top or bottom half of a composite while ignoring the other half (Young et al., 1987). However, since a different face half is typically associated with a different name or response, the interference observed in this task could arise either from holistic processing of face halves or from response conflicts like those observed in Stroop tasks. To dissociate the potential sources of interactions arising at perceptual or response stages, additional conditions can be implemented in the composite task. In the version we use here (Richler et al., 2009b), participants first learn names for four faces. Two faces are assigned the name “Bob” and two other faces are assigned the name “Fred”. At test, a target face half (e.g., top) from one of the four learned faces is paired with an irrelevant half (e.g., bottom) from the same face or a different face. The critical manipulation is the face-name relation between the target and irrelevant halves at test (Figure 1). While the irrelevant half from the same face also has the same name as the target half (same-face/same-name; SFSN), the irrelevant half from a different face may either have the same name as the target half (different-face/same-name; DFSN), or a different name from the target half (different-face/different-name; DFDN).

Figure 1.

Sample face composites used in Experiments 1 and 2. During learning (Phases 1 and 2), participants learned names for four face composites, two “Bob” and two “Fred”. During testing (Phase 3), the irrelevant halves were recombined with the target halves to create composites. In the same-face/same-name (SFSN) condition, both the target and irrelevant halves were from the same studied face. In the different face/same-name (DFSN) condition, the irrelevant half was from a different face that shared the same name with the target half. In the different-face/different-name (DFDN) condition, the irrelevant half was from a different face that was assigned a different name from the target half.

Interference in the composite task is often measured by comparing conditions where the irrelevant half is different in both percept and name from the target half vs. where the halves are misaligned or when both halves are from the same face. However, if the effects of holistic processing and response interference both exist in the composite task and are additive, overall interference for aligned composites may reflect both perceptual interference (i.e., holistic processing) and response interference. Using the different irrelevant half conditions described above, these different types of interference can be dissociated. Holistic processing can be inferred by longer response times (RT) to name the target half when the irrelevant half is from a different face that has the same name as the target face, compared to when it is from the same face (DFSN vs. SFSN). In this comparison, response interference is minimized because the name of the target half is the same as that of the irrelevant half in both conditions (and the response key is also the same). Thus, any interference observed can be attributed to perceptual differences between the irrelevant halves from the same face vs. a different face. Response interference, on the other hand, can be revealed by longer RT to name the target half when there is a conflict in selecting or executing a response, given that the irrelevant half is perceptually different in both conditions (DFDN vs. DFSN).

Using this design, Richler et al. (2009b) found that the interference for intact upright faces arises from holistic processing and not response interference: longer RTs were observed whenever the irrelevant half was from a different face than when it was from the same face, but the names associated with the face halves did not influence the effect. Here, we ask whether temporal integration of face halves reflects holistic processing or response interference. If holistic processing is a cumulative process, information from different face halves maintained in a short-term visual buffer may become integrated into a holistic percept across time (Anaki & Moscovitch, 2007). In contrast, if facial information stored in the visual buffer is not integrated perceptually, temporal integration may instead arise during the response stage.

To examine temporal integration, the target or irrelevant face half was presented either 50ms or 200ms prior to the other half. Our first goal was to replicate the temporal integration effects found in Singer and Sheinberg (2006), where temporal integration is revealed by longer RT for DFDN than SFSN and this effect is larger for aligned than misaligned composites. Next we divided this effect into contributions from holistic processing and from response interference. In addition, if holistic processing (e.g., longer RT for DFSN than SFSN) is the source of temporal integration between face halves, this effect should also be disrupted by misalignment, consistent with the finding that misaligning face composites disrupts holistic processing (e.g., Young et al., 1987; Richler et al., 2008). In contrast, response interference (e.g., slower RT for DFDN than DFSN) may not be sensitive to misalignment (e.g., as in the Stroop tasks, Schooler et al., 1997).

Experiment 1

Method

Participants

Fifty members of Vanderbilt University (27 female; mean age: 22.5 years, SD: 4.5 years; normal/corrected-to-normal vision) were compensated $12 for participation. All participants reached at least 90% accuracy at the end of each training phase. Data from two participants whose performance was below chance in several test conditions were excluded from further analyses.

Stimuli

For each participant, five face tops and five face bottoms from the Max-Planck Institute face database were randomly combined into five composite faces. Name assignment was counterbalanced across participants, with two of the composites assigned the name “Bob”, and two composites named “Fred”. The fifth composite was not assigned a name and was only used during testing.

Aligned composites subtended 4°×3° of visual angle and a white line 2mm thick separated top and bottom halves. Face halves were presented on a gray background. For misaligned composites, the top half of the composite was moved leftward and the bottom half was moved rightward, such that the side of one face half fell in the middle of the other face half.

Procedure

The experiment was conducted using Matlab on Mac minis with 19” CRT monitors with 1024×768 pixel resolution.

In Phase 1 (whole-face learning), participants learned the names of four whole composite faces. All four faces and their assigned names were first displayed on the screen for participants to study for as long as they wanted. Training trials began when participants terminated this study screen. On each trial, a fixation cross (500ms) was followed by a face. Participants were told to press “1” if the face was assigned the name “Bob”, and “2” if the face was assigned the name “Fred”. All participants completed two blocks of 40 trials. If accuracy was 90% or higher, participants moved on to Phase 2. Otherwise, participants completed another block of 40 trials until this criterion was achieved, up to four additional blocks.

In Phase 2 (half-face learning), participants were trained to name face halves. Training was identical to Phase 1, except that a face half was presented in isolation on each trial. Participants named top halves until criterion (90% accuracy) was reached, then repeated the training with bottom halves. This training was included to ensure that names were strongly associated with each learned half (Richler et al., 2009b).

In Phase 3 (testing), the four faces were first presented again on the screen with their assigned names. Test trials began when the subjects terminated the study screen. On each trial, a fixation was presented (500ms), followed by an isolated target face-half or a composite face with one half cued as the target. Composites were composed of a target half and one of the possible irrelevant halves with respect to the target half. Notably, either the target or irrelevant half would be presented 50ms or 200ms prior to the other half. The response cue appeared at the onset of the first face half, even if the target half itself would not appear for another 50 or 200ms. Participants were told to indicate the name of the target half as fast and accurately as possible, while ignoring the irrelevant half. They were not asked to wait for the irrelevant half, to encourage them to ignore it if possible. Face composites were either spatially aligned or misaligned and were presented until a response was made to a maximum of 5 seconds. RT were measured from the onset of the target face-half.

Alignment conditions (aligned/misaligned) were blocked, with the presentation order counterbalanced across participants. There were 8 blocks of trials within each alignment condition, with alternating top-naming and bottom-naming blocks (4 blocks each). There were four stimulus onset asynchrony (SOA) conditions (−200ms/−50ms: the irrelevant half preceded the target half for 200ms or 50ms, and 50ms/200ms: the target half preceded the irrelevant half for 50ms or 200ms) and four irrelevant half conditions (SFSN/DFSN/DFDN/unfamiliar face¹). SOA and irrelevant half conditions were randomized. Note that since both temporal and spatial misalignments were involved, we used blocking and response cues to make sure that participants would not be confused about which half they should respond to. There were a total of 544 trials.

Results

Training performance in Phases 1 and 2 is reported in Table 1. RT in Phase 3 were log₁₀-transformed and analyzed with extreme RT excluded (<200ms or >3s; 1.26% of trials). Mean correct RT for Phase 3 is shown in Figure 2A.

Table 1.

Mean accuracy and correct RT in the last blocks for Phases 1 (whole learning) and 2 (part learning) in Experiments 1 and 2. Standard deviations are included in parentheses. Note that the training criteria were 90% and 95% accuracy with minimum training blocks of 2 and 3 in Experiments 1 and 2 respectively.

Experiment	Learning Condition	No. Blocks to Criterion	Accuracy (% Correct)	Correct RT (ms)
1	Whole	2.36 (.88)	97.49% (2.63%)	1016.30 (264.41)
	Part	2.81 (.80)	95.90% (2.67%)	868.41 (162.19)
2	Whole	3.59 (.98)	97.76% (1.80%)	770.68 (125.01)
	Part	3.44 (.80)	97.68% (1.89%)	734.71 (112.51)

Figure 2.

A) Mean RT in Phase 3 (testing) in all Irrelevant Half conditions across different Stimulus Onset Asynchrony conditions (SOA) for aligned and misaligned composites in Experiment 1. Negative SOA indicates that the irrelevant half was presented first; positive SOA indicates that the target half was presented first. B) To emphasize the effect of holistic processing, the differences between DFSN vs. SFSN in all SOA conditions for aligned and misaligned trials are plotted. C) To emphasize the effect of response interference, the differences between DFDN vs. DFSN in all SOA conditions for aligned and misaligned trials are plotted. The asterisks indicate significant effects (with corrections for multiple comparisons). Error bars represent standard error of the mean.

To separately examine the effects of temporal integration, holistic processing and response interference, three 4 × 2 × 2 repeated-measures ANOVAs were conducted on RT in Phase 3. Each ANOVA involved the factors SOA (−200ms/−50ms/50ms/200ms), Alignment (aligned/misaligned), and Irrelevant Half Condition (SFSN/DFDN for temporal integration, SFSN/DFSN for holistic processing, DFSN/DFDN for response interference). Scheffé's tests were used to follow up significant interaction effects.

Temporal Integration (SFSN vs. DFDN)

Replicating Singer and Sheinberg (2006), the ANOVA revealed a main effect of SOA (F_1,141=31.55, MSE=.0011, ${\tilde{\eta }}_p^2$, p<.0001), with shorter RT when the irrelevant half preceded the target half (−200ms/−50ms) than when the target half was shown first (50ms/200ms) (Scheffé's, ps<.05). RT was also shorter for misaligned than aligned composites (F_1,47=4.40, MSE=.0079, ${\tilde{\eta }}_p^2$, p=.04). The difference between SFSN vs. DFDN approached significance (F_1,47=3.29, MSE=.0015, ${\tilde{\eta }}_p^2$, p=.076). Critically, there was an interaction between the Irrelevant Half Condition and Alignment (F_1,47=6.84, MSE=.127, ${\tilde{\eta }}_p^2$, p=.012): longer RT for DFDN than SFSN was found for aligned (Scheffé's, p<.002) but not misaligned composites (Scheffé's, p>.54). Also, the interaction between SOA and Alignment was significant (F_3,141=3.20, MSE=.001, ${\tilde{\eta }}_p^2$, p=.025), revealing larger SOA differences for aligned than misaligned composites. No other results were significant (Fs<1.6, ps>.19).

Holistic Processing (SFSN vs. DFSN)

For holistic processing, a significant main effect of SOA (F_1,141=32.34, MSE=.0011, ${\tilde{\eta }}_p^2$ p<.0001) revealed shorter RT when the irrelevant half came first or when the target half appeared first for 50ms, compared to 200ms (Scheffé's, ps<.05). The interaction between Irrelevant Half Condition and Alignment was significant (F_1,47=4.75, MSE=.0014, ${\tilde{\eta }}_p^2$, p=.034). Surprisingly, there was no difference between SFSN and DFSN for aligned composites (Scheffé's, p>.46) but overall RT was shorter for DFSN vs. SFSN for misaligned composites (Scheffé's, p=.027). The significant interaction between SOA and Alignment (F_3,141=3.81, MSE=.0012, ${\tilde{\eta }}_p^2$, p=.012) revealed larger SOA differences for aligned than misaligned composites. No other results were significant (Fs<1.40, ps>.24).

Response Interference (DFSN vs. DFDN)

For response interference, a significant main effect of SOA (F_1,141=28.33, MSE=.0013, ${\tilde{\eta }}_p^2$ p<.0001) revealed shorter RT when the irrelevant half appeared first (Scheffé's, ps<.05) RT was also shorter for misaligned than aligned composites (F_1,147=9.82, MSE=.0073, ${\tilde{\eta }}_p^2$ p<.003) and for DFSN than DFDN (F_1,147=13.59, MSE=.0009, ${\tilde{\eta }}_p^2$ p<.0006) Critically, the interaction between SOA and Irrelevant Half Condition (F_3,141=2.99, MSE=.0011, ${\tilde{\eta }}_p^2$ p=.033) revealed that response interference was significant when the target preceded the irrelevant half for 50ms (Scheffé's, p<.001) but not for other SOAs (Scheffé's, ps>.54). No other results were significant (Fs<.97, ps>.41).

Discussion

We replicated the temporal integration effect for aligned composites (Singer and Sheinberg, 2006). This effect was reduced for misaligned composites. While holistic processing and response interference may both contribute to the effect at different SOAs, our results indicate that the integration for aligned composites cannot be accounted for by holistic processing: when irrelevant face halves shared the same name as the target, there was no significant disadvantage for a face half from a different face. However, a reversed holistic effect was found for misaligned composites, presumably because the temporally and spatially separated face halves get assigned to different tokens rather than integrated into a unified whole.

In contrast, response interference was observed when the target half was presented 50ms prior to the irrelevant half, regardless of alignment. This is in sharp contrast to the holistic effect observed for intact faces (Richler et al., 2009b). Experiment 2 directly examines the possibility of a double-dissociation between holistic processing and response interference for these two conditions.

Experiment 2

Method

Participants

Fifty-five members of Vanderbilt University (31 female; mean age: 25.3 years,SD: 6.4 years; normal/corrected-to-normal vision) were compensated $6 for participation. Data from six participants who did not reach the training criterion (95% accuracy, see below) were discarded. All remaining participants performed above chance in all conditions.

Stimuli and Procedures

All stimuli and procedures were identical to Experiment 1 except for the following changes. During learning, the training criterion was raised to 95% accuracy with a minimum of 3 training blocks for each phase to match that in Richler et al. (2009b)². During test, two SOA conditions (0 and 50ms) were blocked and counterbalanced across participants to prevent potential contextual influences.

Results and Discussion

Training performance is reported in Table 1. RT in Phase 3 were log₁₀-transformed and analyzed with trials excluded according to the same criterion as in Experiment 1 (0.6% of trials). Mean correct RT for Phase 3 is illustrated in Figure 3A.

Figure 3.

A) Mean RT in Phase 3 (testing) in all Irrelevant Half conditions across the two SOA conditions for aligned and misaligned composites in Experiment 2. B) To emphasize the effect of holistic processing, the differences between DFSN vs. SFSN in the two SOA conditions for aligned and misaligned trials are plotted. C) To emphasize the effect of response interference, the differences between DFDN vs. DFSN in the two SOA conditions for aligned and misaligned trials are plotted. The asterisks indicate significant effects (with corrections for multiple comparisons). Error bars represent standard error of the mean.

Three 2 × 2 × 2 repeated-measures ANOVAs were conducted on RT in Phase 3. Each ANOVA involved the factors SOA (0/50ms), Alignment (aligned/misaligned), and Irrelevant Half Condition (Composite effect: SFSN/DFDN, Holistic processing: SFSN/DFSN, or Response interference: DFSN/DFDN). Bonferroni-corrected planned comparisons were conducted to examine the effect of holistic processing and response interference for aligned and misaligned composites at 0 and 50ms.

In all three ANOVAs, the main effects of Alignment, Irrelevant Half Condition and the interaction between these two factors were significant (Fs_1,48≥3.95, ps≤.05). The 3-way interaction between SOA, Alignment and Irrelevant Half Condition was significant for holistic processing (F_1,48=5.18, MSE=.019, ${\tilde{\eta }}_p^2$, p=.027) but not for response interference (F_1,48=2.21, p=.14). No other results in the omnibus ANOVAs were significant (Fs<2.11, ps>.15). Planned comparisons revealed that holistic processing (SFSN vs. DFSN) was only observed for aligned composites at 0 (p<.02) and response interference (DFSN vs. DFDN) was only observed for aligned composites at 50ms (p<.04). No other comparisons were significant (ps>.34).

These results confirm a double-dissociation for aligned faces: holistic processing is consistently more important for face halves presented simultaneously and not found for temporally separated face halves. Instead, response interference is reliably more important for temporally separated halves. For misaligned composites, unlike Experiment 1, no reversed holistic effect or response interference was observed, suggesting that interference for misaligned composites is less reliable than for aligned composites.

General Discussion

Integrative processing is thought to be stronger for faces than non-face objects (e.g., Farah et al., 1998), but the contributions of different types of integration have rarely been closely examined. Here we distinguished contributions from holistic processing and from response interference. Our results suggest that holistic processing is mainly engaged when all parts of a test face are shown simultaneously and in the familiar configuration. The interaction between temporally separated face parts instead arises at the response stage. This is consistent with findings in temporal Stroop tasks (Glaser & Glaser, 1981) and word recognition tasks (Forget et al., 2010), suggesting domain-general mechanisms in response interference.

Our finding that holistic processing fails to operate when parts are presented separately in time may be due to the fact that our presentation conditions do not support natural eye movements. During free viewing of a face, eye movements can play an important role in the encoding of facial features (Henderson et al., 2005). However, faces can be processed holistically in the absence of eye movements (Richler et al., 2009a) and extensive eye movements may be necessary only when faces are relatively close to the observer. Interestingly, recent work suggests that holistic processing drops sharply with increasing size at such near distances (McKone, 2009). Our results suggest a possible reason for this: to the extent that large faces require several fixations, the need for temporal integration may limit holistic processing.

Our findings may help explain temporal integration observed in other paradigms. For instance, Anaki and colleagues (2007) presented parts of a face in a brief sequence and found better performance for upright than inverted orientation. Because the percept and response are confounded in those studies, the integration effects may instead be accounted for by response facilitation, since all parts led to the same response. Note also that even with unfamiliar faces, some response processes may still be engaged if a response is required (e.g., “this face is different from the target face”). While our methods do not directly apply to these other designs, our resutls emphasize the importance of investigating the locus of temporal integration in such cases.

Although there is debate about whether holistic processing occurs during encoding (Farah et al., 1998; Tanaka & Farah, 1993) or arises because face parts are not treated independently during perceptual decisions (Richler et al., 2008; Wenger & Ingvalson, 2002), according to both hypotheses holistic processing refers to an integrative process operating during perception, prior to response selection or execution. Our findings with intact faces are consistent with this assumption and provide important temporal constraints for models of holistic processing.