The role of face shape and pigmentation in other-race face perception: An electrophysiological study

Abstract

Adult observers generally find it difficult to recognize and distinguish faces that belong to categories with which they have limited visual experience. One aspect of this phenomenon is commonly known as the “Other-Race Effect” (ORE) since this behavior is typically highly evident in the perception of faces belonging to ethnic or racial groups other than that of the observer. This acquired disadvantage in face recognition likely results from highly specific “tuning” of the underlying representation of facial appearance, leading to efficient processing of commonly-seen faces at the expense of poor generalization to other face categories. In the current study we used electrophysiological (event-related potentials or ERPs) and behavioral measures of performance to characterize face processing in racial categories defined by dissociable shape and pigmentation information. Our goal was to examine the specificity of the representation of facial appearance in more detail by investigating how race-specific face shape and pigmentation separately modulated neural responses previously implicated in face processing, the N170 and N250 components. We found that both components were modulated by skin color, independent of face shape, but that only the N250 exhibited sensitivity to face shape. Moreover, the N250 appears to only respond differentially to the skin color of upright faces, showing a lack of color sensitivity for inverted faces.

Introduction

The so-called “other-race” effect describes the phenomenon in which observers typically find faces difficult to recognize and distinguish from one another if they belong to an ethnic or racial group to which the observer has had little exposure (Sporer, 2001). Anecdotally, this is often expressed as a subjective impression that members of an other-race group look alike (Malpass & Kravitz, 1969; Malpass, 1981). Empirically, observers display distinct impairments in face memory for other-race faces (Meissner & Brigham, 2001), both in terms of recognition accuracy and response bias (Slone, Brigham & Meissner, 2000). In natural scenes, changes to other-race faces are detected more slowly than changes to own-race faces, despite the fact that observers attend to both face types equally (Hirose & Hancock, 2007). Beyond these basic differences in accuracy, observers also appear to process other-race faces in qualitatively different ways. For example, there is some evidence that holistic processing is not applied to other-race faces to the same degree as it is to own-race faces (Michel et al., 2006). Whole/Part effects (Tanaka & Farah, 1993) appear to be more evident for own-race faces than for other-race faces (Tanaka, Kiefer, & Bukach, 2004), and there also appears to be an overall own-race advantage for both “component” and “configural” processing (Hayward, Rhodes, & Schwaninger, 2007). In the context of visual search, other-race faces are easier to detect in a field of own-race distracters than the reverse situation (Levin, 2001; Chiao et al., 2006). This is more or less consistent with other search asymmetry results insofar as a disparity in the fidelity of target and distracter encoding tends to produce similar effects (Rosenholtz, 2001; Rosenholtz, 2004). Finally, category-contingent aftereffects have been reported for own- and other-race faces (Little et al., 2008) suggesting that distinct neural populations may support the processing of these face types.

One simple way to summarize a great deal of the behavioral literature regarding other-race face perception is to say that other-race faces do not appear to be processed by “expert” or “face-like” mechanisms to the same degree as own-race faces. That is not to say they are not perceived as faces, but rather that the efficient, expertise-based strategies adopted for the processing of own-race faces are either applied less skillfully or simply with less success to other-race faces. Conceptually, one could say that other-race faces fall outside the “tuning” of facial appearance used by whatever representation supports recognition behavior. An important question then is to determine the specificity of the neural representation of facial appearance in the context of own- and other-race face perception. We continue by briefly discussing existing results relevant to this issue, concentrating on the literature describing own- and other-race face perception using event-related potentials (ERPs) since this is the methodology we have adopted in the current study.

It is very challenging to summarize previous results concerning the electrophysiological response to own- and other-race faces. To date, multiple studies have compared ERPs to faces belonging to “in-group” and “out-group” faces, but significant variability in task design, experimental stimuli, and analysis techniques make it difficult to condense existing results into a coherent picture of how other-race face processing may differ from that of own-race faces. For example, many studies have been conducted using White observers (and thus White “own-race” faces) but in some cases a comparison between White and Black faces is made (Ito & Urland, 2003; Ito & Urland, 2005) while in others a comparison between White and East Asian faces is made (Caldara et al., 2003; Herrmann et al., 2007; Stahl, Wiese, & Schweinberger, 2008). Few studies have employed more than one “other” race type (see Willardsen-Jense & Ito, 2006 for an exception). Although it is tempting to assume that the processing of all other-race faces is the same, as yet there is no clear evidence that this is true. Furthermore, task demands vary even more dramatically across different studies. In some cases, observers are asked to categorize faces by race and/or gender (Ito & Urland, 2003) while other experiments require an old/new judgment for previously studied items (Stahl, Wiese, & Schweinberger, 2008). Other tasks have required observers to make judgments as simple as a secondary target detection task during recording (Caldara et al., 2003) and as complex as deciding whether or not to shoot at an individual who may be carrying either a cell phone or a firearm (Cornell, Urland, & Ito, 2006). Beyond these substantial differences in task, different studies tend to focus on different ERP components including the N170 (Caldara et al., 2003), the P2 (Ito & Urland, 2003), or the N250r (Herrmann et al., 2007) or adopt tools like Principal Components Analysis (Ito, Thompson, & Cacioppo, 2004), making it still more difficult to determine how different results might fit together into a coherent theoretical package. While many of these studies have reported a variety of differences between the ERPs elicited by own-race faces and other-race faces, we are left with a fairly loose confederacy of results that are not easily relatable to one another.

Given the lack of agreement regarding the nature of the ORE in electrophysiology, the primary goal of the current study was to ask a fundamental question regarding the “tuning” of face-specific ERP components along dimensions relevant to other-race face perception: To what extent do shape cues and pigmentation cues independently contribute towards the modulation of ERP responses to own- and other-race faces? We emphasize that this question addresses how physical differences in the stimulus affect subsequent processing as measured by ERP responses. This is in contrast to studies designed to investigate social or emotional responses to other-race faces (Bernstein et al., 2007), which often attempt to equalize image-level differences between faces belonging to “in-group” and “out-group” categories (Maclin & Malpass, 2001; Chiao et al. 2001). We also have not adopted a complex cognitive task in our design, concentrating instead on how image properties related to racial categorization of the face modulate the neural response in the absence of demanding task requirements. Finally, rather than isolate a particular ERP component for analysis, we present a survey of the P1, N170, P2, and N250 components, each of which have been implicated as taking part in various stages of face processing. Multiple studies have reported larger responses to other-race faces at these components (Itier, 2002; Caldara et al., 2004, Ito & Urland, 2003; Scott et al., 2006) without isolating the cues that are responsible for the differential response. Our study thus represents an attempt to characterize the neural other-race effect in more depth by teasing apart physical stimulus variables that may contribute independently to other-race face perception, concentrating on perceptual factors relevant to the ORE, and simultaneously examining multiple relevant components of the ERP response.

We chose to concentrate on the potentially separate contributions of face pigmentation and face shape to the ORE since previous behavioral work has indicated that these visual cues make independent contributions to face recognition in general (Russell, 2003; Russell & Sinha, 2006; Russell & Sinha, in press) and the social perception of other-race faces in particular (Dixon & Maddox, 2005). These behavioral results raise the important question of how the neural representation of facial appearance is “tuned” along these dimensions, which is of direct relevance to the study of own- and other-race face perception. Since perceived category membership (deciding that a face is White or Black) is a function of both shape and pigmentation data, we suggest that knowing how race-specific visual data of each type modulates behavioral and neural responses is crucial. Thus far, we have little information regarding how shape and pigmentation may separately contribute to the behavioral ORE (Bar-Haim, Seidel, & Yovel, 2008). In many studies, the visual features that define racial categories are confounded either by using natural faces as stimuli or adopting ‘race morphing’ techniques to define a continuum of faces that progress between one category and another via global modification of all differences between face types (Walker & Tanaka, 2003). These techniques have made it difficult to understand how face shape and pigmentation may separately contribute to the ultimate perceptual representation of a face as belonging to one racial group or another.

Our contribution to the characterization of the neural response to other-race faces was therefore to examine the nature of the response to faces which contain dissociated shape and pigmentation information obtained from distinct racial categories. To accomplish this goal, we employed computer-generated stimuli that permitted independent manipulation of face shape and face pigmentation via a “morphable model” of facial appearance (Blanz & Vetter, 1999). We were thus able to create faces that were completely matched for 3D shape, for example, but had race-specific pigmentation applied to their surfaces (and vice-versa). We begin by describing a straightforward behavioral experiment we conducted to ensure that the synthetic faces we created were sufficient to induce a behavioral ORE. We then continue by describing the results of our ERP analysis, which use the same images to examine the neural response to dissociated shape and pigmentation information.

Experiment 1 – A Behavioral ORE for synthetic faces with dissociated shape and pigmentation information

Before examining ERP responses to the synthetic faces we used to dissociate shape and pigmentation information, we first conducted a simple match-to-sample ORE behavioral experiment with our stimuli. This task allowed us to determine whether or not we could obtain a behavioral ORE with computer-generated stimuli, providing an important foundation for our prospective ERP results. We therefore treated this preliminary behavioral assay as a means of determining whether the faces we created had sufficient ecological validity.

An additional question we chose to build into our study was how the ORE (if evident) interacted with the well-known face inversion effect (FIE). Behaviorally, inverting a face makes it harder to recognize (Yin, 1969) and discriminate, impairs observer efficiency (Sekuler et al., 2004), and may induce a less holistic strategy relative to upright face recognition (Freire, Lee, & Symons, 2000). These behavioral consequences resemble accounts of the behavioral other-race effect as discussed above, making the perception of inverted faces a potentially useful “model system” for the perception of other-race faces (Valentine & Bruce, 1986; Valentine, 1991). Besides this, inverted faces confer additional utility as a control for low-level visual differences.

Methods

Subjects

Ten Caucasian volunteers (7 female) participated in the match-to-sample face discrimination task. All participants were between 20-35 years of age and reported normal or corrected-to-normal vision. Furthermore, all participants reported greater exposure to White faces than Black faces (No extensive exposure to a Black family member or friend over a long period of time or an extended stay in a region with a majority of Black individuals). Informed consent was obtained from all participants in accord with the ethical standards enacted by the Children’s Hospital Boston IRB.

Stimuli

Our image set was composed of 200 upright faces and 200 inverted faces. Each of these sets was further divided into four categories of 50 images each depicting: 1) Individuals with White face shape and pigmentation, 2) Individuals with White face shape and Black pigmentation, 3) Individuals with Black face shape and White pigmentation, or 4) Individuals with Black shape and Black pigmentation. These images were created using 3-D graphics software (FaceGen, Singular Inversions Inc.) that allowed for independent manipulation of face shape and reflectance data. The program’s estimates of facial appearance across racial groups are based on physical measurements of 3D faces obtained from individuals belonging to the multiple groups supported by the application’s interface. Generating random faces involves drawing samples from the distributions of faces learned from these data. When manipulating shape/pigmentation relationships, we maintained the variability of pigmentation across all conditions by always creating “hybrid” stimuli using a particular individual’s shape information and a different individual’s pigmentation information (Figure 1).

Figure 1.

We created four distinct types of faces by combining the 3D shape of White and Black model faces with White and Black skin. The figure depicts how an individual face with White shape and color (top left) could be given Black skin (top right). The bottom row depicts the complementary process by which faces with Black shape and skin color were given White skin color. “Shape” as used in the current study always refers to the real 3D form of the computer-generated head.

Procedure

The behavioral test consisted of a match-to-sample face discrimination task. All observers saw faces presented on an LCD monitor in a dark room, with faces subtending approximately 6 degrees of visual angle. All faces were presented on a medium-gray background. On each trial, observers were first presented with a sample face for 500ms (viewed frontally) followed by a delay period of 800ms, and then two simultaneously presented test faces that were visible for 500ms (also viewed frontally). One of these test images depicted the same individual as the sample image and the second of which depicted some new individual. Participants were required to select the test face that matched the sample for identity as quickly as possible. To require the observer to do more than simple template-matching to successfully carry out the task, test images were rendered with a strong side illuminant and sample images were rendered with diffuse frontal lighting (Figure 2). Observers completed 50 unique trials per condition for a grand total of 400 trials. No sample faces were repeated in the same orientation during the experiment, but each face appeared once in the upright condition and once in the inverted condition. This repetition is critical, since it ensures that low-level image properties are perfectly matched across the inversion manipulation. Observers’ accuracy in all conditions was recorded.

Figure 2.

Schematic view of our behavioral match-to-sample task. Observers first viewed a sample face for 500ms before viewing two candidate test images, each rendered from a different lighting direction than the sample. The individual depicted in the sample image was always present at test (in this case pictured at left). The change in illumination was employed between sample and test to discourage simple pattern-matching strategies.

Results

For all observers we computed the percentage of correct responses in each category. A 2×2×2 repeated-measures ANOVA was carried out with face shape (White vs. Black 3D form), face pigmentation (White vs. Black skin color), and orientation (upright vs. inverted images) as within-subject factors. Table 1 displays mean match-to-sample accuracy in all categories.

Table 1.

Average matching accuracy across subjects in our behavioral match-to-sample task. Numbers in parentheses represent the standard error of the mean.

	Upright	Inverted
White Shape/White Pigmentation	75.3% (2.4)	55.3% (3.1)
Black Shape/Black Pigmentation	76.1% (4.6)	68.6% (2.8)
White Shape/Black Pigmentation	76.9% (5.0)	68.0% (5.3)
Black Shape/White Pigmentation	76.1% (3.0)	66.9% (2.6)

The ANOVA revealed a main effect of image orientation (F(1,9)=37.8, p < 0.001) and an interaction between face shape and face pigmentation (F(1,9)=5.97, p = 0.037). A marginal interaction between orientation and face shape was also observed (F(1,9)=3.63, p = 0.089). The main effect of orientation was driven by decreased accuracy for inverted faces, while the interaction between shape and color was driven by significantly poorer accuracy for faces with white shape and color information (confirmed by post-hoc analysis of the 95% confidence intervals obtained by collapsing over face orientation). Given that performance with all upright faces is roughly equivalent, this last effect suggests that the key difference in performance in this task is mostly due to the extremely particularly poor performance for inverted faces with white shape and pigmentation. Indeed, post-hoc tests confirm that this is the only condition in which observers do not perform significantly better than chance (one-sample t-test, p < 0.05 in all other conditions, p =0.14 for inverted white-shape/white-pigmentation faces).

Discussion

The results of our behavioral experiment demonstrate the existence of an other-race effect using our novel synthetic faces that is consistent with previous literature concerning the nature of visual expertise in face processing and generic object recognition. Observers in this task display particularly poor performance for inverted faces that possess White face shape and White skin color. Put another way, when faces possess one or both attributes of the unfamiliar racial group, the impact of image inversion appears to be lessened. This finding that greater expertise leads to an enhanced inversion effect is consistent with previous studies of visual learning for both complex objects (Gauthier & Tarr, 1997) and generic visual patterns (Husk, Bennett, & Sekuler, 2007). Additionally, the finding that changing only the skin color of the faces displayed to the observer is sufficient to induce a change in behavior is intriguing. Observers are known to be less efficient in using the visual information in inverted faces (Sekuler et al., 2004), but our result demonstrates that for images rendered from the same 3D model a change in perceived race induced solely by skin tone is sufficient to further disrupt inverted face processing. Overall, the data from this task indicates that recognition performance is tuned very tightly to the faces dominating observers’ visual experience. Any deviation from a frequently-seen face category along either the shape or pigmentation dimension changes behavior, demonstrating that the underlying representation of facial appearance is highly selective.

Experiment 2 – ERP analysis of the ORE

Having established that the synthetic faces we have used to dissociate shape and pigmentation data in other-race faces are sufficient to induce performance differences related to perceived race, we continued by examining the ERP response to these faces at multiple components thought to be associated with face processing in adult observers.

We chose to examine 2 distinct ERP components, including the N170 and the N250. In each case, our goal is to examine how perceived race (as defined by either shape or pigmentation) affected the neural response. These components were selected since it has been suggested that they have some amount of selectivity for face stimuli (Itier & Taylor, 2004). The N170 is thought to reflect structural encoding of the stimulus as a face (Bentin, 1996), and the N250 hasa previously been discussed as a possible index of visual expertise (Scott et al., 2006), both of which are highly relevant to the current study (though as we shall discuss later, we cannot make some strong claims regarding expertise given the limitations of our design). At each component, we asked whether the race-specific shape or pigmentation properties of the face significantly modulated the magnitude and (where possible) the latency of response. Further, by including inverted faces in the design, it was possible to ask whether or not any observed differences were tuned to upright faces, or generic by-products of the physical patterns used. The consequences of face inversion on ERPs (specifically the N170) are well-known and widely replicated (Bentin, 1996; Rossion et al., 1999, 2000), making it possible for us to ask how the effects of shape and pigmentation compare to well-known and robust differences in the neural response to upright and inverted faces. Moreover, we note again that inverted faces offer an important control for low-level image properties and also sidestep concerns regarding the physical variability of stimuli across categories raised by Theirry et al. (2007).

Subjects

Fourteen right-handed Caucasian volunteers (12 female) participated in the ERP task. All participants were between 20-35 years of age, reported normal or corrected-to-normal vision, and reported greater exposure to White faces than Black faces according to the criteria described above. Informed consent was obtained from all participants in accord with the ethical standards enacted by the Children’s Hospital Boston IRB.

Stimuli

A subset of the stimuli used in the behavioral task were used in this experiment. Specifically, only the faces with frontal lighting were presented in this task. Otherwise, all four categories of faces obtained by crossing Black/White face shape with Black/White pigmentation were shown to observers, including both upright and inverted versions. As in the behavioral task, we included 50 trials of each condition for a grand total of 400 trials.

Procedure

During the ERP testing session, brain activity was recorded while participants viewed the computer-generated faces described above. As in our behavioral task, all observers saw faces presented on an LCD monitor in a dark room, with faces subtending approximately 6 degrees of visual angle. All faces were presented on a medium-gray background. Images were randomly selected without replacement for presentation throughout the experiment, and were displayed for 500ms. Given that our goal was to determine how perceptual factors influence the neural response to faces with varying shape and pigmentation, we opted to use a very simple orientation judgment to assure that our participants were actively attending to our stimuli. Naïve observers are capable of performing this task rapidly and at high accuracy for unfamiliar faces (Balas, Cox, & Conwell, 2007) making it a useful means of assuring that subjects look at each stimulus, but do not need to engage in highly demanding perceptual or cognitive processing during the recording period. On each trial, observers were asked to indicate the orientation of the face (“upright” or “inverted”) via a standard button box held on their lap during the task. The box was held with both hands, such that the left and right thumbs were used to depress the response buttons. Hand assignment for these responses was balanced across observers such that half of our participants indicated “upright” with the left hand and “inverted” with the right, while the remainder responded in the opposite fashion. We note that each face appeared once in the upright condition and once in the inverted condition, roughly equalizing the low-level visual information presented across orientation. Given that we have chosen to analyze the N250 component (which is known to be sensitive to repetition) we pause to consider the potential impact of this aspect of our design on the analysis of the N250. Critically, repetition effects at the N250 are most prominent for familiar faces (Pfutze, Sommer & Schweinberger, 2002), immediate repetitions of an identical stimulus (Schweinberger et al., 2002; Begleiter et al., 1995), and are essentially non-existent when repetitions occur across gaps of many trials (Schweinberger, Huddy, & Burton, 2004). Our own design uses unfamiliar faces that are only repeated across a significant image transformation (inversion) with many interleaving trials, and so we feel confident that the use of repeated stimuli does not have substantial consequences here.

Continuous EEG was recorded using a 128-channel Geodesic Sensor Net (Electrical Geodesics, Inc.), referenced online to vertez (Cz). The electrical signal was amplified with 0.1 to 100Hz band-pass filtering, digitized at a 250 Hz sampling rate, and stored on a computer disk. The data were analyzed offline using NetStation 4.2 analysis software (Electrical Geodesics, Inc.). The continuous EEG signal was segmented into 900ms epochs, starting 100ms prior to stimulus onset. Data were filtered with a 30Hz low-pass elliptical filter and baseline-corrected to the mean of the 100ms period before stimulus onset. NetStation’s automated artifact detection tools were used to examine the data for eye blinks, eye movements, and bad channels. Segments were excluded from further analysis if they contained an eye blink (threshold +/- 70μV…) or eye movement (threshold +/- 50μV…). In the remaining segments, individual channels were marked bad if the difference between the maximum and minimum amplitudes across the entire segment exceeded 80μV. If more than 10% of the 128 channels (more than 13 channels) were marked bad in a segment, we removed the entire segment from our analysis. Alternatively, if we observed fewer than this number of bad channels, these channels were replaced using spherical spline interpolation. Average waveforms from all stimulus categories were calculated for each participant (using only trials on which a correct orientation response was made) and re-referenced to the average reference configuration.

Results

We begin by noting that the behavioral data obtained during this task revealed no effects of face shape, pigmentation, or orientation on either the accuracy of the orientation judgment or the response time to make a correct judgment. Observer performance was near ceiling in all conditions and responses were made rapidly and accurately. We continued by examining only ERPs obtained after correct responses were made.

Grand-averaged ERP waveforms were inspected to identify the components of interest. For each component, sensors and time windows of interest were determined by first identifying where the component was maximal and subsequently identifying a time window that reliably captured variation across subjects (Figure 3 outlines the sensor grouping used for each component). An important distinction between the current study and some previous work (Tanaka & Pierce, 2009) is that we have selected distinct sensors to analyze the N170 and the N250 as opposed to using the same grouping to measure both components. Our motivation for doing so was simply that these sensors were those obtained by separately evaluating where each component was most prominent, which led in this case to separate electrode sites. Putting it more simply, there was not a clear N250 component for all subjects at the sites identified for the N170 analysis. We could thus either attempt to compromise and analyze both components over sensors that would likely be non-optimal for both components, or select non-overlapping groups where each component was clearly evident. We have adopted the latter strategy here, in the interests of maintaining consistent sensor selection criteria throughout our analysis.

Figure 3.

The sensor groups used to analyze the N170, and N250 components. Note that previous research has often used the same sensors to measure the N170 and the N250. We did not do so here because our electode selection criteria depended upon identifying sensors where the relevant component was maximal across conditions, leading to non-overlapping sensor groupings for the N170 and N250. We discuss this point in more detail in the main text.

We also note that In the case of the N250, some subjects did not have a well-defined peak, making it necessary to use mean amplitude to quantify response magnitude rather than the peak of the response. This is not uncommon in studies of the N250 (Tanaka et al., 2006; Tanaka & Pierce, 2009) and we therefore used mean amplitude across both the N170 and the N250 for the sake of consistency. We were however able to measure latency for the N170 based on peak magnitude within the relevant time window, though the magnitude of the peak was not used for any statistical analysis. For each component under consideration, a 4-way 2×2×2×2 repeated-measures ANOVA was carried out with hemisphere, orientation, face shape, and face pigmentation as within-subjects factors. We continue by describing the results for each component separately, specifying the sensors of interest and the time windows used in each analysis. To simplify the statistical report, we have omitted third-order and higher interactions, due to the difficulty in interpreting their meaning.

N170

We observed maximal responses of the N170 over sensors 58, 59, 64, & 65 in the left hemisphere and sensors 91, 92, 96, & 97 in the right hemisphere. Having identified these locations where the component was of greatest magnitude, we defined a time window of 120-175ms for determining mean amplitude and extracting latency at peak amplitude.

Mean Amplitude

N170 mean amplitude was significantly modulated by the pigmentation of a face (F(1,13)=9.7, p=0.008), while the effect of inversion (larger response to inverted faces) was only marginally significant (F(1,13)=3.87, p = 0.071). No other main effects or interactions reached significance. The main effect of face pigmentation was driven by a larger response to Black faces compared to White, independent of the 3D shape of the face (Figure 4).

Figure 4.

The N170 component observed at left and right hemisphere sensors of interest (numbered in the text). Waveforms for upright and inverted faces are presented in the top and bottom rows, respectively.

Latency

The peak latency of the N170 was significantly modulated by both the orientation of the face (F(1,13)=44.7, p < 0.001) and face pigmentation (F(1,13)=17.7, p =0.001). The main effect of orientation was the result of slower latencies to inverted faces (consistent with previous literature) while the main effect of pigmentation was driven by faster latencies to Black faces.

N250

We observed maximal responses of the N250 over sensors 60, 61, 66, & 67 in the left hemisphere and sensors 78, 79, 85, & 86 in the right hemisphere. Having identified these locations where the component was of greatest magnitude, we defined a time window of 230-300ms for the N250 (a time window consistent with Tanaka et al., 2006). Unfortunately, we were unable to determine robust component peaks for all subjects making it difficult to determine peak latency. We therefore have omitted this analysis from our report.

N250 mean amplitude

The mean amplitude of the N250 was significantly modulated by both the shape (F(1,13)=8.7, p = 0.011) and the pigmentation (F(1,13)=12.3, p = 0.004) of the face (Figure 5). Having White shape made N250 magnitudes less negative, while having white pigmentation made them more negative. We further observed a significant interaction between orientation and pigmentation (F(1,13)=4.7, p = 0.048) such that pigmentation differences were more prominent for upright faces than inverted ones.

Figure 5.

The N250 component observed at left and right hemisphere sensors of interest (numbered in the text). Waveforms for upright and inverted faces are presented in the top and bottom rows, respectively.

Discussion

We begin by discussing our ERP results in relation to previous work characterizing typical responses to faces in general and how our data compares to established effects in the literature. First, we do observe an inversion effect at the N170 that is consistent with the existing literature. The N170 response is significantly delayed, while the increase in magnitude was non-significant, which is not an uncommon response profile for this component. Second, the sensor groupings where maximal responses were observed for these components and the time windows isolated for analysis are also in reasonable agreement with previously reported parameters. However, an important feature of our data that is not consistent with earlier results is the absence of a hemisphere effect favoring the right hemisphere over the left. This hemispheric difference has been reported in multiple studies (Rossion et al., 2003) and we lack an a priori reason for expecting it to be absent in the current experiment. It may be related to our use of synthetic faces, or the absence of hair, or any other difference in design between our study and others. We note that may stimulus factors can influence the laterality of the N170, including presentation time (Mercure et al. 2008). Though our own stimulus duration is relatively long (500ms) it is still possible that other factors intrinsic to our design reduced the laterality of this component. For the moment, we can only acknowledge that this difference exists and continue by interpreting our results regarding shape and pigmentation with attention paid to both hemispheres.

To speak to our goal of determining the tuning of various “face-specific” components for race-specific shape and pigmentation, our results revealed differential processing of faces depending on both shape and pigmentation information, though the nature of these differences depended on the component under investigation. The N170 displayed a “shape-blind” response in that only the pigmentation of the face appeared to modulate responses. Black pigmentation speeded the N170 peak, an effect being inconsistent with the possible analogy between other-race face perception and inverted face perception. In general, the N170 to inverted faces is larger (as we observe for the Black pigmentation faces) but is delayed rather than peaking earlier (which the Black pigmentation faces appear to do here.) One must be cautious regarding the interaction of peak amplitude and latency-to-peak, but this is a preliminary piece of data suggesting that the analogy between other-race face perception and inverted face perception is invalid. We also must admit the possibility that these differences (as well as the increased magnitude of the N170 in response to Black pigmentation) may represent nothing more than purely low-level image differences that are not directly relevant to face processing considered broadly, or to other-race face processing in particular. Regardless, the “shape-blindness” of this component is intriguing. Despite physical differences in shape that are readily apparent to observers (and contributed to differential performance in our behavioral task) the N170 does not appear to encode this aspect of race-specific visual information. These results challenge some interpretations of the N170 insofar as our data suggests that the N170 cannot purely encode “2^nd-order” or “configural” face structure. This aspect of the face is not altered by our pigmentation manipulation and is thus insufficient to explain our results.

To contextualize our work relative to earlier studies, previous efforts have reported the absence of such an effect (Caldara et al., 2003), while others indicate the presence of an ORE for the N170 (Ito & Urland, 2005; Stahl, Wiese, & Schweinberger, 2008), but may differ as to the direction of the effect. Interestingly, the direction of our other-race amplitude effect is in agreement with results obtained from Asian faces (Stahl, Wiese, & Schweinberger, 2008) but goes in the other direction as results obtained using African-American faces (Ito & Urland, 2005). We suggest that our use of full-color images may be an important addition to previous work. We note that previous results have largely used only grayscale images (or not reported the color content of their stimuli) and our results strongly suggest that face pigmentation may be a critical factor determining the properties of the N170 response.

We continue by considering our results at the N250 in light of existing literature regarding other-race face processing and expertise at this component. There are multiple interesting features of our data. We find that the N250 is indeed sensitive to both the shape and the pigmentation that we expect our observers to be “experts” with, consistent with previous results regarding this component (Scott et al., 2006). Critically, at this component we also observe an important interaction between the orientation of the face and the pigmentation, such that pigmentation differences are only observed for the upright faces. While we have heretofore been forced to acknowledge that our data may reflect purely low-level process percolating up to these components, this interaction between orientation and pigmentation makes a strong case for face-specific processing at the N250 that is sensitive to pigmentation. If merely low-level processes were in play, we would expect equal differences regardless of orientation, since image-level features do not differ as a function of face orientation. By obtaining this interaction, it would seem that the pigmentation effect we have observed for the N250 may indeed be relevant to aspects of facial appearance beyond low-level image properties.

Finally, we close by noting that we do not find a simple relationship between our behavioral data from Experiment 1 and any of the data obtained from the components examined in Experiment 2. To some extent, this is not of great concern. The behavioral response produced by observers in our behavioral task likely represents the integration of multiple processes, which may mean that looking for a true neural correlate of that behavior in any one component closely associated with visual perception could prove difficult. Nonetheless, this remains an intriguing question for future work.

General Discussion

We have found behavioral and electrophysiological evidence that race-specific face shape and pigmentation may make independent contributions to the modulation of face-specific responses. We conclude by discussing important limitations of the current study and suggest avenues for future work.

The most obvious limitation of the current study is the fact that we only recruited Caucasian observers. While this is typical of other published ERP investigations of the other-race effect (Caldara et al., 2003; Ito & Urland, 2003, 2005; Ito, Thompson & Cacioppo, 2004) it also limits the scope of our results. To demonstrate a true other-race effect it is desirable to look for cross-over interactions between recognition performance and the race (or visual experience) of observers, which we are not able to carry out with our data. We thus cannot conclude that our results are necessarily due to the visual expertise of White observers, but may reflect perceptual processing that is independent of experience and subsequent expert-level processing. Still, our data demonstrates that visual features known to be directly relevant to defining racial categories (within which expertise has been demonstrated by previous studies) can independently modulate the neural response to faces. It is still fair to speak of the “tuning” of these responses in the population we have considered, since the effects we have observed still reflect the sensitivity of these components to some visual properties of the face (pigmentation) and insensitivity to others (shape, in the case of the N170). What we may not do is conclude that the underlying representation has been tuned by experience with a particular group of faces.

Besides this very substantial question of how the representation we have characterized here is acquired, there are many other factors not considered here that may additionally be relevant to the perception of own- and other-race faces, both behaviorally and neurally. First, the stimuli we generated for use in both tasks were reasonable models of typical, natural White and Black faces (and their cross-pigmented counterparts) in which skin color and luminance were confounded. This reflects the natural viewing scenario, but raises an interesting question: Is color of luminance the relevant factor? There are intriguing behavioral effects related to the perception of luminance in own- and other-race faces (Corneille et al., 2004; Levin & Banaji, 2006), but to the best of our knowledge there has been no direct assessment of how these two distinct aspects of skin appearance contribute to the other-race effect. Indeed, many previous studies have used only grayscale images, making it impossible to assess any color-related effects. Additional behavioral and electrophysiological work on this issue would further contribute to our understanding of the scope and limitations of the “expert” face representation. A second somewhat-related issue is the potential for extending this paradigm to include assessment of the ORE for a wider range of racial and ethnic categories. Not all pairs of face categories differ as dramatically in color and luminance as White and Black faces, which may mean that the ORE manifests quite differently when the perception of White faces (by White observers) is compared to East Asian, Indian, or Middle Eastern faces. By and large, there has been little effort to measure the other-race effect across multiple face categories (though see (Kelly et al., 2005) for a developmental investigation of this issue). We also point out that investigating the other-race effect in prosopagnosic patients would lend a further important perspective into how visual information is recruited to define faces as belonging to various groups. Despite over all poor abilities with faces, the extent to which prosopagnosic individuals exhibit “classic” or “hybrid” OREs as defined here would shed light into which aspects of facial appearance are inaccessible to such observers, providing key insight into the neural architecture of face processing.