FMRI analysis of contrast polarity processing in face-selective cortex in humans and monkeys

Abstract

Recognition is strongly impaired when the normal contrast polarity of faces is reversed. For instance, otherwise-familiar faces become very difficult to recognize when viewed as photographic negatives. Here, we used fMRI to demonstrate related properties in visual cortex: 1) fMRI responses in the human Fusiform Face Area (FFA) decreased strongly (26%) to contrast-reversed faces across a wide range of contrast levels (5.3-100% RMS contrast), in all subjects tested. In a whole brain analysis, this contrast polarity bias was largely confined to the Fusiform Face Area (FFA; p < 0.0001), with possible involvement of a left occipital face-selective region. 2) It is known that reversing facial contrast affects three image properties in parallel (absorbance, shading, and specular reflection). Here, comparison of FFA responses to those in V1 suggests that the contrast polarity bias is produced in FFA only when all three component properties were reversed simultaneously, which suggests a prominent non-linearity in FFA processing. 3) Across a wide range (180°) of illumination source angles, 3D face shapes without texture produced response constancy in FFA, without a contrast polarity bias. 4) Consistent with psychophysics, analogous fMRI biases for normal contrast polarity were not produced by non-face objects, with image statistics similar to the face stimuli. 5) Using fMRI, we also demonstrated a contrast polarity bias in awake behaving macaque monkeys, in the cortical region considered homologous to human FFA. Thus common cortical mechanisms may underlie facial contrast processing across ~ 25 million years of primate evolution.

Introduction

It is well known that the polarity of luminance contrast strongly influences facial recognition. Faces of reversed (‘negative’) contrast polarity are much less recognizable than faces of normal (‘positive’) polarity (Galper, 1970; Galper & Hochberg, 1971; Bruce & Langton, 1994; Bruce & Young, 1998; Kemp et al., 1996; Russell et al., 2006; Nederhouser et al., 2007). For instance, otherwise-familiar faces are very difficult to recognize when viewed as photographic negatives (Galper, 1970).

Analogously, one previous study reported reduced fMRI activity in the vicinity of the Fusiform Face Area (FFA, although that area was not independently localized), in response to color reversal of red-green faces (George et al., 1999). A subsequent study (Gilad et al., 2009) demonstrated a reduced response in right FFA in response to contrast reversal in achromatic faces or facial regions. Moreover, a recent study (Nasr, Tootell, 2012) showed reduced recognition-related activity in FFA in response to contrast-reversed faces compared to the normal faces.

Here we further explored this intriguing contrast polarity effect in multiple experiments. In Experiment 1, we first tested for a contrast polarity bias using achromatic faces of quantitatively calibrated contrast and equal mean luminance, across a wide range of contrast levels. This quantitative information was not available from the previous literature. Experiment 1 also furnished baseline results for subsequent experiments.

Although it is often considered a single image manipulation, reversal of contrast polarity actually reverses three independent image properties in parallel: absorbance, illumination, and specular reflection. Although psychophysical studies have debated this issue (e.g. Bruce & Langton 1994; Russell et al., 2006), there are no corresponding fMRI experiments distinguishing which of these propert(ies) produces a polarity bias in the brain. Experiment 2 tested which of these three properties were necessary and/or sufficient to produce a contrast polarity bias.

Previous perceptual studies (Hill and Bruce, 1996; Liu et al., 1999) showed that the contrast polarity bias is also found in faces lacking normal textures (i.e. in face shapes without the normal covariation in surface absorbance). To systematically test for a cortical counterpart of this perceptual effect, experiment 3 measured whether the fMRI contrast polarity bias (experiment 1) is eliminated in response to such ‘shape only’ face stimuli, consistent with the psychophysics. By systematically varying the illuminant position over more than 180 degrees, this experiment also tested for ‘illuminant direction constancy’ in the fMRI responses – as one might predict if FFA responses correlate with processing of faces per se, independent of local contrast variation. Such illuminant constancy has not been reported previously based on fMRI, to our knowledge.

Psychophysically, contrast reversal has little effect on the recognition of non-face objects (Galper, 1970; Nederhouser et al., 2007). Why would contrast reversal affect recognition of faces, but not objects? One idea is that faces have certain universal features based on local contrast that facilitate recognition, whereas objects are variable enough to defy this rule. One exception to this rule is shading: by definition, shadows are always darker (not lighter) than their surround. Experiment 4 tested for a contrast polarity effect in shaded non-face objects. Like faces) these object stimuli included a fundamental lighting property.

Experiment 5 tested whether the contrast polarity bias is specific to humans, or whether it is a general feature of primate vision, including macaque monkeys. This question is especially pressing because human faces differ so markedly from faces of non-human primates.

Materials and Methods

Visual Stimuli

The face/head 3D meshes were generated using FaceGen (Singular Inversions, Canada), imported into Matlab (The MathWorks, US) using customized Matlab programs. The faces were rendered by projecting the absorbance variations onto its 3D meshes, with precise control of lighting, viewpoint, and rotation angle. All stimulus faces were 12.7° in diameter (average of width and height), in frontal view. Stimuli were presented in the scanner via a LCD projector (Sharp XG-P25, 1024 × 768 pixels) using PsychToolbox (Brainard, 1997; Pelli, 1997). All experiments included a common subset of reference faces (frontal view and gaze, centered in the visual field, using eight identities, all with a neutral expression). Stimulus conditions were organized in a block design. Blocks (16 seconds duration) were presented in semi-random order (8 different faces/block, one face/second), along with control blocks of uniform gray. Mean luminance was kept constant (253 cd/m²) for all stimuli, unless specified otherwise. Luminance contrast levels were defined as the root-mean-square (RMS) contrast across the entire face, in equal logarithmic steps ranging from 5.3 - 100%. Compared to other measures, contrast detection is improved when based on RMS (Bex & Makous, 2002).

Visual Stimuli for Experiment 2

Reversal of facial contrast changes three image properties in parallel, not just one. First, surface absorbance is reversed. For instance, the white part of the eyes (the sclera) becomes black, and the black part (the pupil) becomes white. Secondly, the effect of illumination is reversed: shadows (e.g. the nostrils) become bright rather than dark, whereas directly illuminated regions become dark instead of bright. The third property is specular reflection, which is manifest as a ‘shine’ on the facial surface. A prime example is the bright white ‘light in the eye’ (due to the liquid film on the cornea); this bright patch becomes black when contrast polarity is reversed. Moisture on the skin can produce similar specular reflections.

To distinguish which of these image properties contribute to the fMRI-based polarity bias in FFA, the following stimulus conditions were presented at both normal and reversed contrast polarity: 1) normal faces, 2) three-dimensional head/face shapes including reflection variations, but lacking absorbance variations, and 3) two-dimensional maps of absorbance variations, lacking reflection differences. The rationale for condition 2 is as follows. Reflection differences are intrinsically determined by the interaction of shape and illuminant location(s), and such differences are masked by any concurrent differences in the normal face/head. Therefore we tested reflection differences on head shapes, which lacked all absorption variation. The rationale for condition 3 was complementary: absorption differences are independent of (and masked by) concurrent variations in reflection. Such reflection variations were eliminated by rendering the absorbance variations on a 2-D sheet. In condition 2, variations in specular reflection were combined with shading variations, since both properties are differences in reflection.

In condition 4, we also tested ‘chimeric’ faces, in which facial contrast was reversed in the eye region, but not elsewhere in the face (eye-positive chimeras), and vice versa (eye-negative chimeras). A prior study using eye-positive chimeras (Gilad et al., 2009) concluded that the contrast polarity effect in FFA was strongly influenced by the contrast of the eye regions, rather than the contrast of the whole face. However that study did not test the effects of eye-negative chimeras. Here we tested that hypothesis in more detail, by comparing the effect of both polarities of chimeric face, as well as whole faces of both normal and contrast-reversed polarities. According to the previous conclusions, an eye-negative chimera should produce lower fMRI activity, compared to eye-positive chimeras.

A FaceGen face combines two components: a 3D shape mesh including vertices and facets, and a face texture. Those two components were imported into Matlab using a customized Matlab code to render a full face (condition 1 in experiment 2). For condition 2 in experiment 2, an image was generated in Matlab using a 3D mesh data from FaceGen stimuli, with a uniformed gray texture substituted for the face texture. The images for condition 3 in experiment 2 were obtained from the original FaceGen face textures, after smoothing (Photoshop) to reduce detailed shape noise. The images for condition 4 in experiment 2 were generated in Photoshop, as illustrated in Gilad et al. (2009).

Visual stimuli for experiment 3

This experiment tested the effects of variation in facial illumination. We used face shapes rather than faces with texture, to better isolate the effects of illumination apart from texture confounds. The location of the virtual illuminant was varied in 22.5° steps, along a 225° arc from top-lit to bottom-lit and beyond. Because changes in illumination location usually change the mean luminance of the illuminated object, we tested the illuminant effects in two ways, using 1) head shapes (reflection maps) with normal, uncorrected lighting variation (i.e., unequal mean luminance), and 2) head shapes in which mean luminance was equated.

Visual stimuli for experiment 4

Three-dimensional artificial objects (‘blobs’) were computer-generated in two steps. First, a given orientation was added to the 2nd and 3rd harmonics (3D shapes with either two or six equally-spaced convex lobes, respectively), and to a sphere and the 4th harmonic of a sphere, using Matlab. Second, by rotating the 2nd and 3rd harmonics, a toroidal space of smooth, asymmetric 3D blobs was created (Yue et al., 2006). Then the 3D mesh of each blob was imported from Matlab to Blender (www.blender.org) for rendering as an image with or without shadow.

Visual stimuli for experiment 5

The experiment design was analogous to that in the experiment 1. However the monkeys were presented with monkey faces, instead of human faces.

Task, human subjects

A fixation dot was present on the center of all face stimuli and the baseline (uniform gray) images. In addition, a small probe dot was superimposed on all stimuli (face stimuli and uniform gray), throughout functional scanning. The subjects’ task was to detect the probe dot during central fixation throughout the functional scanning, using a button box in the scanner. The probe dot appeared at unpredictable times (100 ms random shift from each stimulus onset), distributed randomly across the display with equal spatial probability. The timing of the probe dot presentation was unrelated to the timing of the face presentation (e.g. Yue et al., 2011). The detectability of the probe dot was manipulated by slightly varying its low/high luminance ratio (decreased local contrast = decreased detection). Threshold was modulated by the staircase method, converging on 75% correct. To reduce response variability, the dot size varied with eccentricity (Yue et al., 2011).

Human subjects

All subjects (n = 29) gave written consent. The number of subjects in each experiment is given in the Results. The experimental protocol was approved by the Institutional Review Board of Massachusetts General Hospital. Subjects had normal or corrected-to-normal vision, and radiologically normal brains. Supplementary Table 1 furnishes additional details of subjects used in different experiments.

Human Imaging

All scanning was done in a 3T Siemens Trio. Functional scans used a gradient echo EPI sequence (TR = 2s; TE = 30 ms; Flip angle = 90 degree; 33 slices with in-plane matrix size 64 × 64. Voxel size is 3 mm isotropic). For each subject, high-resolution anatomical scans (TR = 2530 ms; TE = 3.39 ms; Flip angle = 7 degree; Voxel size is 1 mm isotropic) were also collected for cortical surface reconstruction.

fMRI data Analysis

Each individual brain was inflated using FreeSurfer (http://surfer.nmr.mgh.harvard.edu). Statistical analysis of the functional data was performed with the Free-Surfer Functional Analysis Stream (FS-FAST) (Friston et al., 1995). All functional images were pre-processed with motion correction, slice-timing correction, spatial smoothing (a 5 mm Gaussian kernel), and normalized across sessions individually before submitting to General Linear Model (GLM) fitting. After those pre-processing steps, the functional data was regressed with GLM, where each condition was modeled as a convolution of a boxcar with a hemodynamic response function. The 3 motion measurements generated from the 3D motion correction were also included in the GLM fitting in addition to the experimental condition, to reduce the influence of head movement.

Average signal intensity maps were calculated for each condition, for each subject. Voxel-by-voxel statistical tests were conducted by computing contrasts based on the GLM described above. For averaging across subjects, each subject’s functional and anatomical data were spatially normalized by using a spherical transformation (Fischl et al, 1999).

Localizers

In each subject, area FFA was localized using an independent set of stimuli: group faces versus indoor scenes, of equal retinotopic extent, as described elsewhere (e.g. Tootell et al., 2008; Rajimehr et al., 2011; Yue et al., 2011). MR levels during baseline conditions (an otherwise uniform gray screen including a small central fixation point, 32 seconds duration) were measured at the beginning and the end of each run. The intervening 10 conditions (16 seconds/condition) were presented without intervening baseline conditions. Therefore, each run lasted 224 seconds. Each subject had 8 runs. Each image (scenes or faces) was presented for 1 second. Each condition included 8 images, repeated twice per condition. The order of stimulus conditions (faces vs. scenes) was semi-random. During scanning, the subjects were required to press a key with their index figure on an fMRI-compatible button box every time the fixation color changed from red to green, or green to red. Subjects were instructed to keep their fixation on the fixation dot. As defined here, FFA was bounded by non-face-selective regions dorsally, ventrally and anteriorly, and the terminus of the fusiform gyrus posteriorly. By those criteria, FFA was easily localized in all hemispheres tested in this study. The group averaged map of FFA is presented in Supplementary Figure 1.

Evidence for a topographically-distinct Occipital Face Area (i.e. OFA) (Gauthier et al., 2000) was less clear. Although activity biased for faces (relative to scenes) was often found in the location reported for OFA (averaged Talairach coordinates: (−38, −80, −11) for left OFA, (36, −76, −9) for right OFA based on Steeves et al., 2006; Rossion et al., 2003; Pitcher et al., 2007; Goffaux et al., 2011), such face-selective activity was topographically somewhat scattered and variable across subjects and across hemispheres, at least in our results. Similarly variability in OFA has been noted previously (for review see Pitcher, Walsh, & Duchaine, 2011). Given this variability, ROIs for presumptive OFA were defined here as discrete patches of activity located within ~4 mm of the averaged Talairach given above, defined in the brain volume, using a threshold of p < 10^-3. The number of voxels included in the OFA ROIs ranged from 9 to 25. These volume-based patches were then translated onto the cortical surface. Subsequent ROI analysis is as described for other visual areas. Based on these criteria, presumptive OFA activity was found in 23 of the 28 hemispheres in which OFA activity was relevant and measured (e.g. experiment 1).

Monkey face-selective regions were defined using the same localizer used in the human experiments. Thus this localizer was equated for lower level cues but differed between species based on higher-level cues. Supplementary Figure 2 shows these face selective regions from the two monkeys scanned for this study.

In experiment 4, which probed the responses to non-face objects, the lateral occipital complex (LOC) was also localized using an independent set of stimuli: gray scale objects versus a scrambled version of these same objects (11.5° in averaged diameter). For each subject, data in this experiment was based on 8 runs with 160 seconds/run. Each condition (16 second/condition) was repeated 4 times during each run. Other aspects of the experimental design were identical to those used in the FFA localizer. Subsequent ROI analyses for LOC were based on these individually localized regions. ROIs for V1 were based on two criteria. The peripheral retinotopic limit (anterior-dorsal versus posterior-ventral, along the calcarine fissure) corresponded to the peripheral representation of the face stimuli, based on the subjects’ activity map of faces versus uniform gray activity. The V1-V2 border was defined based on earlier retinotopic data (Sereno et al., 1995; Tootell et al., 1998, Hinds et al., 2008), and the cortical anatomy (e.g. Hinds et al., 2009). The orthogonal border in V1 (the peripheral extent of the stimulus-driven activity) was defined based on the independent localizers of equivalent retinotopic extent.

In all experiments, individual voxel activity for all conditions (vs. uniform gray baseline) was calculated with a univariate General Linear Model (GLM) using the Fs-Fast. The group maps were computed with Fs-Fast using random effects analyses with an uncorrected threshold of p< 0.01. All ROI statistical analyses were performed in SPSS. All error bars in the ROI analysis plot are S.E.M.

MRI in monkeys

Two male rhesus monkeys (6-8 kg) were used in the monkey fMRI experiments. Surgical details and the training procedure are described elsewhere (Tsao et al., 2003a, b; Vanduffel et al., 2001) and summarized here. Each monkey was implanted with an MRI-compatible plastic headset. All surgical procedures conformed to institutional guidelines (Massachusetts General Hospital; National Institutes of Health). After recovery, monkeys were adapted to sit in a sphinx position inside a plastic restraining chair. They were trained to fixate a small (0.35° × 0.35°) central fixation target, and eye position was monitored using an infrared pupil tracking system (ISCAN) at 120 Hz. Monkeys were rewarded for maintaining fixation within a square-shaped central fixation window (2° × 2° in size) surrounding the fixation spot. After 20-40 training sessions, when fixation performance reached asymptote, we began functional scanning. Only scanning sessions with adequately high behavioral performance (> 90% central fixation throughout the duration of each scan) were analyzed further. Before each scanning session, an exogenous contrast agent (MION; 12 mg/kg) was injected intravenously to enhance the contrast-to-noise ratio and functional sensitivity (Vanduffel et al., 2001; Leite et al., 2002). For ease of comparison, the polarity of the MION-based MR response was inverted.

The EPI scan parameters were as follows: TR = 2 s; TE = 19 ms; flip angle = 90°; 50 slices, in-plane matrix size = 84 × 96. Voxel size was 1.5 mm isotropic. As in the human MRI experiments, anatomical scans were also collected from each monkey (TR = 2.5 s; TE = 4.35 ms; flip angle = 8°. 0.5 mm isotropic, MP-RAGE). These data were reconstructed to create the cortical surfaces for each monkey with FreeSurfer.

Results

Human behavioral performance

Across all face-based stimuli, performance on the dot detection dummy task (measured as increases versus decreases in the threshold level) did not vary between normal versus reversed contrast polarity (t₁₃ = −0.77, p = 0.46).

Experiment 1: Contrast level x contrast polarity

Face stimuli were presented at normal versus reversed polarity, at four levels of RMS contrast: 5.3%, 14.1%, 37.6%, and 100% (see Figure 1a), spanning the range of readily visible face stimuli. Fourteen subjects were tested in experiment 1. For each subject, this data was based on 10 runs of 192 seconds/run.

Figure 1.

Stimuli and results from experiment 1. a: Stimulus examples. Top row: faces of normal contrast. Bottom row: faces of reversed polarity. For both polarities, face contrast level was varied in logarithmically equal steps from 5.3 – 100%, calculated as a root mean square (RMS; see Methods). In the experimental display, all faces (including those at the lowest contrast) were readily visible, for all subjects. b: Results in FFA, averaged across subjects (n=14). At all contrast levels, the response was lower for reversed polarity faces, compared to faces of normal polarity. The differences in MR signal were statistically equivalent, across the four levels. Thus the change due to contrast polarity was additive rather than multiplicative (see supplementary Figure 1). c: Results in V1. There was no significant difference between the effects of normal versus reversed polarity, at any contrast level. The error bar indicates the standard derivation calculated from the group data. d: Inter-subject variability in FFA (n=14). For each subject, data was collapsed across the four contrast levels. The bias for normal contrast polarity was relatively consistent; no subject showed a converse bias for reversed polarities.

Although the absolute signal amplitudes were larger in the right FFA compared to the left (Supplementary Figure 5), we did not find significant differences in response properties or amplitude between hemispheres. Thus, data from both hemispheres were averaged together. The faces of normal polarity produced consistently higher activity in FFA, compared with faces of reversed polarity (Figure 1b). This difference was present at all contrast levels (F(1,13) = 113.54, p < 0.0001). The post hoc analysis showed a significantly higher response to normal polarity faces compared to reversed polarity faces at every contrast level (four pairs, all p < 0.01).

Unlike the result in FFA, V1 showed no significant difference between responses to normal versus contrast-reversed faces (t₃= 0.63, p > 0.1, Figure 1c), at any contrast level. This result is consistent with many single unit studies showing that V1 responses are driven largely by edges (local contrast variation), of either/both polarities.

In FFA, the size of the polarity bias remained roughly constant over the entire range of contrasts, rather than decreasing with contrast level. Thus there was no interaction between contrast polarity and contrast level (F(3,39) = 1.9, p > 0.1). The change due to contrast polarity was additive rather than multiplicative (Supplementary Figure 3). By comparison, the change due to contrast level was roughly log-linear, consistent with previous results (Yue et al., 2011).

To increase statistical sensitivity, we averaged the effect of polarity across all contrast levels. This analysis showed that the bias for normal face polarity was relatively consistent across subjects. In all 14 subjects tested, mean values were biased for normal contrast polarity, to some extent (Figure 1d).

The corresponding group averaged random effects maps revealed that the contrast polarity bias was stronger in FFA than anywhere else in visual cortex. At a typical statistical level (threshold = p < 10^-3.5, uncorrected), the preference for normal face polarity was effectively unique to FFA (see Figure 2 and Supplementary Figure 4).

Figure 2.

Map of activity produced to faces of normal contrast polarity, compared to reversed contrast polarity, across visual cortex. Flattened views of the left and right hemispheres are shown on the left and right side of the figure, respectively. The cortical anatomy (sulci = darker gray; gyri = lighter gray) and the activity maps were averaged across all subjects (n = 14). FFA was localized using independent face-versus-place stimuli. The resultant ROI for FFA is shown as a dashed white line (threshold = p < 0.01, random effects). The stimulus-activated portion of V1 is indicated as a solid white line. The activity bias produced by a normal polarity in the left/right hemispheres is coded in red/yellow (threshold = p < 10^-3.5) with the cluster threshold larger than 20 mm; no regions showed the converse bias to reversed polarity faces, at the equivalent thresholds. The contrast polarity bias was higher in FFA, compared to other regions of visual cortex. Local orientation of the brain axes are indicated in white (D = dorsal; V = ventral; P = posterior; A = anterior).

A ROI for right OFA could be localized in 12 out of 14 subjects. Although there was a significant effect of contrast gain (F(3, 33) = 18.39, p < 0.001) in this ROI, there was no effect of contrast polarity (F(1,11) = 6.77, p > 0.01) and no interaction between contrast gain and contrast polarity (F(3,33) = 2.55, p > 0.01) (Supplementary Figure 6a). In the left hemisphere, a ROI for OFA could be localized in 11 out of 14 subjects. In this hemisphere, there was a significant (but weaker) effect of contrast polarity (F(1, 10) = 23.19, p < 0.01) and contrast gain (F(3,30) = 15.46, p < 0.01) (Supplementary Figure 6b).

Experiment 2: Image components underlying the polarity bias

The RMS contrast level was 100% for all stimuli in this experiment. Stimulus examples from the different conditions are shown in Figure 3a. This experiment included 9 subjects. For each subject, the data was based on 12 runs of 192 seconds/run.

Figure 3.

a: Stimulus examples from experiment 2. The leftmost vertical pair (‘Face’) shows faces of 100% contrast, at normal and reversed contrast polarity (top and bottom, respectively). The normal polarity ‘face’ stimuli were equivalent to those in the 100% contrast condition in experiment 1. In the adjacent stimulus pair (‘Reflection’; second from the left), variations in surface reflection (shading and specular reflection) were extracted from the normal ‘face’ stimuli, and presented on 3-d face/head shapes. Normal polarity images (normal illumination) are illustrated in the top; images based on contrast-reversed illumination are shown below. The next pair of stimulus examples (‘Absorbance’; second from the right) show the 2-d map of surface absorbance from the original faces, after removal of the 3-d reflection cues. Again, normal polarities are shown above, and reversed polarities are shown below. The rightmost stimulus pair shows ‘chimera’ stimuli (e.g. Gilad et al., 2009), in which the contrast polarity of the eye region was reversed, relative to that in remaining regions of the face. The top example has reversed contrast polarity in the eye region, and normal polarity in remaining face regions. In the bottom chimera, these polarities were reversed. b: Polarity index for all four classes. The polarity index is high in the normal faces, consistent with the data in Figure 1 and 2. However the index is not significantly different from unity (chance) for any of the other three conditions. c, d: Activity in response to the four different stimulus types, relative to a uniform gray field of equal mean luminance, in FFA and V1, respectively.

Responses from FFA and V1 are shown in Figure 3c, and 3d, respectively. To isolate the contribution of FFA (relative to V1), we defined a contrast polarity index as ((FFA_pos – FFA_neg)/(V1_pos-V1_neg)). An index of one would indicate no polarity effect in FFA, compared to the baseline responses in V1. This normalized polarity index (Figure 3b) corrected for the variation in contrast polarity effects at higher cortical levels (e.g. FFA) which passively reflect activity differences present at lower levels (e.g. V1) (Yue et al., 2011).

As expected from Experiment 1, Figure 3b showed a high polarity index (a robust contrast polarity bias) for the normal faces (t₈ = 3.82, p < 0.01). Normal faces were the only condition that included all three lighting properties at normal polarity. In comparison, the polarity index did not differ significantly from chance (unity) for the other three conditions, which were based on a subset of these image properties (condition 2, t₈ = 0.44, p > 0.1, and condition 3, t₈ = −0.20, p > 0.1), or internally inconsistent polarity (condition 4, t₈ = −0.41, p > 0.1). According to this index, a polarity bias was produced only by an internally consistent combination of all the component image properties: absorbance, shape, and specular reflection. However this ‘synergistic’ conclusion is less apparent if the FFA responses are considered in isolation, without correction for the variation in V1 responses. The OFA ROI did not show any significant contrast polarity bias in any of the conditions, in either hemisphere.

Our data was consistent with that of Gilad et al. (2009), in that FFA responses for eye-positive chimeras were statistically indistinguishable from responses to normal faces (t₈ = 3.24. p > 0.01). However we also found that responses to eye-negative chimeras were not lower than responses to eye-positive chimeras (t₈ = 0.56, p > 0.01) - which indicates that facial regions outside the eye region also contribute to the contrast polarity effect. This conclusion is consistent with results from single unit recordings in macaque (Ohayon, Freiwald, & Tsao, 2012).

Experiment #3: Variations in illuminant location

Examples of the test stimuli are illustrated in Figure 4a and Supplementary Figure 8. These stimuli were tested in a total of 10 subjects (n=6 for stimulus set 1, and n=4 for stimulus set 2). Each subject participated in 12 runs of 208 seconds/run.

Figure 4.

a: Examples of the stimuli in experiment 3, set 1, at normal contrast polarity (top row) and reversed contrast polarity (bottom row). Face/head shapes were equivalent to those in Figure 2, in the ‘Reflection’ condition, except that here, the location of the virtual illuminant was varied. The location was varied in 22.5° steps, from directly above (0°) through directly below (180°), and beyond (225°), along a plane oriented parallel with the line of sight (90°). For brevity, this illustration includes only angular changes of 45°. In this stimulus set, mean luminance was not equated; thus the strongly shaded faces are darker overall. The full set of experimental stimuli (shown in Supplementary Figure 6a, b, c, d) included faces of normal and reversed polarity, both corrected and uncorrected for mean luminance. b: FFA responses from experiment 3. Results from stimulus sets 1 and 2 (c.f. supplementary Figures 4a and 4b) were statistically equivalent, so those two datasets are combined here. Responses were essentially equivalent across the 180° range of frontal-through-profile illuminant locations, relative to a baseline condition (uniform gray field). At greater angles, responses decreased as the head shapes were increasingly silhouetted (‘backlit’), in which shape features were not highlighted.

Consistent with the above results at a single illuminant location (experiment 2, condition 2), we found no contrast polarity bias at any illuminant location tested (for stimuli set 1: F(1,5) = 0.02, p > 0.1; for stimuli set 2, F(1,3) = 0.03, p > 0.1). Despite the enormous variations in the local contrast and mean luminance of these stimuli (see 4a and Supplementary Figures 8 and 9), responses were remarkably invariant in FFA (see Figure 4b and Supplementary Figure 10) and throughout visual cortex. This invariance for illumination extended a full 180°, decreasing only when the illuminant was positioned behind the face (e.g. the silhouette at 225°). This result was obtained both when mean luminance was varied (F(8,40) = 1.62, p > 0.1; Supplementary Figure 10a), and when it was not (F(8,24) = 1.16, p > 0.1; Supplementary Figure 10b). This remarkably robust illuminant constancy is consistent with the general idea that FFA is involved in face processing, because it eliminates one otherwise-complicating variable.

ROIs for OFA did not show a contrast polarity bias across illumination angles, in either hemisphere.

Experiment #4: Contrast polarity bias in non-face objects?

To test whether the contrast polarity bias generalizes to non-face objects, we presented computer-generated objects at positive versus negative contrast. These objects (‘blobs’) were based on lower-order image statistics matched to those of faces (Yue et al., 2006). To emphasize the universal shading properties on such objects, explicit shadows were added in the background in half of the stimulus conditions (see Figure 5a). Experimental procedures were otherwise identical to those in experiments 1-3. This experiment included 8 subjects. Each subject participated in 12 runs of 160 seconds/run.

Figure 5.

Stimulus and results from blob experiment. a: Blob stimuli. Blobs are shown in normal contrast in the top row, and in reversed contrast in the bottom row. Blobs in the left column are shaded but without shadows on the background. Shadows are included in the blobs in the right columns. b: The blobs did not produce a contrast polarity bias in LOC, either with or without background shadows.

Overall these stimuli did not show a bias for normal contrast polarity, in LOC or elsewhere in cortex (F(1,7) = 5.69, P > 0.01) (see Figure 5b and Supplementary Figure 11). These results are consistent with behavioral studies showing that variations in contrast polarity do not affect object recognition (Galper, 1970; Nederhouser et al., 2007). Blobs with shadows produced a slightly larger response than those without shadows (F(1,7) = 10.28, p = 0.015), but only in those specific retinotopic representations predicted by the additional retinotopic extent of the shadows per se (e.g. Supplementary Fig 3 of Yue et al., 2011). Thus shading variations did not affect the fMRI responses in FFA, on either face shapes or non-face shapes (experiments 3 and 4, respectively).

Experiment #5: Facial contrast polarity in the macaque

Finally, we tested whether the normal face contrast polarity bias is unique to humans, or a feature shared in common with non-human primates. FMRI experiments were conducted in two trained macaque monkeys (see Methods). Multiple runs were collected from each monkey across 2-4 weeks (40 runs for the first monkey and 43 runs for the second monkey). Runs were discarded during the data analysis when contaminated by poor fixation during the scans, or napping, or large body motion-induced shim distortions. Each run lasted 192 seconds. Each condition (16 second duration) included 8 monkey faces with a neutral facial expression, and each face presented for 1 second, and repeated twice. 3648 volumes were used for data analysis for the first monkey, and 3840 volumes for the second monkey. Like the human subjects, the monkey subjects were awake, with each monkey fixating the center of frontal views of faces throughout the functional scans. Faces were presented at both normal and reversed contrasts, at four different contrast levels, identical to those used in the human experiments. In the monkey experiments, we used monkey faces as stimuli, rather than human faces. Thus both the human and monkey subjects viewed faces of their conspecifics, so that the face stimuli were matched evolutionarily. In terms of lower level cues, the monkey faces differ significantly from human faces (see Figures 6 and 7a, and Discussion).

Figure 6.

Species-specific differences in the eye region, in humans (top) compared to macaque monkeys (bottom). Eyebrows (the discrete band of hair on the lower edge of the brow) are present in humans, but not in macaques. Also, during frontal gaze, the sclera (the ‘white of the eye’) is prominent in humans, but not visible (covered by the eyelids) in the macaque.

Figure 7.

FMRI tests of facial contrast polarity in awake fixating macaques. a: Stimulus examples for experiment 5. Analogous to the human faces in Figure 1a, the monkey faces were presented at four RMS contrast levels from 5.3-100% (from left to right), in both normal contrast polarity (top row), and in reversed polarity (bottom row). b: Map of fMRI activity in a flattened hemisphere, comparing normal (red-yellow) versus reversed (cyan-blue) contrast polarity. Responses to four contrast levels were combined, for each polarity. The activated region of V1 is indicated with solid white lines. Face-selective regions were localized using the same stimuli used in the human experiments. As in human FFA, the monkey homologue of FFA (dotted white line; threshold = p < 10^-10) is the prominent patch of face-selective activity anterior to retinotopic visual areas; in monkeys this region is located in the lower bank and lip of the posterior superior temporal sulcus. The center of this main face selective patch showed a clear preference for normal face polarity (yellow-red). c: Graph of activity in the main face selective patch, averaged across all hemispheres, at each contrast level tested. The format is similar to that in Figure 1b.

In the activity map in macaque visual cortex, we found a clear bias for normal face contrast polarity in a discrete patch in macaque visual cortex. Figure 7b shows the topographical relationship between this polarity-biased patch, relative to the face-selective patch based on the independent face-place localizer. The face-selective patch in the present study is the large face patch located in the posterior bank and adjacent lip of the posterior temporal sulcus, which has been previously referred to as the ‘middle’ face patch (e.g. Tsao et al., 2006), or the ‘posterior’ face patch (e.g. Rajimehr et al., 2009; Bell et al., 2009; Pinsk et al 2009), or more recently as the ‘middle lateral’ (ML) face patch (Freiwald et al., 2010). Consistent with our hypothesis, the polarity bias was largely confined to a subdivision within that main face-selective cortical patch, which is considered the likely homologue of human FFA (Tsao et al., 2003; Rajimehr et al., 2009).

To quantify this further, ROIs for this face patch were defined in each of the four monkey hemispheres, using an equivalent threshold (p < 10^-10), based on localization stimuli equivalent to those used in the human subjects. Then the contrast polarity responses were averaged from all voxels within those ROIs. The resultant data showed higher activity to normal contrast polarity across all contrast levels (see Figure 7c and Supplementary Figure 12). Consistent with the human data (Figure 1b), the differences across contrast level in monkeys (Figure 7c) were relatively constant. Those differences were also smaller in monkeys compared to those in humans. The smaller polarity bias in monkey may reflect actual species variation, or it could simply reflect sampling differences (2 monkey subjects versus 14 human subjects), and/or task differences. In any event, the overall results suggest that a bias for normal contrast polarity exists in a homologous face-selective cortical area, in both humans and monkeys.

Discussion

Contrast polarity and facial recognition

Previous studies (George et al., 1999; Gilad et al., 2009; Nasr, Tootell, 2012) have emphasized the intriguing parallel between recognition, facial contrast polarity and fMRI activity in/near FFA: both measures decrease when face contrast polarity is reversed. Indeed, the fMRI results here and previously (George et al., 1999; Gilad et al., 2009) showed a bias for normal face polarity even though subjects were not explicitly required to recognize faces. FFA may respond better to faces at normal contrast polarity simply because these stimuli are more ‘face-like’, compared to faces of reversed polarity. All normal polarity faces include consistent contrast-specific features (e.g. white sclera, dark pupils and nostrils). Standard localization experiments are consistent with this: even during passive viewing or attention to non-face stimuli, FFA is activated more by normal faces, compared to less face-like stimuli (e.g. Puce et al., 1995; Kanwisher et al., 1997; Halgren et al., 1999; Hasson et al., 2001; Haxby et al., 1999; Grill-Spector et al., 1999).

It could also be argued that subjects were recognizing faces covertly, although the scanning task required no face recognition (George et al., 1999; Nasr, Tootell, 2012), or actively competed with it (Gilad et al., 2009, and present results). However our evidence suggests that the contrast polarity bias is at least largely sensory-driven, less likely to be a covert recognition processes. We found an equivalent fMRI preference for normal contrast in the much smaller brain of monkeys, who were concentrating intensely on a fixation task for juice reward. Our human subjects also performed a competing (dummy) non-face attention task. However, our current data cannot completely rule out a possible influence of covert recognition during task performance (e.g. Denys et al., 2004).

These data suggest that the facial polarity differences produce an automatic, bottom-up bias in FFA activity, which affects facial recognition at a higher stage (e.g. the anterior face area; Rajimehr, Young, Tootell, 2009; Kriegeskorte et al., 2007; Nasr, Tootell, 2012). This supports a role for FFA in face discrimination (relative to non-face objects) – but not necessarily face recognition (relative to other faces) (Yue, Tjan, Biederman, 2006; Xu et al., 2009; Steeves et al., 2006; Kriegeskorte et al., 2007; Nestor et al., 2011; Nasr, Tootell, 2012).

The ROI analysis also found a weak but significant effect of contrast polarity in the left (but not the right) OFA. One interpretation is that the contrast polarity effect arises (perhaps less selectively) as early as OFA. Another possibility is that this result reflects feedback (top down) activity from FFA back to left OFA.

Contrast gain versus contrast polarity

In FFA, activity increased monotonically with contrast level, approximating a log-linear function for both polarities (Figure 1b). Similar contrast gain functions are present from the retina through early visual cortex (Tootell et al., 1988; Kaplan et al., 1987; Sclar et al., 1990; Tootell et al., 1995; Boynton et al., 1996). Analogous contrast gain functions have been reported in single units recorded in anterior temporal cortex from the macaque monkey, in response to macaque faces (Figure 4b; Rolls and Baylis, 1986). Presumably, the contrast gain function in FFA reflects the contrast gain function in lower level areas such as V1 (Tootell et al., 1995; Boynton et al., 1996; Kaplan et al., 1987).

One fMRI study (Avidan et al. 2002) concluded that pFS (~posterior FFA) and LO showed an ‘increasing tendency towards contrast invariance’, relative to V1. However, when that earlier data is re-plotted on a conventional logarithmic scale, it also showed a near-linear increase in pFS, similar to that presented here. The similarity between contrast gain functions in these two studies is notable, considering the many technical differences between them. For instance, the earlier fMRI study was based on responses to luminance variations of line drawings on a constant luminance background, rather than the equal-luminance contrast variations in the gray level faces tested here. Another fMRI study also reported contrast-varying responses in nearby region ‘LO’ (Murray & He, 2006). Overall, these results suggest that contrast invariance cannot be assumed in FFA, nor likely in other ventral stream areas.

By comparison, the difference in contrast polarity was essentially constant across all contrast levels tested. That is, the polarity change in FFA was additive rather than multiplicative. Towards threshold, FFA responses are increasingly driven by the sign of the contrast, and relatively less by the presence or absence of a face per se. In previous single unit studies at lower cortical levels (e.g. Williford & Maunsell, 2007), additive versus multiplicative response functions have been used to distinguish between common versus distinct inputs, respectively. Here, the additive change is consistent with the idea of distinct brain circuits: the contrast gain reflects input variations from lower levels, whereas facial contrast polarity may be computed locally, in FFA or just prior to it.

Localization

Earlier fMRI studies described higher polarity-related activity in/near the fusiform gyrus (George et al., 1999), or in right FFA (Gilad et al., 2009), but did not systematically test for activity elsewhere. Our whole brain maps showed that the contrast polarity bias was highest in human FFA (e.g. Figure 2 and Supplementary Figure 2), and in its monkey homolog (Figure 7), compared to any other region of the brain. At conventional thresholds (e.g. 10^-3 for the group-averaged human map, and 10^-10 for individual macaque map), the contrast polarity effect was essentially confined to FFA. More sensitive analyses revealed a significant contrast polarity bias in the left OFA. However, considering the large amount of data in the present sample (11,200 functional volumes), and the inverse square law of signal averaging (I = 1/d²), a very large amount of additional fMRI data would be required to resolve statistically robust effects in additional cortical areas. In any event, any hypothetical additional areas cannot logically have a polarity bias larger than that in FFA.

Parsing the polarity bias

Psychophysical studies (Bruce & Langton 1994; Russell et al., 2006) suggest a prominent role for facial pigmentation (absorbance) in the contrast polarity bias. Here, the facial polarity bias was produced in FFA only when all component cues (absorbance, illumination and specular reflection) were combined in an internally coherent manner – i.e. as in a real world face. Experiment 3 confirmed that the effect of shading (illumination) alone is insufficient to produce a contrast polarity bias in visual cortex.

This synergistic effect of the three lighting cues can be interpreted in terms of either lower or higher levels of visual processing. As described for simpler stimuli at lower levels of the visual system, the current result is reminiscent of subthreshold summation (Jancke et al., 2004), or an all-or-none response. In higher-level terms, the contrast polarity bias can be considered an example of ‘holistic’ face processing (Tanaka & Farah, 1993; Farah et al., 1998), because only the most face-like stimuli (the normal faces) produced the contrast polarity effect in FFA. Combining these two levels of explanation, non-linear (e.g. subthreshold) summation may underlie some mechanisms of holistic face processing.

Face processing in FFA

Strictly invariant responses to lower level properties are extremely helpful in face computation (Zhao et al., 2003), because each invariant response is one less complication for the overall computation. However, evidence for strictly invariant responses has been scarce in previous studies of FFA. Here, FFA responses were strictly invariant to at least one cue, the direction of illumination (Figure 4 and Supplementary Figure 10).

Based on the available evidence, different face-related image dimensions can affect FFA activity either: 1) in a graded, systematic way (size, position, contrast gain and rotation in depth; Yue et al., 2011), or 2) not at all (e.g. direction of illumination; Figures 4), or 3) in a binary manner (contrast polarity; Figures 1 and 2). For the former six dimensions, fMRI responses in FFA were qualitatively unrelated to face perception, and/or similar to those in primary visual cortex (V1); i.e. those cues were not ‘face-selective’. By comparison, the FFA response to contrast polarity does appear related to the psychophysics of face perception, at least qualitatively.

Contrast polarity bias in faces versus non-face objects

Psychophysically, the contrast polarity effect is strong for faces, but negligible for non-face objects (Galper, 1970; Nederhouser et al., 2007). This is consistent with the idea that the contrast polarity bias reflects universal features in face stimuli (white sclera, dark pupils/nostrils) - because such universal features are more rare in non-face objects. However even when we tested shaded objects (which, like faces, reflect a universal lighting feature), we found no preference for normal contrast polarity anywhere in brain, in either the non-face blob shapes (Figure 6b and supplementary Figure 8), or in head/face shapes (Figure 3). Thus the fMRI version of this universal hypothesis (that contrast polarity reversal reflects a neural sensitivity to universal features) needs to be qualified: this idea is viable only for faces, in naturalistic conditions in which shading is not the only lighting property that is reversed (e.g. Figure 3).

Evolution

Our fMRI data indicated a homologous contrast polarity bias in the corresponding face-selective region, in both macaques and humans. The simplest interpretation is that a polarity-sensitive cortical mechanism evolved at least ~25 million years ago in Old World primates, and was retained at least in macaques and humans.

However, facial features evolve over time. Consider species differences in the eye region, which is especially implicated in the contrast polarity effect in humans (Tomalski et al., 2009; Sadr et al., 2003; Gilad et al., 2009). Although many primate species share longer hair along the brow ridge, hair is confined to discrete ‘eyebrows’ only in humans (see Figure 6). Scleral visibility is also species-specific. In essentially all non-human primate species, the white part of the human eye (the sclera) is invisible during frontal gaze (see Figure 6). Thus, although the sclera and the eyebrows contribute prominently to the contrast polarity bias in human psychophysics, these uniquely human features cannot contribute to a polarity bias, if an identical hard-wired mechanism mediates this effect across many primate species.

Alternatively, the fMRI bias for normal face polarity could be mediated by a more flexible mechanism that ‘learns’ consistent visual properties, either within each individual’s early lifespan, or more incrementally during the evolution of each species. Such a model would not be limited to specific facial features in a given primate species. The within-lifespan hypothesis is consistent with the time course of face selectivity development in FFA (Golarai et al., 2007), and reminiscent of the expertise hypothesis in FFA (Gauthier et al., 2000).

Outside of the main face patch in monkeys, contrast reversed faces produced more activity than normal contrast faces (Figure 7b, Supplementary Figure 12), compared to that ratio in human visual cortex (Figure 2, Supplementary 4). However such an apparent difference across species could be influenced by multiple factors, including differences in spatial attention during the task and familiarity to contrast reversed faces. Moreover in the monkey, the contrast reversed face images may not be recognized or interpreted as faces. Consistent with the conclusions of Murray et al. (2002), early visual areas may retain information that cannot be interpreted longer than those that can be recognized at a later stage. Further experiments would be necessary to clarify this issue.

Comparison of FMRI and single units:

A recent single unit study (Ohayon, Freiwald, & Tsao, 2012) in macaque demonstrated a positive contrast polarity bias in some cells in the main face patches, which is considered a homologue of human FFA. In some respects this data is consistent with our fMRI. For instance, the fact that some of the face selective cells did not show the contrast polarity bias is consistent with our fMRI results in monkeys, which showed a positive bias restricted to a portion of the main face patch (Figure 7b), and a generally weaker bias in monkey compared to humans overall (Figure 7c). A more sensitive fMRI analysis, such as multiple voxel pattern analysis, might reveal more information that could reconcile the fMRI with the unit recording results. For example, with a spotlight search, it may be possible to tease apart the subregion(s) of the main face patches that process facial information which is invariant from contrast level and/or contrast polarity.

However in other respects the results from that study differ from the present results. For instance, Ohayon et al (2012) reported that modulating the contrast polarity of different facial regions produced a response as strong as to whole faces in 50% of the cells. Those data imply a contrast polarity bias in the fMRI response to chimeric faces. However our data from humans did not confirm that expectation. Possible reasons for this apparent discrepancy are manifold, including differences in species, stimuli, and measurement (fMRI versus single units).

Conclusion

Manipulations of contrast polarity offer a unique window into face processing. Contrast polarity selectively affects face perception and FFA activity in a qualitatively similar way, in a way that several other visual cues do not (e.g. Yue et al., 2011; current Figures 1, 2, 4). Moreover this contrast polarity bias is apparently conserved evolutionarily, which suggests that it is fundamental for facial processing. More practically, the presence of a contrast polarity bias in macaques also makes it amenable to systematic experimental dissection using classic neurobiological system tools.

Highlights.

• *)Reversal of facial contrast polarity produced fMRI decreases in human FFA, across contrast levels
• *)A similar fMRI effect was found in awake fixating macaques, in corresponding face-selective cortex
• *)This fMRI contrast bias reflected a combined reversal of three lighting properties, not just one
• *)FFA shows a strikingly constant fMRI response irrespective of illumination angle
• *)Changes in a different ‘universal’ lighting property (shading) produced no selective response