Cortical representations of bodies and faces are strongest in their commonly experienced configurations

Abstract

Faces and bodies are perhaps the most salient and evolutionarily important visual stimuli and here we show with human functional imaging that the strength of their representations depends on long-term experience. Representations were strongest for stimuli in their typical combinations of visual field and side (e.g. left-field, right-body), though all conditions were simply reflections and translations of one another. Thus, high-level representations reflect the statistics with which stimuli occur.

In natural viewing, observers typically fixate faces¹, and consequently the right side of the observed body often lands in the left visual field and the left side in the right visual field. We investigated whether this natural experience modulates the strength of face and body part representations in extrastriate cortex (Extrastriate Body Area, EBA ²; Fusiform Face Area, FFA³). While prior studies have demonstrated that training modulates responsiveness and selectivity for categories of visual stimuli ^4–8, here we focus primarily on the strength of discrimination across simple affine transformations of otherwise identical stimuli in the absence of explicit training.

In an event-related functional magnetic resonance imaging (fMRI) experiment (Supplementary Methods), 18 participants saw allocentric views of five Body Parts (shoulder, elbow, hand, knee, foot) and Half-Faces (Fig. 1; Supplementary Methods). All participants gave written informed consent approved by the National Institutes of Health Institutional Review Board. We chose allocentric views because prior studies demonstrated that right Extrastriate Body Area (EBA) responds stronger to allocentric than egocentric body views^{9, 10}. We manipulated both Side of body (left, right) and Field of presentation (left, right) by reflecting and translating the stimuli, respectively. We independently localized EBA and FFA, which were both larger and more consistent in the right than left hemisphere (Supplementary Data 1) and here we focus on data from the right (rEBA, rFFA).

Figure 1.

Experimental Design. (a) Sample stimuli from right (top row) or left side (bottom row) of the body. Left and right side stimuli are mirror images. (b) fMRI experiment. On each trial, participants indicated whether the color of the fixation cross matched that of a small circle placed on top of each stimulus (average accuracy, 94%). (c) Behavioral experiment. Same stimuli, sizes and locations as in the fMRI experiment. On each trial, participants viewed two masked presentations of exemplars from a given stimulus type and made an eye movement to the either the green or red target to indicate whether the exemplars were the same or different, respectively.

Average response magnitude in each ROI showed little differentiation between conditions (Supplementary Data 2) and to investigate discrimination for faces and body parts we used a split-half correlation method¹¹ (Supplementary Methods). For each pair of conditions, we calculated the similarity of the response patterns in EBA and FFA across independent halves of the data (Supplementary Data 3, 4).

For each combination of Field and Side we compared the within-condition correlation (e.g. hand to hand) to the between-condition correlations (e.g. hand to leg, hand to shoulder etc.) (Figure 2a, b). Greater within- than between-condition correlation indicates the pattern of response can be used to discriminate between conditions. We calculated an average discrimination index by subtracting the average between-condition correlations from the within-condition for each condition (Supplementary Data 5). Given the category-selectivity of the ROIs, we examined half-faces and body parts separately.

Figure 2.

Discrimination of body parts and half-faces. (a-b) Similarity matrices and summary plots for the right side of the body in the left visual field in rEBA (a) and rFFA (b). Each element in the similarity matrices shows the correlation between two conditions. Bar plots show the within-condition correlations (white bars) against the average between-condition correlations (gray bars) for each condition. Discrimination indices were calculated as the difference between these bars. Error bars indicate the between-subjects standard error. * p < 0.05. (c–e) Interaction of Field and Side for discrimination indices in rEBA (c) and rFFA (d) and for behavioral performance (e). Error bars indicate the between-subjects standard error. * p < 0.05. # indicates significant difference from zero (p < 0.05) and thus significant discrimination.

For discrimination between body parts in rEBA, we observed significant discrimination indices only for right body parts in the left visual field and for left body parts in the right visual field (Fig. 2c). Importantly, these combinations of Field and Side correspond to the commonly experienced configurations. This effect of experience also gave rise to a significant interaction between Field and Side (F_1,17=11.848, p<0.003). This specific pattern of the interaction (Fig. 2c) was observed in 11/18 participants. In contrast, for body parts in rFFA, we observed no significant discrimination indices and no significant effects involving Field or Side (Supplementary Data 6). A direct comparison of rEBA and rFFA revealed stronger discrimination for body parts in rEBA than rFFA (F_1,13 = 10.59, p<0.006). The same pattern of results was observed even when voxel numbers were matched between the regions (Supplementary Data 7). Thus, in rEBA, but not rFFA, there is discrimination for body parts, confirming prior suggestions that EBA contains representations of individual body parts^{12, 13}, but only for the commonly experienced configurations.

Interestingly, discrimination was not equal for all body parts (Fig. 2a and Supplementary Data 8). In particular, for right body parts in the left visual field, we observed stronger discrimination for upper compared with lower body parts (main effect of Body Part, F_4,68= 8.533, p<0.0001). Further, planned comparisons revealed significant discrimination for shoulder (t_1,17= 3.916, p<0.0005), elbow (t_1,17= 4.917, p<0.0001) and hand (t_1,17=4.946, p<0.0001), but not knee or foot (both p>0.07). Thus we observed strongest discrimination for body parts closest to the face, consistent with a long-term effect of preferential fixation of faces.

For half-faces in rFFA, we observed significant discrimination for faces from body parts (Fig. 2d). Similar to the discrimination pattern for body parts in rEBA, discrimination was strongest for right half-faces in the left and left half-faces in the right visual field, leading to a significant interaction between Field and Side (F_1,13=6.185, p<0.027). In contrast, in rEBA, there were no significant effects (p > 0.65), and overall discrimination of half-faces from body parts was stronger in rFFA than rEBA (F_1,13 = 13.376, p<0.003).

Thus, for body parts in rEBA and half-faces in rFFA, discrimination was stronger for the commonly experienced combination of field and side (see Supplementary Data 9 for other face-and body selective ROIs; see Supplementary Data 10 for data from early visual cortex and analysis of low-level stimulus differences).

To confirm that the strength of face and body part representations was contingent on field and side we conducted a separate behavioral experiment (Fig. 1c) in which 13 new participants performed a within body part or half-face discrimination in a delayed match-to-sample task (Supplementary Methods). Participants d'-scores revealed the same interaction of Field and Side (F_1,12=10.34, p<0.007) as observed for body parts in rEBA and half-faces in rFFA (Fig. 2e; Supplementary Data 11). This finding demonstrates that the effect of Field and Side evidenced in the response patterns of rEBA and rFFA impacts behavioral discrimination.

In conclusion, we have demonstrated that the strength and distinctiveness of visual representations in both body and face-selective cortex is determined by long-term natural visual experience. How the effect of field and side we observed for allocentric body views relates to egocentric body views^{9, 10}, to the putative mirror neuron system, and to body representations in parietal cortex^{14, 15} will need to be addressed by future work. Importantly, our findings significantly extend prior evidence for the role of experience in shaping the selectivity of high-level visual cortex^4–8 and suggest that representations in these regions directly capture the statistics with which complex stimuli occur.