The Neural Mechanisms of Face Processing: Cells, Areas, Networks, and Models

Abstract

Since its discovery, the face-processing network in the brain of the macaque monkey has emerged as a model system that allowed for major neural mechanisms of face recognition to be identified – with implications for object recognition at large. Populations of face cells encode faces through broad tuning curves, whose shapes change over time. Face representations differ qualitatively across faces areas, and we not only understand the global organization of these specializations, but also some of the transformations between face areas, both feed-forward and feed-back, and the computational principles behind face representations and transformations. Facial information is combined with physical features and mnemonic features in extensions of the core network, which forms an early part of the primate social brain.

Review

Faces, from a vision perspective, are a special object category with a specific first-order structure [1], and associated with specific motion patterns that can change the shape of the face. Because of their complexity, faces seem an unlikely model system to understand the neural mechanisms of object recognition, but, likely because of their social importance, faces are processed by specialized circuitry, and because of the organization of this circuitry the problem of the neural mechanisms of face recognition has become imminently tractable and become a model for understanding the mechanisms of object recognition at large. Face cells [2] and clusters of face cells [3] had been found across the macaque temporal lobe and in prefrontal cortex [4], seemingly intermingled with cells of very different selectivity, but functional magnetic resonance imaging (fMRI) combined with electrophysiological single-cell recordings showed a hitherto unappreciated organization of areas, several millimeter in diameter each, containing large fractions of face-selective cells [5–7].

This concentration of face cells into face areas is the first of the main principles of the functional organization of the face-processing circuit. The second is the organization of face areas into a large-scale spatial pattern that is anatomically highly conserved across individuals, and even similar between macaque monkeys and humans [5,8], with each face area exhibiting a unique functional specialization: for example, cells within the so-called middle face area ML are strongly tuned to head orientation, while cells in the anterior face area AM are strongly tuned to facial identity even across different head orientations [7]. Population codes across the different face areas are thus easily distinguishable and thus unique [7]. Furthermore, third principle, there is a reproducible overall organization of face areas along functional gradients: from posterior to anterior locations face-representations become more identity-specific, while from ventral to dorsal locations, representations become more motion-selective [9]. And the fourth principle of the functional organization of the system is that it is a network: face areas are selectively interconnected to each other [10,11].

Because of these four principles, information-processing in high-level vision can now be dissected. For example, the systematic progression of decreasing head orientation tuning to increasing facial identity tuning from posterior to anterior locations along with increasing mean response latencies from posterior to anterior areas, and the fact that face areas are directly interconnected, has lead to the proposal of this sub-network as a two-step processing hierarchy [7] (Fig. 1). Furthermore, because of the clarity of organization, the face-processing system can be used to uncover the general organization of object-recognition systems [12,13], suggesting that the computations within the face-processing system might generalize to the realm of object recognition. And, finally, this functional organization also allowed for the demonstration that the system is causally relevant for face detection and discrimination [14–17].

Fig. 1.

Top: Schematic side view of macaque brain with seven areas of face-selective cortex (red) in the temporal lobe together with connectivity graph (orange). Face areas are named based on their anatomical location: AF, anterior fundus; AL, anterior lateral; AM, anterior medial; MF, middle fundus; ML; middle lateral; PL, posterior lateral, and have been found to be directly connected to each other to form a face-processing network. Face-motion area MD, medial dorsal, is shown as well. Its connectivity with the other temporal lobe face areas is currently unknown.

Middle: Classical view of the network operating in a feed-forward hierarchy.

Bottom: Quantification of population tuning to head orientation and identity by three tuning coefficients (arbitrary units): View specificity and mirror symmetry describe the shape of tuning to head orientation, identity selectivity to identity across all head orientations. Population activity in ML is dominated by view specific representations. This tuning is still found in AL, but here a new quality emerges, mirror-symmetric tuning to head orientation. In AM, tuning to head orientation is substantially reduced, and identity selectivity dominates.

Multi-Dimensional Coding of Facial Information

Because of the clear and highly reproducible organization of the face-processing network [5], it is possible to record from the same face area across subjects. And because of the high concentration of functionally homogenous cells within fMRI-identified face areas of known stimulus category preference, it is possible to study the functional properties of face cells at great detail. Can the tuning of face cells explain major properties of human face recognition? Several results from human psychophysics can be explained by the concept of face space: a multidimensional space in which each face occupies a unique position based on its combination of physical features [18–20]. This perceptual space was found to be organized around a center occupied by the average face, which serves as a reference relative to which each face’s characteristics are represented.

How could a neural population generate a neural face space with the main properties of this perceptual space? One way would be through broad tuning curves spanning the entire space and map facial feature dimensions onto their firing rates, e.g. a one-to-one mapping through rampshaped tuning curves (axis coding). Direct evidence for this coding scheme was found in face areas MF and ML [21]. The tuning of cells was probed with a multi-dimensional cartoon face space that varied in features such as eye size, inter-eye distance, or relative face width. The majority of MF/ML cells was tuned to a small subset of features. Most cells exhibited ramp-shape tuning curves with maximal responses at or near one feature extreme and minimal responses at or near the opposite feature extreme. When cells were tuned to more than one feature dimension, tuning to the different dimensions was approximately orthogonal. Importantly, while cartoons were artificial and designed to span a very wide range of features beyond what would be encountered perceptually in real faces, the tuning to carton faces predicted tuning to the much narrower space of natural faces [21]: the tuning to cartoon face width, for example, would predict tuning to (the much smaller) variation of face widths in real faces.

These results suggest a neural explanation for a well-known effect on human psychophysics: the caricature effect [22] describes the fact that identity can be perceived more easily in caricatures by emphasizing the characteristics of a particular identity, i.e. by moving the face further away from the average. In the neural population code, such a shift would increase the heterogeneity of responses and thus the population’s coding capacity [21]. The study [21] found putative neurophysiological correlates of two further psychophysical effects: the face inversion effect [23], and the part-whole effect [24]. Just like humans are worse recognizing faces turned upside down, so tuning strength and consistency of MF/ML neurons was reduced [21]. This neural faceinversion effect appears to be confined to cells inside the middle face patches [25]. And just as humans recognize features better within a face than in isolation, MF/ML cells were less strongly tuned to isolated cartoon features than to the features placed in holes [21]. Even stronger context dependence of facial features occurs in the so-called Thatcher illusion [26]: inversion of local features, noticed easily as distortions in the upright face, are hardly recognizable in the inverted face. A putative neural correlate of this illusion was found in ML [27], where inversion of the eyes reduced the response in the upright but not the inverted face.

Given the evidence for a face-space representation in MF/ML, and given that activity in these areas drives downstream face areas, it could be expected that downstream areas would represent faces in a face space as well. Alternatively, just as head-orientation tuning is reduced from MF/ML to downstream areas, so the representation in face-space might get reduced. For example, AM might represent faces through narrow tuning curves centered around particular individuals (‘exemplar code’). A recent study found that area AM, at the top of the faceprocessing hierarchy, also uses axis coding to represent faces [28••]. While earlier levels or processing are more sensitive to coarse facial features describing overall shape, cells in AM are more sensitive to finer details that dominate facial appearance, in agreement with the increased importance of identity in this area [7]. Thus, facial identity does not appear to be coded by narrowly tuned exemplar neurons, but by broadly tuned neurons. This coarse coding scheme has been argued to be advantageous for higher-dimensional representations [29,30] like those of faces.

While coarse coding of faces is, by now, well-established, what shape this tuning takes, is less clear. In parallel to the work leading to the discovery of ramp-shape tuning [21], at a more anterior location, possibly AM, cells with a very different tuning, V or inverted V shape, were found [31]. And, similarly in ML, some cells also exhibited V-shaped tuning [21]. More recently, very strong evidence for V-shape tuning was found in multiple face areas [32••]. Interestingly, V-shape tuning took longer to develop, possibly resulting from a local normalization operation. Thus an earlier, axis-based representation, which might result directly from stimulus statistics [33•], appears to be transformed with time, through local processing, into an orthogonal representation. (A similar transformation of population codes over time has been observed in the motor system before [34].) This idea provides a mechanistic account for how the center of face space assumes the special status it has in human face perception [18–20]. Yet, these findings raise important computational questions. An axis-based code is linearly decodable [28••], while it is less clear how identity could be efficiently decoded from V-shape representations, as it maps physically distinct identities in opposite parts of face space onto more similar population representations. Some of these questions may be answered with more detailed measurements of the tuning of individual neurons. Approaches to adapt stimuli online to the preferences of a cell under study have provided deep insights into shape-selectivity within high-level visual areas [35], and recent advances using generative deep convolutional networks might prove especially powerful [36•] to determine this tuning. Whether they are able to find the multiple tuning peaks V-shape coding predicts, remains to be seen. Results obtained so far suggest that they might find as preferred stimuli, sometimes, stimuli that do not even look like a face - as predicted by the aforementioned axis coding models, requiring the best stimulus to reside in the outskirts of face space or beyond.

Transformations between the Nodes of the Face-Processing Network

As face areas interact with each other, some properties of face representations within each node of the network stay the same - all nodes are face-selective, axis-based coding prevails – while others change drastically from one face area to the other. Along the occipito-temporal axis, as we have seen, face representations are transformed from a format reflecting the great input variation of faces with changes in pose into a format reflecting the physical identity behind the image. Can this transformation be explained as feed-forward transformations? And does it require deep architectures, which have been proposed as models of object-recognition in inferotemporal cortex [37,38] and have been quite successful explaining population-level representations within this region [39,40]? A computational proof of principle was provided by a feed-forward model that explained this major transformation from MF/ML to AM [41].

Located between MF/ML and AM is another area, AL, which exhibits a peculiar tuning to head orientation: cells that are tuned to both left and right profile views [7]. Can mirror-symmetric tuning be explained by feed-forward processing? It can, and the study that demonstrated this [42••], also uncovered the putative principles behind this transformation. For mirror-symmetry to appear in feed-forward hierarchical models, mathematical analyses demonstrated, the stimulus has to exhibit intrinsic mirror symmetry, and the hierarchy needs to be wired according to a specific learning rule. The network does not have to be deep though. Consistent with this finding, deep convolutional networks that have been examined for this property, do not exhibit mirror-symmetric tuning in their convolutional layers, but only in the subsequent first fully connected layer [43••].

What is the function of top-level area AM? We have seen that it performs an inference of the physical features of the face that gave rise to the particular image in sight. Standard deep neural networks are trained on labels of facial identity, and would thus support this inference as well. But how rich can the inferences be that the face-processing system performs? A computational model that utilizes a deep neural network trained not on identity labels but to invert a three-dimensional face graphics program, can identify the latent variables that generate the specific image in a single fast feedforward pass [43••] (Fig. 2). This model provided a quantitatively better explanation for the degree of head orientation tuning, mirror symmetry, and identity selectivity than standard high-performance face-recognition networks (Fig. 2), and it also explained human psychophysics of face recognition better than standard models [43••]. Thus one of the computational goals [44] of the face-processing network might be to make deep inferences about incoming stimuli and explain their content in terms of an internal three-dimensional face model.

Fig. 2A.

Schematic illustration of two alternative hypotheses about the function of face-processing pathways: The recognition or classification hypothesis (top) and the inverse-graphics or inference network hypothesis (bottom). The first hypothesis, currently dominant in AI and neuroscience, is that perception is best approached using neural networks optimized for classification, trained to recognize or distinguish object or face identities. The second hypothesis posits that face perception in the brain is best understood in terms of an inference network that inverts a causal generative model.

Such an internal model could be generative and not only explain incoming stimuli, but also generate (“hallucinate”) them. These capabilities could be used, through feedback, to compare incoming sensory information to the best current explanation. Theories of brain function like predictive coding have proposed just that: lower-order brain areas compare bottom-up sensory inputs to top-down predictions from higher-order areas and then signal deviations from expectations, leading to more effective representations [45]. Evidence for exactly this operation has been found in the macaque face-processing system [46••]. This evidence for predictive coding was provided with a statistical learning paradigm of sequentially presented stimulus pairs that had been used before to show an unpredicted second stimulus elicited bigger responses in inferotemporal neurons than an expected one [47]. This ‘prediction error’ signal was also present in face area ML, showing that ML computes the deviation of actual from predicted stimuli. But where did the prediction signal come from? The signal was identity specific and view invariant, with hints of mirror symmetrical tuning to head orientation, thus belying its origin in higher-level areas AL and AM. Independent work demonstrated further evidence for predictive coding in the face-processing network without preceding explicit statistical learning [48••]. This study demonstrated an early differential response for typical face stimuli in a high-level area, preceding in time the inverse effect in a low-level area. Predictive feedback might thus indeed serve to generate more efficient codes in the face-processing network. The formal similarities between predictive coding and classical error backpropagation [49•] frequently used to train neural networks, suggest that the measured expectation-error signals might also reflect ongoing learning in the network.

Large-Scale Organization of Face-Processing Systems

While these two studies [46••][48••] point to high-level areas within the face-processing system as sources of feedback, another possibility exists. During presentation of natural (predictable) face motion and temporally scrambled (unpredictable) face motion, areas like ML and AL, exhibited larger responses to scrambled than naturalistic motion [9], in line with predictive coding accounts. Yet an additional area, located more dorsally inside the superior temporal sulcus, MD (Fig. 1), was particularly selective for naturalistic face motion. This dorsal face-motion area might provide another possible source of predictive feedback, especially when facial movements deviate from the predicted path of motion. Yet little is known about the connectivity of this area to the rest of the face-processing system, and if the connectivity pattern of a putative human MD-homolog is any indication [50], its connections to ventral face areas might be weak. The finding thus raises a fundamental question about the internal functional organization of the face-processing system: are there possibly multiple networks with different computational goals? And, an analogous question arises about the external organization of the system: where would putative output areas of the system transmit information to?

Surprisingly, the answer is not obvious. The initial study that found face areas to form a network [10] and a later tracer study [11] describing anatomical connectivity with high resolution, paint a picture of a largely self-contained network with only a few specific outputs, especially to the basolateral amygdala. This is likely because both approaches emphasize strong and focal projections. A complementary approach, resting state connectivity, assesses functional interactions between brain areas. This approach revealed a very different picture, one in which the face areas functionally interacted with larger sets of brain areas [51•]. Each face area showed a unique pattern of functional interactions, e.g. AM with medial temporal peri- and entorhinal cortex, but all shared a common set of brain areas they interacted with, primarily in the frontal lobe. Interestingly, this common network intersected with another resting state functional network, the default mode network [52], in three areas of the brain that are plausible homologues of those areas that in the human brain support high-level social cognition [53]. This finding suggests that face areas could be closely tied to the social brain. Another study, using a very different approach, came to a similar conclusion. Comparing the functional profile of a large number of areas to a large set of videos including those of complex social scenes, and found functional profiles of face areas to be most similar to those of social brain areas, while bodyselective areas, though directly neighboring face areas, exhibited more similarity to the mirror network [54•]. Thus face and body areas might not just be face and body areas, but also the major inputs to two different cognitive brain systems.

From Generic Face Processing to Person Recognition

As the face-processing system interacts with these other brain regions, we expect to see selectivity for combinations of features to emerge. We have already seen this in area MD with selectivity for the combination of faces and another factor, motion, paralleled by another dorsal area at a more anterior region of the STS, AF, which similarly showed selectivity for other stimulus factors [55], including selective for body-context [56]. This combined selectivity became apparent for another quality than motion, one that is at the core of face processing capabilities: the recognition of familiar individuals. Contrasting activation to familiar faces with familiar toy objects, a recent study [57]•• found two additional face areas, one in perirhinal cortex, a region long known to be critical for long-term associative memory [58], and the other in the temporal pole [57••]. To test whether these areas were genuinely selective for familiar studies, the study used a second paradigm [59] that revealed a highly blurred stimulus slowly over time (Fig. 3). Only the two new face areas generated a non-linear surge of activity akin to the moment of recognition – and did so only for familiar faces. These two new areas, probably interconnected to the core face-processing system, are strategically located to form a link between face perception and person knowledge. This is now the clearest example for how the core faceprocessing network serves as an input to the social brain.

Figure 3A.

Top:

Bottom: Stimulus-aligned fMRI time courses within face area ML (left), TP, and PR (right) during the presentation of familiar faces (red), nonfamiliar faces (blue), and objects (gray). Percent signal change shown. Error bars represent standard error. Sigmoidal functions Naka-Rushton function fit to mean time courses are shown.

Fig. 2B.

Top: The inference network (EIG, efficient inverse graphics) inverts a generative model using a cascade of deep neural networks with intermediate steps corresponding to processing stages of the ventral object recognition stream and face areas. Layer f3, the top convolutional layer, corresponds to face areas ML/MF, f4, the first fully connected layer, to area AL, and f6, the second fully connected layer, to face area AM.

Bottom: Tuning coefficients in EIG layers f3, f4, and f5, are very similar to and highly correlated with those in MF/ML, AL, and AM (Fig. 1), respectively.

Fig. 3B.

There are at least two specific additions to the core face-processing system, one for the processing of face motion and the other for the processing of familiar faces.

3-5 Highlights.

• The primate brain contains a dedicated multi-node face-processing network.

• Face cells encode the physical identity of faces as points in face space.

• Feed-forward and feed-back transformations generate new face representations.

• Computational models capture the main tuning properties in the network.

• Two additional face areas in the temporal lobe represent familiar faces.