FACE PROCESSING SYSTEMS: FROM NEURONS TO REAL WORLD SOCIAL PERCEPTION

Abstract

Primate face processing depends on a distributed network of interlinked face-selective areas composed of face-selective neurons. In both humans and macaques, the network is divided into a ventral stream and a dorsal stream, and the functional similarities of the areas in humans and macaques indicate they are homologous. Neural correlates for face detection, holistic processing, face space, and other key properties of human face processing have been identified at the single neuron level, and studies providing causal evidence have firmly established that face-selective brain areas are central to face processing. These mechanisms give rise to our highly accurate familiar face recognition but also to our error prone performance with unfamiliar faces. This limitation of the face system has important implications for consequential situations such as eyewitness identification and policing.

Face recognition, from a computational point of view, is a daunting task. But for most of us detecting a face in a scene, recognizing a friend, and noticing even minute changes in facial expression, is effortless. How is that possible? The primate brain contains specialized cells and circuitry that support face recognition, and some of the computational mechanisms these circuits implement are now becoming clear. Here, we provide a comprehensive overview of primate face processing systems from single cells to circuits and their link to behavior. We highlight the surprising limitations of unfamiliar face recognition and discuss its implications for real-world situations such as eyewitness testimony.

Neural Systems for Face-Processing

Charles Gross’ discovery of cells selective for faces provided first evidence consistent with earlier ideas about single-unit coding of person identity, which Lettvin had famously referred to as grandmother cells (Gross 2002). Gross’ finding called for an explanation: how can such complex shape-selectivity arise from simpler representations in early visual cortex (Hubel 1982)? The natural answer was, and has remained, hierarchies that transform visual information through multiple levels of processing (LeCun & Bengio 1995, Riesenhuber & Poggio 1999). What the discovery of face cells did for the concept of hierarchy, the discovery of entire face areas by functional MRI (Kanwisher et al 1997, McCarthy et al 1997) did for the concept of modularity: it reinvigorated debates on local versus distributed information processing (Haxby et al 2001, Kanwisher 2000). Face-processing systems in primates, as we understand them today, instantiate multiple organizational features, including some that had been thought of as mutually exclusive: face-processing systems are modular and distributed and face-processing appears to proceed in parallel and through hierarchies.

Distributed and Modular Organization

Face cells have been found across temporal (Perrett et al 1992) and prefrontal cortex (Ó Scalaidhe et al 1997, Rolls et al 2006) of the macaque monkey brain, suggesting that faces are represented in a distributed fashion. Local clustering of face cells (Perrett et al 1984) suggested an organization of columns (Fujita et al 1992). Later neuroimaging studies revealed a large-scale organizational feature, face patches several millimeters in diameter (Pinsk et al 2009, Tsao et al 2003, Tsao et al 2008a). These patches are primarily located laterally in the lower bank of the superior temporal sulcus (STS) and medio-dorsally in the fundus (see Figure 1A). Face patches contain very high fractions of face-selective neurons (Freiwald & Tsao 2010, Issa et al 2013, Tsao et al 2006). Whatever the computational reason for local grouping of face cells, they constitute domain-specific specialized hardware, possibly implementing functional modules (Kanwisher 2010). As multiple face areas have been found across temporal and prefrontal cortex (Tsao et al 2008a, Tsao et al 2008b), it has become clear that modular organization goes hand in hand with distributed processing.

Fig. 1. Organization of Face Processing Systems in the Macaque and Human Brain.

a. Face patches in the macaque superior temporal sulcus (STS), shown here (top) opened on a lateral view of the brain, are functionally organized along an occipito-temporal (posterior-anterior) axis and a ventro-dorsal axis (expanded view, bottom). Arrows in upper right corner indicate dorsal (D) and anterior (A) direction. General motion areas (blue) and face areas (red and purple) are shown. Ventral areas (in the ventral bank of the STS and further ventrally) are selective for momentary form of faces (red), while areas at more medio-dorsal locations in the STS (fundus and dorsal bank) are selective for natural facial motion (purple). Hierarchical processing from view selective to view invariant representation is manifested in the posterior-anterior axis: representations are transformed from view-specific into increasingly view tolerant and facial identity-selective ones.

b. Face and motion areas (colors as in a) in the human brain are found ventrally in the lateral occipital cortex and the fusiform gyrus and dorsally in the superior temporal sulcus. A functional organization of hierarchical processing along the posterior-anterior axis and sensitivity to motion along the ventral-dorsal axis is found, similar to the one in the macaque brain. The ventral areas (red) show no sensitivity to motion whereas face areas in the STS are highly sensitive to moving faces. As in the macaque brain, the transformation along the occipito-temporal axis progresses via an intermediate representation that does not differentiate between left and right profile views (bottom). View selectivity was found in the lateral occipital cortex and mirror symmetry in the fusiform gyrus and posterior superior temporal sulcus.

Face Networks, Parallel and Hierarchical Pathways

Distributed face areas do not operate in isolation, but are interconnected into a face-processing network through selective long-distance connections (Moeller et al 2008). Thus, facial information processing is integrated not only locally, but also across the larger distances separating face-selective areas. The spatial arrangement of face areas, relative timing of activation, and functional characteristics highlight two organizational principles within the network (Freiwald & Tsao 2010). First, face areas are organized along a posterior-anterior axis and response latencies systematically progress from early to late areas, suggesting hierarchical organization. Facial information appears to be systematically transformed from early view-specific representations into late identity-specific representations (Freiwald & Tsao 2010, Meyers et al 2015). In the most posterior area, PL, selectivity for a single feature, the eye, seems to dominate (Issa & DiCarlo 2012), at the next level, in areas MF and ML, a wide range of features are represented, and this selectivity is modulated by the embedding of features into the facial whole (Freiwald et al 2009). While face representations in MF and ML are view-selective, this dependence is reduced at the next level, AL, through construction of a mirror-symmetric invariance to head orientation, and reduced even further in AM, where representations for facial identity are robust against variation in head orientation (Dubois et al 2015, Freiwald & Tsao 2010, Meyers et al 2015). Concomitantly, tolerance for stimulus position and size is increased as well.

Orthogonal to this hierarchical principle of information processing, a second functional distinction between face areas along the ventral-dorsal axis exists. Face areas located in the fundus of the STS (dorsal) exhibit a pronounced preference for natural facial motion that face areas located laterally in the lower bank (ventral) lack (Fisher & Freiwald 2015a). These results suggest different kinds of facial information are processed through two parallel streams. However, given the complex connectivity between face areas (Moeller et al 2008) and the fact that information processed in either stream is likely informative for computations in the other, the parallelism is likely not one of functional isolation between streams.

The spatial distribution of face patches requires long-distance connections for them to interact, thus raising the question why the system is organized in this seemingly non-economical manner. The answer may be that the anatomical location of face patches is governed by another organizing principle, their belonging to multiple separate larger-scale object-processing maps. Such a scenario appears plausible given the curious co-localization of color patches directly ventral to some face areas (Lafer-Sousa & Conway 2013). Face and color processing share a foveal bias (Hasson et al 2002), and such retinotopic bias might explain their co-localization (Rajimehr et al 2014). Also abutting many face areas are body areas (Pinsk et al 2009, Popivanov et al 2012, Tsao et al 2003), whose co-localization with face areas may serve to support person perception.

Human and Non-human Primate Face-Processing Systems

Despite an estimated twenty five million years of separate evolution (Stewart & Disotell 1998), human and macaque face processing systems bear remarkable similarities, and some of these similarities might extend to the evolutionarily more distant New World monkeys (Hung et al 2015). In the human temporal and frontal lobes, multiple, spatially separate face areas have been found (for a recent review, see (Duchaine & Yovel 2015), Figure 1B). Like macaques, human ventral face areas are adjacent to body-selective areas (Peelen & Downing 2005, Schwarzlose et al 2005). The overall number of face areas is similar to that in macaques, and so is their spatial arrangement in the temporal lobe with two occipito-temporal series of areas, one in ventral temporal cortex and the other, dorsal one, in the STS (Yovel & Freiwald 2013). Functional specializations along the ventral series of face areas follow a progression similar to that in the macaque monkey, suggesting a hierarchical organization: in the occipito-temporal direction face selectivity increases (Bell et al 2009), position dependence decreases (Hemond et al 2007), mirror symmetric confusion of facial profile views emerges (Axelrod & Yovel 2012, Kietzmann et al 2012), and facial identity selectivity grows stronger (Yang, Susilo, & Duchaine, in press). Similarly, in both macaque and human face areas, the response to faces is augmented in the presence of an anatomically correctly placed body, and this augmentation grows stronger from posterior to anterior face areas (Bernstein et al 2014, Fisher & Freiwald 2015b, Song et al 2013). Furthermore selectivity for facial motion is highly pronounced in dorsal areas, but not in ventral ones (Fox et al 2009, Pitcher et al 2011a). Similar to the macaque, the ventral face areas appear to be interconnected, while connectivity of the two streams appears limited (Gschwind et al 2012) and is still the subject of active research (Yeatman et al 2014). Similarities between the systems extend beyond the temporal lobe. Face-selective neurons and areas have been found in the macaque in three major parts of prefrontal cortex (PFC) (Ó Scalaidhe et al 1997, Rolls et al 2006, Tsao et al 2008b), and later fMRI work confirmed the presence of face-selective areas in the human PFC (Axelrod & Yovel 2013, Chan & Downing 2011, Pitcher et al 2011a). Thus human and macaque face-processing systems, according to at least two sets of criteria, anatomical location and functional specialization, appear to be homologous.

Face Processing: from Cells to Behavior

Face Detection

For facial information to be extracted, the face must first be detected in a scene. Face detection has been proposed, based on psychophysical and computational work, to rely on coarse contrast relationships between the different parts of the face (Sinha 2002). Due to the three-dimensional structure of the face, e.g., with the eyes set back relative to the forehead, the forehead will be brighter than the eyes under most illumination conditions. Since change of appearance due to change of illumination is among the most difficult problems for object recognition, heuristics that are robust against this non-affine transformation are of great use. Furthermore, coarse contrast features do not require facial detail and thus should be easy to compute and relatively independent of facial identity. Humans indeed use twelve contrast-pairs (e.g. forehead brighter than left eye) to detect faces (Sinha 2002). This behavioral characteristic finds a clear correspondence at the single cell level. Half of the neurons in ML and MF, when probed with coarse-contrast face stimuli, showed sensitivity to contrast relationships between pairs of facial parts (e.g., nose and left eye) (Ohayon et al 2012) (Figure 2A). Cells exhibited highly consistent preferences for a specific polarity of each pair (e.g., nose brighter than left eye region). Importantly, the population preferences for contrast included all of the twelve pairs found to be important in human detection behavior, and the population’s polarity preferences matched that of human observers in all twelve cases.

Fig. 2. Mechanisms for Face Detection, Holistic Face Processing, and Face Space.

a. (left) To generate stimuli that probe selectivity for coarse-contrast face-stimuli, a front-view picture of a face was segmented into eleven areas, to which eleven luminance values were then randomly assigned (Ohayon et al 2012). Recordings from the middle face patches (MF/ML, bottom) during stimulus presentation revealed selectivity of cells to the polarity of a large fraction of all possible contrast pairs and dominance of a single polarity for each pair. (Right) Histogram of ten most common contrast polarity features and their relative proportion in the cell population. The top two contrast polarity features express higher luminance for the nose region than left and right eye, respectively. Five of these contrast features and their polarity were predicted by human psychophysics (columns outlined in orange).

b. Effects of “Thatcherizing” faces on face-selective responses in single units from face area ML (left) and the evoked potential from the right occipito-temporal electrode in humans (right). Left: Face cells in ML differentiate between intact and Thatcherized upright faces (blue and red), but not between intact and Thatcherized inverted faces (green and yellow) (Taubert et al 2015b). This is consistent with the behavioral Thatcher illusion first reported by Thompson (Thompson 1980). Right: Event related potentials to Thatcherized and intact upright and inverted faces recorded from a right occipito-temporal EEG electrode in humans also show a difference in the amplitude of the N170 between intact and Thatcherized upright faces but no difference for inverted faces (Carbon et al 2005).

c. i Two dimensional depiction of perceptual face space (Leopold et al 2001). The space is centered on an average face. Each face occupies a particular location in face space based on its deviation from the average face along many dimensions. Faces along the same trajectory originating at the average face all have the same identity, but faces further from the average face have greater identity strength and are therefore more distinctive. An anti-face is a face on the opposite side of the average face from a face with an equal identity strength. Aftereffects for opposite adaptor pairs are greater than aftereffects for non-opposite faces matched for perceptual similarity.

ii Facial feature selectivity of a middle face patch neuron (Freiwald et al 2009). Macaques viewed cartoon faces that varied randomly along nineteen dimensions, each with feature values ranging from one extreme (−5) to another (+5) (valence arbitrary). Typical cells were tuned to small subsets of feature dimensions (here four) and exhibited ramp-shape tuning curves with minimal responses elicited by one feature extreme and maximal responses by the opposite feature extreme. Thus tuning curves spanned face space. The sample neuron shown here was modulated by face aspect ratio (preferring narrow faces), inter-eye distance (preferring narrowly spaced eyes), eye aspect ratio (preferring wide eye shapes), and iris size (preferring large).

iii. Response of a face-responsive cell, recorded likely in AM to stimulus trajectories originating in the average face. The center figure illustrates stimuli shown and their organization along three trajectories (red, blue, black) originating in the center of face space. For all trajectories, the neuron’s firing rate increased with distance from the center of face space.

Holistic Face Representations

A key characteristic of human face recognition is the holistic nature of upright-face processing relative to the more part-based nature of object processing and inverted face processing (Tanaka & Farah 1993). The face-inversion effect is the robust observation that humans perceive face information far less precisely in inverted than in upright faces (Yin 1969), and there is ample evidence to suggest that subjects use a more holistic strategy with upright than with inverted faces (Farah et al 1995, Young et al 1987). In the middle face patches, the response to inverted faces is, on average, delayed, more transient and overall reduced (Perrett et al 1982, Tsao et al 2006). This neural face-inversion effect may be confined to the face patches as it was found to be much weaker for face-cells located outside the face patches (Taubert et al 2015a). How might face-inversion affect the processing of facial features? Cells in ML/MF, when probed with cartoon stimuli, were tuned to more features in the upright than the inverted face (Freiwald et al 2009). Furthermore, tuning to one feature, eye-brows, was lost entirely under inversion, while tuning to the mouth was generated de novo. This pattern of results can be explained with a holistic template matching hypothesis that proposes incoming facial features in both upright and inverted faces are matched against an upright face template. The account also explains how the shape of feature tuning can be preserved upon face-inversion, as was observed for features that do not change shape and do not markedly change position when faces are inverted. Second, the part-whole effects reported in human psychophysical studies, demonstrate that features are better recognized within a face than in isolation (Tanaka & Farah 1993). ML/MF cells exhibit tuning to isolated facial features, but tuning is greatly augmented, when features are part of the entire face (Freiwald et al 2009). Third, maybe the most striking expression of the context dependence of facial features, is the so-called Thatcher illusion (Thompson 1980). Inversion of local features like the eyes or the mouth, which are easily noticed as grotesque distortions in the upright face, are hardly recognizable in the inverted face. A single unit correlate of this effect has been reported in ML (Taubert et al 2015b), where inversion of the eyes reduced the response in the upright, but not the inverted face and similar effects are reported in human electrophysiological studies (Carbon et al 2005) (Figure 2B).

Face Space

Many productive investigations of face processing have utilized the concept of face space: a multi-dimensional space in which each face occupies a unique position based on its combination of features (Leopold et al 2001, Turk & Pentland 1991, Valentine 1991). The space is thought to be organized around a center occupied by the average face, around which facial identities are located along trajectories that reflect how each face’s characteristics deviate from the average face (Figure 2C). The distance of a face along a trajectory is determined by the similarity of a face to the average face, so distinctive faces occupy the fringes of the space.

Two main models have been considered in discussions of how dimensions in face space are coded (Rhodes & Leopold 2011). One possibility is broad tuning curves spanning entire axes of face space. Such ramp-shaped tuning curves of opposite polarity can efficiently encode the position of a face feature along an axis. Second, an axis could be tiled by narrow tuning curves, in which each cell functions as a channel and coding is exemplar-based. Psychophysical results suggest that the center of face space occupies a special position such that identities located in opposite directions are related to each other even though they are perceptually dissimilar: when one adapts to one identity, subsequent perception of the “anti”-identity, but not that of an equidistant control identity, is enhanced (Leopold et al 2001). This effect is compatible with the broad tuning model, not the exemplar-based model. Moreover, these aftereffects are stronger than aftereffects found when faces matched in similarity to face/anti-face pairs are used to create a morph continuum (Rhodes & Jeffery 2006). Consistent with these psychophysical findings, broad tuning was indeed found in two macaque face areas. In MF/ML, the aforementioned feature tuning is predominantly ramp-shaped yielding a maximal response to one extreme and a minimal response to the opposite extreme (Figure 2C), and for almost all feature dimensions, opponent polarities were found across cells (Freiwald et al 2009). More anteriorly, likely in AM, broad tuning curves have also been found in response to facial identities (Leopold et al 2006). Unlike tuning curves in ML/MF, these tuning curves exhibited increased firing as a function of distance from the center of face space (Figure 2C), which provides neurophysiological evidence for the special role of the center of face space. More distinct response patterns to extreme faces in ML/MF compared to average faces might, furthermore, explain why identity has been found in some studies to be perceived more easily in caricatures than the original face (Rhodes et al 1997)

Faces as Sources of Social Information

A multitude of social information can be extracted from fine variations across faces, like gender, age, attractiveness, changing mental states like social attention and mood (Haxby et al 2000), and even inferred character attributes like trustworthiness (Todorov 2008). Many face cells have tuning properties that suggest an involvement in some of these social inferences. Some face cells are tuned to head orientation or gaze (De Souza et al 2005, Desimone et al 1984, Perrett et al 1985), and similar selectivity has been reported in the superior temporal sulcus of the human brain (Carlin et al 2011). This selectivity may support an analysis of the direction of social attention in the dorsal face pathways. Similar head-view selectivity was also found in the parallel, ventral face pathway, but there it may reflect an initial view-dependent step preceding a view-invariant face representation. Other face cells appear to be tuned to facial expressions (Gothard et al 2007, Sugase et al 1999) and yet others to physical characteristics of the face including facial features and identity (Freiwald et al 2009, Young & Yamane 1992). Mechanisms for the processing of expression and identity may overlap in common neural systems (Young & Calder, 2005, Bernstein & Yovel, 2015). Yet other studies have reported neural population codes to reflect another social quality, familiarity (Eifuku et al 2011, Young & Yamane 1992). Thus, specializations of face cells exist that are plausible neural mechanisms supporting the different social dimensions of face recognition.

Causal Links between Face-Processing Systems and Behavior

The links between particular neural processes and behavior discussed above are correlational, leaving open the question of whether these neural processes actually contribute to face recognition. Many regions in extrastriate cortex show a response to faces that is well above baseline, but as we discuss below, a number of findings suggest the response to faces is behaviorally relevant in face-selective but not in nearby areas that are not selective to faces. To determine the behavioral relevance of a neural mechanism, behavior must be assessed while that neural process is disrupted.

Acquired Prosopagnosia

The earliest evidence causally linking neural regions and face processing was provided by patients who acquired face processing deficits due to brain damage. Prosopagnosia can result from lesions extending from the ventral occipital lobe (Bouvier & Engel 2006, Dalrymple et al 2011, Rossion et al 2003) to the anterior temporal lobe (Busigny et al 2014, Dalrymple et al 2011) in the right hemisphere, while unilateral left hemisphere lesions rarely produce prosopagnosia ((Barton et al 2002, De Renzi et al 1994); see (Rossion 2014) for discussion of these cases). Thus, consistent with lateralization of face processing in human fMRI and ERP (Bentin et al 1996, Bukowski et al 2013), this pattern indicates face processing in humans is strongly right lateralized.

Functional imaging has allowed researchers to assess the status of face-selective areas in individuals with acquired prosopagnosia. Patient P.S. suffered from dense prosopagnosia following a closed-head injury, and fMRI revealed the damage overlapped with her right OFA, while leaving the right FFA intact (Rossion et al 2003, Sorger et al 2007). Findings from other prosopagnosics have provided further evidence that right hemisphere posterior face-selective areas are critical to face recognition (Dalrymple et al 2011, Fox et al 2011). However, prosopagnosia following damage to anterior temporal regions that spare posterior face-selective regions has been reported in a number of cases (Busigny et al 2014, Dalrymple et al 2011, Fox et al 2011). Thus, it appears that face-selective regions along the entire ventral pathway causally contribute to face recognition. Less is known about the areas supporting face perception for aspects of faces other than identity. R-ST1, one of the few cases with lesions to the face network restricted to the STS (Fox et al 2011), exhibited deficits with facial expression recognition consistent with the role of STS face areas in the processing of changeable facial information (Haxby et al 2000). Importantly, in identity matching tasks, R-ST1 performed normally when expression was held constant, but had pronounced deficits when expression differed in the images to be matched, suggesting STS face areas contribute to identity computations carried out in ventral stream areas when expression variation is present.

Experimental Disruption of Face-Processing

Experimental disruption during behavioral performance provides a controlled means to demonstrate a causal link between neural mechanisms and behavior, and studies using these methods in both humans and monkeys have demonstrated clear links between particular neural regions and face recognition. Transcranial magnetic stimulation (TMS) provides a non-invasive means to examine the functional role of brain regions in humans. Because of TMS’s limited ability to target regions not close to the scalp, most studies have targeted superficial face areas OFA and pSTS-FA. TMS to the right OFA impairs face shape discrimination but not object or body shape discrimination (Pitcher et al 2009, Pitcher et al 2012, Pitcher et al 2007) (Figure 3). Similarly, TMS to the extrastriate body area (EBA) selectively disrupts body discrimination while TMS over the lateral occipital complex (LO), an area implicated in object recognition, selectively disrupts object discrimination (Pitcher et al 2009). The three category-selective areas targeted in these studies all responded substantially to non-preferred categories (e.g., objects and bodies for OFA) (Schwarzlose et al 2008), so the category-selectivity of these impairments indicates (i) the presence of a response within a region does not imply behavioral relevance (see also (Dilks et al 2013, Pitcher et al 2011b), and (ii) supports a modular view of high-level visual recognition. Consistent with results from acquired prosopagnosics, TMS over both OFA and pSTS-FA has been found to disrupt expression perception, but the effective time window for pSTS-FA disruption is longer than the OFA window (Pitcher 2014), consistent with the notion of hierarchical processing.

Fig. 3. Causal Studies of Face Processing.

a. Effect of TMS over three right hemisphere category-selective areas on sequential face discrimination (Pitcher et al 2009). The behavioral effect was computed by subtracting % correct in the absence of TMS from % correct when TMS was delivered over a given area. TMS over the occipital face area (OFA) disrupted face discrimination whereas TMS over the lateral occipital (LO) object area and extra-striate body area (EBA) had no significant effect on face discrimination. Similar category-selective effects were found for object discrimination at LO and body discrimination at EBA (not shown).

b. Effect of muscimol microinjection on face gender discrimination (Afraz et al 2015). The behavioral effect was calculated by subtracting % correct when faces were presented in the ipsilateral visual field (VF) from % correct in the contralateral VF. Sites had been categorized as “face detector sites” or “other IT sites” based on the response to briefly presented images of faces and non-face objects prior to muscimol injection. Inactivation of face detector, but not control sites, caused a deficit in face gender discrimination.

Intracranial stimulation has a long history in human neuroscience, and recently researchers have applied it to investigate the neural basis of face processing. After mapping the category-selectivity of regions in right middle and posterior fusiform gyrus, Parvizi et al. (2012) electrically stimulated face-selective areas. These stimulations caused pronounced distortions to face percepts (Parvizi et al 2012). Consistent with a modular view, the patient reported only subtle distortions when the face regions were stimulated while he was viewing non-face objects. Evidence for functional lateralization of face-processing was provided by another study from this group which found stimulation to the right hemisphere face areas caused face-selective distortions, but comparable stimulation in the left hemisphere did not distort face percepts but led only to non-face, elementary distortions like phosphenes (Rangarajan et al 2014). Similarly, (Jonas et al 2014) stimulated right OFA, severely disrupting the ability to discriminate simultaneously presented faces (see also (Jonas et al 2012)). Thus electrical stimulation demonstrates a causal role of right hemisphere face-selective areas for face processing. Moreover, the absence of face distortions following stimulation to left hemisphere face areas in combination with the rarity of acquired prosopagnosia after unilateral left hemisphere lesions raise questions about what role, if any, left hemisphere areas make to face processing in humans.

Surprisingly few experiments disrupting face processing in the macaque have been done. Afraz et al (Afraz et al 2006) electrically stimulated clusters of face-selective neurons while macaques made face/non-face decisions on briefly presented stimuli. The macaques were more likely to report the presence of a face when face clusters were stimulated, whereas stimulation of clusters that were not face-selective did not affect behavior. Recently, Afraz et al (Afraz et al 2015) used optogenetics and muscimol in separate experiments to selectively suppress activity in face-selective clusters in the lower bank of the STS, likely in the vicinity of ML. During face gender decisions, suppression in face clusters, but not outside them, weakly disrupted performance (Figure 3A). These effects were restricted to contralaterally presented faces, likely reflecting the position of the inactivated neurons’ receptive fields. Together, the causal studies described above demonstrate that face-selective neurons and face-selective areas play critical roles in primate face processing and suggest that these areas play a much more prominent role than areas that are face-responsive but not face-selective.

Recognition of Familiar and Unfamiliar Faces

Most studies on face processing use unfamiliar faces. Unfamiliar faces are ideal stimuli to isolate the contribution of the visual system to face recognition, because they are associated with far less semantic and emotional information than familiar faces. Nevertheless, the primary goal of face recognition for social creatures is to recognize familiar individuals. Many of the basic phenomena of face recognition are common to familiar and unfamiliar faces (for review see (Johnston & Edmonds 2009)), including the face inversion effect (Yarmey 1971, Yin 1969), but see (Megreya & Burton 2006)), the composite face effect (Le Grand et al 2004, Young et al 1987) and the Thatcher illusion (Thompson 1980). Contrast negation impairs recognition of both familiar and unfamiliar faces (Kemp et al 1996), and distinctiveness enhances recognition of both familiar and unfamiliar faces (Bruce et al 1987). Also, prosopagnosic individuals almost always have difficulty recognizing both familiar and unfamiliar faces (Dalrymple et al 2011, Duchaine et al 2007, Van Belle et al 2010). Despite these similarities, the processing of familiar and unfamiliar faces differs in important ways. First, familiar faces are better recognized across different images based on their internal than external features whereas no such difference is found for unfamiliar faces (Clutterbuck & Johnston 2005, Ellis et al 1979, Young et al 1985). A second difference is that matching of unfamiliar faces across different illuminations (Johnston et al 1992), head orientations (Hill et al 1997, O’Toole et al 1998), and expressions (Bruce 1982) is prone to error whereas comparable tasks with familiar faces are effortless (Figure 4A).

Fig. 4. From Non-Familiar Face Recognition to Person Knowledge.

a. Forty face images, unfamiliar to most, which subjects had to sort by identity (Jenkins & Burton 2011). Although the array includes 20 images of two individuals each, subjects sorted them into an average of seven identities. Subjects familiar with the two individuals successfully sorted the images to two identities. These results demonstrate the challenge of generalizing across different images of the same person for unfamiliar faces.

b. A single neuron in the medial temporal lobe of an epilepsy patient exhibiting robust responses to a wide range of images of actress Halle Barry, including images in which her face is completely masked, and even her name written in letters (Quiroga et al 2005). The response profile demonstrates that non-face related knowledge (i.e. knowledge about the movie she appeared in with this mask, knowledge of her name) shaped the response of this neuron, thus generating a highly invariant person representation beyond the representations that can be obtain from vision alone.

How is this invariance for familiar faces generated? The original Bruce and Young (1986) model proposed that unfamiliar face representation involves a pictorial code, which is primarily image-based, whereas the representation of familiar faces relies on a structural code, which is more abstract and less susceptible to changes in view, expression and lighting. Burton, Jenkins, and colleagues (Burton et al 2005, Jenkins & Burton 2006, Jenkins & Burton 2011) suggested that this representation may be generated by averaging the various images of the same person that an observer encounters, separately for multiple views. These averaged representations will then include the essential, diagnostic features of a given identity. Consistent with this view, averaged faces made from many face exemplars of a given individual are better recognized than averaged faces made from fewer exemplars (Burton et al 2005). Thus representations of familiar faces benefit from rich perceptual exposure to the same individuals. In contrast, when we see an image of an unfamiliar face it can be difficult to disentangle superficial pictorial information from information about the stable characteristics of the face.

Familiar faces are not only rich in perceptual information but also in semantic, emotional, and social information, which can enhance face recognition. For example, participants better remember a face when asked to make a personality judgment (e.g., “Does the face look intelligent?”) than when they make a perceptual judgment (e.g., “Does the face have a large nose?”) (Curtois & Mueller 1979). Similarly, associating new faces with unique labels (i.e., each face with a different letter) increases face recognition compared to associations with similar labels (e.g. all faces are associated with the same letter) (McGugin et al 2011). These findings suggest that social, emotional, and higher-level categorizations all play a role in face recognition abilities. The contribution of conceptual information to the invariant representation that we have for familiar faces is also evident in the response of the so-called “concept cells” (Quiroga et al 2005) that demonstrate robustness to very different pictures of a known individual, even when the face is largely hidden in the image (see Figure 4B). This representation thus reflects invariance beyond purely visual information about the face, and must be based on person knowledge. Thus, combinations of visual and conceptual factors are likely to contribute to the invariant representation of familiar faces.

In summary, whereas recognition of familiar faces is robust to many different variations in the image of a given person, recognition of different images of unfamiliar people is challenging due to the limited amounts of information available.

Face Recognition in the Real World: Challenges and possible solutions

The limitations of unfamiliar face recognition have substantial effects in a number of consequential situations. Eyewitness identification is highly accurate for familiar faces, but when identification of unfamiliar faces is required, errors are frequent (Wells et al 2002) and an important factor in miscarriages of justice (Rattner 1988). In fact, eyewitness misidentification accounts for more than 70% of convictions overturned by DNA testing in the USA (www.innocenceproject.org/free-innocent/improve-the-law/fact-sheets/dna-exonerations-nationwide) (National Research Council (U.S.). Committee on Scientific Approaches to Understanding and Maximizing the Validity and Reliability of Eyewitness Identification in Law Enforcement and the Courts. et al). Another factor often limiting eyewitness identification of unfamiliar faces is the other-race effect: identification is substantially worse for faces of races a subject has had little contact with in daily life (Malpass & Kravitz 1969). Face recognition is also used at border crossings where simultaneous matching of a live person to a passport photo is required. Although simultaneous matching is easier than eyewitness identification, a recent study conducted with passport officers found these highly experienced individuals incorrectly accepted 14% of the passports that did not match their bearers (White et al 2014). Furthermore, the officers’ performance was no better than that of student participants. Notably the passport officers showed great individual differences (range 0–30% errors), and these individual differences were not associated with years of experience in their job.

In the last decade, it has become apparent that great individual differences in face processing exist (for review see (Yovel et al 2014)). In developmental prosopagnosia (DP), individuals with no history of brain damage or general impairments have difficulty recognizing the faces of family and friends (Susilo & Duchaine 2013). On the opposite end of the spectrum are super-recognizers; they rarely forget the faces of people they interact with and regularly recognize faces encountered only briefly years before (Russell et al 2012, Russell et al 2009). The majority of the population lies somewhere between these extremes, and differences among them are stable (Bowles et al 2009, Wilhelm et al 2010) and strongly heritable (Wilmer et al 2010, Zhu et al 2010). Lab-based face identity tests are not only correlated with other lab-based face recognition tests (Germine et al 2012), but also predict performance in more ecologically valid identification situations like eyewitness identification (Andersen et al 2014). Modest relationships have been found between face recognition ability and some known face processing effects such as the composite face effect (DeGutis et al 2013) and face aftereffects (Dennett et al 2012), but the neurocognitive sources of most variance in face processing abilities is unknown.

The reliable and stable individual differences across lab-based and real-life face recognition tasks suggest that when face recognition plays a role in consequential situations, individual ability should be considered. In courts, awareness of an eyewitness’ face recognition skill would allow the judge and jury to better assess the weight they should place on the eyewitness’ testimony. Selection for occupations such as passport officers and police officers would benefit from consideration of face recognition ability, and this suggestion has been successfully applied recently by the London Metropolitan Police (Davis et al 2013). In collaboration with face researchers, officers and staff with especially good face recognition have been identified. These ‘super-recognizers’ used their skills to identify hundreds of suspects from low-resolution surveillance videos taken during the large-scale riots in London in 2011, and a team of ‘super-recognizers’ have watched live video during large festivals to identify and then capture suspects wanted for previously committed crimes.

In summary, the belief that all humans are experts in face recognition has led to the assumption that face recognition can be relied on in real life settings. However although this assumption is, by and large, true for identification of familiar faces, it has detrimental effects on criminal justice, border control, and policing, because we are far from perfect when it comes to identification of unfamiliar faces (Burton & Jenkins 2011).

Conclusion

The dedicated system for face processing in the primate brain has been the focus of intensive basic and applied research in the recent decades. The findings discussed here provide an overview of attempts to bridge the gap between levels of organization from single cells to networks of areas and behavior as well as between lab-based and real-life face recognition. This work shows the promise of these approaches but also makes clear that these are only the initial steps toward a comprehensive understanding of the operations required for successful face recognition.