Neural microgenesis of personally familiar face recognition

Significance

We addressed the open question of how the human brain recognizes personally familiar faces. A dynamic visual-stimulation paradigm revealed that familiar face recognition is achieved first and foremost in medial and anterior temporal regions of the extended face-processing system. These regions, including the amygdala, respond categorically to individual familiar faces. In contrast, activation in posterior core face-preferential regions is associated with the amount of visual information available, irrespective of familiarity. Through integration of core and extended face-processing systems, these observations provide a common framework for understanding the neural basis of familiar face recognition.

Abstract

Despite a wealth of information provided by neuroimaging research, the neural basis of familiar face recognition in humans remains largely unknown. Here, we isolated the discriminative neural responses to unfamiliar and familiar faces by slowly increasing visual information (i.e., high-spatial frequencies) to progressively reveal faces of unfamiliar or personally familiar individuals. Activation in ventral occipitotemporal face-preferential regions increased with visual information, independently of long-term face familiarity. In contrast, medial temporal lobe structures (perirhinal cortex, amygdala, hippocampus) and anterior inferior temporal cortex responded abruptly when sufficient information for familiar face recognition was accumulated. These observations suggest that following detailed analysis of individual faces in core posterior areas of the face-processing network, familiar face recognition emerges categorically in medial temporal and anterior regions of the extended cortical face network.

One of the most important functions of the human brain is the ability to recognize other people by their faces. Humans are capable of astonishing performances; for example, they can identify individuals despite not having seen them for decades (1) and can tell apart familiar from unfamiliar faces in a few hundred milliseconds (2, 3). However, face recognition is one of the most difficult operations performed by the human brain: performance is highly variable across typical individuals (4, 5) and can be severely disrupted following right occipitotemporal brain damage (“prosopagnosia”) (6–8) or atypical development (“congenital/developmental prosopagnosia”) (9, 10).

To date, the neural basis of familiar face recognition in humans remains largely unknown. Several regions in the ventral occipitotemporal cortex exhibit increased activation during perception of faces relative to nonface objects (i.e., face-preferential areas) (11–15). These regions include the so-called “core face-processing system” (16), comprising regions in the middle section of the lateral fusiform gyrus [“fusiform face area” (FFA)], the lateral inferior occipital gyrus [“occipital face area” (OFA)], and the posterior superior temporal gyrus. However, these regions, which are identified with unfamiliar face stimuli, show only weak and/or inconsistent differences between familiar and unfamiliar faces (17). For example, within the right FFA, face familiarity has been associated with either increased or decreased neural activation (18, 19). Moreover, only subtle differences in the patterns of voxels activated for (visually) familiar versus unfamiliar faces have been described in the OFA and FFA, in some but not all studies (20). These relatively weak and inconsistent differences between the neural representations of familiar and unfamiliar faces are at odds with the large differences observed behaviorally (21, 22).

Effects of face familiarity have also been reported in structures located anterior to the “core” regions, in “extended” face-processing regions (16). Such extended regions include structures within the medial temporal lobe (MTL), including the hippocampus, perirhinal cortex and amygdala (11, 23–27), the latter two of which exhibit face-preferential responses (15, 28, 29). Although the hippocampus and perirhinal cortex are traditionally considered to mediate explicit recollection based on episodic memory and diffuse feelings of familiarity, recent studies indicate that these regions are also involved in visual discrimination (26, 29–33). Furthermore, various studies have also reported anterior ventral temporal activation for familiar (famous) face recognition (11), discrimination of familiar from unfamiliar faces (34–38) and face naming (39). Together with neuropsychological reports of impaired familiar face identification subsequent to right anterior temporal damage (40), these findings support the view that anterior temporal regions, including the MTL, are critically involved in storage of biographical information related to faces and feelings of familiarity (41).

Integrating these sets of data from posterior face-preferential regions and anterior temporal regions, referred to as the core and extended face-processing systems, respectively (16, 35), in a common framework is a major challenge for understanding the neural basis of familiar face recognition. According to one view, core face-preferential areas could rapidly assess the “known-versus-unknown” status of faces, sending information forward to anterior regions of the extended face-processing system for storage and affective processing (16, 20, 35). Alternatively, following a detailed analysis of individual faces in posterior regions of the core network, the categorization of faces according to their familiarity might be determined (i.e., emerge) first and foremost in anterior ventral and medial temporal regions.

The present neuroimaging study was designed to address this issue by adopting an approach that departs from typical neuroimaging studies of face perception and recognition in several ways. First, we reasoned that the typically used transient (i.e., abrupt) mode of visual stimulation may account for some of the inconsistencies in previous studies. The sudden onset of a face stimulus evokes a global visual response that includes a large common response for familiar and unfamiliar faces that may obscure differential brain activation related to face familiarity. To overcome this limitation, we isolated the familiarity-related response from the transient visual response(s) by gradually and slowly revealing facial identity. This was accomplished over the course of a stimulation sequence by parametrically increasing the amount of high-spatial frequency (HSF) content to initially severely low pass-filtered face stimuli (Fig. 1). Such a coarse-to-fine dynamic display not only mimics the perception of an individual face approaching an observer but also corresponds to a slow version of a coarse-to-fine model of how faces are perceived in the human brain (8, 42–45). Although, to our knowledge, such coarse-to-fine stimulation has never been used in neuroimaging, previous studies have successfully used parametric variations of stimulus visibility to identify robust differential responses in high-level visual areas (46–48), including category-specific responses in face-preferential regions (49).

Fig. 1.

Experimental design. Each trial consisted of presentation of an average blurred face, replaced by a familiar or unfamiliar face containing the same spatial frequency content (1.5 c/f ). The HSF content of each individual face then increased progressively over 18 TRs until it was presented at full resolution. The average trial length was 27 TRs (TR: 1,250 ms).

Second, rather than presenting pictures of famous or familiarized faces as in previous studies, we used personally familiar faces (i.e., those of participants’ classmates). In doing so, we minimized intersubject variability in the degree of familiarity with individual faces, as well as the iconic (i.e., image-based) nature of famous and familiarized face representations (2, 50). (For a recent discussion of this issue, see ref. 20.) Previous findings suggest that the high saliency and association with numerous past experiences, which are characteristic of personally meaningful stimuli, lead to the involvement of larger proportions of the brain (51), in particular, within medial and anterior temporal regions (30, 52, 53). Additionally, stimulus familiarity has been shown to modulate the interaction between these regions (54).

Finally, we performed in-depth complementary analyses of the neural responses to familiar and unfamiliar faces. Whole-brain univariate analyses aimed at identifying regions showing familiarity-dependent differential neural activation, independently of their face-preferentiality. Univariate region-of-interest (ROI) analyses focused more specifically on the response profiles in both core posterior and extended medial and anterior temporal face-preferential regions, namely, the OFA, FFA, amygdala, and anterior inferotemporal cortex (AIT). Lastly, an original multivariate pattern analysis (MVPA) (55) of single-trial representational (dis)similarity (56) investigated familiar and unfamiliar face representations within these ROIs. This analysis aimed at testing for potential differences between familiar and unfamiliar faces not disclosed by univariate analyses.

Results

Behavior.

On each trial, participants were asked to decide whether the gradually revealed face belonged to a familiar or unfamiliar individual. Participants’ accuracy did not differ significantly between conditions [familiar: 81.58 ± 14.57%; unfamiliar: 83.22 ± 11.14%; t(10) = −0.38, P = 0.71; confidence interval (CI): −0.11; 0.08]. However, average reaction times (RTs) were significantly faster for familiar compared with unfamiliar faces (15,573 ± 920 ms vs. 18,608 ± 1,370 ms, corresponding to stimuli containing 10.09 and 16.97 cycles per face (c/f), respectively; t(10) = −8.87, P < 0.00001; CI: −3.04; −1.82).

Whole-Brain Analysis.

Larger responses for familiar than unfamiliar faces were found mainly in the temporal lobe, encompassing both cortical and subcortical structures (Table 1 and Figs. 2 and 3). Additional frontal lobe foci were identified, as well as one cluster located in the posterior cingulate gyrus. Face familiarity-related activation in right lateralized medial temporal structures encompassed the perirhinal cortex, hippocampus, and amygdala; left lateralized clusters were found in the entorhinal/parahippocampal gyrus and amygdala (Fig. 2 and Table 1). Clusters in the AIT were found in both hemispheres. Most of the right hemisphere clusters were face-preferential, except those located in the hippocampus and perirhinal cortex. Left hemisphere clusters were non-face-preferential, with the exception of the amygdala, the AIT, and orbitofrontal/straight gyrus. The largest clusters exhibiting significantly different responses between conditions were found in the bilateral amygdala (Fig. 3). There were very few regions showing a larger response to unfamiliar than familiar faces, with clusters observed in the left medial occipital and posterior superior temporal gyrus (Table 1), and no clusters in the medial and anterior temporal lobe regions.

Table 1.

Location and face-preferentiality of clusters identified in the univariate whole-brain analysis

Anatomical location	Talairach coordinates			No. of voxels	Face-preferentiality
	x	y	z		F > O	F > Scr
Fam > Unf
HC	17 ± 4	−17 ± 2	−20 ± 1	129	—	—
AIT	18 ± 1	3 ± 1	−42 ± 1	48	*	*
AMY	18 ± 3	−3 ± 3	−20 ± 2	589	*	*
PrC	19 ± 2	−7 ± 1	−30 ± 1	127	—	—
HC	25 ± 1	−31 ± 1	−6 ± 1	29	—	—
ITG	52 ± 2	−57 ± 2	12 ± 1	128	*	*
MTG	61 ± 1	−41 ± 1	12 ± 1	37	*	*
SRG	−5 ± 2	52 ± 1	0 ± 1	54	—	—
CG	−7 ± 2	−56 ± 3	18 ± 1	197	—	—
OFG/SG	−7 ± 1	49 ± 1	−12 ± 3	37	*	*
ITG	−16 ± 1	−13 ± 1	−26 ± 1	32	NS	NS
Ent/PHG	−19 ± 1	−23 ± 2	−23 ± 1	126	NS	F < Scr
AMY	−24 ± 2	−6 ± 2	−17 ± 2	274	*	*
AIT	−31 ± 1	−4 ± 1	−35 ± 2	143	*	*
LOrG	−40 ± 1	28 ± 1	−14 ± 1	41	—	—
STG	−49 ± 4	−57 ± 4	16 ± 4	2834	—	—
MTG	−58 ± 3	−51 ± 3	−6 ± 2	616	—	—
Unf > Fam
AOI/BO	29 ± 2	19 ± 2	−1 ± 2	225	*	NS
OcG	−23 ± 1	−94 ± 1	−2 ± 2	180	*	F < Scr
OcG	−42 ± 2	−82 ± 1	−11 ± 2	158	NS	*
STG	−64 ± 2	−19 ± 1	6 ± 0	38	F < O	F < Scr

Identified clusters are reported sorted by x coordinates. Asterisks indicate face-preferentiality as verified with a random-effects GLM for F > O (faces–objects) or F > Scr (faces–scrambled faces) (P < 0.05). Note that some clusters showed the opposite patterns at the same significance level (F < O, or F < Scr). Dashes indicate that no values are reported for clusters that exhibited deactivation in response to stimuli presented in localizer scans. x, y, and z are Talairach coordinates in mm. AOI, area orbitoinsularis; AMY, amygdala; BO, basal operculum; CG, cingulate gyrus; Ent, entorhinal cortex; ITG, inferior temporal gyrus; LOrG, lateral orbital gyrus; MTG, middle temporal gyrus; NS, not significant; OcG, occipital gyrus; OFG, orbitofrontal gyrus; PHG, parahippocampal gyrus; SG, straight gyrus; SRG, superior rostral gyrus; STG, superior temporal gyrus.

Fig. 2.

Right lateral medial and ventral temporal lobe clusters identified in the whole-brain group analysis as exhibiting larger BOLD responses for familiar compared with unfamiliar faces. The figure shows clusters on transverse slices of an average anatomy of from a large scale localizer study (15) (n = 40; x coordinates are provided); colors indicate face-preferentiality as verified with an external localizer (for details, see Table 1). AMY, amygdala; HC, hippocampus; PrC, perirhinal cortex.

Fig. 3.

Bilateral amygdala clusters identified in the whole-brain group analysis as exhibiting larger BOLD responses for familiar compared with unfamiliar faces. Clusters (for details, see Table 1) are displayed on a coronal slice (y = 7). Average time courses (±SE) are juxtaposed (x axis represents time in volumes, TR) to illustrate the onset of differential responses for familiar and unfamiliar face trials. Vertical dashed lines indicate when familiar face decisions (i.e., average time of behavioral responses) occurred. Per trial face stimuli of increasing spatial-frequency content were presented over 20 TRs on average (see Lower and Fig. 1 for example stimuli). lAMY, left amygdala; rAMY, right amygdala.

Face-Preferential ROIs.

Univariate analyses.

Seven regions of interest were identified with an independent face localizer: bilateral OFA and FFA (i.e., core posterior face-processing regions), as well as the amygdala and right AIT in the extended anterior face-processing system. (See Table S1 for individual ROI details.) Univariate analyses were performed on subjects’ stimulus-aligned time courses derived from these ROIs to investigate familiarity-dependency of neural responses, differences in the onset of significant activation, and their general profile of neural activation (Figs. 4 and 5 and Tables 2 and 3).

Table S1.

Location of individual participants’ face-preferential ROIs

	Talairach coordinates
Anatomical location	x	y	z	No. of voxels
rOFA
S01*	34	−79	−22	230
S02†	29	−84	−10	250
S03‡	33	−84	−18	246
S04§	38	−75	−13	276
S05§	25	−85	−5	264
S06§	34	−89	−18	276
S07§	25	−92	−8	283
S08¶	31	−85	−20	296
S09†	29	−88	−16	257
S10‡	40	−76	−11	287
S11§	36	−76	−7	298
S12‡	30	−90	−7	291
S13
S14§	43	−70	−4	284
Mean	33 ± 5	−83 ± 7	−12 ± 6	272 ± 21
lOFA
S01*	−34	−80	−23	256
S02†	−38	−75	−12	254
S03‡	−39	−77	−19	250
S04†	−46	−74	−9	296
S05
S06¶	−39	−81	−26	263
S07‡	−28	−90	−3	278
S08§	−45	−74	−17	298
S09§	−21	−86	−18	254
S10§	−38	−78	−11	297
S11#	−43	−73	2	282
S12‡	−35	−74	−7	284
S13
S14§	−40	−74	−13	250
Mean	−37 ± 7	−78 ± 5	−13 ± 8	272 ± 19
rFFA
S01¶	36	−45	−27	426
S02†	39	−37	−17	504
S03||	47	−48	−19	443
S04¶	39	−48	−22	487
S05†	35	−42	−20	464
S06§	37	−42	−22	474
S07§	38	−41	−25	497
S08†	36	−46	−22	495
S09¶	41	−52	−22	452
S10§	42	−41	−19	470
S11§	36	−50	−17	473
S12‡	37	−35	−19	462
S13¶	36	−43	−14	496
S14§	40	−41	−15	495
Mean	39 ± 3	−44 ± 5	−20 ± 4	474 ± 2
lFFA
S01§	−37	−50	−15	431
S02||	−43	−38	−19	501
S03¶	−47	−49	−18	449
S04§	−39	−49	−22	486
S05§	−35	−63	−11	429
S06§	−45	−43	−21	471
S07§	−44	−43	−25	491
S08†	−41	−41	−17	464
S09*	−46	−35	−23	447
S10§	−45	−57	−14	459
S11§	−49	−52	−22	472
S12‡	−39	−48	−22	478
S13¶	−35	−43	−16	499
S14¶	−40	−47	−12	487
Mean	−42 ± 4	−47 ± 7	−18 ± 4	469 ± 23
rAMY
S01†	17	−7	−14	234
S02*	16	−6	−12	256
S03
S04*	14	−4	−14	253
S05†	21	−4	−14	158
S06**	13	−6	−22	264
S07‡	16	−9	−14	287
S08**	13	−6	−15	300
S09*	16	−6	−13	253
S10*	19	−8	−12	281
S11*	17	−6	−13	258
S12#	19	−8	−10	166
S13‡	16	−5	−8	261
S14*	18	−4	−7	188
Mean	17 ± 2	−6 ± 2	−13 ± 4	243 ± 45
lAMY
S01†	−18	−7	−11	230
S02#	−19	−4	−13	106
S03
S04#	−21	−7	−14	173
S05*	−21	−8	−13	146
S06**	−20	−4	−20	267
S07*	−16	−9	−13	265
S08**	−15	−5	−13	269
S09*	−21	−5	−14	265
S10*	−22	−14	−11	272
S11*	−16	−7	−16	250
S12*	−20	−4	−10	188
S13*	−18	−4	−8	221
S14#	−21	−9	−10	177
Mean	−19 ± 2	−7 ± 3	−13 ± 3	218 ± 55
rAIT
S01*	44	−3	−22	292
S02*	28	0	−30	282
S03†	30	−3	−27	295
S04*	35	−4	−31	293
S05#	40	−14	−34	297
S06§	25	−4	−35	283
S07‡	28	−3	−32	289
S08#	29	−7	−34	288
S09‡	27	4	−36	288
S10*	35	−4	−29	283
S11*	31	−8	−34	300
S12#	32	−6	−37	296
S13**	17	8	−32	291
S14‡	35	2	−35	292
Mean	31 ± 7	−3 ± 5	−32 ± 4	291 ± 5

Clusters reported here were found after initial transformation of each subject’s data into Talairach space. Symbols indicate statistical threshold applied for individual ROI definition: *P < 0.050 (uncorrected); ^†P < 0.125 (Bonferroni-corrected); ^‡P <0.001 (uncorrected); ^§P < 0.005 (Bonferroni-corrected); ^¶q < 0.050 [false-discovery rate (FDR)]; ^#P < 0.100 (uncorrected); ^||q < 0.100 (FDR); **P < 0.250 (uncorrected). x, y, and z are Talairach coordinates in millimeters; mean coordinates and cluster sizes (±SD) are provided.

Fig. 4.

Stimulus-aligned time courses for familiar and unfamiliar face trials within face-preferential ROIs. Mean time courses along with upper and lower bootstrapped CIs per condition. Time courses depicted here were created by deriving each subject’s average time course per ROI (based on 20 unfamiliar/familiar face trials; aligned to stimulus onset) and then averaging across subjects. Colored squares illustrate mean z-coordinate locations across subjects on transverse slices (center column). lFFA, left FFA; lOFA, left OFA; rFFA, right FFA; rOFA, right OFA.

Fig. 5.

RMSE of first-order (linear), second-order (quadratic), and third-order (cubic) functions for BOLD responses to familiar and unfamiliar faces in face-preferential ROIs. Activation profiles are considered linear if the fit of a first-order linear function is not significantly improved by second- or third-order functions. Conversely, activation is considered nonlinear if either second or third-order functions fit the data better than a first-order function. *P < 0.025; °P < 0.05; ns, not significant. NF, function fitting not possible because of noisy data (slope: 0).

Table 2.

Univariate analyses of familiarity effects within face-preferential ROIs

ROI	TR
		Familiar CI	Unfamiliar CI
rAMY	18	[−0.02; 0.41]	[−0.32; −0.03]
	19	[0.06; 0.48]	[−0.25; 0.01]
lAMY	20	[0.18; 0.55]	[−0.25; 0.17]
rAIT	20	[0.02; 0.15]	[−0.34; −0.01]

Timing (in volumes, TR) of significant differences in neural activation magnitudes for familiar and unfamiliar faces, as well as bootstrapped CIs as plotted in Fig. 4.

Table 3.

Univariate analyses of neural activation onset within face-preferential ROIs

ROI	Familiar		Unfamiliar
	TR	CI	TR	CI
rOFA	13	[0.09, 0.69]	13	[0.09, 0.83]
lOFA	10	[0.03, 0.38]	9	[0.07, 0.36]
rFFA	10	[0.07, 0.23]	9	[0.03, 0.21]
lFFA	10	[0.03, 0.22]	10	[0.01, 0.18]
rAMY	19	[0.06, 0.48]	NA	NA
lAMY	19	[0.07, 0.46]	NA	NA
rAIT	20	[0.01, 0.15]	NA	NA

Timing (in volumes, TR) of onset of significant activation within face-preferential ROIs per condition, as well as bootstrapped CIs as plotted in Fig. 4.

Familiarity-dependent differences in activation.

Increased activation for familiar over unfamiliar faces was observed only within the anterior ROIs: bilateral amygdala and right AIT (Fig. 4; for timing and 95% bootstrapped CIs, see Table 2).

Activation-onset differences.

Significant activation relative to baseline was observed relatively early in bilateral FFA and OFA. In bilateral amygdala and right AIT, significant signal increase was observed later and only in response to familiar faces. (For onset timing and 95% bootstrapped CIs, see Table 3.)

Response profiles.

As further illustrated in Fig. 4, individual ROIs differed regarding their response profiles. Visual inspection revealed that neural responses in posterior regions (OFA and FFA) increased progressively with HSF information, whereas those in anterior ROIs (bilateral amygdala and right AIT) exhibited an activation profile that was more categorical (i.e., familiarity-dependent). To quantify these differences, we fit polynomial functions to the time courses to verify whether linear (first-order) or nonlinear (second- and third-order) functions would best account for the observed activation patterns. The results indicate that responses in bilateral OFA and FFA were better explained by a simple linear function (i.e., neither second nor third-order functions provided a significantly better fit; Fig. 5). In contrast, response profiles in the bilateral amygdala corresponded better to complex nonlinear functions. This was especially true for the familiar condition, because responses for unfamiliar faces were more variable. High noise levels prevented function fitting to data derived from the right AIT. Quantitative analyses therefore indicate that BOLD responses in the posterior ROIs rise proportionately with increasing HSF information, whereas the activation in anterior ROIs emerges more abruptly and in a familiarity-dependent manner.

Multivariate analyses.

Multivariate analyses were performed to further examine potential differences in the processing of familiar and unfamiliar faces that might have been missed by univariate analyses. Our approach involved correlating the multivoxel patterns elicited by different identities, leading to asymmetric matrices, rather than the commonly used symmetric matrices (56, 57). Our goal was not to determine the neural representations of individual faces (i.e., identities) (58, 59) but to capture differences in pattern similarity related to the familiarity of stimuli presented. Therefore, we averaged the correlations of patterns observed for all pairs of familiar or unfamiliar identities, respectively.

As illustrated in Fig. 6, these analyses involved the creation of single trial representational dissimilarity matrices (stRDMs) (see Methods for details), which were computed for neural responses aligned in two ways: (i) to the onset of visual stimulation; or (ii) to individuals’ behavioral responses (i.e., familiarity decisions). Fig. 7 shows the resulting stRDMs. Reported below are the means of the 1 − Pearson correlation coefficient (1 − r) values of repetition times (TRs) showing significant differences between familiar and unfamiliar faces. These differences were revealed by the 95% bootstrapped CIs computed on the Fisher z-transformed mean of the 1 − r values normalized by the SE of the difference between the two conditions (for details and CIs, see Methods and Table S2).

Fig. 6.

Illustration of the procedure of creating stRDMs. The procedure for stRDM creation is described in detail in Methods; for illustrative purposes, the y and x axes of each matrix display the average time course across all voxels in a given ROI (depicted here are matrices for familiar faces from the right FFA of an exemplary subject). 1 − r values were computed among all voxels across all trial (i.e., face) combinations, independently per time point, ROI, subject, and condition.

Fig. 7.

Single-trial multivoxel pattern similarity between individual familiar and unfamiliar faces in face-preferential ROIs. Displayed here are the average stRDMs across subjects per condition for stimulus-aligned and response-aligned multivoxel patterns. Note that these represent asymmetric matrices resulting from intercorrelations of voxel patterns elicited by different identities (Methods). Colder and warmer colors represent relatively lower and higher dissimilarity, respectively. Correlational distance (1 − r) values are scaled independently per ROI. Significant differences between the multivoxel patterns for familiar and unfamiliar face trials are indicated by red squares. For stimulus-aligned stRDMs, vertical and horizontal white lines indicate timing of mean RTs; for response-aligned stRDMs, the lines illustrate the point to which all individuals trials were aligned. For response-aligned stRDMs, differences in the multivoxel patterns were observed only after behavioral responses in the bilateral amygdala as well as lFFA (Fig. 4). Mean z-coordinate locations across subjects are illustrated by colored squares on transverse slices in the middle column (Table S1).

Table S2.

Multivariate analyses of response characteristics of face-preferential ROIs

ROI	CI
Stimulus-aligned
Left amygdala	−0.21; −5.52
Left FFA	−0.03; −5.95
Response-aligned
Right amygdala	−0.66; −7.28
Left amygdala	−0.23; −3.63
Left FFA	−0.25; −5.53

Provided here separately for stimulus-aligned and response-aligned analyses are the CIs of the bootstrapped differences between conditions that reached significance (based on the Fisher z-transformed 1 − r values as described in Methods).

Stimulus-aligned.

The 95% bootstrapped CIs computed on the mean stRDMs aligned to onset of visual stimulation revealed significant differences only within the left amygdala and left FFA. These differences, reflecting higher similarity among the multivoxel patterns elicited by familiar compared with unfamiliar faces, were found between 21 and 25 TRs in the left amygdala (means familiar vs. unfamiliar: 0.92 vs. 0.97) and between 15 and 21 TRs in the left FFA (means familiar vs. unfamiliar: 0.85 vs. 0.89). No significant differences were found in any of the remaining ROIs.

Response-aligned.

The 95% bootstrapped CIs computed on the mean stRDMs aligned to subjects’ behavioral responses also revealed significant differences between the multivoxel patterns in the bilateral amygdala and left FFA. In these regions, the multivoxel patterns elicited by familiar faces were more similar than those of unfamiliar faces, whereas no significant differences emerged for the remaining ROIs. The results for the left amygdala mirrored those observed for the stimulus-aligned analyses, with significant differences emerging between seven and eight TRs after aligned responses (means familiar vs. unfamiliar: 0.89 vs. 0.96). In the right amygdala, these differences occurred earlier, between three and eight TRs after aligned responses (means familiar vs. unfamiliar: 0.92 vs. 0.98). Significant differences were also observed within the left FFA; here, the multivoxel patterns of familiar faces were more similar than those of unfamiliar faces between six and eight TRs after responses (means familiar vs. unfamiliar: 0.85 vs. 0.89) (Fig. 7).

Discussion

Role of Core and Extended Regions in Signaling Face Familiarity.

In the present study, we used an original stimulation paradigm in which spatial frequency information revealing face identity increased gradually throughout the course of visual stimulation. Adopting both whole-brain and ROI analyses, we sought to test whether core face-preferential areas assess the known-versus-unknown status of faces, sending information forward to anterior regions of the extended face-processing system for storage and affective processing (16, 20, 35) or rather if the categorization of faces according to their familiarity emerges in anterior ventral and medial temporal regions.

Increasing HSF content to reveal personally familiar, compared with unfamiliar, faces led to increased activity predominantly within medial and anterior temporal structures, in particular, the bilateral amygdala, right perirhinal cortex, right hippocampus, and AIT regions. Clusters in bilateral amygdala and AIT were face-preferential, as revealed by independent face localizer scans. Importantly, these medial and anterior temporal lobe regions exhibited an abrupt, rather than progressive, increase of activity to gradually revealed familiar faces. In contrast, neural activity within posterior regions (i.e., bilateral OFA and FFA) increased progressively during the dynamic visual stimulation. Furthermore, these latter regions did not exhibit significant familiarity-dependent differences in their overall response. Therefore, taken together, these findings do not support the view that explicit familiarity decisions involve initial assessment of the known-versus-unknown status of faces within core posterior face-preferential areas, which would then be propagated to anterior regions of the extended face-processing system involved in storage of semantic information and affective processing (16, 20, 35). Rather, following a detailed analysis of individual faces in posterior regions of the core network, the categorical distinction between personally familiar and unfamiliar faces appears to emerge in anterior ventral and medial temporal regions.

This conclusion holds even with in-depth analysis of multivoxel patterns of activity. When neural responses were aligned to behavioral responses (see also ref. 60), the patterns elicited by familiar faces were more similar than those elicited by unfamiliar faces only in the left FFA, not the right hemisphere homolog or bilateral OFA. However, this effect, which we interpret as categorical signaling of face familiarity, was also observed more robustly in the amygdala. Moreover, familiarity-dependent differential patterns at the voxel level in the left FFA arose after familiarity discrimination in the amygdala, indicating that they may reflect feedback inputs from this structure and other anteriorly located regions of the temporal lobe, rather than an early signaling of face familiarity.

Previous investigations have largely concentrated on posterior regions in the core face-processing system. This can be partly attributed to susceptibility artifacts, which affect anterior face-preferential regions that have only recently been systematically targeted given the emergence of approaches to recover signal loss (e.g., refs. 61–63). This relative lack of evidence for the extended system is unfortunate, given that anterior regions have been suggested to link perceptual processing with semantic information about unique exemplars (32) or social concepts including familiarity or group membership (64, 65). Furthermore, nonhuman primate studies have reported that the ventral portion of face-preferential AIT is relevant for face identification (53, 66–68). Here, we were able to define face-preferential clusters in the AIT that were modulated by familiarity of the faces. The finding of categorical signaling of face familiarity in this region is in line with neuropsychological studies reporting face recognition impairments following anterior temporal lobe lesions, which have also indicated a crucial role of the right hemisphere (e.g., refs. 40 and 69–72). Early conceptions of prosopagnosia posited a relationship between lesion location and deficits observed behaviorally, with posterior and anterior lesions giving rise to apperceptive and associative deficits, respectively (73, 74). However, closer inspection of patients’ performance has revealed that individual face discrimination (i.e., perceptual processing) is impaired following anterior inferotemporal lesions (69). Similarly, both posterior and more anterior temporal lesions are associated with pronounced difficulties in recognizing both familiar and unfamiliar faces (6–8, 25, 75–77).

Personally Familiar Face Perception and Recognition in the MTL.

The activation of a triad of MTL regions, encompassing the amygdala, perirhinal cortex, and hippocampus, is in agreement with studies that have reported activation within MTL regions to personally meaningful stimuli (32, 51–53). Contrary to the traditional conception of the amygdala representing a threat detector or fear module, this structure is increasingly considered to fulfill a more general-purpose role related to signaling ambiguity, salience, and signal of interest (78–80). Moreover, previous studies have reported differential amygdala responses as a function of personal relevance or familiarity of faces (23, 24, 81–83). Our results support these findings and provide evidence that the amygdala can show the earliest and most pronounced differentiation between faces based on their familiarity.

In line with previous findings (11, 23, 34, 84, 85), we found significantly larger activation for familiar compared with unfamiliar faces within the perirhinal cortex and hippocampus, supporting the involvement of these structures in memory retrieval. However, our findings are also in agreement with more recent evidence suggesting that both structures, which are strongly connected with inferior temporal visual and prefrontal regions (27, 86–88), play a crucial role in perception (25, 29, 89) and assessment of “the significance of entities” (90). The perirhinal cortex, which codes feature conjunctions viewpoint-invariantly (91), is involved in complex visual discrimination (92, 93) and has been proposed as an extension of the representational hierarchy in the ventral system (94–98). These characteristics, along with the anatomical location of the perirhinal cortex, render this structure ideally situated to link perception and semantic memory, as required for generation and maintenance of face representations for face individuation (26) and face discrimination across different viewpoints (30, 31, 99). Our finding of familiarity-related activation increases in the hippocampus mirrors findings suggesting the hippocampus’s role in activating preexisting knowledge about faces (100). Additionally, our visual stimulation paradigm might have been particularly suited to engage this structure, which shows a bias toward pattern completion processes and coarse global representations (101), as well as being involved in complex visual discrimination tasks (54).

Generalization to Other Modalities and Stimulus Material?

Given previous evidence that individual faces are processed in a coarse-to-fine manner (42–44) and that face perception is more sensitive to spatial frequency than visual processing of other complex nonface categories (102, 103), faces arguably represent ideal stimuli for the paradigm used here. However, the extent to which the findings reported here are unique to faces or could be observed for other familiar (non)visual entities remains unknown (39, 104). A related issue concerns the extent to which categorical signaling of familiarity necessitates the use of personally relevant stimuli (32, 51–53) or whether it would also occur for famous face stimuli. Furthermore, using personally familiar and unfamiliar faces and voices, a previous study (104) reported familiarity-related, “modality-unspecific” activation increases in the cingulate gyrus (BA23). Our whole-brain analysis revealed a cluster within this region, which is involved in episodic memory and emotional salience (105, 106) that exhibited increased BOLD response for familiar faces. Therefore, further studies are required to determine whether the findings reported extend other modalities and other types of visual stimuli.

In addition, personal experience with the faces leads to associations between visual representations of these faces and semantic information, or names. Thus, one may argue that MTL structures activated for familiar faces are essentially involved in retrieval of semantic or naming information. However, several elements suggest that the face familiarity effects observed in the MTL and AIT reflect the activation of visual representations, irrespective of or in addition to semantic information and name associations. First, the amygdala showed a face-preferential response as defined by an independent face localizer with unfamiliar faces (i.e., no names associated with these faces). Second, this structure was right lateralized (589 voxels vs. 274 in the left), and the hippocampus was activated only in the right hemisphere for familiar faces. This right lateralization is a signature for face, as opposed to name/semantic information processing. For instance, right lateralized MTL activation has been found for information retrieval based on pictorial representations (107), whereas retrieval of semantic information has been associated with left lateralized activation in MTL structures (108). Moreover, the right anterior temporal lobe is activated when semantic information is accessed based on the face (109, 110), whereas the left anterior temporal lobe is involved rather in linking semantic information to the language system for name production (110). Finally, there is evidence that the reported MTL regions are involved in perception (25, 29, 89), in particular, the perceptual discrimination of faces and objects (e.g., refs. 30, 91, and 99).

Conclusions

To summarize, the present results suggest that both the posterior core regions and the more anterior ventromedial temporal regions contribute differentially to decisions about face familiarity: core face-preferential regions accumulate perceptual evidence that is used to signal familiarity in areas of the extended system, including both face-preferential and nonpreferential medial and anterior ventral temporal structures. Whether the observations reported here also arise without explicit discrimination of familiar and unfamiliar faces (i.e., an orthogonal task) or when face identity and spatial frequency information are not varied in conjunction will have to be determined by future studies.

Methods

Participants.

Fourteen final-year psychology students (two males; mean age: 23 ± 1 y; one ambidextrous) attending the same courses at the University of Louvain, Belgium, participated in the study. Subjects received financial compensation for participation. Written informed consent was obtained following the procedures approved by the University of Maastricht.

Stimuli.

Full-frontal color photographs of 20 female Caucasian students (half unfamiliar) taken under standardized conditions were used for stimulus creation. Per identity, 18 images were produced that contained increasing amounts of HSF information (starting point: 1.5 c/f; increments of 1/4 octave), in addition to an “average” face stimulus created based on all 20 images containing 1.5 c/f (Fig. 1). Paired sample t tests on the entropy (computed independently per face and spatial frequency) revealed no global low-level visual differences across familiar and unfamiliar faces at any spatial frequency.

Procedure.

Participants completed two runs of a block-design face localizer, in addition to two runs of the slow event-related fMRI experiment (20 randomly presented trials; identities repeated across runs; see Familiarity Decision Task for details). Fig. 1 illustrates the procedure with which face identity was revealed gradually via continuous presentation of face stimuli containing increasing amounts of HSF information during the familiarity decision task. This task required subjects to indicate (by pressing one of two defined response keys) whether the identity presented on a given trial was familiar or unfamiliar.

Image Acquisition.

Scanning took place at the Maastricht Brain Imaging Center, using a 3T head scanner (Siemens). T1-weighted structural images were obtained with 1 × 1 × 1 mm spatial resolution (acquisition matrix: 256 × 256), using ADNI (Alzheimer's Disease Neuroimaging Initiative) sequence [echo time (TE): 2.6 ms; TR: 2,250 ms; flip angle (FA): 9°; field of view (FOV): 256 mm). Functional data from localizer scans were obtained from 36 transverse slices (spatial resolution: 3.5-mm isovoxels; acquisition matrix: 64 × 64), using a repeated single-shot echoplanar imaging sequence (TE: 50 ms; TR: 2,250 ms; FA: 90°; FOV: 224 mm). For localizer scans a 25° angle perpendicular to the main magnetic field B0 was used to reduce magnetic artifacts and signal dropout, allowing recording up to the anterior inferior temporal lobe (111). Data from experimental scans were obtained from 20 transverse slices for the sake of maximizing temporal resolution. With exception of slice number and TRs (here: 1,250 ms), the remaining acquisition parameters were identical; slice orientation was adapted individually to ensure covering of the anterior inferior temporal lobe given previous reports of familiarity effects within this region (e.g., refs 11, 34–39, and 41). For some subjects, this procedure implied reduced coverage of the superior temporal cortex.

Data Preprocessing.

Functional images were preprocessed using Brain Voyager QX (Version 2.1.0, Brain Innovation). The first four volumes of each functional dataset were discarded to avoid saturation effects. Preprocessing steps included slice scan time correction, linear trend removal, high pass filtering (removing frequencies lower than two cycles/session, ∼0.003 Hz for experimental runs and 0.005 Hz for localizer runs), and 3D motion correction (with realignment to the respective first volume). Both anatomical and functional data were transformed into Talairach space (112).

FMRI Data Analyses.

Analyses were conducted using Brain Voyager QX (BVQX) (Version 2.8.1; Brain Innovation) and MATLAB 7.5 (2007b). Whole-brain analyses identified clusters exhibiting enhanced BOLD responses for unfamiliar/familiar face trials. ROI analyses aimed at univariately characterizing response profiles as well as the underlying multivoxel patterns within face-preferential regions. MVPA involved computation of representational dissimilarity matrices (56, 57) per subject based on correlations of individual familiar or unfamiliar face trials, to compare the dissimilarity among familiar and unfamiliar faces (see Multivoxel ROI analyses and Fig. 6).

Whole-Brain Analysis.

A separate-subject random-effects generalized linear model (GLM) was carried out independently per voxel using BVQX (Brain Innovation). A predictor time course was obtained by convolution of a condition time course with a two-gamma hemodynamic response function (spatially smoothed data, FHWM: 6 mm). Epochs were extracted from the onset of the average face to the offset of the last face stimulus on a given trial (Fig. 1). Clusters of voxels exhibiting significantly larger signal increase for familiar compared with unfamiliar trials were identified using the appropriate contrast ([familiar–unfamiliar]; cluster size threshold: 15 voxels; P < 0.003). Face-preferentiality of these clusters was examined through separate-subject, random effects GLMs of the time course data extracted from localizer scans. Two different types of contrasts were performed ([faces–cars], [faces–scrambled faces]; Table 1); a cluster was considered face-preferential if both contrasts yielded significant responses and these resulted from differential activation (deactivations were not considered).

ROI Analyses.

Univariate ROI analyses.

For each subject and ROI, we derived average time courses (across all voxels) for each condition by averaging the signal elicited two TRs before stimulus onset to five TRs, on average, after the end of the trial (i.e., −2 to 32 TRs) We then sampled subjects’ average time courses with replacement and computed the respective sample means; the 95% CIs (113) computed for these bootstrapped samples were informative for the estimation of neural activation onset (i.e., above baseline signal changes), as well as differences between conditions. Significant increases in BOLD was inferred when bootstrapped CIs did not include zero; significant differences between conditions by nonoverlapping bootstrapped CIs. The activation profiles in each ROI were quantified by fitting first-, second-, and third-order polynomial functions to the time courses from stimulus onset until peak activation. A leave-one-out cross-validation procedure was applied to assess the degree of fit of each type of function. We computed the mean error of fit (RMSE) between the fitted functions and the test subjects’ time course. RMSE values of the first- vs. second-order and first- vs. third-order functions were then compared with pair-wise t tests in each ROI and condition. An activation profile was considered linear if second- and third-order functions did not significantly improve fit [i.e., had lower RMSE values (P < 0.025, one-tailed, Bonferroni-corrected)]. Conversely, an activation profile was considered nonlinear if either second- or third-order functions provided a better fit.

Multivoxel ROI analyses.

We developed a single-trial fMRI approach (see also ref. 60) to assess the evolution in the degree of dissimilarity between neural representations elicited by different identities within a given face category (i.e., familiar or unfamiliar). Dissimilarity of neural responses associated with unfamiliar and personally familiar faces was investigated by comparing the multivoxel patterns for epochs aligned to either the onset of the stimulus (“stimulus-aligned”) or to participants’ behavioral responses (“response-aligned”). For both stimulus-aligned and response-aligned BOLD time courses, stRDMs were computed on the BOLD percentage signal change independently per subject, ROI, and condition. Within each ROI and condition, we iteratively correlated (Pearson correlation) the values of all of the voxels at one time point with all of the remaining ones among the epochs of two different trials (e.g., the time course elicited by face 1 and that elicited by face 2). Correlational distance (i.e., 1 − r) was then calculated; this procedure was repeated across all possible trial combinations. The resulting matrices were averaged (20% trimmed mean) to obtain the final stRDM. Fig. 6 demonstrates the underlying rationale: lower dissimilarity (cold colors) between the multivoxel patterns elicited by different identities reflects stimuli being relatively less distinct. Higher dissimilarity (warm colors), on the other hand, is indicative of the stimuli’s relatively greater distinctiveness at the neural level.

To test for statistically significant differences between familiar and unfamiliar stRDMs, we implemented an expanding sliding window approach. Firstly, independently per condition (i.e., familiar and unfamiliar), we normalized the otherwise positively skewed distribution of Pearson r values using Fisher z transformation. We started by centering a 2 × 2 pixel window on the first point of the diagonal of both matrices. We then performed a simple linear subtraction (familiar minus unfamiliar) between all of the values in the 2 windows. The resulting matrix was normalized by the SE of this difference to partially account for the relative differences in data points and variance across windows of different sizes. We computed the 10% trimmed mean of the values within this window and performed 95% bootstrap CI analyses on the mean difference between familiar and unfamiliar stRDMs by sampling subjects with replacement (113). Importantly, we adjusted the threshold for determining high and low CIs as a function of the total number of windows to account for multiple comparison problems (i.e., we applied a Bonferroni correction). The analysis was repeated on increasingly larger windows that expanded by one pixel in each direction (when applicable), centered on each point of the diagonal. Differences between conditions were inferred when the bootstrapped CIs did not include zero. This expanding sliding window approach allows investigating whether potential differences across stRDMs encompass a few time points or whether these are sustained over a larger time window.

Localization of Face-Preferential ROIs.

Face-localizer runs (15, 49, 114) served to identify face-preferential areas and verify the response properties of regions revealed by the whole brain analysis. Seven face-preferential ROIs were defined independently per subject based on (nonsmoothed) functional images from two localizer scans transformed into Talairach space. Using the contrast [faces–cars] in conjunction with [faces–scrambled faces] ensured that activation in the ROIs was not related to low-level properties of faces (49, 114). A multisubject, fixed-effects GLM on the (unsmoothed, Talairach-transformed) data from the localizer scans was performed before individual subject ROI definition; clusters identified here [cluster size thresholding: 15 voxels; P = 0.0005 (Bonferroni-corrected)] served as a guideline for individual ROI selection. Mean Talairach coordinates (112) and cluster size of individual bilateral OFA, FFA, amygdala, as well as right AIT are provided in Table S1. Individual ROIs were selected to include a maximum number of face-preferential voxels without surpassing a threshold we set to 300 anatomical voxels for the OFA, amygdala, and AIT and to 500 for the FFA. This measure was adopted to obtain homolog ROIs comparable in size within and across subjects to ensure that any statistically significant difference stemming from a direct comparison across ROIs could not be related in differential signal-to-noise ratio and statistical power.

Familiarity Decision Task.

Each run of the experiment began with the onscreen instructions, followed by a centrally presented fixation cross for six TRs. The average stimulus presented at the beginning of each trial was presented to ensure that observed differences between both conditions could not be attributed to differences in the initially available visual information. On each trial, the same average stimulus (Fig. 1) was presented for one, two, or three TRs, followed by all 18 images created per identity for one TR each. After a one TR blank, a fixation cross was presented for five TRs on average (duration varied between four, five, or six TRs); trials were separated by a one TR blank. On each trial, all 18 frames were presented (irrespective of individual response times) to ensure that the onset of stimulus presentation on consecutive trials was separated by six, seven, or eight TRs. Behavioral data were acquired from only 11 subjects because of a technical error. Stimulus presentation and response registration was controlled using E-prime 1.1 (Psychology Software Tools). Stimuli were back projected onto a screen located over participants’ heads (∼8.54° of visual angle). Within each of the two runs of the familiarity-decision experiment, the same 20 identities (10 un/familiar) were gradually revealed, leading to two trials per identity across the experiment. Given the aim of the study to determine whether familiarity-related (as opposed to identity-related) differences could be uncovered across the brain, this option was chosen over e.g., inclusion of fewer facial identities and larger number of repetitions. Which of the 10 personally familiar faces (part of a larger cohort of ∼30 students) would be presented was not evident before the experiment, nor were subjects aware of whether stimuli would be repeated. Before scanning, subjects were shown a set of unedited photographs—different from those used to create stimuli; all external features visible—of 20 familiar faces (10 familiar identities presented in the experiment and 10 others not presented) to confirm familiarity with classmates.