The face of image reconstruction: progress, pitfalls, prospects

Abstract

Recent research has demonstrated that neural and behavioral data acquired in response to viewing face images can be used to reconstruct the images themselves. However, the theoretical implications, promises, and challenges of this direction of research remain unclear. Here, we evaluate the potential of this research for elucidating the visual representations underlying face recognition. Specifically, we outline complementary and converging accounts of the visual content, the representational structure, and the neural dynamics of face processing. We illustrate how this research addresses fundamental questions in the study of normal and impaired face recognition, and how image reconstruction provides a powerful framework for uncovering face representations, for unifying multiple types of empirical data, and for facilitating both theoretical and methodological progress.

From neural decoding to image reconstruction

A core endeavor in the cognitive neuroscience of face perception is to elucidate the neural and psychological representations that subserve face recognition [1–4]. Accordingly, neural decoding has been extensively used to investigate face perception by relating the structure of neural patterns to the properties of viewed faces. For instance, facial identity decoding has shed light on the cortical locus [5–9], the temporal dynamics [10–13] and the broad characteristics of face representations, such as the relationship between identity and expression information [14]. However, the finer-grained content and structure of face representations remain to be uncovered. Further, the ability of such representations to support multiple aspects of cognition, including perception, imagery and memory, remains to be clarified.

An increasingly popular type of decoding approach has shown promise in addressing the challenges above. Specifically, image reconstruction encompasses a newer set of methods (Box 1) that convert empirical data associated with visual processing into images that capture the visual properties of underlying representations. As a general strategy, reconstruction relies on the combination of a set of visual features, weighted by the amplitude of specific neural and/or behavioral responses, into images. Beyond this general strategy, however, reconstruction methods vary widely in the types of features utilized, corresponding to different mechanistic hypotheses [15,16], as well as in the procedure for feature combination and in their mapping onto empirical data.

Box 1. Neural decoding, image reconstruction and result validation.

Image reconstruction illustrates an increasingly popular type of decoding approach in the analysis of neural data. In the study of perception, decoding serves to predict stimulus properties from different types of neural recordings. For instance, linear classifiers have been extensively used to decode facial identity, expression and orientation from fMRI data associated with viewing faces [108]. Reconstruction evinces a more ambitious aim of recovering not just broad stimulus properties but an approximation of the stimulus itself by predicting its detailed perceptual properties, such as precise shape and color parameters. In turn, broader stimulus properties may emerge from reconstruction outcomes – for instance, gender and identity are perceptual properties visualizable in a given reconstruction [36]. Similar to other decoding techniques, reconstruction operates on neural patterns and is well suited to capturing distributed representations. However, reconstruction tends to be more computationally intensive as it targets a larger number of variables for prediction purposes (e.g., the values of different visual features and, ultimately, the intensity value of every pixel in an image).

Thus, unlike other forms of decoding, reconstruction is, by design, more thorough and fine-grained. Yet, importantly, its aim is not necessarily that of recovering a perfect replica of a stimulus [109]. If the purpose of visual representations is ultimately that of facilitating appropriate behavioral responses [110], rather than thoroughly capturing the visual world, reconstruction cannot generally be expected to fully recover a stimulus. Hence, the comparison between reconstructions and corresponding stimuli provides, when available, an immediate but only incomplete means of evaluating reconstruction results. Additional validation tools, such as behavioral-based measures of reconstruction accuracy by human observers, can be quite valuable in this sense and are commonly used for assessing facial image reconstruction [31,33,36]. Further, regarding memory-based reconstructions, asking participants to recognize their own reconstructions, without appeal to direct comparisons with experimental stimuli, can provide an important and exacting test of reconstruction success [30,78]. Clearly though, future efforts should be directed not only at the design of better reconstruction algorithms but also of additional, rigorous validation procedures.

Of course, the primary goal of reconstruction may not always be the study of visual representations and, then, the evaluation of reconstruction outcomes would need to consider factors beyond representational accuracy – for instance, improving reconstruction quality through dedicated models that boost usability and utility would be highly relevant in the design of reconstruction-based brain-computer interfaces.

To date, reconstruction endeavors have relied primarily on functional Magnetic Resonance Imaging (fMRI) to retrieve the content of natural images [17–20], movies [21,22], and of specific stimulus categories, such as orthographic characters [23–26] and scenes [27]. In the study of face perception, proof of concept has been provided for the recovery of face images [28] and identifiable facial appearance [29] from fMRI patterns. More recently, facial image reconstruction has been conducted with other data types, including behavioral [30,31] and electroencephalography (EEG) data in humans [32,33] as well as single-unit recordings in monkeys [34]. Further approaches based on neural networks have aimed to improve the accuracy and image quality of facial stimulus reconstruction [35,36]. This new body of work can benefit from a close examination to uncover not only its methodological progress but also its theoretical implications, its merits and its challenges.

The present review focuses on faces, as a target domain, to ensure even ground for the comparison of a variety of reconstruction approaches. Notably, faces yield a restricted but important testbed as they provide the epitome of visual expertise while, also, exhibiting a well-constrained visual structure (e.g., by sharing the same parts in similar configurations) [37]. Historically, their use as a target domain has played a seminal role in the development and validation of decoding techniques, such as those based on multivoxel pattern analysis [5,38]. Similarly, image reconstruction could benefit from a targeted assessment within the context of this visual domain.

In the current review, we show how recent investigations provide a coherent and informative perspective on face processing. Notably, these studies open up new avenues for researching memory representations and visual distortions associated with specific deficits. At the same time, we consider the need to distinguish between recovering the appearance of a stimulus and uncovering the genuine content of visual representations. Accordingly, we attempt to clarify and illustrate how image reconstruction reflects not just properties of face stimuli but, critically, those of neural and psychological representations.

From image reconstruction to perceptual representations

The feasibility of facial image reconstruction has served as a powerful demonstration of the richness of visual content underlying face perception and of the ability to extract this content from neural signals [28]. As face identification stands out as the hallmark of face recognition, reconstruction has the opportunity to shed new light on representations of facial identity. Here, we consider three goals essential to rendering reconstruction relevant for the study of such representations: (i) relating reconstructions to the architecture of a representational space and to behavioral performance; (ii) deriving/analyzing reconstructions into fundamental components of face representations, such as shape and surface properties, and (iii) examining the robustness of reconstruction over image variability (e.g., invariance to viewpoint and expression).

The concept of face space (Box 2), a multidimensional space within which the distance between faces corresponds to their perceived dissimilarity [39], is fundamental to our understanding of face representations [40–43]. Accordingly, recent work [29] has not only related reconstructions directly to face space but grounded them in its architecture based on the structure of fMRI patterns and behavioral data (i.e., ratings of pairwise face similarity). In this work, image reconstruction was achieved in a series of steps (Figure 1): first, by assessing the representational similarity [44,45] across faces from neural and behavioral data; second, by converting similarity estimates into face space constructs; third, by synthesizing visual features from the structure of this space; and, last, by linearly combining such features into reconstructions of target stimuli. The twofold success of deriving identifiable face information from fMRI patterns and from behavioral data demonstrated the ability of reconstruction to uncover the visual codes structuring the architecture of face space. This demonstration was reinforced by the correspondence between behavioral and neural-based face spaces and, also, between their respective reconstruction outcomes.

Box 2. . Face space.

The concept of face space has provided a seminal framework in the study of face processing. Originally, face space was introduced as a psychological similarity space [39]. Within this space each face is represented by a location and the distance between faces corresponds to their perceptual (dis)similarity: the more distant faces are in this space, the more dissimilar they are to each other.

An influential norm-based model postulated that faces are encoded relative to a specific prototypical face (i.e., a norm face) located at the origin of the space, with less typical faces placed further away from the origin [111]. Alternatively, in an exemplar-based model [112], faces are represented in the space without reference to a central prototype and the distinctiveness of a face is determined by how densely populated its designated region of space is.

The dimensions of the space correspond to properties or parameters by which faces vary. In the first computational account of face space, dimensions corresponded to eigenfaces derived through the application of principal component analysis (PCA) to face images [68]. PCA-based models have been useful both in psychology, to explain a plethora of behavioral effects, and in computer vision, as the basis for the first generation of automatic face recognition systems.

However, an eigenface-based face space is highly limited in its capacity to support and explain face recognition (e.g., because of the inability of its image-based features to deal with invariance). Subsequent approaches made advances by applying PCA separately to shape and surface properties rather than directly to images [51]. More recently, a new generation of face space models have been constructed from latent features encoded on the layers of DCNNs with various architectures and training regimens [3,61]. Modeling neural-based face space across distinct cortical regions with corresponding spaces from distinct DCNN layers is particularly appealing as a way to capitalize on the hierarchical nature of visual processing in both biological and artificial systems.

Overall, computational approaches in the study of face space are critical to assessing its properties: its dimensionality, its similarity metric, its flexibility to learning and the nature of its dimensions. Image reconstruction provides a new and promising direction of research in the study of these properties.

Figure 1. Facial image reconstruction informs fundamental aspects of face representation.

Notably, reconstruction speaks to the topography of face space and to the separate encoding of shape and surface information. Reconstruction can rely on multiple types of neural and behavioral data as well as directly on image information (e.g., processed by a model or a theoretical observer). An illustration of image reconstruction methodology (adapted with permission from [33]) involves estimating a face space construct from the structure of experimental data, followed by the synthesis of shape and surface features from face space and the combination of such features into image reconstructions of a viewed face (only a representative subset of facial landmarks are displayed for shape; surface features are displayed for L*, a* and b*, which correspond to the lightness, red-green and yellow-blue channels of color vision).

Of note, the demonstration above was conducted with faces that are highly homogeneous in appearance (e.g., front views of adult Caucasian male faces). While reconstruction efforts often consider a wider scope of image variability as introduced, for realistic stimuli, by race, gender, and pose [28,36], face recognition is remarkable in its ability to discriminate even amongst highly similar individuals. Hence, synthesizing discriminable images from data associated with homogeneous stimuli provides a complementary strategy that takes direct aim at fine-grained representations while, also, delivering an exacting test of reconstruction.

An important question, however, is how reconstruction and the underlying face space reflect fundamental properties of face representations. Notably, the segregation of shape and surface information defines a key aspect of visual processing [46], with both types of information supporting critical aspects of face recognition [37,47–50]. A landmark study relying on single-unit recordings in the macaque brain [34] examined this segregation and its contribution to reconstruction. The results showed that shape and surface features are predominantly encoded by groups of neurons in different patches of inferotemporal (IT) cortex (Figure 2a). An examination of shape and surface features, based on active appearance models [51], demonstrated that such features can be efficiently decoded from neural recordings and integrated into image reconstructions (Figure 2b). Importantly, this study demonstrated that IT neurons exhibit tuning to specific features rather than encoding information about face exemplars, such as ‘concept cells’ in the human hippocampus [52,53]. Theoretically, these neurons function as the axes of a space that flexibly supports the distributed representation of any number of faces (i.e., as linear projections). Conversely, a linear combination of feature/axis values, through the readout of corresponding neurons, enables image reconstruction. Hence, these results establish the neural validity of an axis model of face representation relying on separate sets of shape and surface features.

Figure 2. Cortical areas that support identifiable facial image reconstructions across variation in viewpoint and expression.

(a) Sagittal slices showing the location of fMRI-mapped face patches in one monkey along with (b) examples of stimuli and their corresponding reconstructions based on single-unit recordings (SUR) in these areas (adapted with permission from [34]). (c) Axial slices showing the location of cortical areas in the right anterior fusiform gyrus (FG) and the left posterior FG in humans along with (d) examples of stimuli and their corresponding reconstructions based on multivoxel patterns in these areas - additional reconstructions based on behavioral and EEG data are also shown for the same stimuli (adapted with permission from [29] and [33]).

Yet, one critical issue for reconstruction concerns the retrieval of invariant face representations (see Glossary). Our ability to recognize facial identity despite considerable variation in appearance [4], such as that due to changes in viewpoint and expression, suggests that some aspects of representations and the underlying face space topography are invariant to such changes. Hence, one should assess whether reconstruction captures features robust to identity-preserving transformations. Namely, does reconstruction retrieve only lower-level image properties (e.g., average lightness and image contrast) or higher-level facial identity properties? An examination of reconstruction results with front-view and profile faces from single-unit recordings [34] showed that a shared face space topography underlies both sets of reconstructions. A similar conclusion was reached by an examination of EEG-based reconstructions of faces displaying neutral and happy expressions [32]. These findings strongly point to the ability of reconstruction to capture invariant visual information regarding facial identity.

Thus, image reconstruction concurrently validates and capitalizes on fundamental aspects of face representation: face space topography, the segregation of different visual properties and the use of invariant features for recognition purposes. Accordingly, reconstruction may be utilized to probe an entire range of aspects regarding face representations, such as their spatiotemporal profile and their computational underpinnings.

Image reconstruction: where, when and how

What cortical areas subserve image reconstruction? Current work demonstrates that fusiform gyrus (FG) regions in the human brain, including a region anterior to the fusiform face area, support reconstruction [29] and the extraction of underlying shape and surface features [33] (Figure 2c). Considering the presumed homology between the cortical systems for face processing in the human and monkey brain [54,55], these findings are in broad agreement with the corresponding ability demonstrated for middle and anterior face patches in monkey IT cortex [34]. More importantly, the location of these regions concurs with the encoding of invariant features in high-level visual cortex [56].

Yet, neither face selectivity nor successful classification of facial identity guarantees the ability of a cortical region to support reconstruction. For instance, neither the face-selective occipital face area (OFA) nor inferior frontal gyrus (IFG) areas capable of identity classification appear to support identifiable reconstructions [29]. Presumably, this reflects the involvement of the OFA in lower-level face processing and the role of IFG areas in higher-level semantic processing [57]. Critically, these findings illustrate a notable benefit of reconstruction: explicating if and how selectivity and/or classification results rely on relevant visual information as opposed to higher-level conceptual properties or task-related attributes. However, when one considers a broader range of stimuli, encompassing variation due to race, gender or pose, reconstruction relies on a larger network of regions including, for instance, occipital areas [28,36]. Thus, when reconstruction aims to recover visual properties beyond those strictly needed for identification, then it benefits from capturing both lower and higher-level information encoded in the larger network for face processing.

A complementary aspect of image reconstruction concerns its time course as revealed by methods with suitable temporal resolution. In this sense, EEG-based reconstruction has proven not only possible but informative [32]. For instance, both shape and surface features underlying reconstruction can be recovered over an extended period of time, starting around 150ms after stimulus onset and gradually declining after 300ms. Interestingly, a multimodal evaluation showed that EEG-based reconstructions are maximally similar to their behavioral and fMRI counterparts (i.e., shape-based reconstruction based on the posterior FG) in the proximity of the N170 event-related component [33] – see Figure 2c for examples of reconstruction across modalities. This is consistent with the role of the N170 component in identity processing [58,59] and it accounts for the nature of visual information during this time window in the EEG signal. More generally, relating reconstruction efforts across modalities is valuable both for broadening our understanding of face representations and for facilitating the mutual validation of different data types.

However, representational and spatiotemporal aspects of face processing speak only indirectly to its neurocomputational mechanisms. Recently, deep convolutional neural networks (DCNNs) have been explored as models for human face processing [60–63], given their ability to match human-level performance in face recognition [64]. In particular, the hierarchical nature of DCNNs provides an appealing opportunity to compare successive levels of visual processing between such models and the visual system [65–67]. In parallel, DCNNs have also been used as tools for image reconstruction (Figure 3). For instance, recent work [35] has demonstrated the utility of higher-level representations as encoded by higher layers of DCNNs. Specifically, this work used the output of a 14-layer convolutional encoder as latent features for face encoding and decoding. To this end, fMRI responses were related to latent features by inverting a linear mapping of such features onto brain responses. Then, patterns of latent features were converted into reconstructions by inverting the nonlinear transformation from stimuli to features. DCNN features provided superior results when compared to eigenface features, as obtained through the application of principal component analysis (PCA) to face images [68], and utilized by previous reconstruction approaches [28]. Similarly, another study [36] used the output of a 13-layer variational auto-encoder (VAE) to mediate between fMRI responses and stimulus properties. Again, the appeal to VAE features facilitated reconstructions more accurate than those based on eigenfaces and with better image quality. Further, latent feature representations corresponding to occipitotemporal fMRI patterns supported gender classification and pointed to the possibility of face decoding from imagery.

Figure 3. An illustration of deep neural network application to image reconstruction.

(a) An encoder network maps a face image onto latent features, which the generator network converts into a novel face image. The discriminator network evaluates whether a given image, from the original data set or from the generator output, is real or fake (i.e., a generator output). (b) Encoding associates a latent vector with the corresponding brain response pattern. (c) Encoding is inverted to convert a brain response pattern into an estimate of latent face features which the generator network can translate next into a reconstructed face image (adapted from [36]).

Although the studies above focused on boosting reconstruction accuracy and image quality, they also speak to the relevance of face spaces encoded on DCNN layers as models of neural face space. Clearly, a full-blown computational account needs to consider the validity of multiple levels of representations in a DCNN along with the biological plausibility of the transformations performed on these representations [65]. Importantly though, the work above demonstrates the benefit of including reconstruction as a tool in assessing the validity of DCNNs architectures as face processing models.

Perceptual and memory-based face representations

The relationship between perceptual and mnemonic representations defines a key research area in cognitive neuroscience. For example, extensive work suggests that, during mental imagery, visual information is recovered in the ventral pathway via top-down mechanisms for memory retrieval, in contrast to predominantly bottom-up mechanisms employed by perception [69–71]. However, the precise content of memory representations remains to be uncovered. The application of image reconstruction to face memory can be particularly informative given our remarkable ability to remember the appearance of a face. Building on the success of perception-based reconstruction, memory-based reconstruction may elucidate the reliance and the uniqueness of memory representations relative to their perceptual counterparts.

The feasibility of reconstructing face images from working memory has been explored [72] using an eigenface-based approach that linearly maps eigenface features to fMRI patterns (e.g., by fitting a regression model of eigenface scores to multivoxel patterns) [28]. In this study [72], participants imagined the appearance of a target face, singled out by a retro-cue from a pair of face stimuli. Then, corresponding perception and memory-based information was retrieved from the angular gyrus (ANG). Reconstruction outcomes captured information about skin color and emotion and, thus, they were not limited to low-level visual properties. Importantly, reconstruction was used to refute the hypothesis that ANG plays only a content-general role in memory retrieval [73] and to confirm that mnemonic content is represented in the ANG during both encoding and retrieval. However, the diversity of the stimuli (e.g., different races, genders etc.) along with the limited accuracy of memory-based reconstructions precluded definitive conclusions regarding the nature of ANG representations. Namely, while reconstructions were pictorial in format, their success may have reflected the correlation between non-perceptual ANG information and specific eigenfaces (e.g., between semantic information regarding ethnicity and eigenfeatures capturing color variation). In addition, these findings in ANG raise the possibility that other regions, such as medial temporal lobe structures supporting face memory and perception [74], may show more promise for such investigations.

Interestingly, behavioral-based reconstruction [30] has demonstrated the ability of long-term memory to encode fine-grained visual information. A method relying on similarity ratings between visible stimuli and faces retrieved from long-term memory recovered the appearance of both famous faces and newly learned faces. Stimulus homogeneity along with the robustness of the reconstruction results suggested that detailed visual information, rather than broad semantic information, underlies this success. Surprisingly, the accuracy of memory-based reconstruction for unfamiliar faces, learned during the experiment, surpassed that of perception-based reconstruction. Presumably, this indicates that extensive familiarization with a stimulus may lead to memory representations comparably rich in diagnostic content to their perceptual counterparts [75].

In a similar vein, by appeal to similarity ratings between stimuli and faces retrieved from memory, another study [31] provided evidence for the encoding of 3D shape information in mnemonic representations of personally familiar faces. Further, it showed that face representations did not capture only semantic information regarding gender or age, as such factors were modeled separately in the reconstruction procedure. The robustness of the reconstructions was further evaluated by stringent behavioral tests in which participants had to recognize reconstructions over changes in viewpoint and age, confirming the ability of the method to extract invariant visual information about identity.

Thus, current work confirms the visual nature of memory face representations, takes steps to uncover its content, and points to its underlying neural resources. Further, it suggests the reliance on a representational topography shared with that of perceptual representations [30]. However, the precise relationship between working memory, long-term memory and perceptual face representations remains to be clarified [72]. Also, the reliance of these results on restricted samples of healthy participants raises obvious questions regarding their generality.

The veracity of face representations

How veridical are our face representations and how do they differ across individuals? Image reconstruction provides a unique window into the veracity and idiosyncrasy of face representations both in healthy individuals and in those who suffer from perceptual impairments associated with a variety of visuo-cognitive deficits.

A particularly relevant impairment is prosopagnosia, a deficit in face recognition abilities that is not due to more general problems with vision or memory [76,77]. An investigation of dynamic expression perception in acquired prosopagnosia [78] examined emotion representations in a patient with extensive bilateral occipitotemporal lesions [79]. Consistent with the lesion pattern, face recognition abilities — and identification in particular — were severely impaired. However, regions subserving expression recognition, such as the posterior superior temporal sulcus and the amygdala, were intact, suggesting normal expression representations. To evaluate this hypothesis, expression representations were recovered by relating Facial Action Units (AU) [80] to behavioral performance in an emotion categorization task. Reconstructions of dynamic expressions were comparable between the patient and age-matched controls, confirming the hypothesis above. Yet, intriguingly, the patient was impaired at recognizing static versions of his own reconstructions, suggesting that dynamic and static facial information are subserved by different cortical pathways. These results illustrate a valuable use of reconstruction to characterize expression representations and to elucidate the functional organization of the neural system. They also set the groundwork for future investigations of more thorough representations that include diagnostic surface information, such as color [81], and more complex AU information in compound expressions [82]. However, they do not pinpoint the differences in representational content, between the patient and healthy controls, which may account for the patient’s deficits.

Recent work [83] addressed the challenge of identifying representational [85] differences due to aging. A decline in face recognition abilities associated with healthy aging is well documented [84–86] and likely to involve degrading representations of identity [87]. To confirm and clarify such representational changes, behavioral-based reconstruction was conducted in a group of older adults (62–71 years) and younger controls (18–31 years). The study revealed a loss of representational accuracy in older adults for both shape and surface features critical for recognition, such as eye shape and skin tone [43,88,89]. Interestingly though, aging accounted only for a small proportion of variance in reconstruction results, pointing to the marked impact of individual differences on face representations.

While the veracity and effectiveness of visual representations come under scrutiny as a window into the nature of visual deficits, non-veridical representations extend beyond the scope of such deficits. An intriguing and pervasive example is that of ensemble coding, as documented by extensive behavioral work [90]. Whenever we encounter a group of objects or faces (e.g., a crowd), the visual system seemingly synthesizes a summary representation, by averaging visual properties of individual elements, at the expense of encoding every element. For instance, it may synthesize the representation of an individual with an average expression and identity, though no such individual exists in a crowd [91–93]. To establish the existence of neural summary representations, reconstruction was applied to EEG recordings corresponding to the perception of face ensembles (i.e., groups of faces presented simultaneously) [94]. This investigation recovered summary representations (i.e., the average of individual faces) while showing that information about specific individuals is discarded. It further indicated that ensemble face processing exhibits distinct neural dynamics compared to individual face processing. Thus, reconstruction was instrumental in confirming the neural status of summary representations, in visualizing their appearance and, more generally, in validating an extensive body of behavioral work.

Taken together, the results above demonstrate that reconstruction methodology can reveal not just objective stimulus properties, as encoded by our visual system, but also the complexities of visual mental content that may not accurately and entirely reflect the visual world.

Concluding remarks and future directions

The obvious promise of image reconstruction is that of revealing the fine-grained content of visual representations. Yet, its application to face recognition demonstrates considerably more theoretical value and versatility than a quantitative boost in the detail with which it characterizes representations.

Extending neural-based reconstruction to a behavioral counterpart provides an important opportunity to examine the relationship between brain and behavior [95] in face recognition. Similarly, its applicability to different types of neural data serves to combine their relative strengths and validate each other’s conclusions within and across species. Equally important, the use of DCNNs serves a twofold role in modeling visual processing and in facilitating reconstruction.

To date, reconstruction encompasses a variety of approaches as illustrated by their reliance on active appearance features, DCNN latent features, or those synthesized from behavioral and neural data. Yet, despite this variability, current work provides a cohesive perspective on face recognition that emphasizes the reliance on separate shape and surface features, the validity of an axis model of face space, the encoding of invariant features in high-level visual cortex, as well as a shared representational basis for perception and memory. Further, it demonstrates how changes in visual representations account for decrements in face recognition and how misrepresentations of objective information help the system cope with certain tasks (e.g., processing an abundance of redundant information as an aggregate).

Thus, reconstruction has already provided important insights into face recognition. Yet, its prospects remain at least as impressive. Clearly, the scope of its applicability can be expanded substantially (see Outstanding Questions). A notable challenge is that of characterizing genuine distortions in representations as opposed to just informational loss. Misrepresentations of facial appearance in prosopagnosia and prosopometamorphosia [96], negative biases in expression recognition in borderline personality disorder [97] and schizophrenia [98], face illusions and hallucinations [99] along with perceptual deficits in dementia [100,101], provide rich fields of inquiry that can benefit from this methodology.

Outstanding questions box.

• How do representations of familiar faces differ from those of unfamiliar faces? Extensive work documents differences in recognition performance and in the neural profile of processing familiar versus unfamiliar faces. What aspects of image reconstruction (e.g., content, representational structure, spatiotemporal dynamics) capture these differences and how do they account for them?

• Can we achieve neural-based reconstructions of dynamic facial information with suitable temporal resolution? For instance, can we recover detailed visual representations of dynamic expressions capitalizing on the high temporal resolution of EEG data?

• What does image reconstruction reveal about the other-race effect? Faces of a different race are often recognized more poorly than faces of one own’s race. This difficulty presumably reflects the use of less efficient and/or inaccurate visual representations for outgroup faces. What are the specific features impacted in such representations and how well do they account for recognition decrements characteristic of this effect?

• How does perceptual interference alter the structure of mnemonic face representations? For instance, how does a series of face stimuli impact the representation of a target face recovered from memory? Prior work with simple shapes and colors suggests that visually dissimilar interference erases memory representations, while similar interfering items blur these representations. Is face memory equally sensitive or more resilient to such effects?

• Can image reconstruction elucidate the heterogeneity of visual deficits in prosopagnosia? Recent work has emphasized the diverse profile of face processing deficits and, presumably, their diverse etiology in congenital prosopagnosia (CP). Can image reconstruction facilitate a better categorization and understanding of different CP (sub)types?

• How well does reconstruction methodology targeting faces generalize to other visual categories (e.g., visual words)?

Moreover, memory-based reconstruction opens up the possibility of concrete applications, such as neural-based sketch artists. An important challenge in this sense is the amount of data required for reconstruction purposes for any given individual. Exploiting the mapping between neural recordings and image properties across a larger, more diverse sample of individuals could improve reconstruction via transfer learning [102]. However, such attempts require a rigorous characterization of individual differences in face representation and their neural correlates [103–105]. We believe that current efforts offer a solid foundation for progress in this regard.

Methodologically, as different reconstruction approaches exhibit different biases, we also note the need to carefully compare their results. The present review takes a step in this direction, though experimental studies that compare different approaches across the same population and stimuli should be able to provide more precise insights. Further, reconstruction would benefit from its comparison with the outcomes of more traditional methods, such as noise-based reverse correlation (Box 3).

Box 3. Methodological paradigms for the study of visual representations.

Image reconstruction is related to several methodological paradigms in the study of visual representations. In particular, representational similarity analysis (RSA) has been extensively used to uncover the structure of representational spaces subserving object recognition [44,45,113]. Specifically, RSA estimates the pairwise distance between neural measures associated with processing different stimuli (e.g., images of different objects) and infers, based on such differences, the representational (dis)similarity between corresponding stimuli. RSA is capable of targeting a broad variety of representations, including conceptual and task-related ones, in addition to perceptual ones. Yet, such broadness can also yield underconstrained and ambiguous interpretations of similarity-based results [114]. Relative to RSA, image reconstruction is limited in its immediate scope to visual representations. However, in virtue of such focus, reconstruction can resolve the features underlying the structure of representational spaces by extracting and visualizing the informational content responsible for this structure.

Another class of relevant methods includes noise-based reverse correlation and Bubbles. The former estimates a visual template, aimed at approximating a target representation, from patterns of visual noise (e.g., white noise) added to or supplanting a stimulus image [115,116]. The latter estimates the components of an image (e.g., specific areas or spatial frequency bands) that guide behavioral and/or neural responses [117,118]. These two methods have provided important insights into the visual information that underlies face recognition in both healthy individuals and in those with impaired face processing [119,120]. Similar to image reconstruction, these methods uncover diagnostic image information underlying visual representations. In particular, noise-based reverse correlation aims to synthesize the appearance of a representation, though, unlike reconstruction, it relies on visual noise as opposed to more complex visual features. Importantly, noise-based reverse correlation, through its use of visual noise, limits the possibility of specific biases associated with the reliance on inaccurate or incomplete sets of visual features for reconstruction purposes. However, the substantial number of trials required to recover the appearance of a single template, renders noise-based reverse correlation impractical when applied to a large set of target stimuli and corresponding representations (e.g., dozens or hundreds of individual face representations).

Thus, image reconstruction provides a related but complementary approach relative to current methodological paradigms. Of note, certain types of image reconstruction capitalize on the strengths of these different paradigms and, further, they even embed techniques akin to reverse correlation and RSA into image reconstruction [29,30].

While the present review focuses on faces, further evaluation of reconstruction, as applied more generally to natural images and to other categories of interest, will be beneficial. In particular, we note the recent extension of specific face decoding and reconstruction approaches, as reviewed here [32], to other types of stimuli such as words [106,107]. Hence, future work will need to consider the promise and the limits of cross-domain reconstruction.

To conclude, image reconstruction provides an informative and comprehensive account of visual face representations with notable theoretical and practical prospects. This approach can help to answer long-standing questions in the study of face recognition while also guide similar endeavors in other visual domains.

Highlights box.

• The visual content of human face representations can be derived through image reconstruction at an unprecedented level of detail.

• The neural underpinnings of psychological face representations can be elucidated via image reconstruction from behavioral and multiple types of neural data.

• Facial image reconstruction capitalizes on the structure of face space and also serves to validate its representational properties.

• The recovery of face representations from both perception and memory can help to clarify the relationship between these two components of cognition.

• Image reconstruction can pinpoint the loss of information and/or misrepresentations in face perception across a broad population, including individuals with deficits in face processing.

Glossary

Action Units (AUs)

facial movements, each related to a particular muscle or group of muscles, as described by the Facial Action Coding System (FACS). AUs have proven particularly useful in the physical description and recognition of facial expressions. For instance, AU 6 (cheek raiser) is often used to express sadness while AU 25(lips part) is used to express joy

Deep convolutional neural networks (DCNNs)

artificial networks consisting of multiple layers of simulated neurons. When applied to image processing, DCNNs begin with raw images and recode information across layers by convolution (i.e., a linear combination of neighboring pixels according to the weights expressed in a filter). Convolution implements filters with a restricted spatial receptive field, inspired by neural processing in the visual cortex. The hierarchical nature of DCNNs is also broadly inspired by the organization of the visual cortex

Eigenfaces

Components derived from the application of principal component analysis (PCA) to a set of images. PCA can reduce the dimensionality of a dataset by encoding it via a relatively small number of components that capture the main forms of variability in the data. Conversely, a face image can be recovered fairly well from a weighted combination of a limited number of eigenfaces

Ensemble coding

a type of visual coding deployed for groups of similar objects in the visual environment: the leaves of a tree, a basket of apples, a crowd of people. In order to handle such a wealth of information the visual system is believed to capitalize on its redundancy. Specifically, it may encode a statistical summary representation of all the elements, an average apple or an average face, instead of every single apple or face

Invariant face representations

types of representations that encode visual information independently, to some degree, of variability in individual appearance. Considerable variability in the projection of a face image on the retina occurs due to extrinsic factors, such as angle of view and lighting, as well as to intrinsic factors, such as expression and aging

Mental imagery

visual experience whose content does not directly derive from an external stimulus, as it happens in the case of perception, but, instead, it is derived from memory. Imagery and visual working memory are deeply related with each other though their precise overlap remains to be determined

Prosopagnosia

a deficit in face recognition abilities, and identity recognition in particular, that cannot be explained through a more general perceptual or cognitive deficit. Prosopagnosia can occur as a consequence of brain damage in premorbidly normal individuals (acquired prosopagnosia) or as a lifelong developmental disorder (congenital/developmental prosopagnosia)