Reconstructing seen images from human brain activity via guided stochastic search

Abstract

Visual reconstruction algorithms are an interpretive tool that map brain activity to pixels. Past reconstruction algorithms employed brute-force search through a massive library to select candidate images that, when passed through an encoding model, accurately predict brain activity. Here, we use conditional generative diffusion models to extend and improve this search-based strategy. We decode a semantic descriptor from human brain activity (7T fMRI) in voxels across most of visual cortex, then use a diffusion model to sample a small library of images conditioned on this descriptor. We pass each sample through an encoding model, select the images that best predict brain activity, and then use these images to seed another library. We show that this process converges on high-quality reconstructions by refining low-level image details while preserving semantic content across iterations. Interestingly, the time-to-convergence differs systematically across visual cortex, suggesting a succinct new way to measure the diversity of representations across visual brain areas.

Introduction

Successfully reconstructing even a moderately complex image from its evoked brain activity requires a strong and appropriate prior. In (Naselaris, Prenger, Kay, Oliver, & Gallant, 2009), we used a massive image library as the prior. We used an encoding model for early areas to screen the library for images with low-level details that were consistent with brain activity in early visual areas. A high-level encoding model was then used to select images that were also consistent with brain activity in higher visual areas. This “structure before semantics” strategy was an inversion of reconstruction priorities, as it produced semantically correct reconstructions only in cases where low-level details and semantic content were highly correlated in the image library. This made the success of the method dependent on the content of the library.

We use a recently developed generative model (Rombach, Blattmann, Lorenz, Esser, & Ommer, 2022) to impose a strong prior that guarantees naturalistic reconstructions with interpretable content. We further take advantage of this model to induce a “semantics before structure” search. We search through an image library generated by the diffusion model with guidance from a semantic CLIP embedding (Radford et al., 2021) decoded from brain activity. We then use encoding models for early visual areas(St-Yves, Allen, Wu, Kay, & Naselaris, 2022) to iteratively refine the library until convergence on a final reconstruction.

Methods

We analyze the Natural Scenes Dataset (Allen, St-Yves, & Wu, n.d.). For the subject shown here (S1), we partition the data into training ($\mathrm{n}=\mathrm{20,809}$, used for encoding/decoding model parameter estimation), validation ($\mathrm{n}=\mathrm{4,170}$, used for model regularization), hyperparameter ($\mathrm{n}=552$, used to finetune the iterative search procedure), and test ($\mathrm{n}=\mathrm{2,218}$, used to assess decoding performance) sets. Hyperparameter and test sets derive from a collection of 1,000 images that were seen by all subjects in the NSD. Training and validation sets originate from the remaining data. Our training stage prepares a decoding model to extract CLIP image embeddings $c_I$ from brain activity $\mathrm{\beta }$, voxel-wise encoding models (St-Yves & Naselaris, 2017) that map images $I$ to predicted brain activity $f\left(I\right)={\mathrm{\beta }}^{\prime }$, and a denoising autoencoder that maps measured brain activity $\mathrm{\beta }$ to denoised brain activity $\stackrel{\mathrm{ˆ}}{\mathrm{\beta }}$. During inference, we average $\mathrm{\beta }$ across three repetitions of the target image and decode to $\prime c_I$, which is held constant throughout the inference. We denoise the three fMRI data repetitions individually to produce $\stackrel{\mathrm{ˆ}}{\mathrm{\beta }}$, which we use to guide the search process. We use SD to generate a small library $(\mathrm{n}=250)$ conditioned on $\prime c_I$ and a latent $z_{t=0}$ sampled from a uniform Gaussian. We pass each image in the library, $I_j$, through the encoding models to obtain predicted brain activity ${\mathrm{\beta }}_{\mathrm{j}}^{\prime }$. We scored each image by computing Pearson correlation between the predicted activity pattern and $\stackrel{\mathrm{ˆ}}{\mathrm{\beta }}$. We select the images with the highest average correlation across the three repetitions, encode them as latent representations $z_{t=1}$, and generate a new library. With each iteration, we decrease the strength parameter of the diffusion model, which determines the relative level of guidance of the conditioning $c_I$ and $z_t$ variables during diffusion. For the examples shown here, we performed 10 iterations, reducing the strength parameter along a cubic decay schedule from 1.0 to 0.6. We compare our search-based reconstruction procedure to a CLIP-only decoder in which we generate a single sample conditioned on $\prime c_I$, and a simple library search method in which the encoding model is used to select one image from a large (60K images) library of COCO images.

Results

Preliminary experiments with our “semantics before structure” search converge on reconstructions that depict the right objects at roughly the right location and scale (Figure 1) by refining low-level detail across search iterations (Figure 3), outperforming simpler approaches on several metrics (Table 1).

Figure 1:

Pipeline diagram for our method: the top half demonstrates the Training stage for our models, and the bottom half depicts the Inference Stage that deploys our stochastic search procedure.

Figure 3:

Refinement of reconstructions over search iterations (left to right)

Table 1:

Quantitative comparison of methods.

Methods	PixCorr	SSIM	CLIP
Stochastic Search	.215 ± 0.041	.295 ± 0.042	85.1 ± 9.1 %
CLIP Decoding	.067 ± 0.038	.268 ± 0.041	82.1 ± 8.9 %
Best COCO Image	.220 ± 0.043	.277 ± 0.067	82.0 ± 9.0 %

PixCorr: pixel-wise correlation metric. SSIM: structural similarity index measure. CLIP: two-way identification experiment comparing CLIP from reconstruction to CLIP from ground truth target and a random reconstruction of the same type.

We tracked the score for the best image at each iteration for different brain areas relative to the score for the ground truth target image. The number of search iterations required to achieve the score of the ground truth image decreased monotonically with progression from V1-V4 and into “high-level” visual cortex. This time-to-convergence provides a succinct measure of the level of representational invariance across visual cortex.

Conclusion

Our effort is one of several (Takagi & Nishimoto, 2023; S. Lin, Sprague, & Singh, 2022; Ozcelik & VanRullen, 2023; Gu, Jamison, Kuceyeski, & Sabuncu, 2023; St-Yves & Naselaris, 2019) that use recent developments in Al to achieve impressive reconstruction quality. Although further improvement is warranted, these efforts suggest that reconstructing seen images from human brain activity in controlled settings may soon be a solved problem. For better or worse, this brings promise of using brain decoding as a novel modality for communicating internal states incrementally closer to fulfillment.

Figure 2:

Qualitative comparison of our method against a CLIP decoding-only method, and brute-force search through a library of COCO images. (T.-Y. Lin et al., 2015)

Figure 4:

Correlation of predicted and actual brain activity (“score”) for the top image at each iteration for different ROls (curves). Inset: the number of iterations for each ROI to cross score for ground truth image (dashed lines).