Evidence for compositionality in fMRI visual representations via Brain Algebra

Abstract

Electrophysiological and neuroimaging studies have revealed how the brain encodes various visual categories and concepts. An open question is how combinations of multiple visual concepts are represented in terms of the component brain patterns: are brain responses to individual concepts composed according to algebraic rules? To explore this, we generated “conceptual perturbations" in neural space by averaging fMRI responses to images with a shared concept (e.g., “winter" or “summer"). After thresholding to ensure specificity, we applied these perturbations to the neural pattern associated with a base image, forming new brain patterns that incorporate the added concept. These modified brain patterns were then decoded into images using a pretrained fMRI-to-image decoding model. Qualitative and quantitative inspection of the resulting images provides insight into how the brain might combine visual concepts. For example, adding a “winter" perturbation to the brain pattern of a man on a skateboard yields a new pattern representing a man on a snowboard in a winter scene—even when the perturbation modifies only a small subset of voxels. Our findings reveal that compositional processes in neural representations may lead to predictable perceptual outcomes, as interpreted by our decoding model. This suggests that the brain’s combinatory encoding of concepts may follow a systematic, algebraic-like process—what we term “brain algebra." Although our study is model-driven, it opens avenues for future empirical work into the mechanisms of compositionality in the brain.

Algebraic combinations of fMRI patterns reveal compositional neural representations of visual concepts, showing that brain activity can be perturbed in predictable ways to generate interpretable and meaningful perceptual transformations.

Introduction

The compositionality of latent representations in artificial intelligence (AI) systems has contributed to recent advancements in deep learning. Model-based techniques like word embeddings have demonstrated that semantic relationships between concepts can be captured through vector arithmetic—for example, “king” minus “man” plus “woman” yields a vector close to “queen"¹. Similarly, image and text representations in AI models exhibit compositional properties that allow for the manipulation and combination of visual and semantic concepts^2–7.

In neuroscience, machine learning has spurred significant progress through the development of encoding and decoding models. These models have established bidirectional mappings between visual or linguistic inputs and corresponding brain activity^8–22. Notably, the use of larger, multimodal, and more complex models—which often exhibit some amount of compositionality—has led to significant improvements in predicting brain activity with encoding models. This suggests that better embeddings, enriched with compositional properties, capture more nuanced information that aligns more closely with the brain’s representations^23–25. This raises a fundamental question: does the human brain employ a similar compositional structure in its neural code for vision²⁶? Recent evidence suggests that brain-pattern compositionality may indeed occur in specific linguistic contexts²⁷: information regarding analogy questions can be effectively retrieved through the addition and subtraction of functional Magnetic Resonance Imaging (fMRI) patterns. In their study, participants were presented with sequences of related concepts, such as professions, tools, and places (e.g., “doctor", “stethoscope", “hospital"). The researchers demonstrated that the algebraic combination of fMRI activation patterns could reflect analogical reasoning, akin to vector operations in word embeddings (e.g., “mechanic-doctor+stethoscope=wrench").Moreover, the vector space representations utilized by AI models appear to exhibit key properties essential for supporting cognition, such as high-dimensional representations, compositionality, concept distances, and similarity measures²⁶.

Building on these findings, we investigate whether brain-pattern compositionality holds for visual representations. Specifically, we aim to determine whether the neural activity elicited by viewing a composite image can be approximated by algebraically combining the neural patterns associated with its constituent parts—a concept we refer to as "brain algebra." In this framework, adding the neural pattern associated with a particular concept to the neural pattern of a base image should yield a new neural pattern corresponding to the perception of the base image modified by the added concept. This idea mirrors the compositional operations observed in AI models, where vector arithmetic in latent spaces captures semantic relationships between concepts. By testing this hypothesis in the context of visual perception, we aim to uncover whether the brain employs a similar mechanism for combining visual information.

In doing so, we also test a related hypothesis: if compositional structure is embedded in neural representations, then it should manifest in the ability to perturb only a sparse subset of voxels and still observe meaningful transformations in perceptual content. This hypothesis enables us to examine whether compositionality is supported by localized coding, distributed patterns, or a hybrid of both. While we do not aim to resolve the broader debate on semantic brain organization, our findings provide a novel lens through which this question can be revisited.

While compositionality has been extensively studied in the context of language models and visual generative systems in AI^1,6,7, and recent neuroimaging work has shown compositional effects in linguistic reasoning tasks²⁷, little is known about whether similar compositional mechanisms operate in the brain’s visual representations. Our study aims to bridge this gap by testing whether algebraic operations on neural patterns derived from real fMRI data can yield predictable and interpretable visual transformations. In doing so, we extend the investigation of compositionality from semantic and multimodal embeddings to empirical neural representations of visual cognition.

To address this question, we use the Natural Scenes Dataset (NSD)²⁸, a large-scale fMRI dataset where participants viewed approximately 10,000 natural images while their brain activity was recorded using a 7T fMRI scanner. We define “base” brain patterns as the fMRI responses to individual test images from this dataset. We also define “concept” brain patterns, where we average the fMRI responses to multiple training set images that share a specific concept. The presence of these concepts in training images is identified using semantic embedding models (i.e., CLIP;⁶), ensuring that the selected images are strongly related, from a semantic point of view, to the target concept. By algebraically combining these patterns in brain space, we create a perturbed brain pattern that hypothetically represents the base image with the added concept. This is done by adding the thresholded concept pattern to the base pattern. For example, starting with a base brain pattern corresponding to an image of a man on a skateboard, we might add the concept pattern for “winter" to generate a perturbed pattern that should represent a man on a snowboard during winter.

A critical challenge is evaluating whether this perturbed brain pattern truly corresponds to the brain representation of the base image with the added concept. One approach would be to create corresponding composite images—for instance, using generative AI models to synthesize images that combine the base image with the added concept—and then present them to participants in an fMRI experiment. By recording the brain activity elicited by these composite images, we could compare the observed neural patterns with the predicted ones derived from our “brain algebra” operations (analogous to the approach used in ref. ²⁷). However, this method presents several difficulties. First, the number of possible combinations of base images and concepts leads to a combinatorial explosion in the number of stimuli required. Testing a wide range of concepts and their combinations would necessitate a prohibitive number of experimental trials. fMRI experiments are inherently long and expensive, with limitations on how long participants can be scanned and associated financial costs with data acquisition. These logistical constraints make it impractical to collect sufficient data to robustly evaluate compositionality across diverse concepts. Second, even if such extensive experiments were feasible, interpreting the results would remain challenging. Differences between the expected and observed brain patterns could arise from various sources, including fMRI noise, individual variability in neural responses, or limitations in the quality and realism of the generated stimuli. These confounding factors would challenge any definitive conclusions about compositionality in the brain’s neural code.

Additionally, existing neurostimulation technologies do not currently permit to directly manipulate specific voxel activations in the brain to test compositionality. We cannot selectively stimulate or alter precise patterns of neural activity at the voxel level to create the exact perturbed brain patterns hypothesized by our “brain algebra” model. This limitation means that we cannot empirically test the predicted neural patterns by artificially inducing them in the brain.

Given these challenges, we employ an alternative approach that evaluates the “brain algebra” results using a decoding model. By transforming the perturbed brain patterns into reconstructed images, we can indirectly assess whether the algebraic combination of neural patterns corresponds to meaningful composite perceptions. This method leverages existing fMRI data and advanced decoding algorithms to infer the perceptual content associated with the combined neural patterns, providing a practical lens to explore our research question within the constraints of current technology.

Thus, instead of attempting to collect new fMRI data or manipulating brain activity directly, we employ Brain-Diffuser¹⁰, a well-established decoding model that maps brain activity into the latent space of a generative model. By decoding the perturbed brain patterns, we can reconstruct images that represent the hypothetical perception resulting from our brain algebra operations. The resulting images can be assessed qualitatively—through visual inspection—or quantitatively using automated semantic analysis tools based on AI systems like CLIP⁶. In summary, our study explores whether the compositionality observed in AI systems and linguistic brain representations extends to visual processing in the human brain. We introduce a novel method to assess “brain algebra," combining base and concept brain patterns derived from actual fMRI data, and employ the Brain-Diffuser model to decode these patterns into images. Through this method, we aim to provide evidence for or against the existence of compositional neural codes in visual cognition. In the following sections, we detail our methodology for defining and combining the base and concept brain patterns, describe how we employ the Brain-Diffuser model for decoding, and present our qualitative and quantitative analyses of the reconstructed images to evaluate compositionality. See Fig. 1 for a visual explanation of perturbation definition and Fig. 2 for a visual overview of our approach.

Fig. 1. A visual overview of the process used to define conceptual perturbation vectors based on “winter” and “summer” concepts.

First, the textual representations of the concepts are encoded using the CLIP Text model, generating 512-dimensional embeddings. Simultaneously, images from the NSD training set are processed through the CLIP Vision model to obtain vision embeddings. Cosine similarity is calculated between the vision embeddings and the textual concept embeddings (e.g., “winter” and “summer”), allowing us to select the top-matching images that best represent each concept. The fMRI patterns corresponding to these selected images are then averaged to generate concept-specific perturbation patterns in brain space, such as z_winter for winter and z_summer for summer. These perturbation vectors are later thresholded, and combined with base fMRI patterns in the brain algebra framework to modulate visual representations (see Fig. 2).

Fig. 2. Illustration of the “brain algebra” approach used in our study.

The leftmost image represents the initial visual stimulus presented to the participant, with corresponding fMRI activations shown as heatmaps across different brain regions. Perturbations are introduced by summing the base brain pattern with a concept-specific perturbation vector, such as “summer,” “winter,” “man,” or “woman.” The perturbation vector is computed as a thresholded average of brain patterns evoked by visual perception of images with that content (see Fig. 1). The perturbed brain patterns (center) are subsequently decoded using a pretrained fMRI decoder, producing modified images that reflect the added conceptual information (right). The results demonstrate how small changes in neural patterns can lead to predictable and meaningful changes in visual perception, supporting the hypothesis of compositionality in neural representations.

Results

We explored 12 different semantic concepts encompassing themes such as season (winter, summer), gender (man, woman), lighting (night, day), numerosity (empty, crowded), location (indoor, outdoor) and emotions (happy, sad). Each corresponding perturbation vector was thresholded by a variable amount (retaining between 5% and 100% of the voxels, the rest being set to zero), and scaled by various factors α (from 1 to 4) before being summed with base fMRI patterns corresponding to a random subset composed of 100 test images. We begin this section by discussing the qualitative results, focusing on the exemplary Figs. 3 and 4, which were generated using a scaling factor of α = 2 and a 50% threshold to visually highlight the key outcomes. Additional figures with varying scaling values and thresholds are provided in the supplementary materials for a more comprehensive evaluation.

Fig. 3. Qualitative evaluation of brain algebra perturbations applied to base images (images best viewed digitally).

Starting from the central base images, decoded from a (non-perturbed) base fMRI pattern, perturbations corresponding to various concepts---such as “summer,” “winter,” “day,” “night,” “man,” “woman,” and more---are applied to the brain patterns. The resulting decoded images show how the base visual perception is altered by the addition of each conceptual perturbation, reflecting changes in environmental conditions, the presence or absence of people, and other context-specific details. This demonstrates the ability of brain algebra to generate compositional modifications in visual representations based on abstract conceptual inputs.

Fig. 4.

More examples as in Fig. 3.

In the first set of images (top of Fig. 3), showing horses in a field and an indoor bathroom, compositionality is evident. Concepts like “summer," “winter," and “night” alter the landscape and lighting in the horse scene, while “woman” and “man” introduce an additional person. In the bathroom scene, “crowded” and “empty” adjust the number of objects or people, and “summer” and “winter” change the mood. The bottom row, showing a skater and a social gathering, also demonstrates compositional transformations. “Summer” and “winter” modify the environment, “woman” changes the skater’s gender, and “crowded” and “happy” alter social dynamics and expressions.

Similarly, perturbations in Fig. 4 introduce clear modifications based on the perturbation concepts. For the bus scene, “summer” brings brighter environments with outdoor activities (however, the bus is no longer present), while “night” darkens the scene with illuminated elements. The “indoor scene” concept transforms the bus into a more enclosed space, like a terminal. Perturbations applied to the man holding a sandwich show clear changes—“woman” alters the subject’s gender, “empty” reduces the person’s size within the scene, and “crowded” adds individuals to the scene. In the outdoor table scene and paragliding activity, compositional adjustments are also evident. These results suggest that the model effectively generates coherent and predictable changes in response to targeted brain perturbations. Overall, these results support the hypothesis that compositionality in brain patterns can be decoded into visually meaningful images. The perturbations introduced lead to expected modifications in both human-related and environmental aspects, while generally maintaining the integrity of the original base scenes. This indicates that the brain perturbation patterns successfully capture abstract conceptual information and translate it into visual content, providing strong evidence for compositionality in the brain’s visual representations.

While these qualitative examples generally support our hypothesis of visual concept compositionality, they also include some perturbations for which the desired concept did not obviously appear or was difficult to evaluate (e.g., paragliding+day appears similar to the base image stimulus), and perturbations that replaced the initial content rather than complementing it (such as the disappearing bus in the bus+summer perturbation). To provide a more systematic evaluation of compositionality, we quantitatively measured the presence of both the perturbation concept and the base image concepts in the decoded images from perturbed brain patterns, by leveraging cosine similarity in the CLIP latent space as a measure of semantic content (since this metric is shown to be aligned with human judgments on image similarity^29,30).

This quantitative evaluation of the decoded perturbed images reveals clear trends in the similarity between the images and two targets: the original image and the concept used for perturbation. In the bottom panel of Fig. 5, which assesses the similarity (in the CLIP-vision latent space) between the decoded perturbed images and the original images, we observe that similarity decreases as the scaling value increases. This is expected, since larger scaling values apply a stronger perturbation, causing more deviation from the original image. The similarity between the decoded perturbed images and the original images remains significantly above the baseline for all conditions (calculated by contrasting decoded images against randomly chosen base images), indicating that elements of the original image are still retained even as the perturbation intensifies.

Fig. 5.

Top: Average similarity between decoded perturbed images and the target concept (across CLIP-vision and CLIP-text latent spaces, respectively) across different thresholds (0, 25, 50, 75, 90, 95th percentiles of the voxel distribution) and scaling values (0 to 4, with zero corresponding to the base pattern without perturbation). The similarity increases with higher scaling values, reflecting that larger perturbations align the decoded image more closely with the target concept. The green shaded region represents variability (standard deviations across 4 subjects, 100 images per concept, 12 concepts) in similarity, while the red dashed line represents the baseline similarity between random images and the target concept. Bottom: Average similarity (in the CLIP-vision latent space) between decoded perturbed images and the original images across the same thresholds and scaling values. The similarity decreases as the scaling value increases, indicating that larger perturbations deviate more from the original image. The red dashed line shows the baseline similarity between the original image and random images.These results indicate a trade-off between maintaining original image features and introducing conceptual modifications, depending on the scaling value and threshold.

The top panel of Fig. 5, which compares the decoded perturbed images to the target concept (across CLIP-vision and CLIP-text latent spaces, respectively), shows the opposite trend: similarity increases as the scaling value rises. This suggests that larger scaling values make the images more representative of the added concept. Lower thresholds allow the perturbation to have a broader effect, leading to faster increases in similarity to the target concept, while higher thresholds limit the impact of the perturbation, resulting in slightly more modest increases. In all cases, the similarity between the decoded images and the target concept remains consistently above the baseline (calculated by contrasting random images with the target concept), demonstrating that the perturbation effectively introduces the desired conceptual information into the images. Interestingly, even at high thresholds, where only a small portion of the brain’s activity is perturbed, we still observe notable changes in similarity to the target concept. The fact that meaningful conceptual shifts occur even when the perturbation is restricted to higher thresholds indicates that the brain might encode these abstract concepts in localized areas, and only subtle changes in activity within these regions are required to reflect concept-driven modifications in the decoded images. Overall, these results illustrate a trade-off between maintaining similarity to the original image and introducing conceptual modifications. As scaling increases, the perturbed images deviate more from the original content but become more aligned with the target concept. The thresholding mechanism provides a way to control the extent of the perturbation, with higher thresholds preserving more of the original image and lower thresholds allowing for greater conceptual compositionality.

While our focus is on subject-specific decoding, we note that the compositional effects observed in our perturbation experiments generalize across the four participants included in our study. This is evident both in the consistency of decoding trends and in the qualitative similarity of reconstructed outputs across subjects (see Fig. 5).

To further explore the spatial characteristics of conceptual perturbations, we visualized the top 10% most active voxels (thresholded at the 90th percentile) for each of the 12 semantic concepts in a representative subject (Fig. 6). These maps reveal that different concepts elicit distinct spatial patterns in visual cortex. Scene-related concepts such as indoor and outdoor show broad bilateral activation in occipital and parahippocampal regions, while social categories like man, woman, and crowded engage more lateral and ventral regions, consistent with areas implicated in face and body perception (e.g., FFA, EBA). Emotion-related concepts such as happy and sad exhibit more diffuse patterns, yet still evoke reproducible changes in localized patches. Importantly, conceptually opposing categories (e.g., summer vs. winter, crowded vs. empty) result in spatially distinct perturbation maps, suggesting that these brain-based concept representations are separable and consistent with a compositional structure. These findings are in line with our hypothesis that the brain can perform algebraic operations within concept-specific subspaces of neural representation. What appears to matter is not solely the anatomical localization of activity, but rather the patterned distribution of neural responses across the voxel space. This supports the notion that conceptual information is embedded in a vectorial format, enabling systematic operations akin to those observed in semantic embeddings and artificial models of compositionality.

Fig. 6. Spatial distribution of the top 10% most active voxels (90th percentile) for each concept for Subj01, visualized on cortical surfaces.

Distinct patterns emerge across categories---e.g., scene-related ("indoor", “outdoor"), social ("man", “woman", “crowded"), and emotional ("happy", “sad")---highlighting the diversity and specificity of conceptual representations in brain space.

Discussion

The findings from this study provide a promising indication of compositionality in neural representations. By manipulating brain patterns in a “brain algebra” framework—combining a base neural state with a thresholded and scaled perturbation vector—we observed distinct, meaningful changes in the decoded images. This suggests that visual processing in the brain may follow a compositional structure, much like language, where basic elements can be combined to create more complex representations. The ability to successfully decode these perturbations aligns with broader theories on compositionality in cognition, such as in language, where concepts are combined to produce new meanings (e.g., “queen” = “king” - “man” + “woman”). This parallel between vision and language highlights how the brain may generalize compositional principles across different domains of cognition, supporting flexible and dynamic perceptual and cognitive processes^2,4,27.

The use of natural images as stimuli is integral to our approach, as it enables the study of brain patterns in conditions that closely mimic real-world visual experiences. These images contain a variety of visual and semantic elements that reflect everyday interactions with the environment. By leveraging these stimuli, we can investigate how the brain processes complex compositional patterns that are more representative of natural vision, compared to more controlled or artificial stimuli.

There are, however, important limitations to consider. While our results suggest compositionality in neural representations, our evaluation relies on a decoding model to interpret the perturbed brain patterns. Although the two brain patterns we are combining are derived from actual fMRI data, the interpretation of their combination is model-driven because it depends on the decoder’s ability to accurately reconstruct images from brain activity. This means that our conclusions are contingent upon the performance and limitations of the decoding model, and we are not directly observing brain processes in real-time but interpreting them through the lens of the model. This limits the extent to which we can confirm that similar compositional operations happen naturally in the brain. Additionally, we are constrained by the need for sufficient training data—only concepts with ample representation in the training set allow us to generate reliable perturbation vectors. As a result, our exploration of neural compositionality is bounded by the availability of data, limiting the range of concepts we can examine. Furthermore, some concepts are not orthogonal, and their representations in the training set can lead to biased perturbations in brain patterns. For instance, visual inspection shows that adding the concept of “happiness” occasionally introduces food elements, likely because in datasets like COCO, “happiness” is often associated with, or co-occurs alongside, images of food. Similarly, concepts like numerosity or emotion might also be biased due to their frequent co-occurrence with humans. As a result, applying a “crowded” perturbation may add humans to scenes with animals, even when the intended effect is only to increase the number of animals.

One intriguing aspect of our results is that meaningful changes were observed even when the perturbation involved only a small subset of voxels (e.g., the top 5% of voxels). This suggests that relatively small, localized regions of the brain can significantly influence the representation of specific concepts. This finding contributes to the ongoing debate between distributed and localized cortical representations in visual processing^31–33. On the one hand, proponents of distributed representations, such as Haxby and colleagues, argue that visual information is encoded across widespread patterns of neural activity, with object and category information represented in distributed and overlapping voxel patterns³⁴. On the other hand, researchers like Kanwisher propose that certain visual categories are processed in specialized, localized cortical regions—for example, the fusiform face area (FFA) for face perception³⁵. Our observation that concept perturbations are effective even when modifying only a small portion of voxels aligns with the idea that specific cortical areas play a crucial role in representing certain concepts. These perturbations, when decoded, yield image reconstructions that reflect conceptually altered visual content, suggesting a shift in perceptual representation as inferred by the decoding model.

These findings invite a nuanced interpretation of how compositionality and modularity coexist in neural coding. While compositionality typically refers to operations within a single high-dimensional space, our results suggest that different concepts may be encoded in partially distinct, functionally specialized subspaces within that space. This modular organization does not preclude algebraic manipulation but rather supports it: perturbations restricted to voxels most strongly associated with a concept can still generate coherent semantic transformations when combined with unrelated base patterns. This suggests that the brain’s representational geometry may be composed of overlapping but structured subspaces that enable both modular specialization and compositional generalization.

In summary, our findings provide evidence suggesting compositionality in the brain’s processing of visual stimuli. This compositional structure could be key to understanding how the brain flexibly combines sensory information to form different percepts, furthering our understanding of neural coding, perception, and learning.

Conclusions

In this study, we explored the compositionality of neural representations through the novel framework of “brain algebra," combining base fMRI patterns with conceptual perturbations to decode visual representations. Our findings provide evidence that the brain may employ compositional mechanisms similar to those seen in language and cognition, where smaller elements are combined to form more complex representations. The results demonstrate that neural patterns can be manipulated to create distinct and predictable changes in decoded images, aligning with the target concepts, even when only small portions of brain activity are perturbed.

The findings suggest that neural representations of visual concepts involve both localized and distributed processing. The ability to change perceived images by modifying a small number of voxels indicates that certain brain regions are specialized for processing specific visual information, supporting the idea of localized specialization. However, these localized changes also integrate with broader neural networks, aligning with the view that visual perception involves distributed representations. This dual role of localized and distributed coding contributes to the debate in neuroscience and underscores the complexity of how the brain processes and combines visual information.

Overall, this work contributes to a deeper understanding of neural compositionality in vision and highlights the brain’s capacity to integrate conceptual information. By offering new perspectives on how perturbations in neural space correspond to changes in perception, this research provides a foundation for future studies on neural coding, perceptual constancy, and cognitive flexibility. Expanding this line of inquiry could also reveal insights into how the brain composes and generalizes across other cognitive domains, such as language and broader semantic representations. This suggests that the principles underlying “brain algebra” in vision might extend to other brain functions, hence providing a more comprehensive framework for understanding compositional processes in neural representations.

Methods

In this section, we describe the proposed method and the data we used. The data are publicly available and can be requested at https://naturalscenesdataset.org/. All experiments and models were trained on a server equipped with eight NVIDIA A100 GPU cards (80GB RAM each connected through NVLINK) and 2 TB of System RAM.

Data

The study employs the Natural Scenes Dataset (NSD)²⁸, an extensive fMRI dataset gathered from eight participants who were shown images from the COCO21 dataset. Our analysis focused on four subjects, resulting in a specialized training set containing 8859 images and 24,980 fMRI trials per subject, as well as a shared dataset consisting of 982 images and 2770 trials per subject. To reduce the spatial dimensionality of the fMRI signals (with a resolution of 1.8mm isotropic), we applied a mask using the provided NSDGeneral ROI, targeting multiple visual areas. This deliberate selection of ROIs improved the signal-to-noise ratio and reduced data complexity, enabling the investigation of both low- and high-level visual features. Temporal dimensionality was further minimized by leveraging precomputed betas derived from a general linear model (GLM;^36,37) with an adjusted hemodynamic response function (HRF) and a denoising process as detailed in the NSD publication.

BrainDiffuser

The “Brain-Diffuser” model¹⁰ is a two-stage framework designed to reconstruct natural scenes from fMRI signals. In the first stage, a Very Deep Variational Autoencoder (VDVAE) generates an “initial guess” of the reconstruction, capturing low-level details. This guess is then refined using high-level semantic features from CLIP-Text and CLIP-Vision models, and a latent diffusion model (Versatile Diffusion;³⁸) is used for the final image generation. The model takes fMRI signals as input and produces reconstructed images that reflect both low-level properties and the overall scene layout. Brain-Diffuser, was trained subject-wise with data from Subj01, Subj02, Subj05, Subj07). More information about the decoding model is detailed in the original paper. While this specific pipeline is used in our study, our proposed method is universally applicable and can enhance any single-subject decoding pipeline. It offers a versatile, adaptable tool that can seamlessly integrate with novel, advanced pipelines. By focusing on preprocessing input data, our approach enables the underlying pipeline—regardless of its unique aspects—to effectively work with single-subject fMRI data to generate images, without requiring direct modifications to the pipeline itself.

Main experiment

Here we outline our main experiment, which aims to decode synthetic brain patterns derived from the algebraic sum of real brain patterns, and examine compositionality in the decoded images. The NSD dataset provides paired fMRI data and corresponding images. Let’s define fMRI data as z and images as x, giving us a training set of pairs (z_tr, x_tr) and a test set (z_ts, x_ts). Additionally, we have a decoder d, a function that maps z to x, such that x ≈ d(z).

The essence of our work is as follows: we explore the outcome when decoding $z^{\prime }$, defined as $z^{\prime }=z_{base}+\alpha z_{perturb}$, where α is a scaling factor, z_base is a brain pattern drawn from the test set, and z_perturb is a perturbation pattern computed by averaging training set brain patterns associated with a specific concept. One way to investigate compositionality in the brain is to hypothesize that the resulting image, $x^{\prime }=d\left(z^{\prime }\right)$, represents a combination of the base image x_base and an additional concept.

For instance, if the base pattern z_base corresponds to an image of an indoor scene x_base, and we add a perturbation brain pattern z_perturb related to the concept of a man, then decoding $z^{\prime }$ might produce an image of a man in an indoor scene. Similarly, if the perturbation pattern corresponds to a woman, we might expect the decoded image to depict a woman in the scene, and so on. We tested our framework for a random subset (identical for all subjects) of 100 images drawn from the test set.

Pattern definition and thresholding

A natural question arises: how do we define the perturbation pattern relative to a specific concept? We adopted a straightforward approach by filtering the image-fMRI pairs in the training dataset based on their similarity to the concept, which is defined in natural language and measured using a CLIP-based cosine similarity.

We explored 12 different semantic concepts: ["man", “woman", “indoor", “outdoor", “summer", “winter", “day", “night", “crowded", “empty", “happy", “sad"]. These concepts encompass themes such as season, gender, lighting, numerosity, emotions.

Each concept was represented as a word and encoded using the CLIP Text model⁶, producing a 512-dimensional representation (using version clip-vit-base-patch32). We then used the CLIP Vision encoder to encode all the images in the training set and calculated the cosine similarity between each image and all the concepts. This resulted in a similarity matrix of shape (8859, 12). For each concept, we selected the top 100 pairs with the highest similarity scores to extract the corresponding fMRI and image indices. The perturbation patterns were then defined by averaging the fMRI patterns associated with these top-100 pairs, thereby establishing their representation in brain space.

We applied various threshold values to the perturbation patterns based on their percentile values. Specifically, in our experiments, we evaluated the outcomes by thresholding z_perturb and retaining only the values above the 0th, 25th, 50th, 75th, 90th, or 95th percentiles. This allows us to assess whether the representation of the chosen broad semantic concepts is distributed across the entire visual cortex or if only a small region is responsible for encoding changes, with the composition of values in these small regions being sufficient to produce compositional images when decoded. Importantly, it is worth emphasizing that all compositional operations in our framework are performed directly in brain space—that is, on real fMRI-derived neural patterns—before any decoding takes place. This distinction, coupled with sparsity and non-linearity introduced by thresholding procedure avoids circularity and ensures that any emerging semantic effects are a result of the structure present in the fMRI data itself, rather than learned priors in the decoding model. Moreover, our use of voxel-wise thresholding enables us to probe whether concept representations are localized or distributed, and to what extent localized subspaces are sufficient for compositional decoding. This design choice allows us to explore the neural geometry underlying compositionality without assuming full distribution or strict modularity. Please see Supplementary Table 1 for an assessment of Brain-Diffuser performances on threhsolded patterns and Supplementary Figs. for activation patterns for all concepts and subjects.

Evaluation

The first part of our evaluation is qualitative, focusing on visually assessing decoded images from perturbed brain patterns to examine the compositionality of the original stimulus image and the perturbation concept. However, a quantitative measure is necessary to rigorously evaluate this compositionality. Compositionality can be loosely defined as the co-occurrence of two concepts within an image. In our framework, we adopt a practical working definition of compositionality that aligns with approaches in large-scale neural models such as CLIP or GPT. Specifically, we define compositionality as the ability to algebraically combine neural patterns associated with distinct concepts to produce a new, coherent representation that reflects both components. This does not imply formal semantic compositionality in the linguistic sense (i.e., deriving a compound meaning strictly from parts and syntax), but rather refers to the integration of multiple conceptual features into a unified and plausible perceptual scene. Thus, we need to quantify how closely the decoded image resembles the target perturbation concept while retaining similarity to the original content. If the scaling factor α is too large, the perturbation may replace the original content entirely, leading to misleading results if we only measure the similarity between the decoded images and the perturbation concept.

To address this, we calculated the CLIP cosine similarity between the decoded perturbed images and the original stimuli to ensure that the original content was not entirely replaced. Simultaneously, we measured the CLIP cosine similarity between the decoded perturbed images and the perturbation concept. In the first case, we measured cosine similarity between images, while in the second case, the similarity was measured between images and text. As these two metrics may have different baselines, we also computed a random baseline by measuring the cosine similarity between each base image and 100 randomly selected images from the training set. Similarly, we established a baseline for each concept using 100 random images.

Finally, we averaged the results as a function of the scaling factor α for each threshold across all decoded images and subjects.

Statistics and Reproducibility

All experiments were conducted using data from four participants in the publicly available Natural Scenes Dataset (NSD). For each subject, the training set included 24,980 fMRI trials corresponding to 8859 unique images, and the test set included 2770 fMRI trials for 982 unique images. A fixed random subset of 100 test images was used consistently across subjects in the main decoding experiments. For each semantic concept, perturbation vectors were computed by averaging the fMRI patterns of the top 100 training images most semantically related to the target concept (measured via cosine similarity in CLIP-embedding space). Quantitative analyses were based on computing cosine similarity in the CLIP latent space between the decoded images and both the original base images and the textual representations of target concepts. These measurements were averaged across subjects, concepts, and test images, and variability was reported as standard deviation across the four subjects. No formal statistical hypothesis testing (e.g., p-values or confidence intervals) was conducted, as the objective was to characterize systematic decoding trends rather than population-level inference.

Reproducibility is supported by the use of a public dataset (NSD), standard pretrained models (CLIP and Brain-Diffuser), and a fixed experimental pipeline. All custom code and scripts used for preprocessing, perturbation generation, and decoding will be made publicly available upon publication.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Author contributions

M.F. and R.V.R. conceived the study. M.F. implemented the code, conducted the analyses, and drafted the initial manuscript. T.B. contributed to the interpretation of results and critically revised the manuscript. R.V.R. and N.T. provided scientific guidance, mentorship, and overall supervision throughout the project. All authors reviewed and approved the final version of the manuscript.

Peer review

Peer review information

Communications Biology thanks Olaf Hauk and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Jasmine Pan. A peer review file is available.

Data availability

The data supporting the findings of this study are publicly available. The fMRI data and corresponding image stimuli were obtained from the Natural Scenes Dataset (NSD)²⁸. All preprocessing scripts and derived datasets used in the analyses will be public released upon acceptance. Supplementary Data 1 and Supplementary Data 2 provide numerical sources to reproduce Fig. 5.

Code availability

The custom code used for preprocessing fMRI data, generating brain perturbation patterns, and performing image decoding with the Brain-Diffuser model will be made publicly available upon acceptance of this manuscript at this link: https://github.com/matteoferrante/BrainAlgebra. The Brain-Diffuser model itself is described in detail in its original publication³⁹.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s42003-025-08706-4.