Reconstructing music perception from brain activity using a prior guided diffusion model

Abstract

Reconstructing music directly from brain activity provides insight into the neural representations underlying auditory processing and paves the way for future brain–computer interfaces. We introduce a fully data-driven pipeline that combines cross-subject functional alignment with bayesian decoding in the latent space of a diffusion-based audio generator. Functional alignment projects individual fMRI responses onto a shared representational manifold, increasing the performance of cross-participant accuracy with respect to anatomically normalized baselines. A bayesian search over latent trajectories then selects the most plausible waveform candidate, stabilizing reconstructions against neural noise. Crucially, we bridge CLAP’s multi-modal embeddings to music-domain latents through a dedicated aligner, eliminating the need for hand-crafted captions and preserving the intrinsic structure of musical features. Evaluated on ten diverse genres, the model achieves a cross-subject-averaged identification accuracy of , and produces audio that human listeners recognize above chance in 85.7% of trials. Voxel-wise analyses locate the predictive signal within a bilateral circuit spanning early auditory, inferior-frontal, and premotor cortices, consistent with hierarchical and sensorimotor theories of music perception. The framework establishes a principled bridge between generative audio models and cognitive neuroscience.

Introduction

Music exerts a profound influence on the human brain, involving distinct neural networks that modulate emotions, trigger memory recall, and affect various neurological states¹. Understanding how musical information is represented in neural activity has implications for both basic neuroscience and potential clinical applications. For example, Brain–Computer Music Interfacing (BCMI)² explores how musical features can be decoded or modulated from brain signals, potentially supporting personalized auditory stimulation or communication for individuals with motor impairments. In addition, music-based cognitive tasks could improve cognitive functions such as mental flexibility and creativity³.

This study investigates the intricate relationship between neural activity and music, focusing on the feasibility of reconstructing musical stimuli from fMRI data using generative models. A key challenge is decoding high-frequency musical information (in our case within the 0–8000 Hz range) given the lower temporal resolution of fMRI data, which is further complicated by regional variations in the brain’s Haemodynamic Response Function (HRF). The temporal resolution of fMRI inherently constrains the type of musical information that can be captured. Fast-evolving rhythmic or transient features, which unfold on the order of tens to hundreds of milliseconds, are effectively blurred in the BOLD signal. As a result, our reconstructions should be interpreted as reflecting high-level representations rather than fine-grained rhythmic or dynamic aspects of the music.

Denk et. al⁴ examined generative music decoding on the same fMRI corpus, yet our study introduces several novelties in the decoding approach. First, we bypass the semantic bottleneck introduced by textual conditioning: instead of prompting the generator with captions, we project neural activity directly into CLAP’s aligned audio-text space⁵. Second, rather than relying on subject-specific masks derived from anatomical atlases and proprietary embeddings such as MuLan and MusicLM^6,7, we apply a fully data-driven voxel-selection strategy based on cross-validated encoding accuracy, yielding a reproducible mask that generalizes across participants. Third, we formulate brain-to-music reconstruction as a bayesian maximum-a-posteriori inference problem that combines the encoding-model likelihood with the prior imposed by MusicLDM, guiding the diffusion sampler toward acoustically plausible yet neurally consistent solutions. Finally, in addition to objective similarity metrics, we report a forced-choice listening test in which human listeners identify the correct reconstructions, providing a human-centred benchmark of perceptual fidelity. Figure 1 provides an overview of our approach.

Fig. 1.

The proposed pipeline is composed of three stages. In the top section, participants listened to musical stimuli during the GTZAN-fMRI experiment, with concurrent fMRI recordings capturing their brain activity. The middle section involves extracting latent representations of the stimuli using the CLAP-audio model, followed by voxel-wise encoding to correlate brain responses with music, with a correlation threshold identifying music-responsive regions. In the bottom section, a regression model predicts CLAP-text embeddings from these regions, which are subsequently used to reconstruct new musical outputs via the MusicLDM decoder.

Related works

The neural basis of music processing has been explored extensively in classical neuroscience⁸. Zatorre et al.⁹ emphasize the importance of auditory and premotor cortex in music perception and imagery, while Levitin et al.¹⁰ highlight the contribution of distributed neural systems—including reward and memory networks—in supporting music’s emotional and cognitive dimensions. These studies underscore that music engages a broad cortical and subcortical network.

However, recent advancements in artificial intelligence have enabled more detailed and data-driven analyses of brain responses to musical stimuli¹¹. Building upon prior research, significant progress has been made in mapping fMRI activity to latent representations of diverse stimuli, including images, video, language, and music, through techniques such as linear mappings and subject-specific models^4,12–14. Takagi et al.¹⁵ demonstrate high-resolution visual reconstruction from fMRI using latent diffusion models. Our work follows a similar philosophy, extending this concept from vision to audition by coupling fMRI-derived latents with a diffusion-based audio prior. Giordano et al.¹⁶ show that intermediate acoustic-to-semantic representations best explain responses in the superior temporal gyrus (STG), supporting our choice of CLAP-aligned latent representations as targets rather than relying on low-level acoustics or purely linguistic descriptors. Bellier et al.¹⁷ demonstrate that music reconstruction can be performed using both linear and nonlinear approaches to decode the auditory experience using EEG data.

The advent of pre-trained models has further facilitated the extraction of latent representations capable of driving retrieval tasks or conditioning generative models. Notably, the development of text-to-music models has made it possible to generate high-fidelity music in a conditional framework, linking language-based representations with the generation of coherent musical outputs^7,18,19.

Our work extends the approaches of Ferrante et al.²⁰ and Denk et al.⁴, advancing previous methods by decoding brain activity between subjects within a generative framework that operates independently from empirically derived captions.

Results

Encoding regions

We empirically set the Pearson Correlation threshold at 0.05, following a principled trade-off between signal reliability and decoding performance. Lower thresholds (e.g., r < 0.05) increased voxel count but introduced substantial noise, diluting informative signal and decreasing decoding performance. Conversely, higher thresholds (e.g., r > 0.05) excluded a large number of moderately informative voxels, reducing the representational capacity of the decoder. This threshold identified 3043 voxels involved in music processing. See Supplementary Materials for details and statistical validation.

The spatial distribution of these voxels is depicted in Fig. 2. The identified brain regions are primarily located in the temporal lobes and lateral frontal areas, and are highly plausible as music-responsive regions. The superior temporal gyrus, particularly in both hemispheres, plays a crucial role in auditory processing, with the right temporal lobe being especially involved in melody and tonal perception²¹. Bilateral activation is consistent with the known engagement of both hemispheres in different aspects of music, such as pitch, rhythm, and harmony^9,22. The involvement of frontal regions, such as the inferior frontal gyrus (IFG), is also expected, given its association with complex aspects of music cognition, including rhythm processing, pattern recognition, and musical syntax^23,24. These findings align with established research that links music perception with both auditory processing areas and higher-order cognitive regions.

Fig. 2.

Spatial distribution and predictive power of music-responsive voxels. Cortical voxels whose BOLD time series were predicted from the CLAP feature set (using five-fold cross-validated ridge regression) are shown on the inflated fsaverage template brain. The colorbar represents the mean Pearson correlation coefficient (r) between predicted and observed responses across participants; voxels with r below an empirical threshold were excluded, resulting in a final set of 3043 significant voxels. This threshold excluded significant but weakly predictive voxels. The resulting bilateral network spans: (i) the posterolateral superior temporal gyrus and Heschl’s gyrus (early tonotopic cortex), (ii) anterior and posterior inferior frontal gyrus, and (iii) supplementary motor and premotor cortices. Left and right hemispheres are shown in lateral views.

Decoding performance

Table 1 presents the performance of the proposed method, which utilizes functional alignment. The model achieved an identification accuracy of , outperforming several baseline methods from brain2music⁴.

Table 1.

Comparison of test identification accuracy with literature baselines. Our best result reported a 95% confidence interval of [0.887, 0.939]. A statistical comparison against the best baseline yielded z = 2.51, p = 0.006, assuming Gaussian-distributed accuracies.

Embedding	Test identification accuracy
SoundStream-avg
w2v-BERT-avg
MuLantext
MuLanmusic
MusicLDM-anatomical
MusicLDM-functional alignment

The computed similarity matrix is visualized as a heatmap (Fig. 3), where each cell represents the cosine similarity between a real target embedding (row) and a predicted embedding (column). The matrix illustrates that the predicted latent representations of the stimuli are well-aligned with the real ones (along the diagonal) and, in some cases, also between representations of closely related genres, such as rock, reggae, and blues. Visualizing the distribution of cosine similarity values across different genres, we see that classical and jazz exhibited the highest alignment scores, while genres with more complex rhythmic structures, such as dance and hip-hop, showed slightly lower alignment. This suggests that certain musical features, like harmonic progression and melodic consistency, may be more easily decoded from neural activity than rhythmic or textural components.

Fig. 3.

Heatmap of cosine similarity scores between real (rows) and predicted (columns) GTZAN embeddings. The colour scale ranges from − 0.2 (blue) to 1.0 (red), with red indicating higher similarity. The concentration of red along the diagonal indicates a strong match between predicted embeddings and their corresponding real embeddings.

We additionally performed an ablation varying the number of candidate samples and the inclusion of the CLAP audio-text aligner. As shown in Table 2, performance metrics (identification accuracy, cosine similarity, and SSIM) confirm that larger candidate sets improve reconstruction quality with a diminishing returns trend (after 50 candidates), and that CLAP-based alignment provides a measurable gain.

Table 2.

Ablation study on the generative prior. Metrics are averaged over 5 subjects (test set, mean ± SD). Notably, all the configurations showed significant SSIM and cosine similarity (e.g. for our best configuration) compared to a null model obtained by random permutation of brain-audio mappings.

Config.	ID. Acc.	Cosine Sim.	SSIM
Our best (100 samples)
50 samples
20 samples
1 sample
100 NoAligner

Note that the identification accuracy reported in this ablation refers to post-generation evaluation—that is, correlations are computed between CLAP embeddings extracted from the generated audio samples and their corresponding ground-truth embeddings. This differs from the identification metric in Table 1, which is computed directly on brain-predicted embeddings. The post-generation metric thus reflects end-to-end reconstruction fidelity, incorporating both latent prediction and diffusion decoding.

Human results

In the human experiment, participants’ average accuracy was 85.7% with respect to the chance level (50%), indicating that the human ear can reliably detect the correspondence between the original musical stimulus and its brain-generated counterpart. Participants were asked to choose the track they perceived as most similar to the original stimulus before proceeding to the next trial.

Individual accuracies ranged from 0.767 to 0.967 (mean ± SD ). A one-sample t-test against the 0.50 chance level confirmed that human performance was significantly above chance (one-sided ; 95% CI ). Table 3 reports the identification accuracies by musical genre, showing consistently high performance across categories, with particularly strong agreement for classical and metal excerpts.

Table 3.

Human accuracy by musical genre (mean ± SD, 95% CI).

Genre	Acc. (mean ± SD)	95% CI
Blues
Classical
Country
Disco
Hip-hop
Jazz
Metal
Pop
Reggae
Rock

Samples of the generated tracks compared to the original stimuli are available at: https://musicdecod.my.canva.site/decoding-musical-perception.

Discussion

This study presents a principled framework for reconstructing musical stimuli from fMRI data, suggesting potential relevance for future brain–computer interfaces and neuroadaptive applications. By leveraging MusicLDM and CLAP, we obtain natural-sounding, genre-consistent reconstructions that emphasize a “brain signature” of music tracks over temporally precise musical events (e.g., micro-timing, rhythmic transients). This outcome is consistent with the temporal resolution and macroscopic nature of fMRI.

The identification of music-responsive regions aligns with previous studies, with clusters in superior temporal gyrus (STG), inferior frontal gyrus (IFG), and lateral prefrontal cortex. The bilateral distribution across temporal areas indicates that information predictive of our CLAP embeddings is present in both hemispheres. This spatial pattern is consistent with established roles of temporal cortex in auditory analysis (including pitch/timbre) and frontal cortex in higher-order musical structure^9,21,22. We interpret our results as compatible with models in which STG supports spectral–acoustic analyses and IFG contributes to processing of structural regularities and musical syntax, as suggested by prior literature^24,25. While the novelty of our work does not lie in identifying new brain regions, the convergence with canonical auditory circuits supports the validity of our decoding framework. Testing hypotheses about temporal integration windows or motor entrainment would require higher-temporal-resolution modalities (e.g., MEG/iEEG).

Among the competing audio embeddings, the two MuLan variants outrank SoundStream and w2v-BERT, yet are themselves outperformed by the proposed CLAP  MusicLDM latent. The pattern sketches a spectrum: models that bind musical timbre to semantic captions (MuLan) fare better than purely signal-driven representations (SoundStream), but the hybrid embedding that is explicitly optimized for generation captured still more of the variance linked to brain responses. This suggests that cortical populations respond not just to low-level acoustics, nor only to high-level semantic tags, but to an intermediate abstraction that generative diffusion models are well suited to represent. Relatedly, cosine-similarity values reveal that classical and jazz stimuli map most faithfully from cortex to embedding, whereas dance and hip-hop exhibit a lower degree of alignment. A plausible account is temporal: fMRI’s 1.5 s TR smears rapid onsets and syncopated beats, depriving the regression model of rhythmically diagnostic features while leaving slower-evolving harmonic progressions largely intact. Interestingly, reggae, rock and blues cluster together off the diagonal of the similarity matrix, implying that shared back-beat structure and guitar timbres co-activate overlapping voxel subsets; the decoder finds these embeddings exchangeable in latent space, echoing behavioural confusions reported in genre-labelling studies.

Also, participants recognized the correct reconstructions 85.7% of the time, a figure that lags the embedding-level accuracy yet still doubles chance performance. The gap between machine and human metrics likely reflects musicality cues that reside outside the evaluated embedding dimensions—e.g. micro-timing and dynamic range—that listeners notice even when the cosine test is passed. Put differently, the brain-to-embedding mapping identifies “what song is it most similar to?” more often than “does it sound musically convincing?”, highlighting a potential avenue for adversarial fine-tuning of the generative prior ^26,27.

Broader applications

The results emphasize the potential of functional alignment techniques in reducing inter-subject variability, providing a more generalized model of music perception across individuals. This aspect is particularly relevant for applications in brain–computer interfaces (BCI) and neuroprosthetics, where reliable and subject-independent decoding is crucial. Put differently, gains over the best baseline are likely robust, signaling that between-person variability is less intractable than often feared. Functional alignment here acts like a low-dimensional morph of each individual cortical sheet into a shared computational space, suppressing idiosyncratic folding while preserving representational geometry. The ability to reconstruct auditory experiences from neural signals could support non-verbal communication in individuals with severe motor or cognitive impairments, such as patients with locked-in syndrome or advanced neurodegenerative disease.

Limitations

With a TR of 1.5 s, rapid temporal dynamics such as rhythmic transients, syncopation, and micro-timing cannot be faithfully encoded in the neural signal. Accordingly, the present reconstructions should be regarded as capturing static, timbral-genre features that characterize the overall acoustic profile of a piece, rather than temporally precise musical events.

Beyond temporal blurring, the subsampled and noisy nature of fMRI further enforces a macroscopic, averaged representation of neural activity. Under these macroscopic conditions, linear mappings often provide the most reliable predictive performance for fMRI data, as spatial/temporal averaging and measurement noise can mask microscale non-linearities; thus we adopt ridge regression as a principled, well-validated baseline for brain decoding mapping ²⁸. This choice does not preclude non-linear decoders; rather, it reflects the data regime studied here. Future work with higher-resolution modalities or larger datasets could profitably revisit non-linear mappings to capture finer neural effects.

Finally, while CLAP and MusicLDM enable robust feature extraction and generation, they also carry biases from their training corpora, which may influence reconstructed content. We therefore interpret results as reconstructions within these model priors, not as unbiased samples of the full musical space.

Conclusions

Our framework offers several methodological advances. First, cross-subject functional alignment markedly elevates decoding accuracy, enabling music reconstruction that generalizes across listeners rather than being confined to single-participant models. Second, embedding the neural signals in the latent space of a diffusion generator yields natural-sounding reconstructions that preserve perceptual consistency. Third, framing the decoding step as a Bayesian inference problem represents a novel contribution: probabilistic sampling over candidate latents selects the most plausible stimulus, thereby stabilizing the output against noise in the neural data.

Instead of relying on written captions, we map CLAP’s text embeddings directly onto music-domain embeddings via a dedicated aligner. This strictly audio-driven, free-text bridge captures musical features without the biases introduced by external verbal descriptions. Consistent with prior work, we find that genres with pronounced structural signatures—classical and jazz—are encoded more robustly, whereas closely related styles such as rock and blues remain harder to disambiguate, underscoring the need for finer-grained modelling.

Several extensions suggest themselves. Comparative studies spanning varying levels of musical expertise and culturally diverse cohorts could clarify how experience and enculturation shape neural encoding. Incorporating modalities with superior temporal resolution (e.g., intracranial EEG) should improve decoding of rhythmic and transient features that fMRI blurs.

The capacity to reconstruct music directly from brain activity may contribute to several applications: thought-driven composition for individuals with motor impairments, objective indices for music-based therapies in neurodegenerative disease, and diagnostic assays for auditory-processing disorders, among others.

In sum, this study lays groundwork for next-generation neural music decoding at the intersection of generative modelling and cognitive neuroscience, opening new avenues for both basic research and translational applications in technology and healthcare.

Methods

This section details the dataset and methods employed in this study. The principal innovation of this study resides in the decoding pipeline, which integrates feature alignment, generative modelling, and bayesian inference into a multi-stage procedure (see Decoding Model section for details).

First, we train a simple linear projection that maps CLAP’s audio embeddings into its text-embedding subspace⁵. This step establishes a cross-modal bridge: although our conditioning variables originate from audio, they are transformed into the same latent domain that the generative model natively expects for text prompts.

Next, ridge regression model transforms fMRI patterns into CLAP-aligned latent coordinates, establishing a direct interface between cortical signals and the representational space used for music synthesis. In this way, the diffusion process can be conditioned directly by neural activity, without recourse to natural-language captions. MusicLDM, a diffusion model originally developed for text-to-music generation ²⁹, is thus repurposed to render audio waveforms from the brain-predicted latents.

Finally, we cast the decoding task within a probabilistic inference problem: our goal is to infer the most likely musical stimulus given observed brain activity , by maximizing the posterior . Here, models the encoding likelihood, and represents a generative prior over realistic music samples defined by MusicLDM. Inspired by recent work on brain-to-image decoding^15,30, we adopt a two-stage approach: we sample a candidate set of music stimuli from the generative prior, and select the one that best matches the observed activity (maximum-a-posteriori estimation). This method implements a form of rejection sampling that balances perceptual realism (via ) and neural consistency (via ).

Model performance is assessed using identification accuracy⁴ (see “Evaluation” section), defined as the proportion of test samples for which the brain-predicted embedding was closest (in CLAP space) to the true embedding of the corresponding stimulus among all test candidates. This measure, standard in brain decoding studies, reflects the model’s capacity to target the correct stimulus representation from neural activity.

Dataset

We used the GTZAN fMRI public dataset³¹, which consists of fMRI data from five subjects (sub-001 to sub-005) exposed to music stimuli drawn from ten distinct genres: blues, classical, country, disco, hip-hop, jazz, metal, pop, reggae, and rock. Each genre was represented by 54 tracks (i.e. stimuli) sampled at 22.050 kHz. Subjects underwent 18 fMRI runs, of which 12 were used for training and 6 for testing. Each run consisted of 40 music clips, each 15 s in duration. The stimuli were RMS normalized and included a 2-s fade-in and fade-out. During testing, each stimulus was presented four times, and the brain activity (i.e. fMRI signal) was averaged across identical stimuli to enhance the signal-to-noise ratio.

Scanning was performed using a 3.0T MRI scanner with a repetition time (TR) of 1500 ms, yielding 400 volumes per run. Preprocessing included motion correction, co-registration to Montreal Neurological Institute (MNI) space using T1-weighted anatomical images, detrending, and run-level standardization. Brain activity was time-shifted by 3 TRs (4.5 s) to account for the delayed hemodynamic response, and neural representations were averaged over 10 volumes (15 s). This procedure provides a high-SNR estimate of the sustained BOLD response, which is most relevant for track-level decoding^32,33. The final dataset comprised 540 fMRI-stimulus pairs per subject (480 for training and 60 for testing).

Encoding model

Music processing in the brain involves complex, nonlinear mechanisms. To capture this, we used the CLAP model⁵, a multimodal neural network employing contrastive learning for audio and text alignment. CLAP extracts audio features using the SWINTransformer³⁴ and log-Mel spectrograms, as well as text features using RoBERTa³⁵, projecting both into a shared latent space. Similarity is commonly used to measure the correspondence between elements of this shared latent space. In order to identify brain regions most responsive to music, we trained a voxel-wise encoding model that predicts fMRI responses z from CLAP’s audio embeddings h (extracted from each stimulus x). Specifically, Ridge regression with cross-validation was applied independently to each voxel. Model training incorporated a hyperparameter search for the regularization parameter (ranging on a logarithmic scale from to ) and we empirically determined a correlation threshold (in a discrete range of values [0.01, 0.02, 0.05, 0.10, 0.15, 0.20]) to define music-responsive brain regions by choosing a value according to maximizing decoding results and minimizing noise. Pearson correlation coefficients were used to create a voxel-wise correlation map between real and predicted fMRI activity, identifying regions most responsive to music.

Formally, for each subject i, we estimate whole-brain encoding weights to predict brain activity z using the latent representations of audio h as inputs. Specifically, we train a model for each subject. To optimize the weights , we perform nested cross-validation on the training set, minimizing the loss function . In each fold, we predict the held-out data (20% of the training set not used to train that specific model). When predicting on the entire training set, we compute the voxel-wise correlation between the predicted and real brain activity, , selecting only voxels that exceed the correlation threshold for further analysis.

Functional alignment

In order to mitigate individual variability in brain structure and function, we employed cross-subject data aggregation techniques following Ferrante et al.²⁰. Anatomical alignment is widely used in neuroimaging as it facilitates the direct comparison of localized brain activity across subjects. However, relying solely on the brain’s physical structure for alignment and decoding lacks the precision needed for fine-grained tasks due to inherent subject-specific anatomical variability, which may not exactly mirror functional differences.

To account for individual variability in brain responses and to enable cross-subject generalization, we implemented a functional alignment strategy based on Ridge regression, inspired by prior work on linear alignment techniques³⁶. The underlying assumption is that, although fMRI activity patterns vary across individuals, they all encode information about the same stimulus content. Therefore, one subject’s activity can be expressed as a linear transformation of another subject’s activity when exposed to the same stimuli. Concretely, given a source subject and a target subject (sub-001), we sought a linear mapping W such that , where S and T denote the matrices of voxel-wise brain responses for the same stimuli from the source and target subjects, respectively. The mapping W is a matrix in which each column defines the weights for reconstructing one voxel in the target subject as a linear combination of all source voxels. In order to compute W, we employed Ridge regression with five-fold cross-validation to select the optimal regularization parameter . The model was fit independently for each subject, using the training set to estimate W, and then applied to both training and test data from the source subject. Aligned activity was further standardized and rescaled to match the distribution of the target subject (sub-001), ensuring consistency across runs and subjects. The choice of sub-001 as the alignment target followed practical and methodological considerations. This subject had high-quality recordings across all experimental runs and showed robust and representative functional responses.

More elaborate techniques, such as Hyperalignment³⁷, estimate high-dimensional shared spaces or subject-specific transformations; these additional parameters increase model complexity and can hinder interpretability. Hyperalignment performs a statistical shape analysis that iteratively rotates each subject’s representational space so as to minimize pairwise dissimilarities with respect to a common template. In a recent comparison of alignment strategies for cross-subject music decoding²⁰, the regression-based method achieved the highest accuracy while retaining the advantages of interpretability and generalization.

Decoding model

Our decoding framework aims to generate music from brain activity by combining feature alignment, neural regression, diffusion modeling, and probabilistic guidance for reconstruction. This process involves different components, described below.

We first established an alignment between audio and textual embeddings in CLAP’s joint latent space. Specifically, we used embeddings from the MusicCaps dataset⁷ and trained a linear projection from audio to text representations by minimizing the mean squared error (MSE) between them. This transformation yields a set of 512-dimensional aligned embeddings as conditioning vectors for stimuli generation. Given that CLAP is pretrained to produce semantically consistent audio-text representations, a simple linear layer is enough to refine this alignment without introducing overfitting or distorting the structure of the latent space.

Then we trained a regression model to map brain activity to the CLAP-aligned latent space. We used Ridge regression with five-fold cross-validation to predict embeddings from fMRI data in the GTZAN dataset. This regression step effectively creates a bridge between brain signals and musical representations, enabling us to infer CLAP embeddings directly from neural activity. These embeddings are then used to condition the generative model.

For the generation stage, we used the Music Latent Diffusion Model (MusicLDM)²⁹, a state-of-the-art generative model that synthesizes audio from text prompts. Importantly, MusicLDM internally uses CLAP text embeddings as conditioning inputs for its latent diffusion process (U-Net based) and HiFi-GAN vocoder²⁷ for waveform reconstruction. In our framework, we replace these text-derived embeddings with brain-predicted CLAP embeddings, keeping the generative process unchanged. This ensures full compatibility between the decoding model and the conditioning space used by MusicLDM. The complete pipeline is illustrated in Fig. 4.

Fig. 4.

Overview of the music generation framework from fMRI. (1) GTZAN music stimuli transformed by the CLAP-audio model into latent representations, used for inference. (2) MusicCaps stimuli processed by CLAP-audio for music embeddings and CLAP-text encoder for captions. (3) A linear layer maps these embeddings into a 512-dimensional latent space. (4) Music-responsive brain regions are used to estimate new GTZAN embeddings via regression. (5) The predicted embeddings are passed to the MusicLDM decoder for music generation.

Finally we framed the problem within a bayesian approach to decode music from brain activity, providing a structured way to infer the most likely musical stimulus given observed brain activity . Bayesian inference allows us to incorporate prior knowledge and systematically update our beliefs about potential solutions, leading to more robust decoding results. Instead of directly modeling , which is complex due to the stochastic nature of generative models, we factorized the problem using Bayes’ theorem:

where represents how likely the observed brain activity is given a musical stimulus, and is a prior over musical stimuli that constrains the possible solutions to realistic ones. This formulation allows us to infer the musical stimulus by evaluating different candidate songs and selecting the one that best matches the observed neural activity. Specifically, we select from 100 trials the one that maximizes the correlation between the real brain activity and the predicted brain response obtained by passing the candidate song through the encoding model . A direct sampling from would be computationally prohibitive due to the large space of possible musical pieces. To make this approach feasible, we introduce a guidance mechanism that restricts sampling to a structured latent space, reducing the complexity of the problem. Specifically, we define a mechanism based on CLAP latent space alignment:

where is the generative model (MusicLDM) conditioned on representation from CLAP linear aligner, and represents the mapping between neural activity and the CLAP-aligned latent space via Ridge regression. This method provides a computationally tractable way to sample plausible musical outputs while maintaining strong correspondence with neural data.

Evaluation

We evaluated model performance using the identification accuracy metric defined in the Brain2Music framework⁴, which quantifies the correspondence between predicted and target music embeddings using Pearson correlation. A correct identification occurs when the correlation between a predicted and true embedding is higher than with any other target embedding. We computed a correlation matrix , where represents the Pearson correlation between the -th predicted embedding and the -th target embedding. The identification accuracy was computed as:

where is the indicator function. The overall accuracy was averaged across all predictions. This metric ensures the robustness of the model in discriminating between embeddings, which is crucial for applications requiring high precision.

We also evaluated the cosine similarity matrix between the real and brain-predicted CLAP embeddings using the following formula:

where and are the normalized real and predicted embeddings, respectively. Each score represents the degree of alignment between these embeddings, which is critical for evaluating the model’s performance in producing embeddings consistent with the target data. Finally, the structural similarity index (SSIM) was computed between the Mel-spectrograms of the generated and ground-truth audio signals. SSIM quantifies the structural fidelity of the reconstruction in the time–frequency domain, assessing how well the generated sample preserves spectral and dynamic patterns of the original stimulus.

As a qualitative evaluation, we developed a human metric to assess the perceived similarity between reconstructed and original music stimuli. 10 participants listened to pairs of stimuli from the test set and were asked to identify which of the two stimuli was the correct decoded version. Subsequently, we calculated the percentage of correct identifications and averaged across subjects. We used Streamlit, a Python framework, to develop a simple web interface for participants. Streamlit is an open-source library for turning data scripts into web applications without requiring the implementation of a custom front end. The user experience consisted of listening to three distinct audio stimuli: the original stimulus, the brain-decoded version of the target stimulus, and a randomly selected brain-decoded track from the test set. In each trial, the randomly selected song was chosen without regard to genre, which permitted participants to encounter pairs from closely related genres (e.g. ‘hip-hop’ vs. ‘pop’ or even ‘hip-hop’ vs. ‘hip-hop’) as well as contrasting genres (e.g. ‘hip-hop’ vs. ‘jazz’). See Supplemetary Materials for human details.

Author contributions

M.C.: Conceptualization, data curation, formal analysis, investigation, methodology, software, visualization, writing—original draft. M.F.: Conceptualization, formal analysis, investigation, methodology, visualization. N.T.: Funding acquisition, project administration, resources, supervision, writing—review and editing.

Data availability

The dataset is publicly accessible at: https://openneuro.org/datasets/ds003720/versions/1.0.1. The present study makes use of the publicly available GTZAN-fMRI dataset by³⁸, which was collected under informed consent and institutional ethical approval. The authors did not acquire new data. The code implementation is available at this repository: https://github.com/neoayanami/fmri-music-gen. This manuscript represents the authors’ original work, has not been previously published, and is not under consideration elsewhere. All authors have made substantial contributions to the work, take full responsibility for its content, and have approved the final version. The manuscript properly credits prior work and the contributions of all co-authors. The human evaluation study involved only listening to short musical excerpts and providing simple perceptual judgments. No clinical procedures or sensitive personal data were collected, and the task involved no risk or discomfort. All participants were informed about the nature of the study and provided their verbal consent prior to participation.

Declarations

Competing interests

The authors declare no competing interests.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-025-26095-w.