Music evokes shared neural representations of imagined narratives across sensory modalities

Abstract

The human capacity to construct and share subjective experiences from diverse sensory inputs is central to human cognition, and is vividly exemplified through music listening. Direct, focused music perception has been extensively studied while free, imaginative listening, typical of real-world engagement with music, has not. Here, we identify shared neural representations for imaginings evoked by matched excerpts of music and speech across listeners. Twenty participants freely imagine stories while listening to ten music excerpts in an fMRI scanner, then recount their imaginings. Stories for each excerpt are highly similar between listeners. Stimulus-specific spatial patterns are observed in auditory regions for music perception, speech perception, and music report. Triple pattern similarity across all conditions reveals unique spatial patterns in default mode, language, and association regions. These findings demonstrate that music elicits detectable, modality-general brain representations specific to particular excerpts, pointing to a shared neural basis for meaning-making across diverse sensory inputs.

When people listen to music, they often imagine remarkably similar stories. The authors show these shared imaginings selectively rely on stimulus-specific patterns in a set of modality-general brain regions.

Introduction

The cognitive neuroscience of music has revealed much about the processes that underlie the direct, focused perception of musical structure, such as pitch and time relations¹, auditory-motor interactions², and prediction error^3,4. Yet these accounts neglect what happens when the mind drifts during music listening. This neglect likely stems from a presumption that musical daydreams are arbitrary, unrelated to the sound, and too idiosyncratic to study. Recent behavioral studies, however, refute these notions, revealing shared, detailed mental content. This phenomenon offers a unique and tractable window into the brain’s mechanisms for generating and sharing subjective realities from seemingly ambiguous sensory information. Here, using fMRI and verbal reports, we present evidence for shared neural representations in the default mode network (DMN) and association cortices for imaginings evoked by music and speech across listeners. We argue that semantic musical content and the brain systems underlying it should be considered part and parcel of music perception, as much as earlier sensory perceptual processes.

While listening to music, people often spontaneously imagine fictional scenes, recall autobiographical memories, or experience other sensory imaginings^5,6. Rather than constitute mere mind wandering, these imaginings are often tightly cued by the sound. People tend to imagine stories with similar content in response to specific musical excerpts, even when those excerpts are unfamiliar⁷. These imagined stories unfold dynamically across the course of the excerpts, with the onset of events in the imagined story tied to particular moments in the music⁸. Thus, a widespread form of musical experience involves the generation of mental representations whose content and temporal dynamics are shared among individuals.

The human brain’s ability to transform raw sensory inputs into rich, meaningful thought content and shared narratives is a cornerstone of our cognitive and social lives. This meaning-making process by which sensory inputs are transformed into thought content is highly complex; it would be hard to decompose it using behavioral research methods alone. Human neuroimaging, however, can elucidate the sub-processes involved in complex sensory or cognitive functions, where there can be important distinctions between structural and semantic processing⁹. Work in domains outside of music suggests this happens hierarchically, such that sensory stimulus features are encoded in early sensory regions, and subsequent processing shifts towards association cortices and into the default mode network (DMN) for temporal integration^10,11 or abstraction^12,13. Unlike in movies or speech perception, perceiving narrative in music relies on inferring meaning from features that are semantically ambiguous. To characterize the process of meaning-making in music, it helps to ground it in a less elusive domain, such as language, where semantic imagery can be described explicitly with words. Thus, participants in the current study listened to unfamiliar musical excerpts known from previous behavioral research to elicit specific imagined stories and audio recordings of a speaker describing these stories in words. After each listening period (music or speech), participants verbally described the story they imagined.

Previously, regions of the DMN have been shown to encode stable, scene-level spatial activation patterns independent of input modality¹⁴. Consistent and shared neural patterns were observed when participants watched a movie, recalled what they had seen in each scene¹⁵, and listened to other viewers’ verbal descriptions of each scene’s plot¹⁶, demonstrating a modality-independent representation of the movie’s narrative. Furthermore, the shared neural patterns were also produced by a more abstract, anthropomorphized geometric-shaped movie¹⁷, indicating that a more conceptual form can also evoke shared narratives. Here, we extend this to musically evoked imaginings, demonstrating that these patterns arise in music, a domain where narrative constructions have been well documented but are poorly understood¹⁸. These findings provide insights into the modality-general neural architecture supporting shared imagination and meaning-making from diverse inputs.

Results

High Consensus in Participant Reports for Musically-Evoked and Speech-Evoked Imaginings

The experiment was completed on two separate days. It was designed to elicit imaginings during music listening, reactivate them in a report condition, and to elicit similar imagining content using matched verbal stimuli, without any music (Fig. 1A). Twenty participants (9 females) participated in two fMRI sessions each. Participants listened to ten instrumental musical excerpts that previous research had shown were broadly unfamiliar. Participants were instructed to listen to each musical excerpt as if it was telling a story. Following each excerpt, participants verbally recounted the story they imagined. On a separate scan day, participants listened to ten spoken stories. Each story was a verbal rendition of the matching consensus narrative for one of the musical excerpts played in the previous session. Consensus narratives were selected through separate behavioral work, in which participants described the stories they imagined to the musical excerpts. For each excerpt, the most representative story was selected and used as the consensus narrative. The aim was to attempt to prompt similar imagined content through overt spoken exposition in one experimental session, and through indirect musical cueing in the other. While listening to the spoken stories, participants were asked to imagine the story they heard. Following each spoken prompt, participants were asked to verbally recount the story they imagined. In total, this constitutes four experimental conditions: music perception, music report, speech perception, and speech report. Each participant completed all four conditions, the order of presentation of the stimuli was randomized, and fMRI session order was counterbalanced.

Fig. 1. Eliciting music-evoked imagining and measuring neural pattern similarities.

Top Imaginings are evoked using ten unfamiliar musical excerpts that have been shown to cue particular stories in listeners. During two counterbalanced fMRI scans, participants listen to either the excerpts (music-perception) or to explicit verbal renditions of the story, based on stories reported in a previous study (speech-perception). Participants are first instructed to imagine stories to the stimulus, and are then asked to verbally recount their imagining (music-report). Bottom Neural pattern similarity was used to find the shared, stimulus-driven activation patterns that underlie shared imaginings. Spatial activation patterns are first calculated for each participant, in each condition (music-perception, music-report, speech-perception), for each excerpt. Pattern similarity was calculated between individual participants and the rest of the group for each excerpt, and was compared to repeated permutations of shuffled stimuli.

Participant imaginings to each musical excerpt were generally similar. To assess shared imagined content, participant-reports for each excerpt were compared to the consensus narrative for that excerpt. This comparison was a binary classification (match/no-match) according to the classification criteria in Table S2, and only considered a match if both raters agreed. In total, 200 reported imaginings were produced in both the music report and speech report conditions.

For the music report condition, on average, 71% of participants’ reports matched the consensus for each excerpt (Fig. 2A). When testing for potential counterbalancing effects, we observed slightly lower performance in the music-first group compared to the speech-first group (70% vs. 77.5%, respectively) after the exclusion of one outlier with low overall performance. Further analysis of stimulus-level performance revealed that two specific excerpts yielded low consensus scores across participants. In line with participant feedback, indicating that the two excerpts’ beginnings were soft sounding and were obscured by the acoustic noise of the scanner, these two excerpts were excluded from all subsequent behavioral and neuroimaging analyses to ensure data quality (stimulus details in table S1). As expected, similarity ratings were higher for stories cued overtly through speech compared to those prompted indirectly by music. In the speech condition, 94% of participant reports matched the consensus stories.

Fig. 2. Musical imaginings show high consensus narratives match and high stimulus-specificity.

A Consensus match human raters rated each participant report as either matching or non-matching its corresponding excerpt’s consensus narrative. Each point represents one stimulus (n = 10 musical excerpts; n = 10 speech stimuli), with the matching percentage calculated across n = 20 participants. The two lighter-shaded points in the music condition indicate excerpts subsequently excluded from all analyses due to soft sound levels in the scanner (see Results). On average, for each musical excerpt, 71% of participant reports matched the excerpt’s consensus narrative (94% for speech condition). B Stimulus specificity - semantic similarity Semantic similarity was calculated between each participant-generated report and its corresponding consensus narrative, across n = 8 stimuli (after excluding the two soft excerpts) and n = 20 participants. To estimate stimulus-specificity (i.e. are all reports just similar, or are they only similar to their matched consensus narratives), consensus labels were shuffled and similarity was tested between participant reports and shuffled consensus narratives. In both the music and speech conditions, the darker box shows similarity to matched consensus narratives and the lighter box shows similarity to shuffled consensus narratives. Box plots show the median (center line), interquartile range (box bounds), and whiskers extending to 1.5 × IQR. Significance was assessed by two-sided permutation test (1000 iterations of shuffled labels); similarity to matched consensus narratives was significantly greater than to shuffled narratives in both conditions (music report: mean similarity = 0.65, 95% CI [0.64, 0.66], vs. 0.61 shuffled, Z = 6.04, p < 0.001; speech report: 0.75, 95% CI [0.74, 0.77], vs. 0.62, Z = 7.75, p < 0.001). C Example reports match / non-match for a given excerpt. Consensus narrative (top, grey) and three individual participant reports: music-report non-match, music-report consensus match and speech report.

Stimulus-Specificity in Participant Reports for Musically-Evoked and Speech-Evoked Imaginings

Participant reports showed high stimulus-specificity as reflected by significant semantic similarity between participant reports and their respective consensus narratives, similarity that is not present when participant reports are compared to random consensus narratives from other excerpts. Semantic similarity was estimated by extracting the sentence embedding for each report in a sentence transformer e5-mistral-7b-instruct language model¹⁹. Thus, each report was expressed as a 4096 dimensional vector. Semantic similarity between pairs of vectors was measured as cosine similarity. For each report, cosine similarity was calculated between its own vector and the vector representing its respective consensus narrative.

Reports were found to be specific to their respective musical excerpts (Fig. 2B). Semantic similarity was compared between each report and its matched consensus narrative and compared to a distribution of 1000 permutations of shuffled excerpt labels. Reports in the music condition were significantly stimulus-specific (mean similarity to matched consensus = 0.65, 95% CI [0.64, 0.66], vs. 0.61 for shuffled labels, Z = 6.04, p < 0.001). Speech-condition reports were, unsurprisingly, also significantly stimulus-specific (0.75, 95% CI [0.74, 0.77], vs. 0.62, Z = 7.75, p < 0.001).

Stimulus-Specificity for Musically-Evoked and Speech-Evoked Imaginings in Participant Emotion Ratings

Emotion ratings for the music and speech excerpts (comprising scale response assessments of each excerpt’s anger, calmness, happiness, sadness and energy) showed significant group-level stimulus specificity, compared to shuffled emotion ratings. No statistically significant stimulus specificity emerged for narrative engagement ratings (comprising scale response assessments of each imagining’s vividness, immediacy, ease, and clarity). Previous work has shown that narrative engagement and narrative content are independent dimensions²⁰. For each musical excerpt, cosine similarity was calculated between its own emotional and engagement vectors and the vectors for its corresponding speech excerpt. Emotion and engagement similarities were compared to respective distributions of 1000 permutations of shuffled excerpt labels. Matched emotion ratings were stimulus-specific (observed similarity = 0.92 vs. 0.85 for shuffled labels, Z = 10.72, 95% CI [0.91, 0.94], p < 0.001). The corresponding test for narrative-engagement ratings was not statistically significant (observed similarity = 0.98, Z = − 0.22, p = 0.55).

Stimulus-Specific, Modality-Specific Representations Found in Auditory Cortex

Behavioral data show shared imaginings specific to each stimulus, under each condition. The main analysis goal for the fMRI data was to find the shared, stimulus-specific activation patterns that underlie shared imaginings, whether elicited by music, recalled, or elicited by speech perception. This was first performed for each condition separately, calculating pattern similarity between individual participants and the rest of the group for each stimulus (Fig. 1B), compared to repeated permutations of shuffled stimuli. Then, a similar process was performed between the different experimental conditions to detect stimulus-specific, modality-general effects.

Activation patterns were calculated for each excerpt by averaging brain activation across timepoints for each participant in each excerpt under each condition. The analysis was done on brain parcels using the 400 parcel Schaefer atlas²¹. For each participant, their spatial pattern within each brain parcel while listening to each excerpt was compared to the average group pattern for that excerpt (excluding that particular participant) using Pearson correlation. Pattern similarity was averaged across excerpts, then across participants. To ensure stimulus specificity, pattern similarity was evaluated against a null distribution constructed by shuffling stimulus labels, and measuring the resulting similarity. This process was repeated 1000 times. Significance was evaluated against this distribution and corrected for multiple comparisons using Benjamini-Hochberg correction²² at q = 0.05 for the number of brain parcels. To ensure that condition counterbalancing did not drive the results, we tested the differences between the two counterbalanced groups (music-first and speech-first) against 1000 randomly split groups and found no statistically significant results.

Pattern similarity analysis of single conditions revealed significant brain regions where stimulus-specific representations were encoded for all three conditions tested: music perception, speech perception and music report (Fig. 3). All three conditions revealed significant patterns around the auditory cortex, superior temporal gyrus (STG) and the posterior middle temporal gyrus (pMTG). The music perception and music report conditions also showed significant patterns in the posterior cingulate cortex (PCC) and precuneus. Effect sizes (mean pattern-similarity r), 95% confidence intervals and FDR-corrected p-values for every significant parcel are provided in Tables S3–S5.

Fig. 3. Pattern similarity between participants for single conditions.

Significant pattern similarity for individual conditions was found in superior temporal gyrus (including auditory cortex), precuneus and around the temporal parietal junction. Pattern similarity was calculated between individual participants (n = 20) and the rest of the group for each excerpt (n = 8). Significance was assessed by two-sided permutation tests against null distributions of 1000 shuffled-stimulus permutations; observed similarity was z-scored against the null and p-values derived from the Gaussian survival function. P-values were corrected across all 400 parcels using the Benjamini–Hochberg procedure at q < 0.05. Effect sizes (mean pattern-similarity r), 95% confidence intervals and exact FDR-corrected p-values for every significant parcel are provided in Tables S3–S5.

Stimulus-Specific, Modality-General Representations Found in Default Mode Network and Association Cortices

To identify brain activation patterns that are stimulus-specific and shared between all experimental conditions, triple pattern similarity was calculated¹⁶. This procedure looks for voxels that fluctuate together across conditions. This analysis was performed between participants: for each participant, their spatial pattern in one condition was compared against the group’s patterns (excluding that participant) in the two remaining conditions. Similar to the single condition analysis, it is compared to a null distribution of shuffled labels. To ensure that condition counterbalancing did not drive the results, we tested the differences between the two counterbalanced groups (music-first and speech-first) against 1000 randomly split groups and found no statistically significant results.

This analysis revealed a set of significant brain regions that selectively encoded stimulus-driven imaginings. Significant regions included anterior- and posterior middle temporal gyri (aMTG, pMTG), posterior cingulate cortex (PCC), association visual regions (inferior temporal gyrus), and multiple regions around the temporal parietal junction (TPJ), including the supramarginal and angular gyri. Effect sizes, 95% confidence intervals and FDR-corrected p-values for all significant parcels are provided in Table S6. This brain map includes the posterior components of the default mode network, as well as regions commonly associated with semantic processing. This map is also similar to the brain maps previously observed for movie-driven narrative representations^15,16. The map is less similar to the maps derived for single conditions in our study, which were mostly characterized by auditory cortex regions (Fig. 4, Table S6). Stimulus-driven imaginings are a complex process and we believe this reflects the fact that within condition pattern similarity contains perceptual stimulus features perceived or even recalled, while between condition similarity can be attributed to the imagined content.

Fig. 4. Triple pattern similarity between participants was used to identify brain activation patterns that are stimulus-specific and shared between all experimental conditions.

This analysis revealed a set of significant brain regions including parts of the default mode and language networks and association cortices that differentially encoded stimulus-driven imaginings. Significant regions included anterior- and posterior middle temporal gyri (aMTG, pMTG), posterior cingulate cortex (PCC), association visual regions, and multiple regions around the temporal parietal junction (TPJ), including supramarginal and angular gyri. This procedure looks for voxels that fluctuate together across conditions. It is compared to a null distribution of shuffled labels. Results are significant at q < 0.05 corrected for the false discovery rate. Effect sizes (mean pattern-similarity r), 95% confidence intervals and FDR-corrected p-values for all significant parcels are provided in Table S6.

Comparison to Meta-Analytic Maps Confirms DMN Effect for Domain General Representations

The previous sections showed that stimulus-specific activation patterns were found in unimodal regions for single modalities. Conversely, stimulus-specific activation was found across modalities in multimodal regions. Here, we wished to formally test this assertion. This was done by comparing pattern similarity results with meta-analytic maps generated by neurosynth²³. Neurosynth-derived association maps were obtained for four relevant terms, representing well-defined brain maps: Auditory, Sensorimotor, Language and Default. These z-score maps were averaged across cortical parcels as in the main analysis. The pattern similarity maps used were the z-score maps produced by permutation testing for stimulus-specific effects. Correlations were calculated between each neurosynth-derived map and the pattern similarity maps. Significance for each combination of Neurosynth and pattern similarity map was determined and corrected for multiple comparisons within each pattern similarity map.

This analysis (Fig. 5) reveals that stimulus-specific effects for the music perception condition are significantly similar to the auditory (r = 0.27,q < 0.001) and language (r = 0.12, q = 0.028) maps. The music report condition is significantly similar to the sensorimotor network (r = 0.16, q = 0.005), and the speech perception condition is significantly similar to the auditory network (r = 0.27, q < 0.001). The across-condition pattern similarity map is the only one significantly similar to the DMN meta-analytic map (r = 0.16, q = 0.007).

Fig. 5. Network alignment of stimulus-specific brain representations across conditions.

Pattern similarity results were compared with meta-analytic maps generated by neurosynth for the following terms: Auditory, Sensorimotor, Language and Default. Correlations were calculated between each neurosynth derived map and the pattern similarity maps. Stimulus-specific effects for the music perception condition are significantly similar to the auditory (r = 0.27, q < 0.001) and language (r = 0.12, q = 0.028) maps. The music report condition is significantly similar to the sensorimotor network (r = 0.16, q = 0.005), and the speech perception condition is significantly similar to the auditory network (r = 0.27,q < 0.001). The across-condition pattern similarity map is the only one significantly similar to the DMN meta-analytic map (r = 0.16, q = 0.007).

Discussion

This study showed stimulus-specific encoding within conditions in auditory regions, and across-condition encoding in DMN and association cortices. Within individual conditions (music perception, speech perception, or music report) encoding is not likely attributable to low-level musical features: first, because this similarity was found for the story reporting condition as well as the music and speech listening tasks, and second, as analysis was performed using temporally aggregated spatial patterns. Across conditions (triple pattern similarity), however, stimulus-specific activity was found in the default mode network and association cortices. Once musical experience is brought to a common semantic ground, by recording verbal reports of imagined stories and including a matched speech perception condition, excerpt-specific representations extend beyond auditory regions and into higher-level regions also known to house modality-general representations for narrative essence. This work demonstrates that music elicits detectable, modality-general representations, a kind of higher-order “meaning”, integrating both narrative imagery and its associated emotional profile, specific to particular excerpts. Music has rich semantic dimensions that have been largely ignored by neuroscience, with a few exceptions^24,25. By integrating behavioral manipulation, which preserves the same semantic content across three distinct conditions, with brain imaging technology, we could uncover the neural substrate that supports shared semantic processing among people inferring stories from multiple modalities.

The behavioral and brain results presented here suggest two main conclusions: first, that musical imaginings represent a common aspect of musical experience, reflecting stable semantic content that can be studied, and second, that these multimodal representations consistently involve higher order sensory brain regions and the default mode network. These two insights establish that this phenomenon is a tractable object of study. They further lay the groundwork for future research that takes musical semantics seriously, mapping the relationship between musical imaginings and acoustic features within defined populations, and identifying the brain mechanisms that support this content.

Several aspects of music perception have been studied in detail, both behaviorally and neuroscientifically. These include the perception of pitch, melody, harmony, rhythm, as well as joint performance and improvisation, emotional response and learning^1,4,26. At the same time, semantic content or meaning have received less attention. Indeed, a common criticism of work in the psychology and neuroscience of music has been that it neglected this dimension^27–30. Koelsch et al.²⁴ represents a notable exception, identifying ERP signatures of semantic processing in response to linkages between specific musical passages and individual words, but did not extend to more complex modes of meaning construction that extend beyond short-term priming. Similarly, scrambled and intact musical pieces were compared, showing that higher-order brain areas responded selectively to the intact versions³¹. This is consistent with findings of musical event sensitivity in both auditory and DMN regions³². Another study identifying DMN activation during music listening conceptualized it as a signature of stimulus-independent mind wandering³³; here we suggest instead that such activation is specifically triggered by the music and an essential component of the meaning-making process inherent in many kinds of music listening. Some researchers have even proposed that this kind of process is key to music’s origins; Trevor and Frühholz³⁴ postulate that music is “an evolved tool for socio-affective fiction,” lending immersive power to imagined stories.

Our research shows that musical imaginings are common and shared and that they can be identified in fMRI signal, but leaves open questions about the precise mechanisms of operation. We see evidence that within condition stimulus-specific similarity is encoded in auditory regions, while across condition similarity is encoded in the DMN and association cortices. However, this apparent distinction is likely accentuated by statistical thresholding. DMN involvement in narrative construction may also be present within individual conditions, but single condition maps are statistically dominated by stimulus-driven sensory activity. Because the single-condition and across-condition analyses were corrected at the same whole-brain threshold, the greater extent of the across-condition map does not reflect a more lenient criterion, but rather its selective sensitivity to the modality-general signal that is otherwise masked by sensory variance. The absence of DMN regions in thresholded single condition analyses therefore does not imply an absence of meaning-related processing. Rather, robust sensory effects can obscure more subtle, integrative signals in higher-order association regions. Consistent with this interpretation, DMN activation is evident in uncorrected statistical maps for individual conditions (see Figure S1) and only survives conservative whole-brain correction when modality-specific variance is minimized in the across-condition analysis.

By isolating what is common across music and speech, we likely tap into a higher-level representation at the intersection of sound and meaning. Since these representations were tested summatively, across the entire timecourse of each stimulus, it is unlikely that they relate to the low-level temporal unfolding of particular acoustic characteristics. Instead, they likely represent a more abstract semantic “gist.” We posit that there is a dictionary of sorts, where musical motifs are linked to concepts, such as scenes, images, actions, objects and so on. As shown by the emotional specificity shared between music and speech excerpts, emotion likely plays an important part in these representations, consistent with the findings in a recent paper about musically evoked nostalgia³⁵. Because our data do not speak to whether narrative engagement is stimulus-specific, we can only raise the possibility that engagement gates imagination rather than determining its content, consistent with a recent study³⁶. This dictionary could have entries like gentle arpeggios that sound dreamlike, connected to a collection of words like slumber and fantasy that connote a dreamlike state, or majestic-sounding rolling drums and brass instruments coupled with all of the visual aspects of regalia.

Importantly, emotion could possibly have a causal role in this dictionary, as supported by three studies, one showing that emotional responses to music preceded visual imagery³⁷, another showing that music and color associations were mediated by emotion³⁸ and yet another showing that participants could consistently associate movie scores with scene-level cinematic features, also mediated by emotion³⁹. In the context of the current study’s results, a mechanism mediated by emotion could explain the process by which narratives emerge. It could not, however, causally account for the DMN responses found across conditions, as in the speech condition participants only heard stories, meaning emotions have to follow semantic meaning and not the other way around.

A key remaining question regarding these crossmodal representations is the degree of interdependence between these modalities. In line with models of distributed semantic representations (discussed in more detail below), it is possible that these entries exist as high-level concepts that, once triggered, become functionally independent of their associated modalities. Much like reading the word “bark” can activate the concept of a dog without necessarily conjuring a vivid visual scene, a musical motif might tap into a dreamlike state without mandatorily triggering a full visual image (even though some degree of cross-modal cortical activation may accompany such conceptual retrieval even in the absence of conscious simulation^40–42). When interpreting these results we must remain cautious: an absence of evidence for cross-modal activation in the signal is not necessarily evidence of its absence in the experience.

We do not know the size of this proposed dictionary, which could number dozens, hundreds or thousands of entries. Nor do we know the duration of musical motifs that constitute its terms: how long does a piece need to be to elicit a clear image? Furthermore, we are interested in the way this phenomenon plays out over longer duration music listening. Anecdotally, we have heard from people who construct long and elaborate stories in their minds over whole concerts. Our studies use a small set of one-minute excerpts, only scratching the surface of this phenomenon. Similarly, our sample size allowed us to establish the existence of these shared neural representations using nonparametric statistics. However, larger-scale studies will be necessary to account for individual variability. Expanding the participant pool would allow for a more nuanced investigation into how factors like musical training⁴³, personality, or imaginative vividness⁴⁴ moderate the strength and topography of these semantic representations.

This paper demonstrates modality independence for musical imaginings. Now, the specific multimodal interactions can be further elucidated. The reports elicited in the study include imagery with auditory, visual, motor and emotional aspects (as well as complex actions, goals, etc.). These include elements such as frolicking forest creatures, sword fighting knights in armor, falling asleep amid falling snow, among others. This is consistent with identifying significant brain representations in association cortices. The knowledge that auditory information can be encoded in visual areas and vice versa is not new, as is shown, for example in the ample sensory substitution literature^45,46. However, it lends support to the broader claim that higher cognition is multimodal in essence⁴⁷.

Multimodal representation mechanisms have been thoroughly discussed in the context of semantic knowledge, where two theories have been proposed. The first is modality-dependent representations accessed by a shared semantic hub in the anterior temporal lobe (known as “hub and spokes”⁴⁸). The other option is that sensory information flows into “convergence zones” that are amodal⁴⁹, more consistent with the approach described above⁴⁴. A recent study showed support for the latter approach. Using silent videos and spoken narratives, it shows both modality-specific encoding and converging semantic representations in brain areas similar to the regions observed here⁵⁰. In the context of translation, language-agnostic concept representation was demonstrated, independent of other language-specific concepts⁵¹. This cross-linguistic convergence is mirrored in the default mode network, which has been shown to represent narrative gist across typologically diverse languages independently of lower-level perceptual features⁵². Furthermore, recent evidence suggests that human brains and large language models converge on a shared conceptual space, with multilingual embeddings providing a predictive framework for neural representations regardless of the input language⁵³. Since multimodal-input-capable deep learning models are available, it would be interesting to see whether abstract representations can be detected in them. Musical imaginings could be useful in understanding the propagation of information along these pathways with regard to both experience and the brain. To answer the question of the minimum duration required to elicit visual or motor imagery, high temporal frequency brain recordings (such as EEG, MEG or iEEG) could test how long after the beginning of an excerpt visual/motor association cortex activities are detectable.

In conclusion, this study identifies higher-order areas that subserve the generation of imagined narratives while listening to music. It extends what was previously known about the brain representations of narratives communicated by movies, showing that similar brain regions encode the construction of narratives even when induced by musical notes that seem not to carry clear semantic meanings. Given the establishment of this phenomenon in the current paper, future work should seek to identify much more fine-grained mappings of the associational links between music and the imagination. In the temporal domain, possible extensions include: brain information propagation, behavioral temporal characterization for short excerpts (showing the minimal building blocks for semantic musical content), as well as longer excerpts (exploring the upper bound of the complexities of shared imaginings). In the modality domain, future studies could compare brain representations of musical experience to the perception of images or videos showing actions and movement (this is an advantage of using brain data, where experience doesn’t have to be mediated by overt semantic reporting). Future work should also aim to identify the role of development and enculturation in establishing the scope and nature of musical semantic content.

Methods

Participants

48 Princeton students with normal hearing were recruited through campus email lists. They were compensated $10/hour to fill out a pre-screening survey to determine eligibility for this experiment. The pre-screening survey consisted of the Goldsmiths Musical Sophistication Index (GMSI⁵⁴), Four-Factor Imagination Scale (FFIS⁵⁵), and the Absorption in Music Scale (AIMS⁵⁶). Participants were invited to participate in the full study if their scores were no lower than one standard deviation under the population means on the FFIS subscales for frequency and complexity and AIMS. These eligibility criteria were set to ensure that individuals participating in the fMRI experiment would be likely to complete the task of imagining stories while listening to music. 20 individuals (F = 9, mean age 20.4, SD = 2) met eligibility criteria and were compensated $20/hour for participation in the fMRI experiment. The final participant group had a mean GMSI score of 82.43 (SD = 18.24), a mean AIMS score of 136.95 (SD = 15.93), and FFIS scores of 4.39 (SD = 0.85) for frequency and 5.02 (SD = 0.63) for complexity. All participants were fluent in English. All participants gave written informed consent.

Sex and gender were determined by self-report. Of the 20 participants who completed the study, 9 identified as female and 11 as male. Sex and gender were not used as variables in the study design or analyses, as the study was not powered to detect sex or gender differences and the primary research questions concerned shared neural representations across the full participant group. No statistical method was used to predetermine sample size; the sample size of 20 was chosen based on practical constraints and is consistent with prior fMRI studies of narrative processing using similar designs. Two musical excerpts were excluded from behavioral and neuroimaging analyses due to poor audio quality in the scanner environment; no participants were excluded from the final analyses, though one participant with low overall behavioral performance was noted in sensitivity analyses. Stimulus presentation order was randomized across participants. The experiments were not blinded; investigators were not blinded to allocation during experiments or outcome assessment.

Experimental Procedure

The experiment was approved by the Princeton Institutional Review Board, and informed consent was obtained from all participants. The experiment involved two fMRI sessions, spaced exactly one week apart. Participants were randomly assigned to the order of conditions they received, which was counterbalanced. In one session, participants listened to the music condition, and in the other session, they listened to the speech condition. Both conditions followed the same experimental procedure, differing only in the stimuli presented. During each session, participants were asked to listen to each 60-second stimulus while imagining a story (the exact instructions were: “You will now hear a musical excerpt. Try to listen to this excerpt as if it is telling a story. Imagine the story as the musical excerpt progresses. Strive to keep your attention on the story implied by the music.”). After listening, they had 60 seconds to verbally describe what they had imagined, followed by 60 seconds to answer survey questions about their report and emotional engagement with the stimulus. Survey questions for each stimulus included narrative engagement questions (vividness, immediacy, ease, and clarity), familiarity, emotion questions (anger, calmness, happiness, sadness and energy) and an enjoyment question. All survey questions were rated on 5-point Likert scales. This procedure was repeated for each of the ten stimuli in each condition, presented in a randomized order. To maintain consistency with behavioral studies of imagining stories in response to music⁷, participants were not explicitly instructed either to close their eyes or to keep them open. Audio was played using an Avotec SS-3300 Silent Scan in-ear system. Participant report audio was recorded using an OptoAcoustics FOMRI III microphone system. At the end of each scanning day, after the main task, participants performed a short recall task, for use in a different project. Altogether, the experimental session lasted about an hour.

Stimuli

In the music condition, participants listened to ten 60-second instrumental music excerpts, which had been validated for high narrative engagement by previous behavioral studies, summarized in Table S1^7,20,57. Consensus narratives were generated for each musical stimulus by selecting the most frequently reported narratives in these prior studies, resulting in a paired consensus narrative for each of the ten musical stimuli (Table S2). In the speech condition, the stimuli consisted of the ten consensus narratives spoken aloud by one of the authors, a female undergraduate student. Each was recorded to match the 60-second length of the music stimuli.

fMRI data acquisition

Imaging data were acquired using a 3 T Siemens Magnetom Prisma full-body scanner and a 64-channel head coil. High-resolution structural images were obtained using a T1-weighted MPRAGE (TFL) sequence with the following parameters: TR = 2530 ms, TE = 3.3 ms, TI = 1100 ms, flip angle = 7°, FOV = 256 mm, and 1.0 mm isotropic voxels. The anatomical scan was acquired in the sagittal (A/P) orientation using a GRAPPA acceleration factor of 2. Functional images were collected using a T2-weighted Echo Planar Imaging (EPI). The acquisition parameters were: TR = 1000 ms, TE = 30 ms, flip angle = 59°, FOV = 195 mm, and 2.5 mm isotropic voxels. We acquired 56 slices in an interleaved fashion with a multiband acceleration factor of 4. The phase encoding direction was set to Anterior-to-Posterior (A/P).

fMRI data preprocessing

MRI data were preprocessed using fMRIPrep 20.2.0⁵⁸. fMRIPrep included slice time correction, head motion correction and normalization to MNI space. A detailed, automatically generated log can be found in Supplementary Methods. Further preprocessing was performed using nilearn⁵⁹: preprocessed data were standardized and detrended, a high-pass filter of 0.01 Hz was applied to remove low frequency drifts, and motion (6 regressors) and aCompCor (6 regressors) confounds provided by fMRIPrep were removed. The data were then smoothed with a 2.5 mm FWHM Gaussian kernel. After all participants’ data were preprocessed and cleaned, a custom brain mask across all participants was created from the brain masks provided by fMRIPrep for each participant; only voxels that had signal in every run for every participant were included in the custom mask.

Behavioral data analysis

Consensus match

Recorded participant reports were processed by research assistants. Reports were first manually transcribed, then compared to their respective consensus narratives. Each report was binarily labeled as either consensus matching or non-matching. Consensus matches were evaluated using the criteria in table S2. Data for 10 participants (200 trials) were coded twice to check for interrater agreement. Raters agreed on 86% of trials.

Pattern similarity analysis

Data preparation

Pattern similarity analysis was performed using the Schaefer-400 parcellation atlas²¹. The atlas was masked with the custom group brain mask described above. Spatial patterns for each experimental run were calculated by extracting the signal in each run (each stimulus, each condition) for each participant, adding a 5 second delay to account for the hemodynamic response function and averaging the signal in time. All trials are represented by equal length 1D vectors in each brain parcel.

Statistical analysis

Pattern similarity was calculated following the procedure performed in a previous paper investigating movie narratives¹⁵. Within-condition, between-participants pattern similarity was calculated as the Pearson correlation between each participant and the group average for all other participants in each excerpt. Participant similarity was averaged across all excerpts, then averaged across participants. Statistical significance was calculated by constructing null distributions. The procedure described above was repeated 1000 times, shuffling the group excerpt labels (e.g. comparing sub-01, excerpt-01 spatial pattern to mean(sub-02:sub-n, excerpt-05)). Probabilities were calculated under this null distribution. P-values in each analysis were corrected for 400 parcels using Benjamini-Hochberg correction.

To measure pattern similarity for each stimulus across the three conditions, we followed a previously published procedure¹⁶. This analysis was performed between participants. For each participant and each stimulus, three spatial patterns within a parcel were entered into the computation: the individual participant’s pattern in music perception, and the group-average patterns, computed across all other participants, excluding the participant being tested, in the two remaining conditions. Similar patterns mean that voxels co-modulate their activity in similar ways. The computation is calculated in a few steps. First, within each parcel, the brain response is z-scored. Then, if patterns are similar, voxels within the pattern should have the same sign across all conditions. Thus, if a voxel had the same sign for a given stimulus across all three conditions, it was assigned the product of the absolute values across conditions. If it didn’t have the same sign across the three conditions, it was assigned a value of zero. This was calculated for each stimulus, then averaged. To test the significance of this quantity, the same procedure was repeated 1000 times using shuffled excerpt labels. P-values were corrected across all 400 parcels using the Benjamini–Hochberg procedure at q = 0.05, identical to the correction applied in the single-condition analyses.

Comparison to meta-analytic maps

Meta-analytic maps used here are association maps from neurosynth²³. These maps represent term-specific activations, in other words, “the association test maps tell you whether activation in a region occurs more consistently for studies that mention the current term than for studies that don’t.” (neurosynth documentation)

Maps were selected based on relevant terms, selecting a term with a high number of studies, in a mostly non-overlapping manner. The terms used were: “default” (907 studies), “auditory” (1252 studies), “language” (1101 studies), and “sensorimotor” (684 studies). In cases of overlap, the term using more studies was chosen, for example “Music” and “Auditory” yield largely similar meta-analytic maps, but the Auditory map makes use of many more studies (1252 vs. 163, respectively). A similar case was default vs mentalizing (907 vs. 151, respectively).

Neurosynth maps in MNI space were resliced to the same resolution as the functional data, masked with the same brainmask, and signal was averaged across parcels (to match the resolution of the pattern similarity results, calculated per parcel, not voxel).

Reported effect sizes are Pearson correlations between each Neurosynth association map and the stimulus-specificity z-score map; significance was assessed by permutation and FDR-corrected across the four maps within each condition.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Author contributions

I.J. contributed to conceptualization, methodology, formal analysis, visualization, writing the original draft, and review and editing. J.A.W. contributed to conceptualization and formal analysis. K.E.C. contributed to investigation and formal analysis. C.T. contributed to conceptualization and investigation. G.B.S. contributed to conceptualization, investigation and formal analysis. U.H. contributed to conceptualization, methodology, formal analysis, initial manuscript preparation, funding acquisition and supervision. E.H.M. contributed to conceptualization, methodology, formal analysis, writing the original draft, review and editing, funding acquisition and supervision.

Peer review

Peer review information

Nature Communications thanks Sarah Hennessy, Jonas Kaplan, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Funding

This work was supported by NSF BCS 2240330 grant awarded to author EHM, Princeton Catalysis Initiative Funding awarded to authors UH and EHM, and Princeton Dean for Research Innovation Funding awarded to authors UH and EHM.

Data availability

The behavioral and derived neural data generated in this study (per-parcel BOLD pattern vectors, the group brain mask, in-scan ratings, consensus-match annotations, and e5-mistral-7b-instruct semantic embeddings of the verbal reports) have been deposited in a Zenodo database (10.5281/zenodo.20084749). These data constitute the minimum dataset required to reproduce the analyses reported in this article: all reported statistics, figures, and permutation tests are computed from the per-parcel pattern vectors and semantic embeddings provided in the deposit. Participants’ verbatim transcripts are not included in the deposit as they could contain identifiable information; the semantic embeddings derived from them are the representations entered into all reported analyses. Transcripts and raw fMRI data are available on request to the corresponding author (E.H.M., margulis@princeton.edu), who will respond within 60 days. The Schaefer-400 parcellation atlas used in this study is publicly available at https://github.com/ThomasYeoLab/CBIG/tree/master/stable_projects/brain_parcellation/Schaefer2018_LocalGlobal. Neurosynth meta-analytic maps are publicly available at https://neurosynth.org.

Code availability

Analysis code used to process the data and generate the results reported in this study is available at https://github.com/itamarJa/musical-imaginings-fmri-2026 and archived on Zenodo (10.5281/zenodo.21476369)⁶⁰. Preprocessing was performed using fMRIPrep 20.2.0 (https://fmriprep.org) and nilearn 0.6.2 (https://nilearn.github.io). Downstream pattern analyses were implemented in Python 3.13.7 using nilearn 0.12.1, NumPy 2.3.3, SciPy 1.16.3, pandas 2.3.3, scikit-learn 1.8.0, statsmodels 0.14.6, nibabel 5.3.3, and h5py 3.16.0. Semantic similarity analyses used the e5-mistral-7b-instruct sentence embedding model.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at 10.1038/s41467-026-76452-0.