Dissonant music engages early visual processing

Significance

The present study reveals responses in the primary visual cortex modulated by musical information: tonal dissonance recruits early visual processing via feedback interactions from the auditory ventral pathway to the primary visual cortex. We demonstrate that the auditory “what” ventral stream plays a role in assigning meaning to nonverbal sound cues, such as dissonant music conveying negative emotions, providing an interpretative framework that serves to process the audio-visual experience. The findings substantiate the critical role of audio-visual integration in shaping higher-order functions such as social cognition.

Abstract

The neuroscientific examination of music processing in audio-visual contexts offers a valuable framework to assess how auditory information influences the emotional encoding of visual information. Using fMRI during naturalistic film viewing, we investigated the neural mechanisms underlying the effect of music on valence inferences during mental state attribution. Thirty-eight participants watched the same short-film accompanied by systematically controlled consonant or dissonant music. Subjects were instructed to think about the main character’s intentions. The results revealed that increasing levels of dissonance led to more negatively valenced inferences, displaying the profound emotional impact of musical dissonance. Crucially, at the neuroscientific level and despite music being the sole manipulation, dissonance evoked the response of the primary visual cortex (V1). Functional/effective connectivity analysis showed a stronger coupling between the auditory ventral stream (AVS) and V1 in response to tonal dissonance and demonstrated the modulation of early visual processing via top-down feedback inputs from the AVS to V1. These V1 signal changes indicate the influence of high-level contextual representations associated with tonal dissonance on early visual cortices, serving to facilitate the emotional interpretation of visual information. Our results highlight the significance of employing systematically controlled music, which can isolate emotional valence from the arousal dimension, to elucidate the brain’s sound-to-meaning interface and its distributive crossmodal effects on early visual encoding during naturalistic film viewing.

We share with our ancestors the ability to identify other’s intentions based on the available sensory information. The concept of neocortical operations being multisensory is widely accepted and emphasizes the importance of integration in sensory encoding (1). This distributive aspect of sensory processing not only grants evolutionary advantages, such as prompt reactions to threats, but also serves to reduce the uncertainty derived from individual sensory estimates and to endure brain damage or sensory loss (2).

Previous research on audio-visual integration in humans has primarily relied on task-based paradigms employing simple stimuli (e.g., checkerboards or flashes as visual stimuli, and tones or noises as auditory information) (for a review see ref. 3), limiting the generalizability of findings to complex, real-life situations (4–6). Functional MRI (fMRI) experiments during naturalistic film viewing provide an alternative approach, enabling the study of audio-visual integration in ecologically valid contexts and facilitating investigations into high-level processes, such as social cognition (7).

During film watching, viewers effortlessly recognize and follow the mental states of onscreen characters, a process associated with theory-of-mind function (8–10). This seemingly undemanding task, however, can be complex; especially when visual cues are ambiguous. In these cases, viewers strongly rely on contextual cues, such as the accompanying music, to aid their comprehension (11–13). When integrated into a visual context such as film, music’s influence goes beyond mirroring a meaning portrayed by the visual images (14, 15). Music can shape our experience of the film’s narrative, affecting perceptual judgments, emotion, and memory of the events (11–13, 16–22).

The influence of affect in cognitive processes has been widely recognized in past literature. Theorists, such as Isen (23) and Bower (24), proposed that affective states could operate like category names to prime related material in the cognitive system: any event stored in memory is also related to its associated mood, and events with similar affect are bound together into higher-order categories. A specific affective state of an individual or context may enhance the accessibility of this material within the cognitive system, as a consequence of spreading activation in the semantic network (25). This phenomenon, known as a “congruency effect” or “priming effect,” has been consistently observed [e.g., effect of induced mood on the recall of previously learned personality trait words (26), effect of depressed mood on the accessibility of autobiographical memories (27), effect of induced mood on word associations (28)]. Priming effects have long been acknowledged also within the music cognition field (29–31). In the film music research domain, mood congruency effects have been explained by means of the schema theory (11), which holds a strong connection with current views of the brain as a prediction machine, continuously generating inferences that approximate the relevant future (32, 33). Schemas (34) are memory associations built by extracting statistical regularities from our environment, which are thought to be assembled in memory structures that function as the building block for predictions (33, 35–37). Most researchers agree that one of the primary functions of schemas is to provide an interpretative framework, that serves to lessen the amount of attentional effort required for perception and comprehension. According to this perspective, film soundtracks exert a schematic influence on the cognitive processing of visual events (11, 38): the perceptual content is seen as resulting from knowledge-driven predictions that can be modulated by musical information. In other words, film music encourages viewers to predict scenarios that are consistent with its implied mood (11).

Despite the significant impact of music on film, there is limited neuroscientific research investigating the joint processing of music and visual information. Previous studies have shown that the combined presentation of music and visual information increases activation in brain structures associated with emotion (39, 40). However, the lack of control over specific musical structure variables in these studies has made it challenging to determine how such features modulate brain function.

In the present study, we employed fMRI during naturalistic film viewing to examine the neural mechanisms underlying the emotional responses to musical dissonance (41–44). Although one of the most common harmonic manipulations in film music (e.g., Spielberg’s “Jaws” or Kubrick’s “The Shining”), systematically controlled transformations of tonal dissonance have not yet been neuroscientifically studied in regard to affective processing biases of visual contexts. By strictly controlling musical variables such as instrumental timbre, dynamics, rhythm, textural density, and melodic contour, we aimed to investigate the effects of tonal dissonance on the neural processing of emotional valence during mental state inferences (45, 46).

Why certain tone combinations are perceived as more attractive, harmonious, and positively valenced than others has been debated for centuries. The concept of pleasant-sounding (consonant) intervals entailing special numerical properties has been attributed to Pythagoras (47). Empirical evidence indeed shows that musical intervals with simple frequency ratios confer perceptual processing advantages (48, 49). Intervals such as the octave (2:1) and perfect fifth (3:2) are also judged as more consonant (i.e. more pleasant, smooth) than intervals with more complex ratios, such as the minor second (16:15) and tritone (45:32), which have been consistently appraised as more dissonant (i.e. more unpleasant and less smooth) (50–53). Theorists have argued that the singular perceptual status of consonant intervals could derive from the exposure to particular musical cultures or styles [psychocultural hypothesis: (54, 55)] or from their presence in naturally occurring sounds including those of speech [vocal similarity hypothesis: (56–58)]. Recent studies on auditory roughness, a psychoacoustic factor originally studied by Helmholtz (59), suggest that roughness-mediated aversion mechanisms could underlie the vocal similarity model. Multidisciplinary research on human and animal vocalizations has further revealed that auditory roughness is associated with distress and danger, and that it can therefore induce defensive behavioral and neural responses (for a review see ref. 60). Substantial evidence indicates that the degree of consonance/dissonance is strongly associated with the percept of valence (51, 52, 61). Valence judgments have been shown to reliably index the evaluation of consonance/dissonance level in Western musicians and nonmusicians (50, 51, 62), and this association has been also observed in listeners never exposed to Western music [(53, 63–71) although see also (72, 73)].

The present experiment focused on valence judgments during mental state attribution to the main character in an animated short film (Materials and Methods and Fig. 4). The hypothesis was that although participants would watch the same visual scene, tonal dissonance would bias their interpretative framework (11) leading to more negatively valenced mental state inferences. Furthermore, we predicted that early visual processing would interact with higher-level auditory association areas, indicating the influence of music on the visual experience. Accordingly, we hypothesized that the auditory ventral stream (AVS) (74–76), a brain pathway involved in mapping contextual sound cues to semantic associated attributes, would modulate low-level visual encoding via feedback projections.

Fig. 4.

Experimental stimuli and design: After 180 s of fixation, an instruction to attend specifically to mental states is given (10 s), the film clip [Man with pendulous arms” (Laurent Gorgiard, 1997); duration: 125 s] follows with either music condition 1 (consonance) or condition 2 (dissonance) in randomized order. Following the clip participants are asked to rate the valence of the movie character’s intentions (10 s) on a scale ranging from 1 (good intentions) to 4 (bad intentions). Two control conditions are included; i) visual alone category (i.e., film clip with no soundtrack but same instruction as above) and, ii) film clip with an instruction to describe the physical appearance of the character, to control for multimodal sensory processing, working memory, and attentional demands of the task.

To test our hypotheses, we designed an experimental paradigm that instructed participants to ascribe mental states to the main character in the film clip. Through investigating the recruitment of theory-of-mind neural substrates, the influence of tonal dissonance on mental state inferences, the interaction between early visual processes and sound-to-meaning systems, and the potential impact of top–down contextual influences on visual encoding in V1, we aimed to uncover the cognitive mechanisms and the neural underpinnings of the emotional responses to musical dissonance during naturalistic film viewing.

Results

Behavioral Results.

Impact of tonal consonance/dissonance on valence judgments.

During the present fMRI study, 38 subjects watched the same short-film with controlled consonant or dissonant music, in randomized order. Prior to the start of the audio-visual film clip subjects were instructed to think about the intentions of the main character in the following film clip. Following the clip participants were asked to rate the valence of the movie character’s intentions on a scale ranging from one (good intentions) to four (bad intentions). Two control conditions were included; i) visual alone category (i.e., film clip with no soundtrack but same instruction as above) and, ii) film clip with an instruction to describe the “physical” appearance of the character, to control for multimodal sensory processing, working memory, and attentional demands of the task, without cueing subjects to attend specifically to mental states (see Materials and Methods and Fig. 4 for details).

Behavioral results revealed significant differences between the consonant, dissonant, and visual alone conditions (Table 1). A repeated measures MANOVA showed a significant effect of condition on the judgments of valence regarding the character’s intentions (Wilks’ Lambda = 0.509, F(_{2, 36}) = 17.348, P < 0.001). [The data did not violate the sphericity assumption; Mauchly’s test of sphericity was not significant (P = 0.345)]. Post hoc tests (Bonferroni corrected) indicated that the average valence rating for the dissonant condition was significantly more negative than the valence rating for the consonant condition (P < 0.001, d = 0.895) and the visual alone condition (P = 0.013, d = 0.368). There was also a significant difference in valence ratings between the consonant and the visual alone conditions (P < 0.001, d = 0.526). Concretely, subjects ascribed significantly more negative (positive) intentions to the character on-screen when the film clip was paired with dissonant (consonant) music. Polynomial contrasts on the mean ratings for the three categories indicated a significant linear trend (F (_{1, 5.263}) = 15.289, P < 0.001), dissonance > visual alone > consonance (Table 1).

Table 1.

Valence means with SD and 95% CI for each condition (consonance: movie with consonant music; dissonance: movie with dissonant music, visual alone: movie without music)

Valence ratings for the dissonant, consonant, and visual alone conditions
Condition	Mean	SD	95% CI (adjusted)
Dissonance	3.052	0.804	[2.892, 3.213]
Visual alone	2.684	0.620	[2.543, 2.825]
Consonance	2.157	0.973	[1.984, 2.332]

The range for valence ratings was one (good intentions) to four (bad intentions).

The results support previous research that has examined the affective reactions induced by consonance/dissonance (50–52, 61, 66, 77, 78); and, specifically converge with studies assessing the impact of consonance/dissonance on valence judgments during mental state attribution, in which increasing levels of dissonance induce more negatively valenced inferences (79–81).

fMRI Results: Subtractive Analysis.

Effects of the theory-of-mind task compared with the control condition (physical appearance).

The comparison between the audio-visual theory-of-mind (ToM) task (i.e., think about the intentions of the main character) against the audio-visual control condition (focus on the physical appearance of the main character) evidenced signal changes in bilateral supramarginal and inferior parietal gyrus (Table 2A and Fig. 1D). The results are in line with previous studies that have assessed the neural systems supporting implicit nonverbal ToM encoding (82) and corroborate that our task engaged mental state inference processing (45, 46).

Table 2.

(A) fMRI results [at threshold P < 0.001 voxel-level uncorrected, P < 0.05 cluster-level FWE-corrected; (Additionally, * markings show brain regions with significant signal changes at voxel-level FWE-corrected P < 0.05)] of group General Linear Model for the contrasts: dissonance > consonance, consonance > dissonance, consonance > visual alone, dissonance > visual alone, and audio-visual task (Theory of Mind) > audio-visual control (Physical appearance)

Region	Cluster Peak MNI	Voxels	Max t-value (z-value)	Mean t (std.)	Cluster P (FWE)
(A) Subtractive analysis
Dissonance > Consonance
Occipital Mid L*	−15 −94 8	78	5.70 (4.80)	4.05 (0.47)	<0.0001
Occipital Sup L*		17		4.07 (0.64)	Cl.
Occipital Inf L		15		4.23 (0.58)	Cl.
Lingual L		8		3.82 (0.49)	Cl.
Calcarine L		5		3.85 (0.19)	Cl.
Calcarine R*	12 −94 8	37	5.51 (4.80)	3.72 (0.33)	0.037
Cuneus R		5		4.72 (0.81)	Cl.
Occipital Sup R		5		3.79 (0.45)	Cl.
Occipital Mid R		4		3.60 (0.16)	Cl.
Consonance > Dissonance
(No suprathreshold activations were observed)
Consonance > Visual Alone
Temporal Sup R*	54 −16 5	237	8.31 (6.21)	4.76 (1.05)	<0.0001
Heschl R*		44		4.69 (1.18)	Cl.
Temporal Sup L*	−48 −25 8	177	7.05 (5.58)	4.58 (0.87)	<0.0001
Heschl L*		25		5.02 (1.09)	Cl.
Dissonance > Visual Alone
Temporal Sup L*	−48 −16 5	296	8.54 (6.31)	4.74 (0.99)	<0.0001
Heschl L*		42		5.29 (1.34)	Cl.
Temporal Sup R*	60 −13 5	399	8.14 (6.12)	4.78 (1.00)	<0.0001
Heschl R*		56		5.19 (1.01)	Cl.
Occipital Inf L*	−33 −88 −10	96	6.76 (5.42)	4.34 (0.74)	<0.0001
Occipital Mid L*		86		4.33 (0.81)	Cl.
Occipital Sup L		10		3.78 (0.28)	Cl.
Calcarine L		3		4.53 (0.20)	Cl.
Precentral L*	−39 −7 62	144	6.63 (5.35)	4.20 (0.74)	<0.0001
Supp Motor Area L		84		3.93 (0.51)	Cl.
Occipital Inf R	45 −73 −7	38	5.47 (4.65)	3.81 (0.37)	<0.0001
Calcarine R		37		3.68 (0.29)	Cl.
Occipital Mid R		20		3.58 (0.20)	Cl.
Occipital Sup R		2		3.72 (0.06)	Cl.
Parietal Sup L	−30 −52 53	69	5.32 (4.56)	3.79 (0.37)	<0.0001
Parietal Inf L		30		4.13 (0.46)	Cl.
Frontal Inf Tri L	−42 17 26	34	5.01 (4.38)	3.94 (0.44)	0.006
Frontal Inf Oper L		10		3.53 (0.18)	Cl.
Insula L		5		3.76 (0.37)	Cl.
Frontal Inf Tri R	54 32 11	52	4.82 (4.22)	3.74 (0.36)	0.008
Frontal Inf Oper R		6		3.95 (0.47)	Cl.
Insula R		1		3.35 (-)	Cl.
Mid Prefrontal R	27 41 20	17	4.79 (4.20)	3.75 (0.43)	<0.0001
Cingulum Ant R		13		3.88 (0.34)	Cl.
Task (Theory of Mind) > Control (Physical appearance) - ROI analysis: TPJ, mPFC, ACC
Parietal Inf L*	−57 −28 29	100	6.36 (5.20)	4.17 (058)	<0.0001
SupraMarginal L*		43		4.16 (0.72)	Cl.
Postcentral L		19		3.92 (0.40)	Cl.
Parietal_Inf_R	33 −37 41	25	5.44 (4.63)	3.90 (0.38)	<0.0001
(B) Functional connectivity analysis (PPI). Dissonance > consonance
Seed region (6 mm radius sphere) located around highest peak for each subject within the right Calcarine
Temporal Mid R*	51 −70 −4	182	7.86 (5.99)	4.45 (0.92)	<0.0001
Temporal Inf R*		81		4.76 (1.09)	Cl.
Occipital Mid R*		27		4.39 (0.83)	Cl.
Occipital Mid L	−42 −76 −1	94	5.40 (4.61)	4.16 (0.56)	<0.0001
Temporal Mid L		60		4.05 (0.55)	Cl.
Temporal Inf L		3		3.59 (0.21)	Cl.
Seed region (6 mm radius sphere) located around highest peak for each subject within the left middle Occipital
Temporal Mid R*	51 −70 −1	88	6.33 (5.18)	4.29 (0.82)	<0.0001
Temporal Inf R		27		4.01 (0.54)	Cl.
Occipital Mid R		9		4.15 (0.50)	Cl.
Occipital Mid L*	−48 −76 5	62	5.77 (4.84)	4.18 (0.64)	<0.0001
Temporal Mid L*		60		3.96 (0.55)	Cl.
Temporal Inf L		3		3.79 (0.17)	Cl.
(C) Effective connectivity analysis. DCM – BMS (Effects of dissonance) Models				Expected sk|Y	Exceedance ψk
1) right middle temporal gyrus → left middle occipital gyrus				0.6874	0.9104
2) left middle occipital gyrus → right middle temporal gyrus				0.1499	0.0350
3) Bidirectional				0.1627	0.0546

(B) Results of group psychophysiological interactions analysis (PPI) with seed voxels (spheres with a 6 mm radius) located around highest peak for each subject within the right calcarine cortex and within the left middle occipital gyrus (contrast dissonance > consonance). The regions described showed stronger positive functional connectivity with either of these regions during the dissonant condition compared to the consonant condition. (C) Dynamic Causal Modeling (DCM) Bayesian model selection (BMS) results: conditional probability (expected posterior probability representing the probability of a model given the observed data) and exceedance probability (probability compared with other tested models). Model 1 specified a connection from the rMTG to the lMOG (hypothesized). Model 2 specified a connection from the lMOG to the rMTG. Model 3 specified bidirectional connections between both regions. Model 1 obtained the most evidence. Abbreviations: L, left; R, right; TPJ, temporo-parietal junction: supramarginal gyrus, angular gyrus, and superior temporal gyrus; mPFC, medial prefrontal cortex; ACC, anterior cingulate cortex; Cl., areas integrating the above-detailed cluster-level P-value.

Fig. 1.

Effects of tonal dissonance and theory-of-mind function during naturalistic film-viewing. fMRI results (FWE-corrected P < 0.05 for cluster-level inference). Colored areas (red) reflect statistical parametric maps (SPM) superimposed onto a standard brain in stereotactic MNI space. (A) consonance > visual alone and (B) dissonance > visual alone: similar patterns of brain response were observed bilaterally in primary and secondary auditory cortices (AC) including Heschl’s gyri. The contrast dissonance > visual alone further evidenced signal changes in occipital areas. (C) SPM showing voxels in the middle occipital gyrus and calcarine cortex in which the response was higher during the dissonant compared to consonant condition. (D) SPM showing activation in the supramarginal and inferior parietal gyrus (bilaterally) for the comparison between ToM > control conditions.

Effects of music in a ToM audio-visual context, compared to visual alone.

To establish the general effects of music compared to visual (alone) information, we contrasted consonance > visual alone and dissonance > visual alone. These comparisons showed similar patterns of brain response in primary and secondary auditory cortices (AC) (superior temporal gyrus, Heschl’s gyrus) bilaterally (Table 2A and Fig. 1 B and C). These findings converge with previous evidence from studies that have employed music listening paradigms containing harmonized chord progressions (83–85), in which signal changes were observed bilaterally in primary and secondary AC when comparing sound conditions against no-sound (or silent) conditions. The contrast between dissonance > visual alone revealed additional activations in the visual cortex (inferior, middle, superior occipital gyrus, and bilateral calcarine cortices), which we interpret as effects driven by increased tonal dissonance, and not by sound information per se (see following subsections).

The analysis of the contrast dissonance > visual alone further evidenced signal differences in the left superior/inferior parietal, bilateral frontal inferior, bilateral insula, and right medial prefrontal and anterior cingulate cortex (ACC) (Table 2A). It should be noted that signal changes within these attentional (superior/inferior parietal), salience, and emotional evaluation regions (insula and medial prefrontal cortices) were only observed while participants watched the dissonant and not the consonant condition (compared to visual alone). These findings converge with previous empirical evidence that has reported activation of attentional, salience detection, and emotional appraisal networks in response to negatively valenced auditory stimuli (80, 86) or threat-related signals (87, 88).

Effects of music in a control (physical appearance) audio-visual context.

To examine whether the above-reported effect on the visual cortex for the contrast dissonance (ToM) > visual alone was specific for the ToM task, we contrasted control dissonance (physical appearance) > visual alone, which yielded no response in the visual pathway, but only within primary and secondary AC bilaterally. The contrast of control consonance (physical appearance) > visual alone yielded a similar response on primary and secondary AC bilaterally. The comparison of control dissonance (physical appearance) > control consonance (physical appearance) did not show any suprathreshold signal changes (SI Appendix, Table S6). Overall, these results suggest that the previously reported findings of signal changes in early visual processing for the contrast dissonance (ToM) > visual alone might indicate a specific cross-modal modulation effect driven by musical dissonance during intention attribution, and which was not observed during the control (physical appearance) task.

Effects of musical dissonance compared to musical consonance within an audio-visual context.

The contrast of interest, dissonance > consonance, revealed significant signal changes in the visual cortex (bilateral middle and superior occipital gyrus, left inferior occipital gyrus), including the primary visual cortex (bilateral calcarine cortex) (Table 2A and Fig. 1A). The opposite contrast (consonance > dissonance) did not show any suprathreshold signal changes. Given the experimental design, showing identical visual information in both conditions, the results substantiate our prediction of early visual systems modulated by contextual sound cues and indicate a role of the visual system beyond visual recognition (89–91). This study reports activations encompassing early visual cortices (V1) modulated by musical information with systematically controlled manipulations of the tonal consonance/dissonance level.

It is important to note that in our previous work (86), we found no evidence for BOLD signal changes within the early visual cortex when examining the same (auditory only) musical stimuli for the contrast dissonance > consonance. In that experiment (86), we employed fMRI to investigate whether different brain regions would be involved in the processing of musical information with contrasting levels of tonal consonance/dissonance. Participants listened to the same musical materials as in the present experiment within a passive music listening paradigm. We aimed to assess whether, without an explicit instruction for cognitive processing of emotional responses, the brain areas underlying participants’ responses to consonance/dissonance would engage. By adopting this approach, the focus was oriented toward the neural substrates that could inform about the more concealing effects of dissonance, such as those exerted in audiovisual contexts. Increased activations were observed in the left medial prefrontal cortex (mPFC) and the left rostral ACC while participants listened to dissonant compared to consonant music, converging with studies that have proposed a key role of these regions during conflict monitoring (detection and resolution), and in the appraisal of negative emotion and fear-related information. The findings highlighted the involvement of the ACC and the mPFC in the appraisal of negatively valenced stimuli signaled by auditory information. The auditory cortex (bilaterally) showed stronger functional connectivity with the ACC during the dissonant portion of the task, implying a demand for greater information integration when processing negatively valenced musical stimuli. There was no evidence for suprathreshold responses within the early visual stream for the contrast dissonance > consonance (auditory only). These previous results, seem to exclude an alternative, more linear explanation of our present findings (i.e., visual cortex activation for audiovisual-dissonance > audiovisual-consonance) as responses to the auditory stimulus alone, which could have been linked to the potentially higher attentional demands that might be allocated during listening to dissonant sounds. (A detailed description of the methodology used in that experiment is included in SI Appendix, to facilitate a comparison with the present study.)

fMRI Results: Brain Connectivity.

Psychophysiological interaction analysis (PPI, functional connectivity).

The function of the primary visual cortex in visual recognition is known to be strongly modulated by multisensory information (2, 3, 92, 93). In particular, auditory feedback signals are the largest contributor (3, 94). To elucidate why dissonance recruited additional regions and to examine potential neuromodulatory influences on early visual processing, we measured functional connectivity with PPI (95). A PPI analysis identifies voxels that increase their relationship with a seed region of interest (ROI) in a given psychological context, such as when participants watch the film with dissonant compared to consonant music. Seeds ROIs (spheres with a 6 mm radius) were defined around the highest activated peak for each subject in the left middle occipital gyrus and in the right calcarine cortex, based on the significantly activated clusters for the contrast dissonance > consonance. The results (displayed in Table 2B and Fig. 2) showed that, during the dissonant condition (compared with the consonant condition), a cluster with peak in the right middle (extending to the right inferior) posterior temporal gyrus exhibited stronger functional connectivity with the right calcarine cortex and with the left middle occipital gyrus, indicating that tonal dissonance modulated the connection between early visual encoding and the ventral auditory system [represented by the right posterior middle temporal gyrus; (74, 76, 96, 97)], which is known to serve as a sound-to-meaning interface, mapping sound-based cues (e.g., distinct levels of dissonance) to their semantic associated attributes (e.g., emotional valence).

Fig. 2.

Functional connectivity modulated by tonal dissonance. fMRI results (FWE-corrected P < 0.05 for cluster-level inference). Psycho-physiological interaction analysis (PPI): blue color identifies voxels in the middle (extending to inferior) posterior temporal gyrus, which exhibited stronger functional connectivity with the primary visual cortex (PPI seed regions were 6 mm spheres around the highest activated peak for each subject within the right calcarine cortex and the left middle occipital gyrus, for the contrast dissonance > consonance).

The PPI analysis (contrast dissonance > consonance) also revealed a significant coupling between the above-described ROIs and voxels in the middle occipital gyrus. The middle occipital gyrus is considered a spatial processing region in the visual dorsal pathway (98, 99), which has also shown a functional preference for processing spatial properties of both auditory and tactile stimuli in early-blind subjects (100). These results suggest that tonal dissonance further reinforced the integration of information between the primary visual cortex and spatial processing in the dorsal visual stream.

Dynamic Causal Modeling (DCM, effective connectivity).

Methodologically, DCM is an approach used to infer causal relationships of activity patterns between different brain regions (101, 102), which therefore differs from functional connectivity measures, such as PPI, that explore nondirectional statistical dependencies (i.e., patterns of synchronization or correlation) between brain regions (95). To further examine the causal flow of information between early visual processing areas and the ventral auditory system and, specifically, to assess whether dissonance engages top–down or bottom–up interactions, we carried out an effective connectivity analysis using DCM (101). We hypothesized that distinct dissonance/consonance levels would exert differential influences on early visual encoding through feedback projections from the ventral auditory stream (103). To test this hypothesis, DCM analysis was conducted for each participant on two ROIs: i) the left middle occipital gyrus (lMOG), which showed the strongest activation in the subtractive analysis dissonance > consonance; and ii) the right middle temporal gyrus (rMTG), which exhibited the most robust functional coupling with the lMOG (see Materials and Methods for detailed information). The modulatory effect of interest was the dissonance condition, since this category appeared to strongly modulate the functional interaction (PPI) between the aforementioned areas. We modeled the direction of interaction. For each subject, three models were defined. Model 1 specified a connection from the rMTG to the lMOG (hypothesized). Model 2 specified a connection from the lMOG to the rMTG. Model 3 specified bidirectional connections between both regions. Model 1, in which the left middle occipital gyrus received information from the right middle temporal gyrus obtained the most evidence. Bayesian model averaging further indicated a negative modulation effect of dissonance (−0.0002) on the connection from the rMTG to the lMOG that changed the strength of the intrinsic connection (0.0189) (Table 2C). The results are consistent with a role of feedback signals being sent from the AVS to the early visual cortex, modulated by increased tonal dissonance.

Discussion

Impact of Tonal Consonance/Dissonance on Valence Judgments during Social Cognition Processing.

The present study aimed to investigate the cognitive and neural mechanisms underlying the effect of music on valence inferences during mental state attribution. To assess the capacity of our task to recruit mentalizing neural substrates we compared the audio-visual theory-of-mind condition with the audio-visual control condition (focused on physical appearance). This comparison revealed signal changes in bilateral supramarginal and inferior parietal gyrus; areas supporting implicit nonverbal ToM reasoning (82) as well as emotional and visual perspective taking, action observation, social attention, and encoding biases (104–107).

Following each film sequence, participants were asked to rate the valence of the movie character’s intentions (i.e., positive or negative). The results showed that musical dissonance led to significantly more negative mental state inferences, exposing its strong emotional impact. While in line with previous research that has examined the negative affects elicited by dissonance (50–52, 61, 66, 77, 78), our findings extend the evidence for its strong influence on valence judgments during mental state attribution (79–81).

Activation of V1 by Auditory Cues with Increased Level of Tonal Dissonance.

Univariate whole-brain analysis revealed significant bilateral activation of primary visual cortices (calcarine cortex) including the middle and superior occipital gyrus, while participants were watching the film sequence accompanied by dissonant, compared to consonant, music. The results endorse our prediction of modulations in early visual systems driven by musical information and provide demonstration for the engagement of crossmodal low-level sensory substrates in response to tonal dissonance.

The findings are in agreement with previous models which have highlighted the adaptable properties of cortical neurons (89–91). Electrophysiological recordings show that V1 neurons function as spatiotemporal filters encoding elementary visual features, upon which feedforward connections with higher visual areas assist in representing progressively more complex aspects of the visual scenario (108). Although the primary function of V1 is visual perception, its role expands beyond visual recognition and encompasses multisensory aspects (2, 3, 92, 93), with the largest contribution of crossmodal reentrant signals being auditory feedback (3). An illustration at relatively basic sensory processing levels has been reported by Shams et al. (94). The authors showed how a single visual flash could be misperceived as multiple flashes when paired with multiple auditory beeps. Our results go beyond this previous work by demonstrating that dissonant sound cues can qualitatively bias higher-level mental state inference processes, by exerting neuromodulatory effects on early visual encoding. The findings emphasize the role of the brain as a highly distributed processing engine in which areas across multiple hierarchies work in parallel to perform complex cognitive functions.

Top–Down Modulation of Visual Processing.

We investigated whether V1 could be subject to top–down contextual influences modulated by tonal dissonance. PPI analysis showed strong coupling between the right middle posterior temporal gyrus and V1 in response to dissonance. Effective connectivity analysis demonstrated that musical dissonance modulated visual processing via top–down feedback inputs from the right middle posterior temporal gyrus to V1.

The traditional view of sensory information processing is based on feedforward connections. Within the visual system these types of connections characterize a hierarchy of cortical areas initiated in V1 and ascending via the ventral or dorsal pathways (109, 110). Each feedforward connection, however, coexists with a reciprocal feedback connection that carries contextual information. Cortical neurons have been described as “active blackboards” or “adaptive processors” (89, 90), which can modify their function and response according to the behavioral context or the specific demands of the task being carried out. It is well established that the visual cortex can be subjected to diverse top–down influences, including among others, those of attention, reward, and emotion responses (111). Top–down signals represent influences from higher-order cognitive representations that impact earlier stages of information processing. These higher-order representations can carry contextual information that prepares the visual system for optimizing behavioral responses (90, 112).

While prior research has examined the functional effects of auditory information in the primary visual cortex, most of these studies have employed basic stimuli or simplified displays to investigate the role of sound in assisting the spatial localization of visual inputs, consequently reducing the complexity of naturalistic interaction conditions (113). Only few studies have assessed higher-order cognitive representations using naturalistic stimulus. In the study by Vetter et al. (114), 10 subjects wore a blindfold and were instructed to keep their eyes closed at all times and room lights were switched off. Participants listened to three distinct natural sound stimuli: traffic noise (a busy road with cars and motorbikes), a forest scene (birds singing and a stream), and a crowd scene (people talking without clear semantic information). Using fMRI in combination with multivariate pattern analysis (MVPA), the authors showed that category-specific information from complex natural sounds could be read out from early visual cortex activity, particularly V2 and V3 in the absence of feedforward visual stimulation. Using a univariate approach (even at a liberal threshold of P < 0.05 uncorrected) the authors did not find activation in early visual regions. Petro et al. (91) replicated the study by Vetter et al. (114) with an adjusted experimental design, to examine whether contextual auditory information could be decoded from the primary visual cortex during concurrent visual information. In this design variation, participants had their eyes open and the auditory stimuli were paired with uniform visual stimulation: a blank fixation screen. Also, using MVPA, the authors showed that sound imagery could be decoded from the early visual cortex. However, as the authors stated, “it remains to be seen if auditory information can be read out from early visual cortex when this is also receiving a more driving visual stimulus” (91). We hereby broaden these previous findings by using nonuniform, conspicuous visual stimuli (i.e., naturalistic film-viewing), and demonstrating a strong response of the primary visual cortex to musical dissonance via univariate whole-brain analyses.

Linking the schema theory with predictive coding approaches (36, 115, 116), we propose a potential mechanism to explain these modulatory influences. We argue that our manipulation of the musical stimuli triggered high-level contextual representations (i.e., affective valence associated to the distinct levels of consonance/dissonance), which were fed back to early visual cortices to facilitate the interpretation of the visual information. Throughout an audio-visual experience such as watching a film, musical affect can direct the viewer’s attention toward visual aspects that portray a matching connotative meaning. This, in turn, can allow the spectator to develop certain inferences about the characters’ internal motivations, e.g., their intentions, therefore contributing to the comprehension of the narrative. At a behavioral level, this mechanism has been conceptualized as “mood congruency effects” and explained by means of the schema theory (11). Schemas (34) [also described as scripts (117) or frames (118)] are memory associations built by extracting repeating patterns and statistical regularities from our environment. This “related” information refers to events that tend to be linked on some level (such as dissonance leading to negative valence), and is thought to be clustered in memory structures that serve as the building block for predictions (35–37). From a sociocognitive perspective, schemas allow us comprehend the social environment: They help us understand why people behave in the ways they do, and predict what is most likely to happen next. We posit that the predictions set by these memory associations are probably manifest in the signal changes we observe in the primary visual cortex.

Feedback Influences from the Auditory “What” Ventral Stream (Mapping Sound Cues to Meaning) to V1.

The two-stream hypothesis is a model of the neural processing of the cortical organization of vision (119). The model proposes two information-processing pathways originating in the occipital cortex. A dorsal visual stream leading to the parietal lobe, which is involved in processing spatial relationships among objects (“where pathway”); and a ventral visual stream that projects to the inferior temporal cortex and which is crucial for the visual identification of objects (also known as the “what pathway”).

Intersecting Goodale and Milner’s model with Wernicke’s research (120), which suggested that sensory representations of speech should at least interface with two systems: a motor-articulatory and a conceptual system; Hickok and Poeppel (121) advanced an analogous dual stream hypothesis for auditory language processing. The model entails a ventral stream supporting conceptual representations via projections to the middle posterior temporal gyrus, and a dorsal stream entailing motor representations via projections to temporal-parietal regions (75, 103). Although the specific role of the dorsal auditory stream is still a subject of debate with various hypotheses including localization, spectral processing, and auditory–motor integration, there is general agreement on the what processing function of the AVS (74–76). For instance, the prototypical verbal paradigm for targeting the ventral auditory pathway involves listening to meaningful speech versus meaningless pseudospeech (122). The ventral pathway serves as a sound-to-meaning interface by mapping sound-based representations to distributed conceptual representations (74, 76, 96, 97). In our study, functional/effective connectivity analysis revealed a causal flow of information from the right middle posterior temporal gyrus to V1 modulated by tonal dissonance. These findings suggest the involvement of the AVS in the encoding of the above-described schemas: building global representations for perceptual and semantic associated attributes by mapping nonverbal contextual sound cues to meaning (e.g., dissonant music leading to negative valence). These, in turn, provide an internal model or interpretative framework that helps to organize the visual experience into a comprehensible whole via feedback projections to the early visual cortices (Fig. 3).

Fig. 3.

Schematic representation of the findings. Left: The auditory what ventral stream plays a role in assigning meaning to nonverbal sound cues, such as dissonant music conveying negative emotions, providing an interpretative framework that serves to process the audio-visual experience. Right: Functional/effective connectivity analysis showed a coupling between the AVS and V1 in response to tonal dissonance and demonstrated the modulation of early visual processing via top–down feedback inputs from the AVS to V1.

Why does tonal dissonance, rather than consonance, drive stronger the modulatory effects on V1?

It has been consistently acknowledged that emotionally laden stimuli benefit from preferential processing given their adaptive significance (123–133). We have previously shown that the specific harmonic manipulation applied in this study can isolate emotional valence from the arousal and potency dimensions, facilitating a novel access to the neural representation of negative emotion (79, 86). Negative valence can further modulate the interaction between visual processing and attentional systems implicated in the appraisal of behaviorally relevant, unexpected, and potentially threatening events (80, 86–88). We postulate that the negative valence associated with the dissonant musical background can explain the heightened weight on visual sensory evidence. Our findings conceptually expand previous work, which has exclusively focused on the affective value associated with visual stimuli, to provide evidence of crossmodal interactions and enhanced neural responses in early visual pathways signaled by nonverbal auditory information: music.

Future research could examine other ways of inducing negative valence to modulate the neural dynamics associated with mental state attribution. For example, Chen and Spence (134) reported that the valence associated to different odors can exert a strong crossmodal influence over judgments of facial attractiveness. Could this effect also impact intentional attribution if distinct fragrances are concurrently presented during film watching? Film sound design is another captivating area to further investigate. The term, introduced by Walter Murch (135), refers to an often unconventional use of nonverbal sound effects to shape the film narrative. Even very subtle changes to the spectral components of sound effects (e.g., footsteps on a fragile vs. solid wood floor) can elicit a robust emotional effect. Empirical research within this domain of film sound could therefore bring novel insights into alternative approaches to systematically manipulate sound information to modulate mental state inferences.

The ultimate purpose of our work is to develop advanced methods of applying musical information in naturalistic paradigms to characterize complex mental health disorders. Relative to traditional paradigms, naturalistic stimuli, such as the one used in this study, are often more engaging than conventional highly constrained tasks, and therefore allow the observation of brain activity that closely resembles to freeform cognition. Film-watching provides an effective means to study multiplexed neural responses, from low-level sensory processes to high-level components: We showed that a simple, but controlled, manipulation of musical dissonance can elucidate the brain’s sound-to-meaning interface and its distributive effects during social cognition. We envision the use of nonverbal music-based fMRI paradigms for identifying behavioral signatures and measurable neural markers associated with severe psychopathology.

Conclusions

In the present study, we employed fMRI to assess the effects of musical dissonance upon the affective processing of visual information during naturalistic film viewing. Participants watched the same short-film with either dissonant or consonant music. Compared to consonant music, dissonance led to more negative mental state attributions. Neuroscientific analyses revealed that tonal dissonance modulated early visual processing via top–down cortical feedback inputs from the auditory what ventral stream to the primary visual cortex. We offer evidence for the involvement of the AVS in mapping nonverbal sound cues to meaning, providing an internal model that serves to organize the audio-visual experience. Taken together, these findings demonstrate the critical role of multisensory processing, particularly audio-visual integration, in shaping higher-order functions such as social cognition.

Materials and Methods

Subjects.

We conducted an independent prestudy (n = 8) in the same scanner to test the study design, modeling strategy, and for power calculations. Target sample sizes were selected based on fMRI power analysis conducted following (136). The power calculation was based on the primary visual and AC as ROIs, with a P-value threshold of 0.005 for a one-sided hypothesis test. Results showed that a sample size of 34 would be able to detect high effect sizes (Cohen’s d > 0.8) at power = 80%.

Data for the fMRI experiment were obtained from thirty-eight healthy volunteers. The participants (18 females and 20 males; mean age = 27 ± 4) were native German speakers, with no history of neurological or psychiatric illness, or use of psychotropic medication. They reported no long-term hearing impairment. All participants were right-handed. The sample did not include any professional musician. Five participants (two males and three females) reported having received informal musical training for less than 3 y. All subjects gave written informed consent. The study received ethical approval from the Ethics Commission at TU-Dresden (Reference Number: EK 305072016).

Stimulus Material.

The materials employed in the present study were created for, and tested in two previous experiments (79, 86). The musical stimuli comprised two soundtracks (i.e., consonant and dissonant) composed for the same short film. In summary, a choral piece, written in the form of musical variations, was made to sound consonant or dissonant by modifying its harmonic structure (i.e., interval content manipulation) producing two otherwise-identical versions of the same musical piece. The consonant condition consisted of a theme followed by three variations written by Fernando Bravo in a romantic musical style. The dissonant condition was achieved by lowering by a semitone the second violin, viola, and violoncello lines (of the consonant piece), while maintaining the other instruments (i.e., first violin and double bass) at their original pitch (SI Appendix, Fig. S2). The level of tonal dissonance (41, 42) was the manipulated variable, while other factors that are also known to contribute to the building and release of musical tension such as instrumental timbre, dynamics, rhythm, textural density, and melodic contour were strictly controlled (44, 84, 137, 138) (see SI Appendix for detailed technical description of the audiovisual stimuli, which can be downloaded from: https://doi.org/10.6084/m9.figshare.24710442.v2) (139).

Procedure.

fMRI experiment.

Prior to the start of the audio-visual film clip, and based on established methods (140–143), an instruction was given to participants designed to engage affective mental state inference processes. Subjects were instructed to “Please, think about the intentions of the main character in the following film clip” (Fig. 4). The paradigm, therefore, required to ascribe mental states (e.g., intentions) to an actor on-screen (45, 46). After the instruction, the film clip followed with either music condition 1 (consonance), music condition 2 (dissonance) in randomized order (Fig. 4). At the end of the respective clip, a valence inference question was presented (“What type of intentions does the main character have?”), which subjects responded using a 4-point scale (ranging from 1 to 4) with extremes labeled “positive” or “negative” (order counterbalanced).

Two control conditions were included; i) visual alone category (i.e., film clip with no soundtrack) with identical instructions as above to control for basic visual sensory processing, and; ii) the same audio-visual film clips with an instruction to describe the physical appearance of the character on-screen (“Please, focus on the physical appearance of the main character in the following film clip”). This additional control condition was aimed at controlling for multimodal sensory processing, working memory, and attentional demands of the task, without cueing subjects to attend specifically to mental states (141, 143, 144).

Order of presentation: The two main conditions of interest (audio-visual ToM consonance and audio-visual ToM dissonance) were presented first in counterbalanced order, to ensure that each treatment appeared comparably often as the first (or second) viewed audio-visual clip. The visual alone control condition was always at the third position of the sequence, followed by the physical appearance control condition. There was only one presentation of the physical appearance control condition, paired with consonant music when the first condition was the audio-visual ToM consonance condition, and paired with dissonant music when the first condition was the audio-visual ToM dissonance condition.

Even though we did not find differences in the neural encoding when we tested different presentation orders during the independent pilot study (which was also conducted for power calculations), we decided to maintain the audio-visual ToM conditions fixed in the first position as described above, since film watching habits rarely resemble repeated measures designs.

The experiment was run in the Neuroimaging Center (NIC) at TU Dresden. Participants were asked to arrive 45 min before the fMRI scanning in order to undertake a preparatory session of 10 min in a separate room (contiguous to the scanner room). Subjects were familiarized with the task and trained on the procedure, including the use of the 4-point scale response box system. In the fMRI setup, the audio-visual stimulus was projected onto a screen and presented to the subject via a 45° angled mirror positioned above the participant’s head. The auditory stimuli were delivered via MRI-compatible headphones. Sound pressure levels were measured with a Galaxy Audio CM130 Meter, the output volume was set to 70 dB. Participants were instructed to avoid head movements throughout the functional scan, to keep their eyes on the screen throughout the session and to focus on the presented audio-visual stimuli. Following the scanning session, each subject completed a questionnaire in order to collect subject-specific sociodemographic information.

MRI data acquisition.

Scanning was performed with a 3.0T system (General Electric, Signa). Prior to the functional magnetic resonance measurements, high-resolution (1 × 1 × 1 mm) T1-weighted anatomical images were acquired from each participant using three-dimensional fast spoiled gradient-echo (3D-FSPGR) sequence. Continuous Echo Planar Imaging (EPI) with BOLD contrast was employed (TR = 2,000 ms, TE = 25 ms, in-plane resolution = 3 × 3 mm). Each functional volume contained 34 contiguous 3.2 mm-thick axial slices separated by a 0.8 mm interslice gap.

fMRI data analysis.

Data were processed using SPM, version 12 (www.fil.ion.ucl.ac.uk/spm). Following correction for the temporal difference in acquisition between slices, EPI volumes were realigned and resliced to correct within subject movement. A mean EPI volume was obtained during realignment and the structural MRI was coregistered with that mean volume. The coregistered structural scan was normalized to the Montreal Neurological Institute (MNI) T1 template (145). The same deformation parameters obtained from the structural image, were applied to the realigned EPI volumes, which were resampled into MNI-space with isotropic voxels of 3 cubic millimeters. The normalized images were smoothed using a 3D Gaussian kernel and a filter size of 6 mm FWHM. A temporal high-pass filter with a cutoff frequency of 256 Hz was applied with the purpose of removing scanner-attributable low frequency drifts in the fMRI time series.

An event-related design was employed (146), modeling each of the thirty-seven chord presentations per music category using a canonical hemodynamic response function (147) (model specification timings are available together with the fMRI data in the figshare repository). The design matrix for the first-level analysis included the following four regressors: consonance (i.e., movie with consonant music); dissonance (i.e., movie with dissonant music), visual alone (i.e., movie without music); control (i.e., movie with dissonant or consonant music with a control task -physical appearance-). Parameter estimate images were generated. Seven contrast images per individual were calculated: dissonance > consonance, consonance > dissonance, consonance > visual alone, dissonance > visual alone, consonance/dissonance task > consonance/dissonance control, control dissonance (physical appearance) > visual alone, and control consonance (physical appearance) > visual alone. The second-level group analysis was carried out using one-sample t tests. The significant map for the group random effects analysis was thresholded at P < 0.001 for voxel-level inference with a cluster-level threshold of P < 0.05 corrected for the whole brain volume using FWE (family-wise error), which controls for the expected proportion of false-positive clusters. (See SI Appendix for further details, including information about psycho-physiological interactions and dynamic causal modelling analyses.)

Supplementary Material

Appendix 01 (PDF)

Movie S1.

Video_consonance (Audiovisual stimuli employed for the consonant condition).

Movie S2.

Video_dissonance. (Audiovisual stimuli employed for the dissonant condition).

Data, Materials, and Software Availability

fMRI data have been deposited in Figshare (https://doi.org/10.6084/m9.figshare.21345240.v1) (139).