Investigating emotion with music: An fMRI study

Abstract

The present study used pleasant and unpleasant music to evoke emotion and functional magnetic resonance imaging (fMRI) to determine neural correlates of emotion processing. Unpleasant (permanently dissonant) music contrasted with pleasant (consonant) music showed activations of amygdala, hippocampus, parahippocampal gyrus, and temporal poles. These structures have previously been implicated in the emotional processing of stimuli with (negative) emotional valence; the present data show that a cerebral network comprising these structures can be activated during the perception of auditory (musical) information. Pleasant (contrasted to unpleasant) music showed activations of the inferior frontal gyrus (IFG, inferior Brodmann's area (BA) 44, BA 45, and BA 46), the anterior superior insula, the ventral striatum, Heschl's gyrus, and the Rolandic operculum. IFG activations appear to reflect processes of music–syntactic analysis and working memory operations. Activations of Rolandic opercular areas possibly reflect the activation of mirror‐function mechanisms during the perception of the pleasant tunes. Rolandic operculum, anterior superior insula, and ventral striatum may form a motor‐related circuitry that serves the formation of (premotor) representations for vocal sound production during the perception of pleasant auditory information. In all of the mentioned structures, except the hippocampus, activations increased over time during the presentation of the musical stimuli, indicating that the effects of emotion processing have temporal dynamics; the temporal dynamics of emotion have so far mainly been neglected in the functional imaging literature. Hum Brain Mapp 2005. © 2005 Wiley‐Liss, Inc.

INTRODUCTION

During the past 10 years the majority of imaging studies on emotion have focused on the examination of emotions with negative valence. In the course of this research a neural circuit underlying the processing of aversive stimuli has been described in which the amygdala plays a crucial role: Activity changes in the amygdala have been observed during the detection [Adolphs et al., 1994, 1998; Scott et al., 1997] and generation [Lane et al., 1997; Taylor et al., 1998; Zald and Pardo, 1997, 2002] of fear‐related, negative emotions. The amygdala is involved in the emotional processing of stimuli from multiple sensory modalities [Davis and Whalen, 2001; Zald, 2003]. So far, amygdala responses have been investigated with functional imaging studies mainly in the visual modality using images of fearful facial expressions [Breiter et al., 1996; Morris et al., 1996, 1998; Phillips et al., 1997, 1998b; Whalen et al., 1998]. In the auditory modality, the amygdala has been shown to be involved in the processing of affective nonverbal vocal expressions: Involvement of the amygdala in the recognition of emotion has been suggested by functional imaging techniques using PET [Morris et al., 1999] and fMRI [Phillips et al., 1998b; Sander and Scheich, 2001], and case studies on patients with damage to the amygdala suggest that the amygdala supports the appraisal of auditory signals of danger [Anderson and Phelps, 1998; Scott et al., 1997]. With respect to music processing, a recent lesion study reported impaired recognition of fear in patients with amygdala resections [Gosselin et al., 2005]; functional imaging studies with musical stimuli have so far not reported activations of the amygdala.

The amygdala also plays a role in the processing of positive emotions [Davis and Whalen, 2001; Hamann et al., 1999; Liberzon et al., 2003; Zald, 2003], and the amygdala is not the only neural substrate essential to emotion but part of a system that also involves other cerebral structures. The structures implicated in emotion processing have been suggested to include limbic (e.g., amygdala and hippocampus), as well as paralimbic structures (e.g., insular and orbitofrontal cortex; for a review of the anatomical specification of limbic and paralimbic structures see Mega et al. [1997]). Limbic and paralimbic structures are involved in a number of functions underlying the processing of a variety of emotions [e.g., Baxter and Chiba, 1999; Calder et al., 2001; Dolan, 2002; Hamann and Canli, 2004; Mega et al., 1997; Phillips et al., 2003; Robbins and Everitt, 1996; Rolls, 2004].

Although there has been progress in the research of circuits mediating emotions with negative valence, brain imaging studies investigating the neural correlates of positive emotions are still rare [see Nitschke et al., 2004, for an overview] (for imaging studies investigating recognition and classification of positive emotional expressions see Phillips et al. [1998a]; Morris et al. [1996]). The investigation of the neural correlates of emotions with positive valence is challenging, because in an experimental setting these emotions are more difficult to evoke than negative emotions (especially in experimental settings like those required when applying functional imaging techniques such as PET or fMRI).

In the present study we used musical stimuli to evoke emotion because music has been shown to be capable of inducing strong emotions with both positive and negative emotional valence consistently across subjects [Krumhansl, 1997]. However, to date only a few imaging studies have addressed the investigation of emotion with music. Using PET, Blood et al. [1999] investigated the emotional dimension of pleasantness/unpleasantness with sequences of harmonized melodies. The stimuli varied in their degree of (permanent) dissonance, and were accordingly perceived as less or more unpleasant (stimuli with highest permanent dissonance were rated as the most unpleasant). Stimuli were presented under computerized control without musical expression. This paradigm was not intended to induce the full range of (pleasant) musical mood, yet it allowed examination of emotional processing with music while simultaneously excluding effects of musical preference on the perception of the emotional valence of the stimuli. Increasing unpleasantness of the stimuli correlated with activations of the right parahippocampal gyrus and the precuneus, while decreasing unpleasantness of the stimuli correlated with activations of frontopolar, orbitofrontal, and subcallosal cingulate cortex.

The present study takes a similar approach, using pleasant and unpleasant musical stimuli to investigate emotion. In contrast to Blood et al. [1999], the pleasant musical excerpts were not computerized sounds but natural musical stimuli (joyful, instrumental dance‐tunes recorded from commercially available CDs). Unpleasant stimuli were permanently dissonant counterparts of the original musical excerpts. Permanently dissonant signals are usually perceived as more unpleasant compared to mainly consonant signals [Blood et al., 1999; Schellenberg and Trehub, 1994; Trainor and Heinmiller, 1998; Van de Geer et al., 1962; Wedin, 1972; Zentner and Kagan, 1998; these studies do not exclude that cultural experiences can modify judgments about the pleasantness of certain dissonances]. Note that, compared to the stimuli used by Blood et al. [1999], the present stimuli were intended to induce not only unpleasantness, but also pleasantness as a response to the joyful, naturalistic music.

A PET study by Brown et al. [2004] investigated activations elicited by unfamiliar pleasant music. Contrasted to a rest condition, pleasant music activated limbic and paralimbic structures, including subcallosal cingulate cortex, anterior insula, the posterior hippocampus, the superior temporal poles, and part of the ventral striatum (nucleus accumbens). Similarly, another experiment by Blood and Zatorre [2001] measured changes in regional cerebral blood flow (rCBF) during “chills” when participants were presented with a piece of their own favorite music (as a control condition, participants listened to the favorite piece of another subject). Increasing chills intensity correlated with increases in rCBF in brain regions thought to be involved in reward and emotion, including the insula, orbitofrontal cortex, and the ventral striatum. Decreases in rCBF (with increasing chills intensity) were observed in the amygdala, the hippocampus, and the ventral medial prefrontal cortex. Compared to the paradigm used in the study from Blood and Zatorre [2001], the present study has the advantage that identical stimuli were used across subjects, enabling the investigation of emotion independent of personal preferences of listeners. Moreover, in contrast to the studies from both Blood and Zatorre [2001] and Brown et al. [2004], the present study used pleasant as well as unpleasant music, allowing investigation not only of the processing of stimuli with positive but also with negative emotional valence in a within‐subjects design.

We hypothesized that participants perceive consonant musical excerpts as pleasant, and permanently dissonant excerpts as unpleasant. Because the amygdala has been implicated in the processing of aversive stimuli, we expected that unpleasant music elicits activity changes in the amygdala. Based on the findings of the mentioned PET studies on music processing [Blood et al., 1999; Blood and Zatorre, 2001; Brown et al., 2004], we also expected activity changes in other limbic and paralimbic structures, namely, in the hippocampus, the ventral striatum, frontopolar, orbitofrontal, as well as subcallosal cingulate cortex in response to pleasant music, and in the parahippocampal gyrus in response to unpleasant music.

Another aim of the present study was to compare brain activations elicited during early and later stages of the processing of music with emotional valence. The investigation of the temporal dynamics of emotion has so far been rather neglected, and to our knowledge only one psycho–physiological study has investigated this issue with music [Krumhansl, 1997]. In this study, several physiological measures (including cardiac, vascular, electrodermal, and respiratory functions) were recorded while listeners heard musical excerpts chosen to represent one of three emotions (sadness, fear, and happiness). Significant correlations were found between most of the recorded physiological responses and time (measured in 1‐s intervals from the beginning of the presentation of each musical excerpt). The strongest physiological effects for each emotion type generally tended to increase over time, suggesting that the intensity of an emotional experience may increase over time during the perception of a musical excerpt. Thus, differences in brain activations could be expected in the present study when comparing activations elicited during the first half and during the second half of the presentation of each musical stimulus (each excerpt lasted ∼1 min). We expected differences between both halves, especially for the pleasant excerpts, because tender positive emotions might require a certain amount of time to unfold.

SUBJECTS AND METHODS

Participants

Eleven nonmusicians (five females; age range 20–29 years, mean 24.6 years) participated in the experiment after giving written informed consent. All subjects were right‐handed [Oldfield, 1971] (handedness quotient >90 according to the Edinburgh Handedness Inventory), and reported to have normal hearing. None of the subjects had any special musical expertise or musical education; no subject had learned an instrument or had had singing lessons.

Stimuli

Pleasant stimuli were eight excerpts of joyful instrumental dance‐tunes from the last four centuries (all major‐minor tonal music), recorded from commercially available CDs: A. Dvorák, Slavonic Dance No. 8 in G Minor (Op. 46); J.S. Bach, Badinerie (Overture No. 2, BWV 1067); J.S. Bach, Bourrée (Overture No. 1, BWV 1066); J.S. Bach, Rejouissance (BWV 1069); F. Canaro, La Punalada (CD‐ASIN: B00000DXZQ); J. Pastorius, Soul intro (“The Chicken,” CD‐ASIN: B0000C24JN); P.F. Caroubel, Volte (CD‐ASIN B0000247QD); Anonymous, Entree‐Courante (CD‐ASIN B0000247QD).

Unpleasant stimuli were electronically manipulated counterparts of the original tunes (stimuli were processed using CoolEdit Pro software): For each pleasant stimulus, a new soundfile was created in which the original (pleasant) excerpt was recorded simultaneously with two pitch‐shifted versions of the same excerpt, the pitch‐shifted versions being one tone above and a tritone below the original pitch (samples of the stimuli are provided at http://www.stefan-koelsch.de/Music_Emotion1).

Importantly, both pleasant and unpleasant versions of an excerpt (original and electronically manipulated) had the same dynamic outline, identical rhythmic structure, and identical melodic contour, rendering it impossible that simply the bottom‐up processing of these stimulus dimensions already contributes to brain activation patterns when contrasting effects of pleasant and unpleasant stimuli.

Procedure

Participants were presented at least 3 days prior to the functional MRI study with the musical stimuli (i.e., with both original and electronically manipulated versions) to ensure that all participants were similarly familiar with the stimulus material. In the fMRI experiment, pleasant and unpleasant excerpts were presented alternately, duration of excerpts was between 45 s and 60 s (mean duration 55 s). All excerpts were presented twice during the functional measurements to increase the signal‐to‐noise ratio. Each stimulus began and ended with a signal tone (440 Hz, sine wave) of 600 ms, and each excerpt was followed by an interval of 3 s in which no music was presented. Within this 3‐s interval, participants had to indicate how (un)pleasant they felt by pressing response buttons according to a five‐point‐scale (with −2 corresponding to very unpleasant, 0 neutral, and +2 very pleasant). That is, subjects were asked to rate their own emotional state [see also Krumhansl, 1997]; they were not asked to make an assessment of the emotions expressed by the music. Behavioral responses were averaged for the consonant and dissonant excerpts separately for each subject. Then, pleasant and unpleasant ratings from all subjects were compared with a two‐tailed, one‐sample t‐test.

During the presentation of the music, participants were instructed to listen carefully to the music and to tap the meter of the music with their right index finger (for example, during a musical piece with a metronome beat of ∼90 beats per minute, participants tapped with their finger at an average rate of 1.5 Hz). This task allowed us to control if participants paid attention not only to the consonant but also to the dissonant stimuli. For each subject and each excerpt, all intertap time intervals (i.e., time intervals between two subsequent taps) were measured. Subsequently, the variance of these intertap time intervals was calculated separately for each excerpt. To investigate if the subjects' tapping performance was comparably accurate for both pleasant and unpleasant excerpts, t‐tests were used to compare the variances between each pleasant excerpt and its unpleasant counterpart at the group level.

fMRI

Scanning procedure

Inside the scanner, participants were acoustically shielded by placing soundproofing material around the heads of participants and by hearing protection placed over the headphones. Scanning was performed on a 3 T Bruker Medspec 30/100 spectrometer. Nine axial slices (19.2 cm field of view, 64 × 64 matrix, 5 mm thickness, 2 mm spacing), parallel to the AC–PC plane (five slices above AC–PC), using a single shot, gradient recalled EPI (TR 3000 ms) were continuously acquired. The relatively long TR and the relatively low number of slices were chosen to avoid a strong masking of the music by the scanner noise. Two functional runs were conducted in direct succession, each run comprising 360 acquisitions (resulting in a duration of 18 min for each run). Each excerpt was presented once in each run. Each acquisition sampled over the nine slices. The first four acquisitions of each functional run were excluded from data evaluation to compensate for T ₁‐saturation effects. Prior to the functional sessions, two anatomical data sets (MDEFT, consisting of 16 anatomical slices, and an EPI‐T ₁ image) were acquired.

fMRI data analysis

fMRI‐data were processed using the software package LIPSIA [Lohmann et al., 2001]. Functional data were corrected for slicetime acquisition differences using sinc‐interpolation. In addition, data were corrected for motion artifacts. A temporal highpass filter with a cutoff frequency of 1/132 Hz was used for baseline correction of the signal and a spatial Gaussian filter with 5.6 mm FWHM was applied.

To align the functional data slices with a 3‐D stereotactic coordinate reference system, a rigid linear registration with six degrees of freedom (three rotational, three translational) was performed. The rotational and translational parameters were acquired on the basis of the MDEFT and EPI‐T1 slices to achieve an optimal match between these slices and the individual 3‐D reference dataset. This 3‐D reference dataset was acquired for each subject during a previous scanning session. The rotational and translational parameters were subsequently transformed by linear scaling to a standard size [Talairach and Tournoux, 1988]. The resulting parameters were then used to transform the functional slices using trilinear interpolation, so that the resulting functional slices were aligned with the stereotactic coordinate system.

Statistical evaluation for each subject was based on a least‐squares estimation using the general linear model for serially autocorrelated observations [Friston, 1994]. The design matrix was generated with a boxcar (square wave) function, observation data and design matrix were convolved by a Gaussian kernel of dispersion of 4 s FWHM. The output image contains the parameter estimation (beta‐values) which are an estimation for the slope of the regression. These images were evaluated for each subject using a linear contrast. Pictures containing a signal tone were excluded from data evaluation.

As noted before, each individual functional dataset was aligned with the standard stereotactic reference space. A second‐level statistical analysis was performed consisting of a one‐sample t‐test across the aligned contrast images of all subjects. The t‐test indicated whether observed differences between the pleasant and the unpleasant condition were significantly different from zero [Holmes and Friston, 1998]. Additionally, a smoothness estimation was performed [Kiebel et al., 1999]. The resulting t‐map was thresholded at t = 4.14 (P < 0.001, one‐tailed, uncorrected). The smoothness estimation was used to describe the statistical parametric t‐map in terms of excursion sets. The corrected significance of the results are based on cluster‐ and voxel‐level inferences using distributional approximations from the theory of Gaussian fields [Worsley et al., 1996]. Clusters of activated voxels were taken into account when the size of the cluster comprised at least three voxels, and when the reported activations were significant using a threshold corresponding to P < 0.05 (corrected for multiple comparisons) at the cluster level [Worsley et al., 1996].

Both t‐maps and region of interest (ROI) values were first calculated modeling each excerpt as one block. To investigate differences in activation between the first and the second half of each excerpt (see Introduction), each excerpt was then modeled as consisting of two blocks: the first block was modeled for the first half of each excerpt (second 1–30), and the second block was modeled for the second half of each excerpt (second 31 to the end of each excerpt). The t‐maps and ROI values were then computed separately for each block.

ROI‐analysis

To test lateralization of activations, as well as differences in activations between blocks, analyses for ROIs were performed for structures mentioned in the hypotheses (amygdala, hippocampus, parahippocampal gyrus, and ventral striatum). Moreover, analogous ROI analyses were performed for all other structures that were significantly activated in the t‐maps corrected for multiple comparisons. In each participant, individual ROIs were defined as single voxels (3 × 3 × 3 mm) according to each subjects' individual brain anatomy. The definition of ROI coordinates for each subject individually accounts for intersubject anatomical differences that can deteriorate group statistical data, especially when examining activations of small subcortical structures. The exact locations of the individual ROIs were established as follows. The maximum voxel of activation in the t‐map of a group contrast (e.g., unpleasant > pleasant, computed for entire excerpts) was located for each brain structure. Within a search radius of 6 mm from these maxima (and within the anatomical boundaries of the ROI in each subject), the coordinates for the individual ROIs were defined as the maximum voxel of activation in the individual contrasts. For statistical comparisons, ROI‐values were analyzed for each structure and its contralateral homotope (e.g., left and right amygdala) using ANOVAs with factors condition (pleasant, unpleasant), block (first, second), and hemisphere.

RESULTS

Behaviorally, subjects rated the original (mainly consonant) stimuli as pleasant, and the manipulated (permanently dissonant) stimuli as unpleasant. The average rating for the consonant stimuli was +1.1 (on a 5‐point scale ranging from −2 to +2), for the dissonant stimuli −0.6, the difference between ratings being significant (P < 0.001, two‐tailed t‐test). Both stimulus categories (pleasant and unpleasant) were rated significantly different from zero (corresponding to emotionally neutral; P < 0.0001 in each test). Participants showed comparable accuracy in tapping performance for both pleasant and unpleasant excerpts: None of the t‐tests used to compare the variances between each pleasant excerpt and its unpleasant counterpart at the group level (see Subjects and Methods) yielded a significant difference between pleasant and unpleasant excerpts. This indicates that participants performed the tapping task in a comparable manner during both pleasant and unpleasant excerpts, and that, thus, participants not only paid attention to the pleasant but also to the unpleasant excerpts.

In the fMRI data, the t‐maps of entire excerpts (corrected for multiple comparisons, see Subjects and Methods) showed significant activations during the presentation of unpleasant music (contrasted to pleasant music, unpleasant > pleasant) in the left hippocampus, the left parahippocampal gyrus, and the right temporal pole (Fig. 1A, Table I). When analyzing corrected t‐maps of the first and second block separately (guided by our hypotheses), an additional activation was indicated for the (left) amygdala in the second block (Fig. 1B, Table I). In all mentioned structures, activity changes were not only due to an increase of the fMRI signal during the presentation of the unpleasant music, but also due to signal decreases during the presentation of the pleasant music (Fig. 3).

Figure 1.

Activations elicited during the presentation of unpleasant (contrasted to pleasant) music (t‐maps), separately for entire excerpts (A), and for the second block of excerpts only (B, see Subjects and Methods for details). The t‐maps were thresholded using an error probability of P = 0.001 (corrected for multiple comparisons). Unpleasant music activated the hippocampus, the parahippocampal gyrus, the temporal poles, and the amygdala.

Table I.

Activations as indicated by the t‐maps (corrected for multiple comparisons)

Anatomical structure	Left hemisphere			Right hemisphere
	Coordinate	t‐value	mm3	Coordinate	t‐value	mm3
Both blocks
Unpleasant > Pleasant
Hippocampus	−28, −14, −14	6.9	135	25, −14, −14	3.1	27
Parahippocampal g.	−25, −26, −11	5.7	135	22, −26, −13	3.8	27
Temporal pole	−37, 12, −20	3.9	243	37, 9, −23	4.2	27
Pleasant > Unpleasant
Heschl's g.	−37, −26, 14	4.9	81	46, −17, 11	4.6	81
IFG (BA45/46)	−37, 30, 7	4.8	270	37, 30, 7	—
Ant. sup. insula	−29, 18, 8	4.5	135	34, 21, 5	3.4	54
Second block only
Unpleasant > Pleasant
Amygdala	−19, −5, −14	4.7	54	15, −5, −17	3.3	54
Pleasant > Unpleasant
Rolandic op. (BA43)	−49, −4, 8	6.2	324	49, 0, 11	3.2	183
IFG (BA44i)	−52, 9, 2	3.5	27	49, 3, 8	5.1	27
Ventral striatum	−10, 6, −8	4.1	54	10, 6, −4	5.1	81

Values in bold indicate t‐values and size of activations (in mm³) obtained with a threshold of t = 4.14 (corresponding to P‐values smaller than 0.001); plain type values were obtained with a threshold of t = 3.17 (corresponding to a P‐value of 0.005).

IFG, inferior frontal gyrus.

Figure 3.

Percent signal change of trial‐averaged fMRI signals within the structures reported in Tables I and II; fMRI signals were computed for the voxels used for the ROI analyses (see Subjects and Methods), and averaged for all four conditions (first and second block of pleasant and unpleasant stimuli). AL/AR: left/right amygdala, PL/PR: left/right parahippocampal gyrus, TL/TR: left/right temporal pole, VSL/VSR: left/right ventral striatum, HIL/HIR: left/right hippocampus, HL/HR: left/right gyrus of Heschl, IL/IR: left/right insula, RL/RR: left/right Rolandic operculum.

The ROI analysis indicates that activations of amygdala, parahippocampal gyrus, and temporal poles were significantly stronger during the second block (Table II); no difference between blocks was noted for the hippocampus. When analyzing the ROI values of amygdala, parahippocampal gyrus, and temporal poles separately for each block, all mentioned structures were activated during both blocks, although activations of the amygdala were only marginally significant during the first block (P < 0.06). It appears that activations of amygdala, hippocampus, and parahippocampal gyrus were stronger in the left hemisphere, and the activation of the temporal pole stronger in the right hemisphere, but these observations were not statistically confirmed by the ROI analyses (ANOVAs computed with factors condition and hemisphere did not yield any two‐way interaction).

Table II.

Results of ROI analyses

Anatomical structure	Condition	Cond. × Block
Unpleasant > Pleasant
Amygdala	0.002	0.0005
Hippocampus	0.0005	—
Parahippocampal g.	0.005	0.0001
Temporal pole	0.0001	0.002
Unpleasant > Pleasant
Heschl's g.	0.002	0.0001
IFG (BA45/46)	0.001a	0.0001
IFG (BA44i)	0.01	0.0001
Ant. sup. insula	0.001	0.0002
Rolandic op. (BA43)	0.001	0.0001
Ventral striatum	0.005	0.001

Values are expressed as P.

^aInteraction Condition × Hemisphere: P > 0.05.

For each anatomical structure, ROI values were compared at the group level using ANOVAs with factors condition (pleasant, unpleasant), block (first, second), and hemisphere.

IFG, inferior frontal gyrus.

Figure 2A shows the fMRI data of the opposite contrast (pleasant > unpleasant, entire excerpts). The corrected t‐maps indicate activations of Heschl's gyrus, the anterior superior insula, and the left inferior frontal gyrus (IFG, BA 45, 46; see also Table I). When analyzing corrected t‐maps of the first and second block separately, additional activations were observed during the second block in the Rolandic operculum (BA 43, extending into the precentral sulcus), and the inferior portion of the (right) frontal operculum (inferior pars opercularis, BA 44i; Fig. 2B,C, Table I).

Figure 2.

Activations elicited during the presentation of pleasant (contrasted to unpleasant) music (t‐maps), separately for entire excerpts (A), and for the second block of excerpts only (B,C). The t‐maps were thresholded using an error probability of P = 0.001 (corrected for multiple comparisons). Pleasant music activated Heschl's gyri, the IFG (BA 45 and 46), as well as the left anterior superior insula during both halves, and additionally Rolandic and frontal opercular areas (inferior BA 44) during the second block.

ROI analyses performed for these structures, as well as for the ventral striatum (guided by our hypothesis), indicated significant activations for all structures (Table II). Moreover, all structures were activated significantly more strongly during the second block. When analyzing the ROI values separately for the first and second block, all structures except the ventral striatum and the pars opercularis were significantly activated during both halves of the musical excerpts; the activation of the ventral striatum was only marginally significant during the first block (P < 0.06), and no activation of the pars opercularis (BA 44i) was indicated for the first block. No significant lateralizations were found for any structure except the anterior IFG (BA 45/46; Table II).

It was also hypothesized that pleasant (contrasted to unpleasant) music activates (medial) orbitofrontal, and subcallosal cingulate cortex, but no data could be obtained from these structures because of fMRI‐related susceptibility artifacts. To investigate if the arousal [Russell, 1980] elicited by the musical excerpts differs between pleasant and unpleasant excerpts, a different group of subjects (n = 20) was investigated behaviorally using a very similar experimental protocol, except that participants were asked to rate both the valence and the arousal of the stimuli (reminiscent of previous studies, e.g., Bradley and Lang [1994]). Whereas pleasant and unpleasant ratings clearly differed from each other (P < 0.0001), no difference in arousal ratings was indicated. This suggests that differences in the fMRI activations are due to differences in emotional valence, and not due to differences in arousal.

DISCUSSION

The behavioral data indicate that participants perceived the original excerpts as pleasant and their electronically manipulated (permanently dissonant) counterparts as unpleasant, showing that the stimuli used in the present study are capable of inducing pleasant and unpleasant emotion.

During the presentation of unpleasant music, the amygdala, the hippocampus, the parahippocampal gyri, and the temporal poles were activated. The activation of the amygdala is in line with studies showing that stimuli with negative emotional valence induce activity changes in this structure (see Introduction). However, so far the majority of studies investigated the role of the amygdala for emotion processing in the visual domain, and few studies have shown involvement of the amygdala in the auditory recognition of emotion [Morris et al., 1999; Phillips et al., 1998b; Sander and Scheich, 2001; Scott et al., 1997; these studies report involvement of the amygdala in the emotional processing of nonverbal vocal expressions]. The present data demonstrate that activity changes in the amygdala can also be observed in response to unpleasant musical information, corresponding to a study reporting impaired recognition of scary music in patients with resections of the amygdala [Gosselin et al., 2005]. Note that strong deactivations were observed in the amygdala in response to the pleasant stimuli (as well as in the hippocampus, the parahippocampal gyrus, and the temporal poles), supporting the notion that activity changes within the amygdala are not only related to unpleasant, but also to pleasant emotion [see also Davis and Whalen, 2001; Zald, 2003]. With respect to music processing, Blood and Zatorre [2001] also reported deactivations of the amygdala (and of the hippocampus) in response to pleasant stimuli. Thus, the combined findings of the present and the mentioned previous studies on music and emotion [Blood and Zatorre, 2001; Gosselin et al., 2005] indicate that the amygdala plays a role for the processing of complex, meaningful auditory information with both negative and positive emotional valence.

The hippocampus is densely interconnected with the amygdala, and has previously been implicated in emotional processing [for the relevance of the hippocampus with respect to memory functions see, e.g., Greicius et al., 2003]: The hippocampus has been reported to be activated by audiogenic stressors, and to be sensitive to other emotional stressors [Bremner, 1999; Campeau and Watson, 1997; Lopez et al., 1999; Phillips et al., 2003]. It has been suggested that the hippocampus is involved in facilitation and inhibition of defensive behavior, as well as anxiety in response to threatening stimuli [Phillips et al., 2003]. Moreover, the hippocampus is assumed to be involved in affective state regulation, attentional systems, and motivation [Mega et al., 1997; Phillips et al., 2003]. Using PET, Brown et al. [2004] reported an activation of the hippocampus during listening to pleasant music (contrasted to a rest condition), although the activation focus reported by that study was located more posteriorly than the activation observed in the present study. The factors that account for the differences between the present study and the study from Brown et al. [2004] remain to be specified.

The parahippocampal gyrus is also densely interconnected with the amygdala, and has previously been implicated in emotion processing. The activation focus of the parahippocampal gyrus observed in the present study replicates findings of the study from Blood et al. [1999] in which unpleasant (permanently dissonant) music also activated this structure (note the high consistency of coordinates between studies; for a similar, although more dorsal activation of the parahippocampal gyrus during the perception of aversive pictures see Lane et al. [1997]). Likewise, the temporal poles are interconnected with the amygdala through monosynaptic connections [Amaral and Price, 1984], and have been reported to be involved in the processing of acoustic stimuli with negative emotional valence [Zald and Pardo, 2002]. A study from Brown et al. [2004] reported activations of the right superior temporal pole (BA 38/22) during listening to pleasant music, but that activation was more superior than the activation observed in the present study (near, and partly within the planum polare), and possibly associated with the cognitive processing of the musical structure. Both parahippocampal gyrus and temporal poles receive input from auditory association areas [Amaral and Price, 1984; Suzuki and Amaral, 1994], and it appears that the aspect of the parahippocampal gyrus activated in the present study and in the study from Blood et al. [1999], as well as the temporal poles, are part of a paralimbic circuit that is involved in the processing of complex auditory information with emotional valence.

It cannot be excluded that the BOLD responses observed in the contrast unpleasant > pleasant originate from inhibitory rather than from excitatory synaptic processes [Buxton, 2002] and that, thus, the unpleasant stimuli inhibited emotional activity in limbic regions (compared to activity as present during a positive emotional state) rather than activating excitatory processes in those regions. With respect to this, it is interesting to note that the hippocampus is presumably one of the most sensitive cerebral structures because it appears to be the only brain structure that can be damaged by traumatizing stressors (such as strong violence, e.g., Bremner [1999]). Thus, inhibition of neural pathways projecting to the hippocampus during the perception of unpleasant stimuli could represent a neural mechanism that serves the prevention from potential damage of hippocampal neurons.

In summary, clear activity changes were measured in limbic and paralimbic structures (amygdala, hippocampus, parahippocampal gyrus, and temporal poles) that have previously been implicated in emotion processing. The present results indicate that a cerebral network comprising these structures can be activated by the emotional processing of auditory (musical) stimuli.

During the presentation of pleasant (contrasted to unpleasant) music, the Rolandic operculum (BA 43) was activated in both hemispheres (especially during the second half of the musical excerpts, where the P‐values computed for the ROIs were significantly higher than during the first half). It is highly likely that the portion of the Rolandic operculum activated in the present study contains the representation of the larynx (and the pharynx). The larynx is a vocal tract articulator which produces (as an effector) melody, rhythm, and emotional modulation of the vocal timbre during vocal communication. Functional imaging data on the exact localization of the (sensorimotor) representation of the larynx are not yet available. However, it is known that both sensory and motor cortices are somatotopically organized [Nakamura et al., 1998; Penfield and Rasmussen, 1952], with the representation of the larynx being ventral to the representation of tongue, lower lip, and upper lip. The focus within the Rolandic operculum activated in the present study (axial plane through z = 8) is ventral to foci reported for tongue movements [Pardo et al., 1997, z = 20], swallowing [Hamdy et al., 1999, z = 24], and gustatory perception [Zald and Pardo, 2000, z = 22].

Neural circuits comprising Rolandic opercular areas have been described in some functional imaging studies on both overt and covert singing [Jeffries et al., 2003; Riecker et al., 2000; Wildgruber et al., 1996]. It is highly unlikely that the activation of the Rolandic operculum observed in the present study was due to overt singing, because participants were strictly instructed not to vocalize or to move during the experiment; when asked after the experiment, all participants assured that they carefully kept to this instruction. As an additional control, the same experimental protocol was conducted with a different group of subjects while recording the EMG of the larynx. In that experiment no motor activity of the larynx was observed in the EMG data.

The present data suggest that subjects coded vocal sound production (without actual movement) while they perceived the pleasant musical signals. This phenomenon is reminiscent of mechanisms of observation–execution matching that have so far mainly been investigated in the visual domain [see Rizzolatti and Craighero, 2004, for an overview]. In the auditory domain, mirror neurons that discharge in response to the sound of actions have been observed in monkeys [Kohler et al., 2002]; for humans, a recent study from Tettamanti et al. [2005] showed that merely listening to action‐related sentences can activate mirror‐function mechanisms. The present results suggest that the perception of (pleasant) musical information activated an auditory perception–execution matching system that includes representations of vocal sound production. This interpretation is supported by a recent study reporting activation of the Rolandic operculum during the processing of intonation contour while listening to spoken sentences [Meyer et al., 2004], suggesting that premotor programs for the production of prosodic signals (such as speech melody) are already formed when perceiving prosodic signals produced by other individuals.

The activation of the Rolandic operculum was observed during the perception of the pleasant but not of the unpleasant musical information, presumably due to the appetitive quality of the pleasant excerpts (the unpleasant excerpts rather evoked avoidance behavior). It is also possible that the dissonant stimuli were simply more difficult to vocalize than the consonant ones because of the simultaneous presentation of the two pitch‐shifted versions together with the original excerpt.

However, a study from Indefrey et al. [2001] reported an activation of the Rolandic operculum related to the structuring of individual words into phrases and sentences during the perception of spoken sentences. Thus, it cannot be excluded that the activation of the Rolandic operculum in the present study is also related to the processing of music‐syntactic information (due to the positive emotional valence of the pleasant excerpts, it is likely that participants analyzed the structure of the pleasant, but not of the unpleasant excerpts). Previous imaging studies indicate that music–syntactic processing involves the frontal operculum, especially inferior BA 44 [Janata et al., 2002a; Koelsch, 2005; Koelsch et al., 2005, 2002; Maess et al., 2001; Tillmann et al., 2003], which was also activated in the present study; the Rolandic operculum was not activated in those studies. However, in the mentioned studies stimuli were played without musical expression, and it is possible that the Rolandic operculum comes into play during the processing of naturalistic musical stimuli. This issue remains to be specified.

The music–syntactic analysis most likely also required working memory resources. The activation of these resources appears to be reflected in the activation of BA 45/46 [see also Koelsch et al., 2005; Janata et al., 2002b].

The activation of the anterior superior insula is presumably related to the activation of the Rolandic operculum. This interpretation is based on previous studies showing that both overt and covert singing activates neural circuits involving the insular cortex and the Rolandic operculum [Jeffries et al., 2003; Riecker et al., 2000; Wildgruber et al., 1996]. The superior insula has been considered a motor association area [Augustine, 1996], and particularly the left anterior superior insula has been suggested to support articulatory planning [Dronkers, 1996] and the coordination of vocal tract innervation [see Ackermann and Riecker, 2004, for a review]. It is assumed that this area supports the automatic coordination of the activity of the muscles engaged in articulation by elaborating temporospatial innervation patterns of vocal tract musculature during verbal utterances.

It has been reported that this activity is eventually dependent on emotional demands [Ackermann and Riecker, 2004], and the widespread interconnections with both premotor cortex and limbic structures render it likely that the anterior superior insula plays a role in the activation of the Rolandic operculum during the emotional perception of the musical stimuli.

The ventral striatum has previously been implicated in the processing of stimuli with positive emotional valence [Blood and Zatorre, 2001; Brown et al., 2004; Delgado et al., 2000]. It has been proposed that the motivational effects of emotionally significant stimuli are mediated in part by the ventral striatum (specifically, the nucleus accumbens, NAc). The NAc receives input from limbic structures (including the amygdala and the hippocampus), and projects to structures known to be involved in behavioral expression. Therefore, the NAc has been suggested to represent a “limbic–motor interface” [Cardinal et al., 2002]. With regard to this, it is likely that in the present study a motor‐related circuitry (comprising the ventral striatum, the insular cortex, the Rolandic, and possibly the frontal operculum) was activated, which served the formation of premotor representations for vocal sound production during the perception of the pleasant musical excerpts (activations of NAc, insular, and Rolandic cortex have also been reported in a previous study during the singing of melodies [Jeffries et al., 2003]).

Due to the positive emotional valence of the pleasant excerpts, the activation of the primary auditory cortices during the presentation of the pleasant music was presumably modulated by (top‐down) attentional processes (the increased attention leading to a greater activation of Heschl's gyri [Jäncke et al., 1999]). Note that acoustic roughness is decoded at least at the level of the primary auditory cortex [Fishman et al., 2001], most presumably at even lower levels. Thus, it is likely that the negative emotional valence of permanently dissonant auditory signals can be detected even without the contribution of the mentioned cortical structures in the frontal and Rolandic operculum. This view is supported by a lesion study that used experimental stimuli reminiscent of those used in the present study [Peretz et al., 2001]. In that study a patient with bilateral damage to the auditory cortex was found to be unable to distinguish consonant from dissonant musical excerpts.

Interestingly, activations of all structures, except the hippocampus, were stronger during the second block of the musical excerpts, presumably because the intensity of listeners' emotional experiences increased during the perception of both the pleasant and the unpleasant musical excerpts. This finding corroborates the notion that emotion processing has temporal dynamics [Krumhansl, 1997], especially when listening to music which unfolds over time [see also Blood and Zatorre, 2001; in that study musical stimuli selected to evoke chills had a duration of 90 s]. This temporal dynamics is potentially relevant for an appropriate description of the neurophysiological correlates of emotion. However, further investigations are necessary before the temporal dynamics of emotion processing can be described in more detail.

CONCLUSION

In the present study, activations during the perception of unpleasant music were observed within an extensive neuronal network of limbic and paralimbic structures comprising the amygdala, hippocampus, parahippocampal gyrus, and temporal poles. Although clear increases of BOLD signals were observed in those structures in response to unpleasant music, strong signal decreases were also measured in response to pleasant music. This indicates that these structures respond to both pleasant and unpleasant auditory information with emotional valence, and that listening to music has the capacity to up‐ as well as down‐regulate neuronal activity in these structures.

During the presentation of the pleasant music, activations were observed in the ventral striatum, the anterior superior insula, and in the Rolandic operculum. The activity within the Rolandic operculum is suggested to reflect mechanisms of perception‐execution matching during the perception of vocalizable auditory (musical) information. Rolandic operculum, anterior superior insula, and ventral striatum possibly form a motor‐related circuitry that serves the formation of premotor representations for vocal sound production during the perception of pleasant auditory information.

Finally, the present pleasant/unpleasant paradigm allowed us to play the same musical stimulus to all participants, i.e., stimuli were not chosen separately for each subject (according to each subjects' musical taste). Thus, the present study introduces music as a novel tool to generally investigate neural substrates of pleasant and unpleasant human emotion with fMRI, independent of listeners' personal preferences. This provides the possibility to use the experimental paradigm in future research with different groups of subjects such as children, patients, and subjects with different cultural backgrounds.