Serial changes of humor comprehension for four-frame comic Manga: an fMRI study

Abstract

Serial changes of humor comprehension evoked by a well organized four-frame comic Manga were investigated by fMRI in each step of humor comprehension. The neural substrates underlying the amusing effects in response to funny and mixed order manga were compared. In accordance with the time course of the four frames, fMRI activations changed serially. Beginning with the second frame (development scene), activation of the temporo-parietal junction (TPJ) was observed, followed by activations in the temporal and frontal areas during viewing of the third frame (turn scene). For the fourth frame (punch line), strong increased activations were confirmed in the medial prefrontal cortex (MPFC) and cerebellum. Interestingly, distinguishable activation differences in the cerebellum between funny and non-funny conditions were also found for the fourth frame. These findings suggest that humor comprehension evokes activation that initiates in the TPJ and expands to the MPFC and cerebellum at the convergence level.

Humor appreciation is a major part of our daily social interactions and plays a significant role in human communication. The comprehension of humor is a problem-solving cognitive task. According to the incongruity resolution theory proposed by Suls¹, humor comprehension requires incongruity detection and the resolution of incongruity between the expectations built by the story line and the consequence described in the punch line¹. The theory further suggests that the resolution of this incongruity constitutes a problem-solving task in which the perceiver is required to transform nonsense into humorous sense.

Recent neuroimaging studies have investigated the neural basis of humor comprehension while participants viewed cartoons or listened to jokes. For example, fMRI showed that the neural substrates of humor comprehension are located in cortical and subcortical brain areas regardless of joke type, be it verbal semantic or visual².

Goel & Dolan³ isolated the affective components of humor by comparing the activations associated with jokes that subjects reported as funny with the activations associated with jokes reported as not funny. Conjunction analysis indicated significant activation in the ventral medial prefrontal cortex (vMPFC), a region involved in reward processing, and the bilateral cerebellum. They suggested the affective appreciation of humor involves access to a central reward system in the vMPFC regardless of joke type. Mobbs et al.⁴ found that fronto-temporal systems including the vMPFC play a role in understanding humor. The authors also suggested that humor engages a network of subcortical regions including the nucleus accumbens, which forms a part of the dopaminergic reward system. In addition, they found activation in the left temporo-occipital junction (TOJ), inferior frontal gyrus (IFG) and temporal pole (TP) for the funny cartoon. They suggested that the TOJ and left fusiform gyrus (FFG) are involved in processing the incongruent and surprise element of jokes and play a role in the early stages of the humor network⁴. Consistent with those results, Osaka & Osaka⁵ have found activation in the striatal reward areas of participants who listened to mimic words that reminded them of emotional affective facial expressions like laughing.

In accordance with these findings, it is generally accepted that affective appreciation of humor increases activation of the fronto-temporal brain and stimulates the mesolimbic dopaminergic reward system⁶.

Recently, Bekinschtein et al.⁷ compared jokes that contained ambiguous words and sentences that contained ambiguous words but were not funny, as well as jokes that did not depend on ambiguity. Compared with the ambiguous sentences, they found activation of the bilateral temporoparietal junction (TPJ) for the ambiguous jokes. However, they also found bilateral activations in TPJ for unambiguous sentences regardless of the humor. Increased activity in a network of subcortical regions, including the amygdala, the ventral striatum, and the midbrain, was also confirmed when hearing jokes.

Chan et al.⁸ differentiated brain regions according to the comprehension and elaboration of humor stages⁹. They compared the punch line of stories using funny sentences and garden path sentences. Having subtracted the responses of garden path sentences from those of funny sentences, they also showed that the humor elaboration was associated with the left vMPFC and subcortical regions such as bilateral amygdalae and parahipocampal gyri. The feeling of amusement caused by the humor occurred at this elaboration stage. In a separate paper, Chan et al.¹⁰ investigated more details of the humor comprehension process by examining distinct brain regions that are associated with the detection and resolution of incongruities. They found that the detection of incongruities was associated with greater activation in the right middle temporal gyrus (MTG) and right medial frontal gyrus (MFG). Resolving the incongruities correlated with greater activation in the left SFG and left inferior parietal lobule (IPL). Accordingly, they suggested that the humor comprehension process is built by two stages: incongruity detection and incongruity resolution.

However, most of the above and similar studies have investigated only distinct brain regions that are associated with the punch line. On the other hand, how the humor process during the problem solving of the joke or cartoon drives the neural system before the appearance of the punch line has not yet been fully explored. Moreover, of those that do exist, most neuroimaging studies examining humor appreciation have used cartoons of only one or two scenes⁴,¹¹. For example, Bartolo et al.¹¹ presented either funny or neutral pairs of cartoons. They ran a random effect analysis to contrast the activation related with cartoons judged as funny with those judged as unfunny by the participants. Compared with the unfunny pairs, the second cartoon from the funny pairs increased activation in the right IFG, left superior temporal gyrus (STG), left MTG, and left cerebellum, suggesting these areas are related to the attribution of intention. The study did not, however, compare the serial change between the first and second frames. Such a comparison would benefit by using multiple frame cartoons, which would show which regions of the brain change with humor comprehension.

In the present paper, in order to explore the time course of the humor comprehension mechanism before the punch line and how it corresponds to brain activation, we measured brain activation evoked by a four-frame sequence comic Manga. The four-frame manga included an introduction (first scene), development (second scene), turn (third scene) and conclusion (fourth scene, punch line). The most amusing scene appeared in the last converging scene and was essential for understanding the humor of the manga.

To explore the neural substrates underlying the serial change of the humor comprehension, neural activations were serially measured using event-related fMRI while participants read the four-frame manga. Observations were made using two sets of experiments. In the first, we compared the serial changes of a four-framed manga (Funny) with those from a manga in which the order of the frame among the four frames was randomized (Mix). In the second, to compare the neural basis for perceiving the degree of humor, two kinds of manga stimuli were compared: funny and non funny.

Figure 1 shows an example four-frame manga with captions translated into English.

Figure 1. Example of manga used.

The captions of the manga are translated from Japanese. The left and right panel shows non-funny and funny manga samples, respectively. Funny and non-funny manga are under the permissions of Asahi Shimbun Co. (series No.286 on January 21, 1998 and No.4423 on September 21, 2009) and the author Hisaichi Ishii.

Results

In order to measure serial changes of the humor comprehension process, we compared corresponding frames between Funny and Mix. Behavioral data showed that the mean estimates of the four-score funniness rating by the 15 participants were 2.99 (SD = 0.37) and 1.05 (SD = 0.09) for the Funny and Mix manga, respectively. The difference between these two estimates was significant (t (14) = 439.20, p < 0.001).

In accordance with the time course of the four frames, fMRI activations changed serially. Compared with Mix, we found no significant activation in the first frame for Funny (voxel-level threshold, p < 0.001, uncorrected for multiple comparisons, and cluster-level threshold using FWE, see Figure 2 and Table 1). During viewing of the second frame, which is when the story developed, activation increases in the TPJ and FFG in the left hemisphere were observed. In the third frame, which was when the expectation of the manga began to be resolved, further activation was seen in the right TPJ, as too was an increase in the superior temporal sulcus (STS). Moreover, increased activations were found in the MPFC, striatum and thalamus. In the last frame (fourth scene), when the ambiguity was fully resolved and the most amusing scene appeared, strong activation increases were seen in the TPJ, STS, MPFC and ventro lateral prefrontal cortex (VLPFC) of both hemispheres. In addition, we found activation of the brainstem, thalamus and cerebellum in both hemispheres.

Figure 2. Rendered fMRI images of brain areas activated upon viewing the first, second, third and fourth frames of Funny and Mix manga (experiment 1) (voxel-level threshold: p < 0.001, uncorrected for multiple comparisons; cluster-level threshold: p < 0.05, corrected for multiple comparisons).

Table 1. Brain regions that were significantly more activated in response to Funny relative to Mix for each frame of the experiment 1 (voxel-level threshold: p < 0.001, uncorrected for multiple comparisons; cluster-level threshold: p < 0.05, corrected for multiple comparisons). The table shows specific MNI coordinates of the activated regions (local maxima), T-values and voxels of each region.

Region	BA	T-value	Peak Coordinate			R/L	Voxels
			x	y	z
Funny 1 - Mix 1	No activated voxels
Funny 2 - Mix 2
Middle Temporal Gyrus	37	5.46	−48	−58	14	L	355
Fusiform Gyrus	37	7.27	−42	−56	−14	L	210
Funny 3 - Mix 3
Superior Medial Frontal Gyrus	10	4.42	−8	58	32	L	216
Superior Frontal Gyrus	8	5.72	−18	14	56	L	251
Supramarginal Gyrus	40	5.67	56	−48	28	R	519
Caudate	n/a	5.3	12	8	12	R	861
Funny 4 - Mix 4
Superior Frontal Gyrus	9	9.57	−14	52	38	L	2395
Middle Frontal Gyrus	9	4.21	−48	16	46	L	375
Inferior Frontal Gyrus	45	7.26	52	28	2	R	501
Inferior Orbitofrontal Gyrus	38	5.52	−50	22	−4	L	419
Middle Temporal Gyrus	39	8.49	−54	−58	24	L	502
Superior Temporal Sulcus	42	6.39	56	−42	16	R	1008
Thalamus	n/a	8.43	4	−16	4	R	1888
Cerebellum	n/a	5.28	18	−72	−30	R	368
Cerebellum	n/a	4.75	−24	−78	−40	L	344

To compare the neural basis for perceiving the degree of humor, we measured fMRI activations while participants read funny manga and non-funny manga (see Methods).

Behavioral data showed that the mean estimates of the four-score rating by the participants were 2.71 (SD = 0.30) and 2.22 (SD = 0.40) for the funny and non-funny manga, respectively. The difference between these two estimates was significant (t (17) = 7.23, p < 0.001). No differences in activation between each of the first three frames (introduction, development and turn) were found. However, there was a significant increase in activation in the brainstem for the funny manga when the fourth frame was presented (Table 2). Figure 3 shows rendered fMRI images comparing brain areas activated for the fourth frame during the funny and non-funny conditions (voxel-level threshold, p < 0.001, uncorrected for multiple comparisons; cluster-level threshold, p < 0.05, corrected for multiple comparisons). We also observed a large difference in the activation of the cerebellum of both hemispheres when comparing the funny and non-funny manga.

Table 2. Brain regions that were significantly more activated in response to funny than non-funny manga for each frame of the experiment 2 (voxel-level threshold: p < 0.001, uncorrected for multiple comparisons; cluster-level threshold: p < 0.05, corrected for multiple comparisons). The table shows specific MNI coordinates of the activated regions (local maxima), T-values and voxels of each region.

Region	BA	T-value	Peak Coordinate			R/L	Voxels
			x	y	z
Funny 1 - Non funny 1	No activated voxels
Funny 2 - Non funny 2	No activated voxels
Funny 3 - Non funny 3	No activated voxels
Funny 4 - Non funny 4
Cerebellum	n/a	6.64	−6	−60	−44	L	145
Cerebellum	n/a	6.25	18	−36	−40	R	271

Figure 3. Rendered fMRI images comparing brain areas activated for the fourth frame of funny and non-funny manga (experiment 2) (voxel-level threshold: p < 0.001, uncorrected for multiple comparisons; cluster-level threshold, p < 0.05, corrected for multiple comparisons).

To investigate more detail of the serial changes of the humor comprehension processes, we compared fMRI activations while participants read each frame of both the funny and non-funny manga. Because we did not find significant activation differences in the first frame between Funny and Mix (Figure 2 and Table 1) nor between funny and non-funny in the first frame (Table 2), we subtracted the fMRI contrast of the first frame from each of the other three frames. For funny manga, fMRI activations changed serially in accordance with the time course of the frames (Figure 4 (right side panels) and Table 3). We found greater left TPJ activation in the left hemisphere during viewing of the second frame. In the third frame, left TPJ activation was accompanied by activations in other regions including the STS and TP of both hemispheres and the TPJ of the right hemisphere. In the last frame (punch line), strong activation increases were seen in the TPJ, STS and IFG of both hemispheres. In addition, we found strong activation in the MPFC and cerebellum of both hemispheres. These serial changes of activations for funny manga were similar to the activation differences between Funny and Mix.

Figure 4. Rendered fMRI images of brain areas activated upon viewing the second, third and fourth frames relative to activation upon viewing the first frame (experiment 2).

Left side, rendered images of non-funny manga; right side, rendered images of funny manga (voxel-level threshold: p < 0.001, uncorrected for multiple comparisons; cluster-level threshold: p < 0.05, corrected for multiple comparisons).

Table 3. Brain regions that were significantly more activated in response to the second, third and fourth frames relative to the first frame of the experiment 2 (voxel-level threshold: p < 0.001, uncorrected for multiple comparisons; cluster-level threshold: p < 0.05, corrected for multiple comparisons). The table shows specific MNI coordinates of the activated regions (local maxima), T-values and voxels of each region.

Region	BA	T-value	Peak Coordinate			R/L	Voxels
			x	y	z
Funny 2 - Funny 1
Fusiform Gyrus	37	6.66	38	−50	2	R	399
Hippocampus	20	6.11	−38	−18	−14	L	500
Angular Gyrus	22	5.21	−52	−56	24	L	891
Funny 3 - Funny 1
Temporal Pole	21	7.60	52	8	−24	R	753
Angular Gyrus	39	6.57	−48	−52	26	L	446
Temporal Pole	38	6.57	−40	18	−24	L	463
Hippocampus	20	6.08	−42	−30	−8	L	253
Middle Temporal Gyrus	22	5.82	60	−62	22	R	1457
Funny 4 - Funny 1
Middle Temporal Gyrus	39	10.37	58	−68	20	R	6758
Temporal Pole	38	10.32	−50	14	−24	L	8414
Thalamus	n/a	9.32	6	−20	−12	R	2820
Superior Medial Frontal Gyrus	8	9.05	−6	28	56	L	2669
Cerebellum	n/a	6.92	−18	−78	−40	L	516
Caudate	n/a	6.67	12	6	12	R	191
Precuneus	n/a	6.60	−12	−50	44	L	233
Cerebellum	n/a	5.68	26	−74	−42	R	331
Non funny 2 - Non funny 1
Middle Temporal Gyrus	21	9.93	−62	−4	−10	L	4332
Inferior Frontal Gyrus	48	8.80	−54	22	10	L	383
Superior Temporal Gyrus	42	8.52	52	−44	18	R	3404
Precuneus	n/a	7.37	−6	−50	44	L	322
Cuneus	17	6.16	20	−100	10	R	230
Superior Medial Frontal Gyrus	9	6.13	−4	52	40	L	396
Non funny 3 - Non funny 1
Angular Gyrus	22	12.43	62	−56	24	R	3501
Middle Temporal Gyrus	21	7.94	−46	−54	14	L	1428
Superior Frontal Gyrus	9	7.23	−12	42	50	L	526
Middle Temporal Gyrus	21	7.02	−58	0	−18	L	1396
Middle Frontal Gyrus	6	6.49	−38	8	54	L	465
Inferior Frontal Gyrus	48	6.03	−50	18	12	L	588
Inferior Orbitofrontal Gyrus	47	5.21	44	30	−6	R	397
Non funny 4 - Non funny 1
Temporal Pole	38	14.26	48	16	−26	R	5982
Inferior Frontal Gyrus	48	10.35	−48	18	8	L	7478
Supplementary Motor Area	6	8.56	10	12	62	R	2687
Vermis	n/a	8.16	2	−52	−40	R	564
Hippocampus	35	7.39	−10	−14	−12	L	808
Middle Frontal Gyrus	6	7.26	−40	6	54	L	766
Cerebellum	n/a	6.81	−18	−74	−32	L	514
Precuneus	n/a	6.69	−10	−50	42	L	410
Cerebellum	n/a	6.61	28	−76	−38	R	471
Middle Frontal Gyrus	6	5.63	38	2	52	R	279

For the non-funny manga, however, we found the second and third frames elicited activations in the TPJ, STS and TP of both hemispheres in a manner that resembled the activations seen for the third frame of the funny manga. Moreover, increased activations were found in the IFG and medial frontal cortex (MFC) (Figure 4 (left side panels) and Table 3).

For further analysis, we also measured the interaction between the funniness and the serial order of each manga. The interactions were calculated by comparing the second, third and fourth frames with the first frame. A significant interaction was found between the funny manga and non funny manga for the fourth frame in the right cerebellum. We also found a significant interaction between the funniness and the serial order for the third frame in the middle temporal of the left hemisphere. However, the activation in the middle temporal was larger for the non funny manga than funny manga. Table 4 and Figure 5 show brain regions in which a significant interaction was found (voxel-level threshold: p < 0.001, uncorrected for multiple comparisons; cluster-level threshold: p < 0.05, corrected for multiple comparisons).

Table 4. Brain regions that were significantly more activated in response to the second, third and fourth frames relative to the first frame as indicated by interactions between funny and non funny conditions of the experiment 2 (voxel-level threshold: p < 0.001, uncorrected for multiple comparisons; cluster-level threshold: p < 0.05, corrected for multiple comparisons). The table shows specific MNI coordinates of the activated regions, T-values and voxels of each region.

Region	BA	T-value	Peak Coordinate			R/L	Voxels
			x	y	z
(Funny 2 - Funny 1) - (Non funny 2 – Non funny 1)	No activated voxels
(Funny 3 - Funny 1) - (Non funny 3 – Non funny 1)	No activated voxels
(Funny 4 - Funny 1) - (Non funny 4 – Non funny 1)
Cerebellum	n/a	6.44	10	−30	−16	R	177
Brain Stem	n/a	4.78	6	−20	−12	R
(Non funny 2 - Non funny 1) - (Funny 2 - Funny 1)	No activated voxels
(Non funny 3 - Non funny 1) - (Funny 3 - Funny 1)
Middle Temporal Gyrus	22	5.87	−52	−48	14	L	140
(Non funny 4 - Non funny 1) - (Funny 4 - Funny 1)	No activated voxels

Figure 5. Rendered fMRI images of brain areas that were significantly more activated in response to the third and fourth frames relative to activation in the first frame based on interactions between the funny and non-funny conditions (experiment 2) (voxel-level threshold: p < 0.001, uncorrected for multiple comparisons; cluster-level threshold, p < 0.05, corrected for multiple comparisons).

We extracted percent signal changes of the cerebellum (center coordinates: x, y, z = 6, −20, −12; sphere of radius 5 mm) of both the funny and non-funny conditions under the first and fourth frames. Figure 6 shows signal changes of the right cerebellum showing an interaction between the funniness (funny vs. non-funny) and frame (fourth vs. first). The signal changes of the cerebellum were greater in the fourth frames than the first frames across the funny and non-funny conditions. Moreover, the signal change was greater in the fourth funny frame than the fourth non-funny.

Figure 6. Signal changes of the right cerebellum showing an interaction between the funniness (funny vs. non-funny) and frame (fourth vs. first) in the experiment 2.

Signal change in the fourth frames was greater in the funny condition than in the non-funny condition, while such difference was not observed in the non-funny condition.

In addition, we measured the time course of the brain activation for both the funny and non-funny conditions. Region of interests (ROI) were created using contrast images that showed greater activation (p < 0.001) between the 4th frames of the funny and non-funny manga. The right cerebellum showed a different pattern between the funny and non-funny manga. A significant increase in activity in response to the fourth frame relative to the onsets of the first and second frames of the funny manga was seen, but not of the non-funny manga (Figure 7). Repeated ANOVA showed a significant main effect of the time-point (F (30, 510) = 6.63, p < 0.001) and a significant interaction between the time-point and funniness (F (30, 510) = 1.49, p < 0.05). The main effect of funniness was not significant (F (1, 17) = 2.17, p > 0.05). Post-hoc multiple comparisons showed significant differences at 36.0, 37.5, 39.0, 43.5, 45.0, and 46.5 seconds compared with 1.5 and 10.5 seconds in the funny condition. In the non-funny condition, on the other hand, a significant difference was found only between 45.0 and 22.5 seconds. Finally, a clear signal increase was seen in the fourth frame compared with first and second frames in the funny condition. Signal changes did not differ between the funny and non-funny conditions in the fourth frame, when applying the multiple comparisons. Nevertheless, one-sample t-tests showed significant differences between the funny and non-funny condition at the time-point of 34.5, 36.0, and 37.5 second which corresponded to the fourth frame (p < 0.05, uncorrected for multiple comparison).

Figure 7. Time-course signal change of the right cerebellum while viewing funny and non-funny manga.

The ROI was created based on the contrast images that showed greater activation in the 4th funny frame and that in the 4th non-funny frame (experiment 2) (p < 0.001).

Discussion

In the present study we investigated the serial changes of humor comprehension using four-frame manga. The results argue that humor appreciation changes as the manga scene changes serially from its first to fourth (when the humor emerges) frame.

Comparing Funny and Mix manga, we found TPJ and FFG activation in the left hemisphere during viewing of the second frame, which was when the story develops and cues the participants to expect a change in the story. Upon viewing of the third scene, when the ambiguity of the manga began to be resolved, other regions, including the TPJ and STS of the right hemisphere and bilateral MPFC, showed increased activations. In the last scene, when the manga was fully resolved, the TPJ and STS in both hemispheres were strongly activated with additional activation in frontal regions, such as the MPFC and VLPFC.

Because the TPJ was found activated when the change in the story of manga began (second frame), the TPJ may play some role in the early stages of humor comprehension. As the story developed, the activation of the TPJ expanded to wider regions such as the STS and TP.

A wide area around the TPJ, TP and IFG is assumed to be involved in cognitive humor processing, and the TPJ is involved in incongruity resolution¹²,¹³. Additionally, it has been reported that the TPJ relates to the mental state while performing reasoning tasks¹⁴,¹⁵. In addition, both the STS and TP are thought to be key regions for mentalizing circuits in accordance with the MPFC¹⁶,¹⁷. Following these reports, the activation of the TPJ in the present study is thought to correspond to the attribution of intention to the characters in a manga while participants initiate the early stage of manga comprehension. Moreover, considering that activation of the middle temporal regions relates to episodic memory, our results also suggest participants transferred the contents of each manga frame into episodic memory successively¹⁸.

In the last scene, when the manga was fully resolved, frontal regions, such as the MPFC and VLPFC, were also strongly activated, as too was the cerebellum in both hemispheres. The activation of the MPFC found at the fourth frame is consistent with this region playing a role in understanding the punch line and comprehending the humor, as suggested previously⁴. Stronger activation in the anterior MPFC as well as the TPJ and bilateral SFG was also reported in participants processing incongruity-resolution cartoons than when processing nonsense cartoons¹³.

It is interesting that previous research has also indicated that the MPFC plays a role in understanding theory of mind (TOM). Gallagher et al.¹⁴ used fMRI to compare brain activations while participants observed TOM cartoons that required an attribution of a mental state (ignorance or false belief) and non-TOM cartoons. They found increased brain activity in the MPFC bilaterally, the right MFG and the TPJ. Using positron emission tomography, Brunet et al.¹⁹ examined neural substrates involved in attributing intentions to others. For the condition in which participants inferred the intentions of the characters, the authors found increases in regional cerebral blood flow in the right MFG and IFG. They also found increases in right temporal regions such as the ITG, STG, MTG and left cerebellum.

Following these reports, the activation of the MPFC in the present study is thought to correspond to the attribution of intention while participants read the manga. It is interesting that the activation of the MPFC appears in the third frame, a frame in which the attribution of intension had not resolved. Moreover, in the third frame, the punch line of the manga is still unclear. Chan et al.¹⁰ suggested that the vMPFC is associated with humor elaboration and not humor comprehension. In the present results, however, the activation of the MPFC seems to be associated with not only the punch line but also the third frame of humor comprehension (see Figure 2).

Juxtaposing funny and non-funny manga confirmed the increased activation of the TPJ frame between the first and second frames. The juxtaposition also argues that humor appreciation changes as the scene changes serially from its first to fourth frame. For funny manga, we found TPJ activation in the left hemisphere in the second frame, which cued the participants to expect a change in the story. Upon viewing of the third scene, when the ambiguity of the manga began to be resolved, other regions, including the TPJ of the right hemisphere and the STS and TP of both hemispheres, showed increased activations. However, in the non-funny manga, both the second and third frames elicited activations in the TPJ, STS and TP of both hemispheres, a result similar to the activation seen in response to the third frame of the funny manga.

In the funny manga (Figure 1, right), the readers easily expected the story development. For example, in the second frame, they understand the school master will climb the roof to remove the snow, and in the third frame they expect the snow to fall from the roof. The punch line in the fourth frame is that it is not the snow but the school master who falls, contrary to their expectation. Thus, the story development is easy to expect and understandable.

On the contrary, for the non-funny manga, new information is added to the story after the development of the first frame. For example, in Figure 1, left, the new information is about the wait for the roller coaster. Therefore, the viewer does not expect the next scene, confusing the resolution of the incongruity between the expectation and the punch line. Thus, the problem-solving task in which participants transform nonsense into humorous sense is difficult when reading non-funny manga. Following these results, it is suggested that humor comprehension is raised by the strong expectation of the reader.

In the last frame, activation of the IFG was found in both hemispheres. It has been suggested that prefrontal IFG regions maintain attention for achieving task goals of the working memory system²⁰. The experiments presented here required participants hold the first, second and third manga scenes in order to evaluate the humor of the final frame. This task requires working memory, particularly executive function, which is thought to consist of the prefrontal cortex including the IFG²¹,²²,²³.

In the non-funny manga only, activation of the IFG was found in the second and third frames. This result may be because the reader had to memorize more details of the frame as compared with the funny manga, which made it difficult to expect the final scene of the manga. In Chan et al.⁸, IFG activation increased for humor comprehension that required a semantic selection. Thus the activation increase during the second and third frames of non-funny manga in our study may be by the semantic selection, which would reduce the expectation of the punch line in funny manga.

When comparing brain areas activated during the viewing of funny and non-funny manga, we did not find differences in activation for the first, second, or third frames. However, we found a significant increase in activation of the cerebellum in both hemispheres for the funny manga in the fourth frame. Activation of the left cerebellum in response to humor has been reported before¹¹,²⁴. Goel & Dolan³ reported that affective components of humor are related with activation in the ventral MPFC and bilateral cerebellum. Bartro et al.¹¹ also found that the degree of amusement is associated with activation of the bilateral cerebellum. Parvizi et al.²⁴ found by observing a patient with pathological laughter that the cerebellum plays a role in modulating and coordinating the production of laughter. Following these reports, it would seem the cerebellum activation reflects the amusement effect of the humor comprehension. The anatomical work of Middleton & Strick²⁵ has provided evidence that the cerebellum has reciprocal connections with other cerebral areas including the prefrontal cortex via the thalamus.

In the present study, along with the activation in the cerebellum, activations were found in the brainstem for the last scene of the funny manga. In addition, we found strong activation in the Funny condition not only in the cerebellum, but also in the MPFC, brainstem and thalamus for the fourth frame compared with Mix. Therefore, the resolution of incongruity of the punch line in the fourth frame may be brought from the MPFG to the cerebellum via the brainstem and thalamus. Brunet et al.¹⁹ found increases in the left cerebellum when examining neural substrates involved in attributing intentions to others, which suggests the cerebellum activation reflects both the resolution of incongruity of the punch line and attributing intention of the manga. Moreover, because the activation of the cerebellum was larger for the funny manga, we propose that the resolution of incongruity between the expectation and punch line constitutes attributing intention for the participants. In turn, the attributing intention could then strengthen humor comprehension and increase activation of the cerebellum by the corresponding laughter. This conclusion is supported by meta analysis that showed the cerebellum is critical in social cognition²⁶.

Along with finding activation in the left vMPFC and subcortical regions such as the parahippocampal gyrus, Chan et al.⁸, found activation in the bilateral amygdala when comparing stories whose punch lines were marked by unfunny, funny or garden path sentences, and attributed the feeling of amusement caused by the humor to humor elaboration stages. We, on the other hand, could not find activation in the amygdala. The different results may be because we used manga with captions as funny stimuli, whereas they used sentences. Manga with captions are easier to read than sentences, which shortens the duration of the elaboration humor stage and may thus prevent activation of the amygdala. Another explanation may be that in the present study, participants easily attributed intention to the characters, because the characters of the manga showed facial and bodily expressions. Thus, strong activation was seen in the MPFC and cerebellum, but little in the amygdala. Another factor that may explain why we failed to see activation of the amygdala was the smaller difference in the ratings of funny and non-funny manga (see Methods). Further research is required to identify the precise reason.

To summarize, the present study indicates that activation of the left TPJ initiates the early stages of the comprehension process of the humor story line. This activation then reaches the STS and MPFC, which produce humor appreciation for the punch line, and the cerebellum. Thus, expectation before presentation of the punch line plays an important role for the appreciation of humor.

Methods

Participants

In experiment 1, Funny and Mix, 16 healthy undergraduate and graduate students (9 males and 7 females; age range = 21–26 years, mean = 223.56, SD = 1.97) were enrolled and paid to participate. One male participant was removed from the data analysis due to large number of no response trials (30/48 trials). In experiment 2, funny and non-funny manga, 18 healthy undergraduate and graduate students (14 males and 4 females; age range = 20–30 years, mean = 23.83, SD = 0.89) were enrolled and paid to participate. No participants did both experiments, and all gave informed consent in accordance with a protocol approved by the ATR Brain Imaging Center Review Board and the ATR Brain Imaging Center Review Board.

Manga stimuli

In a preliminary investigation, 219 four-scene manga with captions from Japanese newspapers were selected. Ten undergraduate students were asked to read these manga and judge whether they were funny or not using a 7-point emotional rating scale, where 7 was funniest and 1 was least funny. The mean manga score was 3.78 (SD = 0.63). Using this evaluation, 48 manga were selected and divided into two groups: 24 funny manga (score 4.40 to 5.38, mean = 4.59, SD = 0.36) and 24 non-funny manga (score 2.20 to 3.20, mean = 2.50, SD = 0.20).

fMRI task

To investigate brain activities while participants read four-scene manga, fMRI activations were serially measured using a rapid event-related design. Each frame was presented for 3000 ms with a jittered inter-stimulus interval (ISI) of 4500, 6000 or 7500 ms between frames. After participants read the fourth frame, an estimation requirement frame, which showed a four-score rating scheme with 4 being most funny and 1 being least funny, was presented for 3000 ms. During this time, the participants were asked to assign the previously viewed manga a score. The inter trial interval of the manga presentation was also 4,500, 6000 or 7,500 ms. One trial took about 43–50 s. A whole experiment took about 25 min, with 3 min resting time in the middle.

In experiment 1, 24 funny manga (Funny) and 24 mixed ordered manga (Mix) were presented in random order. The mixed ordered manga were composed of the 24 non funny manga and randomized in their presentation order. We measured fMRI activations for each frame and normalized data for the first, second, third, and fourth frames. fMRI activations for each frame between Funny and Mix manga were compared within participants.

In experiment 2, 24 funny and 24 non-funny manga were presented, and fMRI activation for each frame was measured. fMRI data for the second, third, and fourth frames were normalized by subtracting data from the first frame. fMRI activations for each frame between funny and non-funny manga were compared within participants. For these comparisons, fMRI data differences were taken for each frame.

fMRI measurements and analysis

Whole brain imaging data were acquired using a 3.0-T MRI scanner (MAGNETOM Trio, Siemens, Munich, Germany) and a standard head coil. Head movements were minimized using a forehead strap. For functional imaging, a gradient-echo echo-planar imaging sequence was used with the following parameters: repetition time (TR) = 3000 ms, echo time (TE) = 30 ms, flip angle = 80 degree, field of view (FOV) = 192 mm × 192 mm and voxel size = 3 × 3 × 3 mm.

After the image collection, T1-weighted anatomical images (TR = 2250 ms, TE = 3 ms, flip angle = 9, voxel size = 1 × 1 × 1 mm) were collected for all participants. Stimulus presentations were synchronized with an fMRI pulse using Presentation (Neurobehavioral System, San Francisco, CA).

Data analysis was performed using SPM5 (Wellcome Department of Cognitive Neurology, London, UK) running on Matlab 7.3 (MathWorks, Sherborn, MA). Analysis of the fMRI data was initially performed for each participant during the funny and non-funny manga sessions. The six initial images of each scanning session were excluded from the analysis in order to eliminate any non-equilibrium effects of the magnetization. Thus, a total of 360 images were used in the analysis.

All functional images were realigned to correct for head movements, which were less than 4.0 mm within runs. Functional images were normalized and spatially smoothed with an isotropic Gaussian filter (8 mm full-width at half-maximum). Low-frequency noise was removed with high-pass filtering (128 s). Events were modeled with a gamma hemodynamic response function (HRF). The autoregressive AR (1) model was applied to account for serial correlations when running first-level GLM.

Single participant data were analyzed using a fixed-effects model. Group data from the four-scene manga sessions were analyzed using a random-effects model. We used the statistical threshold of p < 0.001 (uncorrected for multiple comparisons) for the voxels level and p < 0.05 (FWE corrected for multiple comparisons) for the cluster level.

Time-course data were extracted by MarsBaR²⁷. For normalization, the signal at the first time point of each trial was set as baseline, and the proportion of signal change at the following time-points was computed. The obtained data were analyzed with a two-way repeated ANOVA, which had factors of the time-point (31 levels) and funniness (2 levels). Bonferroni correction was employed for multiple comparisons. The alpha level was set to 0.05. Statistical analysis was performed with STATISTICA (StatSoft, Inc.).