Neural Basis of Individual Differences in Synesthetic Experiences

Abstract

Little is known about how the properties of our private mental world relate to the physical and functional properties of our brain. Studying synesthesia, where a particular experience evokes a separate additional sensory experience, offers the unique opportunity to study phenomenological experiences as a stable trait in healthy subjects. A common form of synesthesia is grapheme–color synesthesia, where a particular letter or number evokes a particular color experience. We studied the neural basis of qualitative different properties of the synesthetic experience by using individual differences in grapheme–color synesthesia. Specifically, the synesthetic color can be experienced “in the mind” (associator synesthetes) or “in the outside world” (projector synesthetes). Gray matter structure and functioning (imaged using voxel-based morphometry and functional magnetic resonance imaging, respectively) were examined in grapheme–color synesthetes (N = 42, 16 projectors and 26 associators) and nonsynesthetes. Results indicated partly shared mechanisms for all grapheme–color synesthetes, particularly in posterior superior parietal lobe, which is involved in the integration of sensory information. In addition, the nature of synesthetic experience was found to be mediated by distinct neural mechanisms. The outside-world experience is related to brain areas involved in perceiving and acting in the outside world (visual cortex, auditory cortex, motor cortex) as well as frontal brain areas. In contrast, the in-the-mind experience is related to the hippocampus and parahippocampal gyrus, known for their role in memory. Thus, the different subjective experiences are related to distinct neural mechanisms. Moreover, the properties of subjective experiences are in accordance with functional properties of the mediating brain mechanisms.

Introduction

While it is generally accepted that a perceptual experience is mediated by neurological mechanisms, we know little about how particular properties of a subjective experience relate to particular structural and functional brain properties. Synesthesia offers a unique opportunity to study the neural basis of subjective experiences in healthy brains. In synesthesia, particular sensations (e.g., seeing the letter “A”) evoke particular additional sensory experiences (e.g., the color red) (Galton, 1883). Synesthesia is a “real” phenomenon, and is not related to any psychological, neurological, or psychiatric disease (Baron-Cohen et al., 1987; Rich et al., 2005). Familial studies and a genetic association study support a genetic predisposition for synesthesia (Baron-Cohen et al., 1996; Simner et al., 2006; Asher et al., 2009). Synesthetes, compared with nonsynesthetes, show structural brain differences in white matter (WM) and gray matter (GM) properties (Rouw and Scholte, 2007; Hänggi et al., 2008; Jäncke et al., 2009; Weiss and Fink, 2009). These structural differences are obtained in modality-specific regions as well as in other brain areas. At present, both the question of how structural or functional brain differences underlie synesthesia (Bargary and Mitchell, 2008; Cohen Kadosh and Walsh, 2008) and the question of which particular brain areas are crucial (e.g., Nunn et al., 2002; Steven et al., 2006; Rich et al., 2006) remain topics of debate.

In a previous study, we measured structural properties of white matter tracts with diffusion tensor imaging (DTI) (Basser et al., 1994; Rouw and Scholte, 2007). Increased connectivity near the fusiform gyrus (involved in recognizing visual categories) was found related to the nature of synesthetic experience. The increase was stronger in synesthetes who experience the synesthetic color “in the outside world” (projectors) than in synesthetes experiencing their synesthetic color “in the mind only” (associators) (Dixon et al., 2004). Hubbard et al. (2005) found that increased task performance due to synesthesia was correlated with increased activation in early visual areas. Thus, individual differences might underlie different involvement of modality-specific cortex in synesthesia.

We compared structural [with voxel-based morphometry (VBM)] and functional [with blood oxygen level-dependent magnetic resonance imaging (BOLD-MRI)] gray matter properties of grapheme–color synesthetes with those of nonsynesthetes. Moreover, projector synesthetes were contrasted with associator synesthetes. We hypothesized that both shared mechanisms [parietal processes crucial to generating the synesthetic experience (Esterman et al., 2006; Muggleton et al., 2007)] and distinct mechanisms underlie these two subtypes of grapheme–color synesthesia. On the one hand, synesthesia has been found related to modality-specific brain areas (Nunn et al., 2002; Hubbard et al., 2005). Such areas are normally involved in representing information present in the outside world. We expect that stronger dependence on these areas mediates a similar experience, i.e., in the outside world. On the other hand, brain areas involved in further processing, reactivating, and combining information can also mediate synesthesia. Relative dependence on amodal mechanisms is reflected in the in-the-mind experience. These findings are of interest not only to the study of synesthesia, but also to two of the most fundamental issues in cognitive neuroscience: the mechanisms behind cross-sensory integration and the physiological basis of subjective mental experiences.

Materials and Methods

Subjects.

Subjects were 42 females with grapheme–color synesthesia (mean age, 27 years; range, 18–52 years) and 42 controls matched on age, sex, and level of education. From 35 synesthetes in this group, we retrieved further information on what induced the synesthetic color. The great majority of both the associators (17 of 22; 77%) and the projectors (8 of 13; 62%) indicated that, although one can be stronger than the other, both hearing and seeing a grapheme can induce the color experience [there is a slight difference in the percentage of synesthetes indicating that only seeing the letter elicits the color; 4 of 22 (18%) of the associators and 5 of 13 (38%) of the projectors].

Furthermore, synesthetes indicating that only seeing evokes a color experience report that hearing a word or letter can automatically be translated into seeing this grapheme (in the mind's eye), and then into the concurrent color sensation. The exact role of auditory codes and auditory cortex in grapheme–color synesthesia is not yet clear. The current findings are in line with the suggestion that in grapheme–color synesthesia, auditory information can play a role, e.g., if translated into graphemic representations that subsequently influence the color in grapheme–color synesthesia (Simner, 2007). Synesthetes were invited to participate in the study based on the consistency of their synesthetic grapheme–color reports (an unexpected retest that took place at least 3 weeks after the initial test). Our study and inclusion criteria were aimed at grapheme–color synesthesia, but in any sample of synesthetes it is likely that other kinds of synesthesia are present as well. In particular, number–form is found to be very common among grapheme–color synesthetes (Sagiv et al., 2006). We retrieved information about other kinds of synesthesia in 36 of our subjects. A minority (N = 15) of our subjects had number–form. A few synesthetes (N = 6) also reported some type of synesthesia quite different from grapheme–color or number–form. Importantly, in the projector–associator distinction, this factor was balanced out, as the two subject groups had the same proportion of other types of synesthesia (e.g., 9 of 21 of the associators and 6 of 15 of the projectors had number–form).

Synesthetes also filled out the projector–associator (PA) questionnaire (Rouw and Scholte, 2007), which revealed that 26 of the synesthetes could be considered associators (PA value <0) and 16 of the synesthetes could be considered projectors (PA value >0). Projectors did not differ from associators on characteristics such as age (projector: mean age, 26.6 years; SD, 7.3; associator: mean age, 27.7 years; SD, 8.1), level of education (almost all our subjects were studying or had studied at college or university level), or handedness (90% of our synesthetes were right-handed). Eighteen synesthetes were the same subjects (and functional MRI and MRI measurements) reported previously (Rouw and Scholte, 2007). Unfortunately, due to technical problems we could not analyze the WM structural data in the current subject group.

Procedure.

Magnetic resonance images were acquired using a 3-T scanner (Philips). MRI and BOLD-MRI measurements were done in a single scanning session. The subject's head was immobilized using foam pads to reduce motion artifacts and earplugs were used to moderate scanner noise.

Voxel-based morphometry.

We acquired a structural scan for each of the subjects [three-dimensional T1 turbo field echo; echo time (TE), 4.6 ms; repetition time (TR), 9.6 ms; fractional anisotropy (FA), 8°; 182 sagittal slices of 1.2 mm; field of view (FOV), 250² mm; reconstruction matrix, 256²]. Data were analyzed with FSL-VBM, a voxel-based morphometry style analysis (Good et al., 2001) performed with FSL (Smith et al., 2004). First, structural images were brain extracted (Smith, 2002). Next, tissue type segmentation was performed using FAST4 (Zhang et al., 2001). The resulting gray matter partial volume images were then aligned to MNI152 standard space, using the affine registration. The resulting images were averaged to create a study-specific template, to which the native GM images were then nonlinearly reregistered with a method that uses a B-spline representation of the registration warp field (Rueckert et al., 1999; Andersson et al., 2007). The creation of a study-specific template makes it possible to compare the different synesthetes and controls with each other in a balanced way while keeping the average deformation similar between the groups.

The registered partial volume images were then modulated (to correct for local expansion or contraction) by dividing by the Jacobian of the warp field. The modulated segmented images were then smoothed with an isotropic Gaussian kernel with a σ of 4 mm. Finally, a voxelwise general linear model was applied using permutation-based nonparametric testing.

We thresholded the data by taking only those clusters that consisted of 200 continuous (1600 mm³) voxels that had a p value of 0.05 or lower. We defined controls, projector synesthetes, and associator synesthetes as explanatory variables. We also defined two additional nuisance variables to model any differences between scans that were made before and after a hardware upgrade to the MRI scanner (these did not correlate with group membership). We calculated two different models (see next section and Results).

GM structure and GM activation.

We performed the two different models for two reasons. First, we do not know the nature of the relation between the GM BOLD/VBM values and the PA questionnaire values. To avoid biased results, we included both a model optimal for testing categorical differences (F test over contrasts) and a model optimal for finding ordinal relationships (a covariate analysis). Both models were evaluated with permutation testing. Second, controls were included in the first but not in the second model. The first model allowed us to examine which brain areas differentiate between synesthetes and nonsynesthetes. The second model allowed us to examine which brain areas distinguish between projectors and associators, regardless of controls.

Note that differences between associators and projectors are relative differences. When we present increased GM or increased BOLD for projector compared with associator in a particular brain area, this is the same conclusion as discussing decreased GM or decreased BOLD for associator compared with projector in that particular area.

To avoid type I errors, we also performed a post hoc analysis. We tested for differences between synesthetes and controls in a number of areas reported in previous VBM studies (Jäncke et al., 2009; Weiss and Fink, 2009). We therefore performed a post hoc analysis on the left and right anterior intraparietal sulcus (both hIP1 and hIP2) from the Juelich Histological Atlas (Choi et al., 2006) and bilaterally in the fusiform gyrus; occipital fusiform gyrus, temporal occipital fusiform cortex, temporal fusiform cortex (anterior division), and temporal fusiform cortex (posterior division) from the Harvard-Oxford cortical structural atlas. Both atlases are probabilistic and the areas were therefore thresholded at the 25th percentile [the actual regions of interest (ROIs) used contained those voxels present in 25% or more of the population for that area]. These ROIs were used to extract the average GM value per subject, which were subsequently tested for differences between controls and synesthetes and between associators and projectors and for correlations between the PA values and gray matter. We controlled for false-positives using false discovery rate thresholding at a level of p = 0.05, as determined over all these ROIs.

BOLD-MRI.

We measured, with 61 subjects (the 42 synesthetes tested with VBM and 19 controls matched on age and level of education to the first 19 synesthetes tested), BOLD-MRI on a grapheme perception task. Each synesthete was presented with a personalized set of stimuli: eight graphemes (digits, letters, or symbols) that elicited a strong synesthetic color, eight graphemes that elicited a weak synesthetic color, and eight graphemes that had no synesthetic color for that particular synesthetic subject. The graphemes used for the 19 controls were matched to the graphemes used for the first 19 synesthetes. Each stimulus was presented for 500 ms and was followed by a gray isoluminant screen that lasted between 2500 and 9000 ms. The stimuli were presented in randomized order four times per category over two runs (one run lasted for ∼5.5 min). Graphemes subtended a visual angle of 2°. Subjects were asked to press with their index finger a key on a response box if a letter/digit/symbol was presented in italics and with their middle finger if the stimulus was in normal font. The BOLD signal was measured while the paradigm was presented, with a T2* gradient echo-planar imaging sequence (TR, 2.3 s; TE, 28 ms; 35 slices; slice thickness, 3.3 mm; FOV, 220 × 220; in-plane resolution, 96 × 96; duration, 333 s), and the paradigm was presented twice.

Stimuli were projected on a screen at the front end of the scanner table (Philips 3T Intera). The projected image was seen via a mirror placed above the subject's head. A magnet-compatible response box was used to record the subject's response.

BOLD-MRI data were analyzed with FSL (Smith et al., 2004) and Matlab (Mathworks). The functional images were slice time aligned, motion corrected, high-pass filtered (0.01 Hz), and spatially smoothed with a kernel of 5 mm. After this, the functional images were aligned to the structural image acquired at the start of each scanning session (the same as used for the VBM analysis) and transformed, nonlinearly on the basis of this structural image, to MNI space (Rueckert et al., 1999; Andersson et al., 2007).

We modeled the presentations of three different types of stimuli (strong, weak, and no synesthetic color) and their first-order derivatives. We contrasted viewed graphemes that elicit synesthetic color (strong and weak condition) with viewed graphemes (symbols) that do not elicit a synesthetic color experience. We first performed an F test over the between-group differences (projector synesthetes vs controls and associator synesthetes vs controls), as in the VBM analysis. However, this yielded only one significant region of interest around the parieto-occipital sulcus (similar to the region reported in Table 3 below). We therefore subsequently calculated four t tests on the between-subject contrasts of synesthetes versus controls and associators versus projectors. Note that these analyses are based partially on the same data (18 synesthetes and 18 controls) as that in a previous study (Rouw and Scholte, 2007). The nuisance variable that we defined in the VBM analysis was not used in the BOLD-MRI analysis because there were no significant differences between synesthetes' measures before and after the scanner upgrade. Time series statistical analysis was carried out, at the within-subject level, using a fixed-effects model and the improved linear model of the Functional MRI of the Brain (FMRIB) Centre (University of Oxford, Oxford, UK) for local autocorrelation correction (Beckmann et al., 2003). Higher-level analysis was performed using the FMRIB Centre local analysis of mixed effect stage 1 and 2, a mixed-effects model (Woolrich et al., 2004, 2008). Z-statistic images were thresholded using clusters determined by z >2.3 and a (corrected) cluster significance threshold of p = 0.05 (Worsley, 2001).

Table 3.

BOLD-MRI clusters resulting from t test of between-subject contrasts of synesthetes versus controls and associators versus projectors

	x, y, z	Max t	mm3
PROJ, ASSOC > CON
Intraparietal sulcus	−30, −72, 28	3.85	3280
Inferior frontal gyrus and precentral gyrus	−40, 5, 27	6	3872
Parieto-occipital sulcus	−3, −75, 30	3.75	4656
ASSOC, CON > PROJ
Left parahippocampal and temporal fusiform gyrus	−25, −29, −24	4.36	2584
Right parahippocampal and temporal fusiform gyrus	23, −23, −23	5	5112

No significant differences were found in controls > synesthetes and projector > associators. Reported are center of gravity, max t, and size of the clusters. PROJ, Projector synesthetes; ASSOC, associator synesthetes; CON, nonsynesthetes.

Results

Differentiating projectors from associators

To establish the nature of synesthetic experience, synesthetes filled in the projector–associator questionnaire (Rouw and Scholte, 2007). On this questionnaire, synesthetes indicated their experiences on five-point Likert Scale projector-type questions (e.g., “the color seems to be projected on the letter/digit”) and associator-type questions (e.g., “I see the color of a letter/digit only in my head”). One question was removed from the analysis in the previous study, as it had a low correlation with the total projector score, and was replaced with a new question. For each subject, a PA score was calculated by subtracting the mean score on six associator questions from the mean score on six projector questions. Overall, the PA scores had a mean of −0.08 (SD = 2.35) but ranged from −3.10 to 4.00 (maximum range, −4–4). The distribution of PA scores is not normal (one sample Kolmogorov–Smirnov Z = 1.43, p = 0.03). Instead, it shows a bimodal distribution (Fig. 1). The bimodal distribution suggests that projector and associator synesthetes are two different groups rather than extremes from one distribution, but the exact nature of the projector–associator distinction is a topic for future research. For the current study, it is necessary and sufficient that the PA questionnaire allows the degree to which an individual synesthete is a projector or associator synesthete to be discerned.

Figure 1.

Histogram of scores on the projector–associator questionnaire in 42 grapheme–color synesthetes.

Shared mechanisms: grapheme–color synesthetes compared with nonsynesthetes

Gray matter volume and density are measured with voxel-based morphometry. We first addressed the question whether certain brain areas differ between synesthetes and nonsynesthetes, independent of the projector–associator distinction. To test this in a stricter manner, we first discerned all brain areas that differentiate between either one or both types of the synesthetes (compared with the controls). Next, we tested which (if any) of these brain areas do not discriminate between projector and associator synesthetes. For this, we defined two contrasts. The first contrast was between the associator synesthetes and the control subjects and the second was between the projector synesthetes and the control subjects. We then calculated the F values of the combined contrast, irrespective of the direction of the differences. In this analysis, six clusters were obtained. These clusters could thus be based on control subjects differing from (1) both projectors and associators, (2) projectors only, or (3) associators only. We next examined, for each cluster, which of these three possibilities applied (Table 1) by performing three independent-sample t tests (with Levene's test for equality of variances): associator versus controls, projectors versus controls, and associators versus projectors. Furthermore, for each cluster we examined whether GM values for a synesthete correlated with the PA questionnaire score. On the basis of these analyses, we classified two of these clusters as due to differences in GM in projectors, compared with associators and controls (increased GM in right Heschl's gyrus extending in insular and parietal operculum, and decreased GM in left angular gyrus). Two of these clusters showed increased GM in associators, compared with projectors and controls (located in right hippocampus and in the cerebellum). Finally, we found two brain areas that discriminate between synesthetes and nonsynesthetes, regardless of the projector/associator type.

Table 1.

Regions showing GM differences between projector synesthetes, associator synesthetes, and nonsynesthetes obtained with an F test analysis

	x, y, z	F	mm3	PROJ–CON	ASSOC–CON	ASSOC–PROJ	Correlation with PA
PROJ, ASSOC > CON
Left superior parietal cortex	−11, −58, 61	19.8	2944	t(1, 56) = 3.526, p = 0.001	t(1, 66) = 3.459, p = 0.001	t < 1	rs(42) = 0.095, p = 0.548
CON > PROJ, ASSOC
Cingulate sulcus	0, 39, 21	21.01	9344	t(1, 56) = −2.747, p = 0.008	t(1, 66) = −3.877, p = 0.000	t <1	rs(42) = 0.076, p = 0.633
ASSOC > PROJ, CON
Cerebellum, extending into occipital cortex	6, −69, −19	18.42	14832	t(1, 56) = 0.151, p = 0.880	t(1, 66) = 3.726, p = 0.000	t(1, 40) = −3.519, p = 0.001	rs(42) = −0.480, p = 0.001
Right hippocampus, extending into thalamus	17, −30, −1	10.67	1632	t(1, 56) = −1.180, p = 0.243	t(1, 66) = 2.480, p = 0.016	t(1, 40) = −2.885, p = 0.006	rs(42) = −0.314, p = 0.043
PROJ > CON (ASSOC)
Right Heschl's gyrus extending into insular and parietal operculum	40, −19, 13	12.56	4664	t(1, 56) = 3.220, p = 0.002	t(1, 66) = 1.322, p = 0.191	t(1, 40) = 1.914, p = 0.063	rs(42) = 0.369, p = 0.016
PROJ < CON, ASSOC
Left angular gyrus	−50, −64, 30	10.79	1864	t(1, 42.44) = −3.617, p = 0.001	t(1, 66) = 0.112, p = 0.911	t(1, 39.99) = −3.197, p = 0.003	rs(42) = −0.354, p = 0.021

Center of gravity coordinates, F value, and size are reported. The regions are categorized based on subsequent tests showing which group differences underlie this cluster. These subsequent tests are three between-group t tests, and the correlation between mean GM value and PA score (note that these analyses are performed only to categorize the different regions, not to determine their significance). PROJ, Projector synesthetes; ASSOC, associator synesthetes; CON, nonsynesthetes.

The first area showed increased GM values in synesthetes compared with controls in left superior parietal cortex (Fig. 2). As can be seen in Table 1, the mean GM values of projectors and associators differ significantly from that of the controls. But, no differences were found between subtypes (projector and associator) of synesthesia. Previous studies have reported increased BOLD-MRI activation in parietal cortex during synesthesia (Paulesu et al., 1995; Nunn et al., 2002; Steven et al., 2006). Furthermore, interference related to synesthetic experience has been found to diminish after applying transcranial magnetic stimulation (TMS) to the superior parietal cortex (Esterman et al., 2006; Muggleton et al., 2007). The role of the parietal lobe in synesthesia has been related to its function in cross-sensory integration or “binding,” multimodal associations, and multimodal integratory attention (e.g., Muggleton et al., 2007; Robertson, 2003). In a previous DTI study, we found differences between synesthetes and nonsynesthetes in local WM tract structure near the left superior parietal cortex (Rouw and Scholte, 2007). The measurement for directionality of white matter at this location (mean FA value) did not correlate with PA scores, indicating that both projectors and associators show increased WM structure at this location. The current study, which included subjects from that previous study, showed increased gray matter exactly superior in location to this increased white matter (Fig. 2). Furthermore, in the current study, increased GM value in this region does not correlate with PA value (r_s(42) = 0.095, p = 0.548).

Figure 2.

Two clusters from the F test analysis of between-group differences in GM. Depicted are clusters in left superior parietal cortex and in cingulate gyrus. Subsequent testing revealed that the first cluster is based on increased GM in synesthetes compared with nonsynesthetes, and the second cluster is based on decreased GM in synesthetes compared with nonsynesthetes.

In the second cluster, differences are in the opposite direction: controls showed increased gray matter values compared with synesthetes in the GM around the medial region of the cingulate sulcus (Fig. 2), extending into cingulate and paracingulate gyrus.

The post hoc ROI analysis showed two marginal results. Increased GM values in synesthetes compared with nonsynesthetes in left anterior intraparietal sulcus hIP2 (t test with equal variances not assumed) were as follows: t_(73,15) = 1.97, p = 0.053. Increased GM values in nonsynesthetes compared with synesthetes in left temporal fusiform cortex (posterior division) were as follows: t₍₈₂₎ = −2.027, p = 0.046. None of these p values survived multiple-comparison correction using the false-discovery rate.

Distinct mechanisms: projector versus associator synesthetes

Next, we addressed the question of which brain areas differentiate between projector and associator synesthetes. Again, we contrasted gray matter volume and density values, measured with voxel-based morphometry. However, in this analysis we contrasted only projectors with associators. By including only synesthetes, the analysis is not strengthened (nor weakened) by the contrast between synesthetes and nonsynesthetes. Instead, the question at hand is what sets projectors apart from associators.

We performed a covariate analysis on the VBM data based on PA value. This allowed us to use not only the categorical differences (projector vs associator), but also the relative strength (how extreme the score is) on the PA questionnaire. Results of this analysis are presented in Table 2. We found six regions where projectors showed increased GM values compared with the associators and six regions where associators showed increased GM values compared with the projectors.

Table 2.

Results from a covariate analysis between PA score and VBM values, including only synesthete subjects

	x, y, z	Max t	mm3
PROJ > ASSOC
Anterior intracalcarine sulcus	−19, −61, 8	2.42	2008
Right Heschl's gyrus extending into insula and parietal operculum	40, −19, 14	3.01	3776
Left medial frontal gyrus	−27, 43, 29	2.94	8240
Left precentral gyrus	−53, −3, 38	3.39	3440
Right superior frontal gyrus, extending into middle frontal gyrus	23, 25, 50	4.2	9632
Bilateral superior precuneus	5, −67, 53	3.5	3632
ASSOC > PROJ
Cerebellum extending into occipital cortex	9, −76, −28	4.51	22512
Extending from left temporal fusiform gyrus, parahippocampal gyrus and hippocampus, to the posterior part of the thalamus	−28, −21, −15	4.12	8560
Right hippocampus, extending into amygdala	28, −11, −19	3.19	3496
Right hippocampus, extending into thalamus	12, −30, −4	2.74	3080
Left angular gyrus extending into superior temporal gyrus	−47, −61, 33	2.96	2624
Right angular gyrus and intraparietal sulcus,extending into superior parietal cortex	40, −57, 46	3.56	5720

Results show six regions with relative increased GM in projectors and six regions with relative increased GM in associators. Center of gravity, max t value, and size of the regions are reported. PROJ, Projector synesthetes; ASSOC, associator synesthetes.

Projector synesthetes

Increased GM in projector synesthetes was found located in the visual cortex (Fig. 3). This region of increased GM is located at the most anterior GM near intracalcarine sulcus in the left hemisphere (areas V17 and V18). Another region revealed by the covariate analysis is located in the right Heschl's gyrus (primary auditory cortex), extending medial and superior into right insular cortex and right parietal operculum. This region strongly overlaps with the region found (related to projector synesthesia) in the contrast analysis discussed in the previous section. A third region of increased GM in projectors compared with associator synesthetes is in the left precentral gyrus. This region is functionally described as the premotor and supplementary motor cortex. Increased GM for projectors compared with associators was also found in superior precuneus cortex.

Figure 3.

Increased GM in projector compared with associator synesthetes, resulting from a covariate analysis of PA and VBM values. Increased GM was found in modality-specific brain areas and prefrontal brain areas.

Increased activation in V1 has previously been found only in a case study (Aleman et al., 2001); unfortunately it is not clear whether this particular female with colored-hearing had projector-type experiences. The current finding of increased GM in V1 is in line with the notion of Ward et al. (2007) that the defining feature of projectors is binding of color to the location of an object in external space. They proposed a role of V1 in such representation of visual external space. We did not check whether all our projector synesthetes are “surface” projectors, but our data support a relation between projector synesthetes and V1.

In fact, the current study shows that regions related to projector synesthesia are found in several primary and secondary sensory brain areas. Overall, the areas can be summarized as brain regions involved in sensory experiences (auditory in Heschl's gyrus, visual in calcarine sulcus, secondary somatosensory in parietal operculum, and possibly taste in insula) and planning action (precentral gyrus).

The last two clusters of increased GM were found bilaterally in prefrontal cortex. Projectors, compared with associators, show increased GM around right superior frontal sulcus (superior frontal gyrus, extending to middle frontal gyrus). Increased GM in the left hemisphere was located mostly in medial frontal gyrus. Both regions occupy a large anterior–posterior region of the superior part of the frontal lobe.

Associator synesthetes

The covariate analysis found six regions of increased gray matter in associator synesthetes compared with projector synesthetes. Three of these regions are located in the hippocampal area. The first region overlaps strongly with the associator region found in the contrast analysis, reported previously. This region is located in the right hippocampus, possibly extending into the right thalamus. Increased GM was also found in the anterior part of the right hippocampus, extending into right amygdala. The third region is located in the left hippocampus, parahippocampal gyrus, and temporal fusiform gyrus and extends into the posterior part of the thalamus (Fig. 4). This region in left hemisphere is located bilaterally from the two regions in the right hemisphere.

Figure 4.

Increased GM in associator compared with projector synesthetes resulting from the covariate analysis of PA and VBM values, shown in blue. Clusters where associators, compared with projectors, have increased BOLD-MRI are shown in red.

Next to these regions in the hippocampal region, we found increased gray matter in right cerebellum, extending into the occipital lobe and bilaterally in the angular gyrus. The left region is located in angular gyrus, extending into superior temporal lobe. The right region is located in angular gyrus and the intraparietal sulcus, extending into superior parietal lobe (Fig. 4).

The regions related to associator synesthesia were found most prominently in hippocampus and in the angular gyrus. The hippocampus is mostly known for its function in memory and spatial memory. Functions ascribed to the angular gyrus are making an association between different types of information [e.g., in use of language (Geschwind, 1972)], a “core quantity system” (Dehaene et al., 2003), and the use of metaphors (Ramachandran, 2004).

Brain areas activated during grapheme–color synesthesia:BOLD-MRI

In the BOLD-MRI study, we contrasted a condition where subjects saw graphemes that elicited synesthetic color with a condition where subjects saw graphemes (symbols) that did not elicit a synesthetic color experience. The controls saw the same set of graphemes, which to them only had the color of the typeface (gray). Behavioral performance on the task indicated no differences in either reaction time (RT) or percentage correct between the subgroups, i.e., between controls (mean of 86% correct and mean RT of 964 ms) and synesthetes (87% and 952 ms), or between associator (mean of 88% correct and mean RT of 952 ms) and projector synesthetes (87% and 952 ms). Unfortunately, from one synesthete and two controls the behavioral results were lost.

Between-subject contrasts of synesthetes versus controls and associators versus projectors were performed. Three clusters of increased BOLD signal were found in synesthetes compared with nonsynesthetes (Table 3). The first cluster is gray matter around intraparietal sulcus, extending posterior to the parieto-occipital transition zone and occipital gyri, and extending anterior to the superior parietal lobe and angular gyrus. The second cluster is located in the medial part of inferior frontal gyrus and precentral gyrus. The third cluster is located in the gray matter around the parieto-occipital sulcus, mostly in left precuneus cortex. These results indicate that multimodal brain areas, most distinctively in parietal cortex, are activated during synesthesia, for both projector and associator subtype. No increased BOLD signal was found in nonsynesthetes compared with synesthetes.

The contrast between associator compared with projector synesthetes showed increased BOLD signal in both left and right parahippocampal gyrus, extending into temporal fusiform gyrus. These brain areas in temporal cortex are known to interact with the hippocampus on memory encoding and retrieval. These clusters of increased BOLD signal in associators showed partial overlap with, and were located inferior and medial to, the associator regions of increased gray matter in hippocampus (Fig. 4). This suggests a relation between the anatomical differences we found in the hippocampal region and the functional activation in that same region in associator compared with projector synesthetes. The projectors did not show increased BOLD signal compared with associators.

Discussion

We found brain regions that are related to grapheme–color synesthesia, regardless of the projector–associator subtype. One region of particular relevance to the presence of the synesthetic experiences is the superior posterior parietal cortex. Other brain regions, in contrast, mediate the individual differences in the nature of the synesthetic experience.

These findings are not only of interest to synesthesia researchers. Synesthesia provides an extraordinary case study of sensory integration, sensory experiences in the absence of the appropriate external stimulus, and perceptual awareness. One implication (as discussed in the next section) is the crucial role of nonvisual cortex in generating a visual percept, even though in our subjects both the inducer (grapheme) and concurrent (color) sensations are in the visual modality. Another implication of our findings concerns one of the pivotal issues in cognitive neuroscience: the relationship between the physiological properties of our brains and the subjective properties of our mental world. In this study, we have an extraordinary opportunity to study this question. Our subjects show a clear differentiation in their subjective experiences, even though these experiences are elicited by the same external stimulation. We found that the contrast between experiencing the synesthetic color in the mind only (associator) or in the outside world (projector) can be traced to different neural substrates. Moreover, the subjective properties of this sensory experience mimic the functional properties of the underlying neurological mechanism.

Synesthetes versus nonsynesthetes

We found increased gray matter in synesthetes compared with nonsynesthetes in the superior posterior parietal lobe. Previous studies have found increased activation of this region during synesthetic experiences (Paulesu et al., 1995; Steven et al., 2006). In the current study, increased gray matter in the superior parietal cortex was found unrelated to the idiosyncratic nature of the synesthetic experience. This is in line with previous studies examining this region: increased white matter integrity in synesthetes did not covary with the projector–associator distinction (Rouw and Scholte, 2007), and an attenuating effect after applying TMS was as strong for projectors as for associators (Muggleton et al., 2007). TMS studies suggest that this area plays a causal role in synesthesia (Esterman et al., 2006; Muggleton et al., 2007). Functionally, it is proposed to integrate (by hyperbinding, multisensory processes, or spatial/attentional processes) the inducing with the concurrent sensations (Grossenbacher and Lovelace, 2001; Robertson, 2003; Esterman et al., 2006). This is in line with a recent version of the structural brain differences model (Hubbard, 2007; Weiss and Fink, 2008), which proposes two stages: cross-activation at a sensory-specific level, followed by hyperbinding in superior parietal cortex.

In the current study, we did not find increased GM in synesthetes compared with nonsynesthetes in the fusiform gyrus. This should be interpreted with care. Weiss and Fink (2009) contrasted gray matter volumes between grapheme–color synesthetes and nonsynesthetes and initially did not find significant differences. After defining these areas as regions of interest, the authors found increased gray matter values bilaterally in the fusiform gyrus and in the intraparietal cortex. One possible explanation is that only with sufficient power (e.g., enough subjects or using an ROI rather than whole-brain analysis) can effects in the fusiform gyrus be obtained. A factor that might explain different results in ROI analyses (we did not obtain significant effects) is which exact area is selected. Another factor that might influence significance level is variability between subjects, as different patterns of variation between individuals are found in different brain areas (Mechelli et al., 2005).

Jäncke et al. (2009) found increased gray matter in the fusiform gyrus and adjacent regions in grapheme–color synesthetes. In addition to the fusiform gyrus, they found differences in the precuneus, superior occipital cortex, orbitofrontal cortex, vicinity of the central sulcus, insula, and hippocampus. Increased white matter coherence near the fusiform gyrus was also found after lowering the threshold (to p < 0.05). This is in accordance with findings in our previous study using DTI (Rouw and Scholte, 2007).

Projector grapheme–color synesthesia

We found structural brain differences related to projector synesthesia in primary and secondary modality-specific brain areas, involved in perceiving of (Heschl's gyrus, the calcarine sulcus, somatosensory association cortex) and acting in (premotor cortex) the outside world. The increased gray matter in visual cortex and Heschl's gyrus might reflect the role of auditory and visual codes in grapheme–color synesthesia. However, this does not provide a complete explanation. A more extended set of modality-specific brain areas showed increased gray matter, and this was in projector (compared with associator) synesthetes. This indicates that the subjective properties of the experiences are related to functional properties of the mediating brain areas. In particular, the outside-world experiences of projector synesthetes are related to those brain areas normally involved in perceiving of and acting in the outside world.

Interestingly, a VBM study of an interval–taste and tone–color synesthete reported increased volume of corresponding modality-specific brain areas (Hänggi et al., 2008). It is currently not clear whether similar neurobiological mechanisms underlie different kinds of synesthesia. Current results raise the question of whether a similar projector experience was present in this synesthete with other kinds of synesthesia.

Next to modality-specific brain areas, increased GM in projector (compared with associator) synesthetes was found in bilateral prefrontal brain areas. Previous studies found activation of prefrontal brain areas during synesthesia (Paulesu et al., 1995; Schiltz et al., 1999; Aleman et al., 2001; Nunn et al., 2002; Sperling et al., 2006; Beeli et al., 2008). One possible explanation is that prefrontal and modality-specific brain areas interact in creating the synesthetic experiences. This can be through associative learning (Passingham et al., 2000) and memory retrieval (Fletcher et al., 1998), or in terms of supporting the (change in) conscious perception (Kleinschmidt et al., 1998; Lumer et al., 1998; Rees, 2001; Vuilleumier et al., 2001). An alternative explanation is that the prefrontal brain areas are not involved in creating the synesthetic experiences per se, but reflect increased dependence on control mechanisms (Ridderinkhof et al., 2004). Projector synesthesia is more similar to (i.e., in the same, external reference frame) (Ward et al., 2007), and therefore interferes more with (Dixon et al., 2004), the real color experience. In this interpretation, although all synesthetes need prefrontal control (i.e., to distinguish real from synesthetic colors), projector synesthetes rely more heavily on these mechanisms.

Associator grapheme–color synesthesia

Most grapheme–color synesthetes report that the synesthetic color is located only in the mind and not in the outside world. As far as we know, the current study is the first indicating which set of brain areas underlie associator grapheme–color synesthesia. We found both increased volume/density and increased activation in and near the hippocampus. Associator synesthetes furthermore showed increased GM bilaterally in the angular gyrus. This area is known as a multimodal association area and is also associated with a quantity system (representing numbers).

These findings shed a new light on the relationship between the hippocampus and synesthetic experiences. Previous studies found activation (Gray et al., 2006) or increased white matter integrity (Jäncke et al., 2009) in the hippocampal region, but examined only the contrast between synesthetes and nonsynesthetes. In cognitive (neuro)psychology literature, the hippocampus is mostly known for its implication in memory functions, including episodic and declarative memory (Scoville and Milner, 1957; Squire, 1992; Eichenbaum, 2004), but also spatial memory (O'Keefe and Nadel, 1978; Maguire et al., 2000). Again, the findings indicate that the properties of the synesthetic experiences are related to functional properties of the mediating brain areas. In the case of associator synesthetes, the relatively stronger involvement of the hippocampus mediates a synesthetic experience more similar to retrieving a memory, i.e., an “internal” experience.

What underlies the development of synesthesia?

The innate predisposition to develop synesthesia (Asher et al., 2009) is general rather than to a particular kind of synesthesia (Barnett et al., 2008). Indeed, specific synesthetic associations must develop as an interaction between innate (Baron-Cohen et al., 1996; Rich et al., 2005; Ward and Simner, 2005) and environmental (Simner et al., 2005; Witthoft and Winawer, 2006; Beeli et al., 2007) factors. The presence of different distinct forms (e.g., grapheme–color, or taste–hearing) of synesthesia within the same family (Ward and Simner, 2005) suggests that common mechanisms may be shared across synesthetes, but developed or expressed into different forms. Interestingly, although our subject group consisted of grapheme–color synesthetes, results showed increased GM in several modality-specific brain regions in the projector synesthetes. Similarly, all possible associations can arise from hippocampal learning mechanisms found related to associator synesthetes. It is therefore possible that both mechanisms can in principle mediate different kinds of synesthetic associations. As environmental factors influence which particular synesthetic associations develop, the commonality of grapheme–color synesthesia might simply be due to the relative prominence of letters and numbers in our society. An interesting topic of further research is therefore whether projector–associator mechanisms can be found in different kinds of synesthesia.