Projectors, Associators, Visual Imagery, and the Time Course of Visual Processing in Grapheme-Color Synesthesia

Abstract

In grapheme-color synesthesia, seeing a particular letter or number evokes the experience of a highly specific color. Here we investigate the brain’s real-time processing of words in this population, by recording event-related brain potentials (ERPs) from 15 grapheme-color synesthetes and 15 controls as they judged the validity of word pairs (“yellow banana” vs. “blue banana”) presented under high and low visual contrast. Relative to high contrast words, low contrast words elicited ~30ms delayed P1/N170 visual ERP components in both groups. When color concepts were conveyed to synesthetes by a string of achromatic graphemes (“55555 banana”) individually chosen for each synesthete, visual contrast effects were like those in color words: P1/N170 components were delayed 30ms, but unchanged in amplitude. When controls saw equivalent strings of colored graphemes, visual contrast modulated the amplitude of P1/N170, leaving their latency unchanged. Color induction in synesthetes thus differs from color perception in controls. Independent from the experimental effects, all orthographic stimuli elicited larger N170 and P2 in synesthetes than controls. While P2 (150–250ms) enhancement was similar in all synesthetes, N170 (130–210ms) amplitude varied with the subjective experience of synesthetes and their self-reported imagery ability. Results suggest the extent of immediate cross-activation in visual areas processing color and shape is most pronounced in so-called projector synesthetes whose concurrent colors are experienced as originating in external space.

A neurological condition in which numbers and letters are experienced as colored, grapheme-color synesthesia has general implications for the relationship between conscious and unconscious processes of perception (Cohen Kadosh & Henik, 2007). While synesthesia has been an object of inquiry for over 200 years (Jewanski, 2013), only recently have cognitive neuroscientists begun to reveal what makes the synesthetic brain unique (Eagleman, Kagan, Nelson, Sagaram, & Sarma, 2007; Ward, 2013). Grapheme-color synesthetes and nonsynesthetes differ, for example, in white and gray matter volume of several brain areas (Banissy et al., 2012; Hube, Bordier, & Dojat, 2012; Weiss & Fink, 2009), and measures of local and long-range structural connectivity (Rouw & Scholte, 2007; Whitaker et al., 2014). Here we explore the manifestation of these anatomical differences as revealed by the brain’s real time processing of written linguistic stimuli.

Indeed, the heart of synesthesia is a process that unfolds in time – the mapping of one kind of information to another. When a grapheme-color synesthete perceives a letter or number, a specific visual form is mapped to a particular color, raising the question of how synesthetic color induction relates to processes of color perception and imagery in the typical brain. A central issue in studies of synesthesia concerns the nature and timing of the form-to-color mapping. One prominent account involves a feed-forward mechanism of nearly immediate cross-activation of color areas during visual form perception (Brang, Hubbard, Coulson, Huang, & Ramachandran, 2010; Hubbard, Brang, & Ramachandran, 2011). Others argue that the induction of synesthetic color requires feedback from higher-level association areas to bind the information in the two streams, i.e. form and color (Esterman, Verstynen, Ivry, & Robertson, 2006; Grossenbacher & Lovelace, 2001; Kadosh & Henik, 2006).

Adjudication between models emphasizing feed-forward versus feedback mechanisms thus requires more information about the time course of processing from measures with a high temporal resolution, such as electroencephalogram (EEG) (see Ward, 2012 for a review). To date, many EEG studies on synesthetes have focused on congruity effects that encourage top-down processing strategies. Brang et al. (2008), for example, recorded ERPs as participants read sentences ending with a grapheme whose synesthetic color rendered the sentence either congruent or incongruent (“The coca cola logo is white and 9”), and found congruity effects on early (N1, P2) as well as later (N400) components in synesthetes. Follow-up studies (Brang et al., 2011) showed that the visual N1 effects were very similar to those elicited by colored graphemes in non-synesthetes, and only the P2 effects were unique to synesthetes. However, because congruity effects, by definition, depend on contextual expectations, they leave open the extent of differences in ‘bottom-up’ aspects of orthographic processing in synesthetes and nonsynesthete controls.

Recent evidence hints that we are unlikely to understand the neural mechanisms of grapheme-color synesthesia if we treat all synesthetes as a uniform population. The subjective synesthetic experience can be characterized along a continuum from “projectors”, who report seeing colors projected onto the page or screen, to “associators”, who consistently associate letters or numbers with specific colors, reported as being in their ‘mind’s eye’ (Dixon, Smilek, & Merikle, 2004). The projector continuum appears related to individual differences in visual imagery (Simner, 2013), as grapheme-color synesthetes report more vivid and greater use of visual imagery than do nonsynesthetes (Barnett & Newell, 2008; Meier & Rothen, 2013; Spiller, Jonas, Simner, & Jansari, 2015).

Van Leeuwen and colleagues (2011) have suggested that individual differences among synesthetes may be the source of conflicting accounts of synesthesia as predominantly driven by ‘top-down’ versus ‘bottom-up’ activation. They scanned synesthetes viewing achromatic graphemes and performed dynamic causal modeling of the fMRI data. Whereas projector synesthetes exhibited activation consistent with the near immediate cross-activation of V4 via a bottom-up pathway in fusiform gyrus, associator synesthetes exhibited activity more consistent with top-down feedback from the parietal lobe. These results led to the suggestion that while grapheme processing activates a similar network of brain regions in all grapheme-color synesthetes, their dynamic interaction differs in projectors and associators.

Present study

In the present study, we examined ERPs to orthographic stimuli in synesthetes and non-synesthetes in a paradigm that minimized the import of contextual expectations. In this paradigm, grapheme-color synesthetes and matched controls made judgments about two kinds of knowledge in a go/nogo decision task that relied to varying degrees on the synesthetic concurrent. Participants responded to object names (“lime”) preceded either by valid color names (“green”) or locations (“kitchen”), and withheld responses for invalid colors and locations. For color decisions, synesthetes saw achromatic grapheme strings (“55555”) individually designed for each participant based on their responses to a color consistency test; control participants saw colored grapheme strings designed to mimic the perceptual experience of synesthetic participants (“55555”), and to enable a semantic color decision task. This physical stimulus difference between groups admittedly precludes certain inferences about the elicited ERPs, but enables others of specific interest here.

The decision task can inform to what extent, if any, grapheme-color synesthesia impairs decision-making performance when the eliciting stimuli evoke synesthetic experiences. If successful form-to-color mapping occurs only after grapheme processing is complete (Mattingley, Rich, Yelland, & Bradshaw, 2001), then the speeded decision task here might lead to slower and less accurate performance in synesthetes than controls. Further, especially in the color name condition, it is possible that the form-to-color mappings could interfere with the form-to-meaning mappings required for the task. One could imagine that in the color name condition (e.g. “blue ocean”), different synesthetic colors associated with each grapheme in “blue” might interfere with mapping this word form to the concept blue.

In addition, ERPs to the initial stimulus in each condition (grapheme strings, color names, location words) will yield direct comparisons of the sequelae of electrical brain potentials evoked by graphemes and words across synesthetes and nonsynesthetes. Importantly, the grapheme strings in the present study will not benefit from contextual expectations, but rather create them. Inspection of these ERPs also can inform Brang et al.’s (2008, 2011) suggestion that the P2 component reflects neural processes unique to the emergence of the synesthetic color experience. Because the grapheme strings were designed to evoke a unified synesthetic color experience that was more intense than the mixture of colors evoked by the letters in color and location words, we predicted that the P2 component would differentiate grapheme strings from words in synesthetes, but not in controls.

Finally, we investigated the extent that individual differences along the associator-projector continuum and in visual imagery could explain amplitude differences in ERPs to orthographic stimuli. If cross-activation occurs only in projector synesthetes (Van Leeuwen, et al., 2011), we might expect synesthetes to display greater within-group variability than controls in the amplitude or scalp distribution of early visual ERP components such as the P1, with projectors the most divergent, and associator synesthetes exhibiting more similar ERPs to non-synesthete controls.

Method

Participants

Table 1 shows descriptive statistics for several variables characterizing synesthetes and control participants. All participants were fluent speakers of English. Participants provided written informed consent prior to the experiment and received course credit and/or $9/hour for participating.

Table 1.

Variables matched as closely as possible across grapheme-color synesthetes and nonsynesesthete controls.

Variable	Controls (N = 15)	Synesthetes (N = 15)	t-score (p-value)
Age	20.73 (2.99)	20.20 (2.46)	0.53 (0.60)
Laterality quotient	61.87 (63.39)	53.00 (46.37)	0.44 (0.67)
Object style (imagery)	3.37 (.40)	3.64 (.67)	1.34 (0.19)
Spatial style	2.85 (.67)	3.27 (.47)	2.02 (0.05)
Verbal style	3.0 (.27)	3.05 (.53)	0.35 (0.73)

Synesthetes.

Fifteen grapheme-color synesthetes (12 females) were recruited from flyer distribution and announcements in UCSD undergraduate classes. Participants had normal or corrected-to-normal vision, and reported no major neurological or health problems. Synesthetic experience was tested with standardized color-grapheme consistency matching and speeded congruency judgments (Eagleman, Kagan, Nelson, Sagaram, & Sarma, 2007). Synesthetes ranged from 0.25 to 0.92 on the consistency matching test where scores below 1.0 are strongly indicative of synesthesia, and ranged between 80% and 100% accuracy (M = 91%, SD = 6.1%) in speeded congruency judgments. The Eagleman et al. battery also contains a questionnaire designed to assess how synesthetes experience their synesthetic percepts, where “projectors” describe the color as physically inhabiting a particular spatial location (e.g., on the page or screen) and “associators” report the color is evoked in their ‘mind’s eye’. A positive score on this measure is more consistent with the projector experience and a negative score is more consistent with the associator experience. Synesthetes in the present study ranged from −3.3 to 1.7, largely consistent with the greater prevalence of associator versus projector synesthetes (Dixon & Smilek, 2005). Table 2 shows descriptive statistics for individual difference measures for synesthetes.

Table 2.

Means, standard deviations, and correlations for synesthetes’ individual difference measures

Variable	M	SD	1	2	3	4
1. Object style (imagery)	3.64	0.67
2. Spatial style	3.27	0.47	−.16
3. Verbal style	3.05	0.53	−.56*	−.02
4. Consistency score	0.55	0.16	−.23	−.11	.63*
5. Associator/Projector score	−1.31	1.47	.49	.19	.13	−.02

Note:

^*indicates p < .05. M and SD represent means and standard deviations, respectively.

Control participants.

Fifteen controls (12 females) were recruited from Psychology and Cognitive Science courses at UC San Diego. Controls were matched to synesthetes for age, sex, and handedness (see Table 1). Participants had normal or corrected-to-normal vision, and reported no major neurological or health problems.

Materials

One hundred and sixty-eight objects were paired with one valid color and location, resulting in 336 property-concept pairs. Eleven color names (“black”, “blue”, “brown”, “gold”, “green”, “orange”, “pink”, “purple”, “red”, “white”, “yellow”) were selected from feature production norms (McRae, Cree, Seidenberg, & McNorgan, 2005) wherein at least 7 participants produced the color as a feature of the concept. Words denoting 21 locations (“backyard”, “battlefield”, “desk”, “dresser”, “farm”, “forest”, “fridge”, “garden”, “grass”, “house”, “kitchen”, “ocean”, “party”, “pond”, “street”, “swamp”, “tree”, “tropics”, “water”, “winery”, “zoo”) were selecte from the same dataset and in part by the experimenters. Each color name and location name was paired with between 2 and 33 (M = 10.5, SD = 8.7) object names. We also created 336 invalid concept-property pairs (e.g., red lime) by shuffling the valid pairs, widely distributing the invalid properties across different objects to avoid list-level associations between valid and invalid pairs (e.g., concepts paired with the color name “red” were paired with a variety of different location properties in the invalid pairs). We attempted to minimize the difficulty of determining whether a trial was valid or invalid by avoiding semantically similar pairs (e.g., if valid pairs were “yellow banana” or “kitchen banana” the invalid pairs would not be “orange banana” or “bedroom banana”).

The stimuli were displayed on a CRT monitor (ViewSonic P220f), presented slightly above center in Helvetica font (each character subtended about 0.8 degrees of visual angle in height and 0.6 in width). Visual stimulus contrast for achromatic stimuli was manipulated by presenting either white (luminance: 47.6 cd/m2), or light grey (luminance: 42.6 cd/m2) text against a constant slightly darker gray background (luminance: 39.9 cd/m2). Visual contrast for the 11 chromatic graphemes shown to controls was manipulated by presenting the stimulus at RGB values according to the CSS3 “X11 color” specifications, and for low contrast by alpha compositing: alpha of the foreground layer was decreased to 50% against the constant background.

Design

We created a 3 (property type: grapheme string, color name, location name) x 2 (visual contrast: high, low) factorial within-subjects design (Figure 1). Property type was blocked, and contrast was randomized within blocks. Each of the 168 object names appeared once in every block. Visual contrast was split evenly between low and high contrast within and across blocks, resulting in two versions of each block. Within each block the order of trial presentation was selected at random with the exception that trials requiring a given response type (i.e., go or nogo) never appeared more than four times in succession. Each participant performed six blocks (two versions of each block) in which they responded (go trials) to valid pairs and withheld a response to invalid pairs. Block order across participants was determined by random selection without replacement from a 6×6 Latin Square, which repeated every six participants.

Figure 1.

Object names were preceded by either grapheme strings, color names, or locations, and each stimulus pair was presented under high and low visual contrast. Whereas control participants viewed chromatic grapheme strings, synesthetes viewed achromatic grapheme strings known to elicit a particular color for that person based on their consistency test results. Each trial began with the property for 200 ms, followed by a 300 ms blank screen, followed by the object name for 200 ms. Participants responded (go trials) to valid pairs and withheld a response (nogo trials) to invalid pairs.

Whereas the color name and location name blocks were identical for all participants, the grapheme string condition was not. Synesthetes viewed achromatic strings of five graphemes (e.g., “33333”, “SSSSS”) known to elicit a particular color for that synesthete based on their consistency test. Not all synesthetes reported associations for every color included in the design; therefore, the number of colors varied slightly across participants (range: 9–11). Control participants viewed chromatic strings of five graphemes chosen at random for each participant, with the only constraint that letters and numbers were sampled according to the total frequency of each (i.e., letters were more likely to be selected than numbers). The validity manipulation was identical to the color name condition.

Procedure

Prior to the EEG experiment, all participants completed the Oldfield handedness inventory, a demographic and health information questionnaire, and a computerized version of the Object-Spatial Imagery and Verbal Questionnaire (OSIVQ), a self-report instrument with high reliability and sensitivity designed to assess cognitive style along object imagery, spatial imagery, and verbal dimensions (Blazhenkova & Kozhevnikov, 2009). We were particularly interested in the object imagery dimension (imagery of objects and scenes in terms of their shape, color, texture, etc.), since synesthetes may experience more vivid visual imagery (Barnett & Newell, 2008; Spiller & Jansari, 2008; Spiller, Jonas, Simner, & Jansari, 2015; Whitaker et al., 2014), and be more likely to emphasize a visual imagery cognitive style (Meier & Rothen, 2013), than age-matched controls.

Participants were tested individually while seated in a dimly lit, sound attenuating, electrically-shielded chamber, in front of the CRT monitor at a viewing distance of about 110 cm. At the beginning of the EEG experiment the participant was shown seven word pairs in low contrast, and asked to name each of them aloud to ensure visibility. Before the first block of each condition the experimenter explained the decision criterion, showed the participant three examples of valid and invalid trials in high and low contrast, and ensured that the participant understood the correct decision for each. Synesthetes were shown the grapheme-color mappings selected based on the consistency-matching test and verbally approved the selections. Participants then completed 26 practice trials (13 low contrast/13 high contrast) identical to the experimental trials with the exception that the experimental and practice object names did not overlap.

Each trial began with the property for 200ms, followed by a 300ms blank screen, followed by the object name for 200ms. A blank screen then appeared for a randomly selected interval between 2200 and 2400ms. The words appeared above a small gray fixation square subtending about 0.5 degrees of visual angle in height and width, that remained on the screen throughout each trial.

EEG Recording and Analysis

The electroencephalogram was continuously recorded from 26 geodesically-arranged tin electrodes (see Ganis, Kutas, & Sereno, 1996) embedded in an ElectroCap (impedances were kept below 5 kOhms), and referenced to the left mastoid. Eye movements and blinks were monitored with electrodes placed on the left and right lower orbital ridge, and left and right external canthus. The EEG was digitized at a sampling rate of 250 Hz and bandpass filtered between 0.01 and 100 Hz with James Long amplifiers (www.JamesLong.net). Potentials were re-referenced offline to the mean of left and right mastoid electrodes. Averages were obtained for 1000ms epochs including a 200ms baseline period prior to stimulus onset and screened for different kinds of artifacts. Trials containing amplifier blocking for at least 30ms at any channel were automatically discarded. Trials containing blinks were identified by polarity inversions, operationalized as the absolute difference between the maximum difference and mean difference between the left/right lower eye channel and the left/right medial prefrontal channel (rejection criteria across subjects ranged from 40 to 80 microvolts; median = 60). Trials containing lateral eye movements were identified by computing peak-to-peak amplitudes at the horizontal electro-oculogram (rejection criteria ranged from 40 to 96 microvolts; median = 72). Finally, six trials containing excessive drift at one or more electrodes were identified by peak-to-peak amplitudes (rejection criteria ranged from 64 to 260; median = 190). All trials identified for removal were visually inspected to determine and verify the appropriateness of artifact rejection criteria before removal. The proportion of rejected trials did not differ statistically between groups or stimulus type, (Fs < 0.6), but did differ slightly between high contrast trials (M = 15.5%, SD = 10.6%) and low contrast trials (M = 14.2%, SD = 10.3%), F(1, 28) = 8.50, p < .001.

Analysis of EEG data was based on the literature of relevant ERP components. Accordingly, the P1 component was assessed by measuring the mean amplitude of ERPs recorded at the left and right lateral occipital channels between 90 and 130ms, as in (Mangun, 1995). As in (Rossion et al., 2003), the N170 was measured at the left and right lateral occipital channels between 130 and 210ms. The fronto-central P2 was measured as the mean amplitude of ERPs at eight lateral and medial frontal and prefrontal channels between 150 and 250ms (e.g., Federmeier & Kutas, 2001). As detailed below, the N200 component was of interest only for its onset latency, and we followed the same fractional area analysis procedure as in (Amsel, Urbach, & Kutas, 2014).

Visual inspection of ERPs also revealed an unanticipated effect on the anterior N1 elicited by target stimuli. This difference resembled that reported by Vogel & Luck (2000), and was assessed in a similar fashion via mean amplitude measurements between 75–120ms at left/right/middle central sites and the middle parietal site.

Results

Task Performance

Table 3 shows descriptive statistics for all behavioral measures.

Table 3.

Descriptive statistics for all behavioral measures (means, with standard deviations in brackets)

Group	Visual contrast	Stimuli	RT	Hit rate	d-prime	Response bias
Controls	High	Color	821 (138)	0.75 (0.05)	2.51 (0.25)	0.05 (0.01)
		Grapheme	772 (122)	0.74 (0.04)	2.37 (0.23)	0.05 (0.01)
		Location	861 (137)	0.7 (0.06)	2.4 (0.27)	0.05 (0.02)
	Low	Color	848 (135)	0.74 (0.05)	2.42 (0.33)	0.05 (0.02)
		Grapheme	824 (112)	0.68 (0.06)	2.16 (0.28)	0.06 (0.02)
		Location	897 (157)	0.65 (0.1)	2.09 (0.44)	0.07 (0.03)
Synesthetes	High	Color	757 (140)	0.73 (0.06)	2.36 (0.32)	0.05 (0.02)
		Grapheme	745 (116)	0.68 (0.09)	2.21 (0.37)	0.06 (0.03)
		Location	807 (157)	0.72 (0.05)	2.38 (0.33)	0.05 (0.02)
	Low	Color	799 (153)	0.72 (0.07)	2.29 (0.47)	0.06 (0.03)
		Grapheme	800 (123)	0.63 (0.12)	2.08 (0.41)	0.07 (0.02)
		Location	844 (160)	0.68 (0.09)	2.28 (0.29)	0.06 (0.02)

Accuracy.

Sensitivity¹ (d’) and response bias² (beta) in go/nogo task performance were estimated from hit rates (H) and false alarm rates (FA) in each condition after applying the log-linear rule to correct for extreme proportions (Hautus, 1995). We conducted three-way ANOVAs of response sensitivity (d-prime) and response bias, with one between-subjects factor (group) and two within-subjects factors (stimulus type, contrast). The sensitivity ANOVA revealed a main effect of contrast, as participants exhibited decreased sensitivity under low contrast versus high contrast, F(1, 28) = 13.7, p = .001, η²_G = .05³, and a main effect of stimulus type, F(2, 56) = 7.25, p = .002, η²_G = .05, as participants exhibited lower sensitivity to target words preceded by graphemes than color names, t(58) = 2.53, p = .01, with locations falling in the middle and not differing from either of the other stimulus types (ts < 1.5). A similar ANOVA of response bias revealed only a main effect of contrast, F(1, 28) = 10.25, p = .003, η²_G = .04, as participants were more likely to miss valid targets presented in low contrast than high contrast, and less likely to make false alarms.

Figure 8.

Scatterplots representing the linear associations between synesthetes’ and controls’ imagery style, and P1 and N170 amplitude for each condition. Data are aggregated over low and high visual contrast. In all conditions, synesthetes scoring higher on imagery style exhibit smaller P1 amplitudes and larger N170 amplitudes, whereas trend in the opposite direction. In most cases the difference between synesthetes’ and controls’ correlations was itself statistically significant. See text for details.

Decision latency.

Response latencies (go trials) were measured from the onset of the object name, and responses occurring after 2000ms were not registered. A three-way ANOVA with one between-subjects factor (group) and two within-subjects factors (stimulus type, contrast) was computed on mean decision latencies for correct go (valid) trials. Participants were faster to respond to targets presented in high contrast than low contrast, F(1, 28) = 106.2, p < .001. There was also a main effect of stimulus type, F(2, 48) = 10.1, p < .001, such that participants responded more quickly to targets when they were preceded by graphemes than locations, t(58) = 2.53, p = .06, whereas color names did not differ from either condition.

NoGo – Go N200 effect.

The ERPs at prefrontal sites are more negative when elicited by withholding a response to an invalid attribute (nogo trials) versus responding to a valid attribute (go trials). The onset latency of the derived N200 effect (nogo – go trials) has been taken as an upper limit of when sufficient semantic information has become available from a stimulus to determine whether or not to make a response (Amsel, Urbach, & Kutas, 2014; Hauk, Coutout, Holden, & Chen, 2012; Müller & Hagoort, 2006; Schmitt, Munte, & Kutas, 2000). We operationalized the N200 effect onset latency for each condition as the time point that corresponds to 20% of the area under the curve given by the negative polarity ERP between 200 and 700ms following target onset (i.e., fractional area latency). A mixed ANOVA with one between-subjects factor (group), and within-subjects factors of stimulus type, contrast, and electrode site was conducted on fractional area estimates at five (lateral, medial and midline) prefrontal sites. The only significant effect was of visual contrast, F(1, 28) = 4.87, p = 0.04, η²_G = .02, where low contrast trials were delayed by 14ms (see Figure 6).

Figure 6.

A) N200 effects (nogo – go difference wave) collapsed across stimuli type are shown for low and high contrast trials in both groups. ERPs are averaged from five sites over frontal cortex. B) Bar graphs of N200 effect onset latencies (time point that corresponds to 20% of the area under the curve given by the negative polarity ERP between 200 and 700 ms following target onset confidence intervals). Error bars show Fisher’s least significant difference.

Brain Responses to Stimuli

Properties (stimulus 1).

Effects of stimulus contrast.

Figure 2 shows evoked potentials to grapheme strings at left occipital sites where we expect to see the P1 (Mangun, 1995) and N170 (Rossion, Joyce, Cottrell, & Tarr, 2003) based on canonical studies of these components. An approximately 30ms delay under low contrast was visible in both components in the color name and location conditions for both groups. Consequently, analyses of ERPs time-locked to the initial stimulus in the low contrast color name and location conditions involved a 30ms delay in the time windows used to capture mean amplitude, (e.g., the P1 was quantified by measuring ERPs to high contrast stimuli from 90–130ms, and from 120–160ms for low contrast stimuli). The motivation for this step is that a single time window encompassing components elicited by high contrast stimuli and delayed components elicited by low contrast stimuli was found to capture surrounding components (e.g., the low contrast P1 and high contrast N170 overlap considerably), thereby distorting the estimated amplitudes.

Figure 2.

Effect of decreasing visual contrast for each stimulus type at a representative left lateral occipital site. Viewing low contrast monochromatic grapheme strings (viewed by synesthetes) and low contrast words (viewed by both groups) delays early visually evoked components (P1, N170) by approximately 30 ms. Decreasing the contrast of chromatic grapheme strings (viewed by controls) causes no measurable delay in the same components. For plotting only, ERPs are lowpass filtered with a 3rd order Butterworth filter using a 20 Hz cutoff. In this and all subsequent ERP plots negative is plotted up.

Low contrast graphemes elicited a similar 30ms delay in synesthetes’ P1 and N170 components, presumably because they viewed achromatic graphemes, but not in controls, who viewed colored stimuli (see Figure 2). Consequently, the latency adjustment described above was applied to measurements of synesthetes’ ERPs to low contrast graphemes but not of controls.

Lateral occipital P1 and N170.

Figure 3 shows the P1 and N170 components at the left occipital site for all conditions in both groups. Synesthetes’ ERPs are visibly more negative than controls’ ERPs beginning on the downslope of the P1 component, and this difference persists throughout the 500ms window prior to stimulus two onset. Four-way ANOVAs with one between-subjects factor (group) and three within-subjects factors (stimulus type, contrast, hemisphere) were computed on mean amplitudes at the left and right lateral occipital channels between 90 and 130ms⁴ to capture the P1 component (Mangun, 1995), and between 130 and 210ms to capture the N170 component (Rossion et al., 2003).

Figure 3.

A) P1 and N170 components are visible following the onset of the initial stimulus (see arrows with labels) and subsequent target word, at left and right lateral occipital sites for all conditions in both groups. Synesthetes’ ERPs are visibly more negative than controls’ ERPs beginning already on the downslope of the P1 component, and this difference persists throughout the 500 ms window prior to target word onset. Mean P1 and N170 amplitudes were significantly different across groups. In both groups the P1 was larger under high versus low contrast, but the N170 was not modulated by visual contrast. For plotting only, ERPs are bandpass filtered between 0.5 and 30 Hz with a 3rd order Butterworth filter. B) Bar graphs of mean P1 and N170 amplitude elicited by the initial stimulus; see text for measurement details. Error bars show Cousineau-Morey within-subject 95% confidence intervals (Baguley, 2011).

There were large group effects on the P1, F(1, 28) = 6.2, p = .02, η²_G = .14, and N170, F(1, 28) = 13.3, p = .001, η²_G = .23: P1 amplitude was approximately 2 microvolts larger for controls than synesthetes, and N170 amplitude was three microvolts larger for synesthetes than controls. P1s were significantly larger under high versus low contrast, F(1, 28) = 10.30, p = .003, η²_G = .01, whereas N170s were not modulated by contrast (F < 1). Stimulus type interacted with contrast on the P1, F(2, 56)⁵ = 4.7, p = .013, η²_G < .01, and N170, F(2, 56) = 9.20, p < .001, η²_G < .01.

Given the stark difference between controls’ and synesthetes’ waveforms, we explored this stimulus type by contrast interaction by computing Holm-Bonferroni-corrected paired samples t-tests separately for each group. P1 amplitude did not differ across stimulus type in either group under either high or low contrast. N170 amplitude in synesthetes, however, was larger for graphemes than color names under high contrast, t(14) = 4.0, p = 0.001 (Figure 3).

Frontocentral P2.

Figure 4A shows the P2 at a representative medial frontal site. Measurements of the P2 were subjected to a three-way ANOVA with one between-subjects factor (group) and two within-subjects factors (stimulus type, contrast). Main effects of group, F(1, 28) = 7.28, p = .012, η²_G = .19, and stimulus type, F(2, 56) = 4.40, p = .017, η²_G = .01, were qualified by a group by stimulus type interaction, F(2, 56) = 4.69, p = .013, η²_G = .01. The interaction was explored by computing Holm-Bonferroni-corrected paired samples t-tests separately for each group. For controls, P2 amplitude did not differ across stimuli (ts < .31), whereas for synesthetes the P2 elicited by graphemes was larger than that elicited either by color names, t(14) = 2.6, p = .02, or locations, t(14) = 2.92, p = .01. The uniquely large grapheme-elicited P2 in synesthetes is clearly visible under high and low contrast (Figure 4A and 4B).

Figure 4.

A) The P2 component elicited by the initial stimuli at a representative medial frontal site. For plotting only, ERPs are lowpass filtered with a 3rd order Butterworth filter using a 20 Hz cutoff. B) Bar graphs of mean P2 amplitudes; see text for measurement details. Error bars show Cousineau-Morey within-subject 95% confidence intervals. The P2 is not only larger in synesthetes than controls, it is also larger for graphemes than either of the word stimuli in synesthetes only. The unique processing pattern of grapheme strings in synesthetes was not modulated by decreasing visual contrast.

ERPs to target (second) word.

Although there were differences in a subset of the properties shown to the two groups, (that is, the synesthetes viewed achromatic graphemes, and the controls viewed colored ones), the target words were identical for all participants in all three conditions. Our initial analyses examined the early visually evoked components (P1, N170) elicited by target words in the same ERPs as above (i.e., baseline corrected prior to property onset). The P1 and N170 analyses were conducted on mean amplitudes 575 to 625ms and 630 to 710ms, respectively, following onset of the first stimulus (i.e., the commensurate time windows as the P1/N170 analyses above, including the 30ms low contrast latency adjustment). The pattern of a smaller P1 component, F(1, 28) = 9.1, p = .005, η²_G = .20, and larger N170 component, F(1, 28) = 12.0, p = .002, η²_G = .23, in synesthetes versus controls following target word presentation mirrored the pattern following initial stimulus presentation (see Figure 3). The P1 exhibited a group by stimulus by contrast interaction, F(2, 56) = 13.9, p < .001, η²_G = .01, that was explored with post hoc mixed ANOVAs on ERPs elicited by each stimulus type. Analyses with one between-subjects factor (group) and one within-subjects factor (contrast) revealed a significant group x contrast interaction in the grapheme condition, F(1,28) = 26.56, p<.001, η²_G = .02, that was not present for the color names, F(1,28) = 0.00, p=.98, η²_G < .01, or locations, F(1,28) = 0.01, p=.93, η²_G = .01. Among controls, high contrast targets in the grapheme condition elicited larger P1 than low (2.53 versus 0.57 μV); this was not the case for synesthetes (−1.84 vs. −1.34μV).

The N170 exhibited a group by stimulus by contrast interaction, F(2, 56) = 8.9, p < .001, η²_G < .01, that was explored with post hoc mixed ANOVAs as above. As for the P1, these analyses suggested group differences in the contrast effect in the grapheme condition, F(1,28) = 14.23, p<.001, η²_G = .01, but not for the color names F(1,28) = 0.83, p=.37, η²_G < .01, or for the locations, F(1,28) = 0.01, p=.93, η²_G = .01. As for the P1, N170 was larger (i.e., more negative) for high than low contrast targets in controls (−3.30 versus −1.38μV), but not in synesthetes (−7.63 versus −7.38μV).

Next, to identify any effects unique to target word processing, we examined the ERPs to target words after baseline correction with the 200ms pre-target word interval. Visual inspection of ERPs averaged across Go and NoGo trials revealed a prominent negativity peaking around 100ms at centro-parietal sites (N1 component) followed by a P2 and presumably a centro-parietal N400 (Figure 5). In synesthetes, the N1 appears to be larger when targets were preceded by graphemes, and this increased negativity appears to sustain throughout the visible 700ms epoch. However, high-pass filtering revealed that the amplitude difference resolved following the N1, and thus we only analyze N1 amplitudes.

Figure 5.

A) Target word ERPs reveal a large group difference in centrofrontal N1 amplitude under both levels of visual contrast. The midline occipital site is shown for comparison. Under low contrast, target words preceded by graphemes elicited the largest N1s in both groups. Under high contrast, when target words were preceded by graphemes the N1 was significantly larger in synesthetes versus controls, whereas when target words were preceded by word stimuli the N1 did not differ across groups. For plotting only, ERPs are bandpass filtered between 2 and 20 Hz with a 3rd order Butterworth filter. B) Bar graphs of mean N1 amplitudes; error bars show Cousineau-Morey within-subject 95% confidence intervals.

A mixed ANOVA with one between-subjects factor (group), and three within-subjects factors was conducted at left/right/middle central sites and the middle parietal site between 75 and 125ms (Vogel & Luck, 2000). Within-subjects factors were stimulus type, contrast, validity (valid/go vs. invalid/nogo), and electrode site. Only correct trials were retained for analysis. We found a main effect of group, F(1, 28) = 5.0, p = .03 η²_G = .08, such that synesthetes’ N1 amplitudes were about 3 microvolts larger than those of controls. Valid trials were about 0.5 microvolts larger than invalid trials, F(1, 28) = 5.1, p = .032, η²_G = .01, but validity did not interact with any other factors. There was also an interaction between stimulus type and contrast, F(2, 56) = 5.4, p = .007, η²_G = .02, which we explored by conducting a separate two-way ANOVA for low and high contrast stimuli. There was a main effect of property type for low contrast targets, F(2, 56) = 7.0, p = .005, η²_G = .04, such that N1s were largest when targets were preceded by graphemes, followed by color names, followed by locations (in both groups). For high contrast targets, there was a property type by group interaction, F(2, 56) = 6.7, p = .002, η²_G = .04. Independent samples t-tests with Holm-Bonferroni correction were used to compare groups’ N1 amplitudes for each property type. N1s to targets preceded by grapheme strings were much larger for synesthetes (M = −5.7) than controls (M = −2.2), t(28) = 3.3, p < .01, whereas group differences when targets were preceded by color names or locations were not statistically significant.

Individual Differences

Synesthetes scored higher than controls on imagery, spatial, and verbal dimensions of the OSIVQ, but no difference was statistically significant. Among synesthetes, Holm-Bonferroni adjusted analyses revealed positive correlations between verbal style and grapheme-color consistency score, r(13) = .63, p < .01, and between object imagery style and associator-projector score, r(13) = .49, p < .01 (higher imagery was associated with stronger evidence of being a projector). The latter correlation is consistent with the possibility that the associator-projector distinction falls out naturally from individual differences in mental imagery (Simner, 2013).

Mental imagery generation modulates the amplitude of at least two visual ERP components (Ganis & Schendan, 2008; Qiu, Li, Liu, & Zhang, 2007), the P1 and N170. Accordingly, we examined correlations between the by-subject mean amplitudes reported above and (1) by-subject imagery cognitive style scores for both groups, and (2) associator-projector scores for synesthetes. Given the more exploratory nature of these analyses, we have elected to present scatterplots and correlation coefficients with uncorrected p-values.

We began by examining P1 and N170s to the initial stimuli (at the same sites and aggregated across the same factors as reported earlier). As initial analyses revealed that visual contrast had a minimal influence on the magnitude of correlations, ERP amplitudes were aggregated over visual contrast to increase the signal to noise ratio. Higher associator-projector scores (i.e., those indicative of projector synesthetes) were associated with smaller P1s to graphemes, r(13) = −0.54, p < .05, color names, r(13) = −0.54, p < .05, and locations, r(13) = −0.46, p = .08, and conversely with larger N170s to graphemes, r(13) = −0.54, p = .04, color names, r(13) = −0.50, p = .06, and locations, r(13) = −0.38, p = .16. Figure 7 contains scatterplots showing correlations between associator-projector score, and P1 and N170 amplitude for each condition.

Figure 7.

Scatterplots representing the linear associations between synesthetes’ associator-projector scores, and P1 and N170 amplitude for each condition. Data are aggregated over low and high visual contrast. In all conditions, synesthetes reporting a greater projector tendency exhibit smaller P1 amplitudes and larger N170 amplitudes. See text for details.

In view of the positive association between associator-projector score and imagery style reported above, we expected higher imagery style to be associated with smaller P1s and larger N170s among synesthetes. This expectation was confirmed, where higher imagery style score was associated with smaller P1s to graphemes, r(13) = −0.65, p < .01, color names, r(13) = −0.56, p < .03, and locations, r(13) = −0.56, p = .03, and conversely, larger N170s to graphemes, r(13) = −0.44, p = .10, color names, r(13) = −0.38, p = .16, and locations, r(13) = −0.40, p = .14.

To test whether the difference between correlation coefficients (for synesthetes versus controls) was itself statistically significant, the Fisher r-to-z transform was employed. Analysis revealed group differences in the correlation between imagery style and P1 amplitude for graphemes, z(13) = 2.57, p = .01, color names, z(13) = 2.57, p = .01, and locations, z(13) = 3.09, p < .01. Similar results were obtained for the difference in correlations between imagery style and the N170 component (graphemes: z(13) = 2.03, p = .04; color names: z(13) = 1.75, p = .08; locations, z(13) = 1.94, p < .05). Figure 8 contains scatterplots showing correlations between imagery style, and P1 and N170 amplitude for each condition, in each group of participants. Beyond P1/N170 components, neither the P2 elicited by the first stimulus, nor the ERP components elicited by the target word, exhibited any reliable relationships with individual difference measures.

Discussion

Synesthetes and controls performed a property verification task regarding the typical colors and locations of common objects. Although behavioral measures of task performance were very similar in the two groups, the accompanying ERPs revealed several striking differences in their visual processing of orthographic stimuli. Relative to controls, in synesthetes, visually presented words elicited reduced amplitude P1 components and enhanced N170. Moreover, among synesthetes, the amplitudes of both the P1 and N170 components were systematically related to their scores on the associator-projector continuum and their preference for visual imagery. These data link differences in the subjective experience of grapheme-color synesthetes to underlying neurophysiological differences in their visual processing of words.

Decision Task

In the color word and the location blocks, both the property words and the target words were identical for the synesthetes and the controls. Performance in these conditions thus serves as a baseline for comparison with the grapheme blocks that involved differences in the property stimuli presented to each group. In fact, synesthetes and controls did not differ in sensitivity, response bias, decision latency, or N200 latency in either the color word or the location word conditions. These data attest to the similarities in the time course of orthographic and semantic processing in the two groups, and suggest the form-to-color mappings in synesthetes do not interfere with the form-to-meaning mappings central to reading.

In the grapheme condition, our decision task pitted synesthetic induction of color from achromatic graphemes against the visual perception of color from chromatic ones. Remarkably, synesthetes’ behavioral performance was no different from that of controls. Examining brain activity related to task performance, the onset of the N200 effect also was virtually identical in the two groups of participants. This ERP effect is a marker of when sufficient information has accumulated to make the semantic decision required by the task. Consequently, its similar latency in synesthetes and controls suggests the time course of the conceptual task was essentially the same, whether the color property was activated by synesthetic induction or by color perception.

However, ERPs to the target words – physically identical for all three conditions – revealed group differences in the grapheme blocks, though not in the color word or location blocks. During the grapheme blocks, we observed differences in the anterior N1 elicited by targets (see figure 3). Anterior N1 effects have been reported in tasks that require participants to make a discriminative response (see Mangun, 1995 for a review). Vogel & Luck (2000) have shown that early effects like that reported here can be dissociated from the slightly later N1 components, (e.g. the N170), evident at posterior and lateral sites, and suggest that only the latter index visual processing, while the anterior N1 indexes anticipatory motor-related activity.

Larger anterior N1 to these targets in synesthetes suggests that despite the absence of performance differences, the synesthetic induction required to derive color from achromatic stimuli may have impacted the way they performed the task. Indeed, perhaps the synesthetes’ rapid responses to targets in the grapheme blocks were possible because their arousal levels were higher, and they engaged anticipatory motor processes to a greater degree than in the color word and location blocks.

Another group difference observed in ERPs to targets concerned sensitivity to the visual contrast manipulation. Although both groups showed similar ERP contrast effects on targets in the color word and location conditions, synesthetes exhibited reduced sensitivity to contrast during the grapheme blocks. Both groups viewed the exact same target stimuli (e.g. lime), yet high contrast targets elicited larger amplitude P1 and N170 in controls, but not in synesthetes. The absence of contrast effects in synesthetes is consistent with our speculation that task performance in grapheme blocks required a high degree of sustained visual attention that compensated for differences in visual contrast.

Differences in visual evoked responses

In any case, the sensitivity of synesthetes’ early visual potentials to contrast has implications for the suggestion (e.g., by Barnett et al., 2008), that such responses reflect altered connectivity in their visual systems. Barnett and colleagues (2008) found that at contrast levels above 8%, flashing checkerboards elicited larger P1 components in synesthetes than controls. In the present study, synesthetes’ P1 components were less positive than those in control participants. These disparate findings may reflect differences in the way that checkerboards versus orthographic stimuli engage the visual system (Gauthier, 2000; Malach, Levy, & Hasson, 2000). Further, unlike our word stimuli, the Barnett et al. stimuli could not be mapped to any semantic content.

Barnett et al. (2008) suggested that their P1 group effect reflected increased parvocellular responsiveness in synesthetes, possibly arising from altered connectivity in early visual regions. The parvocellular responsiveness hypothesis predicts that synesthetes should evince larger amplitude P1, and be more sensitive than controls to the contrast manipulation in the present study. However, not only were our synesthetes’ P1 components less positive than those in control participants, visual contrast effects were either similar in the two groups (e.g., during the color word and location blocks), or were less evident in synesthetes than controls.

Visual contrast effects on ERPs to properties (stimulus 1) are also of interest, though effects on the graphemes should be interpreted with caution. In the color word and location conditions, decreasing visual contrast led to reduced amplitude P1 and delayed its onset by 30ms in synesthetes and controls alike. This 30ms delay was also present in synesthetes viewing achromatic graphemes, but not in controls viewing colored ones (see figure 2). As in target detection tasks involving colored bars (see e.g., Johannes, et al., 1995), the contrast manipulation here impacted controls’ P1 amplitude without modulating its latency.

Thus, when task performance required reading the words, the contrast manipulation delayed P1 onset in both synesthetes and controls. Onset delay was similarly present for achromatic graphemes in synesthetes, but absent for colored ones in controls. This suggests a subtle difference in the timing of synesthetic induction and color perception, in that synesthetic induction is more closely tied to visual factors that impact letter recognition. However, because the contrast manipulation here involved different physical stimuli for the synesthetes and controls, no firm conclusions can be drawn.

Immediately following the P1, the N170 component peaking at approximately 170ms was, on average, several microvolts larger in synesthetes than controls in every comparison (see figure 3). N170 effects have been reported in case studies of grapheme-color synesthetes (Sagiv et al., 2003; Kadosh, Kadosh, & Henik, 2007). To our knowledge, however, there is only one prior report of group amplitude differences in this component, unrelated to any experimental manipulation. Sinke, et al. (2014) recorded ERPs as synesthetes and controls viewed line drawings of objects with and without accompanying sounds. Drawings elicited larger N170 in synesthetes than controls, regardless of whether they were accompanied by sounds, and, when accompanied by sounds, regardless of their congruity with those sounds. Enhanced N170 amplitude in synesthetes’ ERPs can thus be observed without the induction of synesthetic colors, and could reflect structural differences between occipitotemporal generators engaged both by object identification and reading.

Unlike the P1 component, EEG and MEG activity in the N170 window is sensitive to experience-dependent word form processing in left occipitotemporal cortex (Brem, et al., 2010), can differentiate orthographic (letter) from non-orthographic stimuli (Bentin, et al., 1999; Tarkiainen, Cornelissen, & Salmelin, 2002; Tarkainen, Helenius, Hansen, Cornelissen, & Salmelin, 1999), and can even discriminate between non-word consonant strings, pseudo-words, and words (Kim & Lai, 2012; Kim & Strakova, 2012). Indexing visual processing important for reading, observed differences between the N170 in synesthetes and controls may be a direct manifestation of neural processes that contribute to synesthetic induction of color.

The associator-projector continuum, imagery, and visual processing

A recent model of grapheme-color synesthesia suggests that cross-activation occurs in “projector” synesthesia, in which synesthetic color experiences are more like perceptual ones, while top-down activations are more important for “associator” synesthesia, in which synesthetic experiences are more varied (van Leeuwen et al., 2011). Following van Leeuwen and colleagues (2011), the present study suggests that perhaps both models are correct, as feed-forward activation occurs in projector synesthetes, and feedback connections are more important for associators.

If the tendency to experience synesthetic colors in an external location (i.e. “projectors”) stems in part from superior ability to form a mental image (Simner, 2013), we might expect variation in both measures to modulate ERP components known to be sensitive to visual imagery. The lateral occipital N170 component elicited by visual objects is larger when participants first imagine objects versus see them (Ganis & Schendan, 2008), and colored pictures of common objects elicit smaller P1 components in people who report more vivid visual imagery (Hirschfeld, Feldker, & Zwisterlood, 2012).

In keeping with Simner’s (2013) suggestion, synesthetes who scored higher on the object imagery component of the cognitive style questionnaire also tended to score higher on the projector dimension of the associator-projector battery. Further, the synesthetes who exhibited a P1/N170 amplitude pattern most like the non-synesthete controls were those who reported a lower preference for visual imagery and scored higher on the associator dimension. Among synesthetes, but not controls, a greater preference for visual imagery style was systematically associated with reduced P1 and larger N170 amplitude. Similarly, synesthetes’ scores on the projector dimension of the associator-projector battery were linearly related to P1 and N170 amplitude, as projectors had the smallest P1 and largest N170.

Individual differences in our synesthetes’ reported experiences – that is, whether synesthetic colors were perceived as part of the graphemes, a patch floating above them, or merely an internal sensation of color – were correlated with the amplitude of their visual ERP components. This finding is in line with van Leeuwen et al.’s (2011) suggestion that differences in the subjective experience of synesthetes may be depend on whether retinotopically organized color areas contribute to the early stages of grapheme processing. The P1/N170 differences among synesthetes in the present study might reflect variation in whether the ventral stream activity manifested in visual ERPs includes contributions from V4. Consistent with this suggestion, Brang, et al., (2010) used MEG recorded from projector synesthetes to show MEG activation in V4 by 120ms, within 10ms of the onset of grapheme form processing. An intriguing possibility is that the enhanced amplitude of our synesthetes’ visual N170 in part reflects V4 cross-activation during grapheme perception.

Time course of the synesthetic color experience

We found a large amplitude increase on the N170 component for grapheme-color synesthetes. However, given that line drawings also elicit large amplitude N170 in synesthetes without a concurrent color experience (Sinke, et al., 2012), it seems unlikely that the underlying neural activity is itself sufficient to produce the subjective experience of synesthetic color. The present study supports the suggestion that a slightly later ERP component (frontcentral P2) reflects processes involved in the conscious experience of synesthetic color (Brang, et al., 2008, 2011). Indexing parietally mediated activity in the visual system, P2 components elicited by graphemes were larger in synesthetes than non-synesthete controls (as in Schiltz et al., 1999). Further, whereas controls did not exhibit differences in P2 amplitude across stimuli, all synesthetes exhibited a larger P2 for achromatic graphemes than achromatic words (see figure 4), presumably because graphemes produced the most intense synesthetic experience. While the stimuli in every condition were composed of graphemes and induced concurrent color sensations, pilot testing on several of the synesthetic participants confirmed that the grapheme strings uniquely facilitated a unified synesthetic color rather than a mixture of competing colors.

To conclude, we suggest that after about 110ms, the visual N170 ERP component in part reflects cross-activation of area V4 following achromatic grapheme perception in synesthetes, but that the probability or degree of cross-activation is modulated by the factors that underlie the associator-projector continuum. Like Brang et al., 2011), we suggest that the P2 component is a more direct index of the emergence of a conscious synesthetic concurrent, and is indifferent to whether V4 activation arises from top-down feedback from association cortex, or direct connections from neighboring visual regions. Given the ubiquity of reentrant projections and the import of inferential factors for all perceptual processes (Friston, 2010), synesthesia researchers might wish to move beyond disputes regarding “bottom-up” versus “top-down” processing, to address how these alternative hierarchical coding mechanisms support efficient reading in synesthetic brains.