The perceptual reality of synesthetic colors

Abstract

Synesthesia is a remarkable, rare condition where an individual has multimodal perceptual experiences from a unimodal sensory event. We have studied such an individual, an adult male for whom achromatic words and alphanumeric characters are seen in vivid, reliable colors. We used a variety of perceptual tasks to document the perceptual reality of synesthetic colors and to begin to localize the stage of visual processing where this anomalous binding of externally specified form and internally generated color may take place. Synesthetic colors were elicited by forms defined solely by binocular cues or solely by motion cues, which implies a central locus of visual processing for synesthetic binding of form and color. Also included among our measurements was a difficult visual search task on which non-synesthetic subjects required an effortful search through the visual display. Our subject, in contrast to non-synesthetic subjects, accomplished the task with relative ease because the target of the search had a different synesthetic color from the distractors. Thus, synesthetic experiences appear to originate from a binding of color and form that takes place within central stages of visual processing.

Different aspects of the visual scene—color, shape, optic flow—are registered in widely distributed, richly interconnected brain areas (1–4). However, this distributed representation of object properties is rarely evident in perceptual experience: the visual appearance of objects is characteristically unitary and coherent. How, then, is distributed neural activity within diverse brain areas coordinated, or “bound” together, in the interests of object recognition and identification? This is the so-called “binding problem,” and it has received extensive discussion in recent years (e.g., ref. 5).

Although a fundamental property of visual processing, understanding how and when binding occurs in the visual system has proved quite difficult. One way of studying the binding process is to exploit stimulus conditions where anomalous binding tends to occur in normal observers, such as imposed attentional loads that can produce illusory conjunctions (6). Another strategy is to examine unusual individuals for whom anomalous binding regularly occurs (7). We have used this strategy by administering a battery of tasks to an individual who experiences the rare condition termed synesthesia, whereby anomalous binding creates perceptual experiences outside of reality (8–13). This adult male (WO) has “lexical synesthesia” in which achromatic words and numbers reliably appear colored, and results from our studies imply that these colors are bound to visual forms during visual processing itself.

We have determined under what conditions WO's synesthetic experiences can be elicited by using displays that isolate binocular mechanisms and displays that isolate motion mechanisms. Furthermore, we have examined how WO's performance on several perceptual tasks is affected by his obligatory, synesthetic colors, by using tasks for which real colors either facilitate performance or hinder performance in normal observers. Results from these experiments document the perceptual reality of synesthetic colors and provide pointers to the possible neural locus of synesthetic binding within the visual processing system.

Experiments and Results

An Initial Characterization of WO's Synesthesia.

WO is an adult male who has experienced lexical synesthesia since early childhood: for him, individual words and alphanumeric characters printed in black-and-white evoke vivid, reliable colors. For that matter, spoken words also evoke color sensations, although WO sees these colors in his “mind's eye.” His late mother, maternal grandfather, and maternal great uncle also experienced synesthesia throughout their lives, but neither his children nor his siblings possess this unusual perceptual ability although “they tried repeatedly” to learn how to see colors in words. WO has normal visual acuity and good stereopsis as assessed by the Bausch and Lomb Orthorator. Based on his performance on the Munsell color arrangement test and the Ishihara plates, it is evident that WO possesses normal trichromatic color vision.

To document the stability of WO's color associations, we started with a phonetically balanced (14) list of 100 common monosyllabic words. In two test sessions separated by more than a month, we asked WO to provide the synesthetic color elicited by each of the words in this list. For comparison, we administered the same test twice during a 2-week period to 22 undergraduate volunteers, instructing them to “provide the color that seems to go with each word” on both occasions. Results confirmed that WO's associations were enduring, with his pairings between words and color descriptors being remarkably consistent: he was 97% consistent across the two repetitions, with the only “errors” being substitutions of beige for off-white or light brown. This result compares with just 43% average consistency for non-synesthetic subjects tested in the same way (range 21% to 60%). To be certain that these associations truly involved color experiences—not just color naming—we asked WO to select the particular color that matched his synesthetic experience when individually viewing each of 12 words. His selections were made by using a Pantone color palette available in adobe photoshop 6.0 (Adobe Systems, Mountain View, CA). On two test sessions separated by 10 days, WO selected identical colors both times. Because many of the experiments reported below used digits as stimuli, we also asked WO to select the particular colors associated with the digits 2–9 (0 and 1 do not elicit vivid chromaticities), using a complete Pantone database containing more than 1,000 color samples. Fig. 1 displays the color selections he made on two occasions separated by more than a week. In all cases, the synesthetic color experienced by WO depends entirely on the physical form of the visual input, not its meaning; for example, 2 is orange but “two” is blue, and 9 is yellow but “nine” is orange.

Figure 1.

Synesthetic colors associated with the digits 2–9 by WO on two separate occasions.

Synesthesia Elicited by Local and Global Forms.

To study the range of conditions eliciting synesthesia in WO, we tested him by using several unusual formats for portraying alphanumeric characters. In one test, we created large characters that were made out of small characters (15), such as a 5 made out of a large number of small 2s (see Fig. 2a). When WO attended to the global form, he reported the synesthetic color of the global form (e.g., light green for a 5), but when he was asked to instead attend to the local forms, he reported that the synesthetic color suddenly switched to that of the local forms (e.g., orange for the 2s).

Figure 2.

(a) An illustration of synesthesia elicited by local/global forms. (b) An illustration of a cyclopean numeral (a “2” in this example) created by random dot stereoscopy (readers with access to red-green glasses can confirm the presence of the digit). (c) An illustration of a numeral defined by motion. The arrows indicate the directions in which dots moved within regions of the animation defined as “figure” and as “background.” In the actual animations, all dots—those defining the figure and those defining the background—were black against a white background, the numeral being defined solely by differential motion of the dots. In this schematic, different color and shading are used for illustrative purposes only.

Synesthesia from Binocularly Defined Stimuli.

In another test, we created dichoptic figures in which complementary parts of letters or digits were presented separately to the two eyes, thus requiring binocular integration for recognition of the complete character. When viewing these displays, WO readily perceived “colored” characters, suggesting that his synesthetic associations arise at a central level of visual processing after information from the two eyes has been combined. In a more critical test of this conclusion, we created individual digits by using random-dot stereograms (16), whereby the “digit” was visible by virtue of the disparity between clusters of a subset of dots in the two half-images of the stereogram (see Fig. 2b). These stereograms were presented to WO by using red/green glasses to achieve anaglyphic stimulation. Over a series of test presentations, not only did WO readily identify all of the digits, he immediately noted that each appeared colored, with the hue of a digit being the same as that associated with normal, luminance-defined digits. Synesthesia elicited by these “cyclopean” characters definitively points to a central locus for the unique binding of color and form.

Synesthesia from Motion-Defined Stimuli.

In another experiment, we presented WO with numerals defined exclusively by visual motion information. The animation portraying these numerals consisted of 400 small, black dots appearing within a rectangular region 4° × 6°. All dots falling within a virtual “numeric” region of the display moved up and to the right, and all dots falling outside this region moved down and to the left. Once a dot left the virtual “numeric region” it was extinguished and replaced by a new “figure” starting at the opposite edge of the “numeric region.” Background dots were similarly extinguished and replaced, the result being an essentially constant number of dots moving in the two opposite directions. On each 1-s presentation, the shape of the “numeric region” closely resembled one of the digits 2–9 (see Fig. 2c). Thus, the “number” was defined solely by differences in direction of motion, and over a series of 1-s presentations, WO reported what color, if any, he experienced when viewing different “numbers.” Again, WO readily identified the digit and immediately saw his associated color for each of the digits. Normal observers viewing these same displays saw the numerals with no hint of color. The evocation of synesthetic colors by these motion-defined contours implies that areas of the brain presumably involved in the computation of structure from motion (e.g., ref. 17) interact with brain regions that synthesize color in WO.

Stroop Interference by Synesthetic Colors.

In a standard Stroop task (18, 19), subjects experience significant interference when naming ink colors of written words when those words are color terms incongruent with the ink color (e.g., “red” is written in green ink). Such results imply that word reading is automatic and leads to interference with color naming. In a modified Stroop task, we looked for interference when naming ink colors when the words gave rise to a synesthetic color that was incongruent with the ink color. Such results imply that synesthetic binding is automatic and leads to interference with color naming as well. WO and non-synesthetic controls viewed a series of different-colored, monosyllabic words, the task being to name the color of the ink in which each word was written as quickly and as accurately as possible (words were displayed via computer, so “ink” actually refers to the color of the text on monitor). On some trials (congruent), words were written in ink colors that were congruent with WO's synesthetic colors for those words. Thus, for example, the word “moose” was printed in pink ink and the word “death” was printed in green ink. On other trials (incongruent), words were printed in ink colors incongruent with WO's synesthetic colors for those words. Thus “charge” (which for WO is blue) was printed in green ink and “pest” (which for WO is yellow) was printed in pink ink. On other trials (control), strings of nonalphanumeric characters ($, %, &, etc.) were printed in a particular ink color (punctuation marks provide no synesthetic colors for WO). We measured the time taken to name the color of each word by using a computerized voice key that provided millisecond-accuracy timing. For non-synesthetic controls, there was no significant difference in naming ink colors in the congruent (M = 641 ms, MSe = 46.8 ms), incongruent (M = 654 ms, MSe = 47.7 ms), and control (M = 630 ms, MSe = 8.4 ms) conditions. By contrast, like other individuals with lexical synesthesia (20–22), WO was significantly slowed when the ink colors were incongruent with his synesthetic experiences (M = 1,124 ms, MSe = 91.4 ms) compared with the congruent (M = 786 ms, MSe = 65.5 ms) and control (M = 812 ms, MSe = 24.7 ms) conditions.

In another modified Stroop task, we asked WO instead to name his synesthetic colors associated with each word as quickly and as accurately as possible (non-synesthetic controls cannot provide such judgments). The congruent and incongruent trials were the same as those used above, with words displayed in colors congruent or incongruent with WO's synesthetic experience. The control trials consisted of words printed in a neutral gray. WO was significantly slowed to provide his synesthetic colors when the ink colors were incongruent with his synesthetic experiences (M = 1,107 ms, MSe = 31.1 ms) compared with the congruent (M = 881 ms, MSe = 22.0 ms) and control (M = 847 ms, MSe = 17.4 ms) conditions. It is interesting to note that WO was just as fast to name his synesthetic colors associated with words printed in a neutral gray as to name his synesthetic colors associated with words printed in their appropriate synesthetic color.

Although demonstrating that binding of synesthetic colors to forms occurs automatically, results from these modified Stroop tasks do not reveal whether this binding occurs during visual processing or during later more conceptual processing. To address this particular issue, we next tested whether WO's color synesthesia could facilitate his performance on a difficult digit search task that is thought to tap visual processing (23). As with the Stroop task, we sought to learn whether WO's synesthetic colors associated with viewing achromatic forms would yield effects comparable to those produced when non-synesthetic subjects viewed chromatic forms.

Increased Visual Search Efficiency from Synesthetic Colors.

This experiment is motivated by the well-known observation that an object of one color stands out conspicuously within an array of background objects of a sufficiently different color. It is commonly believed that this represents a form of stimulus-driven, attentional capture (24–27). Are WO's synesthetic colors sufficiently salient and are they bound to visual forms at a sufficiently early stage of processing to lead to an efficient visual search in a similar fashion? The answer appears to be yes; when first shown a visual array consisting of a white embedded among a set of white s (Fig. 3a), WO commented that the target digit seemed to “pop out” from the distractors because for him it was a different color (Fig. 3b). We documented this important observation by performing a visual search experiment in which WO and seven non-synesthetic volunteers searched arrays of white digits seen against a black background, judging as quickly as possible whether a designated target appeared among a variable number of identical distractors (see Fig. 3a).

Figure 3.

(a) An example of an actual search display used in the visual search experiment. (b) A depiction of this search display with WO's synesthetic colors. (c) Correct target-present response times as a function of display size for non-synesthetes. (d) Correct target-present response times as a function of display size for WO.

Each subject searched for a among s or an among s. Displays contained 16, 25, or 36 digits. A display was created by starting with a 7 × 7 grid and randomly placing the target (if present) and the requisite number of distractors throughout the grid. Once placed on the grid, the actual location of a digit was moved in a random direction by a random amount (the maximum distance moved was selected so as to guarantee that no two digits would ever overlap one another). At a typical viewing distance, each digit subtended ≈1.2° × 0.6°, and the maximum extent of the display of targets and distractors was ≈14° of visual angle. On each trial of the experiment, if a target was found, the subject was instructed to respond PRESENT as quickly as possible without making an error, and otherwise to respond ABSENT. Responses were recorded by key presses on a computer keyboard equipped with a millisecond accuracy timer. The different target-distractor sets were presented randomly throughout the experiment, subject to the constraint that each set was used the same number of times within a block of trials. A target was present on half of the trials within each block. Each session was divided into four blocks of 120 trials, with a rest break between each block. Each subject completed two experimental sessions for a total of 960 search trials.

For all non-synesthetic subjects, search times increased linearly with set size regardless of target-distractor pairing; this finding simply verifies the difficulty of the two search tasks (Fig. 3c). For every non-synesthetic subject, searching for a among s was as difficult as, or more difficult than, searching for an among s, as evidenced by numerically steeper search slopes and numerically longer average response times for every individual subject. WO, too, experienced difficulty searching for an among s, which both appear bluish to him (Fig. 3d). By comparison, when searching for a among s, however, WO showed a significantly smaller effect of set size on search time and made significantly faster responses (Fig. 3d). He several times volunteered that the task was simplified by the “orange” color of the against the backdrop of the “green” s. Indeed, he described the search process as one of oftentimes noticing a uniquely colored patch that drew his attention to the target digit. WO's overall error rates were comparable to those of non-synesthetic subjects, and he made numerically fewer errors when searching for a among s than when searching for an among s, ruling out a speed/accuracy tradeoff.

Unlike true pop-out results often produced by real colors in visual search (23), the slope of WO's search function for a among s was not completely flat. One possible explanation for this result is that WO may perform a serial-like search through the visual display, like non-synesthetic individuals, but is able to reject distractors far more quickly because they have the wrong synesthetic color, unlike non-synesthetic individuals. To assess this possibility, we next tested WO and non-synesthetic controls on visual search displays in which the distractors had no synesthetic color. On each trial, subjects searched for either a (which is orange for WO) or a (which has no color for WO), responding as quickly and as accurately as possible as to which target was present in the display. The distractors on every trial were 36 s (which have no color for WO). In this case, both WO and non-synesthetic controls showed no difference in searching for either type of target among the nonsense distractors.

Thus, binding of synesthetic colors to forms does not seem to occur in parallel across the entire visual field. The pop-out effect reported by WO may not be exactly analogous to the pop-out effect experienced with real colors. Rather, it appears that, as attention is allocated to a part of the visual field (a single digit or a small cluster of digits), synesthetic color seems to be bound to a visual form as that form is being recognized, allowing WO to quickly reject items that have the wrong synesthetic color. This may have given WO the impression of a pop-out when searching for a among s because the search was so much more efficient than searching for an among s.

Conclusions

Interference of synesthetic colors with color naming in a Stroop task indicates that binding of synesthetic colors to forms occurs automatically and cannot be suppressed even if it interferes with performance (20–22). But such Stroop interference is not informative as to when this anomalous binding of color and form takes place. Indeed, such Stroop interference could occur even if synesthetic colors have no perceptual reality whatsoever.

Our other results do argue for the perceptual reality of synesthetic colors and provide some pointers to where this anomalous binding of color and form could (and could not) occur in the visual system. Like other synesthetes, WO has a remarkable consistency and specificity in selecting his synesthetic colors associated with words and alphanumeric characters, consistent with his claims of the perceptual reality of his synesthetic experiences. Our results on synesthetic binding to cyclopean characters generated by random dot stereograms and our results on synesthetic binding to motion-defined characters do limit the earliest extent of binding of synesthetic color to forms, in that it must take place after binocular fusion and after shape from motion has occurred within the visual system. By “after,” we mean that form information defined by disparity or defined by motion must be synthesized before an alphanumeric character can take on its unique color. It is entirely possible, of course, that this form information, once synthesized, can flow to visual areas that register color information. Indeed, just such an idea has been advanced by Smilek et al. (28) to account for their color synesthete's relatively poor performance at digit recognition when the synesthetic color of a “target” digit matched the real color of the background against which it appeared. Our visual search results are entirely congruent with the findings of Smilek et al. (28).

A recent functional brain imaging study reported BOLD (blood oxygenation level dependent) signal modulations during synesthesia within the earliest cortical visual area, area V1 (29). This observation is somewhat puzzling, for it is generally thought that color information is explicitly represented not in V1 but, rather, in higher visual areas within the ventral stream (e.g., ref. 30). In the present study, WO's immediate, vivid synesthetic perceptions could be triggered readily by motion-defined and by disparity-defined numerals. What does this imply about the involvement of V1 in synesthesia? Functional MRI (fMRI) results indicate that shapes from disparity in random dot stereograms (31, 32) and shapes from motion (33) are registered in higher visual areas, not area V1. This result, in turn, implies that synesthetic binding cannot be occurring at such an early stage of visual processing. It is possible, of course, that feedback pathways from higher visual areas to area V1 produce modulations in neural activity during synesthetic perception.

Our visual search results also implicate a central stage of visual processing as a more viable locus for synesthetic binding of color and form. In order for synesthetic colors to assist visual search, converting a very inefficient search into a more efficient search (34), binding of color and form must take place before the explicit recognition of elements within the visual display. If a or a must be explicitly recognized as a two or a five before synesthetic colors could be bound to those digits, then those digits can be selected or rejected immediately without using synesthetic color to guide the search. In that case, no search advantage for our synesthete would be expected. Instead, it appears that the binding of color and form takes place during the process of form recognition itself, with synesthetic colors available before the explicit recognition of the digits. Our synesthete was able to quickly reject distractor digits on the basis of their synesthetic color more quickly than our non-synesthetes were able to reject distractor digits on the basis of their identity.

Our conclusions are somewhat at odds with those of Mattingly et al. (21). They had synesthetes name the color of a target patch that was preceded by an alphanumeric prime. When the prime was presented for half a second, a normal Stroop-like interference pattern was observed, in that synesthetes were slowed when the prime had a different synesthetic color from the color patch to be named. However, when the prime was presented very briefly, so that it rarely could be explicitly identified, this Stroop-like interference disappeared. The briefly presented alphanumeric characters did interfere with performance in a separate character naming task, demonstrating that, although briefly presented, these alphanumeric characters were processed, perhaps unconsciously, to a sufficient degree to influence performance on other kinds of tasks. Mattingly et al. concluded that overt recognition and attentional selection of inducing stimuli are crucial for synesthesia to emerge. Again, it is difficult to understand how synesthesia could improve visual search performance if a digit needed to be overtly recognized before synesthetic color could be bound to that digit. Once selected and recognized as a target or a distractor, the search can either terminate or move on to the next digit in the display without waiting for the synesthetic color to emerge. But, for the largest display sizes, WO was nearly 500 ms faster on average to find a target that differed from the distractors in synesthetic color. It seems more likely that synesthetic color influenced what was actually selected or rejected during the search itself.

Several models of synesthesia have been proposed, all within the neuroanatomical framework consisting of hierarchically organized cortical sensory pathways (11). Our results are consistent with a version of this model in which synesthetic binding arises from anomalous cross-wiring between color and form areas specifically (and between neighboring cortical maps more generally), as recently suggested by Ramachandran and Hubbard (35). Among other results, they observed that lexical synesthetes showed Gestalt-like grouping of ambiguous displays on the basis of synesthetic color, allowing them to more readily detect global forms composed of particular letters among a set of distractors when the letters differed in synesthetic color. Our results go beyond this finding by documenting how the synesthetic binding of color and form unfold during visual information processing. For one thing, we can conclude that synesthetic binding of the sort exhibited by WO occurs before the explicit recognition of form information, for otherwise synesthetic colors would be irrelevant in the visual search task. For another, we show that this form information can arise from contours defined by stereopsis, motion, or luminance. The most parsimonious account of this finding is that synesthetic binding involves activation of cue-invariant form processing areas. Such areas do indeed exist, most notably the lateral occipital complex (36).

Thus, a convergence of evidence indicates that binding in lexical synesthesia occurs during central visual processing and not during later more conceptual processing (although other “higher” forms of synesthesia based on conceptual information rather than visual form information are also possible; see ref. 13). This finding supports the impressions made by synesthetes, such as WO, that their anomalous perceptual experiences are truly perceptual in nature, and are not analogous to forms of cognitive associations between form and color that non-synesthetes may regularly experience.