BINDING, SPATIAL ATTENTION AND PERCEPTUAL AWARENESS

Abstract

The world is experienced as a unified whole, but sensory systems do not deliver it to the brain in this way. Signals from different sensory modalities are initially registered in separate brain areas —even within a modality, features of the sensory mosaic such as colour, size, shape and motion are fragmented and registered in specialized areas of the cortex. How does this information become bound together in experience? Findings from the study of abnormal binding — for example, after stroke — and unusual binding — as in synaesthesia — might help us to understand the cognitive and neural mechanisms that contribute to solving this ‘binding problem’.

Different areas of the cortex receive sensory information through different receptors (for example, the eyes, ears or touch receptors), showing that different areas of the cortex respond to different features of the sensory mosaic. Investigations of multimodal integration have a long history, but a particularly interesting binding problem arose when reliable evidence began to emerge that, within the brain, different areas are specialized for encoding different features of the visual modality, such as colour, shape, size and motion in vision¹. Electrophysiological recordings from monkey cortices showed that visual neurons in different areas responded with different strengths to different features². This specialization has received support from functional imaging studies in humans³. Moreover, this evidence is consistent with more than a hundred years of neuropsychological reports that lesions in different areas of the human brain can result in relatively isolated deficits in visual processing — for example, ACHROMATOPSIA or visual NEGLECT^4,5. This modular organization of the brain led to the question of how features that are registered separately are reunited to produce our unified experience of the world^6,7. Some researchers have argued that this type of ‘binding problem’ is not a problem at all^8,9, but recent evidence has shown that it can become a real problem in everyday life when certain areas of the brain are no longer functioning¹⁰.

In this review, I will discuss evidence — from both normal perceivers and neurological patients — that binding features within vision is a problem in humans, and is not just a theoretical construct. I will then describe the role that spatial attention is believed to have in this type of binding, emphasizing the spatial functions of the parietal lobes¹¹. Finally, I will explore how binding might change if two or more specialized FEATURE MAPS were to merge. This final issue has been approached within a cognitive neuroscience perspective by testing shape/colour synaesthetes — otherwise normal individuals who perceive internally generated colours (induced by particular visual shapes) as if they were being received by sensory receptors (so that the colours are perceived as ‘out there’)¹². For instance, the letter A might induce a certain shade of blue. It has been hypothesized that synaesthesia results from developmentally abnormal connections between specialized cortical feature maps¹³. I will review the experimental evidence in this area and discuss how it might relate to theories of binding as well as to the conscious perception of objects.

The binding problem is a real problem

As early as 1977, cognitive psychologists were intrigued by findings that colours and letters could be miscombined in normal perception¹⁴. Coloured letters (for example, a red O, blue T and yellow X), shown briefly on a computer screen, could have their features recombined and be perceived with high confidence as, say, a blue O, yellow T and red X. Normal perceivers could experience the illusion that an X was conjoined with the colour red only when red was a property of another letter in the display; hence the term ‘illusory conjunction’¹⁵. Illusory conjunctions occur when spatial attention is diverted, when displays are briefly presented, and/or with peripheral presentation where spatial resolution is decreased^16,17. They cannot be attributed to guessing¹⁸. Although input must be limited for normal perceivers to produce conjunction errors, illusory conjunctions are an example of a ‘real’ binding problem (Fig. 1).

Figure 1. Illusory conjunctions.

Features such as colour and shape (letters in this figure) can recombine to form an illusory conjunction in perception. Illusory conjunctions can be corrected for guessing by using intrusion errors (for example, the report of another letter or colour in the stimulus set). Intrusion errors seldom occur, for either normal perceivers or patients with spatial deficits. More elaborate corrections for guessing have been proposed (REF. 18), but true illusory conjunctions, like the example shown, have still been supported.

Why might the visual system recombine features of shape and colour incorrectly? Investigators of visual information processing had long studied how wholes and parts could be integrated into the perception of a shape or object, but the idea that a surface feature such as colour could be erroneously attached to a shape had not been considered. Experiments in neuroscience had not yet demonstrated specialized cortical feature maps within vision, cognitive psychologists had not yet ‘discovered’ neuropsychological syndromes and the discipline of cognitive neurosciences had not yet been conceived. But within this context, a theory of feature integration (FIT) was proposed in which different properties of the visual input were encoded in separate feature maps and were combined in perception by integrating separate features through spatial attention¹⁹.

This theory predicted that deficits in spatial attention would produce feature binding problems, and they do. A stroke that leads to unilateral visual neglect or EXTINCTION²⁰ results in spatial attention being diverted to one side²¹, and stimuli presented on the other side (when detected) produce elevated rates of illusory conjunctions²². Consistent with this, studies using visual search methods (FIG. 2) with the same population have shown that detecting a target on the neglected side is a problem when it is the conjunction of two features (such as colour and shape), but not when detecting basic features (colour or shape alone)^23,24. When the movement of attention into an area of space is disrupted, visual search for a conjunction is impaired and illusory conjunctions are increased in the unattended areas of space. Unilateral neglect comes in various forms and severity²⁵, but the common denominator is that patients miss information presented on the contralesional side²⁶. Increased binding problems in these patients therefore occur under similar conditions to those observed in normal perceivers when experimental manipulations are used to divert attention. In neglect, nature has intervened to divert attention away from stimuli on the side of space contralateral to the lesion.

Figure 2. Examples of displays used in visual search studies.

Detection of a red X among green distractors is easy when colour or shape alone can be used to discriminate the target from distractors, whereas detecting the red X among green Xs and red Os is more difficult. For the feature search display feature maps that detect colour and those that detect shape need not interact to perform the task, while in conjunction search they must. The time taken to detect a feature is typically independent of the number of distractors (as the target ‘pops out’) whereas the time taken to detect a conjunction increases linearly as the number of distractors increases. According to feature integration theory conjunction search is more difficult because spatial attention is needed to co-localize the two features.

The most extreme example of binding errors occurs when both parietal lobes are damaged, resulting in a condition known as Balint’s syndrome²⁷, in which spatial information can disappear almost completely (BOX 1). Although patients with this condition continue to perceive one object (but only one, a symptom known as simultanagnosia) at any given moment, they do not know where it is located. They act as though they have no external spatial frame of reference on which to hang the objects they see. Obviously, if the brain no longer computes a spatial map, the use of space to localize, individuate and select a location for attention should be impossible. Furthermore, if feature binding requires the co-localization of features through spatial attention to form a correct conjunction, features such as colour, shape, motion and size should randomly combine when spatial information is absent. However, detection of a feature in a display should remain possible because binding would not be necessary. Even binding to a location would be unnecessary to know whether a particular feature was present, as detectors in specialized cortical feature maps should continue to signal the presence of their preferred stimulus properties.

Box 1. Balint’s syndrome.

Balint’s syndrome is a neuropsychological disorder that results from damage to both parietal lobes⁷⁷. Clinically, it includes three main symptoms: simultanagnosia (the inability to see more than one object at a time); optic ataxia (the fixation of gaze with severe problems in voluntarily moving fixation); and optic apraxia (the inability to reach towards the correct location of perceived objects)⁷⁸. Balint’s syndrome is seen in some cases of dementia⁷⁹, but in these cases it is typically accompanied by confusion, memory loss and other symptoms. For this reason, cognitive investigations have been most fruitful when testing patients with damage produced by bilateral stroke affecting both parietal lobes, but leaving primary sensory processing, language, memory and judgment intact (see illustration of the brain below showing a reconstruction of RM’s lesions). Patients with Balint’s syndrome lose spatial information outside their own bodies and are functionally blind except for the perception of one object in the visual scene at a time. Their spatial abilities are so impaired that they cannot locate the item they can perceive, nor can they tell when an item is moved towards or away from them. They lose explicit spatial awareness, resulting in the defining symptoms of Balint’s syndrome. It is as if “there is no there, there”⁴⁵.

It has been argued that Balint’s syndrome is a type of double neglect⁸⁰. Unilateral neglect most often arises from damage to right hemisphere parietal areas, producing inattention to the left side of displays and objects. For instance, a patient with neglect might overlook the left side of a room but also the left side of a flower⁸¹. Balint’s patients neglect both sides of the room but they can see a single object. They do not neglect a portion of the objects they see. In fact, they see nothing but objects. So the relationship between the two might not be as straightforward as it first seems.

In addition, although the areas of damage in the two syndromes can overlap, this might be more a consequence of their proximity than their function. Balint’s syndrome has been most often associated with occipital–parietal damage and centred in the angular gyrus⁸², but neglect has been more often associated with temporal–parietal damage and centred in the supramarginal gyrus⁸³. It therefore seems plausible that unilateral neglect and Balint’s syndrome reflect different underlying deficits. It is possible that parietal areas along the angular gyrus are more involved in computations of space itself, while supramarginal areas support attentional selection mechanisms that operate on that space⁴⁵.

A series of studies with a patient (RM) who has bilateral parietal damage and severe spatial deficits confirmed these predictions^10,11. RM is a neurological patient who suffered two sequential strokes in the occipital–parietal region of the right, then left hemisphere, producing nearly symmetrical lesions. As a result, he showed all of the symptoms of Balint’s syndrome (BOX 1). Notably, the calcarine cortex, both temporal lobes, the somatosensory and motor cortices, and both frontal lobes were anatomically intact, as were both supermarginal gyri of the parietal lobes.

To examine illusory conjunction rates, we presented two coloured letters on a computer screen for up to 10 seconds, and asked RM to report the letter he saw and its colour. In initial testing sessions, he produced 38% illusory conjunctions. It is unclear why this rate was less than chance (50%). However, the high rate with long exposures is impressive evidence for a binding problem, especially considering that RM was looking directly at the screen, was focused on the task for up to 10 seconds and was highly motivated to perceive the stimuli correctly. RM produced elevated illusory conjunction rates under similar conditions for motion and shape, colour and shape and colour and size^10,11,28. Elevated illusory conjunction rates have been confirmed in other patients with Balint’s syndrome²⁹.

According to FIT, increased rates of illusory conjunctions in free viewing should also lead to problems in visually searching for a conjunction of two features in a cluttered display, because this would require co-localization of the two features through attention and so would require binding (FIG. 2). When an explicit spatial map is missing, the spatial localization required to conjoin the features is deficient and conjunction search should suffer, but detecting individual features should remain easy because this only requires a signal that the feature is present.

These predictions were confirmed. When the target was a conjunction, RM’s attention was captured, seemingly randomly, by either the target or one of the dis-tractors. He was unable to move attention voluntarily to another item in the display and therefore saw the target only if it happened to be the first item that entered his awareness. However, when the target was a distinct feature, it became the item that entered his awareness. Nevertheless, consistent with his diagnosis of Balint’s syndrome, he was no better than chance at locating either the feature or the conjunction.

In summary, behavioural evidence shows that there is a real binding problem to solve. Normal perceivers see illusory conjunctions when spatial attention is diverted, as do patients with unilateral damage that diverts attention to one side of space. Most striking is that when bilateral damage disrupts the computation of an external spatial map, binding becomes a prevalent problem in everyday life, leaving patients with multiple feature signals that can combine inaccurately in perceptual awareness.

Potential solutions to the binding problem

I have discussed binding in vision as if there is only one binding problem to be solved, but of course, even within this modality, there are many³⁰. Binding objects to each other to form a unified scene, binding features across time and binding parts together to form what we see as an object all require integration³¹. Also, frequent exposure to a conjunction stimulus (for example, colour and shape) can produce binding that does not require attention. This exposure seems to stimulate the formation of new feature maps.

The binding problems observed with parietal lesions are spatial but not temporal. Binding of features across time remains intact, even when the overall time to process two stimuli is reduced for displays separated in time rather than in space¹⁰. When the red A and blue X of FIG. 1 are presented sequentially in the same location, illusory conjunctions do not emerge. Balint’s syndrome also does not produce problems in binding contours into a single shape, such as the four lines and angles of a square, or binding occluded parts to form an object, such as a square behind an occluding circle³². Patients are equally likely to see a square, whether it is in front of or behind the circle. These binding problems can be solved without the attentional mechanisms of the parietal lobes, but spatially binding objects to each other, and binding surface features to shapes, does require these parietal attentional mechanisms³³. Even with large lesions in both parietal lobes, VENTRAL PROCESSING STREAMS seem to solve several binding problems, one of which is to form individual objects. The spaces that support object binding must be implicitly present for this to occur, and implicit spatial abilities have been demonstrated even in patients with Balint’s syndrome¹¹.

Lesions of other regions that are involved in spatial attention also disrupt feature binding. In one case, damage to the pulvinar of the thalamus (which is strongly connected to the parietal lobes) produced a lesion that affected spatial functions (without neglect or extinction) in one visual field but not the other. In this case, feature binding errors also increased in the affected field, but not in the unaffected field³⁴. The extent to which other spatial areas are involved in this type of feature binding is not known, but the evidence suggests that spatial areas that seem to be part of a cortical–subcortical network are crucial for properly binding features registered in specialized ventral cortical feature maps.

The neuropsychological evidence places limits on potential solutions to the binding problem but also clarifies the roles of important cognitive and neurobiological players in the integration of information from specialized cortical feature maps. The DORSAL PROCESSING STREAM, thought to determine ‘where’ or ‘how’³⁵, was obviously disrupted in both hemispheres in RM, whereas the ventral processing stream, thought to process ‘what’, was anatomically intact³⁶. As predicted, registration of different features, such as shape and colour, remained intact even with severe spatial deficits from large parietal lobe lesions, but the features were often bound incorrectly without spatial input from the dorsal stream.

Space and binding

Potential solutions to the binding problem can be approached from both cognitive and neurobiological levels of analysis, but the evidence I have discussed so far suggests that, whatever level is favoured, spatial information must be considered. At a cognitive level, one solution to the binding problem has been that spatial attention is involved in accurate feature binding (as first expressed, for example, in FIT). FIT has been modified over the years since its introduction³⁷, but the proposition that spatial attention is crucial for correct feature binding has remained. The theory has been the topic of some debate, especially regarding the question of whether features are processed in a qualitatively or quantitatively different way from conjunctions^38–40. But the fact that spatial attentional deficits disrupt binding, while leaving feature detection relatively intact, lends strong support to the premise that feature detection and conjunction detection occur in qualitatively different ways. The evidence from neuropsychology also supports the proposition that feature detectors can function without spatial awareness but that detecting a conjunction of features requires explicit spatial information⁴¹.

Evidence from functional imaging studies with normal perceivers supports the role of the parietal lobe in binding. The superior parietal cortex is activated when subjects search for a conjunction target in a cluttered display, but not for a feature⁴². When subjects are asked to find a conjunction of colour and motion, both posterior temporal areas and parietal areas show increased activation over baseline control conditions, but when they search for the presence of either colour or motion in a display, only the temporal lobe areas are activated. When features are present in a display, whether in conjunction or feature search, the temporal lobes register their presence, as would be expected. However, when finding the target requires binding, parietal functions are recruited. Consistently, TRANSCRANIAL MAGNETIC STIMULATION (TMS) over parietal areas in normal perceivers disrupts conjunction search but not feature search⁴³. Although parietal involvement in searching for a conjunction can change as a function of practice⁴⁴, and RM’s ability to bind improved over time, this might reflect the recruitment of spatial maps in areas outside the occipital–parietal cortex⁴⁵, as processing becomes more automatic.

Functional imaging, TMS and neuropsychological evidence all support the claim that posterior interactions between dorsal and ventral pathways (what and where) are necessary to produce normal binding of surface features¹¹. Parietal functions provide the spatial coordinates that allow attention to co-localize features as well as to separate object shapes. When both parietal lobes are damaged, the object that is perceived is not spatially related to other objects, and the other objects drop from awareness. It is not that objects require perception of space to rise above the threshold of perceptual awareness — both features and objects can be perceived without spatial information provided by parietal input — but rather that awareness of the relations between features represented in different cortical feature maps and the spatial relations between objects requires explicit spatial information.

Opponents of this view argue that parietal functions do not contribute to binding per se, but rather are increasingly active when task difficulty increases. These arguments have been made mostly on the basis of demonstrations that searching for a conjunction target in a cluttered display can be made easy and searching for a feature can be made difficult, reversing psychophysical functions and affecting the degree of parietal activation⁴⁶. There is no question that making a target more difficult to locate can engage the spatial attentional mechanisms of the parietal lobes^47,48. A salient stimulus should not require spatial attention, and practice is one method that can increase stimulus salience. Binding only becomes a problem when the assignment of features to a common location is ambiguous.

The difficulty argument was recently tested by equating difficulty of features and conjunctions in a matching-to-sample task. In this functional magnetic resonance imaging (fMRI) study, only conjunctions increased activation in parietal areas⁴⁹. Regions of interest were first identified using an attentional procedure. Not surprisingly, these regions were in the superior parietal cortex and intraparietal sulcus, areas that are commonly activated in spatial attention studies⁵⁰. In the second part of the experiment, where difficulty was controlled, conjunctions increased activity in both of these regions but features did not.

Another leading theoretical account of binding, the biased competition model (BCM), proposes a continuum of competitive interactions between neurons and their preferred features⁵¹. According to this theory, a conjunction takes longer to find as more distractors are added to the display. Feature search is unaffected by the number of distractors because conjunction displays are more likely to induce competitive interactions within and between cortical feature maps.

In this view, the more difficult it is to discriminate the targets and distractors — that is, the less salient the target is — the more the target will have to compete for attention. The evidence that feature and conjunction targets continue to influence parietal activity differentially, even when difficulty is equated, limits the generality of this account. The evidence as a whole points to a modular interactive network in which dorsal spatial and attentional functions interact with ventral feature maps to produce the unity of the multiple objects that we see.

Time and binding

An influential theory of binding proposes that integration is based on temporal correlations. At the cellular level, binding is argued to occur through temporal synchronization of oscillating neural responses⁵². The factor of temporal synchrony has also been adopted by sensory psychophysicists, who argue that binding happens when stimulus events simultaneously change in phase or have synchronous onset/offset times^53,54. This idea is consistent with the Gestalt psychologists’ examples of how common motion can bind a group of elements together to form a unit in perception⁵⁵. For instance, if a random group of dots were scattered across this page and every other dot began to move at the same time and in the same direction, they would appear as a bound unit against a background of stationary dots. The evidence that introduced temporal synchrony among neurons as a plausible correlate of perceptual binding was based on synchronous cellular responses to common as opposed to uncommon motion⁵⁶. When two bars moved across the visual field in the same direction at the same time, neurons with receptive fields that were centred on each bar began to oscillate in phase. This did not happen when the two bars moved in opposite directions.

Common motion is not the only grouping factor that produces synchronized neural activity. Grouping by similarity, good continuation and closure also produce synchrony. For instance, if half the dots from the previous example turned red (grouping by similarity) rather than moving together, they too would be perceived as a group separate from the non-red dots. Likewise, if half the dots were connected to one another by lines (good continuation) or by enclosing half the display (closure), grouping would also occur. In every case, the cells that encode different grouped dots must communicate with one another to ‘know’ that the objects being coded belong together. Temporal synchrony between distributed neural assemblies is one explanation. In humans, this grouping can be reflected by increased synchrony of the γ-BAND responses of the electroencephalogram (EEG)⁵⁷.

Temporal synchrony conveys a signal that the items from different spatial locations form a coherent whole. The cells that are synchronized can be quite distant from one another (even in different hemispheres⁵⁸), and there is no other documentation of long-range synchronous activity⁵⁹.

If we apply this evidence to the issue of binding surface features registered in specialized cortical feature maps, it follows that, to know that two features belong together (red and circle), synchrony between neurons that register colour and those that register shape (both associated with ventral processing) would be necessary. However, direct links between these neurons are not sufficient to explain surface feature binding, because damage to the parietal lobe disrupts such binding. An external source that synchronizes neural responses between cortical feature maps seems to be necessary, and this source seems to be spatial attention mediated by the parietal lobe. Consistently, manipulations of spatial attention modulate γ-band activity of the EEG over occipital–parietal sites⁶⁰. These effects have been observed with visual, auditory⁶¹ and somatosensory⁶² stimuli.

Despite the evidence for synchrony in grouping and parietal contributions when surface feature binding is necessary, neither synchronous activity nor parietal input is sufficient to account for the perception of bound features. There must be sustained activity to allow enough time for awareness of bound units to occur, and there must be two-way communication between the transmitter and receiver if synchronous activity is to be sustained, which is also consistent with recent neuropsychological evidence⁶³. There must also be selection, as synchrony can occur without awareness⁵². Communication and selection might occur through RE-ENTRANT PATHWAYS and help to sustain attention, which, among other things, leads to binding⁵⁹. Notably, perceptual awareness itself seems to correlate with temporal synchrony between the thalamus and cortex⁶⁴.

Together, the cognitive, neuropsychological and physiological evidence point to an interesting picture of binding. Some types of binding are supported by a posterior cortical dorsal–ventral and thalamic network, with synchronous and sustained activity correlated with spatial attention and awareness. This synchrony might even be augmented by massive inhibitory connections that resolve competition. This type of distributed system could be repeated throughout the brain. For instance, grouping or binding parts together to form an object does not require parietal–temporal interactions, but seems to need synchronous activity between neurons that do not interact with parietal lobes. Consistently, neither grouping nor object formation seems to require parietal function.

However, parietal input is necessary when attention must select one object among many. Without intact parietal lobes, object selection is nearly impossible to control, resulting in perception of only a single object at a time (probably bound within the ventral pathway), or perception of only simple features (registered in specialized areas within the ventral pathway). Likewise, without spatial maps, which are used to control selective attention, single objects and surface features are not bound to locations and seem to enter awareness randomly. This scheme does not mean that biased competition (for example, through familiarity or salience) and/or synchronous oscillations do not contribute to binding. However, the neural systems involved are different when explicit spatial information is required, as in conjunction search, and when it is not, as in feature search and grouping.

Synaesthetic binding

The discussion of binding has been centred on how information that is delivered through sensation and registered in different cortical areas is integrated to form the unified world that we perceive. However, there is at least one case of binding in which features are perceived together but one feature is not present in the stimulus: synaesthesia. Synaesthesia comes in many forms⁶⁵: the most typical is phonemic/chromatic, in which sounds induce the perception of colour. Another common form of synaesthesia is graphemic/chromatic (or shape/colour), in which shapes, such as letters or digits, induce the perception of colour. For example, the written letter A might induce blue whereas the digit 7 might induce green. The induced colour can appear in different places relative to the inducer, perhaps ‘hovering’ for one person but bound within the contours of the shape for another⁶⁶. Synaesthesia is automatic and consistent throughout life, and on the surface seems to be a case of feature binding that does not require attention. Functional imaging studies have shown that areas within the ventral pathway that normally register shape, colour and words are activated in synaesthetes, but in addition, there is also parietal activity, which for the most part has been downplayed^67–69. Some have argued that the automaticity and consistency of the synaesthetic experience represent direct connections between cortical feature maps¹³, perhaps through synaptic connections that fail to undergo normal synaptic pruning during development^70,71. This explanation of synaesthesia has been favoured by various investigators⁷², and is consistent with findings from behavioural experiments suggesting that synaesthetic binding occurs before attention^72,73.

In many respects, binding in synaesthesia represents an inverse problem from that of Balint’s syndrome patients. These patients see only one shape (object), but they do not know its location, and because they cannot attend to a space that has not been computed by the brain, they often bind features incorrectly. Conversely, synaesthetes see all the shapes in a display and know their locations, but even when only one feature (such as shape) is present it induces the perception of a different feature at the same time (such as colour). We tested two synaesthetes (CP and AD), who reported that colour and shape were tightly bound (FIG. 3), with two distinct percepts: the objective colour bound to the letter and the induced colour bound to the same letter.

Figure 3. Synaesthetic colours.

Examples of colouring-in done by synaesthete AD when asked to demonstrate where her synaesthetic colours appeared. The outlined black areas were drawn by the experimenter and presented to AD and she was then asked to place the colours exactly as she saw them.

One explanation for how synaesthetes might discriminate the objectively bound colour and form from the synaesthetic one is by positing a different entry path into awareness. If normal binding relies on attention but synaesthetic binding does not (that is, it is PREATTENTIVE), then discrimination could be made by the visual system on this basis. For instance, if the neural response to the letter T were directly connected to the neural response of green (the synaesthetic colour), then T would always induce the perception of green but the objective colour and the letter would also be registered through the normal mechanisms (BOX 2). One prediction of a direct connection model is that attention should not be required for binding of the inducer (T) and the induced percept (green). Early evidence was consistent with this prediction. However, more recent findings have shown that awareness of the inducer is necessary in almost all cases of synaesthesia.

Box 2. Dorsal–ventral interactions in binding.

The two synaesthetes we tested reported that their synaesthetic colours did not replace the objective colour, but rather were present in the same place at the same time. How binding occurs in this type of synaesthesia, and why attention seems to be required, need some explanation.

According to both feature integration and biased competition models, binding becomes a problem when two or more features (either in feature maps or in receptive fields of neurons) are signalled. If only one colour and one shape are present, there is no ambiguity about what colour should be assigned to what shapes, and attention and/or competitive inhibition is not required.

The schematics (a–c) show a simplified diagram of how binding between colour and shape might theoretically occur for normal perceivers. The top pathway of each diagram can be likened to occipital–parietal function, and the bottom pathway to occipital–temporal function. Dashed lines represent re-entrant pathways between them. (Cortical–pulvinar circuits that probably sustain activity have been omitted for simplicity.) The stimuli presented are on the left. Dorsal and ventral pathways do not need to interact when a single coloured letter is presented because no spatial ambiguity exists (a). However, when two different coloured letters are presented (b), the binding problem arises and interactions occur to resolve it (c). The special role of space (L) is implemented to co-localize the features through excitation and inhibition of information at each location.

For synaesthetes who see colours tightly bound to letters, there is only one location containing one shape but there are two colours (d); one presented and one induced. This sets up a different set of conditions for solving the binding problem. Suppose a blue O is presented to a synaesthete. Assume that binding of the letter and the objective colour occurs normally. But when the synaesthete becomes aware of the O, green — the synaesthetic colour — is automatically signalled (presumably because it is registered in the same colour map as blue)⁶⁷. There is then ambiguity regarding whether the O is green or blue. For normal perceivers, the spatial abilities of the parietal lobe are engaged when co-localization of the colours is required, but in this case there is only one letter and only one filled location. The neural signal to that location is excitatory, but there is no other location that is inhibiting the synaesthetic colour (e). The result is a representation in the brain of two bindings at the same location where there would normally be one.

Mattingley and colleagues⁷⁴ reported a series of experiments that clearly showed that awareness was necessary to produce the synaesthetic experience of colour induced by sounds or shapes. They tested a group of 15 synaesthetes, and visually presented them with inducing letters or digits that were either above or below the detection threshold. The inducer was followed by an isolated colour that was always presented above the visual threshold and was either consistent or inconsistent with the synaesthetic colour of the inducer. For example, if T induced green for a particular subject, a grey-scale T would be briefly presented and would be followed by a patch of green (consistent) or red (inconsistent). The time taken by the subjects to name the colour patch on each trial was recorded. Synaesthetes were slower to name the inconsistent than the consistent colour patches when the letters and digits were detected, but when the inducers were below detection threshold, consistency had no effect.

These findings suggest that synaesthetes must be aware of the inducer for it to produce synaesthesia¹². That is, they must attend to the letter or digit for it to be bound to its synaesthetic colour. One could argue that the inducer was not registered by the visual system when it was not detected, so of course it could not affect the time taken to name the subsequent colour patch. However, undetected letters and digits did affect the response time required to name a subsequent letter (for example, if an ‘a’ followed an undetected ‘A’ it was named faster than a ‘b’). The letters were implicitly encoded and registered in the visual system even to the point of implicit identification, but there was no evidence for implicit binding of the colour and the letter.

We also found evidence for attentional contributions to synaesthetic binding (Sagiv, N. et al. submitted). Our studies with RM showed that parietal attentional functions were invoked when binding across specialized cortical feature maps was required. Given this observation and the published literature on synaesthesia at the time, we expected that attention would not be necessary for synaesthetic binding (in other words, that feature maps were directly connected). However, contrary to these expectations, attention was crucial.

In one experiment we motivated two synaesthetes, CP and AD, to focus attention narrowly by having them name the colour of four dots close to fixation as fast as possible (FIG. 4). In another condition, the four dots were presented further from the centre to encourage attending to a larger area. Two hundred milliseconds before the dots appeared, a pair of grey-scale digits that induced colours were presented. CP and AD were told that they could ignore these digits, and that their task was to name the colour of the dots as fast as possible. When the digits were outside the focus of attention, the difference in naming speed between a consistent and an inconsistent dot colour was weaker than when the digits were inside the focus of attention.

Figure 4. Examples of stimuli used to determine whether inducers were influenced by the size of the attentional window in synaesthetes.

First, the colours that were induced by the numbers 2 and 7, for each of the two synaesthetes, were established by having them adjust colours on a computer screen to match their synaesthetic colour. In the study, two achromatic digits were displayed. Two-hundred milliseconds later, four ‘target’ dots were presented that were either the colour induced by the 2s or the colour induced by the 7s and were different for each synaesthete. Dot colour was either consistent or inconsistent with the induced colour of the digit. The digits were irrelevant for the task and were always located 8° to the left and right of fixation. Only the target dots varied in location. For both synaesthetes, naming the colour of the dots took longer when they were inconsistent with the induced synaesthetic colour than when they were consistent, but more importantly this difference was greater when attention was distributed widely over the visual field (165 ms on average) than when it was focused narrowly (52 ms on average). The digits were more likely to induce synaesthesia when they were located within, rather than outside, the spatial area being monitored.

This study showed that the area of the visual field that was monitored by attention could modulate the strength of the synaesthetic experience, consistent with physiological results showing that attention can modulate temporal synchrony between neurons⁵² as well as the receptive fields of single cells³⁸. However, this study was inconclusive in addressing the question of synaesthetic binding before detection. In addition, other studies began to be reported showing that visual search in an array of grey-scale letters or digits was easier for synaesthetes than for non-synaesthetes, presumably because the induced synaesthetic colours helped to segregate the target from the distractors before attention was engaged. However, in both of these cases, the distractors were letters or digits that also induced a colour^72,73. This means that when a distractor was attended it would induce its colour and might be more easily rejected, speeding the time to find the target. A study we performed with AD and CP showed that this was indeed the case. When distractors were not inducers themselves but targets were, synaesthesia did not reduce search time.

The one remaining bit of evidence that awareness of the inducer is unnecessary for synaesthetic binding was reported in another synaesthete known as C (REF. 66). When objectively black shapes were shown on a coloured background, detection was better when the induced colour was different from the background than when it was the same. The background seemed to camouflage the letter, leading to poor detection. For this to occur, the colour and shape must have been induced before awareness. Although we tested AD with the same procedure and were not able to replicate these results, it is still possible that C represents a special case of pre-attentive binding. Functional imaging data of C’s brain have not yet been reported, so we do not know whether parietal activation will appear (it is present in AD; Hubbard, E., personal communication), but C might represent a synaesthete who confirms the hypothesis that synaesthetic binding can be a result of direct physiological connections between specialized cortical feature maps. There are many stages in neural processing where synaesthesia could arise⁷². One suggestion has been that re-entrant connections send an extra signal to sensory systems that results in synaesthesia⁷⁵. It could be that the point at which re-entry has its effect is before detection of the inducer in some cases but after detection in others.

Whatever ultimately accounts for synaesthesia, the bulk of the evidence shows that features that are not present in the stimulus — have not been registered in specialized cortical areas through the usual sensory pathways — conform to the rules of binding as if they were present (BOX 2). They typically require attention and activate areas within dorsal as well as ventral processing pathways. The parietal regions that have shown activation in synaesthetic binding overlap with areas that produce binding deficits when damaged, and the evidence as a whole supports the role of parietal functions and spatial attention in surface feature binding.

Conclusions

The study of binding, and especially of its relationship to synaesthesia, is in its infancy, and conclusions will surely change as more data are collected. However, binding between specialized cortical feature maps is a fundamental ability in everyday life that seems to occur when the attentional functions of the parietal lobes are engaged. Many questions remain, including whether areas of the parietal lobe are involved in binding per se, or are required only in cases in which spatial information is needed to solve the binding problem. To what extent does binding between modalities mimic and/or engage the systems that are responsible for binding features within modalities (BOX 3)? How do the spaces involved in binding parts into objects and elements into groups differ from those that bind surface features or ‘properties’⁷⁶ to each other and individual objects to separate locations? What genetic, neural and environmental events are involved in the development of specialized brain systems and how can these account for variations in normal binding? What can be done to repair deficient binding when it creates problems in everyday life, as occurs through brain injury? These questions and others are at the heart of the binding problem, a real problem for the sensory, cognitive and neural sciences.

Box 3. Cross-modal binding.

The question of how integration across sensations occurs has a long history and, although not traditionally posed as a binding problem, the way in which the different senses merge does pose such a problem. Different areas of the cortex are specialized for early processing of the different forms of information received through different sense receptors. This information is registered in diverse areas of the brain, and therefore requires binding. There is ample physiological evidence that signals from different modalities converge on multimodal cells. Two, and sometimes three, modalities converge on a single cell to produce changes in activity that are more than the sum of the inputs⁸⁴. Some of the better understood multimodal regions are within the superior colliculus, where cellular responses to auditory and visual stimuli are excited or inhibited when signals from the two modalities converge in close temporal and spatial proximity. This structure has been targeted as a possible area in the production of auditory/visual (for example, phonemic/chromatic) synaesthesia⁸⁵. There are also multimodal convergence areas in the cortex, where cells show similar responses to those of the colliculus and are also candidates for the production of cross-modal synaesthesia. Notably, normal cross-modal integration in the colliculus is affected by cortical signals⁸⁶, implying that there are multiple levels at which cross-modal synaesthesia might arise. In addition, both physiological and behavioural evidence have shown the importance of experience in the development of auditory–visual binding during early life. Without such experience, poor auditory localization ensues⁸⁷. Given the larger probability of colour induction from verbal input in the synaesthete population, as well as synaesthesia’s consistency throughout life, cross-talk that produces synaesthesia between modalities might have a similar critical period.

Glossary

ACHROMATOPSIA

The inability to perceive colours despite otherwise intact vision

NEGLECT

A neurological syndrome (often involving damage to the right parietal cortex) in which patients show a marked difficulty in detecting or responding to information in the contralesional field

FEATURE MAP

A distinct population of specialized detectors that respond to basic perceptual features (for example, colour) yielding immunity to distractors and feature migrations from one object to another

EXTINCTION

Lack of awareness of stimulation on the side contralateral to brain injury when stimulation is also presented on the ipsilateral side

VENTRAL PROCESSING STREAM

A stream of processing arising in the parvocellular system and traversing along the occipital–temporal cortex, often referred to as the ‘what’ system

DORSAL PROCESSING STREAM

A stream of processing arising in the magnocellular system and traversing along occipital–parietal–frontal cortex, often referred to as the ‘where’ or ‘how’ system

TRANSCRANIAL MAGNETIC STIMULATION

(TMS). A technique that is used to induce a transient interruption of normal activity in a relatively restricted area of the brain. It is based on the generation of a strong magnetic field near the area of interest, which, if changed rapidly enough, will induce an electric field that is sufficient to stimulate neurons

γ-BAND ELECTROENCEPHALOGRAM

Electrophysiological activity measured in the 30–50 Hz range from scalp electrodes

RE-ENTRANT PATHWAYS

Feedback from higher levels of hierarchical processing to lower levels, creating reiterative interactions between higher and lower levels of processing

PREATTENTIVE

Processing that occurs before attention is engaged and is therefore capable of affecting performance without awareness