How the brain decides what we see

Until recently it was thought that vision shows us what is actually 'out there' at any moment. It was supposed that all the images printed on the retina are coded and transmitted to the cortex; thus every detail of our visual experience would mirror faithfully the ongoing events in the stimulus field. The first clues that things are not that simple came from clinical investigations by Ramachandran and others.¹ These showed that visual scotomata are not experienced as blank areas in the visual field but that the brain fills them in with (hallucinatory) sensations that the brain computes would most probably have been in this locus if there had been no scotoma. Clearly the brain mechanisms responsible for this 'filling in' were not evolved merely to fill in scotomata. They must have some wider function. Next, experiments in visual psychology have revealed that the brain will often construct sensations that override the visual input if the brain judges that the latter is too improbable. A notable example is reported by Kovács et al.² These workers took two photographs, one of a monkey's face and the second of a leafy tropical jungle. They converted these into two pastiches each composed of portions of each photo so that in the location where one photo showed part of the monkey's face the other showed leafy jungle. Then each pastiche was shown separately to each retina so that retinal rivalry occurred. Under these circumstances, the subject did not see what was actually there—i.e., the two pastiches alternating—but rather a complete monkey face alternating with a complete leafy jungle. Clearly the brain had suppressed the improbable mixed pastiche in favour of what it was familiar with (and thus computed what was more probable).

It was then discovered that, during a saccade (rapid movement of the eye), information coming from the eye is suppressed and what we see is largely virtual reality created by the brain from memory.³ Kleiste et al.⁴ expressed it thus:

'When you look into a mirror and move your eyes left to right, you will see that you cannot observe your own eye movements. This demonstrates the phenomenon of saccadic suppression: during saccadic eye movements, visual sensitivity is much reduced.'

In this paper they presented evidence, from psychophysical and fMRI studies, that this saccadic suppression selectively targets the magnocellular visual stream. They concluded: 'The range of effects in our data and in single cell data, however, argues against a single thalamic mechanism that suppresses all cortical input. Instead, we speculate that saccadic suppression relies on multiple mechanisms operating in different cortical areas.' In other words, 'filling in' has a temporal as well as a spatial dimension, based on a widespread network that includes the superior colliculus and parts of the thalamus. Moreover, eye movements are closely linked also to attention shifts. For example, Deubel and Schneider⁵ report a fundamental difference between the preparation for saccades and the preparation for manual reaching—i.e., the difference between responses to the instructions 'in 5 seconds flip your eyes to the target' (which is delayed saccade) and 'in 5 seconds put out your hand to touch the target'. For delayed saccades, attention is pinned to the saccade target until the onset of the response. This is not the case for manual reaching.

Thus what we see does not depend entirely on what is out there but also to a considerable extent on what the brain computes to be most probably out there. So we can ask how extensive is this filling-in process, what is its normal function and how is it done? Some light on these questions may come from digital television compression technology. This seeks to send the maximum possible signal with the minimum computational power compatible with an adequate picture. Sending every detail of every scene over the TV system is efficient but carries a high computational cost. This is called the I programme. So TV engineers have supplemented this with a second programme, P, that derives the most probable scenes following scene 'A' based on the system's memory and its previous experience of such scene sequences. This is supplemented by a third ('P frame') programme that records only differences between scenes. The P programme is inefficient but cheap in computational cost. So the art is in finding the optimum mixture of I and P.

The brain seems to use a similar system. If nothing much is happening the brain may increase its use of the memoryderived P system at the expense of its retina-derived I system (P>I). Then, if a new and potentially significant stimulus (S) is received, it may switch over more to its I system (I>P) so that the new S can be more effectively analysed and dealt with. Also the change can be partial and localized. For example, the P system may continue to supply the background, leaving the I system to focus on the new S, now at the centre of attention.

The question then arises, what is the physiological mechanism that controls the P:I ratio? Yu and Dayan⁶ have suggested that acetylcholine (ACh) promotes the I programme (or exact inference model) and inhibits the P programme (or naïve inference model). When a new and potentially significant S is received, ACh neurons in the nucleus basalis in the basal forebrain are activated. These are the sole suppliers of ACh to the cortex. Yu and Dayan suggest that ACh potentiates thalamocortical conductance (essential for the I programme) by a nicotinic post-synaptic stimulation of cortical cells in layer IV of the cortex. At the same time it inhibits cortico-cortical conductance (essential for the P programme) by a muscarinic activation of inhibitory presynaptic receptors in layers I/II of the cortex.

This mechanism may explain why the ACh projection to the cortex uses mainly targeted synaptic transmission,^7-9 but with some diffuse volume transmission as well.^10-12 This is unlike the projections used by the other 'neuromodulators' dopamine, norepinephrine and serotonin that use exclusively volume transmission in the cortex. (In volume transmission a dense meshwork of fine axons bear many boutons-en-passage along their lengths that do not make synapses with the receiving neurons but simply spray their transmitter load into extracellular space where it diffuses to the nearest neurons.) The dual action described by Yu and Dayan means that ACh must be able to stimulate nicotinic but not muscarinic receptors in layer IV of the cortex, and vice versa in layer I/II. A diffuse volume transmission system could not accomplish this.

Yu and Dayan⁶ suggest that the role of ACh in the cortex is to report uncertainty in top-down (P) information—with the effect of boosting bottom-up sensory (I) information at the expense of top-down expectations. However, I would like to suggest that ACh may have an executive (motor) function in addition to any signalling role. This executive function would be to switch the brain's programmes from P>I to I>P, either globally or locally.

ACh also plays a role in saccades that are the motor expression of increased interest in the locus to which the saccade is directed. During the actual saccade, as I have indicated, the P system is potentiated and the I system is reduced. So do cortical levels of ACh fall during a saccade? A saccade is initiated by a cholinergic signal from the brainstem pedunculopontine nucleus that activates nicotinic receptors in the intermediate (motor) layer of the superior colliculus. Kobayashi et al.¹³ reported that a subset of pedunculopontine neurons (13/70) varied their activity levels with task performance, discharging at a higher rate in successful trials than in error trials. A combination of responses related to saccade execution, reward delivery, and task performance was observed in pedunculopontine neurons. Thus a saccade should both enhance I (at the saccade location immediately before the saccade) and enhance P (during the saccade).

In the muscarinic realm the sparse data do not as yet allow any such analysis.^14,15 We await information on the central issue of whether reduced cortical cholinergic activity during a saccade is responsible for, or is associated with, reduced activity in the thalamocortical projection and/or increased activity in the cortico-cortical projection. So how could the observed P>I alteration during a saccade be achieved? The projection of the pedunculopontine nucleus to the nucleus basalis is massive and uses glutamate as its transmitter. The nucleus basalis contains large numbers of inhibitory GABA-ergic neurons as well as its large cholinergic neurons. It is not known whether the excitatory pedunculopontine projection to the nucleus basalis targets the ACh neurons or the GABA-ergic interneurons (Mesulam M, personal communication). If the latter is the case, this could provide the necessary mechanism. The pedunculopontine nucleus would initiate the saccade by its projection to the superior colliculus and subsequently inhibit the I system and promote the P system in the cortex by exciting inhibitory GABA-ergic cells in the nucleus basalis to reduce ACh release in the cortex. Clearly further experiments are needed to test this hypothesis.

An added complication is provided by the fact that the superior colliculus has a second cholinergic input—from the parabigeminal nucleus—that would appear to be functionally different from its input from the pedunculopontine nucleus. This second input goes to the (sensory) stratum griseum superficiale where it activates excitatory presynaptic nicotinic receptors on glutamatergic terminals (of axons from retinal neurons) that synapse on GABA-ergic interneurons.^16,17 Binns and Salt¹⁶ conclude '... the ultimate effect of nicotinic receptor activation is to depress visual responses'. Thus here the nicotinic ACh system would appear to have the paradoxical effect of reducing the upward flow of collicular activity. The significance of this second cholinergic input to the superior colliculus is not at present clear except that this nucleus may be an integral part of a midbrain circuit that generates target location information.¹⁸

There is another intriguing possibility relating to the role of saccades, this time during rapid-eye-movement (REM) sleep. The saccades that are so prominent a feature of REM sleep are commonly thought to be due to the changing focus of attention on dream images during the course of the dream. However, REM sleep is an expression of pure freewheeling P activity. We have seen that, during waking saccades, P is promoted and I inhibited. Perhaps one function of saccades during REM sleep is to do the same? As we saw above, the initiation of saccades has the opposite effect. Dominance of one or other of these results might depend crucially on the timing or duration of these effects.

This hypothesis has implications for the philosophy of perception. At present most commentators assume that what we see is driven entirely by the sensory input. In other words we see what is actually out there. This position is inconsistent with the experimentally demonstrated fact that much of what we see in normal vision is virtual reality driven by the brain's P programmes.¹⁹