The Medawar Lecture 2001 Knowledge for vision: vision for knowledge

Abstract

An evolutionary development of perception is suggested—from passive reception to active perception to explicit conception—earlier stages being largely retained and incorporated in later species. A key is innate and then individually learned knowledge, giving meaning to sensory signals. Inappropriate or misapplied knowledge produces rich cognitive phenomena of illusions, revealing normally hidden processes of vision, tentatively classified here in a ‘peeriodic table’. Phenomena of physiology are distinguished from phenomena of general rules and specific object knowledge. It is concluded that vision uses implicit knowledge, and provides knowledge for intelligent behaviour and for explicit conceptual understanding including science.

1. Part I

(a) Knowledge for vision

It is generally accepted that the image of a camera is meaningless without a brain to interpret it. Yet, and this is a curious fact, there are theories of perception which fail to recognize that this applies equally to the images in eyes. Without the computing power and memory that brains bring to bear, retinal images would be meaningless patterns of limited use—hence the importance of knowledge for seeing.

Visual science may not always recognize the importance of knowledge for giving significance to vision, as its origins in Greek philosophy set ways of thinking before images were discovered in eyes. Retinal images were hardly appreciated before Descartes and Kepler early in the seventeenth century (Wade 1998). The later notion of the senses as transducers, signalling to the brain with slowly travelling neural activity, was hardly appreciated before Hermann von Helmholtz in the middle of the nineteenth century. Then, it became clear that physiological channels separated perception and experience from the object world, though the implications are not always recognized.

Before these key discoveries, it was generally supposed that vision was directly related to objects of the external world, either by ‘fingers’ of light shooting out and touching objects, or by objects sending expanding ‘simulacra’ of themselves to the observer (Ronchi 1957/1991). This tradition of perception, as directly related to the world of objects, has not altogether died (Gibson 1950, 1966, and followers). Yet, the most striking fact is that perceptual experience is far richer than available retinal images; and though neural signalling is slow, it is not usually delayed in time. From these shadowy ghosts in our eyes we see hard solid objects with properties beyond optics. This depends on knowledge of objects, and how they interact, allowing behaviour to be appropriate to what is known or assumed, rather than limited to what is being sensed. This is where knowledge comes in, as the past enriches the present, and allows some prediction into the future.

This is ‘perceptual’ knowledge of object properties, rather than ‘conceptual’ knowledge, which may be abstract and far removed from perceptual experience. Perceptual and conceptual knowledge can be very different, and may disagree.

(i) Outline evolution

Vision was not the first sense. Almost certainly, the first was testing the chemical environment, with what for us is the sense of taste. The evolutionary steps from monitoring properties immediately essential to life, to sensitivity to light and fully fledged vision, are not at all completely known. Evolutionary theory demands that each step must have some survival advantage; however, as the advantages can change, this does not imply neat linear development.

It is likely that receptors sensitive to light evolved from chemical and touch receptors, eyes starting as concentrations of receptors in gradually deepening pits, optical components starting as transparent membranes to keep the eye-pits from filling with dendritus. These protective windows could transform into lenses by at first increasing shadow contrast and then producing sharp images on closely packed receptors. In spite of Darwin's famous ‘cold shudder’, at how critics would receive his account along these lines, it is now generally accepted that shadow contrast was enhanced by gradually deepening eye-pits, which suddenly transformed into pinhole cameras when they almost closed. This demanded a dramatic reorganization, as all movements were reversed in the camera-eye. We seem to see this today with crossovers in the mammalian brain (Land & Nilsson 2002).

Although imaging is necessary for human-like vision, simple detection of light is a great deal better than nothing. By signalling changes in light, it gives a warning that something is happening; or if distant, is about to happen. When direction of light and forms of shadows and images become available, further advantages are gained—though they need signal processing, and calling up knowledge from the past.

Although the Greeks did not know that eyes work with images that need to be interpreted, this is very much as Plato described his cave, with its moving shadows (Republic, vii. 514 A–521 B). The prisoners in the cave guessed what objects outside might be casting the shadows, so learning something of the outside world. However, what could they learn only from shadows? Some more direct knowledge seems essential for seeing things from shadows, or images.

The evolution of visual experience—when and how consciousness started and developed—is mysterious. Yet it is hardly too fanciful to say that we can time-travel through evolution, for evolutionary time is spread out on the human retina. We can travel back to primitive vision by looking with the edge of the retina, then travel forward in time by looking more centrally, and finally, with the recently developed fovea. This may be done by looking ‘with the corner of the eye’ at, for example, a television picture, and then moving the eye gradually to central vision. First colour goes, and then form is lost. Near the periphery, only movement will be seen with its direction, until this is lost—as at the far periphery, movement is seen without direction. This must surely be the earliest vision (though we cannot really say this is what primitive organisms experienced). Finally, in the extreme periphery, consciousness will be lost, though some reflex behaviour remains.

More than half the human cortex is involved in vision, which is some indication of its importance. How it evolved is only known in patches; however, a crucial step must surely have been learning associated conditions, and events that could not be directly sensed. Then, limited sensory sampling of the object world can be extended and enriched. A key feature of perception, especially vision, is predictive power. There is prediction to ‘unsensed’ properties of things and into the immediate future, making anticipation possible. Anticipation allows behaviour to be in real time, in spite of the slow conduction of nerves, and to continue through gaps of signalled information, as when objects are temporally hidden or the eyes are directed elsewhere.

Prediction to unsensed properties allows behaviour to be widely appropriate, even though what is sensed is limited. We see wood as heavy and hard, knives as sharp and therefore useful; food as edible or dangerously ‘off’, which may have been a major spur to primate colour vision. All this depends on knowledge—some learned as innate, through natural selection and inherited genetically. Much is individually learned. As knowledge from learning is so important, it seems unfortunate that perception and learning are investigated separately, in different laboratories, the results published in different journals.

(ii) Ancient and modern streams of brain processing

Primitive behaviour is largely passive responses to stimuli, with tropisms and reflexes, which became more and more elaborate and with inhibitory mechanisms switching them off when not appropriate. However, more ‘advanced’ perception is active guessing of what might be out there, behaviour being generally appropriate for a wide range of objects and conditions, as appreciated from knowledge. How did active cognitive perception of objects arise from passive responses to stimuli?

Recent experiments suggest that the human brain retains primitive perception for simple rapid movements, with cognitive perception added on, in more recent brain systems. Evidence comes from the effects of brain damage (Schneider 1967, 1969) in the hamster; lesions of the visual cortex impairing form perception but leaving localization of objects intact, lesions of the superior colliculus having the opposite effects. Mishkin et al. (1983) proposed two cortical streams, a dorsal stream for where, and a ventral stream for what, is present. Milner & Goodale (1995) suggest that the evolutionarily early dorsal stream serves simple rapid behaviour, without consciousness—the later ventral system serving full cognitive vision, with recognition of objects for planning behaviour, sometimes with consciousness. Evidence for these different systems has been sought from visual illusions affecting cognitive vision but not related behaviour. The same-size inner circles of the Titchener (or Ebbinghouse) illusion, appear to be different sizes (figure 1), but would they be grasped differently by the fingers? This was found by (Aglioti et al. 1995), but not all investigators agree (Franz et al. 2000). Perhaps this visual distortion is too small, and the grasping behaviour may be too slow for the primitive dorsal system—so switching in the newer ventral, cognitive stream.

Figure 1.

Titchener illusion. The two inner circles are the same size though appear different, by contrast with the larger and the smaller circles. There are related contrast effects, sometimes showing up as illusions, for all modalities and all the senses.

Recently, we have looked at a far larger illusion—reversal of depth with the Hollow Face (figure 2)—with a rapid reaching and ‘flicking’ task. This cognitive reversal of depth gives dissociation between appearance and rapid reaching behaviour. Targets on the hollow mask are touched and flicked correctly, though seen as more distant in the depth-reversing illusion (Grzegorz et al. 2005). This is an example of how useful illusions can be for investigating perception.

Figure 2.

Hollow face illusion. A hollow face mask is seen as a convex face. This is striking evidence of the power of visual knowledge; the more realistic the face, the stronger the effect. There is a general tendency to convexity (as most objects are convex), but this is most dramatic for faces. It is clearly a cognitive phenomenon, the significance of which has only recently been realized.

(iii) Seeing pictures

Together with language, making pictures is uniquely human. A painting is a flat pattern of shapes and colours and yet it is seen as very different objects, lying in a different space and time. A portrait is seen as almost alive, about to speak and interact with the viewer much as a person would, yet we do not attempt a conversation. (It is only a joke that the starving artist will eat the fruit in the still life.) The point is that seeing blobs of paint as very different objects—flowers, ships, people—depends on knowledge of these objects, derived through years of interacting with things and using all the senses. So pictures are seen as far more than blobs of paint, and vision is far richer than the eyes' images.

Just as seeing patterns of pictures as objects depends on knowledge from prior experience, so does everyday ‘seeing’ from retinal images. There is, however, a difference. Unlike a photograph or painting, the retinal image is a picture we never see. It is not an object for vision, for there is no ‘inner eye’ to see it, or there would be an infinite regress of images and eyes. The brain sees from the eye's image, but never sees the image itself as it sees an object or painting or photograph. Retinal images are not seen, but are sources of information for seeing external objects, including pictures.

A picture has a curious double reality, for we see its shapes, colours and textures as more-or-less familiar objects, while at the same time, we know we are seeing blobs of paint. A photograph has already been imaged in the camera before being imaged in the eye, so it is imaged twice. Normal objects are only imaged once, in the eye. A photograph has the perspective features of a retinal image, though usually from a different distance; but as a painter cannot see the image in their own eye, the painting is from their brain's perceptual processing and their knowledge. There is no ‘innocent’ eye and so no innocent paintings. As photographs are innocent of knowledge or prejudice, the camera is useful for artists—back to devices such as the camera obscura, which gives perspective but has no knowledge to enhance or distort or edit and so is quite unlike perceptions.

The artist is free to invent unknown and even impossible objects—to distort perspective, and combine visual clues in bizarre ways not found in nature. Thinking of seeing as intelligent inference from clues fits a Sherlock Holmes account of how the perceptual brain works, and why many illusions occur. Visual and other sensory clues may be helpful or misleading even to creating paradoxes (figure 3). Artists can weave wonders by presenting clues just as they wish, so they can go beyond and comment on nature. Artists are Dr Watson to Sherlock Holmes—not always infallible.

Figure 3.

Impossible triangle model. Although it exists, as a three-dimensional object, it looks impossible. It looks impossible from view points where the sides seem to touch at the corners. This is because of the visual rule that things touching are (probably) the same distance. Here they are not, so here this touching rule misleads. So the perceptual hypothesis is generated from a false assumption, giving a cognitive paradox.

(iv) Knowledge

The word ‘knowledge’ often implies conscious understanding. However, following Herman von Helmholtz's perceptions as ‘unconscious inferences’ (Helmholtz 1856–67), we may allow that knowledge can be implicit and not conscious. This allows us to say that primitive nervous systems may have knowledge, as do computers. It implies that we do not know by introspection what knowledge we use for perception. Experiments are essential for finding this out. It turns out that phenomena of illusions are very useful for this (Gregory 1997), so illusory phenomena will be discussed here.

Visual knowledge is in two forms: specific knowledge of particular objects and kinds of objects, and general rules applying to almost all objects. An example of a rule is perspective, used for seeing distances of any objects. We may use the word ‘knowledge’ for both particular objects and events, and for general rules. Visual knowledge need not be, and generally is not, conscious, although generally useful visual knowledge, including rules, can mislead to generate a rich variety of cognitive illusions (Gregory 1970; Hoffman 1998).

(v) ‘Physiological’ and ‘cognitive’

Knowledge implies cognitive processes and explanations. How do these fit with physiology and its explanations? This is an important and tricky distinction. An analogy from a familiar situation might help. Consider directing a motorist from, for example, London to Cambridge. All that needs to be specified is the route, as it may be assumed that he has a working car. It does not matter whether it is petrol or diesel, as both should be available, so this need not be specified. Although a functioning car is essential for the journey, how it functions can usually be ignored. Generally, all that matters for the instructions is the route. This may be in a map; but the map may be out of date. Then the motorist will be misdirected by knowledge from the past, when it does not apply to the present. And of course the map may have been wrongly drawn, or in the heat of the moment may be misread.

The cognitive map directs the ‘physiology’ for the journey. In normal conditions it is the knowledge, coded in the map, that matters; but the means for carrying out the instructions cannot always be ignored. The car's functioning may be ignored until its limitations make the instructions hard to carry out, as in special conditions such as snow, or when impaired by damage. Delay in arriving may be owing to cognitive or to physiological shortcomings. These are very different and would need to be dealt with differently (as for psychological or physiological problems).

The car will be driven by a human. For the analogy we may imagine a robotic car, controlled with a computer whose software will contain the necessary knowledge. The same considerations will apply to robot and human. The electronic map in the computer may have errors, and will never be fully detailed, lacking information such as other cars to be avoided. As for human driving, bottom-up sensing of the surroundings will be needed in addition to the top-down knowledge of the map.

(vi) Ins-and-outs of vision

The cognition of vision may be simply expressed with a diagram such as figure 4. We have sensory signals from the world of objects, which may be called ‘bottom-up’. These are carried out by many specialized channels, signalling movement, position, colour and so on. Objects are not signalled as such, but have to be inferred from sampled characteristics, as conveyed by the various channels. Then we have ‘top-down’ visual knowledge (such that faces are convex), and ‘sideways’ rules applying to nearly all objects and situations. Thus, perspective with its rules sets distance for any kind of object.

Figure 4.

Ins-and-outs of vision. The general notion is that perceptions are hypotheses of what might be out there. We suppose that bottom-up signals, top-down knowledge and sideways rules combine to generate object perceptions as predictive hypotheses. Failures of behaviour can correct knowledge. Prevailing perceptual hypotheses can work downwards, to modify even basic experience, or ‘qualia’ such as brightness or colour. (This is found from illusions of ‘flipping’ ambiguity, where perception changes though the input stimuli are unchanged.) Perceptions are much like hypotheses of science: filling gaps of data; being predictive to unsensed features, and into to the future; selected by probabilities and conferring probabilities (Gregory 1970, 1980; Hoffman 1998; this is essentially Bayesian).

How behaviour arises from perception is not simple, as perception may entertain many possibilities though only one action is possible. Much, even of human behaviour, is ‘automatic’ from input stimuli, with little or no cognitive processing. One might say that perception is richer than behaviour, rather as theoretical science considers more possibilities than can be carried out or found to be true. Much of the knowledge in both is implicit, and not conscious.

(vii) Illusions

Illusions are phenomena of perception which deviate from truths. But what truths? Accepted truths serve as references for illusions, but just as what is true is not always clear or identified, there can be uncertainty of what is illusory.

If physics is accepted for reference truths, we must recognize that physics frequently changes its mind. If ‘deep’ physics is accepted, there is a danger of calling all perceptions illusory, for accounts of deep or fundamental physics are very different from any appearances. However, to call all perceptions illusory is not helpful. So we may accept for references ‘kitchen’ physics; using simple instruments such as rulers and clocks, generally with common sense interpretations. Although not altogether satisfactory, this seems the best we can do.

Visual errors are sometimes called optical illusions, but this term is best restricted to disturbances of light between the object and the eyes, as for mirages or rainbows, and so on. These are phenomena of physics and studied and explained as physical phenomena, though they might be called ‘illusions’ when they mislead belief or behaviour—as when the viewer expects to walk under the rainbow, or find water in the mirage.

(viii) Classifying illusions

As illusions are perceptual departures from physics, so they reveal perception and its processes and limitations. As illusions can isolate perceptual from physical phenomena, we can use them to reveal and study processes of physiology and mind. As classifying objects and phenomena is important in the physical sciences, so classifying may be useful for explaining perceptual phenomena and how they are related. It is not supposed that these, or indeed any phenomena, show causes directly. For surely phenomena cannot ‘speak for themselves’, but must be interpreted from knowledge and assumptions. There is a circularity here, as theories are needed for explaining phenomena, and yet phenomena suggest and test theories. This circularity seems to underlie all science. Perhaps looking for conceptual similarities and differences by classifying may help to break through the circularity. It is contexts that give meaning. This is no substitute for detailed experiments, but may help to interpret results of experiments, as well as suggesting where to look for new phenomena and explanations.

We may start with lack of perception—non-sense of various kinds of blindness—to end with fictional perceptions of nothing.

(b) Non-sense

(i) Total (bottom-up) blindness

Lives depend on reliably not seeing, so that not seeing corresponds to nothing there. The driver moves off when not seeing another car or pedestrian in the way. Although much of behaviour depends on not seeing, no evidence is different from evidence of nothing. It is likely that older drivers are slower and more cautious, to ensure that seeing nothing corresponds with nothing there to be seen. Ensuring absence demands an even heavier memory load than perceiving what is there.

The long-term blind do not see blackness, they see nothing. The nearest a sighted person can get to this is imagining what is behind one's head. One does not see black (which is a colour), one simply sees nothing, though one may guess what is behind one's head and this can be vitally important.

The experience of becoming blind is different for different people. Becoming blind gradually, over several years, has been meticulously described by Hull (1991) in Touching the rock. For him, visual imagery completely disappeared, as over a period of several years he sank into ‘deep blindness’. There are, however, very different accounts (perhaps more from sudden blindness) of visual images evoked synaesthetic-like by the other senses. Following blindness, there is a crucial choice-point of whether to reject or to hang on to visual imagery (Sacks 1993). This seems to be a personal decision, the mind changing its very centre according to the choice. Fortunately, even in middle age, the brain can adapt to new uses, partly under the control of the individual.

Rare cases of adult recovery from blindness at birth or infancy are remarkable human stories, though unfortunately associated with disappointment and serious depression, with partial or complete rejection of the new sense. These abnormal responses to vision acquired by an adult are a warning that these rare cases are different from normal vision and its development in babies and children, though they are interesting and suggestive.

For many centuries, the only available cure for blindness was removal of the lens, as made opaque from cataract (von Senden 1960). These operations gave an effective retinal image only gradually, as the eye had to settle down for weeks or months. This slowness to see suggested that there is normally little or no vision without prolonged learning. However, recent cases of corneal transplants, which give immediate retinal images, show considerable immediate vision (Gregory & Wallace 1963; Gregory 1966; Fine et al. 2003).

It appears that there can be immediate vision when there is prior knowledge, especially from touch experience during the years of blindness. This is cross-modal transfer, from touch to vision. This makes development of postponed vision different from normal development in babies, though it does suggest that hands-on experience is important for normal visual development.

Learning to live with sight after years of blindness seems to be harder than the far more common, opposite journey, into blindness. Although the sense of sight is lost, the brain is now free to create rich and wonderful experiences that only the blind can know. Other senses are enhanced, starting within hours of blindness, or even after simply being kept in the dark for hours or days. This is an interesting experiment.

(ii) Agnosia (top-down blindness)

Named by Freud, as meaning lack of knowledge for seeing, agnosia owing to brain damage is the inability to read meanings into perceptions. There may be failure to recognize even common objects when visual knowledge becomes unavailable. A famous account is Sacks' (1985) The man who mistook his wife for a hat. Removal of visual knowledge is highly suggestive for its normal uses, so agnosia has special interest here, though will not be discussed further.

(iii) Neglect

There can be neglect of regions of space and of parts of objects. Most often, the left visual field is damaged with a parietal lesion. Amazingly, the left halves of objects are missing wherever the eyes are looking (figure 5).

Figure 5.

Neglect. The left half of objects is missing. Half is missing even though the eyes continue to move freely. This must surely be significant for considering how objects are seen.

(iv) Blindsight

The ability to point to objects even without conscious vision. An alternative, more primitive pathway seems to come into play when there is loss of primary processing. As consciousness is missing, this throws light on the mysterious role of consciousness for normal perception and behaviour (Weiskrantz 1997).

(c) Instability

A major miracle of perception is the stability of visual and other sensory experience, though we move, and objects move and rotate in complicated ways around us. Much of this is achieved by ‘low level’ processes following simple rules, but there are also important higher level processes called ‘constancies’.

Stability or ‘constancy’ through changes of stimuli, is given by dynamic compensations, and scaling for distance and sizes and shapes of objects. These processes tend to stabilize and correct (or normalize) object perception. But, when not appropriate to the situation, or misapplied, these normally correcting processes can create just the kinds of illusions they were designed to avoid. The give-away is that these inappropriate or misapplied corrections produce errors of the opposite sign to what they normally correct or compensate. When the compensation is activated, though there is nothing needed to be compensated, these processes produce the same errors but in the opposite direction. Then we experience, quite directly, perceptual processes that are normally hidden.

(i) Border locking?

A dramatic, recently discovered example of visual instability is the Ouchi illusion (figure 6). With small movements of the whole figure, the central region moves separately from the surrounding pattern. This is striking and surprising; but perhaps the surprise should be that this only occurs for special patterns or objects. The many parallel channels giving bottom-up signals have different delays, delay increasing with reduced intensity of light. As this is a large effect, one might expect scenes to break up, with regions moving differently whenever there is a shift of the image across the retina. Yet this only occurs for certain patterns, such as the Ouchi illusion. Here, the contours in the inner and outer patterns are orthogonal and have different spatial frequencies.

Figure 6.

Ouchi illusion. The two regions are not locked together, but move independently. The inner region moves around independently when the figure is moved, or with movements of the eyes. Why does this not generally occur? For the visual channels have different delays, depending on level of illumination and dark adaptation. Is there a mechanism of ‘border locking’, which fails for this figure, with its regions of orthogonal lines and different spatial frequencies?

Is there a special mechanism for registering visual channels? The notion of ‘border locking’ has been proposed (Gregory & Heard 1979), with luminances and colours normally being registered at borders by active ‘locking’, though this can fail as in the Ouchi illusion. It also fails when there is colour contrast but no luminance contrast (iso- or equi-luminance displays). It is suggested that the Café Wall distortion (figure 7) is owing to inappropriate border locking, causing distortion in this special case of luminance contrasts across narrow neutral-luminance gaps, which the visual system might mistake for errors needing correction. However, although these phenomena are dramatic, this explanation is speculative.

Figure 7.

Café wall illusion. Found in the tiles of a nineteenth century café wall in Bristol. The parallel ‘mortar’ lines appear to converge alternatively, forming long wedges. For this effect, the mortar lines must be thin (less than 10 s of arc at the eye). There must be luminance—not only colour—contrast of the tiles. The distortion of the ‘mortar lines’ reverses when alternative rows of tiles are shifted sideways by half a cycle.

(ii) After-effects

When any sense is given intense prolonged stimulation, there is adaptation, or ‘fatigue’. When only a single neural channel is involved, there is simply a loss of sensitivity with corresponding de-calibration. Where several channels are involved, adaptation phenomena are rich and can be highly informative. Adaptation to a colour will produce after-images of the complementary colour, as colours are seen as mixtures from three channels, as Thomas Young discovered in 1801. For example, red light mixed with similar brightness green light produces yellow. Adapting to bright red or green changes the appearance of the yellow—just as it changes the intensities of the mixed yellow or green light—though in the opposite direction.

Stimulation of the eye with continuous movement will produce an illusory after-effect of movement, in the opposite direction to the adapting stimulus. A rotating spiral appearing to expand will shrink as an illusory after effect when the rotation is stopped. There may be 30 or so ‘vector’ channels for signalling movement in any direction. Adapting to one direction unbalances the system, giving apparent motion in the opposite direction. Such adaptations can apply to complex perceptions, such as tilt or curvature of lines, and of size (Blakemore & Sutton 1969).

Much of the physiology of receptors, and their cortical representations, is known in considerable detail. This has come largely from recordings with micro-electrodes, especially by the experiments of the American physiologists David Hubel and Torstin Wiezel, who received the Nobel Prize in 1981 for this work (Hubel & Wiesel 1962).

Different adaptations to parallel channels can produce paradoxes. The rotating spiral gives an after-effect of movement, but without change of position. This is impossible for an object; but can occur as an illusory perception, because it is not objects that are signalled to the senses. What are signalled are various characteristics that may or may not come from a single object. Vision combines signalled features as hypotheses of objects, using general rules (such as the Gestalt laws; figure 8), and knowledge of objects (which may be hard to see, such as the Dalmatian dog; figure 9). This notion of parallel channels, from receptors signalling many kinds of information—of movement, position, orientation, colour, texture, and so on—is a key bottom-up concept. Making use of present signalled information, and knowledge from the past, is the task of top-down cognitive processes for perceiving and interacting with objects.

Figure 8.

Gestalt laws of grouping. Dots form patterns from ‘closure’, ‘proximity’, ‘common fate’ (when moving, as leaves of a tree) and so on. These may be innate rules.

Figure 9.

Dalmatian dog. The dog is hard to distinguish from the pebbles. All object perception is problem-solving—here, one sees the difficulty.

Rules of organization for early stages of object vision were recognized by the Gestalt psychologists in the first half of the twentieth century. They stressed the innateness and inheritance of many of these laws of organization, though some could be individually learned or at least fine-tuned. They have turned out to be important for programming computers to recognize objects, though this has a long way to go before computer vision rivals even quite simple brains.

As both after-effects and misapplied constancies are generally ‘negative’, they may easily be confused, though they are different. A defining difference is that adaptations only occur when sensory systems are over-stimulated, increasing gradually through time, while the constancies occur immediately and with normal stimuli.

(iii) Jazzing

The remarkable jazzing effects of repeated high contrast lines of Op Art, especially associated with Bridget Riley's pictures and Mackay's Rays (Mackay 1957), have been explained in various ways. These include: saturation by informational redundancy; directly stimulating ‘movement centres’ of the brain; small eye movements, or ‘hunting’ of the lens for accommodation, shifting the pattern across its own transitory after-image, so ‘beating’ to produce moiré patterns, and stimulating ‘on–off’ movement receptors of the retina.

Jazzing instability can occur from object knowledge. Thus, a face drawn with four eyes is disturbing.

(iv) Eye movements

If an eye is pushed gently with a finger on the lid, the world will seem to move. But with normal, voluntary eye movements, the surrounding scene will remain stable. There are two very different explanations: (i) shifts of the retinal image with normal voluntary eye movements are compensated from monitoring the command signals to move the eyes; and (ii) that the movements are ignored as they contain no surprise. It is often hard to know whether experiencing nothing happening is owing to ignoring what is going on or whether there is active compensation. Here, there is evidence for both accounts, and both may apply.

(v) Auto-kinetic effect

This is more-or-less random apparent motion of a small light viewed in darkness. It occurs even though the eyes are not moving. It is generally explained using the system that normally compensates eye movements, which select needed information (Yarbus 1967) to give stability to the world when the eyes are moved, though sending small fluctuations signals when the eyes are held still. If the eyes are held hard over to one side for a few seconds, the little light usually swings around violently in the opposite direction, as the system is now unbalanced. But why does the auto-kinetic effect work only for a small light in darkness? Why does a whole room not look unstable? This takes us back to the cognitive assumption of stability; that a rich world is assumed to be stable, so small imbalances are ignored. A tilted room is disturbing as it violates such assumptions. The general assumption of a stable world seems very important for perception, hence the trauma of earthquakes.

(vi) Self-movement and object-movement

Normally, we know from vision whether we are moving or surrounding objects are moving around us, which is remarkable as all movements are relative. For surviving in the complex world of objects, perception has to decide—relative to what? This may be external objects, or the observer. Proprioception from the limbs helps to determine observer motion, but identifying self-motion can be achieved by vision alone—as when our feet are off the ground in a moving car, or skiing, so vision can solve the problem of what is moving.

Perceptual rules originally suggested by the Gestalt psychologists (Ellis 1938) come into play, such as large distant objects being accepted as stationary ‘references’ for nearer moving objects. Dramatic illusions of induced movement occur when large distant objects move, as these are generally accepted as fixed. Induced motion is dramatic in the fair ground Haunted Swing: an oscillating room, in which the stationary observer seems to move, to fall over, even to be turning upside down, though they are standing still.

The decision of what is moving (and all movements are relative) can be taken early on by vision with simple rules. However, object knowledge comes into play. This is useful for film animators as movements of familiar kinds of objects, such as people and animals, can be conveyed realistically with minimal information. So-called ‘biological motion’ was dramatically demonstrated by the Swedish psychologist Johansson (1973), with small lights placed at the joints of otherwise invisible people moving in a dark room. When stationary, the lights formed a meaningless pattern; but with motion they leapt into life, clearly seen as moving figures. Very soon their gender, and what they are doing, can be seen in the moving lights. This works far less well for non-biological objects. Presumably, identifying people and animals in low light was very important for survival, and we still live with this ability.

(vii) Motion parallax

There are instabilities associated with one's own motion. While being carried along in a train, or a car, the scenery rotates against the motion, around the fixation point. This is optical motion parallax. Although this is an optical phenomenon (clearly seen with moving video) there is a cognitive component. For the apparent direction of motion reverses when far and near are perceptually reversed in depth.

This is seen very clearly with a wire cube (figure 10a), the truly three-dimensional cube will spontaneously reverse in depth, but as the faces of the wire cube lie at different distances, there will be motion parallax, which is not present for the flat figure. The apparent direction of the motion reverses with the flip in depth. The cube appears to rotate with the viewer's movements (at twice the speed) when depth-reversed, which is opposite to normal parallax.

Figure 10.

(a) Necker cube. This flat figure appears to be a three-dimensional cube, although it has no perspective. As there is no evidence for which face is near or far, the brain entertains the alternative possibilities—flipping spontaneously from one to the other. These are alternative perceptual hypotheses, entertained in turn by the brain as it cannot make up its mind. (b) Duck–Rabbit. The best-known ‘ambiguous object’ is seen sometimes as a duck, and at others, as a rabbit. Presumably this ‘flipping’ depends on evidence for a duck and for a rabbit of equal probabilities. As it cannot be two things at once, it flips between the duck and the rabbit possibilities.

This shows that there is a cognitive component of motion parallax, as it depends on assumptions of relative distances. As knowledge or assumption of what is far and what is near determines the seen direction of motion, this is in part a cognitive phenomenon even though parallax itself is optical.

(viii) Stereoscopic vision

The two eyes give slightly different perspective views with parallax. When the two images from their horizontally separated viewpoints are combined in the brain, depth is seen (Howard & Rogers 2002). This is somewhat similar to depth from motion parallax, though stereo does not have the cognitive component of assuming what is far or near, as the base given by the separation of the eyes is fixed, and which eye is which is almost certainly hard-wired in the brain. So stereo is not subject to the ‘flipping’ changes of motion parallax.

Switching the eyes optically or reversing the pictures in a stereoscope will, however, reverse stereo depth. But here, a remarkable component of knowledge emerges—the brain refuses reversal of depth, for objects highly unlikely to be hollow. It is practically impossible to see a face as hollow, when viewed stereoscopically with switched eyes. Conversely, it is practically impossible to see a photograph of a hollow mask as hollow, with normal vision. This is a particularly powerful demonstration of the power of knowledge in vision, and useful experimentally. Parts of objects (including a face) may reverse independently other parts ‘refusing’—reflecting the power of probabilities for parts, as well as of whole objects.

In general, stereo can override and correct false perceptual assumptions of distances and shapes. Thus, the odd-shaped Ames Room (figure 11), seen with both eyes, will tend to look its truly odd shape, rather than the normal rectangular room it appears with one eye.

Figure 11.

Ames Room. An odd-shaped room designed so that from a critical position its image (in a single eye) is the same as a normal rectangular room with each feature being expanded linearly with increased distance. As the eye's image is the same, it must look like a corresponding normal room, until objects are placed in it. Then the brain must decide whether the room is an odd shape or the objects are odd sizes. The room usually wins.

Exploring the odd-shaped room by touch, or with a stick, also tends to make it appear its true odd shape (Ittelson 1952). The senses check up on each other; providing knowledge other senses can use; though seldom removing such errors completely.

(ix) Pseudo-parallax

There are subtle compensations to motion parallax. One sees these compensations when there are apparently different distances without true depth. Moving while looking at a strongly perspective picture gives an extraordinary illusion. The entire scene of the picture may swing round to follow one, as one moves. Although it is flat and physically fixed, it appears to rotate with the viewer's movements. Evidently, one is seeing the compensation for normal parallax, as set by the apparent depth of the picture. So, this ‘pseudo-parallax’ is in the opposite direction to true parallax.

This is most easily investigated with three-dimensional projected pictures on a large screen. They rotate with observer motion, around the distance of zero disparity, which can be set by changing the toe-in of the projectors. Objects at infinity slide across the screen without rotation.

These kinds of effects are seen in an after-image of a dark room—changing in shape, as one walks with the after-image stuck in one's eyes. One is seeing constancy compensations, which normally give stability against movements, but here generating illusory motion, as there are no retinal image changes to be compensated. So, in these illusions, we experience normally hidden dynamic processes underlying all normal vision.

(x) Portrait eyes

The effect of seeing the eyes of a portrait following one's self is owing to object knowledge, rather than rules of perspective or other general cues. The knowledge here is that eyes keep aimed at the moving viewer by rotating. The general rule knowledge, and specific object knowledge, give opposite effects. For one is compensating, though not the other.

(xi) Reverspectives

The London artist, Patrick Hughes, paints perspective scenes on pyramid shapes that stick out, though appear depth-reversed, as the scenes are painted in reversed perspective—further features being painted larger. The result is strangely effective, with the painted scene rotating in weird ways, to follow the observer as he moves across the picture (Papathomas 2002). Scenes painted on curves, such the inside of a dome, move and change in remarkable ways that have not been fully investigated.

(d) Contrast

The senses signal differences rather than absolute values, of intensity or length or weight and so on. The smallest difference that can be detected increases proportionally to the intensity, or the length, and so on. So, in a dark room, lighting a single candle has a huge effect, and adding one more to a few candles will markedly increase the brightness; but add one candle to 10 or more and the difference is hardly noticeable. This logarithmic relation is Weber's law.

$$(\mathrm{\Delta }I/I=\text{const}.)$$

Perception works very much from contrasts. It is contrasts at edges that provide most visual information. So it is not too surprising that simple line figures such as cartoons can be so effective. There are many contrast illusions, of brightness, colour, size, orientation and so on. The first two are primarily at the signal processing level, though these are complicated phenomena, which can be associated with real or illusory distances. Contrast illusions of size seem to be cognitive, the Titchener illusion (figure 1) being a simple example.

(e) Confounded ambiguity

Unfortunately the word ‘ambiguity’ is itself ambiguous. ‘Ambiguity’ may mean failing to distinguish different stimuli or objects. Or very differently, it may mean creating different perceptions from one stimulus or one object. We may name these ‘confounded’ and ‘flipping’ ambiguities. Confounded ambiguity—failure to distinguish differences—can be owing to lack of contrast. Contrast must be sufficient to give signals that are significantly different from residual randomness or ‘noise’ of the nervous system. Neural noise impairs discrimination, very much as electronic noise limits sensitivity of detecting and measuring apparatus.

For distinguishing brightness differences, the areas of the comparison fields of light affect discrimination just as R. A. Fisher described statistical functions used in agriculture for showing effects of fertilizers on fields of various sizes (Fisher 1935). Both have the same square root functions, including maximal discrimination when the background and test fields have the same area (Gregory unpublished work).

(i) The Ames Room

This is an odd-shaped room—full size or a model—which gives the same retinal image as a normal rectangular room by using reverse perspective, further features being made correspondingly larger. As it gives the same image (to one eye) as a normal room, it must look the same. But this raises the question: why does a normal room look rectangular, though an infinity of differently shaped objects would give the same image? The limits and frequencies of alternative flipping perceptions are given approximately by probabilities depending on knowledge. Exploring the Ames Room by touch, or with a long stick, tends to make it appear its true shape. This information needs to be introduced into perceptual systems; conceptual understanding, as by testimony from other people, does not generally correct visual perception, and does not affect the Ames Room. (This makes experiments on illusions easy to perform, as the truth does not generally have to be kept secret, though it is important to check the observers' knowledge or belief of what is going on.)

The false appearance of the Ames Room is corrected with stereo vision, as with two eyes the room appears its true shape. An important general use of stereopsis may be to check and correct assumptions of monocular vision where distances are poorly signalled, so assumptions are important.

The Ames Room becomes interesting when there are objects, such as people, inside it. A distant person can be made to look the same distance as a nearer person, but will have a smaller image in the eye. This presents the brain with a question: are the people the same size, or is the room an odd shape? Generally the room ‘wins’ by looking normal, though it is not, and the people looking different sizes, though they are the same size. The brain has to assess probabilities, and in this situation guesses wrongly.

(f) Flipping ambiguity

Perception can spontaneously flip between alternatives when confronted with incompatible signals, or with equally plausible hypotheses of what may be out there. A well-known example of incompatible signals is retinal rivalry. When the eyes receive very different colours, or different patterns, the brain is unable to combine them into a single stable perception. First one then the other is seen, sometimes with changing combinations of various regions.

There are many well-known figures that change spontaneously, flipping from one alternative to another. These phenomena reveal most clearly the dynamic nature of perception. When the brain creates alternative hypotheses only one can be accepted at a time, as only one behaviour is possible. These are entertained in turn when the brain cannot make up its mind.

Flipping ambiguity depends on relative probabilities of alternatives. When there are only two likely possibilities, as for the Necker cube (figure 10a), or the Duck–Rabbit (figure 10b), flipping is dramatic.

Inkblots, however, provide hundreds of slowly changing perceptions, and meanings which one may say are ‘projected’ onto the blots (figure 12).

Figure 12.

Ink blots. The changing patterns show dynamics of perception. Whether or not these are sufficiently characteristic of the individual to be clinically useful is a controversial topic.

A question is: why are inkblots so visually labile, so evocative of alternative ‘hypotheses’? Why do all patterns and objects not change in this sort of way? Although not fully understood, we may assume that the shapes of inkblots are too vague to specify particular hypotheses. This must tell us a lot about art. Perhaps random patterns are useful references for meaning. Vision creates meanings even where none are present or intended.

(i) Wire cube

A three-dimensional wire cube is a fascinating and revealing ambiguous object. (The wires should be black, to minimize the occlusion cue of the nearer wires hiding the further.) When it reverses in depth, it seems to stand up on a corner and rotate to follow one as one moves around it. Also, the cube changes shape. When flipped in depth, the apparently further face looks too large. So instead of appearing as a cube it looks like a truncated pyramid. This change of shape with change of apparent distance is strong evidence that constancy scaling can work ‘downwards’ from the prevailing perceptual hypothesis (Gregory 1970).

Flipping can occur against evidence from other senses. Holding a wire cube in the hand while seeing it depth-reversed is a peculiar sensation. When the hand is gently rotated, the cube seems to rotate against the hand's movement. This feels (though painless!) as though one's wrist is broken. Recent experiments with functional magnetic resonance imaging are beginning to show where this flipping—this perceptual decision making—is processed.

(ii) Hollow face

The most dramatic demonstration of the power of knowledge affecting vision is the hollow face illusion—a hollow mask appearing as a convex face, until viewed closely with both eyes (figure 2).

We may see this as conflict between bottom-up information that it is hollow, and top-down knowledge that faces are convex objects (Gregory 1970, 1973). At an intermediary distance it flips spontaneously between hollow and convex, as bottom-up information that it is hollow balances top-down knowledge that it should be convex. The bottom-up information is primarily stereo, and, when the lighting is from above, shape-from-shading. As both are easily controlled, simple experiments are revealing and convincing. If the face is shown inverted, the cognitive top-down contribution is weaker (Hill & Bruce 1993), no doubt because the face-probability is reduced.

The phenomenon was known to Helmholtz, but though he did think of perceptions as inferences, top-down knowledge was not then an accepted concept. Helmholtz (1856–67) attributed the effect, which he noted in converse medals, to confusing shadows. However, it works fully with a back-illuminated transparent mask where there are no shadows, and it resists opposed shape-from-shading by lighting from below. It seems safe to attribute this powerful visual phenomenon to knowledge, knowledge that faces are convex objects.

(g) Distortion

These are perhaps the best-known illusions, especially systematic errors of length and curvature of lines. They are often called ‘optical illusions’, but this is a misnomer, except for example by stigmatism. After a century of intensive investigation, explanations for many of these effects remain controversial. Controversy centres on whether these are signal processing physiological errors or are cognitive. This important distinction is not always easy to make. There are examples of both, which we will look at in turn.

(i) Physiological distortions

An example of a signal processing physiological distortion is the Café Wall illusion (figure 7). This is markedly affected by brightness differences and by the thickness of the lines (especially the width of the ‘mortar lines’) which suggests a retinal signal processing origin. When the ‘mortar’ is darker or lighter than the ‘tiles’, the distortion is lost. When the tiles are alternately coloured, for example, red and green, with no brightness difference, again there is no distortion. The distortion reverses when alternate rows of tiles are shifted by half a cycle. This figure allows a variety of changes, and yet is not too complicated for analysis (Gregory & Heard 1979), so it is useful.

It may be noted that the Café Wall has parallel lines, and lines at right angles; but no converging lines as of perspective, or any other depth cue. This makes the Café Wall different from the classical ‘geometrical illusions’. The Café Wall is a symmetrical figure—as any region may be exchanged for any other region, without change of the effect. So it seems to contradict Curie's principle of physics that symmetry cannot produce systematic asymmetry.

So how can these long asymmetrical wedges be produced by the symmetrical pattern of tiles? There are small-scale asymmetries, for each pair of dark and light tiles, across the mortar. These local asymmetries produce small wedges (that can be seen individually with smaller tiles). The local wedges are integrated along the figure, as a second process, to give the long wedges of the mortar line. As suggested by Fraser (1908) for the Fraser Spiral illusion.

What causes the small-scale ‘primary’ distortions of the Café Wall? Opposite-contrast tiles seem to suck together, across the neutral mortar. This effect can be isolated and studied in detail. A grey rectangle, with a narrow light stripe at one edge and a narrow dark stripe at the opposite edge, will move dramatically when made brighter or darker than the surround, and is displaced according to its brightness. This seems to be the basic phenomenon. It turns out that the plotted functions for change of position, velocity, and stereo (when oppositely shifted rectangles are viewed in a stereoscope) are all different. These seem to reflect different channel characteristics for signalling position, velocity and stereo (Gregory & Heard 1983).

(ii) Cognitive distortions

The principal candidate for explaining many distortion illusions is the inappropriate size constancy scaling theory. The notion is that objects generally remain quite constant in size over a wide range of distance, as apparent size is scaled by distance cues or (top-down) from seen assumed distance. This is shown from flipping ambiguities, such as the wire cube which changes apparent shape when depth-reversed, without change of bottom–up signals (see §1f(i)).

The so-called geometrical illusions have perspective–depth features, such as the converging lines of the Ponzo illusion, and the arrowheads of the Müller-Lyer illusion, which may represent perspective corners (Gregory 1963). It is suggestive that for all these distortion illusions represented, distance is associated with illusory expansion. This is the opposite of the normal shrinking of retinal images with increasing distance. As in normal scenes, objects appear much the same size at different distances, their images shrink to half with each doubling of distance. The suggestion is that in flat pictures presenting depth cues such as perspective, these cues set constancy scaling inappropriately, as the picture is flat. In all cases, depicted distance gives illusory expansion. This is to be expected, as scaling is normally set to compensate shrinking of the retinal image with increased object distance (Gregory 1963, 1968, 1970). Scaling is inappropriate—producing related distortions—when it is set by depth cues on the flat picture surface, or for flat objects having these perspective shapes.

It seems that scaling may either be set ‘upwards’ from depth cues or ‘downwards’ from seen distance. This is shown by the depth-ambiguous wire cube changing shape. When flipped in depth, the apparently further face looks too large, though there is no change of stimulus. Here, constancy scaling is working by following apparent distance top-down.

The simplest and clearest example of bottom-up scaling from perspective is the Ponzo illusion (figure 13). The converging lines represent depth, increasing with the perspective convergence; features signalled as more distant are expanded. This works for a simple line figure (the Ponzo illusion) or a photograph of a perspective scene (figure 13).

Figure 13.

Ponzo illusion. The simplest and clearest perspective distortion illusion. As for all these illusions, features depicted though not necessarily seen as more distant, are expanded.

A similar though less obvious example is the Müller-Lyer illusion (figure 14), which is a perspective drawing of corners.

Figure 14.

Müller-Lyer illusion. The double ‘arrow’ with diverging ‘fins’ looks longer than the converging-fins arrow, though they are equal. The theory accepted here is that this is a perspective illusion, the ‘fins’ being perspectives of corners, inside and outside, respectively. Like the Ponzo (figure 13), this represented distance gives expansion: expansion for the inside corner and contraction for the outside corner, according to the perspective depths signalled by the ‘arrows’, accepted as perspective corners.

These distortions should not occur when depth-cues are appropriate. So what happens for these figures when they are truly three-dimensional and seen correctly in three dimensions? The distortions are lost (Gregory & Harris 1975). This seems good evidence for the inappropriate constancy theory.

The Judd Illusion (figure 15) presents a corner (like the Müller-Lyer, figure 14), but as viewed from one side, and so tilted asymmetrically in depth. The dot in the centre is displaced—again by illusory expansion associated with represented distance.

Figure 15.

Judd's illusion. Here, the ‘arrows’ are not opposed, as in the Müller-Lyer (figure 14), but point in the same direction. This gives tilt in depth, and the central dot is displaced by the usual expansion with depicted depth (normally giving size constancy).

What applies in physics may not apply in perception, as perceptions are essentially separate from the physical world and can take-off from physical reality. But can general physical principles (such as Curie's principle, that systematic asymmetry cannot be generated from symmetry) occur in perception, though impossible in physics? The repeated pattern of the Café Wall (figure 7) generates large-scale asymmetries by two stages: (i) local asymmetries giving small wedge distortions, followed by (ii) summing of the little wedges along the mortar lines to form the long wedges. As these are seeded by local asymmetries, Curie's principle is not violated.

The Zöllner illusion (figure 16) lacks elements for producing small-scale asymmetries, which might, by adding successively, give this large-scale asymmetry, though the pattern is symmetrical as it is repeated. One might think that the short cross-lines could do the trick by displacements of the Poggendorff illusion (figure 17), but these displacements are in the wrong direction, and are minimal with these thin lines.

Figure 16.

Zöllner tilt illusion. The repeated short tilted lines signal depth, as perspective corners, like the risers and treads of a staircase. It is suggested that the represented tilted depth introduces asymmetry, much as for the Judd illusion (figure 15). Again the asymmetry is in the representation, not in the pattern, which is symmetrical, as it repeats. Does this ‘save’ Curie's principle?

Figure 17.

Poggendorff displacement illusion. The thin oblique line (which seems to be a single perspective line) is displaced across the horizontal rectangle.

The Poggendorff displacement seems to be another perspective effect, though from a single oblique line. The displacement is practically lost when the long thin line is vertical. Try rotating the page to make it vertical. (A three-dimensional wire model of this shows no displacement, as the oblique goes off into the distance as a continuous line or wire.)

The Café Wall distortion (figure 7) is sensitive to luminances, and thickness of its lines and other features, but this is not so for the Zöllner, and the Zöllner does not seem to be successive tilts or displacements like the Poggendorff. So how can the Zöllner have its asymmetrical distortion?

The short lines form repeated perspective corners, essentially like the Müller-Lyer. Like the Müller-Lyer they represent depth. But they represent a tilted depth, like the repeated treads and risers of a staircase viewed from the side. Again, greater distance is associated with expansion. (This is the same as Judd's variant of the Müller-Lyer; figure 15.) So there is global asymmetry—but it is in the represented tilted depth not in the repeated pattern presented to the eye. Does this ‘save’ Curie's principle? Only by taking the principle outside physics into perceptual hypotheses.

(h) Grouping

The Gestalt psychologists stressed the importance of grouping with their laws of organization. Working mainly with patterns of dots, they showed that there are strong tendencies to group dots according to proximity, continuity, closure and common fate movements (figure 8). These principles can be used in nature and art to conceal and confuse, as in camouflage.

Grouping can also be from meaning. A well-known example is the Dalmatian dog (figure 9). Once recognized as a dog, it is clear which dots are part of the dog and which are pebbles on the beach. However, it is very hard to see the dog when presented upside down. This is evidence of a cognitive effect, as dogs are seldom upside down and so this is unlikely from our knowledge of dogs.

Although seeing seems simple and easy, all object perception is puzzle-solving. The Dalmation dog shows we are aware there is a problem. Missing parts of the dog are constructed, perhaps partly from Gestalt laws of organization and from top–down from our knowledge of dogs.

(i) Glass effect

Leon Glass noted (ca 1970) that a random dot pattern superimposed on itself (with a transparency) and somewhat rotated or displaced, appears as circles or lines (Glass 1969; figure 18).

Figure 18.

Glass effect. A random dot pattern, superimposed on itself and slightly rotated, appears to have circles. Simple displacement gives lines by visual grouping. This seems to be a low-level grouping phenomenon.

(j) Impossible

It is significant that although we tend to see what is likely, we can see things so unlikely they are impossible, even paradoxical. Paradoxical perceptions can arise bottom-up when one or more parallel channels are upset, so disagreeing with other channels. (This is like witnesses disagreeing at a trial. The Judge may reject the less likely evidence, or have to accept that a car is moving at two speeds at the same time, and is red and blue!) We found this for the after-effect of movement, when motion is seen without change of position, as the motion channel adapts though the position channel is unaffected by the prolonged moving stimulus (see §1c(ii)).

This is a bottom-up effect, but paradoxes can also be cognitive. Perhaps the first example of a paradoxical picture is Hogarth's engraving The fisherman (1754). Simpler examples are the impossible triangle and impossible staircase figures of Lionel and Roger Penrose (Penrose & Penrose 1958). These are the basis of many of Mauritz Escher's remarkable pictures.

It is remarkable that we can experience a visual paradox, even while knowing the answer conceptually. The impossible triangle looks impossible when the ends of the sides optically touch at the corners, even when they lie at different distances—though when a depth difference is seen, the paradox disappears. This knowledge must come from vision. Knowing conceptually that the ends are separated in depth does not destroy the visual paradox.

That conceptual understanding does not correct the illusion shows that perception can be powerless to correct errors of understanding, and the reverse is also true. This shows modularity of the brain. It also highlights why we need scientific method to gain reliable understanding from perception, and to guard against illusions.

If we could see only probable objects, we would be blind to the unlikely; but this would be dangerous, as unlikely events do sometimes occur. Indeed, if we could see only expected things and events, there could hardly be perceptual learning.

The ability to see impossibilities raises issues of how the brain represents. The Gestalt psychologists thought the brain represented objects with similar-shaped brain traces. Such isomorphism is a commonly held notion; though it is deeply flawed, as recognized over 2000 years ago by Theophrastus. Theophrastus (ca 372–286 BC) criticized Empedocles' isomorphism for hearing:

It is strange of him (Empedocles) to imagine that he has really explained how creatures hear, when he has ascribed the process to internal sounds and assumed that the ear produces a sound within, like a bell. By means of this internal sound we might hear sounds without, but how should we hear this internal sound itself? The old problem would still confront us.

Although as noted by Theophrastus, the notion of isomorphism gives only the semblance of explanation, and generates confusions and paradoxes, it has been held ever since even by scientists of great distinction. The axioms of physiological psychology proposed by G. E. Müller in 1896 posited cortical activities similar to perceptions:

To an equality, similarity or difference in the constitution of the sensations…there corresponds an equality, similarity or difference in the constitution of the psychophysical (brain) process, and conversely.

The historian of psychology, Boring (1951), points out that this isomorphic account was shared by many others whom he references.

How could an impossible perception be represented isomorphically? A model Penrose impossible triangle appears impossible when its true three-dimensional shape is hidden; but an equivalent model in the brain would need an equivalent ‘eye’, at a position where the true shape of the model is hidden. Theophrastus would love it!

Could representation in digital coding, such as language, escape this problem? Surely yes, for an impossible perception can be described, though not made. If Curie's principle can be violated, a perceptual internal model would be impossible, but it could be described symbolically, as symbols can have any form.

The word mum is symmetrical, yet one's mother is not symmetrical fore and aft. Conversely, circle or sphere do not need to be asymmetrical to represent their symmetrical objects. There would seem to be no problem for language-like or digital asymmetrically shaped brain states representing symmetrical objects or patterns, or for symmetrical states to represent asymmetries. Like words, such brain-symbols would have conventional meanings unrelated to their shapes. This might work for visual representations, with lists of characteristics like Irving Biederman's (1987) geons—characteristic features combined in various ways. Biederman's papers show geons as pictures; but this does not mean they are pictures in the brain. They could have quite different forms, read as a code for shapes, symmetrical or not, combined like words to describe objects.

(k) Fiction

Perceptions are far richer than available sensory data, being enriched by knowledge. Perceptions may be largely fictional, but this does not mean they must be wrong. Fictions can be importantly true, though not based on immediate data. Leaps beyond data are common in science, very often justified by later evidence. Fictions fill gaps and raise questions, which enrich and inspire science, though of course they may mislead.

(i) Filling-in

There is a large blind region of each retina where the optic nerve leaves for the brain, yet we do not normally see a great black cloud hovering before the eyes. There are two likely reasons, and both may be true. The American philosopher Daniel C. Dennett (1991) suggests that the blind spot is not seen because the brain ignores this region, as it never gives useful information (like ignoring a boring person at a party). Yet there is evidence of active filling-in, by copying the surrounding retinal pattern (Ramachandran & Gregory 1991). It will complete patterns, but will not, for example, add missing noses. So this is hardly a high-level cognitive effect. It would be interesting to look for clear evidence of filling-in from meaning. The Dalmatian dog (figure 9) could be an example.

The Italian artist–psychologist Kanizsa (1979) and Petry & Meyer (1987) produced superb examples of illusory contours and surfaces, which put these beautiful phenomena on the map—though they have been known to psychologists since about 1900 and in art from cave paintings. Fictional contours do not stimulate brain cells in the first stage (V1) of processing. They seem to be quite low-level cognitive phenomena, obeying simple rules. Although not physically present, illusory contours can give just about every perceptual effect of true contours, including inducing distortion illusions. Some may be owing to assumptions of occlusion, surprising gaps being attributed to eclipsing, by some nearer object or surface—which the visual system creates as a fiction.

It is interesting that a pair of such figures having somewhat different ‘slice’ angles, and so different curvatures, viewed in a stereoscope will produce three-dimensional illusory surfaces stemming from the centre of the ‘cakes’, and not incorporating the edges of the ‘slices’. So the physical lines that give rise to the stereoscopic perception are abandoned in favour of illusory three-dimensional contours, generated as fiction (figure 19; Gregory 1972).

Figure 19.

Kanizsa's ghostly triangle. The missing ‘portions of cake’ of the black discs, as they line up, are accepted as evidence for a (non-existent) nearer occluding surface–which is created by the visual brain. Probability favours this, but here the brain has made the wrong bet. This seems to be a quite low-level cognitive creation.

Subtle effects of ‘neon-spreading’ of colour leaking through gaps in contours should be useful for artists. To modify a Dan Dennett remark: these mental figments extend the pigments available to painters.

(ii) Phantasms

Vision easily takes off from control by stimuli, as in dreams, with psychedelic substances, with inkblots when visual clues are weak or contradictory. Perceptual fantasies may be projected into the world of objects, and indeed frequently are: the man-in-the moon, ships-in-clouds, faces-in-the-fire, inkblots (figure 12). These bizarre phenomena show the creative insubordination of vision.

(iii) Peeriodic table

To bring order to knowledge it often helps to classify phenomena. Classifications can reveal gaps, and relate experiments and observations to theories. We will now suggest (with apologies to Dimitri Mendeleyev) a peeriodic table (see table 1) of kinds and causes of visual phenomena, especially illusions.

Table 1.

Peeriodic table: elements of perception and illusion.

kinds of phenomena	causes (reception/perception/conception)
bottom-up signals	side-ways rules	top-down knowledge	explicit understanding
nonsense	blindness Loss of signals, or cortical processing damage. Long-term blindness has no sensation—like behind one's head. colour anomaly From loss of colour channels, or spectral shift of cone receptors. There may also be ‘cross-talk’ between channels. channels All perception starts from receptors and neural channels, signalling restricted features of the external world, and states of the organism. Most of the universe is dark matter to the senses.	nonsensical perceptioninappropriate rules Though handled by physical systems, perceptual rules are not laws of physics. They generate perceptions (like Chomskian grammar?) so can generate nonsense, such as paradoxes, from false knowledge or assumptions. When rules are inappropriate, illusions occur with normally functioning physiology. Then explanation must be from rules (or misleading knowledge) and not from the physiology, when this is working normally.	agnosia Lack of visual knowledge. Failure to recognise even familiar things, when visual knowledge not available. change-blindnessirrelevant changes not seen. Perceptual hypotheses may continue until ‘internally’ checked or challenged. inattention-blindness As in conjuring's controlled attention. familiarity blindness Surprise is necessary for information. Low information signals can be ignored as useless.	ignorance Without understanding, the world looks like a conjuring trick. But perceptual experience can be very different from conceptual understanding, and they may conflict. Understanding does not always correct perceptual errors. Many illusions remain in spite of knowing the truth; the truth does not need to be hidden for most experiments on perception, (though explicit knowledge and belief must always be considered.)
context	relative Visual signals code differences, rather than absolute values. Edges and borders carry most information, so are very important. We see brightness, size, distance, etc., by relative judgements, from statistically significant differences. Regions of constant brightness are seen by no change beyond a border.	discrimination Contrast sensitivity is the key to all vision functions. Generally limited by available photons, and neural signal/noise ratio. Discrimination for differences is given by a constant ratio, Weber's Law (ΔI/I=const.). contrast illusions Contrast from surrounding regions may be enhanced by neural interactions: colour constancy, land colours.	clues Visual clues depend on context (like Sherlock Holmes). context-illusions Context can give illusions such as Titchener's circles. (Surrounding dwarfs, make the pyramids look larger!) induced movement Large distant objects are accepted as stationary references—giving induced movement when they move.	meaning Conceptual hypotheses of science are explicit while Perceptual hypotheses are implicit. Context gives meaning for both for perceptions and conceptions. Phenomena cannot ‘speak for themselves’, but need to be interpreted, from knowledge or assumptions, for conceptual significance.
instability	jazzing McKay rays, Op. art jazzing of repeated patterns (stimulate on–off cells with eye tremor.) retinal rivalry Shimmering on polished metal. local drifting At isoluminance. Ouchi illusion. (lack of ’border-locking’?)	grouping Dot patterns group and regroup with Gestalt Laws. Unstable, with inadequate or competing Laws, or Rules: closure, contiguity, common fate, etc. glass effect Random dot pattern, superimposed on itself and slightly displaced, shows lines, or if rotated shows striking circles.	constancy The world generally looks stable, in spite of observer and eye movements. ‘Constancies’ compensate. But when inappropriate, they can generate illusory motion and many other illusions. pseudo parallax Perspective pictures rotate with observer.	conceiving objects Criteria for perceiving and conceiving objects can be very different. Visual objects are not signalled as units, but are brain-constructed, from selected characteristics transmitted by parallel sensory channels, and from past knowledge. Conceptual objects (such as electrons) may be unsensed, but inferred.
confounded ambiguity	distinguishing stimuli Distinguishing differences is limited by neural noise, and lost with overlapping response curves (red+green must look same as monochromatic yellow light, as the R and G responses overlap).	distinguishing objects When their stimuli are the same, different objects must look the same. Ames Room Has the same retinal image as a normal room—so must look the same. But interesting, when objects are inside it.	classifying objects Different kinds of objects are often confounded when not familiar or understood. E.g. fossils, or makes of cars. special knowledge with defined differences is important for classifying.	explaining phenomena As science develops, understanding and appearances separate and often conflict. So classifications change, with new science. This can be circular—as phenomena suggest explanations and explanations interpret phenomena. (Cladistics aims to avoid circularity in evolutionary accounts, by being theory-neutral).
flipping ambiguity	epilepsy The ‘Sacred disease.’ Spontaneous neural activity. Neural nets are dynamic and can be physically unstable, especially when inhibition or negative feed-back fail. migraine Visual disturbances, associated with headache.	figure-ground The most basic decision, is whether there is or is not an object present. This is seen dramatically in flipping figure-ground ambiguity, when the brain cannot make up its mind. Object recognition starts from general rules such as the Gestalt Laws, but when they are inadequate, or conflicting, figure-round is unstable and flip	flipping perceptions Perception flips to alternatives when the brain cannot make up its mind. E.g. Necker Cube, Duck–Rabbit—set by probabilities. hollow face Probabilities normally give stability but can mislead—Hollow face looks convex. stereoscopic vision Resolves ambiguities of distance and form—challenging rules and knowledge.	‘collapsing’ reality Objects do not ‘flip’ to other objects—except in quantum physics. Measurements or perceptions are supposed to ‘collapse’ many possibilities into particular realities. This creation of reality by perception selecting from overlapping possibilities, has been ascribed to consciousness. But this is totally mysterious.
distortion	signal distortions Many visual illusions are due to signalling errors such as cross-talk and lateral inhibition. after-effectsIllusory tilts, curvatures, colours, motion, etc. After-effects may serve to re-calibrate the senses, but can be wrong.	cognitive distortions Inappropriate constancy scaling distortions: Ponzo, Müller–Lyer, Judd, Poggendorff, Hering, Horizontal–Vertical, Harvest Moon, etc. Features signalled as more a distant are expanded, as depth cues set size-scaling to represented distances in pictures. (Scaling is also set by perceived depth, changing with depth-ambiguity.)	anticipation Prediction is essential for cognitive perception—but can mislead size-weight illusion Smaller objects feel heavier than larger objects of the same scale weight. False expectation, as larger objects are usually heavier—setting too much muscle power when actually the same.	reference truths An object cannot itself be distorted, but may differ from accepted references. thus a ruler is bent, or too long or too short by reference to some other ruler, accepted as ‘true’. Reference to non-illusions are essential for measuring illusions; though illusions and errors can show up as internal inconsistencies.
impossible	conflicting signals Neural channels signal various object properties and states of play. Parallel channels can disagree, as when some are adapted differently. Then perceptions may be impossible. The spiral after-effect expands or shrinks, yet without changing size.	impossible objectsThe impossible triangle can exist yet from certain positions it appears impossible. Perceptual hypotheses generated from false assumptions can be paradoxical. The sides of the impossible triangle touch optically at the corners, though some are separated in depth. The false assumption of touching physically generates the paradox.	knowledge conflictsMagritte's painting of the back of a man's head in a mirror, instead of his face. This disturbs, as it is counter to one's implicit visual knowledge of reflections. Conflict of signals with failed predictions, are key to correcting present and evoking new perceptions and, especially, new conceptions.	cognition and physics Though the brain is a physical system, its ‘virtual realities’ are not limited to physics—making it possible to imagine and to see even impossibilities, including paradoxes. The same holds for computer software. Software rules are not laws of physics, and physics is not a guide for cognitive research.
fiction	spurious signals After-images persist, as though from external objects. Projected into space, they appear as external objects though they are retinal ‘photographs’. phi movement Alternating separated lights appear to move, by stimulating normal movement systems, which are tolerant of gaps.	grouping Dots group into object-like shapes, especially with the Gestalt Laws. ghostsKanizsa triangle and many others. Object-shaped gaps can be evidence of nearer occluding objects, here creating fictional surfaces and contours, ‘postulated’ to ‘explain’ the gaps.	phantasmsFaces-in-the-fire; man-in-the-moon; inkblots. These show the creative dynamics of perception, when alternative hypotheses are evoked from minimal evidence. As perception is dynamic, perceptions can break free from stimulus control with a life of their own.	perceptions and conceptions Perceptions are much like predictive hypotheses of science, so may be called ’perceptual hypotheses’, both are derived from interactive experiments; are indirectly related to reality; change with changed knowledge or belief; are subject to errors and illusions—some useful for art, and for investigating perception and mind.

2. Part II

(a) Vision for knowledge

Having considered knowledge for seeing, we turn now to the converse: seeing for gaining knowledge. As this will be short on evidence and on conclusions, we can be short on words. There will be more questions than answers, but these might hint at some needed ideas and experiments.

It is well known that the Scottish philosopher David Hume (1711–76), suggested that all ideas depend on prior sensations (Enquiry into human understanding 1748). He wrote (§2, Of the origin of ideas):

Nothing, at first view, may seem more unbounded than the thought of man, which not only escapes all human power and authority, but is not even restrained within the limits of nature and reality. To form monsters, and join incongruous shapes and appearances, costs the imagination no more trouble than to conceive the most natural and familiar objects.

Hume suggested that philosophical ideas are dubious when not apparently derived from sensations. Although: ‘we need but enquire, from what impression is that supposed idea derived?’ with ‘reasonably hope to remove all disputes’. This now seems optimistic, if only because theoretical science has shown how far abstract ideas are removed from sensations. Yet it may be that when, far removed, although useful, they cannot be imagined. Also, sensations seem virtually useless until given significance by knowledge. We have said that retinal images are mere patterns of form and colour, becoming significant when enriched with knowledge of non-optical properties of things.

A well-known illusion illustrates visual knowledge affecting behaviour: the size–weight illusion. Smaller objects feel heavier than larger objects of the same scale weight. This is based on the knowledge (presumably derived from hands-on experience) that larger objects are generally heavier than similar smaller objects. More muscle power is called up to lift the larger object, though as here it has the same weight as the smaller, the smaller feels and is judged to be heavier. This is a large (over 20%) effect, robust and easily measured.

How much does vision itself add to knowledge? This is a practical question for education, for television, for museums and for hands-on science centres. The assumption behind hands-on science centres, and practical classes in schools and universities, is that passive seeing alone is not adequate for conveying concepts. This challenges TV science programmes; though it would be absurd to claim they lack any power to convey knowledge, or inspire interest. But informative television programmes are more than visual. In addition to the pictures, they have presenters using spoken and sometimes written words. The pure case for evaluating vision alone would be programmes without presenters or language, but these hardly exist.

Perhaps they do not exist simply because unaided vision is ineffective. The obvious experiment is to turn off the sound. Then, the programme is practically impossible to follow. Yet this is not quite fair, as the programme was planned to have spoken explanations. Could silent TV be produced to be adequate for science, or other informative programmes? Perhaps this is an experiment waiting to be tried. But we may ask: how much more informative is TV than radio? The joke is of course that the pictures are better on radio. Is this just a joke? Inputs can inhibit imagination. The point though, is that vision is of little use on its own. We believe that vision developed from senses of taste and touch, and in general, pictures are useless without supporting knowledge. Evidently, vision needs knowledge to provide further knowledge, and this can be infinitely enriching. Surely this should be the basis of informative television, museums and teaching in schools and science centres.

Pictures are seen with the viewer's interactive knowledge of objects, derived through years of experiencing with the various senses and discovering causal relations hands-on. Knowledge from interactions with objects allows images to stand for solid functional things. This is so for retinal images of normal objects, as for paintings and photographs. But representational pictures have a conflict absent from usual retinal images. For pictures evoke objects visually, while we know conceptually they are not there, but in another space and time, or are figments of the artist's imagination (Gombrich 1956).

The most clearly conceptual pictures are graphs; but a graph is useless unless it is known what it is a graph of. A rising curve may be an increase of anything—temperature, or income, perhaps—and the same rising curve may represent the opposite trend, such as cooling or debt. Accepted units are needed, such as degrees centigrade or pounds or dollars. So, although graphs convey knowledge, they need additional knowledge to be seen usefully. This seems generally true of all pictures. Multi-dimensional graphs are particularly interesting: how many dimensions can be visualized? How many can be accepted conceptually?

We have distinguished between implicit visual knowledge and explicit conceptual knowledge. For picture languages, such as Egyptian hieroglyphics, as the language develops, the pictures become arbitrary symbols whose meanings have to be specially learned. Our letter m may have started as a picture of water, or the sea, but this is entirely lost as a letter of the alphabet. Cartoons convey conceptual meanings widely, as gestures are largely international; though political cartoons only work in a context of shared knowledge.

How much can vision add to pure abstract conceptions? The obvious example to consider is geometry. At least for most people, diagrams are essential for understanding the steps of a proof. There may be exceptions of course—Sir Francis Galton (1883) reported that the very best geometers had poor visual imagery (with an inability to describe their breakfast table). Some illusions can make geometrical diagrams misleading—as does Saunder's Parallelogram (figure 20).

Figure 20.

Saunder's parallelogram. The inner oblique lines XA and AY are the same length, though they look different (XA looks much longer). This visual illusion could upset conceptual ‘seeing’ of a geometrical proof.

Engineering is dependent on diagrams and pictures, being weak in words for structural shapes. Vision for surgery is guided and informed by knowledge of anatomy. Artists such as George Stubbs, painting horses, studied anatomy in detail. Diagrams are also important in chemistry, though more theoretically based, combining structures with functions, which have to be appreciated conceptually. The same holds for circuit diagrams.

Causes are typically conceptual and often hard to determine; yet some simple causes can be seen. This was investigated by the Belgian psychologist, Albert Michotte (Miles & Miles 1963), with pre-computer graphics of moving blobs, apparently hitting in causal sequences. But generally, vision only conveys causes from explicit conceptual knowledge, which is often counter-intuitive and surprising.

The extreme case of a purely visual science is astronomy; yet its appearances have often proved misleading. The Sun, appearing to move round the Earth, misled astronomy for thousands of years. It is remarkable that appearances, with their implicit knowledge, may be opposed to explicit conceptual knowledge, and yet both are accepted. A familiar example is seeing the evening sun sinking, though we know the horizon is moving up to meet it, as the Earth we are standing on rotates. The visual experience is clearly opposed to its conceptual understanding, and yet, we live happily with the conflict.

How the brain's perceptions and conceptions are related, and often divorced, has been investigated following the lead of Piaget (1929). Much remains for future experiments. These issues are clearly important for methods and aims of education and public understanding of science (Bodmer et al. 1985; Bodmer 1987). This involves schools and universities, and also science centres attracting children and adults to explore phenomena, sometimes with explanations. The principal pioneer is the founder of the San Francisco Exploratorium, Frank Oppenheimer, who opened it to the public in 1969. The first in Britain was the Bristol Exploratory, opened ten years later, soon followed by others including Cardiff's Techniquest (Gregory 1988).

The most dramatic claim for vision to be effective is the remarkable notion of quantum mechanics, that observations set reality. The notion that reality presents alternative possibilities, selected forever by an observation, may suggest that perception itself is outside physics; even though the brain that is necessary for perception is a physical system, obeying laws of quantum physics. This has suggested to some that although consciousness is brain-based, consciousness is outside physics, though it is where ‘the buck stops’ for reality. The fact that quantum physics and consciousness are both mysterious is of course hardly evidence for thinking they are the same mystery. However this may be, the realization that perceptions are outside physics and may run counter to physics gives pause for thought.

The realization that illusions of vision and all the other senses occur so frequently, even for the most careful observers, in optimal conditions, should promote tolerance to alternative views. More than vision for knowledge, surely this is understanding illusions for survival. Our best counter to being misled by illusions is science. We may hope that a general understanding of the methods of science will save future generations, if not from all dangers of mis-seeing and misunderstanding, at least from believing it is only other people who are deluded.