The third dimension in the primary visual cortex

Abstract

Anatomical superposition of the cortical projections from the overlapping visual fields of the two eyes does not make it obvious how the disposition of objects in the third dimension is encoded. Hubel and Wiesel's demonstration that units in the primary visual cortex of the mammal respond preferentially to elongated contours of specific orientation encouraged the inquiry into whether binocular disparity might not similarly be represented as an attribute interdigitated within the orderly progression of position. When this was found to indeed be the case, this entrained a brisk research activity into the disparity of receptive fields of single units in the primary visual cortex and the influence on their response of the three-dimensional locations of outside world stimuli. That cells’ preferred orientations covered the whole gamut whereas space perception required only horizontal disparity was an apparent paradox that needed resolution. A connection with an observer's stereoscopic performance was made by the discovery that cells in the primate primary visual cortex display good tuning to the disparity in random-dot stereograms. But a wide gap still remains between the properties of these cortical units and human stereo thresholds in simple target configurations, let alone depth judgments in which perceptual and cognitive factors enter. When the neural circuits in the primary visual cortex that are involved in processing depth are eventually traced in detail they will also need to have properties that allow for the plasticity in learning and experience.

With our eyes pointing forward and the visual fields of the two eyes receiving largely overlapping images on their two-dimensional retinal surfaces, it is not obvious how depth is encoded. After the idea of lines of sight and retinal images had been understood at the time of Kepler early in the seventeenth century, it was another two centuries before Wheatstone had the insight to draw separate and slightly different configurations to present to the two eyes and give explicit formulation of stereoscopic vision: we have the ability to disambiguate the minute dissimilarities that result from viewing objects with eyes that are a few inches apart. Not that there aren't many other cues giving hints about the disposition within the world in front of us: perspective, relative size of familiar items, the direction of shadows, nearer contours partially obscuring farther ones, and so on. These ‘monocular’ clues to depth, known and applied by painters since the middle ages, are a subject apart (Gibson, 1950). The 98% of the population with adequate levels of stereopsis need only close one eye and try to thread a needle to recognize what is in play here. When considering the neural basis of these faculties, the higher perceptual and cognitive ones were at first ignored until some success was achieved with the simpler ones, now often subsumed under the rubric ‘early vision’.

The retinotopic organization of the visual input into the central nervous system, firmly anchored by anatomical research in the nineteenth century, does not at first help us understand the physiology involved. Signals arriving in the cortex through the two eyes are overlaid in a two-dimensional cortical map of visual space. Subjectively there is the demonstration of ‘corresponding points’ on the two retinas, which when stimulated give an observer the sensation of identical location. Superposition of spatial signs from the two retinas in a single ‘cyclopean’ cortical map constitutes a dilemma when confronted with the undoubted ability to distinguish within them a third dimension. This was apparent by the middle of the nineteenth century, when the elite of visual physiologists, led by Panum and Hering, articulated and agonized over the problem of binocular single vision in the face of appreciation of depth. Helmholtz was aware of another dissonance: how, he argued, do we see singly with both eyes yet preserve information of the difference in eye of origin that is manifested in such phenomena as binocular luster and retinal rivalry?

Central to the issue is the concept of retinal disparity (Fig. 1), easily given quantitative expression from the geometry of the situation: the image separations on the two retinas differ when two objects are in different fore and aft locations. It is possible to measure disparity and to generate visual stimuli with defined values in defined locations. Ever since Wheatstone, psychophysical investigations of the relation between disparity and perceived depth have been one of the most productive topics in visual science. The ‘horopter’, the location in object space of target arrays that appear to an observer to be fronto-parallel (Ogle, 1950), occupied the attention of scientists for many generations; that difference in depth has finer discrimination thresholds even than visual acuity has been more than a curiosity since it was discovered in the 19th century (Stratton, 1898).

Figure 1.

Geometrical layout defining the disparity δ when two objects are at distances x and x + dx from R and L, the observer's two eyes, with interocular separation a.

Because information coming in from the two eyes is a prerequisite to stereopsis, and because the streams from the two eyes remain separate even in the lateral geniculate nucleus (LGN) and do not converge to common sites until the visual cortex, neural investigations had to await the availability of tools for probing the central nervous system. Evoked electrical responses were at the outset not sensitive enough for the purpose. The advent of single unit recording from the mammalian cortex in the 1950s finally provided a start.

Of the two interfaces of an organism with its environment, the sensory systems yielded to investigation more readily than the motor system. This should not be surprising. Sherrington and his school, by concentrating on the simplest motor routines in the isolated spinal cord, made many discoveries but in the end were unable to reveal much about how an intact animal's movements are organized in the highly-intertwined pathways in the mesencephalon and above. By contrast, sensory signals enter in discrete bundles, and are subject to relatively well-understood ‘pre-processing’– for vision in the retina and LGN – before fanning out into the richly interconnected central nervous system. In the intact organism, sensation can be investigated using definable and repeatable stimulation, leaving the researcher in full control of the experiments.

First experiments in the cat visual cortex

The mammalian preparation on which the first findings were reported was the anaesthetized cat. The significant concept of ‘trigger features’ which had emerged from previous work had shifted the emphasis in the selection of stimuli when recording from neural entities. Up to then physiologists had looked to physics, not biology or psychology, for the stimuli to use in their experiments. For the eye, the variables of interest had been light energy in ergs or later in quanta, or in dimensions measured in units of wavelength or square degrees. In seminal studies, Kuffler in the cat (Kuffler, 1953) and Barlow in the frog (Barlow, 1953) showed that the Sherringtonian concept of receptive fields, as introduced by Hartline (Hartline, 1938) into the world of vision, had in retinal ganglion cells the additional property of the centre/surround antagonism which could be interpreted as enhancing the contrast with which small objects and edges are represented in the signals sent by the retina to the sensorium.

The first research with single-unit recording in the mammalian cortex failed because that lesson had not yet sunk in (Jung & Baumgartner, 1955). When it had, cortical units immediately revealed their preference not for ‘[room] light on’ or ‘off’ (the latter not without betraying a certain sophistication) but for oriented edges and direction of motion (Hubel & Wiesel, 1959). That is, right at their entry port into the cortex the sensory signals had undergone a rearrangement: instead of representing the visual field point by point, they evidenced a sorting operation and the 50th anniversary of its first full publication is appropriately commemorated in this issue of The Journal of Physiology in which it first appeared. Looking into the neural circuits by which this is accomplished may have kept researchers busy to this day, but right away it cleared the road to postulate other attributes of visual features, beyond oriented borders and movement, for which similar sorting may take place. Binocular disparity was one that had been in the mind of Peter Bishop, a prominent Australian neuroscientist, for some time.

The implication of gathering information from individual points of the visual spatial manifold into more elaborate templates cannot be trivialized. It is true that retinal ganglion cells have receptive fields composed of centre and surround; nevertheless they retain individual and single local signs. But once the oriented receptive field of a cortical neuron has been assembled, the position signal has been compromised to some extent. Reconstitution – or at a minimum preservation of information – of visual space from an ensemble of boundary tokens constitutes a considerable challenge, especially when it is realized just how exquisite our discrimination of position and orientation can really be (Westheimer, 1996).

Accommodating neural compartments for several attributes all sharing the same visual space while retaining global retinotopy is achieved by intermixing. The gamut of orientations is represented piecewise, location by location. This seems to be seamless, the occasional crunch – called pinwheel – notwithstanding. Could the local sequestration of orientation representation contained within an overall progression of locations be prototypical? In particular, how about binocularity and its consequence, stereopsis? There had been conjectures on how the inputs from the two eyes are overlaid to produce, at least grossly, a single ‘cyclopean’ retinotopic map. At one time the stria of Gennari (responsible for the term ‘striate cortex’) was suggested to mark a division between the right and left eyes’ projection. Single-unit studies in the cat instead showed that in any cortical location the neurons generally do not respond equally to inputs from the two eyes. Anatomical tracing revealed ocular dominance columns and the stage was set for deprivation experiments (Wiesel & Hubel, 1963), the first example of a new generation of neuro-developmental studies, important pointers in the aetiology of the clinical conditions of amblyopia and strabismus that continue on to this day.

But preservation of eye of origin is not yet stereopsis, though a precondition for it. Still, it provided the impetus for the enquiry: if border-orientation is a criterion for sorting and reassembling the cortical input according to a parameter set that interrupts but does not disarrange the orderly progression of distance in visual space, could this also be the case for binocular disparity? Were the local signs in the streams entering from the two eyes everywhere in perfect register, the cortical representation would be two-dimensional like the retina's.

In psychophysics one can define ‘corresponding points’ on the retinas operationally. With an observer binocularly fixating a point in space, they are the locations which when alternately exposed are seen in the same visual direction with the two eyes. Target points in front or behind them in depth will be imaged with crossed or, respectively, uncrossed disparity. In studying cortical units in the anaesthetized animal, the procedure is reversed. A screen is placed in front of the head, an electrode records from a cortical unit and the receptive field is plotted on the screen through the right and left eyes. They will not in general coincide because in the anaesthetized or pharmacologically immobilized preparation the eyes usually diverge. The relevant question becomes whether the right/left location differences are the same for all units after allowing for a variety of factors such as stability during the course of recording from several cortical units and deviations due to tangential projections. If not, one can talk of receptive field disparities (sometimes referred to as incongruities) even if the value for exact correspondence or zero disparity remains uncertain.

This was the research programme envisaged by Peter Bishop and carried out both in Sydney and Berkeley through the participation in both laboratories of Jack Pettigrew. And indeed neurons in the striate cortex of the anaesthetized cats were found with a range of such receptive field differences. But as distinct from the guarded title of the Sydney publication ‘Analysis of retinal correspondence by receptive fields of binocular single units in cat striate cortex’ (Nikara et al. 1968), the title of the Berkeley publication (Barlow et al. 1967) proclaimed ‘The neural mechanism of binocular depth discrimination’. It reflected the different scientific personae of the two senior investigators: Peter Bishop, ever the cautious and painstaking experimentalist who resisted being carried away by flights of his imagination; Horace Barlow, an originator and tireless champion of the concept that neurons read meaning into the physical stimulus pattern that excites them. In his PhD dissertation Colin Blakemore went on to search for depth columns as companions to the orientation and ocular dominance columns embodied in the popular ‘ice-block’ cartoon of cortical structure. The idea that binocular cortical neurons with particular ranges of retinal disparities are clustered was subsequently revived in Pettigrew and Dreher's proposition that binocular cells in cytoarchitectonic area 17 respond preferentially to zero disparities whereas those in parastriate and prestriate cortex (cytoarchitectonic areas 18 and 19) code for crossed and uncrossed disparity, respectively (Pettigrew & Dreher, 1987).

It took a little while before consensus was reached, because Hubel and Wiesel, when they returned to the cat cortex, found these disparity-sensitive neurons to be more prominent in cat area 18 than in area 17 (Hubel & Wiesel, 1973). Questions of eye stability needed to be addressed and the knotty problem of vertical disparities was opened: functionally, binocular depth is coded only by disparities in the horizontal directions, whereas in the data by Nikara et al. (Fig. 2) and later by Ferster, horizontal and vertical scatter of interocular receptive field location differences is about equal (Ferster, 1981). It took years for the next generation of investigators to get to grips with this problem.

Figure 2.

Distribution of receptive field disparities of single cells in the striate cortex of the anaesthetized cat in the horizontal and vertical directions. Shaded data: raw values; filled data: after allowing for unsteadiness of eye position (after Nikara et al. 1968).

Primates

Throughout all the literature devoted to the presence of neurons in the cat visual cortex with receptive field disparity, their possible role in stereopsis was invoked. Though the disposition of the eyes and the neural projections in the cat allow the utilization of binocular disparity to appreciate depth (Timney, 1990), the rich repertoire of stereoscopic abilities had been plumbed only in the primate, which differs quite significantly in the confluence of the streams from the two eyes as they enter the cortex. In the monkey there are layers of strictly monocular and not yet orientation-specific cells. When Hubel and Wiesel looked for receptive field disparity in the primary visual cortex of anaesthetized the monkey they could not immediately convince themselves that they were present (Hubel & Wiesel, 1970). At the same time they were able to find cells sensitive to binocular depth in area V2.

Any residual doubts about disparity coding, however, were resolved a few years later by Gian Poggio, working in the difficult alert primate preparation. The situation is somewhat more nuanced than merely finding neurons whose binocular receptive field disparities were ‘tuned’ to a range of disparities (Poggio & Fischer, 1977). Instead, units in the monkey cortex fell into categories of those excited by far stimuli and inhibited by near ones, or vice versa, or those that had narrow excitatory or inhibitory depth zones flanked by zones with the opposite depth polarity (Fig. 3). Poggio, however, was ecumenical as to the anatomical provenance of his cells; many were not in V1 but in V2.

Figure 3.

Schematic depiction of the four kinds of disparity tuning of single units in the awake macaque primary visual cortex described by Poggio & Fischer (1977). Ordinates: response level of neurons. I, tuned for near; II, tuned for far; III, narrowly tuned inhibitory; IV, narrowly tuned excitatory.

It may seem straightforward to study disparity tuning in a manner equivalent to orientation tuning: instead of changing the orientation of the stimulus contour, one would scroll through the range of disparities using a Risley prism or the equivalent, and arrive at an optimum disparity, i.e. one with the maximum response, and perhaps also at a tuning width. However, there are complications associated with the fact that the receptive fields of the involved cortical neurons have spatially contiguous excitatory and inhibitory subregions. Moreover, because the receptive fields are all elongated and the right/left ‘incongruities’ were looked for in the direction normal to the elongation, what should one do with neurons whose receptive fields are oriented horizontally? Further in this connection, would possible interocular orientation differences (Blakemore et al. 1972; Bridge & Cumming, 2001) suit the detection of the depth-tilt of stimuli such as a rod in the median plane slanting towards or away from the observer?

Receptive field properties

The diverse possibilities of interocular differences in the receptive field properties of cortical units pose a considerable challenge. Spatial dissection of receptive fields into adjoining regions through which the unit responds to bright and dark bands on a grey background is nowadays accomplished by the back-correlation procedure. It differs from the traditional stimulus → response methodology, wherein the experimenter shapes the stimulus while observing responses, and homing in on what appears to be the optimum stimulus by trial and error. Instead, in a long train of thousands of sequential exposures of bright and dark lines randomly displayed in all likely locations, each of the unit's impulses is correlated with the stimulus components that preceded it and that are assumed to help trigger it. Thus a picture of the receptive field's structure is synthesized statistically off-line as a set of adjoining bright and dark bands. It has become customary to fit it into the mould of a Gabor patch, composed of a sinusoidal grating with specified orientation, spatial frequency, phase and amplitude, modulated by a Gaussian envelope, specified by centre location and the standard deviations in two directions which do not necessarily coincide with the grating's orientation (Ohzawa et al. 1996). One talks of position disparity when the receptive field is exactly the same in the two eyes with merely a position shift (relative to a predefined zero disparity in which they exactly overlap), and of phase disparity when there is a difference between the two eyes in the relative disposition of the dark and bright bands. Both types are found in macaque V1 (Tsao et al. 2003). Broadly concordant results are said to be obtained with the direct and back-correlation methods of plotting the receptive fields, but the differences may matter when conclusions are drawn from models elaborated from them.

One of these is the so-called ‘disparity-energy’ model (Ohzawa, 1998). Its basis are a set of four types of receptive field: because both sine and cosine terms are needed for a Fourier transformation to be complete, a pair is required for each eye, one with odd and the other with even symmetry (Fig. 4). Where disparity is involved, there will be interocular differences between the members of each pair. In the theory, the visual cortex is tiled with a set of these four kinds of receptive fields, linked by summing and rectifying operations. The disparity ‘energy’ of a visual stimulus with arbitrary spatial intensity distribution then is the signal in the domain of disparity generated by the response it engenders in the families of the four types of receptive field.

Figure 4.

Top: geometrical sketch showing how vertical disparity occurs when a target AA′ is located near an observer and to one side of the midline CC′ and hence closer to the right eye R than the left eye L. Bottom: the observer's view: *, straight forward; AA′_R, the right eye's view; AA′_L, the left eye's view.

The aim of the exercise is to model neural circuits whose output will match the kind of disparity-tuning profiles first described by Poggio using bar stimuli. One goal in particular, in view of the simple cell/complex cell dichotomy originally proposed by Hubel and Wiesel, is to develop propositions about the channelling of signals through sub-units. However, once a Gabor-function-fitted mathematical model of receptive fields is favoured, it becomes tempting to follow it up with visual stimuli such as sinusoidal gratings or Gabor patches. Because such stimuli are multi-lobed and at any given disparity will cover different excitatory and inhibitory receptive field zones in the two eyes, quite complex response functions may result, allowing room for elaborate computational schemes (Cumming & DeAngelis, 2001).

The impact of Hubel and Wiesel's discovery of orientation selectivity of neurons in the cat striate cortex, namely the establishment of the oriented line, bar or grating as the canonical probing stimulus, nevertheless was carried forward into the investigations of the disparity attribute with its overtones of representation of the third stimulus dimension of the visual world, from its beginning right through the ascendency of the analytical phases just described. These are, however, still based on line orientation. Yet stereopsis is present for the small non-oriented target, a direction into which the subject was given a nudge, almost contemporaneously with Hubel and Wiesel's first paper, by the random-dot stereogram proposition of Bela Julesz (Julesz, 1960). One can see depth in fused uniocular panels without overt boundaries. In an extraordinary feat of neural processing, often called ‘solving the correspondence problem’, depth is disambiguated from the multiple disparities of many small tokens. That the visual cortex is involved in more than a purely receiving capacity had been demonstrated by the changes that occur in the evoked electro-encephalographic readings to random dots when they carry disparity compared to when they do not (Regan & Spekreijse, 1970). It was, therefore, of particular interest when it was found that visual cortical neurons in the alert monkey, already in V1, evidenced horizontal disparity tuning for random-dot stereograms (Poggio et al. 1985), regardless of their receptive field orientation as tested with bars. But they also have disparity tuning when in the stereograms the dots for the two eyes are of opposite contrast polarity, though the human observer then sees no depth (Cumming & Parker, 1997).

Vertical disparities

A re-examination was also required of the many findings that disparities of striate cortical cell receptive fields occur in all possible directions, not only the horizontal usually associated with seeing depth with two eyes laterally displaced from the head's mid-sagittal plane. There had been an early interest in vertical disparity when it was first realized that some astigmatic spectacle corrections that increased the vertical size of only one eye's retinal image caused the patient to experience illusory spatial distortions. Vertical disparity can be detected by the human observer (Ogle, 1950): it usually manifests itself in space perception as a compensating horizontal disparity, the ‘induced size effect’, but it could also lead to a misperception of the subjective straight forward (Berends et al. 2002; but see also Banks et al. 2002). Hence the vertical component of cortical cell receptive field disparity could act as the neural substrate for the detection of vertical disparity. It has been cogently proposed that this phenomenon, geometrically consonant with a height difference of close-up vertical targets when they are to one side and hence nearer to one eye than the other (Fig. 5), is of utility in sensing the degree to which the two eyes need to converge on such a target (Mayhew & Longuet-Higgins, 1982). Vertical receptive-field disparity, therefore, would not be an irrelevancy and its channelling into the horizontal disparity processing apparatus apropos, though some psychophysical evidence demands that it be accomplished in differentiated ways in the four quadrants of the visual field (Westheimer & Pettet, 1992). Further, in their global effects the two classes of disparity differ markedly (Stenten et al. 1984).

Figure 5.

Four components used in ‘disparity energy’ model for position disparity, here shown for the uncrossed (far) situation (after Ohzawa et al. 1996). Top, right and left eye cosine (even-symmetrical) receptive fields, displaced in opposite directions with respect to zero disparity. Bottom: similarly positioned pair of sine (odd-symmetrical) fields. The disparity energy is calculated by summing the squares of the two classes in each eye and subjecting this value to a differencing operation. For ‘phase-disparity’ cells, the two pairs of components are all centred on the zero disparity lines, but the internal distribution of the excitatory and inhibitory regions is different in the two eyes.

Binocular fusion

From Wheatstone's day there has been soul-searching about the apparent discord between (a) the identity of apparent visual direction when pairs of specific retinal points in the two eyes are stimulated, i.e. binocularly corresponding points, (b) the need for stimulation of non-corresponding points for purposes of stereopsis, and (c) the lack of diplopia for targets seen with small degrees of disparity. The word used is fusion: the melded impression of a target imaged on corresponding points of the two eyes is described as fusion, but it is more than just singleness, because dissimilar targets shown to corresponding points are seen as occupying the same visual direction yet are not seen as combined. Targets much closer or much farther than the fixation plane and hence imaged on non-corresponding points are seen in diplopia; but targets with small disparity that lead unquestionably to the experience of depth are seen as single. The disparity range over which this holds is called Panum's fusional area and in the fovea it has an extent of 10 minutes of arc or more and, moreover, is subject to training and other modifying factors (Mitchell, 1966).

To resolve the dilemma it helps to recall that targets shown with substantial disparity, very evidently seen double, also have depth associated with them (Westheimer & Tanzman, 1956) and, when newly introduced in the visual field, will entrain vergence eye movement in the correct direction (Westheimer & Mitchell, 1969). Moreover, some patients with permanent constant strabismus since birth see double when a target is presented simultaneously to their two foveas, a syndrome called abnormal or anomalous correspondence (Jennings, 1985). Hence it seems best to decouple the experience of singleness (rather than diplopia) from disparity and from the experience of depth. Fusion nevertheless represents a distinct sensory quality that might very well have a neural foundation in the ‘obligatory binocular cells’ whose firing requires simultaneous input from both eyes, unlike most of the cortical cells described so far which show some kind of response on uniocular stimulation. Reports of obligatory binocular cells come from studies of V2 (Hubel & Wiesel, 1970; Ts’o et al. 2001).

A reconciliation is also needed of neurophysiological findings of ocular dominance of cortical cells and the perceptual reports on the seen brightness (Fechner's paradox) and colour (binocular yellow) of overlaid panels with binocular differences in these attributes.

Neural-psychophysical parallels: receptive field disparities and stereopsis

That the disparity cues about the three-dimensional object world are teased out of the signals sent by the retina to the cortex is, of course, a truism, since they are utilized in perception. But 50 years ago, when Hubel and Wiesel first successfully demonstrated response specificity for border orientation and movement direction in neurons in the mammalian primary visual cortex, it was not so obvious just how far along the processing stream one would have to go before neurons are found that respond preferentially to stimuli with different disparities. The conviction, on the basis of biological intuition, that it might be quite early on was borne out by the experiments of the groups in Sydney and Berkeley. After some initial hesitation, the demonstration of disparity tuning in some units in the striate cortex of the alert monkey consolidated the topic.

There followed a very active ‘inward-looking’ research phase with the interest centred on the nature of the receptive field differences in the two eyes, using models in which each eye's field structure was fitted by mathematical functions and expressing in quantitative form the receptive field disparity and the response to disparate stimuli in terms of linear systems theory. A similar approach in the study of movement selectivity of cortical cells (Adelson & Bergen, 1985) led to insightful experiments in which intracellular recording clarified the underlying temporal–spatial confluence of synaptic input (Priebe & Ferster, 2005). It may be expected, therefore, that in a similar manner we will gain a deeper insight about the convergence of impulses from the two eyes on disparity-selective cells.

Once disparity selectivity had been demonstrated in the striate cortex, it is of course not surprising to find it abundant also in higher visual processing areas. Disparities are prominent in area V2 (Ts’o et al. 2001) and in V3 (Adams & Zeki, 2001) and have been reported widely in higher visual areas, in particular the middle temporal lobe (MT) (DeAngelis et al. 1998). As has been the case with motion as a stimulus attribute (Albright, 1984), the question of whether higher cortical areas ‘inherit’ the disparity signal from V1 or whether it re-emerges de novo is knotty (Cumming & DeAngelis, 2001). Though it has not yet shown up, a cortical area or circuitry specializing in the third visual dimension, encompassing disparity along with perspective and some of the monocular cues for three-dimensional object disposition, would be of major interest. Not only are these cues used to complement each other by an observer, but some such arrangement is suggested by evidence of cross-talk between them (Gogel, 1954).

The gap separating even the fullest description of neural signals in the striate cortex from the rich repertoire of sensory capabilities of an observer in stereoscopic tasks remains vast. Barlow, Blakemore and Pettigrew's title ‘The neural mechanism of binocular depth discrimination’ 40 years later (Barlow et al. 1967) still constitutes more of a challenge than a finding. Many psychophysical and perceptual experiments in the realm of stereopsis tell us about the rules which these circuits, when they are traced, must be found to obey.

The original unease about whether disparity is first manifested in V1 or in V2 neurons is emblematic also of the question of the extent to which an observer's performance in a stereoscopic task might be expected to be reflected in the response properties of these neurons. Because this is an ultimate objective in this kind of investigation, and because experiments in non-human primate preparations as well as in the human by non-invasive neural probing are very demanding, the researcher, in planning experiments and posing the questions they are to answer, needs to be aware of the salient features that characterize the capability in the realm of stereopsis. While at this stage it is not at all clear that V1 neurons match the latter even minimally, the final word in this matter has to await detailed knowledge of the effect of top-down influences, which may readily endow such neurons with response properties that depend on the prevailing state of other segments of the central nervous system (with respect to, for example, attention, expectation, context and previous experience) and that may make them more attuned to the immediate stimulus conditions than when in a neutral setting or under anaesthesia.

What follows is a brief enumeration of just a few points that ought to be kept in mind, based predominantly on some of the reviewer's experiments in this field spanning the same half century that has elapsed since Hubel and Wiesel's paper that is here being celebrated.

That on their first presentation the depth in random-dot stereograms takes a while to emerge, whereas on subsequent viewing it is seen instantly, is merely one example of the need for any proposed circuit to include modifiable elements. There are many reports of perceptual learning (Fendick & Westheimer, 1983) and of strange temporal effects (e.g. Gillam et al. 1988) in stereopsis.

An observer's ability to judge the depth location of a free-standing object without head or eye movements may perhaps match the disparity tiling in the cortical representation (Westheimer, 1994), but the finest stereo thresholds are more than an order of magnitude better than that, in the hyperacuity range of a few seconds of arc, and obviously the result of refinement of these signals by clever circuitry. They demand sharp, discretely articulated features (Westheimer & McKee, 1980); performance is poorer with the Gabor patches that are so widely advocated as a basic visual stimulus and implemented in the ‘disparity-energy’ model (Westheimer, 1998). They emphasize that disparity differences rather than raw disparities are employed most prominently in judging the three-dimensional placement of an object. The distinction between relative and absolute disparity has only just begun to be tackled in cortical processing (Roe et al. 2007).

As demonstrated with random-dot stereograms, the association of feature fragments in the two eyes into a set of binocular tokens for the purpose of assembling a coherent global structure is a difficult task involving the selection among many possible pairing states (Marr & Poggio, 1979). This process needs to be considered in connection with a variety of other interaction phenomena including pooling and repulsion operations in the domain of disparity (Westheimer, 1986b) and the pairing of unmatched features (Westheimer, 1986a; Nakayama & Shimojo, 1990) when in a natural environment a surface partially occludes one eye's view of an object. A beginning has already been made in seeking cortical cell correlates to surface segmentation, though it appears that they occur only in V2 and not yet in V1 (Bakin et al. 2000). Eventually the neural circuits advanced as subserving depth processing will also have to accommodate interaction within and input from other cortical regions as demanded by this wide variety of perceptual phenomena.

The view forward

The original interest and still the main aim of the enterprise is to define how much of the whole organism's ability to make discriminations about depth in its visual world takes place through processing in the first neural location at which the signals from two eyes come together. To provide an adequate model of the neural substrate of the processing revealed in human observers, findings should preferably come from the difficult alert primate preparation, a scarce resource. Single-cell recording has been the procedure of choice because it focuses on the elementary units composing the sensorium, but from the outset, it has been understood that neural ensembles – made up to be sure from individual elements – are involved. They call for more distributed modes of analysis which will no doubt encompass new generations of non-invasive imaging techniques with sequentially better temporal and spatial resolution.

Finally, it has been clear for a long time that cortical plasticity, evidenced by effects of training and previous experience, plays a more important role in the processing of disparity than of, say, brightness or colour. As neuro-developmental, indeed even genetic, modes of study are applied, fertile ground for tracing developmentally aberrant pathways may be found in such clinical conditions as anomalous correspondence (Lyle & Jackson, 1949) in which widely distant retinal regions in the two eyes seem to have become homologous over the course of years of associated common exposure.