How The visual system turns things the right way up

The output of the visual brain usually shows no hint of doubt about what you see, which is why those rare situations of ambiguity are troubling—is it a face or a vase? Are the colors of the dress white and gold or blue and black? The retinal projection of any object, such as a fork, will vary dramatically as your physical relationship to it changes, either because it moves or you do. Like two snowflakes, precisely the same retinal image might never be encountered twice. And yet, we effortlessly recognize the fork. The disjunction between the input to the visual machine—the retinal images—and its output—the perception of a fork—underscores a key neural operation: the brain’s alchemy that transforms signals represented by retinal coordinates into representations defined by behaviorally relevant coordinates (Conway, 2018). This is, of course, adaptive: we wouldn’t want perception to be a simple linear read-out of retinal images any more than we would want the conception of snow to be defined by one configuration of ice crystals. There is nothing veridical about retinal images—they are upside down and backwards. The paradox of vision is that it depends on ephemeral and flipped retinal images yet gives rise to stable percepts that are the right way up. Davida, the young woman studied by Vannuscorps, Galaburda, and Caramazza, provides fascinating clues to how this paradox unfolds.

Davida experiences ambiguity about the orientation of drawings of shapes, stripped of their three-dimensional context, depicted on a flat surface in high contrast. When shown a drawing of an arrow, a letter, or an object, Davida sees it as a dynamic hybrid of rotated versions, either flipped vertically or mirror reversed. But if the image is blurred or the contrast is greatly reduced, the object depicted in the image has a stable orientation. She has no difficulty seeing objects in the real world.

Classic work in visual cognition uncovered the dorsal and ventral streams, which interpret signals from retinotopic cortex toward behaviorally relevant operations of “where” and “what”, or “action” and “object”. Meanwhile, classic visual neurophysiology revealed two main subcortical channels carried by magnocellular and parvocellular cells of the lateral geniculate nucleus, which relay retinal information to visual cortex. It has been tempting to attribute perceptual experiences (and the operation of neurons in the dorsal and ventral streams) to specific neurophysiological properties of the lateral geniculate nucleus, leading to the notion that magno signals feed most dorsal-stream operations, and parvo signals supply most ventral-stream operations. The integrated framework favored by Vannuscorps to explain Davida’s experiences succumbs to this temptation, for, as they argue, Davida’s anomalous perception is restricted to images of high contrast, which the authors would like to ascribe to the high-contrast sensitivity of parvocellular neurons. We present an alternative account.

What the retina tells the brain, and the locus of Davida’s deficit

Magnocellular and parvocellular neurons innervate V1 in parallel. But these inputs are very soon resampled. Most, if not all, individual V1 cells receive inputs from both magnocellular-derived and parvocellular-derived subcortical channels (Sawatari & Callaway, 1996). V1 does not keep the subcortical signals separate as it does its work to generate parallel outputs targeting different extrastriate streams; moreover, despite dogma and contrary to the arguments put forth by Vannuscorps et al., spatial frequency by itself is insufficient to isolate magnocellular and parvocellular channels (Blakemore & Vital-Durand, 1986; Skottun, 2015). The framework adopted by Vannuscorps et al. traces subcortical signals through V1 as one might trace migrating birds from the Amazon to Maine. A better analogy for how V1 works is as a baker, who combines different ingredients to make different baked goods: the ingredients are readily distinguished, just like the subcortical channels, and the baked goods are readily distinguished, just like the extrastriate cortical streams, but there is no trivial correspondence between the different ingredients and the different baked goods. Evaluating the output of V1 in terms of subcortical channels will be about as useful as evaluating baked goods in terms of their ingredients: it will miss the point that the subcortical channel inputs undergo irreversible operations that transform retinal signals into new representations. Indeed, careful lesion studies of subcortical channels in monkeys show that extrastriate areas receive equally strong inputs from magnocellular and parvocellular layers (Ferrera et al., 1994). And functional imaging studies in humans shows that areas in both dorsal and ventral streams are selective for objects defined by both high and low spatial frequency (Vaziri-Pashkam et al., 2019), so spatial frequency is not only incapable of distinguishing subcortical channels but also insufficient to isolate the extrastriate streams. Davida’s case may point to a selective deficit within some component of the visual-processing apparatus, but it does not provide support for the notion that subcortical channels remain segregated through V1 toward the extrastriate streams.

So, what causes Davida’s deficit? Do the data point to an anomaly in the subcortical inputs to V1, in V1, in extrastriate circuits, or perhaps in other systems such as the superior colliculus, cerebellum, and basal ganglia that direct attention? Or alternatively, is the deficit not within one stage, but rather in how the stages interact to turn retinal signals the right way up? Davida can see stimuli, and she can recognize objects, suggesting that her subcortical channels are working and her high-level visual-processing regions are fairly normal. This localizes the deficit to the machinery that turns the input the right way around.

Could the deficit derive from a problem in retinotopic cortex, perhaps within V1? This possibility is suggested by the fact that Davida has abnormalities in binocular fusion. A set of red, green, and achromatic spots were viewed with a red filter over one eye and a green filter over the other eye. People with normal binocular fusion will see all the spots. Davida saw the red and green dots flashing rapidly, asynchronously, which implies an anomaly in how signals from the two eyes are integrated. The first stage in the visual-processing pathway at which signals from the two eyes are combined is V1, which implicates V1 in Davida’s deficit. But the object-based nature of her experience shows that V1 is not entirely to blame.

Horizontal connections in retinotopic cortex connect cells with similar orientation preferences, providing a substrate that is likely important for the representation of object contours. This machinery seems normal in Davida, since she can accurately describe the relationship of joined marks in a scene. Where Davida encounters problems is in interpreting the relationship of the bounded elements to a meaningful reference frame. Normal daily life may provide sufficient redundant cues to the reference frame (other visual elements, the ground and sky, the vestibular system), allowing Davida to compensate. To identify the putative extrastriate locus of Davida’s deficit, the authors rely on the observation that Davida’s perception is only peculiar when viewing drawings of high contrast. We concur that this is a key observation, although we are not sure that it necessarily indicates a deficit in processing high spatial frequencies since pure high spatial frequency stimuli were not tested (high-contrast drawings contain a mix of spatial frequency components). And in any event, differences in perception of images depicted in different spatial frequency bands would neither isolate magno and parvo contributions nor extrastriate visual streams, as discussed above.

What seems puzzling is that Davida sees blurred images unambiguously. We wonder if Davida may have no difficulty seeing the orientation of stimuli that are blurred or of low contrast because these image manipulations provide cues used by the brain to compute scene structure. This insight is suggested by a strategy that artists use to more clearly see the spatial organization of a scene: they squint, which blurs the image. High-contrast information that usually dominates attention is reduced and largescale luminance variations are enhanced. These “value relationships,” as artists call them, define the scene’s reference frame and are presumably conveyed predominantly by peripheral vision. The artists’ trick suggests that even in normal observers, high-contrast lines can mask cues needed to interpret scene structure. These considerations raise a different interpretation to the one provided by Vannuscorps et al.: that Davida’s deficit does not arise because the stimuli are high contrast and high spatial frequency, but because they lack cues needed to assign an appropriate spatial reference frame. Without these cues, the visual system struggles to assign the correct up–down and left-right flip of the retinal image, and perceptual read-out juggles among the restricted set of options. The perceptual instability that Davida reports is analogous to the instability of multi-stable displays, such as the 3-D configuration of the Necker cube or the colors in #thedress (Lafer-Sousa et al., 2015), which suggests that perception of scene content and scene structure depend on different neural operations that dynamically interact.

Multiple pools of neurons competing for a coherent percept

Moving around a scene, stationary objects don’t seem to change color, size, position, or orientation. Instead, when we move relative to fixed objects, we interpret the changes in the projected retinal images to the shifts in our own perspective. This shows that what we see must depend on more than the retinal images. One idea is that the process of seeing can be understood probabilistically, where each retinal image is mapped onto a restricted set of possible interpretations that are sifted through given the context of other cues. One intriguing aspect of Davida’s reports is that her perceptual difficulty manifests only with two-dimensional stimuli, where other cues are limited, and the difficulty involves switching between a restricted set of possible interpretations. The phenomenology resembles binocular rivalry and multi-stable percepts, such as #thedress, which are only evident under impoverished or restricted viewing conditions and similarly involve a restricted set of perceptual interpretations.

Consider for a moment the strict feedforward view of visual processing. Wavelengths from an object are absorbed by photoreceptors in the retina. Retinal ganglion cells process photoreceptor responses to encode low-level features, including luminance polarity, contrast, and spatial frequency. At subsequent stages, neurons compute a representation of simple shapes. And at the highest levels, neurons show tolerance to changes in low-level features to represent object categories, colors, faces, and places.

It is difficult to reconcile Davida’s experience with this strict feedforward view. How could damage to any stage lead to a multi-stable perception? Instead, the data from Davida require a dynamic interplay across stages in the putative visual-processing hierarchy that is engaged as her visual system attempts to sort through the plausible interpretations of the retinal image. There are bi-directional connections within and between almost all layers and areas of the hierarchy, providing the required hardware for this interplay across stages. Davida’s perception of the arrow, seeing it merge between alternative possible configurations, suggests her deficit lies in connections between pools of neurons that assign the correct scene reference frame. Her experience reinforces the idea that perception needs to be understood in terms of how the parts of the system work together, just like a baked good can only be understood as the interaction of the ingredients and how they are cooked.

There is one other striking feature of Davida’s experience: she is more likely to report the wrong orientation of a picture of anarrow, rather than simply a random draw of the possible orientations generated by flipping or mirror-reversing the retinal image (how she can report her ambiguous percepts through drawing raises a set of separate questions that we do not have space to discuss). This suggests that excitation of a set of neurons representing a particular arrow direction in Davida leads to abnormally strong inhibition by neurons representing the wrong interpretation of the scene structure on neurons representing the correct interpretation. These inhibitory interconnections seem to fluctuate, as if the circuit is waiting for other cues to resolve the scene orientation, leading to illusory instability that is not unlike the jitter of equiluminant displays. That the ambiguity arises when using high-contrast stimuli suggests that contrast or the presence of multiple spatial frequencies in the stimulus drives an imbalance of inhibitory and excitatory connections, potentially involving an imbalance in how signals from central and peripheral vision are integrated. Davida’s case suggests that vision in normal observers must work through a competition between alternative interpretations and highlights the importance of low spatial frequency signals for resolving which way is up.

In sum

Vannuscorps et al. present the fascinating neurodevelopmental case that offers insight into how visual signals are transformed from a retinal coordinate frame into a behavioral reference frame. As one explanation for the case, the authors invoke a hypothesis now known to be inaccurate that links channels in the lateral geniculate nucleus with operations in extrastriate areas. The function of any extrastriate area, and probably all perceptual experiences, cannot be understood in terms of the isolated activity of subcortical channels. Instead, we think the case underscores the importance of interactions between stages in the putative visual-processing hierarchy, and it raises important questions about the nature of the nonlinear operations that give rise to specialized response properties in extrastriate areas and how the interactions among areas give rise to perception.