Davida reorients intermediate visual processing

Vision is subserved by a hierarchical system, where increasingly complex representations are formed at each processing stage (Van Essen et al., 1992). This collection of interconnected brain regions is understood to work in concert to build a meaningful representation of our surroundings. Beginning with point-wise light detection by retinal ganglion cells, representations are pieced together in successive stages, forming scene segmentation and object representations. The earliest processing stages, involving the retina, lateral geniculate nucleus and early visual cortices, are thought to act as filters selective for various stimulus dimensions, giving rise to features such as orientation in areas LGN and V1 (Cheong et al., 2013; De Valois et al., 1982; Hubel & Wiesel, 1968). These early visual areas have been heavily studied and are understood to encode information about our surroundings in a retinotopic coordinate system, which preserves information in retinal coordinates centered on the focus of gaze. However, at some stage in the process, our visual system clearly must transition away from purely retinotopic frames of reference, to those which are of use in actually interacting with the objects around us – spatiotopic and body-centered reference frames. That is, world coordinates, which do not necessarily match retinotopic ones. Higher visual areas, whose cells’ receptive fields can be so large as to cover the entire visual field, likely operate in such reference frames (Melcher & Morrone, 2015).

How is the transition from retinotopic to world coordinates achieved? Perhaps we can draw from what we know about the formation of representations within the early visual system of neurotypical humans. Even within V1, we begin to see departures from the one-to-one correspondence of a strict retinotopic coordinate system. Indeed, one of the hallmark transformations within V1 is the transition from simple cells to complex cells, which goes from spatial phase-variant to phase-invariant selective preferences. This primitive conversion of spatial coordinate systems involves gains and losses in information. In V1, the gains are clear: complex, phase-invariant representations endow these populations with the ability to encode motion direction and stereoscopic depth – both essential building blocks necessary in order to build an understanding of our environment. But, there is a loss in information with the construction of these representations, as well. In gaining the ability to process motion and depth, complex cells lose the ability to identify the precise spatial phase of visual stimulation. And yet, we are generally still able to consciously access that information when asked to identify the precise spatial phase of a visual stimulus, presumably by relying on information that resides within simple cells, or even the LGN. This principle of gains and losses likely repeats itself throughout the visual hierarchy: as abstract, higher-level representations are gained in extrastriate cortices, specific, lower-level information is lost. In terms of spatial representations, the losses are well-known along the visuocortical hierarchy, with retinotopically-defined receptive fields increasing substantially in size along the hierarchy (Dumoulin & Wandell, 2008; Yoshor et al., 2007). These growing receptive fields effectively act as increasingly aggressive lowpass spatial frequency filters, which end up limiting knowledge of the retinotopic specificity of an item’s location.

While the loss in retinotopic spatial coordinates along the hierarchy is clear, due to larger receptive fields, we lack a very good understanding of what information is gained in representation in these areas, particularly in intermediate stages of visual processing. One suggestion is that a transitional stage of processing exists where the visual scene is divided into mutually exclusive, bounded 2-D shapes based on segmentation processes guided by principles of “uniform connectedness” (Palmer & Rock, 1994), and that at this stage shapes may be represented in shape-based coordinates, as opposed to retinotopic or spatiotopic (Olson, 2003). The intermediate stages of processing and their hypothesized object-based reference frames have proven more difficult to study than either early or high-level visual processes. Most of the evidence comes from neuropsychological studies of patients with specific deficits in object representation – for example, one patient was found whose visual neglect consisted of ignoring one side of individual objects in scenes, as opposed to one side of space (Driver & Halligan, 1991). There is also evidence that these shape representations are formed in multiple parallel pathways, each processing different types of visual cues, such as motion or color; patients have been reported who had difficulty perceiving shapes from some cues but not others (Cowey & Vaina, 2000; Rizzo et al., 1995).

Vannuscorps and colleagues (2021) report a patient, Davida, whose disorder is consistent with these ideas and offers additional information about some of the properties of shape-centered representations in intermediate vision. Davida’s disorder appears to be highly selective for the perception of 2-D shapes whose characteristics strongly suggest involvement of the parvocellular pathway (sharp edges, medium-to-high contrast) and a processing stage in which shape-centered representations have been formed and need to be mapped onto body-centered or spatiotopic frames of refence. For instance, she can correctly judge the tilt magnitude of a line on a computer screen and match it to a neighboring line in a staircase procedure, but when asked to grasp the outer points of a line drawn on a board, most of the time she responds as if the line were rotated by 90 degrees. In general, her visual deficit is characterized by perceiving sharp-edged 2-D objects (shapes) as if they were inverted, mirrored across their axes or reversed by 90, 180 or 270 degrees around their center, as the authors demonstrate in numerous tasks.

Interestingly, Davida reports that her perception of different shape orientations is often unstable, alternating gradually between shape-centered-axis representations, akin to the perceptual oscillations that define binocular rivalry (Vannuscorps et al., 2021). This perceived fading in and out of the various orientations, combined with Davida’s inability to correctly interact with these shapes, is intriguing, as it suggests that the precise information about the orientation of the shape is inaccessible to whatever process is responsible for incorporating shape information into higher-complexity reference frames. At the same time, this information has clearly been computed by and is present in earlier visual areas, since Davida matches controls on some tasks which require correct orientation perception and likely rely on retinotopic representations (e.g. tilt magnitude discrimination). This suggests that the visual areas where shape-centered representations are mapped onto world coordinates do not themselves represent information about the orientation computed in the preceding stages. Instead, Davida’s case suggests that when shape information is being pieced together, it is in pursuit of orientation invariance in its own axis coordinates. This is reminiscent of what occurs in V1, with the transition from phase-variant (simple cells) to phase-invariant (complex cells) preferences, losing the retinotopic information obtained in the previous stage. In intermediate stages, while gaining object-centered representations and the ability to interact with objects in the real world by virtue of mapping the axes of the shape representation onto higher coordinate frames, evidently some areas lose the ability to construct the specific orientation of that exemplar shape. And while neurotypical individuals can access the preceding stages to know specifically what the orientation of the exemplar is, in Davida’s case, these areas appear to have lost the ability to explicitly access this information, making the formation of stable 2-D shape representations impossible.

The high specificity of Davida’s visual disorder helps shed light on some of the processing that occurs in intermediate stages of object perception and the properties of intermediate shape-centered representations (ISCRs), while presenting a range of new questions. Specifically, what are the neural areas representing ISCRs and mapping them onto higher order frames? And why, in this case, do they lack access to retinotopic orientation information? Previous research suggests that human visual areas LO1-LO2 (V4d in monkey) are well situated within the visual processing stream and could be suitable candidates for computing ISCRs, since they can encode information about shapes in both retinotopic and object-centered reference frames (El-Shamayleh & Pasupathy, 2016; Nandy et al., 2013; Pasupathy & Connor, 2001), and maintain segregation between groups of neurons which respond to shapes from different cues (Tanigawa et al., 2010; Tootell & Nasr, 2017). Vannuscorps and colleagues further suggest that the parietal dorsal stream is critical in providing information about shape axis correspondence and object location (Friedman-Hill et al., 1995; Harris et al., 2001; Priftis et al., 2003), both essential for mapping shape representations. However, the neural underpinnings of Davida’s unique perceptual experience have not been directly tested. There are a variety of functional neuroimaging studies that could be deployed with Davida to help identify the neural loci of ISCR processing. One could, for instance, relate the perceptual switches Davida experiences when viewing 2-D shapes with brain activity, to test whether the switches in subjective percept are correlated with neural oscillations in specific regions along the visual stream (Lumer et al., 1998; Polonsky et al., 2000; Wunderlich et al., 2005). Experiments such as these this may help localize the deficit within the visual processing stream, more directly vetting the hypothesis laid forth by Vannuscorps and colleagues that lateral occipital and dorsal stream regions play a putative role in intermediate shape processing.