The where, what, and how of object recognition

Global shape is a representation that describes an object’s form via the spatial arrangement of its features without describing the appearance of the features themselves. In our Opinion paper [1], we presented evidence that the ventral visual pathway does not represent global shape information but is instead sensitive to the appearance of local features. On the basis of recent findings [2], we further argued that, because of its involvement in visuospatial processing more generally, the dorsal visual pathway is ideally suited to compute the relations among object features and form global shape percepts.

A key prediction of this hypothesis is that patients with damage to the dorsal pathway should exhibit deficits in perceiving global shape and patients with ventral damage should exhibit spared shape perception abilities. However, Goodale and Milner [3] rightly point out that, in the majority of cases, damage to the dorsal pathway causes deficits in visually guided action, whereas damage to the ventral pathway causes deficits in object recognition.

As we discussed in our Opinion [1], the evidence from patients is more nuanced than typically asserted. For instance, there are many cases where patients with extensive bilateral temporal lobe damage and object agnosia can nevertheless discriminate and match objects on the basis of their shapes [4,5]. Moreover, there are studies that demonstrate a double dissociation wherein patients with circumscribed dorsal damage exhibit deficits in perceiving the global, but not local, aspects of objects, whereas patients with circumscribed ventral damage exhibit the reverse pattern of deficit [6]. Studies using causal methods with healthy participants (i.e., transcranial magnetic stimulation) also show that disruption to the dorsal pathway impairs the perception of global form [7] and impacts performance on tasks commonly linked to the ventral pathway, such as configural face perception [8].

Yet, as Goodale and Milner [3] point out, there are also many cases where dorsal damage seemingly does not impair object perception. As an example, they highlight a study with R.V., a patient with bilateral dorsal damage, and, for comparison, D.F., a patient with bilateral ventral damage. When presented with simple geometric shapes (see examples in Figure 1 and [3]), R.V. exhibited a deficit in grasping objects (consistent with the ‘vision for action’ characterization of the dorsal pathway), but not discriminating them, whereas D.F. exhibited a deficit in discriminating the objects, but not grasping them (consistent with the ‘vision for recognition’ characterization of the ventral pathway) [9]. How can we reconcile these results and the many other such examples within our proposed framework?

Figure 1. Using computational models to predict patient performance.

(A) A schematic illustrating how visual similarity is computed for a deep neural network (DNN). (B) A correlation matrix showing the perceived similarity between each object shown to patients [3] as computed by CORnet-Z, a feedforward DNN pretrained to categorize objects from the ImageNet database. (A) First, each image was inputted into CORnet-Z and the responses (i.e., activity patterns) from the model’s final layer (‘AvgPool’) were extracted. These activity patterns correspond to the highest-level representation of the object within the model prior to the output. Similarity between each object was then computed by correlating (Pearson) the activity patterns for each object with one another in a pairwise fashion, creating (B) a symmetric correlation matrix. Cells of the matrix with a higher correlation value are object pairs the DNN perceived as more similar, and cells with a lower correlation value are object pairs the DNN perceived as more different. Cells along the diagonal have a correlation value of 1. Like patients with an intact ventral pathway, CORnet-Z perceived curvilinear shapes as more similar to one another than rectilinear shapes and vice versa.

One possible explanation has to do with the stimuli used in these experiments. Specifically, these stimuli can be discriminated on the basis of local features, without invoking the dorsal pathway and a global shape representation. For instance, it has repeatedly been shown that deep neural network (DNN) models do not represent an object’s global shape [10] but are instead extremely sensitive to local features [11]. Yet, despite representing local features alone, these models exhibit high accuracy on many object recognition tasks, and the response profile of their internal units is well matched to neural processing in the ventral pathway. Indeed, using the shape stimuli depicted in Goodale and Milner’s commentary [3], we found that a shallow feedforward DNN (CORnet-Z) readily discriminated between the curvilinear and rectilinear shapes that were presented to patients (Figure 1). This suggests that a global shape representation may not be needed to discriminate these objects, and, hence, it would be expected that a patient with an intact ventral, but not dorsal pathway (e.g., patient R.V.), would also be able to discriminate these objects. Thus, although large-scale resections of the ventral pathway cause object recognition impairments, it’s not clear whether these impairments are rooted in a shape perception deficit per se.

We might then ask, in what kinds of object recognition tasks should patients with dorsal damage be impaired? As we suggest in our Opinion paper [1], the dorsal pathway and global shape representations may be especially crucial when local features are not diagnostic of identity, such as when we encounter a novel object exemplar or when learning a new object category. Indeed, global shape may be particularly important in infancy, when children do not yet have much object experience to draw on [10]. Moreover, it is important to note that the dorsal pathway’s contribution to object recognition may extend beyond global shape as it is defined here. Indeed, a well-studied function of the dorsal pathway is depth and 3-D shape perception [12], which may support the formation of invariant representations of objects more generally.

Altogether, these findings illustrate a common false equivalence in the neuropsychological patient literature. Although it is true that the dorsal pathway is more strongly implicated in visually guided action and the ventral pathway is more strongly implicated in visual recognition, it does not mean that the visual properties needed to accomplish each task are restricted to that visual pathway. Future work should examine not only what kinds of tasks patients succeed at but also how patients accomplish the tasks. As we have illustrated here, DNNs and other computational models [10] provide a powerful method by which researchers can generate predictions about patients’ performance on various tasks and the visual properties they may rely on.