Visual Functions of Primate Area V4

Abstract

Area V4-the focus of this review-is a mid-level processing stage along the ventral visual pathway of the macaque monkey. V4 is extensively interconnected with other visual cortical areas along the ventral and dorsal visual streams, with frontal cortical areas, and with several subcortical structures. Thus, it is well poised to play a broad and integrative role in visual perception and recognition-the functional domain of the ventral pathway. Neurophysiological studies in monkeys engaged in passive fixation and behavioral tasks suggest that V4 responses are dictated by tuning in a high-dimensional stimulus space defined by form, texture, color, depth, and other attributes of visual stimuli. This high-dimensional tuning may underlie the development of object-based representations in the visual cortex that are critical for tracking, recognizing, and interacting with objects. Neurophysiological and lesion studies also suggest that V4 responses are important for guiding perceptual decisions and higher-order behavior.

INTRODUCTION

Human and nonhuman primates rely heavily on their visual system to make critical decisions in daily life. When a monkey swings through the rainforest, or when humans manipulate objects, interpret facial expressions, or look skyward to assess the rain clouds, detailed processing of a visual scene is important. Success depends on perceiving the shape of objects and the textural quality of surfaces, what Adelson & Bergen (1991) refer to as the things and the stuff, respectively, in a visual scene. In the primate brain, this is based on information processing along the ventral visual pathway, which runs from cortical area V1 through V2 and V4 to the subregions of the inferotemporal (IT) cortex. In this review, we focus on V4, a mid-level processing stage along this pathway. Our thesis is that area V4 is important for parsing objects from visual scenes and for generating goal-oriented representations of objects to facilitate perception, decision making, and behavior. In other words, V4 is where a representation for “things” begins to emerge. Based on supporting anatomical, neurophysiological, behavioral, and lesion literature, we argue that:

1. V4 is important for the emergence of object-based representations in the ventral visual cortex.

2. V4 representations guide perceptual decisions and behavior. In this sense, they are goal-oriented representations (as opposed to simply representing the retinal image).

ANATOMICAL CONNECTIONS OF AREA V4

V4 is characterized as a mid-level visual processing stage that receives inputs primarily from area V2 and sends outputs to IT cortex. However, this characterization belies V4’s extensive connectivity (Figure 1). While much of peripheral V4 receives feedforward projections primarily from V2, central representations receive projections from V1 that bypass V2 (Nakamura et al. 1993). This connectivity could explain the short latencies of many V4 neurons (Figure 2). V4 also sends feedback projections to both V1 and V2. V4 is reciprocally interconnected with areas in the dorsal stream, including V3, the middle temporal area (MT), the medial superior temporal area (MST), and the lateral intraparietal area (LIP), and thus may be important for processing spatial information and dynamic stimuli (Figure 1). V4 is also directly interconnected with frontal areas, frontal eye fields (FEF), and the ventrolateral prefrontal cortex (vlPFC) (Barbas & Mesulam 1985, Ninomiya et al. 2012b, Ungerleider et al. 2008), consistent with its proposed role in attentional processing and decision making. Recent studies with trans-synaptic rabies virus have demonstrated disynaptic and trisynaptic feedback to V4 from the medial temporal lobe structures and the hippocampus (Ninomiya et al. 2012a). Anterograde and retrograde tracing studies have identified afferent, efferent, and bidirectional connections between V4 and several subcortical structures, including the caudate and putamen of the basal ganglia, the amygdala, the pulvinar, the lateral geniculate nucleus, the claustrum, and the superior colliculus (Gattass et al. 2014). Among the visual cortical areas that have been well studied (in terms of anatomical connectivity), V4 is the most interconnected, with robust pathways to 21 areas, suggesting that V4 may play a pivotal role in different aspects of cortical processing (Felleman & Van Essen 1991).

Figure 1.

Interconnections between V4 and other brain regions. Areas connected with V4 are highlighted on the lateral surface of the macaque monkey brain. (a) Visual cortical areas along the ventral stream, many along the dorsal visual stream and frontal areas, and (b) the medial temporal lobe and many subcortical structures are interconnected with V4. Figure created using data from Felleman & Van Essen (1991), Parker (2007), Ungerleider et al. (2008), Gattass et al. (2014), and Ninomiya et al. (2012a,b). Part of the figure was adapted with permission from Pasupathy et al. (2018). Abbreviations: AIT, anterior inferotemporal area; CIT, central inferotemporal area; DP, dorsal prelunate area; DR, dorsal raphe; FEF, frontal eye field; FST, fundus of the superior temporal sulcus area; IP, intraparietal areas; LC, locus coeruleus; LGN, lateral geniculate nucleus; MR, medial raphe; MST, medial superior temporal area; MT, middle temporal area; nbM, basal nucleus of Meynert; PIP, posterior intraparietal area; PIT, posterior inferotemporal area; PO, parieto-occipital area; R, thalamic reticular formation; SC, superior colliculus; V3a, visual complex V3 part A; V4t, V4 transition zone; vlPFC, ventrolateral prefrontal cortex; VT, ventral tegmentum.

Figure 2.

Distribution of response onset latencies of V4 neurons. (a) Onset latencies of 276 V4 neurons from three monkeys. Latency measurements were based on the responses of single V4 neurons in animals engaged in a passive fixation task. As animals fixated a variety of shape stimuli were presented within the receptive field of the cell under study. Peristimulus time histograms (PSTHs) were constructed based on evoked responses of each neuron and onset latencies were calculated based on the half-height method, i.e., as the time point at which the PSTH exceeded the mean of the peak and baseline rates. Onset latencies ranged from 25 ms to 200 ms; mean latency was 76.6 ms. For further details, see Zamarashkina et al. (2020). (b) PSTHs of two example V4 neurons studied during passive fixation, displaying approximately 100 ms difference in onset latency. PSTHs were aligned on stimulus onset and smoothed with a Gaussian kernel (σ = 4 ms).

This extensive connectivity supports a comprehensive role for V4 not only in representing visual stimuli, but also in recognition and attention. V4 is uniquely positioned to execute computations associated with both an early processing stage and a high-level stage. This is reflected in the broad range of response latencies observed in V4 (Figure 2): Many neurons respond soon after stimulus onset with latencies that rival neurons in MT and FEF, while many others respond with long latencies comparable to neurons in IT cortex.

ENCODING IMAGE STRUCTURE IN A VISUAL SCENE

Studies over decades have demonstrated that, in V1, the first stage of cortical visual processing, neurons encode visual stimuli in terms of local orientation and spatial frequency (Albrecht et al. 1980; Hubel & Wiesel 1959, 1968; Movshon et al. 1978a,b). Specifically, each small patch of the retinal image is represented by a subpopulation of V1 neurons that signals the orientation and scale of image features at a particular position in visual space, and thousands of such V1 subpopulations that tile the visual field represent the full retinal image. In the next processing stage, in V2, many neurons may be specialized for encoding the structure in visual textures (Movshon & Simoncelli 2014). Texture may be thought of as an aggregate of similar elements in a somewhat ordered spatial arrangement (e.g., a crowd of people, a bunch of leaves, etc.) that together give the impression of a homogeneous surface despite lacking uniformity in color or luminance (Van Gool et al. 1985). Because of its ordered structure, texture can be described in terms of higher-order correlational statistics across position, scale, and orientation (Portilla & Simoncelli 2000). V2 is the first stage in visual cortex to exhibit sensitivity to multipoint correlations in binary pixel images (Victor & Conte 2012, Yu et al. 2015) and may thus be well-suited for encoding naturalistic texture (Freeman et al. 2013, Okazawa et al. 2017, Ziemba et al. 2016). Neurons in V2, unlike those in V1, respond more strongly to image patches that contain the correlational statistics found in natural visual texture than to images that lack them. This preferential encoding in V2 is also evident in human functional magnetic resonance imaging (fMRI) results and is predictive of human ability to detect naturalistic texture. V2 neurons also respond similarly to a family of textures that share correlational statistics, despite variations in local image detail, providing a neuronal basis for texture classification. Taken together, these results support the idea that V2 neurons selectively and differentially encode naturalistic texture and may provide the neuronal basis for its perception.

IMPORTANCE OF OBJECT-BASED REPRESENTATIONS

Selectivity for naturalistic texture in V2 has typically been characterized with homogeneous, circular texture patches. Thus, these results do not reveal how the presence of a boundary within the receptive field (RF), defined by a texture contrast or, more generally, a heterogeneous texture, might influence V2 encoding. However, based on psychophysical studies (Freeman & Simoncelli 2011, Rosenholtz et al. 2012), the dominant view has been that V2 neuronal responses largely reflect the summary statistics within the image patch, without an explicit representation of contrasts that signify object boundaries (but see Wallis et al. 2019). Object boundaries may influence responses insofar as they influence the correlational statistics, but there is no proposed mechanistic basis for a distinction between object boundaries and texture.

In natural vision, however, segmenting object boundaries and representing the shape of component objects are critically important for several reasons. First, they facilitate shape perception, which is critical for object recognition. Even though familiar objects can be recognized based on color and texture, shape appears to be the fastest and most important cue (Elder & Velisavljevic 2009). Shape perception is also important for guiding motor plans for manipulating or interacting with objects (Cooke et al. 2007, Schaffelhofer & Scherberger 2016).

More generally, object segmentation, recognition, and tracking could provide a framework for the visual system to interpret the abundant information in crowded and dynamic scenes. Object-based representations may be critical for attentional selection (Kahneman et al. 1992, Treisman et al. 1983), for continuity of scene perception across saccades, for saccade planning, and for overall scene understanding. For example, in natural scenes, preferential viewing locations and corrective saccades are associated more strongly with object center than with salient image features (Nuthmann & Henderson 2010, Schut et al. 2017). Object representations can also serve as anchors to enable interpretation in conditions where visual information is missing or ambiguous: Attention can be directed to illusory as well as real objects (Martinez et al. 2007), and inferred scene structure, not raw image content, drives reconstruction of corrupted natural scenes (Neri 2017). Additionally, objects, rather than object features, may function as fundamental units in visual working memory (Balaban et al. 2019, Luria & Vogel 2011, Markov et al. 2019).

Overall, the establishment of object-based representations in area V4 could enable several functions that maintain a stable and unambiguous percept of the visual world.

PRECURSOR TO AN OBJECT-BASED CODE-BOUNDARY ENHANCEMENT

Two characteristics of V1 surround modulation-collinear facilitation and flexible surround suppression-may be important first steps for enhanced encoding of object boundaries in subsequent processing stages in visual cortex. Several studies have demonstrated that collinear elements outside the RF could facilitate V1 neuronal responses, especially in the presence of randomly oriented distractors (Bauer & Heinze 2002, Kapadia et al. 1995, Polat et al. 1998). Collinear facilitation processes in V2 appear to integrate depth information consistent with the segmentation of surfaces and partially occluded boundaries (Bakin et al. 2000). Many neurons exhibit tuned surround suppression when the RF center and surround are activated by stimuli with similar characteristics (see, e.g., Blakemore & Tobin 1972, Cavanaugh et al. 2002, Kapadia et al. 1995, Knierim & Van Essen 1992, Levitt & Lund 1997, Nelson & Frost 1985, Nothdurft et al. 1999), which could enable the preferential encoding of texture boundaries that may be compared to uniform texture regions (Nothdurft et al. 1999), observed even in an anesthetized animal. Furthermore, signals related to texture boundaries emerge at roughly the same time in V1 and V4 (Poort et al. 2012), and they emerge in V1 even when V2 is inactivated (Hupé et al. 2001), suggesting that edge enhancement is based on local computations rather than on feedback mechanisms. Coen-Cagli and colleagues (2015) recently demonstrated that the strength of surround suppression in V1 for natural images depends on image similarity in the center and surround of the RF-suppression is stronger for homogeneous images and weaker for images that differentially stimulate the center and surround. Taken together, these results support the idea that contextual modulations in the visual cortex serve to detexturize images-enhancing contour representations at the expense of textures-and thus facilitate the signaling of object boundaries (Gheorghiu et al. 2014). Such enhanced encoding of contrast boundaries (as opposed to texture elements) may be a critical first step for the emergence of object-based representations.

EVIDENCE FOR OBJECT-BASED REPRESENTATIONS IN V4

Several lines of evidence support the hypothesis that object-based representations begin to emerge in V4. In a series of experiments, first with isolated angles and curves (Pasupathy & Connor 1999) and then with simple shapes (Pasupathy & Connor 2001), Pasupathy & Connor demonstrated that V4 neurons are selective for specific boundary conformations. Two examples are shown in Figure 3a. Both neurons respond strongly to some shapes and weakly to others. For the top neuron, the preferred shapes included a sharp convexity to the upper right, while shapes with a broad convexity at that same location evoked weak responses. Such shape selectivity could be based on an RF hotspot, i.e., stimulation of a specific RF subregion to the upper right could underlie strong responses. To rule out this possibility, several past studies have investigated whether shape preference was position invariant. Figure 3b shows the results of such a test, where the rotation tuning of a neuron is probed at different positions within the RF. Responses were strongest at a central position and weaker when the stimuli were shifted in either direction from the center. However, the shape preference was maintained across all positions-the shape with the sharp convexity pointing to the lower left evoked the strongest responses. This pattern of responses was typical across V4-many V4 neurons exhibited shape tuning that is independent of spatial position (El-Shamayleh & Pasupathy 2016, Gallant et al. 1993)-thereby disproving the possibility that shape preference is due to RF hotspots.

Figure 3.

Position-invariant tuning for boundary conformation in V4. (a) Examples of tuning for boundary conformation. Responses of two example V4 neurons (top and bottom) to simple 2D shapes recorded in a fixating animal (for further details, see Kim et al. 2019a). Response frequency histograms (right) show that the shape stimuli evoked a broad range of responses from both neurons. The 20 shapes that evoked the strongest (PREF, red) and weakest (NPREF, blue) responses are shown for each neuron (left). For the top neuron, all of the preferred shapes included a medium-sharp convex feature to the upper right; shapes that evoked a weak response did not include this feature (compare PREF and NPREF shapes). For the second neuron, preferred but not nonpreferred shapes included a concavity at the top of the shape. Responses of both neurons can be well-explained by a two-dimensional angular position × boundary curvature model that captures the conformation of the shape boundary (Kim et al. 2019a). Panel adapted from Kim et al. (2019a) (CC BY 4.0). (b) Example of position-invariant shape tuning. This neuron shows strong narrow tuning for the orientation of a shape stimulus, responding best when the sharp convex projection is to the lower left. Shape selectivity was consistent irrespective of the absolute position of the shape within the receptive field (compare tuning for the different lines). The response magnitude varied across position, but the preference remained the same.

Another possibility is that the shape preference illustrated in Figure 3a is based on a simple preference for feature conjunctions. For example, preference for shapes with a sharp convexity pointing up could be built by pooling signals from V1 or V2 neurons sensitive to 45° and 135° orientations and then repeating such a template across positions to build position-invariant tuning for feature conjunctions (Figure 4a). The hierarchical MAX pooling (HMAX) model for invariant object recognition proposed by Poggio and colleagues (Cadieu et al. 2007, Riesenhuber & Poggio 1999, Serre et al. 2005) is based on such a strategy to build selectivity and invariance in alternate stages. However, if this were the case, V4 neurons would respond similarly to shapes with a convex projection and those with a concave indentation where the convex and concave parts have matched contour conformations (Figure 4b). This is not what we find. Instead, V4 responses are sensitive both to the contour conformation and to its position relative to object center. For example, in Figure 4c, the neuron responds strongly to stimuli with a concavity to the right of the shape (orange arrow) but not to shapes with a convexity to the left (blue arrow), even though the boundary conformations are identical. This is true even when the local contrast is flipped. Thus, V4 neurons encode information about a feature and its position along the object boundary, i.e., they represent contour features in an object-centered reference frame. Pasupathy & Connor (2001) characterized such object-centered preference for shape in terms of tuning in a 2D shape space for boundary curvature and object-centered angular position. In this scheme, the top neuron in Figure 3a encodes a sharp convexity to the upper right of the shape, and the example in Figure 4c encodes a sharp concavity to the right. Implicit in this model is the idea that the responses of a V4 neuron reflect the shape of the boundary as it relates to the rest of the object-whether a specific contour is a convex projection or a concave indentation requires knowledge about the rest of the object-and where it sits along the object boundary.

Figure 4.

Object-centered tuning for boundary conformation in V4. (a) Preference for a sharp convexity pointing up could be achieved by pooling signals from units tuned to an appropriate combination of orientations (feature template) and repeating the template at multiple positions [translation in the receptive field (RF)]. (b) This model would predict similar responses to the two shapes shown, one with a convex projection at the top of the shape and the other with a concave contour at the bottom of the shape, and would therefore not represent an object-centered code. (c) Example of object-centered tuning for contour conformation. Responses of this neuron show clear rotation tuning for a shape with a concave feature along the boundary. Responses are strongest (orange arrow) when the concavity is to the right of shape center. When the same boundary conformation forms a convex projection to the left of shape center, responses are weak (blue arrow). Thus, responses are not dictated by contour conformation alone.

To test this idea further, Bushnell et al. (2011b) investigated whether V4 neurons respond strongly to the preferred contour feature when it is formed by the accidental juxtaposition of an occluded and an occluding object boundary. In Figure 5a, the convexity, Θ, is a real contour feature that is part of a crescent-shaped object. In Figure 5b, however, the same convexity is the result of partial occlusion of an orange circle by a blue circle. Bushnell et al. found that many V4 neurons that respond preferentially to a real contour feature are unresponsive if that feature is the result of partial occlusion, as illustrated in Figure 5c. Such suppression cannot be explained on the basis of low-level factors of local chromatic or luminance contrast, number of stimuli, etc. (Bushnell et al. 2011b). These results support the hypothesis that V4 neurons encode only those features in a visual scene that provide information about the true shape of objects. Such suppressed encoding of accidental contours may also form the basis for the emergence of border ownership signals in earlier stages (Pasupathy et al. 2018, Zhou et al. 2000).

Figure 5.

Representation of real but not accidental object boundaries in V4. (a, b) Accidental versus real contours. The angle Θ in panel a is a real contour, while in panel b, it is formed by the accidental occlusion of one circle by another. Accidental contours bound the occluded object, but they carry no information about its true shape, and they are perceptually discounted. (c) Responses of two example neurons that encode real but not accidental contours. We studied responses of neurons to crescent shapes in eight orientations (x axis), either in isolation (orange) or in combination with a circle (magenta); the circle was also presented in isolation (blue) at the same eight positions in the receptive field as in the combination stimulus. The neuron on the left responds preferentially to shapes with a sharp convexity at the bottom, as reflected in the tuning curve for the crescent alone (orange). When the crescent is formed by partial occlusion in the case of the combination stimulus, responses are suppressed (magenta) because the preferred sharp convexity is now an accidental feature. The neuron on the right responds best to shapes with a broad convexity to the upper right, consistent with the strong responses to the circle alone (blue). In this case, responses are not suppressed with the addition of a circle because the preferred broad convexity remains a real contour (for further details, see Bushnell et al. 2011b). (d) A population of V4 neurons can provide a complete representation of isolated shapes based on the component contour features. For example, the cat may be encoded by V4 neurons selective for sharp convexities to the upper right and upper left; concavity to the top; and broad convexities to the right, left, and bottom (see Pasupathy & Connor 2002). Panels a-c adapted from Bushnell et al. (2011b) (CC BY-NC-SA 3.0).

Finally, across the V4 population, a greater number of neurons are tuned to sharp convexities and concavities than to shallow convex or concave curves. This is consistent with a sparse object-coding scheme where uncommon but diagnostic regions of acute contour curvature are preferentially encoded (Carlson et al. 2011). Such encoding of object boundaries first emerges in V4 and later in V1, possibly via feedback (Chen et al. 2014).

Taken together, these results support the idea that neurons in area V4 encode parts of an object in terms of the contour features using an object-centered code. In the image in Figure 5d, some neurons encode the sharp convexity to the upper right, others encode the concavity to the top, etc. Together, a population of V4 neurons could provide a complete and accurate representation of the entire shape (Pasupathy & Connor 2002) and could thus contribute to its recognition and perception.

JOINT ENCODING OF BOUNDARY AND SURFACE PROPERTIES TO FACILITATE OBJECT SEGMENTATION

Not all V4 neurons are sensitive to object shape. Many neurons are sensitive to surface properties including hue, saturation and luminance contrasts (Bushnell et al. 2011a, Conway et al. 2007, Namima et al. 2014, Schein & Desimone 1990, Zeki 1973), luminance gradients (Hanazawa & Komatsu 2001), and surface texture. Previous studies have documented V4 selectivity for Cartesian and non-Cartesian gratings (Gallant et al. 1993) and in response to brief, sequential presentations of small oriented elements that may be considered dynamic texture (Nandy et al. 2013). Studies have also demonstrated selectivity in V4 for homogeneous naturalistic textures (Arcizet et al. 2008), and such selectivity can be explained on the basis of higher-order correlational statistics (Okazawa et al. 2015). In addition, stimulus dimensions critical for human texture perception, namely, coarseness, directionality, and regularity (Kim et al. 2019b), dictate the texture responses of many V4 neurons. Such V4 sensitivity to surface characteristics is expected, since much of our visual environment is indeed composed of surfaces rather than object boundaries, and processing and accurately perceiving such information are critical for decision making in everyday life (Adelson 2001).

Information about object shape and surface characteristics is often multiplexed in V4 neurons. While some V4 neurons exhibit exclusive tuning for shape or texture, a majority exhibit joint, independent tuning for both stimulus attributes (Kim et al. 2019a). In fact, many neurons that exhibit shape-selective responses for stimuli defined by luminance contrast relative to the background exhibit weak, nonselective responses to shapes defined solely by an outline without an interior fill, and vice versa. These results support the hypothesis that shape selectivity in V4 is dictated both by the boundary and by surface characteristics (Popovkina et al. 2019). Roughly 15% of shape-selective V4 neurons respond best to shapes with blurred boundaries (Oleskiw et al. 2018), and roughly 20% respond selectively to shapes defined solely by a chromatic contrast (Bushnell et al. 2011a). Altogether, these results suggest that individual V4 neurons are tuned in a high-dimensional space that allows the joint encoding of shape and surface characteristics of object parts. In natural vision, object boundaries are defined by contrasts in luminance, texture, and/or chromaticity, and three-dimensional objects that curve away from the plane of fixation may have blurred boundaries. The diverse V4 response properties described above-sensitivity to luminance, texture and chromatic contrasts, blurry boundaries, and luminance gradients-may be well-suited to segment from the background object parts defined by a variety of stimulus cues (Figure 6).

Figure 6.

Joint encoding of multiple features for object segmentation. The input image on the left includes a variety of cues; the bottom images are filtered to include information from each cue in isolation and illustrate that form information may be encoded by a contrast in luminance, color, or texture. When viewing such a stimulus, responses of individual V4 neurons are dictated by tuning in a high-dimensional stimulus space defined by shape, luminance, color, texture, blur, depth, etc., which facilitates the effective segmentation of visual objects from the background that may be defined by contrast along a variety of stimulus dimensions.

GOAL-ORIENTED REPRESENTATIONS OF OBJECTS

Above, we focus on how simple stimuli-isolated shapes and textures-are encoded in V4 during passive fixation. These experiments (simple stimuli, fixation task) and analysis methods (average activity during stimulus presentation) largely capture feedforward processing in V4. However, V4 is heavily interconnected with other brain areas (see Figure 1), and there are extensive recurrent connections within V4. Experiments that engage these circuits with targeted stimuli and/or behavioral manipulation, and analyses that probe response dynamics, reveal that V4 representations are goal-oriented in several important ways (Figure 7a): (a) They are more closely aligned with the perceived rather than the retinal image; (b) they may guide the choice for the next saccade target; (c) they reflect the targeted processing of stimuli of interest, i.e., those that are the focus of attention; and (d) they may contribute to the sequential comparison of visual stimuli, i.e., comparing the memory representation of a stimulus to the visual representation of another. We discuss each of these below.

Figure 7.

Goal-oriented representations. (a) When confronted with the challenge of spotting bananas in a cluttered produce aisle, the subject may saccade to different locations with yellow objects (dashed trajectory) and compare the shape of the object at the attentional focus (circles) with a remembered object. Area V4 is thought to be important for all aspects of this process. (b) Size illusion. The retinal sizes of the two sasquatches in this image are identical, but the perceived sizes are dramatically different. This is because the surrounding context suggests that the sasquatch at right is farther away from the observer; thus, the same retinal size would imply a much larger sasquatch farther away.

Building a Percept from the Retinal Input: Solving the Inverse Problem

The image of an object can induce strikingly different percepts depending on the surrounding context. For example, the two sasquatches in Figure 7b are identical, yet the one on the right appears to be larger. This simple illusion serves to illuminate the complexity in the computations that transform a retinal image into a percept. The retinal image is a function not just of the properties of the objects and surfaces in the visual world (e.g., shape, size, texture), but also of the viewing conditions (illumination, viewing distance, and angle) and of object-object and object-surface interactions (occlusion). Thus, for example, objects of different sizes at different distances from the observer could cast identical images on the retina. To produce a percept that is informative and actionable from this entangled retinal representation, processing in the successive stages must solve an inverse problem that produces best estimates for the true attributes of the objects and surfaces after accounting for viewing conditions. We do not know how the brain solves this computationally challenging, often ill-defined, inverse problem, but contextual modulations and feedback from higher processing stages are thought to be important (Lamme et al. 1998). Investigations of color constancy and the processing of partial occlusions indicate a role for V4 in this process.

A body of work on human patients, as well as physiological and lesion studies in monkeys, suggests that color constancy-the percept of true color independent of the illuminant-is based on contextual modulations in V4. Several studies have identified patients whose color constancy mechanisms have failed (Clarke et al. 1998, Kennard et al. 1995, Zeki et al. 1999), i.e., their color perception is based on wavelength composition and is not independent of the illuminant. All of these patients suffered damage to regions of the cortex that include human V4 (Walsh 1999). Color constancy is also impaired in monkeys with V4 lesions (Heywood et al. 1992, Schiller 1993, Walsh et al. 1993). In neurophysiological studies, Zeki and colleagues (Kusunoki et al. 2006) demonstrated that V4 color tuning peaks shift in predictable ways when the illumination of a multicolored, Mondrian-like background was changed. Similar shifts in the perceived color were also observed in humans and monkeys when assayed with a match-to-sample task. These results support the hypothesis that regularities in reflected wavelength patterns due to illuminants in the surround influence color tuning functions in V4 and thus contribute to the perception of color constancy.

Geometric regularities in image features caused by occlusions may also be discounted by contextual modulations in V4. V4 neurons tuned to a sharp convexity are more strongly suppressed by adjoining contextual stimuli than are neurons tuned to a broad convexity (Figure 5c). This could reflect the geometric prior that curvature discontinuities are a diagnostic feature of partial occlusion but do not carry any information about the true shape of the occluded object. Thus, the greater suppressive influence of adjoining stimuli on the representation of sharp convexities could facilitate the preferential encoding of real object boundaries, rather than accidental contours. Maier and colleagues (Cox et al. 2013) have demonstrated elevated responses in a subpopulation of V4 neurons to illusory Kanizsa surfaces, supporting the hypothesis that active perceptual completion of surfaces and shapes draws on the selective enhancement of activity within V4. This is also consistent with V4 lesion studies demonstrating an impairment in the perception of illusory contours (De Weerd et al. 1996).

Interactions between the visual and frontal cortex may also contribute to solving the inverse problem of deriving a percept from a retinal image. When monkeys discriminate shapes under partial occlusion, the initial feedforward transient burst in V4 declines with increasing levels of occlusion (Figure 8a). Nevertheless, a second transient burst is observed in many neurons that is stronger for occluded stimuli (Figure 8a); the net result is that the average response during the first and second transients is less dependent on the occluders and more dependent on the occluded stimulus that needs to be discriminated. This is captured by the neurometric curve, which is much improved when based on activity in the 50–175 ms duration, as opposed to activity in the 50–125 ms duration (Figure 8c). A similar improvement is not observed in neurons without a second peak (Figure 8b,d). Thus, over time, V4 activity more strongly encodes the stimulus that needs to be discriminated than the occluders, which are irrelevant in this task. The latency of the first and second transient peaks in V4 and timing of peak activity in the ventrolateral prefrontal cortex (vlPFC) support the hypothesis that feedback from the vlPFC may give rise to the second peak in V4 (Fyall et al. 2017). This is also supported by demonstrations of coupling in local field potential (LFP) between V4 and the vlPFC during memory maintenance (Liebe et al. 2012).

Figure 8.

Interactions between the visual and frontal cortex during shape discrimination. (a, b) Responses of two example V4 neurons that exhibit different response profiles to partially occluded shape stimuli. PSTHs in panel a exhibit two transient peaks (black and red bars), while those in panel b exhibit only one transient peak (black bar). During the first transient in both neurons, responses decline with increasing levels of occlusion (line colors; decreasing percent visible area). During the second transient in panel a, which may be based on feedback from frontal cortex, responses are stronger for intermediate levels of occlusion (for further details, see Fyall et al. 2017, Kosai et al. 2014). Adapted from Fyall et al. (2017) (CC-BY). (c, d) Psychometric and neurometric curves based on the data in panels a and b respectively. In both, psychometric performance (dark gray dotted line) declines with increasing levels of occlusion. The neurometric curve based on a larger time window (orange line) shows improved performance at intermediate levels of occlusion for the neuron in panel a but not the one in panel b due to the enhanced shape selectivity for occluded stimuli during the second transient peak.

Choosing the Next Saccade Target

Several lines of evidence suggest that the V4 representation maintains a saliency map that informs the next saccade. Mazer & Gallant (2003) have demonstrated that, prior to a saccade, a majority of V4 neurons exhibit a presaccadic enhancement that reflects the bottom-up salience of RF features, rather than an oculomotor command. A majority of V4 neurons also exhibit a top-down modulation based on target selection. Mazer & Gallant argue that such convergence of bottom-up and top-down processing streams in area V4 results in an adaptive, dynamic map of salience that guides oculomotor planning during natural vision. Burrows & Moore (2009) demonstrated that V4 responses to an oriented, colored bar within the RF were stronger when the surrounding stimuli were designed to produce perceptual popout for color, orientation, or conjunction, but no such enhancement was observed in V1. Schiller & Lee (1991) also demonstrated that V4 lesions impair the ability of monkeys to saccade to the odd-one-out target from an array of distractors, especially if the targets were of low contrast, smaller in size, or slower (see also De Weerd et al. 1999). Thus, V4 responses appear to be critical for identifying stimuli that are different from surrounding stimuli, potentially as targets for the next saccade, even when those stimuli are not the brightest, largest, or fastest.

Attentional Modulations in Area V4

Some studies have shown that behavioral allocation of attention produces robust modulation of the strength of neuronal responses in V4, their synchrony, and their noise correlation (Cohen & Maunsell 2009, Mitchell et al. 2009). V4 attentional effects may be both spatial position-based and feature-based (for a detailed review, see Maunsell 2015). Briefly, studies have demonstrated (a) a shrinking of the V4 RF about the object that is the focus of attention (Moran & Desimone 1985); (b) enhanced responses in a small neighborhood surrounding the focus of attention (spotlight of attention) (Connor et al. 1996); (c) enhanced gamma-band synchronization in V4, which could enhance the postsynaptic impact of V4 neurons driven by the attended stimulus (Fries et al. 2008); and (d) enhanced responses in V4 when a preferred feature in the RF matches the target feature (for a review, see Maunsell & Treue 2006). Feature-based attention could also alter tuning. David and colleagues (2008) demonstrated shifts in orientation and spatial frequency tuning peaks of many V4 neurons toward the direction of the orientation and spatial frequency content of the sought target. Popovkina & Pasupathy (2019) demonstrated that many V4 neurons also exhibit broadening of tuning width for an irrelevant tuning dimension (color) when animals made shape judgements. Such broadening cannot be explained by a gain change alone. Thus, attentional modulations at the level of V4 facilitate the targeted processing of a spatial location, an object, or even a specific stimulus dimension. Feature- or object-based (see Pooresmaeili et al. 2014) attentional mechanisms may be especially important for the maintenance of internal representation of targets and thus for achieving behavioral goals (Hayden & Gallant 2005).

Sequential Comparison of Simple Visual Stimuli

In everyday situations, we often compare items that we are looking at with what we have seen previously, as when we ask, “Have we met before?” or “Have I been here before?”, or when we look for a familiar object in a cluttered environment (see, e.g., Figure 7a). Scientists have studied the processes underlying this ability with sequential comparison tasks: Two stimuli are presented in sequence, separated by a delay, and the animal reports whether they are the same or different. To perform this task, animals have to compare the sensory representation of the second stimulus with the memory representation of the first. Because of the involvement of working memory, the prefrontal cortex (PFC) is a plausible site for this comparison (Fuster 1989, Kim & Shadlen 1999, Romo & de Lafuente 2013), but for simple stimuli, V4 may be as well. In a version of the sequential comparison task with simple 2D shape stimuli, Kosai et al. (2014) found that V4 signals carry all task-relevant variables: the sensory representation of the second stimulus, the memory representation of the first stimulus, and the outcome of the computation (Figure 9). Crucially, the memory representation of the first stimulus arises just prior to the onset of the second stimulus, suggesting that it may be communicated from elsewhere in the brain. This idea is consistent with evidence for increased functional connectivity between V4 and the PFC during memory maintenance (Liebe et al. 2012). Furthermore, outcome-related, same-versus-different selectivity emerges soon (approximately 125–200 ms) after second stimulus onset, supporting the hypothesis that V4 may be a site for such a computation (Kosai et al. 2014). Memory- and outcome-related signals have been previously reported in other paradigms (Haenny & Schiller 1988, Hayden & Gallant 2013, Ogawa & Komatsu 2004, Sligte et al. 2009), even when the first stimulus was tactile (Haenny et al. 1988).

Figure 9.

Memory encoding in V4. Responses of an example neuron during the performance of a sequential shape-discrimination task are shown. Stimulus 1 was presented at central fixation (outside the receptive field of the neuron), followed by an inter-stimulus interval (ISI), and stimulus 2 was presented within the receptive field (RF). The animal had to report whether stimuli 1 and 2 were the same or different. Stimulus 1 and stimulus 2 could be shapes A or B, for a total of four conditions (for details, see Kosai et al. 2014). Responses of this neuron includes three task-relevant pieces of information. First, the responses provide a sensory representation of stimulus 2 with stronger responses for shape A than shape B (compare the blue solid line and orange dashed line with the other two lines). Second, a memory representation of stimulus 1 is also evident during the ISI (blue arrow), with stronger responses when stimulus 1 was shape B (blue lines) rather than shape A (orange lines). Finally, responses also reflect whether stimuli 1 and 2 were the same or different with stronger responses during the presentation of stimulus 2 when stimuli 1 and 2 were different (solid lines, black arrows).

FUTURE DIRECTIONS and OUTSTANDING QUESTIONS

Processing in V4 is likely important for several additional aspects of visual perception that have not been investigated in depth in prior studies. We outline three of these below. We also discuss outstanding questions related to how V4 response properties may be built and how neurons with different response properties may be arranged across V4.

Dynamic Stimuli

Most neurophysiological studies focused on the processing of dynamic stimuli target dorsal stream areas (Manning & Britten 2017). However, 15–30% of V4 neurons exhibit direction bias (Ferrera et al. 1994, Tolias et al. 2005), i.e., they respond more strongly to one motion direction than to the opposite one, and V4 is strongly interconnected to dorsal stream areas MT and LIP. Bigelow et al. (2019) demonstrated that V4 neurons signal the direction of motion when an object is intermittently displaced across the display with large spatial steps (dx > 0.5°), providing the first known neuronal correlate of long-range apparent motion. Thus, V4 may be important for segmentation and tracking of dynamic objects, and future experiments will need to investigate how this might be achieved. In-depth studies are also needed to determine how V4 motion signals are computed and whether form and motion direction signals are multiplexed in the responses of single neurons.

3D Stimuli

V4 is likely critical for building a 3D percept of objects and surfaces, since V4 lesions in monkeys appear to impair this ability (Merigan & Pham 1998); however, this has not been extensively explored. Many V4 neurons are disparity tuned (Hinkle & Connor 2001), and many encode the slant of a bar in depth (Hinkle & Connor 2002). V4 neurons also encode nondisparity cues that are important for building a 3D percept. For example, when an object curves away from the plane of focus, the boundary is blurry, and many shape-selective V4 neurons respond preferentially to such blurry boundaries (Oleskiw et al. 2018). V4 neurons are also selective for surface luminance gradients (Hanazawa & Komatsu 2001), which could contribute to the recovery of 3D information from shading. Consistent with this observation, Arcizet and colleagues (2009) demonstrated that a population of V4 neurons can differentiate between 3D shapes defined by surface shading and corresponding 2D control stimuli. Future studies will need to delineate how 3D information from a variety of cues, e.g., disparity, shading, blur, and occlusions, may be integrated to build a percept that is invariant to illumination direction.

Visual Crowding

Because most previous studies have focused on the encoding of isolated stimuli, we do not know whether V4 encoding is lossy when there are multiple nearby objects. Studies in human subjects have explored the limits of object recognition in crowded displays (Balas et al. 2009, Levi 2008, Pelli & Tillman 2008). For accurate recognition in humans, the minimum spacing between objects scales with eccentricity [Bouma’s law (Bouma 1973): approximately 0.3–0.6 × RF eccentricity]. Because monkey V2 and V4 RF diameters also scale with eccentricity at approximately 0.3 and 0.6 × RF eccentricity (Gattass et al. 1981, 1988), respectively, the limits of recognition in clutter may be a direct consequence of the representation in the mid-level visual cortex. We do know from prior studies that surround modulation can be strongly suppressive in V4 (Schein & Desimone 1990) and that responses to pairs of stimuli may be modeled as the maximum of the responses to the component stimuli in isolation (Gawne & Martin 2002). More extensive and systematic studies are needed to document how the number, distance, and properties of the neighboring stimuli modulate V4 responses and how this could underlie perceptual limitations due to crowding. Such experiments would be critical for us to begin to understand how natural scenes may be encoded in V4.

Functional Organization

The distinct patterns of input and output connections of V4 subregions (DeYoe et al. 1994, Felleman et al. 1997, Zeki & Shipp 1989) and the similar preferences of nearby neurons recorded on a single electrode penetration (Gallant et al. 1996, Watanabe et al. 2002) hint that an underlying functional organization exists. Recordings with laminar probes also suggest segregation of units encoding sensory information and eye-movement planning in the superficial and deep layers (Pettine et al. 2019). Experiments employing functional neuroimaging methods (e.g., fMRI and optical imaging) have demonstrated that V4 contains clear functional domains in terms of preference for color, orientation, spatial frequency, size, motion direction, and disparity (Conway et al. 2007, Fang et al. 2019, Ghose & Ts’O 1997, Li et al. 2013, Lu et al. 2018, Tanigawa et al. 2010). To investigate how multiple feature dimensions may be arranged across V4 (i.e., microarchitecture of functional domains), advanced methods that facilitate high-density sampling of single-unit activity are required. Recent studies using two-photon calcium imaging have revealed neuronal clusters associated with curvature or corners (Jiang et al. 2019) and 3D shape encoding (Nielsen 2019). Future studies with high-density electrode recording and imaging techniques could explain how the neurons that encode various stimulus dimensions or participate in attentional modulation and behavioral influence are laid out across the cortex.

Models

A major outstanding question is how V4 responses are built from responses in V1 and V2. Several models have been proposed, typically to explain responses to one specific class of stimuli, but they often fail to generalize to other classes. For example, the HMAX model for object recognition can provide a good fit for tuning for boundary curvature by pooling from a set of V1 neurons tuned to appropriate orientations (see Figure 4a). However, such a contour template model does not capture the object-centered nature of shape coding in V4 and fails to achieve the level of position invariance observed in real neurons (Bair et al. 2015). It also fails to capture the diversity in fill-outline invariance observed in V4 responses (Popovkina et al. 2019). In contrast, the texture statistics model of Okazawa and colleagues (2015) can capture tuning for texture, but it fails to generate the texture-invariant shape tuning observed in V4 (Kim et al. 2019a). Sharpee and colleagues (Nandy et al. 2013) have proposed that V4 tuning for curved contours could be produced by pooling of heterogeneous orientation signals from earlier visual areas. However, such a model cannot produce translation invariance in shape tuning, a fundamental property of many V4 neurons (El-Shamayleh & Pasupathy 2016, Gallant et al. 1993, Pasupathy & Connor 2001). The spectral RF model of Gallant and colleagues (David et al. 2006) can achieve translation invariance but not tuning for boundary curvature (Oleskiw et al. 2014).

In recent years, with the advent of efficient learning algorithms for deep learning neural networks, computer vision has made great strides with regards to object recognition: On some tasks, networks have reached levels that are comparable to humans. Encouraged by this, some studies have begun to compare the internal representations of these models and the responses of visual cortical neurons (e.g., Pospisil et al. 2018, Yamins et al. 2014). While deep networks are not models of the brain (in terms of architecture, cell types, and functionality), analyzing the network can be insightful because the emergence of similar encoding features in models and neurons could imply similar computational strategies. Dissecting the underlying architecture of model units could provide insights into how response properties arise in the brain. Furthermore, detailed study of model units could promote the development of more targeted experiments with neurons; this could be invaluable due to the practical constraints that limit experimental time.

Efforts to date are already beginning to bear fruit. Comparison of responses of single units in deep networks and monkey V4 neurons has identified the current best model for position-invariant tuning for boundary curvature in V4 (Pospisil et al. 2018). Deep networks, in a closed loop with primate physiology experiments, have been used to validate models by testing response predictions to novel synthetic stimuli (see, e.g., Bashivan et al. 2019). Future experiments that compare deep networks and the brain in terms of unit responses and behavior, especially leveraging their differences, could provide deeper edification about the why and the how of mid-level visual cortical representations and functions.

CONCLUSION

Much remains to be discovered about how mid-level visual cortex represents stimuli and how these representations underlie visual perception and object recognition and guide our behavior. The time is right to make progress-more labs, armed with more experimental and analytical tools, are undertaking this quest. To move forward toward a comprehensive understanding of the functional role of area V4 in vision, we need to conduct more experiments with a variety of parametric, artificial stimuli that serve to constrain, falsify, and update models; these models could then be validated by evaluating their responses to natural stimuli. The biggest obstacle to making progress in this manner is the constraint of experimental time. Thus, the biggest technological advance would be the ability to study the same neuron over days, weeks, and months. With this innovation, we could evaluate and characterize the responses of a single neuron to tens of thousands of stimuli; we could also study the responses of neurons under a variety of behavioral conditions and contexts. These data could then constrain more elaborate and physiologically sound models of form processing that include feedforward, recurrence, and feedback circuits.