Low-level properties of natural images predict topographic patterns of neural response in the ventral visual pathway

Abstract

Neuroimaging research over the past 20 years has begun to reveal a picture of how the human visual system is organized. A key distinction that has arisen from these studies is the difference in the organization of low-level and high-level visual regions. Low-level regions contain topographic maps that are tightly linked to properties of the image. In contrast, high-level visual areas are thought to be arranged in modules that are tightly linked to categorical or semantic information in the image. To date, an unresolved question has been how the strong functional selectivity for object categories in high-level visual regions might arise from the image-based representations found in low-level visual regions. Here, we review recent evidence suggesting that patterns of response in high-level visual areas may be better explained by response to image properties that are characteristic of different object categories.

The organization of the ventral visual pathway

The ventral visual pathway emerges from the primary visual cortex (V1), continues through a series of early visual areas (V2, V3, V4), and eventually reaches the ventral temporal cortex, where high-level visual regions are located. Neural processing in high-level regions of the ventral visual pathway is thought to play a key role in the perception and recognition of visual objects (Milner & Goodale, 1995; Ungerleider & Mishkin, 1982). Lesions to this region of the brain often result in difficulties in the perception and recognition of different categories of objects (McNeil & Warrington, 1993; Moscovitch, Winocur, & Behrmann, 1997). Although a large number of studies have investigated the functional properties of high-level regions in the ventral visual pathway, an overarching framework for how information is represented topographically is still lacking (Grill-Spector & Weiner, 2014; Op de Beeck, Haushofer, & Kanwisher, 2008).

The topographic organization of early visual areas is based on maps that are tightly linked to the properties of the visual image. The retinotopic organization in V1 provides a clear example of a topographic map in which preferred stimulus position changes smoothly across the cortical surface (Engel et al., 1994). Indeed, this retinotopic organization is used to define the boundaries of early visual areas (Wandell, Dumoulin, & Brewer, 2007). At a finer scale of cortical organization, topographic maps have also been found for orientation. For example, change in the preferred stimulus orientation of neurons has been shown to occur gradually across the cortical surface (Bonhoeffer & Grinvald, 1991; Hubel & Wiesel, 1968). The superimposition of retinotopic and orientation maps allows for the representation of all orientations at each location of the visual field (Hubel & Wiesel, 1968).

Further downstream, cells are tuned to properties that appear to combine features encoded in earlier visual areas that are statistically characteristic of natural images (Connor, Brincat, & Pasupathy, 2007; Tanaka, 1996). Within higher level regions, local functional specialization is seen with neuronal specificity for specific stimulus categories, such as faces (Perrett, Rolls, & Caan, 1982; Tsao, Freiwald, Tootell, & Livingstone, 2006). These observations are consistent with the idea of increasing abstraction, so that at the distal end of the ventral stream, neural responses are highly invariant to low-level sensory properties and increasingly sensitive to properties that characterize identity and semantics (Quian Quiroga, 2012).

In contrast to early visual areas, the organizing principles that underpin the topography of high-level visual areas are less clear. One problem is that, although electrophysiological studies have provided critical insights into the selectivity of neurons in this region of the brain, the sparse nature of the recordings makes it difficult to determine with certainty the critical dimensions along which responses vary and to understand the precise spatial organization. However, due to improved spatial coverage of the brain, neuroimaging studies have begun to shed some light on this question. Functional MRI (fMRI) studies have shown that regions of the ventral visual pathway are specialized for different categories of objects (Kanwisher, 2010). The location of these regions is broadly consistent across individuals, suggesting that common organizing principles underpin the topography. For example, some regions of the ventral visual pathway are more responsive to images of faces than to images of nonface objects (Kanwisher, McDermott, & Chun, 1997), whereas other regions are selective for images of places (Epstein & Kanwisher, 1998), body parts (Downing, Jiang, Shuman, & Kanwisher, 2001), visually presented words (Cohen et al., 2000), and inanimate objects (Malach et al., 1995). These regions are typically localized by contrasting responses to one category of stimulus with those to another. This form of univariate comparison reveals voxels which are selective for a given category, and since statistically significant differences are tightly localized, these experiments create the impression that high-level vision engages a mosaic of discrete modules.

At first sight, these observations appear very different from the maplike organization of early visual cortex. However, the presence of functional localization does not place many constraints on the form of the underlying organization. For example, if one were to compare brain responses observed when stimulating a particular part of the visual field (say the upper right quadrant) with those seen when stimulating a different part (say the lower right quadrant), one would also expect to see discrete regions of activity within the left calcarine cortex. This does not indicate that upper and lower quadrants are represented in discrete modules, because we know that there is a larger, continuous structure (i.e., a retinotopic map) which explains these patterns of response in a unified way.

Another limitation of a modular account of organization in the ventral visual pathway is that specialized regions have only been reported for a limited number of object categories (Downing, Chan, Peelen, Dodds, & Kanwisher, 2006; Op de Beeck et al., 2008). More recent studies using multivariate fMRI analysis methods suggest that the pattern of response may provide a better indication of how objects are represented across the ventral visual pathway. A number of studies have shown that the spatial pattern of response across the entire ventral stream can distinguish a greater range of object categories than has been reported with previous univariate analyses (Haxby et al., 2001; Kriegeskorte et al., 2008; Spiridon & Kanwisher, 2002). The importance of the spatial pattern of response is demonstrated by the fact that the ability to discriminate particular object categories is still evident when the most category-selective regions are removed from the analysis (Haxby et al., 2001).

The distributed nature of the fMRI response to different categories of objects within the ventral visual pathway has been interpreted as showing a topographic map of object form, otherwise known as object form topography (Haxby et al., 2001). This organization is thought to be analogous with the continuous topographic maps found in early stages of visual processing, which are tightly linked to the properties of the visual image. However, it is not clear what dimensions are important for object form topography. A variety of evidence has suggested that patterns of response in the ventral visual pathway are linked to the categorical or semantic information that the images convey (Clarke & Tyler, 2014; Connolly et al., 2012; Kriegeskorte et al., 2008; Naselaris, Prenger, Kay, Oliver, & Gallant, 2009; Walther, Chai, Caddigan, Beck, & Fei-Fei, 2011). Evidence for other organizing principles can be found in the large-scale patterns of response to animacy (Chao, Haxby, & Martin, 1999; Kriegeskorte et al., 2008) and to the real-world size of objects (Konkle & Oliva, 2012). However, it remains unclear how these higher level categorical or semantic properties in the ventral visual pathway might arise from the image-based representations found in early visual regions (Op de Beeck et al., 2008).

In this article, we describe two recent studies in which we have asked whether more basic principles could underpin category-selective patterns of response in the ventral visual pathway (Rice, Watson, Hartley, & Andrews, 2014; Watson, Hartley, & Andrews, 2014). The approach is based on the representational similarity analysis developed by Kriegeskorte et al. (2008). We have used this approach to ask whether the dimensions that are encoded in spatial patterns of neural response can be explained by the image properties that are characteristic of different object categories. First, we measure the patterns of response to different categories of objects in the ventral visual pathway using fMRI and compare the similarity of these patterns of response (Figure 1A). Next, we measure the image properties from the same categories of objects and compare the similarity of the image properties (Figure 1B). Finally, we ask whether similarities in the patterns of fMRI response between different categories of objects could be predicted by corresponding similarities in the image properties. If the patterns of response in the ventral visual pathway are linked to the image properties of objects, then we would predict a correlation between the patterns of fMRI response and the low-level image properties.

Figure 1.

Schematic diagram of multivoxel pattern analysis and image analysis procedures. (A) A leave-one-participant-out method was used to measure patterns of response to different stimulus conditions. In this analysis, the pattern of response elicited by one participant is compared to the pattern generated by a group analysis of all remaining participants. This procedure is repeated for all combinations of stimulus conditions and participants. (B) The calculation of a GIST descriptor for an example image. A series of Gabor filters across eight orientations and four spatial frequencies is applied to the image. Each of the resulting 32 filtered images is then windowed along a 4 × 4 grid to give a final GIST descriptor of 512 values (right).

Low-level image properties of visual objects predict patterns of neural response across category-selective regions of the ventral visual pathway

In the first study (Rice et al., 2014), we probed the relationship between low-level properties of five categories of objects (bottle, chair, house, face, shoe) and the patterns they elicit in the ventral visual pathway. These image categories were selected because they have been used in previous studies (Haxby et al., 2001; Spiridon & Kanwisher, 2002). Figure 2A shows the group averaged patterns of response to each object category across the ventral visual pathway. We found distinct patterns of response to each object category. Responses were normalized such that values above the mean are shown in red and values below the mean are shown in blue. We then compared the similarity of these topographic patterns across participants using correlation-based multivoxel pattern analysis (Haxby et al., 2001; see Figure 1A). Figure 2B shows the correlation matrix for the fMRI analysis across all combinations of object category. This correlation matrix shows that the patterns elicited by the same category were more similar than the patterns elicited by different categories. Nevertheless, there was also significant variation in the within-category and between-category correlations.

Figure 2.

Low-level image properties predict patterns of neural response to different objects in the ventral visual pathway. (A) Topographic patterns of response to different object categories (left) in the ventral visual pathway. Red/yellow and blue/light blue represent positive and negative fMRI responses relative to the mean response across all objects. Correlation matrices showing the correlations in (B) fMRI response across the ventral visual pathway and (C) image properties of different object categories. (D) Scatter plot showing a strong positive correlation, r = 0.79, between the correlation matrices in (B) and (C), demonstrating that patterns of fMRI response are closely linked to low-level image properties.

To determine whether the variance in the patterns of neural response could be explained by differences in the image statistics of different object categories, we measured the low-level properties of all images in the fMRI experiment (see Figure 1B). The image statistics of each object were computed using the GIST image descriptor developed by Aude Oliva and Antonio Torralba (2001). Using the GIST descriptor, we calculated the average orientation energy at different spatial frequencies and spatial positions within each object category. We then determined the within- and between-category correlations in GIST values across images from different categories. Figure 2C shows higher correlations in the image properties of within-category comparisons relative to between-category comparisons. However, similar to the fMRI patterns, there was significant variation in the within-category and between-category correlations.

We then conducted a representational similarity analysis (Kriegeskorte et al., 2008), in which we compared the similarity matrix for the fMRI response with the corresponding similarity matrix in the image properties. We found a strong positive correlation between the similarity matrices in the ventral visual pathway, r = 0.79, t = 24.23, p < 0.001 (Figure 2D). Importantly, the correlation between low-level image properties and the neural pattern of response was still evident when the within-category comparisons were removed from the correlation, r = 0.53, t = 9.28, p < 0.001. This finding shows that low-level image properties can explain systematic variation in the patterns of fMRI response irrespective of category label.

Next, we asked whether this relationship between image properties and fMRI response varied across different anatomical regions within the ventral stream. To avoid any biases about the functional organization of the ventral stream, we used anatomical masks defined by the Harvard–Oxford Atlas (http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Atlases). We found a significant positive relationship between the patterns of response and the image properties in the following regions: temporal pole, r = 0.54, t = 2.94, p < 0.008; middle temporal gyrus–temporal occipital, r = 0.67, t = 5.55, p < 0.001; inferior temporal gyrus–posterior, r = 0.58, t = 3.09, p < 0.006); inferior temporal gyrus–temporal occipital, r = 0.63, t = 6.55, p < 0.001; lateral occipital, r = 0.66, t = 21.44, p < 0.001; parahippocampal gyrus–posterior, r = 0.61, t = 5.70, p < 0.001; lingual, r = 0.72, t = 9.64, p < 0.001; fusiform gyrus–posterior, r = 0.59, t = 9.10, p < 0.001; fusiform gyrus–temporal occipital, r = 0.66, t = 21.81, p < 0.001; and fusiform gyrus–occipital, r = 0.74, t = 15.33, p < 0.001. When the within-category comparisons were removed from the analysis, the following regions showed significant positive correlations: inferior temporal gyrus–temporal occipital, r = 0.64, t = 4.14, p < 0.001; lateral occipital, r = 0.57, t = 14.92, p < 0.001; lingual, r = 0.66, t = 4.01, p < 0.001; fusiform gyrus–temporal occipital, r = 0.35, t = 7.55, p < 0.01; and fusiform gyrus–occipital, r = 0.63, t = 7.81, p < 0.001.

The link between the patterns of fMRI response to objects and their image properties has important implications for understanding how the ventral visual cortex is organized. The strong, linear relationship between low-level image properties and the spatial patterns of neural response suggests that more basic principles could underpin category-selective patterns of response in the ventral visual pathway. These findings are consistent with previous studies that used principal component analysis (PCA) to show that neural responses to different object categories in inferior temporal cortex can be predicted by variance in the principal components of the images (Baldassi et al., 2013; O'Toole, Jiang, Abdi, & Haxby, 2005). However, this need not be counter to a categorical representation, given that a category typically contains objects that are visually similar. The key finding from this study is that the correlation between the pattern of neural response and the low-level properties was still evident when the within-category correlations were removed from the analysis. If the organization of the ventral visual cortex is solely dependent on categorical principles, then the linear relationship between neural and image properties should not extend to between-category correlations when the within-category correlations are removed from the analysis.

These results are consistent with studies using fMR-adaptation to investigate selectivity to different object categories within the ventral stream (Avidan, Hasson, Hendler, Zohary, & Malach, 2002; Ewbank, Schluppeck, & Andrews, 2005; Grill-Spector et al., 1999). For example, it has been shown that adaptation to different object categories is not restricted to regions that respond maximally to those objects (Ewbank et al., 2005). Rather, the pattern of adaptation to different object categories shows distinct by overlapping patterns. Our interpretation of these findings is that the pattern of adaptation reflects the activation of an image-based representation in the ventral visual pathway.

It is not clear why previous studies have not found a link between low-level image properties and patterns of response in the ventral stream (cf. Kriegeskorte et al., 2008; Naselaris et al., 2009). This may reflect methodological differences between studies. In our study, we used a blocked fMRI design that measures the average pattern of response for a particular category. Similarly, measurements of image similarity involved comparing each image with the average image properties across all images in that category. It is possible that these differences provide a more generalizable estimate of both the spatial pattern of fMRI response and the image properties of different object categories, thus revealing the link between them.

Low-level image properties predict patterns of neural response within a category-selective region

In a second study (Watson et al., 2014), we asked whether image properties could also explain variation in the patterns of response to subordinate levels of one object category (scenes). Neuroimaging studies have found a number of regions of the human brain that respond selectively to visual scenes. The parahippocampal place area is a region in the parahippocampal gyrus that displays preferential activity to images of scenes compared to images of objects and faces (Aguirre & D'Esposito, 1997; Epstein & Kanwisher, 1998). Other place-selective regions include the retrosplenial complex, located immediately superior to the parahippocampal place area, and the transverse occipital sulcus or occipital place area, on the lateral surface of the occipital lobe (Dilks, Julian, Paunov, & Kanwisher, 2013; Epstein, 2008). Studies using fMRI multivoxel pattern analysis have found distinct patterns of response in these regions to different types of scene (Walther et al., 2009, 2011). These patterns of neural response have also been shown to correlate with patterns of behavioral response, but not with the low-level image properties of the images (Walther et al., 2009). This suggests a dissociation between the perceptual and neural representation of scenes and their underlying image properties. However, other multivoxel pattern analysis studies have shown that variation in the pattern of response in scene-selective regions are fully explained not by categorical differences in scenes but rather by the spatial layout of the scene (Kravitz, Peng, & Baker, 2011; Park, Brady, Greene, & Oliva, 2011). To address this discrepancy, we used the representational similarity analysis approach to determine whether patterns of response in scene-selective regions could be explained by variance in image properties.

First, we defined scene-selective regions by comparing the response to scenes relative to scrambled scenes. We then measured the patterns of response to city, indoor, and natural scenes within this scene-selective region of visual cortex. These subcategories of scenes were chosen because they are similar to those used in previous studies (Kravitz et al., 2011; Park et al., 2011; Walther et al., 2009, 2011). Figure 3A shows the normalized group response to city, indoor, and natural categories across this scene-selective region. Each category of scene had a distinct pattern of response, which was similar in appearance across the two cerebral hemispheres. We then compared the similarity of these topographic patterns across participants using correlation-based multivoxel pattern analysis (see Figure 1A). Figure 3B shows the correlation matrix for the fMRI analysis across all combinations of scene type. This correlation matrix shows that the patterns elicited by the same category were more similar than the patterns elicited by different categories. Nevertheless, there was also significant variation in both the within-category and between-category correlations.

Figure 3.

Low-level image properties predict patterns of neural response to different visual scenes in scene-selective regions. (A) Topographic patterns of response to city, indoor, and natural conditions on lateral and ventromedial surfaces of the brain. Red/yellow and blue/light blue represent positive and negative fMRI responses relative to the mean response across all objects. Correlation matrices showing the correlations in (B) fMRI response and (C) image properties of different scene categories. (D) Scatter plot showing a strong positive correlation, r = 0.91, between the correlation matrices in (B) and (C), demonstrating that patterns of fMRI response are closely linked to low-level image properties.

To determine whether the variance in the patterns of neural response could be explained by differences in the image statistics of different object categories, we measured the low-level properties of all images in the fMRI experiment (see Figure 1B) using the GIST descriptor. Figure 3C shows that there were higher correlations in the image properties for within-category compared to between-category images. However, similar to the fMRI patterns, there was significant variation in the pattern of the within-category and between-category correlations.

We then compared the within-category and between-category correlations for the fMRI response with the corresponding correlations in the image properties. We found a strong positive correlation across scene-selective regions, r = 0.91, β = 0.57, p < 0.001 (Figure 3D). A similar pattern of results was evident when the region of interest was subdivided into scene-selective regions—parahippocampal place area: r = 0.64, β = 0.42, p < 0.001; retrosplenial complex: r = 0.55, β = 0.17, p = 0.065; transverse occipital sulcus: r = 0.78, β = 0.26, p = 0.004. This suggests that the spatial patterns of response to different categories of scenes are linked to low-level image properties that are characteristic of those scenes.

Our findings show how the spatial properties of a scene could be linked to the patterns of response. However, these results contrast with those of a study by Walther et al. (2009), who found no significant correlation between neural responses and image similarity. One reason for this discrepancy could be the way that the image properties were measured. Those researchers used a different measure of image similarity, based on correlating pixel values across images. However, such a method is unlikely to provide a neurologically plausible measure of image similarity. In contrast, the GIST descriptor used in our main analysis was devised to more accurately reflect statistics encoded by the human visual system and to capture the critical spatial variables used to distinguish scene categories (Oliva & Torralba, 2001). The difference in the results may reflect the differences between these measurements of image properties.

A framework for explaining the topographic organization of the ventral visual pathway

The findings from these studies provide a new framework in which to consider the topographic organization of high-level visual cortex. Previous attempts to characterize the topographic properties of visual cortex beyond the early stages of visual processing have needed to include categorical or semantic information about the images (Connolly et al., 2012; Kriegeskorte et al., 2008; Naselaris et al., 2009). However, a fundamental problem in the endeavor to find a continuous representation across the cortical surface is that categories and semantics are discontinuous variables. The key result from our studies is that within these higher level regions, patterns of activity are parametrically related to statistical properties of the images. The low-level image properties of the GIST descriptor reflect variation in spatial position, orientation, and spatial frequency across the image. With this lower level framework of stimulus representation, it is more straightforward to determine how a continuous map could emerge (Grill-Spector & Weiner, 2014; Op de Beeck et al., 2008).

How do we explain the strong category selectivity found in the ventral visual pathway (Kanwisher, 2010)? We suggest that the appearance of discrete regional selectivity may emerge from the characteristic combinations of low-level image properties that co-occur in natural stimuli. Rather than reflecting a specific categorical layout across the cortical surface, regional selectivity could be explained by an overlap of continuous topographic maps that are based on image properties. Because images from different object categories have distinct image properties, images from a particular object category would activate specific intersections of these maps, giving rise to a spatially selective pattern of response (Op de Beeck et al., 2008).

We illustrate this framework in Figure 4 to show how distinct patterns of activity could emerge in a topographical map encoding several different visual properties. In this schematic example, the map (Figure 4A) represents three such parameters: Stimuli with high levels of Parameter 1 (yellow) tend to activate the rightmost part of the map, while low levels (red) activate the leftmost part; high levels of Parameter 2 (cyan) activate the upper part of the map, low levels (green) the lower part. It is not possible to map a third stimulus dimension to the two-dimensional surface of the cortex, so the organization of Parameter 3 (blue/pink) must be arranged on a somewhat finer scale so that all combinations of Parameters 1–3 can be represented by some part of the map (this is analogous to the way that orientation tunings must be organized on a finer scale within early visual retinotopic maps). The distinct visual properties of stimuli drawn from different categories are represented in Figure 4B. For example, Category 2 is typified by stimuli that show relatively high levels of Parameters 1 and 3 but low levels of Parameter 2. Stimuli in Category 3 tend to show the opposite pattern. These categories thus activate very different parts of the map. Figure 4C shows the patterns of activity elicited, on average, by each stimulus category, with regions of the map showing greater-than-average activity represented in shades of red/yellow and regions showing less-than-average activity represented in shades of blue. Because of their different loadings on the parameters that are mapped within this region of cortex, each stimulus category produces a distinctive pattern of activity. Moreover, the relative similarity of the different patterns reflects the underlying similarity of their image properties.

Figure 4.

Framework for understanding how continuous maps could explain spatial patterns of response in the ventral visual pathway. (A) Three maps of the same square patch of cortex show the continuous organization of tunings to three stimulus parameters, each indicated using a distinct color scale—Parameter 1: red/yellow; Parameter 2: green/cyan; Parameter 3: blue/pink. (B) Stimuli drawn from different categories have characteristic loadings on each parameter. For each category (A–H), the three parameter values associated with each stimulus are represented by different symbols (circles for Parameter 1, squares for Parameter 2, and triangles for Parameter 3) whose colors correspond to those used to indicate local tunings in the cortical maps. (C) Patterns of activity elicited, on average, by each stimulus category, with regions of the map showing greater-than-average activity represented in shades of red/yellow and regions showing less-than-average activity represented in shades of blue. Because of their different loadings on the parameters that are mapped within this region of cortex, each stimulus category produces a distinctive pattern of activity. Moreover, the relative similarity of the different patterns reflects the underlying similarity of their image properties.

This framework is consistent with other work showing low-level biases in the magnitude of the responses of high-level visual regions. For example, spatial frequency (Rajimehr, Devaney, Bilenko, Young, & Tootell, 2011) and orientation (Nasr & Tootell, 2012) biases have been found in regions involved in processing scenes. Other studies have shown central visual field biases in regions associated with face and objects, but a peripheral visual field bias in regions associated with buildings (Levy, Hasson, Avidan, Hendler, & Malach, 2001). However, it has not been clear whether this reflects a modification of the underlying categorical organization based on the way natural object categories are viewed or whether spatial properties themselves represent a fundamental organizing principle in the ventral visual pathway (Kanwisher, 2001).

It is important to note that the representation of image properties in high-level visual areas need not be similar to the way information is represented in lower-level regions. It seems likely that there will be an overrepresentation of low-level image properties that are more commonly found in natural images (Kayaert, Biederman, & Vogels, 2003; Op de Beeck et al., 2001, 2008). It is also important to note that the low-level image description used in our studies may not account for all of the variance in the magnitude and patterns of response. Evidence from psychophysical (Loffler, 2015; Peirce, 2015), electrophysiological, and neuroimaging (Kourtzi & Welchman, 2015; Wilson & Wilkinson, 2015) studies have all revealed the presence of midlevel image representations. So it seems that models of image properties that incorporate these midlevel properties of objects are likely to provide a more complete account of cortical organization in the ventral visual pathway. Indeed, a number of fMRI studies have shown a link between patterns of fMRI response in object-selective regions and the shape of objects (Drucker & Aguirre, 2009; Haushofer, Livingston, & Kanwisher, 2008; Op de Beeck et al., 2008). Consistent with these findings, Wilkinson et al. (2000) have shown that face-selective regions re more selective for concentric compared to radial or oriented gratings. Presumably, this reflects the predominance of concentric shape information in faces.

Conclusions

In summary, previous neuroimaging studies have revealed strong selectivity for object categories, such as faces, in the human visual system. However, it has never been clear whether this regional selectivity is driven solely by tunings to discrete object categories or whether it reflects tunings for continuous low-level features that are common to images from a particular category. Here, we show a clear link between patterns of response in higher level visual cortex and the image statistics characteristic of each category that cannot be explained solely in terms of discrete categorical organization. Further investigation will be necessary to determine whether these findings generalize beyond visual cortex to other sensory domains (Giordano, McAdams, Zatorre, Kriegeskorte, & Belin, 2013).