Multiscale Pattern Analysis of Orientation-Selective Activity in the Primary Visual Cortex

Abstract

Although orientation columns are less than a millimeter in width, recent neuroimaging studies indicate that viewed orientations can be decoded from cortical activity patterns sampled at relatively coarse resolutions of several millimeters. One proposal is that these differential signals arise from random spatial irregularities in the columnar map. However, direct support for this hypothesis has yet to be obtained. Here, we used high-field, high-resolution functional magnetic resonance imaging (fMRI) and multivariate pattern analysis to determine the spatial scales at which orientation-selective information can be found in the primary visual cortex (V1) of cats and humans. We applied a multiscale pattern analysis approach in which fine- and coarse-scale signals were first removed by ideal spatial lowpass and highpass filters, and the residual activity patterns then analyzed by linear classifiers. Cat visual cortex, imaged at 0.3125 mm resolution, showed a strong orientation signal at the scale of individual columns. Nonetheless, reliable orientation bias could still be found at spatial scales of several millimeters. In the human visual cortex, imaged at 1 mm resolution, a majority of orientation information was found on scales of millimeters, with small contributions from global spatial biases exceeding ∼1 cm. Our high-resolution imaging results demonstrate a reliable millimeters-scale orientation signal, likely emerging from irregular spatial arrangements of orientation columns and their supporting vasculature. fMRI pattern analysis methods are thus likely to be sensitive to signals originating from other irregular columnar structures elsewhere in the brain.

Introduction

There is growing interest in the ability of multivariate pattern analysis methods to extract reliable signals from distributed patterns of brain activity (Haxby et al., 2001; Haynes and Rees, 2005; Kamitani and Tong, 2005; Kay et al., 2008; Kriegeskorte et al., 2008), especially regarding the possibility of detecting signals originating from cortical columns. In the primary visual cortex of the macaque monkey, the full range of orientation preferences is represented over a distance of less than a millimeter (Hubel et al., 1978). However, recent human neuroimaging studies show that reliable orientation information is still found even in comparatively low-resolution (3 mm) functional images (Haynes and Rees, 2005; Kamitani and Tong, 2005).

The presence of a reliable orientation signal in these functional magnetic resonance imaging (fMRI) activity patterns may therefore appear to pose something of a puzzle. In the original study by Kamitani and Tong (2005), the authors proposed that random variability in the spatial distribution of orientation columns or their supporting vasculature could lead to weak, idiosyncratic biases in individual voxels. In the frequency domain, such variability in the columnar map would be expected to manifest as residual power across a range of spatial frequencies at and below the fundamental alternation frequency of the cortical columns. Such weak, low-frequency bias signals are indeed observed in optical imaging maps of orientation columns in the macaque monkey (Obermayer et al., 1992; Obermayer and Blasdel, 1993; Erwin et al., 1995).

However, others have argued that the signal detected in fMRI decoding studies might instead primarily reflect very large-scale biases in the V1 orientation map (Sasaki et al., 2006; Op de Beeck, 2009), such as a stronger response to cardinal than oblique orientations (Furmanski and Engel, 2000; Serences et al., 2009) or a retinotopically organized preference for radial orientations (Sasaki et al., 2006, Leventhal, 1983). In contrast to the columnar irregularities model, these proposals imply that activity patterns on scales of millimeters, substantially finer than these more global biases, should carry little or no reliable information about the viewed orientation.

Here, we used high-field, high-resolution fMRI and pattern classification methods to determine the spatial scales at which orientation information can be found in the visual cortex of cats and humans. We used a multiscale pattern analysis approach, in which cortical activity patterns were first spatially filtered to remove image components across a range of spatial scales, and then analyzed using multivariate pattern classification to predict the viewed orientation. A demonstration of reliable orientation decoding for filtered activity patterns on spatial scales of millimeters would support the columnar irregularities model, while an orientation signal primarily on very large spatial scales (∼1 cm and up) would indicate a strong role for global orientation biases.

Materials and Methods

These experiments were approved by the Institutional Animal Care and Use Committee of the University of Pittsburgh (cat data) and the Vanderbilt University Institutional Review Board (human data). The experimental methods used in the cat studies have been fully described in a previous publication (Moon et al., 2007). Detailed descriptions of experimental methods can be found online as supplemental material.

fMRI study of cat visual cortex.

Two anesthetized and paralyzed cats were presented with high-contrast, full-field drifting square-wave gratings displayed at one of 8 equally spaced orientations. Gratings were shown in 10 s blocks, with orientations presented sequentially in counterclockwise order. This continuous cyclic presentation causes orientation-nonspecific responses to saturate, resulting in improved sensitivity relative to blocked stimulation alternating with rest (Fukuda et al., 2006, Moon et al., 2007). Animals were scanned in a 9.4 Tesla Varian MR system using a custom-built surface coil over the primary visual cortex. Functional images were acquired in a single 1 mm-thick slice using a 2D gradient echo (GE) echo-planar imaging (EPI) sequence, with an inplane resolution of 0.3125 × 0.3125 mm and a temporal resolution of 0.5 s (Moon et al., 2007).

fMRI study of human visual cortex.

Four subjects discriminated rapidly presented letters at fixation while oriented square-wave gratings (0°, 45°, 90°, or 135°) were displayed in the left and right visual fields. Orientations were determined independently for each visual field to minimize any possible effects of residual eye movements on the subsequent classification analysis, as no singular pattern of eye movements could lead to reliable decoding of independent orientations in both hemifields.

MR images were acquired on a Philips Achieva 7-Tesla MRI scanner at the Vanderbilt University Institute of Imaging Science (VUIIS). A T2*-weighted 3D fast field echo (FFE) sequence was used for functional imaging to minimize susceptibility-induced geometric distortion, which can be severe with conventional single-shot EPI sequences at high field strengths. Images were acquired with an isotropic spatial resolution of 1 mm³ and a temporal resolution of 4.5 s.

Data analysis.

The fMRI data underwent preprocessing (global trend removal for the cat data and motion correction for the human data; see supplemental Methods, available at www.jneurosci.org), and were then spatially filtered into distinct highpass and lowpass components. Three types of spatial filtering were used: ideal volumetric filtering, 3D Gaussian smoothing, and an iterative surface-based smoothing method (Hagler et al., 2006). Results in the main text refer to ideal filtering, except where otherwise noted. For ease of comparison, all filter cutoff frequencies are expressed in terms of the full width at half maximum (FWHM) of the corresponding Gaussian smoothing kernel (see supplemental Methods, available at www.jneurosci.org). Complementary highpass filtered images were generated for each lowpass filter by subtracting the lowpassed data from the original images.

For pattern classification analysis, response amplitudes for each stimulus block were estimated by a general linear model (GLM). Linear support vector machines (SVMs) were then trained to predict the viewed orientation based on block amplitude data from all but one experimental run, with the remaining run held out as an independent test set. This iterative leave-one-run-out cross-validation procedure was repeated across all runs. For the human data, activity patterns in left and right V1 were used to train separate SVMs, with each pattern labeled according to the orientation presented in the contralateral visual field.

Results

Cat data

Individual orientation columns were clearly evident in very high-resolution fMR images (0.3125 × 0.3125 mm in-plane) of cat visual cortex (Moon et al., 2007). We applied varying levels of spatial filtering to these images, separating the original functional data into distinct fine- and coarse-scale components by complementary sets of ideal spatial highpass and lowpass filters (Fig. 1A). The orientation columns were obscured by levels of lowpass filtering exceeding the 1.2–1.4 mm spacing of iso-orientation domains in cat visual cortex (Löwel et al., 1987). However, a significant orientation bias was still found in most voxels even after several millimeters of lowpass filtering, indicating the persistence of an orientation signal at spatial scales considerably larger than the average spacing between columns.

Figure 1.

Orientation information at multiple spatial scales in cat visual cortex. A, Orientation-selective activity after ideal highpass and lowpass filtering; colors indicate voxels with statistically reliable orientation preference (p < 0.05, Bonferroni corrected). B, Orientation classification accuracy plotted as a function of filter size for highpassed (red triangles) and lowpassed (blue circles) data from each of two cats. Shading indicates 95% binomial confidence intervals, while the dashed line shows chance-level (12.5%) performance.

We assessed the reliability of the orientation signal present in these images across different spatial scales by using linear classifiers to predict which of 8 orientations (presented in counterclockwise order) was shown during each 10 s block of stimulation. The original unfiltered images contained considerable orientation information, as indicated by classification accuracies of 71% and 64% (chance level 12.5%) for each of two cats (Fig. 1B, leftmost data points). Accuracy dropped steadily with increasing amounts of lowpass filtering, indicating a gradual loss of orientation information. Similar results were obtained using Gaussian smoothing (supplemental Fig. 1A, available at www.jneurosci.org as supplemental material). Critically, performance remained well above chance level even with several millimeters of lowpass filtering, confirming the existence of reliable orientation information at spatial scales much larger than the underlying columns. Performance did not fall to near-chance levels until the filter size approached the ∼1 cm width of cat visual cortex.

Analysis of the complementary highpassed images indicated the reliability of fine-scale orientation-selective activity patterns after coarse-scale components were removed. Classification accuracy rapidly improved as additional fine-scale components were included (Fig. 1B, red curves), reaching levels similar to that of the original images once scales in the range of 1.2–1.4 mm, corresponding to the average spacing between cat orientation columns, were included. The addition of still coarser scales did not significantly improve classification accuracy beyond the level achieved at these columnar scales, suggesting that the information available on scales of several millimeters is largely redundant with that present at columnar levels. Together, the lowpassed and highpassed orientation decoding data are consistent with an orientation-selective signal that peaks at the alternation frequency of the cortical columns, but includes a coarse-scale, broad-spectrum bias component extending to scales of millimeters.

Human data

We applied the same multiscale pattern analysis to human fMRI data acquired at 1 mm³ resolution, to determine whether a millimeters-scale orientation signal would also be found in the human visual cortex. Though individual columns were not expected to be resolvable in these functional images, low spatial frequency biases similar to those observed in the cat data could still generate a reliable orientation signal at this spatial resolution.

Four subjects viewed gratings of independent orientation in the left and right visual fields while performing a letter discrimination task at fixation (Fig. 2A). We plotted the preferred orientation of every visually responsive voxel on an inflated representation of the subjects' cortical hemispheres (Fig. 2B). Visual inspection of the orientation preference map suggested the presence of both fine-scale and larger-scale patterns, though no statistical thresholding was used for these plots. Activation patterns were separated into distinct fine- and coarse-scale images by a set of ideal spatial filters (Fig. 2C), then linear classifiers were used to predict the orientation presented in the contralateral hemifield from the fine- and coarse-scale activity in right and left V1.

Figure 2.

Orientation information at multiple spatial scales in human visual cortex. A, Human subjects viewed 18 s blocks of independent oriented gratings in the right and left visual fields. B, Unthresholded orientation preference of individual voxels rendered on the inflated cortical surface of the right hemisphere of a representative subject, with all visually responsive (p < 0.01) voxels shown. Inset shows the folded cortical surface from the same viewpoint. C, Fine- and coarse-scale activity patterns revealed by ideal volumetric highpass and lowpass filtering. D, Orientation classification accuracy plotted as a function of filter size for highpassed (red triangles) and lowpassed (blue circles) data. Leftmost blue circle, classification of original data; rightmost blue circle, classification of mean activity in contralateral V1. Error bars indicate ±1 SEM based on N = 4 subjects. Dashed line indicates chance-level (25%) performance.

Spatial lowpass filtering led to a progressive loss of orientation information (Fig. 2D), with 1–2 cm of volumetric filtering leading to chance-level classification accuracy in individual subjects (supplemental Fig. 2, available at www.jneurosci.org as supplemental material). At the group level, accuracy remained greater than chance with up to 12 mm of ideal lowpass filtering (one-tailed T(3) = 2.83, p = 0.03). These results suggest that considerable information exists at the millimeters-scale, with also some information available at large spatial scales exceeding 1 cm.

Of greater interest, we found substantial orientation information at relatively fine scales on the order of millimeters (Fig. 2D, red curve) after large-scale biases were removed by spatial highpass filtering. Classification accuracy was significantly greater than chance at a group level when patterns on scales of only 2 mm or less were considered (T(3) = 3.00, p = 0.03), and rapidly improved to levels comparable to that of the original, unfiltered images once patterns on scales finer than ∼1 cm were included. These results indicate that classification methods are sensitive to millimeters-scale orientation biases, and do not require the presence of large-scale biases (>1 cm) for reliable decoding.

Similar trends were observed for Gaussian filtering applied to the 3D volume and an iterative cortical surface-based smoothing algorithm (Hagler et al., 2006) with effects similar to Gaussian filtering (supplemental Fig. 1B, available at www.jneurosci.org as supplemental material). In both cases, performance steadily declined with greater lowpass or highpass filtering, though at a somewhat more gradual rate than when using ideal filters. Gaussian filters attenuate but do not fully remove undesired spatial frequency components in the Fourier domain, which remain available to the classifier, albeit with reduced amplitude (supplemental Methods, available at www.jneurosci.org). These differences are most evident in the highpassed data, which remains well above chance with Gaussian filtering, most likely due to the presence of coarse-scale signals not effectively removed by the highpass filter. The results of surface-based filtering were similar to those of the 3D Gaussian filter, but with a still more modest decline in classification accuracy with greater filtering. Surface-based smoothing does not suffer from the partial voluming effects of 3D filters, which will average gray matter voxels together with white matter and space outside the brain, presumably contributing only additional noise. Additionally, due to cortical folding, the spatial scale of cortical activation patterns is likely substantially larger when measured along the 2D cortical surface rather than in the 3D imaging volume.

We performed additional univariate analyses to examine whether preferences for cardinal versus oblique orientations or radial bias might account for the residual orientation information found at large spatial scales. Surprisingly, average responses in V1 were significantly greater for oblique than cardinal orientations (Fig. 3) in a majority of hemispheres (supplemental Fig. 3, available at www.jneurosci.org as supplemental material), in contrast to previous reports (Furmanski and Engel, 2000). Regions preferring oblique orientations in V1 were very pronounced after 12 mm of lowpass filtering, even though orientation classification accuracy was only marginally above chance at this level. Highpass filtering largely removed any evidence of an oblique preference (supplemental Fig. 3, available at www.jneurosci.org as supplemental material). We tested for radial bias by comparing responses to 45° and 135° orientations (Sasaki et al., 2006). Although evidence of radial bias was difficult to see in the original data, lowpass filtering helped reveal activity consistent with a radial bias in many hemispheres. As with the oblique preference, radial bias effects were largely removed by highpass filtering (supplemental Fig. 4, available at www.jneurosci.org as supplemental material). These results indicate that orientation signals associated with the oblique effect and radial bias are most prominent at large spatial scales.

Figure 3.

Oblique effect. A, Average signal change across all visually responsive voxels in V1 shows higher amplitude activation to oblique orientations rather than to cardinal orientations. Error bars indicate ±1 SEM for 4 subjects. B, Activation maps for the contrast of cardinal versus oblique orientations show greater responses to oblique orientations predominantly on spatial scales >12 mm. The preferential oblique response is much reduced in the complementary highpassed images. Statistical maps are thresholded at a level of p < 0.05 (uncorrected), for sensitivity to weak effects.

Discussion

We used high-field, high-resolution fMRI and multiscale pattern analysis to clarify the origins of the orientation-selective signal previously found at conventional fMRI resolutions. Our results reveal that the majority of orientation information in high-resolution fMRI activity patterns can be found at spatial scales ranging from the size of individual columns to about a centimeter. Submillimeter-resolution data from the cat visual cortex revealed a broad-spectrum orientation signal that appeared strongest at the scale of cortical columns, but extended to much lower spatial frequencies, corresponding to scales of several millimeters. A strong millimeters-scale (∼2–10 mm) orientation signal was also found in the human visual cortex. This was indicated by a progressive loss of orientation information with spatial lowpass filtering over this range, and by the highly reliable orientation signal found in highpass activity patterns that preserved information at these scales. Together, our results demonstrate the presence of a strong millimeters-scale orientation signal, consistent with an origin in spatial irregularities of cortical columns and their supporting vasculature.

These findings address an ongoing controversy regarding whether fMRI decoding of orientation primarily relies on signals originating from columnar structures or from large-scale global biases (Op de Beeck, 2009; Kamitani and Sawahata, 2009). We found that large-scale biases (>1 cm) in the human visual cortex provided only a small amount of orientation information. Plots of orientation-selective activity on the cortical surface did reveal some evidence of radial bias, consistent with previous reports (Sasaki et al., 2006), but these effects could only be seen in our data at liberal statistical thresholds after considerable lowpass filtering. Although radial bias can provide some information for orientation decoding, both the present study and previous work have shown that the absence or removal of radial bias effects does not impair orientation decoding accuracy (Kamitani and Tong, 2005; Harrison and Tong, 2009; Mannion et al., 2009). We also investigated whether overall response amplitudes might differ across orientations, and were somewhat surprised to find greater responses to oblique than cardinal orientations, in contrast to previous reports (Furmanski and Engel, 2000; Serences et al., 2009). Although neurophysiological studies have found that neurons preferring vertical or horizontal orientations are overrepresented in regions subserving the central visual field (Mansfield, 1974; Orban and Kennedy, 1981), human behavioral studies have produced mixed results, including reports of better sensitivity to oblique than cardinal angles when viewing broadband or natural images (Essock et al., 2003; Hansen and Essock, 2004).

The parallel analysis of cat and human visual cortex revealed common evidence of millimeters-scale biases in the orientation signal across both species. Functional imaging of the anesthetized cat not only allowed for greater fMRI sensitivity, but also minimized the possibility that head motion would blur any fine-scale activity patterns, allowing us to detect orientation signals extending from the scale of cortical columns up to several millimeters. High field strength human imaging allowed us to resolve orientation signals on similar scales of millimeters, as well as to investigate larger-scale biases over the full expanse of human V1. Although we did not attempt to resolve individual orientation columns, one would expect that submillimeter-resolution imaging in humans would also reveal the strongest orientation signal at columnar scales. Despite the fact that the size of both cortical columns and the primary visual cortex itself varies widely between cats and primates, the presence of relatively low spatial frequency bias in columnar maps (Obermayer et al., 1992; Obermayer and Blasdel, 1993; Erwin et al., 1995) seems likely to be a general property of cortical organization. This would suggest that classification methods should be broadly applicable to signals originating in columnar structures elsewhere in the brain.

Our results can be contrasted with those of a recent commentary by Op de Beeck (2009). In this study, oriented gratings were presented to two subjects scanned at intermediate resolutions (2.1 × 2.1 × 2.2 mm, or 9.7 mm³). Surprisingly, Gaussian smoothing of up to 8 mm FWHM, the most investigated, did not impair orientation classification accuracy. These results led Op de Beeck (2009) to conclude that orientation decoding primarily relies on large-scale biases in V1. Our results do not support these claims, as we observed a substantial decrease in orientation classification accuracy with increasing spatial smoothing. We considered whether this discrepancy might reflect the higher spatial resolution used in our study (1 mm³ vs 9.7 mm³), given that a lowpass filter of fixed size (e.g., 8 mm FWHM) will remove much more information from high-resolution data. To investigate this possibility, we reanalyzed the orientation data of Kamitani and Tong (2005), collected at 3 mm isotropic resolution. Orientation decoding accuracy for the four subjects revealed a significant decline with 8 mm Gaussian smoothing (supplemental Fig. 5, available at www.jneurosci.org as supplemental material), again failing to replicate the null effect of spatial smoothing reported by Op de Beeck. Although the reason for this discrepancy across studies cannot be conclusively determined, a possibility is suggested by the data from one of our subjects who exhibited over 2.5 mm of translational head motion over the scanning session. This subject actually showed some benefit in decoding performance with modest spatial lowpass filtering of 2 mm, and no impairment in performance until filtering exceeded 8 mm (supplemental Fig. 2, available at www.jneurosci.org as supplemental material). Such head motion would likely disrupt the reliability of fine-scale pattern information, which could prove difficult or impossible to recover using post hoc motion correction.

Previous reports of successful orientation decoding at standard fMRI resolutions have sometimes been misinterpreted in terms of spatial aliasing, an image processing artifact in which columnar-scale signals could lead to power at lower spatial frequencies in the low-resolution functional images. Such proposals of spatial aliasing (or “hyperacuity”) are unlikely for several reasons. First, Fourier-based MRI is inherently band-limited in the phase-encode and frequency-encode dimensions, rendering similar “super-resolution” effects either impossible (Greenspan et al., 2002), or at best severely limited (Carmi et al., 2006; Mayer and Vrscay, 2007). Second, an aliased signal can change dramatically with small shifts in relative phase. Kamitani and Tong (2005) demonstrated successful orientation decoding across imaging sessions a month apart, implying that subvoxel precision in image alignment is not required for reliable orientation decoding. Their model simulations further showed that head motion would severely disrupt the reliability of voxel responses to a regular array of orientation columns, but that voxel preferences could remain robust to minor head motion when sampling irregular arrays of columns. Finally, in the present study individual cat orientation columns were clearly resolvable by the 0.3125 mm functional voxels, eliminating the possibility that these columnar signals could be aliased into lower spatial frequencies. We observed reliable orientation information after the fine-scale columnar signals were removed by lowpass filtering. This indicates that orientation biases truly exist at the millimeters scale, and cannot be attributed to low-frequency spatial aliasing artifacts.

A more plausible alternative is that the source of the relatively coarse-scale orientation signal could lie in large draining veins (Gardner, 2009), which might effectively transpose high spatial frequency columnar signals into lower frequency veinous signals. Shmuel and colleagues have recently addressed this issue using high-resolution GE-fMRI images of ocular dominance columns (Shmuel et al., 2009), finding that gray matter voxels as well as voxels identified as veins both convey information about the stimulated eye. These results suggest that veinous contributions to orientation decoding are unlikely to provide the exclusive source of the millimeters-scale orientation signal, though the relative contribution of blood vessels and gray matter in generating these signals remains an area for future research.

This study also has broad implications for understanding the potential of fMRI pattern analysis methods. Our findings suggest that fMRI pattern classification is likely to be sensitive to signals originating from other columnar structures, even at conventional scanning resolutions of several millimeters. The signals observed here on scales of millimeters likely arise as a consequence of random variability in the developmental formation of orientation columns (Swindale, 1982); such variability might well be expected during the formation of other columnar systems. If anything, V1 exhibits a more orderly arrangement of orientation columns than higher visual areas (Vanduffel et al., 2002). However, we consider it important to note that reliable decoding of millimeter-scale activity patterns should not be taken to imply the presence of columnar organization. Direct demonstration of columnar structure likely requires conventional mapping at resolutions substantially finer than the true width of a cortical column (Cheng et al., 2001; Fukuda et al., 2006; Moon et al., 2007; Yacoub et al., 2008). The present study indicates the challenges of determining the true spatial scale of submillimeter columnar structures with fMRI, given that reliable signals and activity patterns can still be found at spatial resolutions lower than that of individual columns.