The primary visual cortex fills in color

Abstract

One of the most important goals of visual processing is to reconstruct adequate representations of surfaces in a scene. It is thought that surface representation is produced mainly in the midlevel vision and that area V1 (the primary visual cortex) activity is solely due to feedback from the midlevel stage. Here, we measured functional MRI signals corresponding to “neon color spreading”: an illusory transparent surface with long-range color filling-in, one of the important mediums in reconstructing a surface. The experiment was conducted with careful controls of attention, which can send feedback signals from higher visual areas. Activity for filling-in was observed only in V1, whereas activity for illusory contours was observed in multiple visual areas. These results indicate that surface representation is produced by multiple rather than single processing.

The ability of the visual system to represent surfaces is fundamental to visual perception (1–3). Where is this surface representation produced in visual cortical areas? It has been suggested that the surface representation level resides at a midstage of visual processing, beyond the level of local filtering (2–7). However, contradicting empirical and theoretical reports have also been proposed (8–12). One reason for this controversy may be the tacit assumption that surface representation is accomplished by single processing rather than multiple processing. Surface representation could be a result of many different aspects of processing (13–15). It comprises various subcomponents such as contours, brightness, color, depth, and motion, each of which could work differently or interact with each other to engender perception of a surface in the brain (16–19). Another reason for the controversy may be that most studies have not controlled the effects of attention on a surface. Attention itself can activate multiple visual areas (20, 21). Thus, it is necessary to examine how subcomponents of a surface contribute to surface representation with attentional effects controlled.

One of the possible mediums to generate surface perception is featural filling-in. A famous illusion that induces filling-in is neon color spreading (22), in which a colored transparent disk is observed even though no transparent surface is physically present (Fig. 1). Neon color spreading is thought to be due to interactions between mechanisms for two surface subcomponents: long-range color filling-in and illusory contours. Neon color spreading and has been used extensively to understand the mechanisms of surface representation (2, 5, 12, 23–27). In addition to the apparent transparent surface quality in the color spreading (28), the luminance rules for occurrence of transparency (26) also govern the neon color spreading. Thus, it has been suggested that the mechanism for transparency is involved in neon color spreading (2, 12, 23, 25).

Fig. 1.

The displays and results of psychophysical sessions in experiment 1. (A) A test stimulus that induces an illusory transparent surface. An illusory blue transparent disk is perceived in the central area; the blue color appears to spread out of the blue inducing segments (26). (B) A control stimulus. The same blue segments rotated by 180° produce no illusory transparent surface. In both the test and control stimuli, the backgrounds are black, except for a small fixation point at the center of the stimulus. (C) The results of the first condition of psychophysical sessions. Graph shows the percentage of subjects perceiving a transparent surface as a function of the distance between the center of the display and the center of an inducing circle (center-to-center distance). The vertical bars represent SEs across the subjects. As indicated by the arrow, all of the subjects perceived a transparent disk at the same center-to-center distance (10.6°) as in the fMRI session. (D) The results of the second condition of psychophysical sessions. The chromaticities (x, y) of the white and blue parts of the stimuli and the chromaticity of the uniform circle in each of four subjects (a, b, c, and d) adjusted to match the filled-in color.

In particular, neon color spreading can be a strong tool when used with functional MRI (fMRI). Because neon color spreading induces a surface that is located physically in the same field as a control stimulus that does not induce surface perception, the activity difference between the neon color spreading (illusory transparency) figure and the control figure can be regarded as purely due to the surface reconstruction process in the brain.

In the present study, we used 3T fMRI that has a fine spatial resolution so that brain activity corresponding to both illusory contours and filling-in as surface subcomponents could be spatially dissociable if surface representation occurred in the retinotopic visual areas. To eliminate or decrease the attentional component of feedback signals, subjects performed an attentionally challenging task unrelated to the surface perception (29, 30).

We found that the activity in the color filled-in region was observed only in primary visual cortex area V1 when attention was controlled, whereas the activity in both illusory and real contours was not confined to V1. These results suggest that surface representation is not a result of single processing but of more complex multiple processing.

Experiment 1

Subjects. Six healthy subjects with normal or corrected-to-normal vision viewed visual stimuli in the magnetic resonance (MR) scanner. All subjects gave informed written consent. The study was approved by Massachusetts General Hospital Human Studies Protocol 2000p-001155.

Stimulus and Procedure. There were fMRI and psychophysics sessions.

fMRI procedure. The test and control stimuli are shown in Fig. 1 A and B, respectively. (Note that the colors and luminances in the printed figures are not necessarily identical to those used in the experiment). In the test stimulus, the distance between the center of the display and the center of each of the four small disks (8.4° in diameter) was 10.6°. In the control stimulus, each disk was rotated by 180° to show a similar stimulus without inducing a filled-in transparency. The luminance of each of the white, blue, and black parts of the stimuli was 62.74 cd/m², 32.44 cd/m², and 0.4 cd/m², respectively.

To control attentional effects (29, 30), we presented a small bar (0.3°) on the top of the fixation point and asked the subjects to perform an orientation indication task. In each trial, subjects were instructed to attend to a series of tiny vertical or horizontal bars presented intermittently near the fixation point throughout the test and control conditions. A random sequence of presentations of “|” or “—” was displayed for 200 ms each, and subjects had 350 ms to depress a button corresponding to each stimulus.

The subjects lay in an fMRI scanner and observed an image projected from outside the scanner onto a mirror inside the scanner. The test and control stimuli were presented alternately every 16 s, and each stimulus was followed by a blank screen with a fixation point for 16 s. The blank screen was necessary to minimize the afterimage of the inducing figure and the illusory transparent surface (31). Thus, each block consisted of one test stimulus, one control stimulus, and two blank intervals. Each scan consisted of four blocks, and each subject performed 8–14 scans. Many scans were collected to quantify relatively small but consistent signal changes related to filled-in transparency.

General fMRI procedures. Experimental details were similar to those described in ref. 32. Scans were acquired by using an Allegra 3T scanner (Siemens, Iselin, NJ). A custom-built, quadrature-based, semicylindrical surface coil was used to acquire high-sensitivity MR images, including occipital, parietal, and posterior temporal lobes bilaterally. Voxels were 3.1 mm² in-plane and 3 mm thick. Functional MR images were acquired by using gradient echo sequences (echo time = 30 ms) with 128 images in 25 contiguous slices oriented approximately orthogonal to the calcarine sulcus. Repetition time was 2 s for the main fMRI experiments, and each fMRI scan took 256 s. Subjects were run for 8–14 scans, and the signals across the scans were averaged. In additional scans, we used a whole head coil to check whether brain activation differed from that obtained with a surface coil, because the coil might be more sensitive to the brain region that is closer to the surface. We confirmed that brain activation was not dependent on the coil we used.

Retinotopy. In a separate session, retinotopic visual areas and borders were mapped by using phase-encoded stimuli and field sign analysis as described in ref. 33. We identified visual areas in V1, V2, V3/VP, V3A, V4v, and MT+. Because no significant relevant activity was observed in MT+ in this study, we excluded MT+ from further analysis.

Flattening the visual cortex. In a separate session, structural images of the whole brain were obtained with high resolution (1.0 × 1.0 × 1.3 mm³) to provide data for 3D brain reconstruction (34), which allowed us to generate an unfolded and flattened cortical surface for each subject. The software used is available at www.nmr.mgh.harvard.edu/freesurfer.

fMRI data analysis. The statistical maps were generated by using the Massachusetts General Hospital NMR Center fMRI processing stream (fs-fast; http://surfer.nmr.mgh.harvard.edu), based on selective averaging and deconvolution after motion correction (35). This method models the blood oxygen level-dependent (BOLD) signal as a linear combination of time-invariant hemodynamic responses imbedded in Gaussian noise, estimating the shape of the hemodynamic response by using a least-squares method on a voxel-by-voxel basis. t tests were conducted to compare activation amplitudes between conditions. The P values were projected onto the flattened activity maps.

Psychophysics. We also conducted psychophysical experiments to assure that illusory transparency (the neon color spreading) in the test stimulus was indeed observed while the subjects were performing the task in the fMRI sessions. There were two conditions. In the first condition, the experimental procedure was identical to that of the fMRI sessions, except for the following two aspects. First, the distance between the center of the display and the center of an inducing circle (center-to-center distance) in both the test and control stimuli was varied in six steps (8.7, 9.3, 10.6, 13, 17.9 and 26.8°). Each condition was repeated 30 times. The order of presentations of the conditions was randomized from subject to subject. Second, in each trial, immediately after a stimulus disappeared, the subjects were asked to report whether they perceived a blue disk in the center. In the second condition, the center-to-center distance in both the test and control stimuli was a constant 10.6°, which was the same distance used in the fMRI sessions. In each trial, a test or control stimulus was presented on the right or left side of the display while a uniform disk (8.5° in diameter) was presented on the opposite side. The distance between the centers of the displays was 35°. The subjects were instructed to match the color and luminance of the disk to those of the central area in the test or control stimulus. This matching was done by using the method of adjustment, that is, by pressing one key for increment or the other key for decrement for each amount of red, blue, and green. The subjects were clearly told that the central area in the test or control stimulus to be matched to the disk did not include any part of the four inducing circles. Twenty trials for each of the test and control stimuli were conducted for each subject (n = 4). The order of the presentations of the two stimuli was randomized from subject to subject.

Results and Discussion of Experiment 1. The results of the first condition of psychophysics (Fig. 1C) indicate that in the center-to-center distance used in the fMRI sessions (10.6°), illusory transparency was observed at 100% of probability by all of the subjects. The mean percentage correct of the task was 74.3%, which was very close to the result obtained in the fMRI session (75.0%). Fig. 1D shows the mean matched chromaticity of the uniform circle to that of an illusory transparent disk for each subject in the second condition of the psychophysical sessions together with the chromaticities of the white and blue areas in the stimuli. The mean matched luminance was 16.58 cd/m² ± 3.7 SE. The results indicate that all of the subjects perceived the illusory transparent disk as blue, although the color was less saturated and less bright than the blue color presented in the inducing circles.

The results of the fMRI sessions indicate that only in V1 were mean fMRI signals significantly larger in the region retinotopically corresponding to the illusory transparent surface in the test condition than in the control condition (Figs. 2 and 3). Both the activity in the cortical regions that retinotopically correspond to the blue sectors in the illusory transparency condition and the activity in the corresponding area in the control condition were excluded from the calculation of the results of all of the experiments. However, greater activity was found in all measured areas (V1, V2, V3/VP, and V4v) in the region corresponding to the illusory and real contours found in the test condition compared with the control condition (Fig. 3). The differences in fMRI signal amplitudes were relatively small, but they were statistically significant and consistent across subjects. Note that this amount of differential fMRI signals is comparable to the amount found in V1 in fMRI studies (36–38).

Fig. 2.

The results of the fMRI session in experiment 1. Only V1 was activated more in the illusory transparency condition than in the control condition (P < 0.01) in the results of the first experiment in which attention was controlled. Note that this figure shows fMRI signal changes between the (excitatory) test and control conditions, not compared with fixation-only conditions; MR activity in each of the visual cortical areas (V1, V2, V3/VP, V3A, and V4v) in the control condition was subtracted from that in the illusory transparency condition. The activity in each area was calculated based on the cortical region that retinotopically corresponds to the illusory transparent surface from the fovea to 10.6° in visual angle. The activity in the cortical regions that retinotopically correspond to blue sectors (filling-in inducers) was excluded from the calculation. The average fMRI signal levels during the fixation conditions corresponded to –1.6%. **, P < 0.01.

Fig. 3.

A representative fMRI activity map in experiment 1. (A) MR signals on the flattened format of the occipital cortex of a representative subject in the control condition subtracted from those in the neon color condition. White and dotted lines represent the cortical locations retinotopically corresponding to the real and illusory contours of the illusory transparent surface, respectively. Although the activity of the region retinotopically corresponding to color filling-in is only observed in V1, the activity of the region corresponding to the real and illusory contours is found in all of the measured visual areas. Asterisks mark the location of the fovea in each cortical area. (B) Schematic illustration of the neon color stimulus used in the main experiment. White arcs represent real contours, and dotted arcs show illusory contours.

As previously mentioned, the mean performance of the attention task was 75 ± 2% SE, indicating that the task was challenging (29, 30).

These results are not in accord with the previous studies, which indicated activation in areas including V2, V3/VP, V3A, and V4v (39–43). Why does this discrepancy occur?

One crucial difference is that other studies did not clearly separate activity for contours from activity for internal surface features such as color, brightness, and texture. Our results of activity for illusory contours are in accord with the previous findings (39). However, activity for filling-in was observed only in V1 in the present study.

Another difference is whether attention was well controlled. Whereas our first experiment controlled attention by giving the subjects an attentionally challenging task, previous studies instructed the subjects to view a surface stimulus with either no or weaker attentional control, which may have allowed the subjects to direct attention to the surface (41) and/or to induce stronger top-down activity to elaborate the surface representation.

Experiment 2

Methods. To clarify these points, we conducted a second experiment with six subjects that replicated the first experiment except that it had no attentionally demanding task. In each trial, the subjects were asked to maintain their gaze at the fixation point.

Results. In all of the measured areas (V1, V2, V3/VP, V3A, and V4v), the fMRI signals in the region corresponding to the filled-in color were significantly higher in the test condition than in the control (Fig. 4). The results of experiments 1 and 2 indicate that the activation of visual areas V2, V3/VP, V3A, and V4v in experiment 2 was due to attention modulation. Therefore, the results that only V1 activity was observed when attention was controlled in the first experiment indicate that V1 activity related to surface processing is not solely due to feedback from higher cortical stages or attention modulation, although contributions of the higher cortical stages and attention modulation to surface processing are not denied.

Fig. 4.

Results of experiment 2. Mean MR signals in each of the visual cortical areas in the control condition were subtracted from those in the neon color condition. Attention was not controlled in either condition. All of the measured visual areas, including V1, were more activated in the filled-in transparency condition than in the control condition (P < 0.001 for V1, V2, V3/VP, V3A, and V4v). No significant difference was found in the activity among the visual areas.

Experiment 3

Methods. Does the visual system respond to physically uniform surfaces in the same way as illusory transparent surfaces? To address this question, we examined how activity in the early visual areas changes in response to a stimulus that physically contains a uniform blue disk that matches the perceptual quality of the illusory transparent disk. Two subjects who had participated in the psychophysical experiments in experiment 1 served as subjects in the present experiment. In addition to the test and control stimuli used in experiment 1, a physically uniform disk whose size was the same as the illusory transparent disk was presented with the four inducing circles in the test stimulus (Fig. 5A). The luminance and chromaticity of the uniform disk were set to be the same as those adjusted by each subject in the second condition of the psychophysical experiment. The three stimuli (the illusory transparent disk stimulus, uniform disk stimulus, and control stimulus) were used with the same procedure as in experiment 1.

Fig. 5.

Stimuli and results in experiments 3 and 4. (A) Three stimuli used in experiment 3. In addition to the test and control stimuli used in experiment 1, a uniform blue disk was shown in the region in which the color filled-in transparency is observed. The luminance and chromaticity were the same as those of the filled-in transparent surface perceived by each subject. (B) Mean MR signal in the control condition subtracted from that in the uniform surface condition and that in the filled-in transparency condition in experiment 3. (C) Two stimuli used in experiment 4. (D) Mean MR signal in the control condition subtracted from that in the brightness filled-in transparency condition in experiment 4.

Results. As shown in Fig. 5B, only V1 showed significantly higher activity both with the neon color filled-in and uniform disk stimuli than with the control stimulus.

Experiment 4

Methods. In the aforementioned experiments, the activity for colored (blue) filling-in was examined. It has been found that brightness filling-in occurs with an achromatic version of the same stimulus (e.g., refs. 25 and 28). In the present experiment, we examined whether similar activity would be observed in brightness filling-in.

The experimental procedure was the same as that in experiment 1 except that the blue parts of the inducing circles in both test and control stimuli were replaced with gray (Fig. 5C). For brightness or color filling-in to occur, the luminance of the internal parts of the inducing circles in the test stimulus must be between the luminances of the background and the external parts of the disks (28). The luminance of the gray parts was 30.28 cd/m², which met the luminance condition for brightness filling-in.

Results. Fig. 5D shows that basically the same pattern of activity was found as in experiment 1.

General Discussion

A prevailing hypothesis is that surface representation is formed in midlevel stages. Although recent findings have shown that V1 in the monkey brain is activated for surface processing (9, 10, 40, 41), it has been suggested that this activation results solely from feedback from higher cortical stages. However, when we controlled for attention, which could give feedback signals from the higher-order visual areas, significant fMRI activity was observed only in the part of V1 that retinotopically corresponds to a color filled-in area of an illusory transparent surface. The results are at odds with the hypothesis that V1 activity for surface representation is solely from feedback from higher cortical stages. V1 may play a more important role in surface formation than previously thought.

Whereas only V1 was activated for the color filled-in area, the regions retinotopically corresponding to both the illusory and real contours bounding the color filling-in were found to activate V2, V3/VP, and V4v as well as V1 (Fig. 3). This result is consistent with previous human studies (39), although in animal studies it is disputed whether V1 is activated with illusory contours (41, 44, 45). A long-held view is that surface representation is produced by interactions between the long-range filling-in system and the contour system (12, 46). The present results show that contours and the filling-in of a surface feature are processed separately.

Why was the activity of visual areas higher than V1 observed for color filling-in when attention was not controlled but such activity was not obtained when attention was controlled? One possible explanation is that the higher-area activity was due to subjects directing their attention to a surface or figure that stands out from the homogeneous background (40). Another possibility is that attention control reduced overall top-down signals, including those for elaborating surface representation (20, 47–49). Whereas attending to the center of the target in the first experiment might have caused the visual system to distribute fewer resources to the process of forming the transparent surface, the visual system could have distributed attention to the entire display in the second experiment. The consequence might be that more resources would be available to the process of forming the transparent surface. In any case, the results of experiment 1 showed that while the subjects were conducting an attentionally challenging task, they reported the 100% occurrence of the color filling-in (Fig. 1C), and only V1 MR activity was observed (Fig. 2). These results suggest that V1 activity for surface representation is not solely from feedback from higher cortical stages.

How much can the present results for the color filled-in transparency be generalized? The results of experiment 4 suggest that the same or similar mechanisms are involved in the brightness and color filling-in for a transparent surface. However, the present finding of V1 being activated with filling-in for an illusory transparent surface may not be applied to all of the filling-in effects. Activity for filling-in across the blind spot was found to be in V1 of monkeys and humans (10, 50) and has been attributed to activity of V1 cells with large receptive fields encompassing the inducing contours across the blind spot. The color filling-in in the present study is spatially too extensive (≈10°) to be explained by activity of the cells with large receptive fields. Filling-in of visual phantom activates extrastriate areas as well as V1 in the human brain.§ Filling-in with the Craik–O'Brien edges did not activate V1 but did activate V3 in the human brain.∥ Thus, the mechanisms for visual phantom and the Craik–O'Brien effect may not be exactly the same as those for the color filling-in for transparency, although they are phenomenologically similar.

What is the underlying mechanism for brightness and color filling-in for transparency? The spatially global nature of the activity for the filling-in only in V1 suggests that long-range horizontal connections in V1 (53), which previously have been found only with contour signals and edges (53), may occur even with non-contour-based feature signals such as colors and brightness. The result of the activity with the uniform disk stimulus in experiment 3 is basically the same as that with color filled-in transparency in experiment 1 and also is in accord with the recent finding that V1 activity in response to a uniform surface is the highest in early visual areas (43). These results suggest that long-range horizontal connections are involved in both the color filling-in processing and uniform surface processing.

The present finding, though limited to brain activity corresponding to color or brightness filling-in and illusory contours of an illusory transparent surface, is consistent with other findings that showed V1 activity for uniform surface and brightness (9, 10, 54) in the sense that roles of V1 neurons are not restricted to be a simple orientation filter of lines (8). V1 seems to play a greater variety of roles than was once thought. Our results show that the processing of an internal surface feature such as color or brightness filling-in is not solely due to the midlevel stage and is not necessarily the same as processing of contours (12). However, this result does not indicate that V1 is solely responsible for the processing of surface features, either. First, the difference between experiments 1 and 2 raises the possibility that attention modulation plays a role in surface formation. Second, it has been reported that brightness, as well as color perception, can be drastically different depending on the context (13, 14). Thus, we conclude that surface representation formation cannot be explained by a single mechanism in one visual area in solely top-down or bottom-up processing. Careful examinations of activity for different components in different features are necessary in future research.