Responses to lightness variations in early human visual cortex

Summary

Lightness is the apparent reflectance of a surface, and depends not only on the actual luminance of the surface but also on the context in which the surface is viewed [1-10]. The cortical mechanisms of lightness processing are largely unknown, and the role of early cortical areas is still a matter of debate [11-17]. We studied the cortical responses to lightness variations in early stages of the human visual system using functional magnetic resonance imaging (fMRI), while observers were performing a demanding fixation task. The set of dynamically presented visual stimuli included the rectangular version of the classic Craik-O'Brien stimulus [3,18,19] and a variant that led to a weaker lightness effect, as well as a pattern with actual luminance variations. We found that the cortical activity in retinotopic areas, including V1, is correlated with context-dependent lightness variations.

Results

A rectangular version of the Craik-O'Brien (CO) stimulus [3,18,19] was used to investigate the cortical responses to context-dependent lightness variations (Figure 1, “Illusory”). In the “Illusory” CO stimulus, the left and right flanking regions appear to have different lightnesses due to a central region composed of a contrast border with oppositely signed luminance gradients on each side. We refer to this illusion as the lightness effect. The lightness effect largely diminishes but does not completely vanish if the stimulus is placed on a gray background (Figure 1, “Control”) [8,20]. The set of visual stimuli used in both behavioral and fMRI experiments included the “Illusory” stimulus, the “Control” stimulus, and a pattern with a real luminance difference between the flanks (Figure 1, “Real”). In the fMRI experiments, cortical activity was measured in regions corresponding to the flanks of dynamically presented versions of the stimuli. In the case of the “Illusory” and “Control” conditions only the central portion varied, the flanks remained constant in luminance during the dynamic display. Even though the flanks remained constant in luminance, their apparent lightness varied in the “Illusory” condition. This lightness effect was much weaker in the “Control” condition. In the “Real” condition the luminance of flanks alternated homogeneously.

Figure 1. Stimuli used in the behavioral and fMRI experiments.

Behavioral and fMRI responses to the lightness effect caused by the “Illusory” Craik-O'Brien (CO) stimulus were measure and compared with responses to the “Real” condition in which flanks physically differed. The response to the “Illusory” stimulus was also compared with that to the “Control” stimulus that caused a weak lightness effect. Luminance profile along a horizontal cross-section is shown below each stimulus.

Recent studies on human observers, as well as studies on anesthetized animals, have produced mixed results regarding the role of early cortical areas in lightness processing. Several studies provided evidence in favor of lightness responses in these areas [11-15], whereas others found no such evidence [16,17]. In this study we address three specific questions: 1) Does the cortical activity correlate with context-dependent lightness variations in regions where locally the physical stimulus remains constant? 2) If so, how does it compare with activity in response to actual physical variations? 3) Do the cortical responses occur without attention directed to the lightness effect?

Behavioral measurement of the lightness effect

In order to compare the cortical activity to context-dependent lightness variations with that to actual physical variations, we first determined the contrast of the “Real” stimulus that is perceptually equivalent to the “Illusory” stimulus for each observer, using an adaptive (1-up 1-down staircase) two-interval forced choice procedure and maximum likelihood techniques [21] (Figure 2). We repeated the same procedure for the “Control” condition. For the “Illusory” stimulus, three levels of CO border contrast were investigated: 0.3, 0.6 and 0.9 (Contrast=(L_max-L_min)/L_mean, where L is luminance). For the “Control” stimulus only the 0.9 contrast level was tested. We found that the effect gets stronger with increasing contrast, and that the “Control” stimulus leads to a smaller lightness effect consistent with previous literature [20] (Table 1). Two experienced observers participated in a dynamic version of the experiment, in which the stimuli were dynamically presented with the same temporal characteristics as in the fMRI experiment (see Supplemental Methods). We found that the dynamic version of the “Illusory” stimulus led to a slightly smaller lightness effect for those two observers (Table 1).

Figure 2. Behavioral measurement of the lightness effect.

The contrast of the subjectively equivalent “Real” stimulus was determined in an adaptive two-interval forced-choice experiment (2IFC). Observers were asked to indicate the interval in which the luminance difference between the flanks, within the square frames, was larger while maintaining fixation at a central mark. The same measurement method was repeated for the “Control” stimulus.

Table 1. Behavioral results.

Contrast of the subjectively equivalent “Real” stimulus reported for all observers (standard errors of the means are reported in parentheses). BC: contrast of the central border in “Illusory” and “Control” stimuli. Results from the dynamic version of the experiment are shown for the two observers who participated in that experiment (KD and HB).

Observer	“Illusory”			“Control”
	BC = 0.3	BC = 0.6	BC = 0.9	BC = 0.9
KD - Static Dynamic	0.20 (0.004 ) —	0.27 (0.006 ) —	0.32 (0.006 ) 0.23 (0.008)	0.15 (0.017) 0.14 (0.006)
FF	0.20 (0.003 )	0.25 (0.011)	0.24 (0.005)	0.07 (0.020)
HB - Static Dynamic	0.21 (0.009 ) —	0.33 (0.008) —	0.37 (0.006) 0.21 (0.006)	0.13 (0.009) 0.13 (0.001)
RWS	0.16 (0.005)	0.35 (0.006)	0.39 (0.015)	0.12 (0.008)
ST	0.21 (0.008)	0.29 (0.003)	0.40 (0.010)	0.10 (0.003)

Cortical responses to context-dependent lightness variations

In the fMRI experiment, the “Illusory” and “Control” stimuli with 0.9 contrast level were used, since that level produced the strongest lightness effect for all observers. The contrast of the “Real” stimulus was set for each observer to perceptually match the “Illusory” stimulus based on his or her behavioral data, as described above. Each stimulus was first presented statically for 18 seconds followed by the same pattern but with 12 second square wave modulated counter-phase flicker (0.66 Hz, Fig. 3A). Throughout the entire scan, observers were asked to perform a demanding fixation task that required them to detect a target letter (“X”) among distracters (“Z”, “L”, “Q”, “J”) during rapidly changing presentation of these letters (120 to 130 ms for each letter) (Figure 3A). The success rate across observers was 79% and mean reaction time was 533 ms.

Figure 3. Design of the fMRI experiment.

Cortical responses were measured to dynamically presented versions of the stimuli shown in Figure 1. The contrast of the “Real” stimulus was adjusted to match the lightness effect for each observer based on his or her behavioral data.

(A) Each stimulus was first presented statically for 18 seconds followed by a square wave modulated counterphase flicker for 12 seconds. During the entire scan observers performed a fixation task that required them to detect a target letter (X) among distracters during rapidly changing presentation of these letters.

(B) Region of interest (ROI) definitions. Counterphase flickering checker patterns were used to localize ROIs. The left panel shows dimensional parameters of the “Illusory” stimulus and the positions of the ROI localizers. The right panel shows the areas of the visual cortex activated by the localizers for one observer (p(corr.)<10⁻⁴). Red lower ROIs, blue upper ROIs. Because the cortical areas corresponding to upper ROIs were much closer to the vertical meridian and therefore to the contrast border in the stimuli, only MR data in lower ROI pairs are presented here (See Supplemental Figure 1 for results from both ROI pairs).

Regions of interest (ROIs) were defined using square wave modulated contrast reversing (8 Hz) black and white checks covering an area of 2 by 2 degrees in observers' periphery (Figure 3B). ROIs were identified by selecting the voxels that respond most strongly to corresponding checker stimulus using a general linear model (p(corr.)<10⁻⁴). Because the voxels corresponding to upper ROIs were much closer to the boundaries between V1 and V2, and therefore to the central region of the CO stimuli, we only present the analysis of the signal within the lower ROIs (see Supplemental Data Figure 1 for the results in both ROI pairs). Using images acquired in a separate scanning session, retinotopic areas were identified using techniques developed by Engel et al. [22] and Sereno et al. [23].

For these ROIs, the MR signal was first normalized by the average of the last two time points of all static presentation blocks in a scan, and then event-related averaged according to the type of stimulus (using the same baseline for all conditions in a scan). The average response from 3^rd through 6^th time points (6^th to 12^th second) of the event-related signal after the onset of flicker was computed and defined as the cortical response to that condition for each observer and ROI.

Possible outcomes

If the cortical activity correlates with the physical properties of the stimulus we would expect to find no MR signal in the “Illusory” condition, because there is no localized physical variation at the flanks where we measured the activity. Alternatively, if the activity is correlated with the lightness variations, we would expect to find a positive signal in the “Illusory” condition. But this test alone may not be sufficient to prove the activity is to lightness variations per se. The activity we find could be a consequence of direct responses (neural or vascular) to the variations in the central portion of the stimulus. Because the luminance profile along a horizontal cross-section is identical in the “Illusory” and “Control” condition, and because the “Illusory” but not the “Control” stimulus produces a strong lightness effect, we can test this possibility. If the cortical activity correlates only with the distant contrast and luminance variations, we would expect to find no difference between the activity in “Illusory” condition and the activity in “Control” condition. Alternatively we would expect to find a stronger activity in the “Illusory” condition because of the lightness effect. Finally, because we used the “Real” stimulus that is perceptually equivalent to the “Illusory” stimulus, we can compare the activity to context-dependent lightness variations with the activity to subjectively equivalent actual physical variations.

Figure 4A shows the results averaged across observers in V1, V2 and V3. Average response to the “Real” condition was significantly higher than that to the “Illusory” condition in V1 (t(4)=2.63; p<0.05), but not in V2 and V3. Critically, the V1, V2 and V3 responses to the “Illusory” condition was significantly higher than that to the “Control” condition (t(4)=6.99; p<0.01 in V1, t(4)=4.57; p<0.01 in V2 and t(4)=9.8; p<0.001 in V3). Therefore we reject the possibility that the activity in V1, V2 and V3 in the “Illusory” condition can be explained as a direct response to distant luminance variations.

Figure 4. FMRI results.

(A) V1, V2, and V3 responses averaged across observers. The average response to the “Illusory” condition was smaller than that to the “Real” condition. However this difference was statistically significant only in V1. Importantly, the MR signal to “Illusory” condition was larger than that to the “Control” stimulus in V1, V2 and V3. The difference between the MR signals to “Illusory” and “Control” conditions suggest that the activity in “Illusory” condition can not be explained as a direct response (neural or vascular) to distant contrast and luminance variations. (See Supplemental Figure 1 for the analysis of MR data in both ROI pairs).

(B) Individual responses in V1. For two observers out of five, MR signals to “Real” condition was statistically significantly larger than that to the “Illusory” condition. For all observers except one, the “Illusory” stimulus evoked a significantly larger MR signal than the “Control” stimulus. Error bars represent one standard error of the mean (s.e.m.). Statistically significant differences are indicated (* p<0.05, ** p<0.01, *** p<0.001).

In V1, individual observer responses to the “Real” condition tended to be higher than to the “Illusory” condition. However the difference was significant for only two observers (Figure 4B). For all observers except one (ST), the response in V1 to the “Illusory” condition was significantly higher than that to the “Control” condition. For ST the difference between the “Illusory” and “Control” conditions was close to significant (t(126)=1.23; p=0.053).

We often found negative MR signals in the “Control” condition (Figure 4). Such negative signals are pervasive in fMRI studies and are usually detected beyond the stimulated regions of visual cortex. The negative signal could be due to either a purely vascular origin (for example, ‘vascular blood steal’) or suppression of neuronal activity [24].

Discussion

The cortical activity correlates with context-dependent lightness variations where locally the physical stimulus remains constant.

This finding is consistent with previous studies that provided evidence in favor of lightness responses in retinotopic cortex [11-15]

The response to actual physical variations is larger than that to perceptually equivalent context-dependent lightness variations in V1 but not in V2/3

This leaves open the possibility that mechanisms underlying the responses to actual luminance variations, at least in V1, might be different than those to context-dependent lightness variations. In a recent study Roe et al. [14], using the same CO stimulus and anesthetized monkeys, found that V2 neurons respond to both actual luminance variations and context-dependent lightness variations, but V1 neurons responded only to actual variations. Taken together with our results, this may indicate a difference between V1 and V2/3 in their role in lightness processing. Other than the difference in species, the disagreement between our results and Roe et al.'s [14] in V1 could be a consequence of the anesthesia. Our results in V1 agree with findings of Rossi et al. [13], although one must be cautious in comparing their results with ours because of the differences in the stimuli.

The cortical activity is present when the observers' attention is directed away from the lightness effect

One might be tempted to view our results as supporting the idea that lightness responses in V1 does not involve a top-down flow of information; however, our findings cannot distinguish between the roles of feed-forward and feedback mechanisms. For example, one cannot rule out higher level feedback even in the absence of attention [25,26], or in the absence of conscious perception [27]. Experimental designs and techniques that are sensitive to the timing of activity are promising candidates to distinguish between these possible explanations and determine how exactly information propagates within V1 [15].

Cornelissen et al. [17] argued that the neural activity to context-dependent lightness (or brightness) variations found in previous studies could be explained by a so called “long range edge response mechanism”, in which contrast-sensitive neurons with large receptive fields respond directly to spatially distant temporal contrast variations (i.e. greater than 1.5 degrees away), and not as a response to lightness variations per se, and concluded that there is no evidence to support a spatial filling-in of lightness in early retinotopic visual cortex. The difference we find between the “Illusory” and “Control” conditions argue against such a “direct long range response” to the central contrast flicker (note that static edges do not contribute to the MR signal averages reported here, because the measurement is done after 24s of the onset of any static edge, which is sufficiently long for any transient response to turn back to baseline). The disagreement between the studies could be due to differences between the experimental designs. Unlike in our design, Cornelissen et al. [17] could not directly compare the responses to context-dependent lightness variations against those to distant luminance and contrast variations. It is possible that contrast variations evoke a much stronger MR signal that hides the responses to context-dependent lightness variations in the particular stimulus used in their study.

There is a close and subtle relationship between lightness and brightness [5,28,29]. Brightness is defined as the perceived luminance of a surface, and the effect in our “Illusory” stimulus could be considered as a brightness effect as well as a lightness effect. In fact, often it is not even possible to distinguish between lightness and brightness in simple scenes where information about illumination is not available. Because the reflectance, but not luminance, of an object is invariant over viewing conditions, it is reasonable to assume that the visual system functions to estimate lightness rather than brightness. Therefore we described the cortical activity in terms of a response to lightness, rather than brightness variations. However, in the behavioral experiments we use a luminance comparison task as an objective measure of the lightness effect in our stimuli. This task favors low level mechanisms [5], consistent with the fMRI experiment. With the current design, we cannot say whether lightness and brightness processes are separate, or whether one precedes the other.

In summary, our results show that the activity in early visual cortical areas, including V1, correlate with context-dependent lightness variations, not only localized luminance variations. We show, for the first time, that the activity in these early cortical areas in response to lightness variations cannot be explained by means of direct responses (neural or vascular) to distant variations in the stimulus alone. These findings are consistent with earlier studies showing that cortical activity in early visual areas correlates with the perceived rather than physical properties of the visual stimuli [30-32].

Supplementary Material

Supplementary Data

Supplemental Data including the details of the Experimental Procedures, the data from both pairs of ROIs, and a discussion regarding the absence of sub-cortical activity to context-dependent lightness variations are available at http://www.current-biology.com/content/supplemental