Image luminance changes contrast sensitivity in visual cortex

SUMMARY

Accurate measures of contrast sensitivity are important for evaluating visual disease progression and for navigation safety. Previous measures suggested that cortical contrast sensitivity was constant across widely different luminance ranges experienced indoors and outdoors. Against this notion, here, we show that luminance range changes contrast sensitivity in both cat and human cortex, and the changes are different for dark and light stimuli. As luminance range increases, contrast sensitivity increases more within cortical pathways signaling lights than those signaling darks. Conversely, when the luminance range is constant, light-dark differences in contrast sensitivity remain relatively constant even if background luminance changes. We show that a Naka-Rushton function modified to include luminance range and light-dark polarity accurately replicates both the statistics of light-dark features in natural scenes and the cortical responses to multiple combinations of contrast and luminance. We conclude that differences in light-dark contrast increase with luminance range and are largest in bright environments.

Graphical Abstract

In Brief

Rahimi-Nasrabadi et al. demonstrate that image luminance has opposite effects on the contrast sensitivity of cortical pathways signaling lights than darks. It impairs luminance discrimination for the brightest stimuli of the scene while improving it for the darkest stimuli, a mechanism that is needed to efficiently sample natural scenes.

INTRODUCTION

Contrast sensitivity and visual acuity are two of the most important measures of visual function in visual disease (Chung and Legge, 2016). While there is strong evidence that visual acuity changes with luminance (Riggs, 1965; Shlaer, 1937), cortical contrast sensitivity is thought to remain constant across vastly different luminance ranges that we experience indoors and outdoors (Frazor and Geisler, 2006; Geisler et al., 2007; Mante et al., 2005; Shapley and Enroth-Cugell, 1984). To some extent, cortical responses to contrast need to remain constant across luminance changes; otherwise, image features would change from dark to light when light intensity increases. At the same time, cortical contrast sensitivity needs to be continuously adjusted if the brain wants to capture the most informative contrasts, which are different in dark and bright scenes. All of the past studies of cortical contrast sensitivity used stimuli that were several orders of magnitude dimmer than those experienced outdoors (Albrecht and Hamilton, 1982; Boynton et al., 1999; Tolhurst et al., 1983) while assuming that the measurements generalized across the entire luminance range. However, this assumption remains untested.

Another common assumption from previous studies is that cortical contrast sensitivity is similar for dark and light stimuli (i.e., stimuli darker or lighter than their background). Consequently, most scientists continue to measure cortical contrast sensitivity with grating stimuli that cannot distinguish dark from light contrast. In fact, the most common neuronal contrast measurement, Michelson contrast, computes a difference between maximum and minimum luminance that is always positive and disregards light-dark polarity. Similarly, the most common image contrast measurement, root-mean-square (RMS) contrast, computes a squared difference between pixel intensity and pixel intensity average that also disregards light-dark polarity. Weber contrast computes a difference between stimulus and background luminance that can distinguish between dark (negative Weber contrast) and light contrast (positive Weber contrast). However, Weber contrast is usually restricted to stimuli presented on homogeneous backgrounds and has a limited application to the natural images that we perceive.

The disregard of light-dark polarity in cortical contrast measurements is probably rooted in the belief that ON and OFF pathways segregate in the retina and thalamus but fully combine in the visual cortex. However, this common belief is rapidly changing. There is strong evidence that ON and OFF pathways segregate in the visual cortex (Jin et al., 2008; Kremkow et al., 2016; Lee et al., 2016; McConnell and LeVay, 1984; Najafian et al., 2019; Norton et al., 1985; Zahs and Stryker, 1988) and that the cortex represents the OFF pathway better than the ON pathway (Jansen et al., 2019; Jimenez et al., 2018; Jin et al., 2008; Wang et al., 2015; Xing et al., 2015; Yeh et al., 2009). There is also evidence that dark stimuli generate stronger, faster, and more spatiotemporally precise cortical responses than light stimuli (Jansen et al., 2019; Jin et al., 2011; Komban et al., 2014; Mazade et al., 2019; Rekauzke et al., 2016) and that contrast sensitivity is higher within ON than OFF cortical pathways (Kremkow et al., 2014; Pons et al., 2017).

Previous studies demonstrated that ON-OFF differences in cortical contrast sensitivity are more pronounced when using the maximum or minimum luminance of the monitor as background (e.g., minimum for ON, maximum for OFF) than when using a common middle-gray (Kremkow et al., 2014; Pons et al., 2017). However, these studies did not investigate the possible reasons for this difference. Middle-gray backgrounds could reduce ON-OFF contrast asymmetries either because they keep the luminance background of ON and OFF measures constant or because they reduce the monitor luminance range by half (e.g., 200 versus 100 cd/m² in a monitor with a maximum luminance of 200 cd/m²). Here, we investigate what image parameters affect ON and OFF contrast sensitivity by measuring the responses in both cat and human visual cortex to stimuli that vary in luminance background, luminance range, luminance contrast, and contrast polarity. Our results demonstrate that, against current belief (Frazor and Geisler, 2006; Geisler et al., 2007; Mante et al., 2005; Shapley and Enroth-Cugell, 1984), cortical contrast sensitivity changes with luminance range and the changes are different for light and dark stimuli.

RESULTS

As we navigate through our visual environment, our brain is continuously adjusting its visual responses to large variations in luminance range. The luminance range, defined as the difference between maximum and minimum luminance in an image sequence, is continuously changing from sunrise to sunset and can be several orders of magnitude larger in outdoor than indoor environments. As the luminance range increases (Figures 1A and 1B), the visual cortex shifts its maximum response toward greater values of stimulus luminance to signal the entire contrast range (Figure 1C). This mechanism could make contrast responses fully independent of global changes in luminance, as past studies assumed (Albrecht and Hamilton, 1982; Boynton et al., 1999; Tolhurst et al., 1983). However, because past studies used dim stimuli, the measurements may not apply to the much brighter stimuli experienced outdoors. To investigate contrast-luminance interactions in much brighter environments, we used a monitor that could reach 1,000 cd/m² of maximum luminance and measured responses in the cat visual cortex to multiple stimulus combinations of luminance contrast, contrast polarity, background luminance, and luminance range (Figure 1D). The stimuli were large dark or light squares covering the receptive fields from multiple groups of neighboring cortical neurons that were simultaneously recorded (Figure 1E, see Method details).

Figure 1. Measuring the effect of luminance range on contrast sensitivity.

(A) Natural image with low luminance range.

(B) The same image with higher luminance range.

(C) Cartoon illustrating how an expansion of luminance range affects the neural signaling of contrast range.

(D) Stimulus combinations of contrast polarity, background luminance, and luminance range. Each rectangle represents a sequence of stimuli with different luminance contrast but the same polarity (red: light, blue: dark in all figures). The longer horizontal line on the top or the bottom of the rectangle illustrates the background luminance. The longer side of the rectangle illustrates the luminance range.

(E) Stimulus temporal sequence.

(F) Naka-Rushton function fit to the cortical responses and its main parameters: the luminance that generates half-maximum response (L50), the exponent (n), and the response generated by the maximum contrast (R100). L50b is L50 minus the background luminance. L50n is L50b divided by the luminance range. Lb is the stimulus luminance minus the background luminance.

Effect of luminance range on contrast response functions

For each specific combination of luminance background, range, and light-dark polarity, we measured cortical responses to multiple luminance contrasts. We then fit these responses with a Naka-Rushton function (Figure 1F) and measured the cortical contrast sensitivity as the absolute value of the luminance generating half-maximum response (L50), the L50 minus the background (L50b), or the L50b divided by the luminance range (L50n). The L50 is closely related to the contrast sensitivity at threshold and is less vulnerable to noise; therefore, we used this measure as an estimate of contrast sensitivity throughout the article. We used three different measurements of L50 (L50, L50b, and L50n) to make the data analysis as transparent as possible and verify the replicability of results across different measurements. The L50 describes the absolute stimulus luminance, which is important to report because cortical neurons respond to luminance temporal changes (Mazade et al., 2019; Wang and Wang, 2016; Xing et al., 2014). The L50b and L50n describe the luminance contrast as the difference between stimulus and background luminance (L50b), and the difference divided by the luminance range (L50n). Throughout the article, we refer to cortical contrast sensitivity as the contrast sensitivity measured (not originating) in the visual cortex. We also report the response to the maximum contrast (R100) and the exponent of the function (n). As in previous measurements with dimmer stimuli, cortical responses increased with luminance contrast (Albrecht and Hamilton, 1982). However, unlike previous studies, we found that the shape of the contrast-response function was strongly dependent on the luminance range and the light-dark polarity of the stimulus (Figures 2A–2F). ON cortical responses increased their saturation to light contrast as the luminance range expanded from 300 cd/m² (Figure 2A) to 600 cd/m² (Figure 2B) and 1,000 cd/m² (Figure 2C). Conversely, OFF cortical responses decreased their saturation to dark contrast as the luminance range expanded from 300 cd/m² (Figure 2D) to 600 cd/m² (Figure 2E) and 1,000 cd/m² (Figure 2F).

Figure 2. Example ON and OFF cortical response functions.

(A) Top: peri-stimulus time histograms of ON responses measured in a cortical site (luminance range: 300 cd/m², background luminance: 0 cd/m², 30 trials per contrast). The first peak is the response to the stimulus onset (onset response) and the second the response to the stimulus turned off (rebound response). See luminance values in Method details. Bottom: onset responses (circles) fitted with a Naka-Rushton function (red line, L50b: black dotted line). The bottom right corner shows goodness of fit (R²), L50n, and non-linearity index (NL).

(B and C) Same for 600 and 1,000 cd/m² luminance range.

(D–F) Same for OFF responses measured in different cortical site (background luminance: 1,000 cd/m²).

For a Figure360 author presentation of this figure, see https://doi.org/10.1016/j.celrep.2021.108692.

To quantify the saturation non-linearity of the contrast-response functions, we computed a non-linearity index (NL) that estimates the difference between the measured function and an ideal linear function. The NL equals 0 when the function is a straight line with no contrast saturation, and equals 1 when the function deviates the maximum possible from a straight line and saturates at very low contrasts (Koch et al., 2016). The OFF function measured at the 1,000 cd/m² range was very close to the ideal linear function. For example, the OFF function illustrated in Figure 2F had an NL of 0.07 (0 in ideal function) and an L50n of 0.5 (0.5 in ideal function). Unlike the OFF contrast-response function, the ON function measured at the 1,000 cd/m² range was the opposite of an ideal linear function. For example, the ON function illustrated in Figure 2C had an NL index of 0.64 and an L50n of 0.15. As the luminance range increased, the ON contrast-response function decreased its L50n and increased its NL (L50n/NL: 0.38/0.19, 0.31/0.39, and 0.15/0.64 for 300, 600, and 1,000 cd/m², respectively, in Figures 2A–2C). Conversely, the OFF contrast-response function increased its L50n and decreased its NL (L50n/NL: 0.31/0.38, 0.36/0.25, and 0.5/0.07 for 300, 600, and 1,000 cd/m², respectively, in Figures 2D–2F).

We investigated the effects of luminance range on the cortical contrast-response function by averaging the responses across all luminance backgrounds for each luminance range. Increasing the luminance range from 300 to 1,000 cd/m² shifted the average cortical ON and OFF L50b (Figures 3A and 3B, top) toward a higher stimulus luminance, allowing the cortex to continue signaling the entire contrast range. However, the shift in L50b was more pronounced for OFF than ON cortical responses (Figure 3C, top) increasing the OFF-ON differences in L50b (Figure 3D, top). The same conclusion can be reached if we divide the L50b values by the luminance range (Figures 3A–3D, center). Because expanding the luminance range from 300 and 1,000 cd/m² increased the L50b less for ON than OFF cortical responses, the ON L50n decreased (Figure 3A, center) while the OFF L50n increased (Figure 3B, center), enhancing the OFF-ON L50n differences (Figures 3C and 3D, center). The same conclusion can be reached if we measure the absolute L50 luminance value (Figures 3A–3D, bottom). Because expanding the luminance range from 300 and 1,000 cd/m² decreased the L50 average across all backgrounds for ON responses (Figure 3A, bottom) but increased it slightly or did not change it for OFF responses (Figure 3B, bottom), the OFF-ON L50 difference became larger (Figures 3C and 3D, bottom). Therefore, expanding the luminance range increased the OFF-ON differences in contrast sensitivity (Figure 3D), mostly because the ON L50b did not increase enough to compensate for the change in luminance range. The ON L50b became closer to the lowest contrasts, causing an increase in contrast sensitivity and high-contrast saturation for light stimuli. Conversely, the OFF L50b followed more tightly the changes in luminance range, becoming closer to middle-contrast values, and causing a reduction in both contrast sensitivity and high-contrast saturation to dark stimuli.

Figure 3. Effects of luminance range on cortical response functions.

(A) Average L50b (top), L50n (center), and L50 (bottom) measured at 3 different luminance ranges from ON onset responses (n: 670, 685, and 110 cortical recording sites listed from lowest to highest luminance range).

(B) Same for OFF onset responses (n: 1,113, 818, and 217 cortical recording sites listed from lowest to highest luminance range).

(C) ON and OFF values superimposed for comparison.

(D) Difference between OFF and ON values. *p < 0.05, p < 0.001; ns, not significant (bootstrap resampling, 50,000 repetitions).

(E–H) Same for NL (top), exponent (center), and response at maximum contrast (R100, bottom).

For (A)–(C), *p < 0.05, **p < 0.001; ns, not significant (2-sided Wilcoxon rank sum test).

See also Figures S1 and S2.

Increasing the luminance range from 300 and 1,000 cd/m² also enhanced the average OFF-ON differences in the linearity of the contrast-response function. The average non-linearity increased in the ON function (Figure 3E, top) while decreasing in the OFF function (Figure 3F, top), making the OFF-ON differences larger (Figures 3G and 3H, top). Increasing the luminance range also reduced the exponent of the ON and OFF contrast-response functions (Figures 3E–3H, center), a finding that is consistent with the exponent increase previously demonstrated at low light (Pons et al., 2017). Expanding the luminance range also increased the maximum ON (Figure 3E, bottom) and OFF cortical responses (Figure 3F, bottom). OFF responses were significantly stronger than ON responses at the 300 cd/m² but not at the 600 – 1,000 cd/m² luminance range, suggesting that brighter environments may make the strength of ON and OFF cortical responses more balanced (Figures 3G and 3H, bottom). Based on these measurements, we conclude that expanding the luminance range affects all of the parameters of the contrast response function, including the contrast sensitivity, contrast saturation, exponent, and maximum response while increasing the ON-OFF differences in contrast sensitivity and saturation. Similar results can be demonstrated if we do not average the responses across backgrounds and restrict the background luminance to 0 cd/m² for ON responses and 1,000 cd/m² for OFF responses (Figure S1). Similar results can also be demonstrated if we measure responses to stimuli turned off (rebound responses) instead of responses to stimuli turned on (onset responses). Rebound responses are weaker and show less stimulus specificity than onset responses (Komban et al., 2014; Mazade et al., 2019), but, as demonstrated here, they are similarly affected by luminance range (Figures S2A–S2E).

Effect of background luminance on contrast response functions

Unlike the luminance range, the background luminance did not cause consistent changes in the contrast response function. When the luminance range was kept constant at 300 cd/m², changes in background luminance also kept the ON and OFF L50b relatively constant (Figures 4A and 4B, top). Most 100 cd/m² increments in background luminance did not cause significant changes in ON L50b (Figure 4A, top), and although some 100 cd/m² increments were associated with significant changes in OFF L50b, the changes were small (~7% average) and unrelated to luminance increments (R² = 0.05, p = 0.563; Figure 4B, top). The OFF L50b was larger than the ON L50b for all backgrounds (Figure 4C, top), consistent with our previous measurements (Kremkow et al., 2014; Pons et al., 2017). The OFF-ON differences in L50b were significantly higher at 400 cd/m² than at other luminance backgrounds (Figure 4D, top), probably by chance, since the p values were relatively modest (p = 0.026 and 0.011 for 300–400 and 400–500 cd/m² comparisons, n = 85, 91, and 71 pairs of ON-OFF functions measured at 300, 400, and 500 cd/m² background luminance, respectively). Consistent with this interpretation, the ON-OFF differences in absolute L50 remained remarkably constant across changes in background luminance (Figures 4C and 4D, bottom). Moreover, the increase in both ON and OFF L50 (Figures 4A and 4B, bottom) was well predicted by linear functions with a slope of 1 (R² = 0.977 for ON and 0.982 for OFF). Notice that the absolute L50 (without background subtraction) is larger for ON than OFF responses because the luminance at half-range is also larger (i.e., the middle value between a 600-cd/m² target in a 300-cd/m² background is 450 cd/m², but the middle value between a 0-cd/m² target in a 300-cd/m² background is 150 cd/m²).

Figure 4. Effects of background luminance on cortical response functions.

(A) Average L50b (top) and L50 (bottom) from ON onset responses measured at different luminance backgrounds (luminance range: 300 cd/m², n: 81, 90, 88, 85, 91, 71, 81, and 83 cortical recording sites measured at 8 different backgrounds listed from lowest to highest luminance).

(B) Same for OFF onset responses (n: 146, 138, 148, 149, 145, 135, 129, and 123 cortical recording sites measured at 8 different backgrounds listed from lowest to highest luminance).

(C) ON and OFF values superimposed for comparison.

(D) Difference between OFF and ON values.

(E–H) Same for NL (top), exponent (center), and response at maximum contrast (R100, bottom). LB: background luminance. Statistical tests and symbols as in Figure 3.

See also Figure S2.

The background luminance also did not affect the non-linearity of the ON and OFF luminance response functions. As in our previous measurements (Kremkow et al., 2014; Pons et al., 2017), the nonlinearity was consistently more pronounced for ON than OFF functions at all backgrounds (Figures 4E–4G, top). Most of the 100 cd/m² luminance increments did not cause significant changes in the ON or OFF NL (Figures 4E and 4F, top), and increments in background luminance were not significantly associated with changes in non-linearity (Figure 4H, top). Most 100 cd/m² increments in background luminance also did not cause significant changes in the ON and OFF exponent (Figures 4E–4H, center). However, the exponent increased slightly when the background luminance increased from 300 to 700 cd/m² (Figure 4G, center), as if increasing the background luminance in a constant luminance range was similar to reducing the luminance range. Finally, the OFF maximum response was consistently stronger than the ON maximum response at all of the backgrounds measured at a 300-cd/m² luminance range (Figures 4E–4H, bottom, see Figures S2F–S2I for measurements at a 600-cd/m² luminance range). Therefore, we conclude that the contrast response function is modulated by luminance range more than background luminance.

ON and OFF functions in human visual cortex

To investigate the implications of these findings for human vision, we performed electroencephalographic (EEG) recordings from the human visual cortex. Our measurements in humans are generally consistent with the most extensive measurements in cats. In the example subject illustrated in Figure 5, doubling the luminance range from 250 to 500 cd/m² increased the contrast sensitivity and saturation of the ON function and had the opposite effect in the OFF function. In the ON function, the L50n decreased from 0.34 to 0.21 and the NL increased from 0.26 to 0.34 (Figures 5A and 5B). In the OFF function, the L50n increased from 0.28 to 0.34 and the NL decreased from 0.25 to 0.18 (Figures 5C and 5D). Similar results could be demonstrated in two other subjects (Figure S3). As in the cat visual cortex, expanding the luminance range significantly increased the ON-OFF differences in L50b (Figure 5E) and L50n (Figure 5F). The human pupil size was not fixed, but at the time of the cortical measurements (Figure 5G and 5H, arrows), its size variation was small and not correlated with stimulus luminance (r < 0.05 for each subject, see Method details). The variations in pupil size across subjects were larger and allowed us to demonstrate a strong logarithmic relation between the range of retinal luminance (in trolands) and the OFF/ON L50n ratio (Figure 5F; R² = 0.93, slope = 0.6, 3 subjects, 2 luminance ranges per subject).

Figure 5. Luminance response functions in human visual cortex.

(A) Top: ON cortical responses to different stimulus contrasts measured in a human subject (luminance range: 250 cd/m², 30 trials per contrast for each background luminance, light-dark polarity, and luminance range). Bottom: cortical responses (circles) fit with a Naka-Rushton function (line). Top left corner shows goodness of fit (R²), L50n, and NL.

(B) Same for luminance range of 500 cd/m².

(C and D) Same for OFF responses.

(E) OFF-ON differences in L50b measured in the same subject (white) and subject average (gray). *p < 0.05 (bootstrap resampling, 50,000 repetitions).

(F) Logarithmic relation between luminance range and OFF/ON L50n ratio (bootstrap resampling, 50,000 repetitions).

(G) Pupil response of same subject (top) measured at 250 (dotted line) and 500 cd/m² luminance ranges (continuous line) at the peak (arrow) of the ON cortical response (bottom).

(H) Same for OFF responses.

See also Figure S3.

ON and OFF density functions in natural scenes

To further investigate the general implications of our results for luminance vision, we measured the cumulative density functions of light (ON) and dark (OFF) stimuli in natural scenes. We hypothesize that if the cortex processes visual information efficiently, then the ON and OFF cortical functions should match the ON and OFF cumulative density functions of light and dark contrast in nature. To test this hypothesis, we measured the ON and OFF density functions from 4,167 monochrome calibrated images (van Hateren and van der Schaaf, 1998). We calculated the cumulative density functions as the accumulated number of pixels with increasingly larger values of dark or light contrast relative to the median image luminance (i.e., equivalent to background luminance in cortical measures). If all contrast values are equally represented in a natural image, then the density function should be linear as the OFF cortical function. Conversely, if low contrast values dominate, then the density function should saturate as the ON cortical function. To facilitate the comparison, we normalized the cortical functions by the maximum response (Figure 6A), the density functions by the total number of pixels (Figure 6B, see Method details), and the x axis of both functions by the luminance range. Since each image has ~1.5 million pixels with different luminance values, we defined the image luminance range as the standard deviation of the luminance distribution instead of the difference between two pixels with maximum and minimum luminance (see Method details). We then compared image sets with luminance standard deviations between 100 and 200, 200 and 400, and 400 and 800 cd/m².

Figure 6. Dark and light contrast in natural scenes.

(A) Average luminance response functions for ON (red) and OFF (blue) cortical responses measured at 300 cd/m² (dashed lines, n: 670 recording sites for ON, 1,113 recording sites for OFF) and 1,000 cd/m² luminance ranges (continuous lines, n: 110 recording sites for ON, 217 recording sites for OFF).

(B) Average ON and OFF image density functions of natural images with different luminance ranges (dashed lines: 0–200 cd/m²; continuous lines: 400–800 cd/m²; see sample sizes in D and E).

(C) ON L50n from image density functions plotted against OFF L50n. Each dot represents an image (n: 3,061 images with median luminance >50 cd/m²). The diagonal shows the unity line.

(D) Average L50b (top), L50n (center), and nonlinearity (bottom) for ON image density functions measured in natural images with different luminance ranges (n: 617, 429, and 643 images listed from lowest to highest luminance range).

(E) Same for OFF image density functions (n: 2,227, 534, and 253 images listed from lowest to highest luminance range).

(F) Superimposed ON and OFF values.

(G) Difference between OFF and ON values.

(H) Normalized bootstrap histograms of OFF/ON L50n log ratios from natural images (left), cat visual cortex (center), and human visual cortex (right). High/low range: ratio of luminance range. Adaptation slope: logarithm of OFF/ON L50n ratio divided by the logarithm of the high/low range (bootstrap resampling, n: 50,000 repetitions). Statistical tests and symbols as in Figure 3.

Increasing the luminance range made the OFF L50n larger than the ON L50n in the image density function (Figure 6B), closely replicating the findings in the visual cortex (Figure 6A). Moreover, as in the visual cortex, the OFF L50n was larger than the ON L50n in most images (Figure 6C). A luminance range expansion from 0 to 200 cd/m² to 400–800 cd/m² increased the OFF L50b more than the ON L50b (Figures 6D and 6E, top) making the OFF-ON L50b differences larger (Figures 6F and 6G, top). It decreased the ON L50n and slightly increased the OFF L50n (Figures 6D and 6E, center) also making the OFF-ON L50n differences larger (Figures 6F and 6G, center). It decreased the non-linearity of the OFF functions and increased it in the ON functions (Figures 6D–E, bottom), also enhancing the ON-OFF differences in non-linearity (Figures 6F and 6G, bottom). Therefore, expanding the luminance range caused a significant increase in the ON/OFF L50n ratio of natural images, which resembled the increase demonstrated in the cat and human visual cortex (Figure 6H). The adaptation slope of this increase (log increase in OFF/ON L50n per log increase in luminance range) was similar in natural images (0.28) and cat cortex (0.29) but higher in human cortex (0.73) for reasons that remain unclear (e.g., species differences, small number of human subjects tested, sampling limitations of electroencephalographic recordings). Based on these results, we conclude that changes in luminance range affect similar ON-OFF functions in natural images and visual cortex.

Simulating human luminance vision

Our measurements of ON and OFF cortical functions could help to simulate more accurately human luminance vision and prevent bright digital images from appearing too dark (Figure 7A). To pursue this goal, we developed an ONOFF algorithm that uses separate ON and OFF cortical functions to process images. The first stage of the ONOFF algorithm converts the luminance range of any image into a common neuronal range that spans from 0 to 1 (maximum firing rate). The second stage splits each image into an ON and OFF version by subtracting the ON and OFF backgrounds and passing the ON image through a function with higher contrast sensitivity than the OFF image. Finally, the third stage combines the ON and OFF images to simulate the image that we see (see Method details).

Figure 7. ONOFF image processing and visual contrast.

(A) The left panels show a natural image (van Hateren and van der Schaaf, 1998) normalized by its maximum (top) and a histogram of image pixel intensity (bottom, 1,024 × 1,531 pixels). The other panels show the same for images processed with IMADJUST, CLAHE, and ON-OFF algorithms.

(B) Contrast gain from IMADJUST (black), CLAHE (green), and ON-OFF (blue, n: 4,167 images).

(C) Correlations between the mean luminance of the original image and the pixel intensity of the processed image (n: 4,167 images).

(D) ON and OFF L50 (x axis) can be accurately predicted (y axis) with linear regression models (equations at the top) for both onset responses (left) and rebound responses (right). Notice that for each combination of background and range, the model returns a single value, but the measured L50 is more variable (n: 331 recording sites for ON onset responses, 520 recording sites for OFF onset responses, 275 recording sites for ON rebound responses, and 321 for OFF rebound responses).

(E) Left: ON and OFF L50 of the image density functions (x axis) and prediction (y axis) with a linear model (equations at the top, n: 3,247 ON images and 3,247 OFF images with mean luminance >50 cd/m²). Right: same as left but for the exponent of the ON and OFF functions (n: 3,112 ON images and 3,175 OFF images with accurately fitted luminance response functions, R² > 0.9).

(F) Correlations of the image L50n with the skewness (left) and kurtosis (center) of the luminance distributions, and model predictions (right, equations at the top, n: 3,247 ON images and 3,247 OFF images with mean luminance >50 cd/m²).

(G) Definition of ONOFF contrast. DVC, dark visual contrast; LVC, light visual contrast; Range, luminance range; Sb, stimulus luminance minus background luminance.

See also Figure S4.

We compared the performance of the ONOFF algorithm with two other algorithms that are commonly used to enhance contrast in image processing analysis: contrast-limited adaptive histogram-equalization (CLAHE) and MATLAB Image adjustment (IMADJUST). The ONOFF algorithm simulated human luminance vision better than the two other algorithms in images with a bright sky, which appeared too dark with IMADJUST and had unnatural dark regions around the clouds with CLAHE (Figures 7A and S4). To quantify this comparison, we processed the 4,167 monochrome calibrated images (van Hateren and van der Schaaf, 1998) with the 3 algorithms. Then, we calculated the contrast gain achieved by each algorithm across different spatial frequencies. The contrast gain was consistently higher for the ONOFF than the IMADJUST algorithm across all spatial frequencies. CLAHE outperformed the other algorithms at the highest spatial frequencies but did very poorly at low spatial frequencies. At low spatial frequencies (i.e., 0.04–0.4 cycles per degree), the ONOFF algorithm consistently outperformed the other two algorithms that we tested (Figure 7B). Low spatial frequencies dominate the power spectrum of natural scenes (Burton and Moorhead, 1987; Field, 1987); therefore, they need to be accurately reproduced to avoid contrast distortions in large surfaces such as sky patches (e.g., CLAHE in Figure 7A). The more balanced performance of the ONOFF algorithm across high, middle, and low spatial frequencies highlights the benefit of processing images separately with ON and OFF contrast functions. In addition, if the ONOFF algorithm incorporates an adaptation factor based on luminance range (ONOFF adaptive), it can reproduce more closely the mean luminance of the original scene than the other algorithms (Figure 7C; Method details).

ON-OFF model of visual contrast sensitivity

Our measurements could also help develop better models of visual contrast sensitivity. To search for the model that best predicted the L50 measurements, we fitted the cortical data with a weighted sum of three terms: luminance background, luminance range, and the interaction between light-dark polarity and luminance range. This analysis revealed a model that explains 78%–90% of L50 variance for all stimulus combinations tested, regardless of whether the stimulus was turned on (Figure 7D, left) or off (Figure 7D, right). In the model, the L50 for the stimulus onset equals the background luminance plus a fraction of the luminance range that is lower for light than dark stimuli (0.29 versus 0.45). Therefore, the L50b equals 0.29 * Lrange (lights) and 0.45 * Drange (darks) and the L50n equals 0.29 and 0.45 (notice that the L50-background difference is negative for darks because the background value is larger than the L50).

The analysis of the image density functions revealed a very similar model that explains 80%–83% of the L50 variance (Figure 7E, left). In this model, the L50 equals again the background luminance plus a fraction of the luminance range that is lower for light than dark stimuli. The analysis of natural images also revealed a model for the exponent of the OFF function that explains 90% of the variance (Figure 7E, right). This model could not be replicated with the cortical data because the exponent variation was very limited (1.5–2.5). However, in the image model, the OFF exponent increases when the luminance range decreases, which is exactly what we found in cortical responses (Pons et al., 2017).

To further investigate the image factors that affect contrast sensitivity, we fit the image data with another linear model that included two shape descriptors of the luminance distribution— kurtosis and skewness. The kurtosis is a measure of distribution sharpness, which is zero for Gaussian distributions and positive for sharper distributions than a Gaussian. The skewness is a measure of distribution symmetry, which is zero when the distribution is symmetric, negative when it decays slower toward the darkest values, and positive when it decays slower toward the brighter values. The model from this third analysis explains 54%–78% of the L50n variance across all images (Figure 7F) and is very similar to the other two models. As in the previous models, the L50 equals the background plus a fraction of the luminance range. Because the skewness and kurtosis are different for ON and OFF luminance distributions (Figure 7F), the fraction is also different.

These three analyses indicate that contrast sensitivity is closely associated with the background luminance, luminance range, and the shape of the image luminance distribution. However, they do not capture the effect of the luminance range on the different parameters of the contrast response function that we measured. To develop a model that replicates our experimental results more closely, we fit a Naka-Rushton equation to the average contrast response functions measured in the cat visual cortex at the 300 and 1,000 cd/m² luminance ranges. We then extracted the parameter values from these fits (L50n, exponent, and gain) and used them to define a contrast metric that we call ONOFF contrast. The parameters of ONOFF contrast are different for dark and light stimuli (DVC and LVC in Figure 7G) and vary with luminance range (see Method details). ONOFF contrast falls within the same theoretical space as Weber contrast and its two most important variants, W and logW (Kingdom and Moulden, 1991; Whittle, 1986). However, unlike all of the previous contrast metrics, ONOFF contrast changes with luminance range and reaches its maximum value in bright environments.

DISCUSSION

Luminance contrast is traditionally defined as a normalized luminance difference between two adjacent regions of an image. The difference is calculated between the luminance of a stimulus and its background in Weber contrast, the maximum and minimum luminance in Michelson contrast, or the luminance of each image pixel and the pixel mean in RMS contrast. These traditional measurements assume that the cortical processing of luminance contrast is independent of the dark-light polarity of the stimulus and the amount of light in the scene. Against this current belief, here, we show that cortical contrast sensitivity is higher for light than dark stimuli and the difference increases with light intensity. Based on these results, we introduce a measurement of visual contrast (ONOFF contrast) that closely reproduces our cortical measurements and reaches its highest value in bright scenes.

ONOFF contrast changes with luminance range

The practical implications of our results can be better illustrated with an example of a contrast measurement outdoors. A subway sign in New York City has white letters on a black background with a letter/background luminance ratio that can vary from 4/0.3 cd/m² in the evening to 2,500/180 cd/m² in the middle of the day (Pons et al., 2017). The Weber and Michelson contrasts of this subway sign is always the same regardless the time of the day (~90%). In comparison, the ONOFF contrast increases from the evening (~72%) to the middle of the day (~100%). If contrast was independent of image brightness as is currently thought, contrast adaptation and the shape of the contrast response function should be the same in monitors with very different luminance ranges. However, our results demonstrate that this is not the case. Increasing the luminance range from 300 to 1,000 cd/m² causes pronounced changes in multiple parameters of the contrast response function, including the contrast sensitivity, contrast saturation, linearity, exponent, and maximum response. Moreover, these changes are different for dark stimuli than for light stimuli, and the differences increase by up to three times when the luminance range expands. Accurate measurements of luminance contrast play a major role in a wide variety of fields, including vision research, clinical vision, medical imaging, image processing analysis, and architectural design (especially when designing spaces for the elderly and visually impaired). Therefore, the ONOFF contrast measurements that we introduce could have broad practical implications.

ONOFF contrast predicts that strong sunlight should make it difficult to discriminate large luminance differences between the brightest values of the scene but easier to discriminate subtle differences between the darkest values. For example, two bright reflections in a shiny car should appear equally bright even if they have different luminance because they fall in the saturating portion of the ON luminance response function and generate cortical responses with similar strength. Conversely, strong sunlight should make it easier to discriminate a dark fly standing on a dark table because slightly different dark stimuli fall within a linear contrast response function and generate cortical responses with different strengths. Our results also indicate that changes in luminance range affect cortical contrast sensitivity more than changes in background luminance. Previous studies demonstrated that increasing the background luminance from black to middle-gray in a monitor reduced cortical contrast sensitivity to light stimuli (Kremkow et al., 2014; Pons et al., 2017). However, these studies increased background luminance at the expense of reducing luminance range. Here, we show that when the luminance range is constant, changing the background luminance from black to middle-gray does not cause pronounced changes in cortical contrast sensitivity.

It is important to emphasize that our definition of luminance range includes both space and time because it is measured as the maximum minus the minimum luminance in a temporal sequence of images. In a static image, the luminance range equals the spatial luminance range if the eye movements are contained within the image. However, blinks often interrupt fixation and can make the luminance range expand (from a luminance close to zero when the eyes are closed to a high luminance when the eyes open in front of a bright sky). Whereas traditional measurements of contrast are frequently restricted to luminance differences across visual space, temporal contrast is equally important. Contrast measurements depend heavily on the temporal history of the stimulus. A high-contrast stimulus presented at a given time reduces the cortical responses (and visibility) of a stimulus with lower contrast presented immediately after (Albrecht et al., 1984; Gardner et al., 2005; Ohzawa et al., 1982). This continuous adjustment of contrast sensitivity has a time course that can span several seconds and is remarkably similar between humans and cats (Albrecht et al., 1984). Consistently, our experiments show that expanding the luminance range across time also causes a similar increase in the dark-light differences of contrast sensitivity in both species.

ONOFF contrast changes with the light-dark polarity of the stimulus

Traditional measurements of luminance contrast frequently disregard the light-dark polarity of the stimulus. For example, the field of vision research has been using sinusoidal grating patterns to measure neuronal contrast sensitivity for many decades (Albrecht and Hamilton, 1982; Chung and Legge, 2016; Mante et al., 2005; Shapley and Enroth-Cugell, 1984). However, most grating measurements cannot separate responses to dark or light contrast because the grating sinewaves have equal dark and light deviations from the mean. RMS contrast also disregards differences in dark-light polarity because it squares the difference between each pixel luminance and the luminance mean of an image. Whereas traditional measurements assume that cortical responses to light and dark contrast are similar and can be averaged, our results demonstrate that this assumption leads to inaccurate measurements, especially outdoors. We notice that the separate processing of dark and light contrast is one of the best evolutionarily preserved neuronal operations in vision. Dark and light contrast are separately measured by ON and OFF pathways already at the very first synapse made by the photoreceptors in the visual pathway. Moreover, the light-dark separation at the first synapse is present in all species that can form retinal images, from flies to humans (Kremkow and Alonso, 2018).

ON-OFF contrast asymmetries are likely to be inherited in part from the retina (Cooper, 2016; Kremkow et al., 2014) and may depend on parameters that we did not investigate such as spatial frequency (Jansen et al., 2019; Kremkow et al., 2014; Onat et al., 2011), visual eccentricity, and brain state (Azimi et al., 2020). There is increasingly stronger evidence that ON and OFF pathways remain cortically segregated (Kremkow and Alonso, 2018) and operate relatively independently. For example, the inactivation of the ON pathway in both humans and non-human primates affects vision for light stimuli on dark backgrounds, leading to night blindness, but does not affect vision for dark stimuli on bright backgrounds (Dryja et al., 2005; Schiller et al., 1986). Moreover, most visual cortical functions such as orientation and direction selectivity are preserved when the ON pathway is inactivated in primates, cats, or mice (Sarnaik et al., 2014; Schiller, 1982; Sherk and Horton, 1984). Cortical responses are tuned to dark and light stimuli with different spatiotemporal properties (Komban et al., 2014; Onat et al., 2011) and show a preference for dark stimuli that are fast and light stimuli that are slow (Mazade et al., 2019). Similarly, humans are more sensitive to slow stimuli that are light (Luo-Li et al., 2018), and fly navigation relies on light stimuli moving slower than dark stimuli (Leonhardt et al., 2016).

A large body of psychophysical studies have also demonstrated that dark and light stimuli contribute differently to human vision. Dark targets are more salient than light targets when presented under low light (Blackwell, 1946; Pons et al., 2017; Short, 1966) or superimposed in noise (Komban et al., 2011, 2014; Pons et al., 2017). Humans show a remarkably pronounced preference for dark contrast instead of light contrast when judging luminance variance across texture surfaces (Chubb et al., 1994, 2004; Nam and Chubb, 2000). Moreover, humans read black letters on white backgrounds faster than white letters on black backgrounds (Buchner and Baumgartner, 2007), have different contrast thresholds for dark than light targets (Kingdom and Moulden, 1991; Whittle, 1986), and use different scaling to classify levels of darkness than lightness (Whittle, 1992). Eye doctors have also been measuring visual function with dark or light stimuli instead of gratings for more than a century. The eye chart to measure visual acuity uses dark contrast (i.e., black letters on white background) and the perimetry test to measure visual sensitivity uses light contrast (i.e., bright spots on white background). Even more recent tests to measure contrast sensitivity (Pelli et al., 1988) or crowding (Levi, 2008) use letters instead of gratings. The use of dark or light contrast in the eye clinic is very consistent with the dark-light asymmetries demonstrated in visual cortex. Since cortical spatial resolution is higher for dark than light stimuli (Kremkow et al., 2014, 2016; Mazade et al., 2019; Pons et al., 2017), measures of spatial resolution (or contrast sensitivity loss at high spatial resolution) should be best assessed with dark contrast. Conversely, since cortical contrast sensitivity is higher for light than dark stimuli (Kremkow et al., 2014; Pons et al., 2017), loss of contrast sensitivity should be best assessed with light contrast.

Efficient cortical sampling of light and dark contrast in natural scenes

Our results also demonstrate that light-dark differences in cortical contrast sensitivity are closely matched to light-dark contrast differences in nature, including their dependence on luminance range. Previous studies demonstrated that neurons efficiently sample image contrast in multiple species, ranging from invertebrates such as the blowfly (Laughlin, 1981) to mammals such as cats and macaques (Tadmor and Tolhurst, 2000). However, these previous studies measured contrast with gratings and did not separate dark from light contrast. Our results extend this previous work by demonstrating that neurons sample efficiently the light-dark contrast differences found in nature (Cooper and Norcia, 2015; Ratliff et al., 2010).

Our results also explain why previous studies did not find a relation between contrast and luminance in neuronal responses (Mante et al., 2005). If the variation in luminance range is limited by a dim monitor, then changes in mean luminance will have a negligible effect on contrast sensitivity. Similarly, contrast and luminance can be independent in static natural scenes and across images with similar luminance range (Frazor and Geisler, 2006). Luminance range changes by several orders of magnitude in nature, from sunrise to sunset. As shown here, these slow temporal variations adjust the contrast response function of cortical neurons to capture the most informative contrasts at each luminance level. There is a limited advantage in knowing that two bright specular reflections have different luminance in a brightly lit sea. However, a subtle luminance change at night may signal the movement of an approaching predator. Therefore, the visual cortex needs to signal both contrast and luminance changes (Dai and Wang, 2012; Mazade et al., 2019; Smith et al., 2015; Wang et al., 2015; Xing et al., 2015), and, as shown here, these two cortical signals are continuously interacting to efficiently sample our visual world.

STAR★METHODS

RESOURCE AVAILABILITY

Lead contact

Further information can be requested to the Lead contact, Jose Manuel Alonso (jalonso@sunyopt.edu).

Materials availability

This study did not generate new unique reagents.

Data and code availability

All datasets, custom scripts and functions used in this manuscript are available from the Lead contact (jalonso@sunyopt.edu) upon request.

EXPERIMENTAL MODEL AND SUBJECT DETAILS

Animal model

Adult male cats (Felis catus, 4–7 kg, age: 9 months to 1.5 years old, n = 6) were housed in groups and allowed periods of free roaming within a private room. They were fed Purina cat food, provided with enrichment items, and had daily interaction with the personnel at the animal facility. All animal experiments and procedures were performed following the guidelines of the United States Department of Agriculture (USDA) and were approved by the Institutional Animal Care and Use Committee (IACUC) at the State University if New York, College of Optometry.

Human subjects

We recruited three human subjects to measure contrast response functions in human cortex: a 25 years old male, a 27 years old male, and a 28 years old female. All experiments in human subjects were approved by the institutional review board at the State University of New York College of Optometry and followed the principles outlined in the Declaration of Helsinki.

METHOD DETAILS

Multielectrode recordings from cortical spiking activity

Surgical procedures were similar to those described in previous studies (Kremkow et al., 2016; Mazade et al., 2019). Briefly, the animals received intramuscular injections of acepromazine (0.2 mg per kg) and ketamine (10 mg per kg). Then, an intravenous catheter was inserted in each limb to administer continuous infusions of propofol (3–6 mg per kg per hour), sufentanil (10–20 ng per kg per hour), vecuronium bromide (0.2 mg per kg per hour), and saline (1–3 mL per hour). The nictitating membranes were retracted with 2% neosynephrine, the pupils dilated with 1% atropine sulfate, and the eyes fitted with contact lenses to protect the corneas and focus visual stimuli on the retina. The contact lenses had an artificial pupil of 3 mm. For consistency, the paper gives all the luminance values in cd/m² except in Figure 5F. The cat retinal illuminance (in trolands) can be calculated as the luminance of the stimulus (in cd/m²) times the area of the pupil (radius² * pi = ~7.1 mm²).

A small craniotomy was performed over the primary visual cortex to give access to electrophysiological recordings. Throughout the experiment, we continuously recorded and monitored multiple vital signs including heart rate, blood pressure, electrocardiogram, temperature, pulse oximetry, expired CO2 and electroencephalogram (EEG) activity. Cortical recordings were performed using 32-channel linear multielectrode arrays (0.1 mm inter-electrode distance, Neuronexus) with one, two or four shanks (200–400 microns of inter-shank distance). The multielectrode arrays were introduced nearly parallel to the cortical surface and centered in layer 4, around 2 mm lateral to the brain midline and 4–5 mm posterior to stereotaxic zero. At these cortical locations, most receptive fields were located between the horizontal meridian and the lower visual field within 10 degrees of eccentricity. The electrophysiological spiking activity was amplified, sampled at 40 kHz, and filtered between 250 Hz and 8 kHz.

We presented visual stimuli on a high-dynamic-range monitor (TRU-Vu, SRMH-15-AR series, maximum luminance: 1,400 cd/m²) placed at 0.57 m from the animal. The monitor was gamma calibrated and the stimulus luminance range was restricted to 1,000 cd/m² (the monitor output luminance deviated from a standard gamma function for luminance values larger than 1000 cd/m²). All stimuli were generated with MATLAB Psychtoolbox. We used sparse noise stimuli to map the aggregate receptive fields from each group of neighboring cortical neurons sampled with each recording site of the multielectrode array. The sparse noise was made of dark (~0 cd/m²) and light (1000 cd/m²) squares (2.8 deg/side) presented for 33 ms in a pseudorandom sequence. The ON and OFF receptive fields were calculated by spike-trigger averaging the light and dark stimuli. We measured luminance response functions with large dark and light squares (8 deg/side) of different luminance that covered the receptive fields from all simultaneously recorded neurons. The squares were presented in sequence, each one lasting 133 ms and separated from the next one by a blank period without stimuli that also lasted 133 ms. Each sequence started with a uniform background lasting 10 s with no stimuli (adapting background). The same background was used for the entire duration of the stimulus sequence. The stimuli within each sequence had 15–30 randomized luminance values with one of two possible polarities (dark or light), one of three different luminance ranges (300, 600 or 1000 cd/m²), and one of 10 different luminance backgrounds (0 to 1000 cd/m², 100 cd/m² interval). Each stimulus luminance value was repeated 30 times. The luminance values illustrated in Figure 2 are, from bottom to top: 0, 43, 64, 107, 129, 171, 193, 236, 257 and 300 cd/m² for a and d; 0, 62, 124, 186, 248, 310, 372, 434, 497 and 559 cd/m² for b and e; and 0, 125, 208, 333, 417, 542, 625, 750, 833 and 958 cd/m² for c and f (notice that only a subset of all values tested are shown in the figure).

Analysis of spiking activity

We generated peri-stimulus histograms (PSTH) of the multi-unit activity recorded with each recording site of the multielectrode array. The spike counts were averaged in time bins of 1 ms and smoothed with a Gaussian kernel of 20 ms. We calculated the signal-to-noise of each recording site as the average response to all stimuli within a sequence divided by the average baseline activity of sequence periods with no stimuli. Only recording sites with a signal to noise ≥3 were included in the analysis. To generate luminance-response functions, we measured the peak response of the PSTH within 30–110 ms following the stimulus onset. We also generated functions from rebound responses measured within a time window of 130–210 ms following the stimulus onset. Both peak and rebound responses had subtracted baseline activity.

We fit each luminance response function with a Naka-Rushton function (Equation 1) using the MATLAB function fmincon.

$$R=R_{max}\frac{L^n}{{L_{50}}^n+L^n}$$ (Equation 1)

Where Rmax is the maximum response of the function, n is the exponent, L is the stimulus luminance, and L50 is the luminance that generated half-maximum response. Notice that the fitting parameter Rmax is different from the R100 that we used in our measurements. Rmax is the multiplicative factor extracted from the fit whereas R100 is the response generated by the largest luminance contrast (100%). We only included in the analysis cortical response functions that had a reasonable goodness of fit (R² > 0.7).

HUMAN ELECTROENCEPHALOGRAPHIC RECORDINGS

We used a customized 6-channel electroencephalogram (EEG) headset with dry electrodes (Wearable sensing) to measure the neuronal activity of the human occipital cortex with the same density as a 128-channel EEG (electrode locations: Oz, O1, O2, POz, PO1, PO2). EEG recordings were sampled at 300 Hz and low-pass filtered at 100 Hz. We tracked eye position and pupil size with Eyelink 1000, and asked subjects to fixate on a point at the center of the screen during the entire stimulus presentation. While the subjects maintained fixation, we presented a sequence of stimuli. The stimuli were checkerboard patterns made of 20 squares per side. Each square had a side of 0.5 degrees and could have a variable luminance (square target) or the background luminance (square background, 500 cd/m²). The luminance of all the square targets within each checkerboard was the same. The luminance of the square targets from different checkerboards could take one of eleven possible values that spanned either 250 or 500 cd/m² of luminance range. In the 250 cd/m² luminance range, the luminance of the square targets varied from 500 to 250 cd/m² for dark stimuli, or from 500 to 750 cd/m² for light stimuli. In the 500 cd/m² luminance range, the luminance of the square targets varied from 500 to 0 cd/m² for dark stimuli, or from 500 to 1,000 cd/m² for light stimuli. The luminance was sampled at 25 cd/m² intervals in the stimulus sequences with 250 cd/m² luminance range and at 50 cd/m² intervals in the sequences with 500 cd/m² luminance range. Each stimulus trial lasted 6 s and contained four stimulus cycles. The checkerboard was turned on for 0.75 s and turned off for 0.75 s in each stimulus cycle. When the checkerboard was turned off, a uniform surface with the background luminance was shown. The checkerboards were presented in a sequence of 100 trials. Within each sequence, each checkerboard could have either light or dark square targets and the two checkerboard types were randomly presented. Each subject performed 400 trials (100 trials × 2 polarities × 2 luminance ranges) that contained 40 min of visual stimulation. Electroencephalographic recordings have a signal to noise much lower than recordings from spiking activity, which is a serious limitation when measuring responses to low contrast stimuli. Therefore, we used checkerboards (instead of the squares from the cat experiments) to generate the strongest cortical responses possible. It should be noticed that the electroencephalographic activity that we measured originates from the average of multiple cortical neurons with receptive fields in both fovea and visual periphery. Therefore, given the small receptive fields of foveal neurons in human visual cortex, the central square of the checkerboard stimuli (centered at the fixation point) was a large surface for many of the neurons contributing to the cortical response. At the time of the cortical measures, changes in checkerboard luminance did not drive strong pupil responses and, therefore, the luminance of the stimulus was not correlated with pupil size variations in any of the subjects (r ≤ 0.05 for each subject). The average variation in pupil size was also small (6% for the subject illustrated in Figure 5 and 8% across subjects). For consistency, the paper gives all the luminance values in cd/m² except in Figure 5H. However, the retinal illuminance (in trolands) can be easily calculated by multiplying the luminance of the stimulus (in cd/m²) times the area of the pupil. The retinal illuminance was ~7.4 mm² for the subject illustrated in Figure 5 (average pupil diameter: 3.1 mm), ~8.1 mm² for the subject illustrated in Figure S3 (average pupil diameter: 3.2 mm), and ~5.5 mm² for a third subject (average pupil diameter: 2.7 mm).

A stimulus trial was aborted and repeated if the subject blinked or interrupted fixation in the middle of a trial. We selected subjects that were able to perform the task with a minimum number of aborted trials. Maintaining visual fixation on the monitor through the entire task was very important in these measurements to avoid contaminating the luminance range with the dark room environment (i.e., if the eye moves out of the monitor many times, the average luminance range becomes determined not only by the monitor but also by the dark room). The stimuli were presented in the same high range monitor used for the experiments in cat visual cortex (TRU-Vu, SRMH-15-AR series, maximum luminance: 1400 cd/m²). The monitor was placed at 60 cm distance from the observer and the stimuli were generated with MATLAB Psychtoolbox.

To calculate the luminance response functions, the EEG signals were first processed with a low-pass filter of 5 Hz cut-off frequency and high-pass filter of 25 Hz cut-off frequency. We then calculated the current density (D) at the central position of the electrode array (between Oz and POz) by performing a voltage subtraction across the electrodes. The averaged voltage from the flanking electrodes, (VO1+VPO1+VO2+VPO2)/4, was subtracted from the average voltage of the central electrodes, (VOZ+VPOZ)/2, as shown in Equation 2.

$$D=\frac{(\text{VO}1+\text{VPO}1+\text{VO}2+\text{VPO}2)}{4}-\frac{(\text{VOZ}+\text{VPOZ})}{2}$$ (Equation 2)

The acquired signals were averaged over 30 trial cycles and the response measured as the maximum minus the minimum value within a time window of 0.05 to 0.3 s after the stimulus onset. At the time of the cortical measurement, the signal to noise was high and the variations in pupil size small. Trial cycles with signal amplitude higher than 50 microvolts were classified as recording artifacts and removed from the analysis (e.g., movements, muscle contractions). The average responses were fit with a Naka-Rushton function (Equation 1) to extract the parameters of the luminance response function.

ANALYSIS OF NATURAL IMAGES

To estimate the ON and OFF functions that best match the luminance variations in nature, we measured the cumulative density of light and dark pixels from 4,167 monochrome calibrated natural images (van Hateren and van der Schaaf, 1998). We then subtracted the median luminance of each image from all its pixel luminance values and classified the positive values as lights and negative values as darks. By choosing the median luminance as the classifier threshold, we obtained an equal number of pixels with light and dark values. However, similar results were obtained if we chose the mean instead of the median luminance as the classifier threshold. After classifying the pixel values, we computed the histogram of the luminance distribution for lights and darks in each image using a luminance bin of 1 cd/m². The luminance histograms were then transformed into cumulative luminance density functions by calculating the accumulated number of pixels with increasingly larger values of dark or light contrast (i.e., density of darks or lights). The light and dark density functions were then normalized between 0 and 1 to estimate the dark or light contrast associated with the half-maximum density (L50). As for cortical functions, we calculated the L50b as L50 minus the median image luminance (i.e., background luminance in cortical measurements). We also calculated the L50n as L50b divided by the luminance range (maximum minus minimum image luminance). To obtain reliable comparisons of maximum and minimum luminance across images, we truncated the sides of the luminance distribution from each image (top and bottom 2.5 percentiles) and then measured the maximum and minimum. The L50 of the luminance density function describes the accumulated density of dark or light luminance at half-maximum pixel density. The maximum darkest or lightest luminance describes the luminance driving the strongest cortical response (R100). As the eye scans an image, the average cortical response should be strongest when the receptive field moves from a region with median luminance (i.e., the most frequent luminance value in the image) to a region with darkest or lightest luminance. Instead, the average response should be weakest when the receptive field moves from a region with median luminance to a different region that has also median luminance. To convert pixel image intensity to luminance in cd/m², we used the conversion factor provided by van Hateren and van der Schaaf (1998) for each image (pixel luminance = pixel value × conversion factor). The conversion factor was calculated by van Hateren and van der Schaaf (1998) based on the camera settings (aperture, shutter time, and International Organization for Standardization ISO number). They used the image of a gray overcast sky simultaneously measured with a Minolta luminance meter as calibration.

To compare the ON and OFF image functions measured at different luminance ranges, we divided the images into three groups based on their luminance standard deviation: low (0–200 cd/m²), middle (200–400 cd/m²), and high luminance range (400–800 cd/m²). Since each image has a large number of luminance values (~1.5 million pixels), the standard deviation of the luminance distribution provided a more accurate measure to classify images with different luminance range than a two-pixel luminance difference. The standard deviation is also less vulnerable to the noise of the camera sensors and luminance measurements than the two-pixel luminance difference. We selected for this analysis only images with median luminance higher than 50 cd/m² to restrict our measurements to the photopic range and avoid errors associated with image measurements at low light. The ranges of standard deviations were selected to keep the sample size of ON functions relatively constant (617, 429 and 643 images for low, middle, and high luminance respectively) and have a reasonable minimum sample size of OFF functions per group (2,227, 534 and 253 images for low, middle, and high luminance respectively). Other ranges of standard deviations increased the sample size disparity across groups because the mean luminance standard deviation is lower for OFF than ON functions in nature (103 versus 665 cd/m² in this image dataset). The average maximum-minimum luminance range for the group of images with low/medium/high luminance was 246/781/1,550 cd/m² for ON functions and 233/836/1,630 cd/m² for OFF functions. For comparison, the sky luminance can range from 1,000 cd/m² at sunset to 15,000 cd/m² at noon in a bright day. The difference between the maximum and minimum luminance was strongly correlated with the luminance standard deviation in the image dataset (r = 0.99 for both ON and OFF).

ONOFF image processing

The luminance distribution of digital images generated by cameras, monitors or other devices needs to be modified with gamma functions or other algorithms to better simulate the images that we perceive. However, current algorithms are poorly related to biological function and frequently fail to simulate human vision, particularly in bright scenes. Here we introduce an ONOFF algorithm that modifies image luminance using simplified versions of the ON and OFF functions measured in visual cortex (without compromising processing speed). The ONOFF algorithm was tested with calibrated images in gray scale but it can also be used to process color images by adjusting the lightness dimension of the image in HSL color space (HSL: hue, saturation, lightness). As retinal photoreceptors, the first stage of our ONOFF algorithm converts different image luminance ranges into a common neuronal range (0 to maximum firing rate) after normalizing each image by its maximum luminance. As ON and OFF bipolar retinal cells, the second stage of the algorithm splits each image into ON and OFF images by selecting pixels with higher (ON) or lower values (OFF) than the ON and OFF backgrounds. The ON and OFF backgrounds are chosen as the 50−α and 50+α percentiles of the image maximum luminance, respectively with α determining the luminance overlap between the ON and OFF functions (set at 20 percent for all the measurements in this paper). Since the OFF cortical function is roughly linear, the algorithm maps the OFF luminance values with a linear function that spans from a minimum dark value of 0 to a maximum dark value of 1. Since the- ON cortical function saturates at high contrasts, the algorithm maps the ON luminance values with a piecewise linear function that saturates at the top β percent of the luminance distribution from each image (β was set to the 95 percentile of the luminance distribution in all the simulations illustrated in this paper). Notice that, because the luminance distribution can have a long tail at the highest values, the maximum luminance is frequently much larger than the luminance at the 95 percentile. As in visual cortex, the third stage of our ONOFF algorithm merges the ON (ION) and OFF (IOFF) images into the single image percept (I), as shown in Equation 3. Merging the ON and OFF images requires setting a midpoint weight (w) that specifies the relative ON and OFF response strength. The midpoint weight is calculated based on the number of ON (nON) and OFF pixels (nOFF) in the image, as shown in Equation 3.

$$\begin{gathered} I=W-I_{OFF}W+I_{ON}\left(1-W\right) \\ W=\frac{n_{OFF}}{n_{OFF}+n_{ON}} \end{gathered}$$ (Equation 3)

We used images from the van Hateren database (van Hateren and van der Schaaf, 1998) to compare the performance of our ONOFF algorithm with other algorithms that are commonly used in image processing analysis: contrast-limited adaptive histogram-equalization (CLAHE) and MATLAB Image adjustment (IMADJUST). CLAHE stretches the luminance distribution in multiple local regions of an image (local histogram equalization), and applies bilinear interpolation to eliminate the boundaries between local regions. It also limits the contrast within each local region to reduce noise. IMADJUST also stretches the luminance histogram while saturating the values at the two histogram sides by a specific percentage. To evaluate the performance of CLAHE and IMADJUST, we selected the parameters that generated the most natural images in both (CLAHE: 8 × 8 local tiles and contrast limit of 0.05; IMADJUST: 1% saturation). To compare the performance of the three algorithms, we calculated the power frequency spectrum from each image with a two-dimensional Fast Fourier Transform. We converted the frequency spectrum to spatial frequency in cycles per degree by using 1 min of arc per image pixel (van Hateren and van der Schaaf, 1998). We then calculated the difference between the logarithms of the power frequency spectrum from processed (Pprocessed) and original images (Poriginal), as shown in Equation 4. This subtraction estimates the contrast gain obtained with each method.

$$\log (\mathit{\text{gain}}(f))=\log \left(P_{\mathit{\text{processed}}}(f)\right)-\log \left(P_{\mathit{\text{original}}}(f)\right)$$ (Equation 4)

We also developed an ONOFF algorithm variation that adapts to changes in luminance range (e.g., from a natural image in the outside world to the same image displayed in a monitor or other device). Images in most conventional displays have much lower luminance range than in nature. Our results demonstrate that, as the luminance range decreases, the OFF/ON ratio of contrast sensitivities also decreases. Therefore, the ONOFF-adaptive algorithm iteratively adjusts the ON and OFF luminance ranges in the display to match the change in OFF/ON contrast sensitivity. The algorithm converges to this match in 20 iterations. At the first iteration, it subtracts the mean of the original image $\left(\overline{{\mathrm{I}}_{\text{t}}}\right)$ from all image pixels and assigns the positive pixel values to the ON image (IONt) and the absolutes of the negative pixel values to the OFF image (IOFFt). Then, it calculates an ON/OFF adapted luminance ratio (ALR) as the ratio between the ON and OFF luminance ranges (standard deviations) of the original image multiplied by an adaptation factor (Equation 5). The adaptation factor equals the ratio between the maximum luminance of the display (LDmax) and the maximum luminance of the original image (LImax) elevated to the power of 0.29, which is the adaptation slope measured in cat visual cortex (Figure 6H). The ON and OFF images are then weighted and combined as in the regular ONOFF algorithm but the weight is calculated based on the ON and OFF luminance ranges and ALR (w_t in Equation 5). Because the combined image is slightly different in each iteration (It), its mean is also different and the algorithm keeps generating slightly different ON and OFF images until the IONt/IOFFt luminance ratio converges to ALR at the 20^th iteration (R²: 0.93 across 4,167 images). Notice that, when LDmax equals LImax, ALR equals the ratio of luminance ranges in the original image because the adaptive factor (LDmax/LImax)^{^0.29} equals one. It is only when LDmax is smaller than LImax that the ON/OFF ratio of luminance ranges changes to match ALR.

$$\begin{gathered} ALR=\frac{std\left(IO{N}_t\right)}{std\left(IOF{F}_t\right)}{\left(\frac{\mathit{\text{LDmax}}}{\mathit{\text{Llmax}}}\right)}^{0.29}\{t=1\} \\ {w}_t=\frac{std\left(IO{N}_t\right)}{std\left(IO{N}_t\right)+ALR\ast std\left(IOF{F}_t\right)}\{1\le t\le 20\} \\ {I}_t=w_t+IO{N}_t\left(1-w_t\right)-IOF{F}_t{w}_t\{1\le t\le 20\} \end{gathered}$$ (Equation 5)

ONOFF VISUAL CONTRAST

We fit a linear model to the L50 measurements obtained from the cortical functions (Equation 6). The model was a weighted sum of three different terms applied to the luminance background (a), the luminance range (b) and the light-dark polarity of the stimuli (c). Then, the L50b was calculated as L50 minus the background and the L50n was calculated as L50b divided by the luminance range. As shown in the results, the background coefficients resulting from the fits to L50 were always one and the coefficients for the luminance range were always lower for light than dark stimuli.

$$L50=a\mathit{\text{Background}}+(b+c\mathit{\text{Polarity}})\mathit{\text{Range}}$$ (Equation 6)

We also fit the L50b and L50n of the image functions using two different linear models (Equation 7). The L50b was fit as a weighted sum of two different terms applied to the luminance range (b) and the light-dark polarity of the stimuli (c). Then, the ON L50 was calculated as L50b + background and the OFF L50 as background - L50b. The L50n was fit as a weighted sum of three terms: a constant (b) and two coefficients applied to descriptors of the image luminance distribution, kurtosis (c) and skewness (d).

$$\begin{gathered} L50b=(b+c\mathit{\text{Polarity}})\mathit{\text{Range}} \\ L50n=b+c\mathit{\text{Kurtosis}}+d\mathit{\text{Skewness}} \end{gathered}$$ (Equation 7)

We also tested different linear approaches to fit the exponent of the image function. The best fit was different for ON and OFF exponents and both were inversely correlated with luminance range (Drange for darks, Lrange for lights). The OFF exponent was strongly related to the OFF contrast (OFFc, mean difference between the luminance of each individual OFF pixel and the median OFF luminance). The fit for the ON exponent was poorly correlated with the image measures but seemed to be related to the background luminance (Bg, median luminance of ON pixels in Equation 8).

$$\begin{gathered} ON\mathit{\text{exponent}}=1+0.4Bg/\mathit{\text{Lrange}} \\ OFF\mathit{\text{exponent}}=-1.5+1.7\mathit{\text{OFFc}}/\mathit{\text{Drange}} \end{gathered}$$ (Equation 8)

These linear models revealed close relations between contrast sensitivity and the image luminance distribution. However, they could not accurately reproduce the L50 changes that we measured. To obtain a measurement of visual contrast that more closely reproduced our results, we used a Naka-Rushton function similar to the one used to fit our experimental data (Equation 9). In this function, VC is the ONOFF visual contrast, C is the contrast of the stimulus (calculated as stimulus luminance minus background luminance divided by the luminance range), C50 is the stimulus contrast that generates half-maximum perceived contrast (calculated as the C50 luminance minus background divided by the luminance range), n is the exponent of the function, and G is a gain correction factor that makes the response at 100% contrast (R100) equal to our measurements. As in our results, C50, n and G change with luminance range and stimulus contrast polarity. Therefore, the perceived ONOFF contrast also changes as a function of these stimulus parameters. It is important to notice that, without G, the ON contrast would be always higher than the OFF contrast because C50ⁿ is at the denominator of Equation 9 and both C50 and n have lower values in ON than OFF functions. Therefore, the gain parameter G is needed to make dark visual contrast stronger or equal than light visual contrast and match our R100 measurements in visual cortex.

$$VC=\frac{G\ast C^n}{C{50}^n+C^n}$$ (Equation 9)

To estimate the values of the C50, n, and G parameters, we fitted the Naka-Rushton function to the average contrast response functions measured in cat visual cortex at low (300 cd/m²) and high luminance ranges (1000 cd/m²) with dark and light stimuli. The fits (Equation 10) were very accurate for all stimulus conditions; the R² was 0.99 for dark visual contrast at 300 cd/m² (DVC indoors), 0.99 for dark visual contrast at 1,000 cd/m² (DVC outdoors), 0.99 for light visual contrast at 300 cd/m² (LVC indoors), and 0.98 for light visual contrast at 1,000 cd/m² (LVC outdoors). Notice that small changes in the parameters of Equation 9 can have a major effect on the measurements of ONOFF contrast. The measurements of VC indoors provide an approximation of the ONOFF contrasts experienced indoors or outdoors at night (e.g., < 500 cd/m²) whereas the measurements of VC outdoors provide an approximation of the ONOFF contrasts experienced outdoors at daylight (e.g., >500 cd/m²). Human psychophysical measurements with brighter monitors may allow us to approximate these parameters more accurately in the future.

$$\begin{gathered} DVC\mathit{\text{indoors}}=\frac{0.9\ast C^3}{{0.4}^3+C^3}LVC\mathit{\text{indoors}}=\frac{0.8\ast C^2}{{0.3}^2+C^2} \\ DVC\mathit{\text{outdoors}}=\frac{1.2\ast C^2}{{0.5}^2+C^2}LVC\mathit{\text{outdoors}}=\frac{1.1\ast C^2}{{0.3}^2+C^2} \end{gathered}$$ (Equation 10)

QUANTIFICATION AND STATISTICAL ANALYSIS

Statistical comparisons between central values of distributions (e.g., Figures 3, 4, and 6) were performed with two-sided Wilcoxon rank tests (MATLAB ranksum function). For comparisons between distributions with different sample size (e.g., Figures 3D–3H, 4D–4H, 5E, 6G, and 6H), the distribution with the largest sample size was resampled 50,000 times with the lower sample size of the other distribution. After resampling, we calculated the difference between the two distributions and obtained the 95% confidence intervals of the distribution difference (shown as error bars in Figures 3D–3H, 4D–4H, and 6G). We also calculated the probability that two distributions are different (p value) as the area of overlap between distributions divided by their sum.

KEY RESOURCES TABLE

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Experimental models: organisms/strains
Adult cats, Felis catus (4–7 kg)	Liberty Research, Inc., Waverly, New York, USA	N/A
Software and algorithms
MATLAB	MathWorks	R2016a
Psychtoolbox-3	Brainard, 1997	v3.0.12
Other
Custom 32-channel multielectrode arrays	NeuroNexus	N/A
OmniPlex Neural Recording Data Acquisition System	Plexon	N/A

Highlights.

• Changes in luminance affect differently light and dark contrast in visual cortex

• Light-dark differences increase with light intensity and should peak around midday

• Relations between contrast and luminance are similar in neurons and natural scenes

• A modified Naka-Rushton function provides the best metric of visual contrast