Human Visual Cortex Responses to Rapid Cone and Melanopsin-Directed Flicker

Abstract

Signals from cones are recombined in postreceptoral channels [luminance, L + M; red-green, L − M; blue-yellow, S − (L + M)]. The melanopsin-containing retinal ganglion cells are also active at daytime light levels and recent psychophysical results suggest that melanopsin contributes to conscious vision in humans. Here, we measured BOLD fMRI responses to spectral modulations that separately targeted the postreceptoral cone channels and melanopsin. Responses to spatially uniform (27.5° field size, central 5° obscured) flicker at 0.5, 1, 2, 4, 8, 16, 32, and 64 Hz were recorded from areas V1, V2/V3, motion-sensitive area MT, and the lateral occipital complex. In V1 and V2/V3, higher temporal sensitivity was observed to L + M + S (16 Hz) compared with L − M flicker (8 Hz), consistent with psychophysical findings. Area MT was most sensitive to rapid (32 Hz) flicker of either L + M + S or L − M. We found S cone responses only in areas V1 and V2/V3 (peak frequency: 4–8 Hz). In addition, we studied an L + M modulation and found responses that were effectively identical at all temporal frequencies to those recorded for the L + M + S modulation. Finally, we measured the cortical response to melanopsin-directed flicker and compared this response with control modulations that addressed stimulus imprecision and the possibility of stimulation of cones in the shadow of retinal blood vessels (penumbral cones). For our stimulus conditions, melanopsin flicker did not elicit a cortical response exceeding that of the control modulations. We note that failure to control for penumbral cone stimulation could be mistaken for a melanopsin response.

SIGNIFICANCE STATEMENT The retina contains cone photoreceptors and ganglion cells that contain the photopigment melanopsin. Cones provide brightness and color signals to visual cortex. Melanopsin influences circadian rhythm and the pupil, but its contribution to cortex and perception is less clear. We measured the response of human visual cortex with fMRI using spectral modulations tailored to stimulate the cones and melanopsin separately. We found that cortical responses to cone signals vary systematically across visual areas. Differences in temporal sensitivity for achromatic, red-green, and blue-yellow stimuli generally reflect the known perceptual properties of vision. We found that melanopsin signals do not produce a measurable response in visual cortex at temporal frequencies between 0.5 and 64 Hz at daytime light levels.

Introduction

In humans, signals originating in the short- (S), middle- (M), and long (L)-wavelength cones of the retina provide chromatic and luminance information (Stockman and Brainard, 2010). The L and M cones are summed (L + M) to form a luminance channel and differenced (L − M) to form a “red-green” chromatic pathway, whereas signals from S cones are opposed to the sum of L and M cones to create a “blue-yellow” chromatic pathway (Boynton, 1979; Fig. 1A).

Figure 1.

Overview and experimental design. a, Postreceptoral mechanisms arising from additive and opponent combination of cone signals. b, Spectral sensitivities of the L, M, and S cones and melanopsin. c, Left, Our stimuli all used a common, neutral background with the spectral power distribution shown. Right, Modulation spectra were added to the background. The L + M + S modulation is illustrated. Modulation spectra varied sinusoidally between an extreme positive (black line) and negative (red line) modulation. d, The maximal positive and negative modulations for each set of targeted photoreceptor(s) studied. Maximal contrast values are given in square brackets. Wavelength axis is as in c. e, Schematic of experimental apparatus and design. The subject viewed sinusoidal spectral modulations produced by the digital spectral integrator through an artificial pupil with a pharmacologically dilated eye. During fMRI scanning, different stimulus frequencies targeting a given class of photoreceptor(s) were presented in 12 s blocks. The sinusoidal flicker was ramped at the start and end of each block by a 3 s half-cosine. The stimulus was occasionally interrupted by an attention event.

The luminance and chromatic pathways differ in their temporal sensitivity as measured by psychophysics. The chromatic pathways have maximal temporal sensitivity at lower frequencies than the luminance pathway (Kelly, 1974, 1983), although the exact shape of the temporal contrast sensitivity functions depends on the spatial frequency (Robson, 1966; Kelly, 1979), mean luminance (De Lange, 1958; Kelly, 1961), and eccentricity of the stimulus (Kelly, 1984).

fMRI has also been used to study the temporal properties of the postreceptoral mechanisms. For example, Engel et al. (1997) found strong BOLD responses to L − M flicker in areas V1 and V2 for flicker between 1 and 10 Hz, whereas responses to S-directed flicker were reduced at 10 Hz. They found similarities in the temporal dependence of the BOLD response and psychophysical results for stimuli at 1 and 4 Hz, whereas, at 10 Hz, the relative response to L − M flicker in V1 and V2 was larger than psychophysics would predict.

Recent discoveries in the biology of the retina lead us to revisit these findings. Melanopsin is a light-sensitive molecule expressed in some retinal ganglion cells, rendering them intrinsically photosensitive. The melanopsin-containing, intrinsically photosensitive ganglion cells (ipRGCs) also receive synaptic inputs from rods and cones, thus combining signals from the inner and outer retina. A major projection of the ipRGCs is to brainstem and hypothalamic sites, where they influence pupil response (Lucas et al., 2003; Gamlin et al., 2007; Tsujimura et al., 2010; Spitschan et al., 2014a) and circadian rhythm (Berson et al., 2002). The ipRGCs project as well to the lateral geniculate nucleus (Dacey et al., 2005; Brown et al., 2010; Brown et al., 2012; Allen et al., 2014) and recent studies suggest that humans are able to perceive variation in melanopsin contrast as variation in brightness (Zaidi et al., 2007; Brown et al., 2012).

This raises the possibility that prior studies of psychophysical and fMRI cone temporal sensitivity include a contribution of melanopsin-mediated signals because the spectral sensitivity of melanopsin overlaps extensively with that of the cones (Fig. 1b). Although the responses of the ipRGCs are notably delayed and prolonged (Dacey et al., 2005), these cells are capable of more rapid signaling. As stimulus intensity rises, the ipRGCs manifest initial transient responses that peak between 200 and 2000 ms (Do et al., 2009). Recent studies have raised the possibility that humans are able to perceive rapid melanopsin stimulation. For example, subjects were able to detect cone-silent flicker at rod saturating light levels in the visual periphery up to 40 Hz (Horiguchi et al., 2013).

Here, we used fMRI in conjunction with cone-directed, spatially uniform flicker to measure the temporal sensitivity profiles of the visual cortex representation of cone and melanopsin signals. Using a digital spectral device with 56 effective primaries (Spitschan et al., 2015), we constructed cone-directed modulations while controlling for activation of melanopsin. We measured cortical temporal transfer functions for the fMRI response to these modulations presented at logarithmically spaced flicker rates between 0.5 and 64 Hz. We then constructed a modulation that targeted melanopsin while controlling for activation of the cones. We examined the perceptual properties of this modulation and its ability to drive neural and pupil responses. Cortical responses to this stimulus were compared with those from control modulations that accounted for imprecision in our stimulus specification.

Materials and Methods

Subjects

Three male subjects (age 43, 26, and 28 years; subjects S01, S02, and S03; the first two are authors of this study) participated in the fMRI experiments. These three primary subjects and 12 additional subjects (8 male, 4 female; age = 30 ± 2 years; subjects S04–S15) participated in perceptual nulling and pupillometry experiments. The data from one additional subject who participated in the nulling and pupillometry experiments were discarded because of poor pupil tracking. All subjects had corrected-to-normal acuity of 20/40 or better and normal color vision (Ishihara, 1977). The research was approved by the University of Pennsylvania Institutional Review Board and conducted in accordance with the principles of the Declaration of Helsinki. All subjects gave informed written consent.

Visual stimuli

Digital light synthesis and silent substitution.

We used the method of silent substitution with a digital light synthesis engine (OneLight Spectra) to stimulate targeted photoreceptors. Our device produces stimulus spectra as mixtures of 56 independent, ∼16 nm full-width half-maximum primaries under digital control and can modulate between these spectra at a rate up to 256 Hz. In the fMRI experiments, we used a refresh rate of 256 Hz, whereas, in the pupillometry experiments, we used a refresh rate of 64 Hz. Details regarding the device, stimulus generation, and estimates of precision may be found in Spitschan et al. (2015). Our estimates of photoreceptor spectral sensitivities were as described in Spitschan et al. (2015) and account for the size of the visual field and subject age. Cone spectral sensitivity adjustments for photopigment bleaching were made for all modulations except those used in BOLD fMRI Experiment 1.

All stimuli were modulations around a common, rod-saturating background (Fig. 1c; chromaticity x = 0.40, y = 0.43). Background luminance was 3700 cd/m² in BOLD fMRI Experiment 1, 3000 cd/m² in perceptual nulling and pupillometry, and 2900 cd/m² in BOLD fMRI Experiment 2. Modulations were between positive and negative spectra designed to increase and decrease excitation of the targeted photoreceptor(s) and produced sinusoidally modulated contrast relative to the background. The spectra are designed such that excitations in “silenced” photoreceptors are held constant. The maximum contrast available upon the targeted photoreceptor(s) is limited by the degree of spectral overlap of the photoreceptors, the gamut of the device, and the chromaticity of the background. Several different modulations were studied as part of the BOLD fMRI and pupillometry studies (Fig. 1d, Table 1).

Table 1.

Predicted contrasts of the modulations

	L + M + S (±45%)	L − M (±10%)	S (±45%)	L + M (±45%)	L + M (±2%)	Mel (±17)	L* + M* (±2%)	Mel* (±32%)	Light flux (32%)
L	45%	10%	0%	45%	2%	0%	0%	0%	32%
M	45%	−10%	0%	45%	2%	0%	0%	0%	32%
S	45%	0%	45%	0%	0%	0%	0%	0%	32%
Mel	0%	0%	0%	0%	0%	17%	0%	32%	32%
Rods	17%	−10%	0%	17%	1%	10%	3%	20%	32%
L*	42%	14%	0%	42%	2%	0%	2%	2%	32%
M*	43%	−7%	0%	42%	2%	0%	1%	2%	32%
S*	38%	0%	36%	0%	0%	0%	0%	7%	32%

Values are given as contrasts of the modulations targeted to a 32-year-old subject (spectral power distributions are shown in Figure 1). Bolded numbers indicate contrast that was targeted for the respective modulation direction; italic numbers indicate contrast that was not specifically controlled. Modulation names designated with an asterisk produce differential spatial contrast on penumbral cones and thus a Purkinje tree percept if flickered rapidly.

Modulations studied in BOLD fMRI Experiment 1.

We measured visual cortex responses to seven different photoreceptor-directed modulations at a range of temporal frequencies. A set of cone-directed stimuli targeted L, M, and S cones with equal contrast (L + M + S, 45% contrast), a red-green chromatic channel (L − M, 10% contrast), S cones (45% contrast), and L and M cones in isolation (L + M, 45% contrast). A melanopsin-directed modulation (Mel, 17% contrast) was also studied. This modulation was designed to stimulate melanopsin and to silence, not only the cones with their expected spectral sensitivities, but also those cones that have (effectively) altered spectral sensitivity as a consequence of being shadowed by larger retinal blood vessels (Spitschan et al., 2015). We designate these “penumbral” cones that receive hemoglobin-filtered light with an asterisk (i.e., L* indicates L-cones in the shadow of blood vessels). A rapid (8–16 Hz) melanopsin-directed modulation that does not silence the penumbral cones produces an entopic percept of the branching retinal blood vessels (Spitschan et al., 2015).

A primary concern in studies of this kind is that inadvertent contrast upon the nominally silenced classes of photoreceptors might produce cortical responses that are mistakenly attributed to the action of the targeted photoreceptor class or classes. We have estimated that this undesired photoreceptor contrast “splatter,” which can arise from biological variability, misspecification of the cone fundamentals, or imperfections in device control, is on the order of a few percent in our stimulus system (Spitschan et al., 2015). We examined two control modulations in BOLD fMRI Experiment 1 to address the contribution of contrast splatter from various sources. A scaled (2% contrast) version of the L + M stimulus was selected to correspond to the uncertainty present in our silencing of photoreceptors. Therefore, the cortical response to a cone-directed modulation at this contrast level provides a threshold for “stimulus noise” in our BOLD fMRI measurements of wide-field photoreceptor-directed flicker. We also measured the cortical response that can arise from isolated stimulation of penumbral cones. A modulation was constructed that selectively targeted the L* and M* cones with 2% and 1% contrast, respectively, while silencing the L, M, and S cones as well as melanopsin (Spitschan et al., 2015). We refer to this as the 2% L* + M* modulation.

Modulations studied during pupillometry.

We measured the pupil response to slow (0.1 Hz) spectral modulations. Data were collected from all subjects using cone- (L + M + S; 32% contrast) and melanopsin (Mel*; 32% contrast)-directed modulations. The Mel* modulation was designed to silence the cones but not their penumbral variants because the slow modulation studied for pupillometry does not produce a percept of the retinal blood vessels. In earlier work, we have shown that the pupil response to a melanopsin modulation is not appreciably altered by the silencing of the penumbral cones (Spitschan et al., 2014a, supplemental Figure S4b). These modulations were tailored for each subject during a psychophysical calibration session (nulling procedure, described below) to remove residual chromatic and (in the case of the melanopsin modulation) luminance components.

In addition to these two modulations, the three primary subjects also were studied with a light flux modulation (32% contrast).

Modulations studied in BOLD fMRI Experiment 2.

Four additional modulations were examined at a single temporal frequency (4 Hz) during BOLD fMRI scanning for the 3 primary subjects. Immediately before scanning, each subject conducted the psychophysical nulling procedure upon the 17% contrast penumbral-cone silent Mel stimulus. The chromatic and luminance nulling values for each subject were noted. During scanning, the subject was presented with the same, un-nulled Mel modulation used in BOLD fMRI Experiment 1, as well as their individualized, perceptually nulled Mel modulation. The difference between the nulled and un-nulled Mel modulations constituted a third modulation condition, allowing measurement of the cortical response to the inadvertent cone contrast present in the un-nulled Mel stimulus. Finally, a high-contrast (90%) light flux modulation was included as a positive control.

Psychophysical calibration (nulling).

In informal observations, we noted that, although the melanopsin-directed and L + M + S-directed modulations theoretically silence postreceptoral cone opponent mechanisms, both melanopsin and L + M + S modulations produced reddish-greenish changes in color appearance for most of our subjects. We considered the most likely cause of these change to be differential contrast splatter onto the L and M cones. Because human vision is highly sensitivity to L − M modulations (Chaparro et al., 1993), not much residual contrast is required to produce a perceptually visible color change. Although contrast splatter is not the only possible explanation for a perceived color change in response to our melanopsin modulation (e.g., melanopsin could in principle contribute to perceptual redness), we sought to measure the putative cone splatter using a psychophysical nulling procedure.

For the melanopsin-directed modulation, subjects were asked to null luminance and red-green color appearance changes. For the L + M + S-directed modulation, subjects nulled only the red-green changes. Luminance nulling was realized by alternating the positive or negative excursions of the modulation spectra with the background spectrum in a square-wave fashion at 30 or 40 Hz pulsed for 1 s (40 Hz for S01–S03; 30 Hz for S04–S15). Subjects were then instructed to add or subtract (nominal) L + M + S contrast using a response pad until the flicker was minimally visible. Red-green nulling was realized by presenting the positive or negative modulation spectra as 500 ms steps from the background, whereas subjects were asked to remove the red-green appearance by adding or subtracting (nominal) L − M contrast to the modulation spectra. In pilot experiments, we also examined nulling blue-yellow appearance by adding or subtracting (nominal) S cone contrast. We found that our subjects did not perform this task reliably in our initial attempts and so did not further pursue blue-yellow nulling.

Subjects nulled the positive and negative excursions of the melanopsin-directed and L + M + S-directed modulations independently. In any given nulling trial (i.e., melanopsin-directed positive, melanopsin-directed positive, L + M + S-directed positive, L + M + S-directed negative), subjects nulled luminance and red-green in alternation (melanopsin trials) or red-green only (L + M + S trials) until no more change was made. The “nulled” modulation spectra were then averaged across the positive and negative arms to obtain an average “nulled” modulation. These nulled subject-specific modulations were then used in the pupillometry experiment and in fMRI Experiment 2.

Eye piece and pupil dilation.

The stimulus was viewed within an MRI-compatible eye piece that provided a circular, uniform field of 27.5° diameter. The central 5° diameter was obscured to avoid variation in photoreceptor spectral sensitivity across the visual field caused by the presence of the foveal macular pigment (CIE, 2006). Grid marks (vertical, horizontal, and two annular) were present to assist the subject in maintaining fixation at the center of the stimulus field. Pupil dilation was achieved with 0.5% proparacaine hydrochloride as a local anesthetic followed by 1% tropicamide. In BOLD fMRI Experiment 1, subjects viewed the stimulus with their pharmacologically dilated eye through an artificial pupil (4.7 mm); the pupillometry, perceptual nulling, and BOLD fMRI Experiment 2 used pharmacological dilation but did not use an artificial pupil. For these experiments, the pupil diameter was taken to be 4.7 mm for purposes of computing the amount of cone photopigment bleaching to be used in correction of cone spectral sensitivities. A patch was worn over the unstimulated eye during fMRI scanning.

BOLD fMRI

Experimental design.

During BOLD fMRI Experiment 1, subjects viewed photoreceptor-directed modulations flickering sinusoidally at different frequencies (0.5, 1, 2, 4, 8, 16, 32, and 64 Hz; Fig. 1e). During each of many scanning runs of 588 s, a single photoreceptor directed modulation was studied. Each scanning run was composed of multiple, 12 s blocks during which the modulation was presented at a particular temporal frequency and the order of presented frequencies was counterbalanced within a run. Within each block, the sinusoidal profile of the flicker was ramped on and off using a 3 s cosine ramp. Each run included blocks in which only the static background spectrum was presented; these periods served as the reference condition in data analysis. The subject adapted to the background spectrum for 10–15 min at the start of each session during collection of anatomical images and was returned to the background spectrum between scanning runs. Scanning was conducted for each subject during three separate sessions. During the first session, we examined stimulus frequencies between 2 and 64 Hz (2 runs each for L + M, L − M, S, L* + M* and scaled L + M, and 3 runs for melanopsin-directed). In the second session, we probed stimulus frequencies 0.5, 1, and 2 Hz (2 runs each for L + M, L − M, S, and scaled L + M, 3 runs for melanopsin-directed). In the third session, we examined the L + M + S modulation for the entire range of frequencies (2 runs per frequency set).

The design of BOLD fMRI Experiment 2 was the same as Experiment 1 except that a single modulation frequency (4 Hz) was used and the four modulation types were intermixed in a single scan in a counterbalanced order.

Attention task.

Subjects performed a detection task during scanning. During each 12 s block, there was a 33% probability of a brief (250 ms) interruption during which the stimulus would transition to a static presentation of a lower luminance (∼50% reduced) background. The target events were constrained so as not to occur during the first and last 2 s of the block. Subjects were asked to report these brief dimming events by pressing a button on a keypad. If the subject did not detect the dimming of the stimulus, then we assumed that the 12 s block was invalid and did not include these blocks in further analyses.

fMRI acquisition and data analysis

Data acquisition.

We collected echoplanar (EPI) BOLD fMRI data on a 3 T Siemens Trio using a 32-channel array coil at a TR of 3 s, with 2 mm isotropic voxels over 34 oblique axial slices (TE = 30 ms, FA = 90°). The imaging slab was centered over the occipital pole. A high-resolution anatomical image (3D MPRAGE) (160 slices, 1 mm isotropic voxels, TR = 1.62 s, TE = 3.09 ms, TI = 950 ms, FOV = 250 mm, FA = 15°) was acquired for each subject at the start of each scan. Magnitude and phase B0 maps were also collected (52 slices, TR = 1.2 s, TE 1 = 4.06 ms, TE 2 = 6.52 ms, FA = 60°).

Preprocessing.

fMRI data processing in volumetric space was performed using FEAT (FMRI Expert Analysis Tool; version 6.00, part of FSL). Motion correction relative to the midpoint reference image of the 4D BOLD data was performed using FSL's MCFLIRT. B0 unwarping was performed within FSL using the FUGUE tool.

Regression.

The data from each scanning run were modeled separately. Each modulation frequency across blocks was modeled with a separate square wave regressor relative to the static background blocks. Blocks in which the subject failed to detect the attention target were modeled with a separate, nuisance regressor; this was an infrequent occurrence (a total of 4/438, 4/473, and 21/438 blocks missed by subjects S01, S02, and S03, respectively). An additional regressor of no interest modeled the occurrence of the attention target as a delta function. These regressors were convolved with a double-gamma hemodynamic response function. Motion outlier volumes were identified using the root-mean-squared intensity difference of each volume to a reference volume (threshold 75^th percentile + 1.5 interquartile range; FSL tool fsl_motion_outliers) and modeled as nuisance regressors in the form of a delta function. In the first-level analysis, a volumetric map of the response (expressed as percentage BOLD fMRI signal change) was obtained for each stimulus regressor of interest for each run.

For each of the cortical regions of interest (described below), we obtained the response for a given stimulus frequency by averaging the response across voxels in that area. Within subject, we obtained mean response across runs and hemispheres and also the average across subjects. The set of the responses across frequencies constitutes the temporal transfer function.

Registration.

T1 structural image volumes were segmented into white and gray matter and flattened and inflated with FreeSurfer. Volumetric, voxelwise maps of regressor weights were first registered to the subject anatomy, which was in turn coregistered to the fsaverage_sum standard space. The registration of the EPI slab volume to anatomy was performed using Advanced Normalization Tools (ANTs; http://stnava.github.io/ANTs/) and bbregister (FreeSurfer) as follows. The EPI slab reference image was registered to FreeSurfer anatomy using boundary-based linear registration. The registered EPI slab scout image was then binarized to create a binary mask for the anatomy, thus creating a corresponding slab anatomical image. We then used diffeomorphic registration in ANTs to map functional runs to the masked partial FOV anatomy, which was coregistered to the whole-brain anatomy. This procedure was performed for every run, which had a characteristic reference image. In this dataset, we found that using diffeomorphic registration outperformed boundary-based registration based on visual inspection.

Definition of ROIs.

We defined the cortical location of visual areas V1, V2, and V3 using an anatomically informed template of retinotopic organization (Benson et al., 2014). We restricted the ROIs to the eccentricity range of 3–13° (unilateral eccentricity) to correspond to the visual extent of our stimulus, with a 1° inner margin around the centrally occluded spot. Because we found similar responses in areas V2 and V3, we averaged the data from these two regions.

The scanning sessions included runs that served as functional localizers. To identify the MT complex, subjects viewed a circular field of static or moving random dot stimuli (white dots on black background) that alternated every 15 s. To identify the lateral occipital complex (LOC), subjects viewed 15 s blocks of presentations of pictures of faces, buildings, general objects, and scrambled objects (1 s per picture). The LOC was identified by the contrast of objects with scrambled objects. These localizer stimuli were presented using an uncalibrated LCD projector (Sanyo SXGA projector, 4200 lumens) with a Buhl long-throw lens for projection on a Mylar screen and viewed by the subjects via a mirror mounted on the head coil in the scanner. The functional localizers were analyzed for each subject using the same processing pipeline as described above. After combining hemispheres and runs, we identified statistically significant voxels for these contrasts in FreeSurfer (cluster-wise p-value threshold <0.01) for each subject. We then constructed a common ROI for the group of three subjects by selecting vertices in which at least two of the three subjects had significant responses to the motion (MT) or object (LOC) stimuli, respectively.

Missing volumetric data.

In some scans for some subjects, the position of the imaging slab was too superior or inferior, causing dorsal or ventral regions of the visual areas within our target eccentricity range to be incompletely imaged. Because signal artifacts arise at the edge of the imaging slab due to head motion, we elected to identify and remove the data from the runs and regions in which this partial imaging occurred. For subject S01, we discarded data from 6/28 of the runs in V2d (left hemisphere), 12/28 of V2d (right hemisphere), 7/28 of V3d (left hemisphere), and 28/28 of V3d (right hemisphere). For subject S02, we discarded data from 6/28 of the runs in V1d (left hemisphere), 21/28 of V1d (right hemisphere), 9/28 of V2d (left hemisphere), 24/28 of V2d (right hemisphere), 24/28 of V3d (left hemisphere), and 24/28 of V3d (right hemisphere). For subject S03, we discarded 11/28 runs of the data from V3d (left hemisphere) and 11/28 runs of the data from V3d (right hemisphere).

Pupil measurements

The approach used to make these measurements has been described in detail previously (Spitschan et al., 2014a). A brief description follows.

Viewing geometry.

Subjects viewed stimuli through the eyepiece and visual stimulation set-up described above (see “Visual stimuli”), whereas the consensual pupillary response of the unstimulated eye was measured using an infrared eye tracker (Video Eye Tracker, sampling rate 50 Hz; Cambridge Research Systems). The pupil detection algorithm supplied in the Video Eye Tracker Toolbox was used and further processing was performed offline. For consistency, the three main subjects, who also took part in the fMRI experiments (S01, S02, and S03), viewed the stimulus with their right eye while their left eye was tracked, whereas the additional 12 subjects viewed the stimulus with their left eye while their right eye was tracked. As noted above, the stimulated eye was pharmacologically dilated.

Procedure.

After adapting to the mean background for at least 5 min, subjects viewed sinusoidal, cosine-windowed (3 s) stimulus modulations at 0.1 Hz in a counterbalanced order with randomized phase (0°, 90°, 180°, or 270°). During each 45 s trial per run, the stimuli would either be melanopsin directed (Mel*, 32% contrast) or L + M + S directed (32% contrast). The three primary subjects (S01, S02, and S03) also had trials in which a light flux (32% contrast) modulation was presented. The melanopsin-directed and L + M + S-directed modulations were individually nulled in chromaticity and brightness (melanopsin-directed) or only chromaticity (L + M + S-directed) by each subject [see “Psychophysical calibration (nulling)” above]. Subjects S01–S03 took part in two runs of 27 trials each (9 trials each of L + M + S-, melanopsin-directed, and light flux), yielding a total of 18 repeats for each stimulus modulation direction. Subjects S04-S15 took part in three runs of 24 trials each (12 trials each of L + M + S- and melanopsin-directed), yielding a total of 36 repeats for each stimulus modulation direction. Subjects initiated each trial with a button press.

Data analysis.

Raw traces of pupil diameter were smoothed and resampled using a seventh-order polynomial Savitzky–Golay filter. Missed pupillary samples due to blinks or eye movements were identified using a “spike detection” algorithm (2 SD from mean pupil size or 20% overall within a 10-sample window). Trials with >10% of invalid samples were discarded altogether. The first 5 s of each 45 s trial were removed from the data. Amplitude and phase of the pupillary responses to the sinusoidal stimuli were obtained by fitting sine and cosine waves to the average response for a given condition at the stimulation frequency (0.1 Hz) using least-squares spectral fitting. The stimulus phase jitter used in the experiment was accounted for when averaging responses across trials. The SEM of the amplitude and phase values was estimated using a bootstrap procedure in which trials were resampled randomly with replacement and amplitude and phase obtained from each set of resampled trials using the same method applied to the actual trials. The SD of the bootstrapped values was then used as the SEM.

Results

Responses to whole-field, cone-directed flicker

We measured the cortical temporal transfer functions between 0.5 and 64 Hz for flicker that targeted the L + M + S, L − M, and S cone pathways while silencing melanopsin (Fig. 2A). We studied three primary subjects in detail over several fMRI sessions and found consistent responses across these subjects (Fig. 2, insets).

Figure 2.

Temporal transfer functions for the postreceptoral cone pathways. a, Temporal transfer functions for L + M + S, L − M, and S. The shaded region indicates ±1 SEM across the three subjects. The solid line is a fourth-order polynomial fit to the data. Views for ROIs at left are medial for V1 and V2/V3, lateral for MT, and ventral for LOC. Insets provide the response profile for each subject. b, Temporal transfer functions for L + M. The dashed red line is the difference of the L + M + S and S temporal transfer functions scaled to best (least-squares) match the L + M modulation.

For the L + M + S direction (Fig. 2a, left), we found a band-pass response that increased in peak frequency between area V1 (∼16 Hz) and the extrastriate areas LOC and MT (∼32 Hz). In the L − M direction, a more broadband response was observed that also increased in frequency of peak sensitivity between area V1 (8–16 Hz) and area MT (32 Hz), but not LOC (Fig. 2a, center). The S-cone stimulus in general elicited a lower amplitude response and a broad, band-pass structure between 2 and 8 Hz in areas V1 and V2/V3 (Fig. 2a, right). We did not find measurable S cone responses in areas LOC or MT.

An L + M-directed stimulus is sometimes used to examine the cortical representation of the luminance pathway (Engel et al., 1997; Liu and Wandell, 2005). Although this is an effective stimulus for the luminance pathway, it also has the property of producing negative loading upon the blue-yellow mechanism and thus a chromatic percept. We considered the possibility that the cortical response to an L + M modulation would resemble a linear combination of the responses to the L + M + S and the S stimuli. We measured the cortical temporal transfer function to an L + M-directed stimulus. Interestingly, the cortical response to the L + M modulation (Fig. 2b) was effectively identical to that evoked by L + M + S-directed flicker. This was the case even in areas V1 and V2/V3, where a substantial S-cone-driven response is measured when an S-cone modulation is presented in isolation.

The failure of a simple additive combination of postreceptoral channel responses to account for the results is illustrated by the dashed line in Figure 2b. The measured temporal transfer function for the S-cone-directed stimulus was subtracted from the function obtained for the L + M + S stimulus. This predicted response function does not resemble the measured response to the L + M modulation in areas V1 and V2/V3, the areas where we observed a nonzero response to the S-cone-directed stimulus.

Psychophysical measurement of apparent cone contrast within a melanopsin-directed modulation

In informal observations, we noted that the melanopsin-directed modulation had a red-green chromatic component despite being nominally cone silent. We characterized the perceptual properties of melanopsin-directed modulation in terms of apparent cone contrast using a psychophysical nulling procedure. Red-green (L − M) contrast was measured using a 0.5 Hz melanopsin-directed modulation, whereas luminance (L + M + S) contrast was measured using rapid flicker (30 or 40 Hz depending on subject).

Across 15 subjects (Fig. 3a), slightly less than 2.5% chromatic (L − M) contrast was on average required to null red-green from the melanopsin modulation; an increase in un-nulled melanopsin contrast was seen as reddish and a decrease in melanopsin contrast as greenish, with this difference reflected in the sign of the nulling contrast obtained for the positive and negative arms. A similar degree of luminance (L + M + S) contrast on average was removed from the Mel modulation to null the cone flicker component. There was modest variation in these measures across subjects. Our three primary subjects had nulling values that were typical of the larger studied population.

Figure 3.

Psychophysical nulling. a, Perceptual nulling data for a positive (black points) and negative (red points) 32% melanopsin (Mel*; nonpenumbral-cone silent) modulation in a population of subjects (n = 15). Primary subjects (S01, S02, and S03) are indicated with a star symbol. Some plot points are overlapping. Ellipse indicates ±1 SD across subjects. b, Perceptual nulling data for a 32% cone-directed (L + M + S) modulation in the same population of 15 subjects. Error bars indicate ±1 SD across subjects.

We note that our psychophysical nulling does not render the nulled melanopsin modulations completely invisible at low temporal frequencies. This may be the result of imperfect nulling (as could be the case either because of nulling variability or if there are retinal inhomogeneities in cone spectral sensitivities) or it could represent a subtle contribution of melanopsin to perceived brightness. We do not consider this issue further here.

The chromatic and luminance percept in the un-nulled melanopsin-directed stimulus could be a consequence of melanopsin stimulation itself or of inadvertent contrast upon cones due to imperfect spectral silencing. In support of the latter view, we measured a similar degree of chromatic content within the nominally achromatic L + M + S modulation (the positive arm of the un-nulled L + M + S modulation appeared greenish and vice versa for the negative arm; Fig. 3b). Although we consider it a priori unlikely that melanopsin stimulation itself is responsible for the measured chromatic and luminance components, we cannot discount this possibility completely. Therefore, we examined the neural response both to the un-nulled melanopsin-directed stimulus (fMRI Experiment 1) and to both the un-nulled and nulled melanopsin-directed stimulus (fMRI Experiment 2). We also adopted the strategy of comparing the amplitude of response to the melanopsin-directed stimuli with the response produced by control stimuli consisting of low-contrast, cone-directed modulations, with complementary instantiations of this strategy implemented in the two experiments. A neural response to the melanopsin modulation greater than that evoked by the control stimuli and not eliminated by the psychophysical nulling might reasonably be interpreted as resulting from melanopsin stimulation itself and not from inadvertent cone contrast.

Visual cortex responses to melanopsin flicker and control conditions

We tested whether flicker directed at melanopsin produces a measurable cortical response. In the first set of measurements, we used an un-nulled 17% contrast melanopsin modulation. Seventeen percent contrast was the maximum available with our device given our choice of background and the requirement to nominally silence both the L, M, and S cones and their counterparts shadowed by retinal blood vessels (the penumbral cones, L*, M*, and S*). Silencing of the penumbral cones is necessary because rapid flicker of a melanopsin-directed modulation that does not do so produces a structured, entopic percept of the retinal blood vessels (Spitschan et al., 2015). Such an entopic percept could itself produce a cortical response and confound our measurement of the response to melanopsin, a possibility that we address explicitly in a later control measurement.

Minimal responses to the 17% melanopsin-directed stimulus were found across visual areas and temporal frequencies (Fig. 4a). Within area V1, there was the suggestion of a band-pass pattern of responses, with the largest average response found for the 4 Hz stimulus. In the other visual areas, the average response did not exceed zero by appreciably more than 1 SEM at any temporal frequency. We considered the possibility that the small measured response in area V1 was the result of inadvertent cone contrast. To address this possibility, we measured cortical responses to a low-contrast (2%) L + M control modulation. We found that the cortical response to this control modulation was of small but consistently positive amplitude across the studied cortical regions (Fig. 4b), although it had no systematic structure over temporal frequency. We regard the BOLD responses to this control modulation as the degree of measured response that might be attributed to inadvertent cone contrast. Importantly, the response evoked by the melanopsin modulation did not reliably exceed the upper limb of the ±1 SEM confidence interval for the response to the low-contrast control cone modulation at any frequency in any visual area.

Figure 4.

Temporal transfer functions for melanopsin-directed (a) and cone control flicker (b). Same format as Figure 2. Dotted red horizontal lines indicate the peak response (+1 SEM) for the scaled (2% contrast) L + M modulation for each area.

The largest candidate response to the melanopsin modulation was found for 4 Hz flicker in primary visual cortex (V1). We examined this stimulus condition in greater detail in an additional BOLD fMRI study (fMRI Experiment 2). A high-contrast, light flux modulation at 4 Hz served as a positive control and evoked a robust response within V1 cortex (Fig. 5). Instead of using a fixed degree of cone contrast as a control for inadvertent cone stimulation, we measured immediately before scanning the L + M + S and L − M contrast needed to produce a psychophysical null of the 17% contrast, penumbral-cone silent melanopsin modulation for each subject. The nulling cone contrast was then presented as a 4 Hz control modulation to the subject. On average, this low-contrast (1–2%) cone flicker produced a small, positive V1 response. We repeated the measure of V1 cortical response to the 4 Hz, un-nulled 17% contrast melanopsin-directed modulation. The measured response was smaller than that evoked by the cone contrast control and did not differ from zero in this replication. Finally, we created a nulled 17% contrast melanopsin modulation for each subject based on his or her psychophysical nulling. When presented at 4 Hz, this modulation also failed to produce a V1 response different from zero.

Figure 5.

V1 response to melanopsin and control modulations. BOLD amplitudes shown as average across the two V1 hemispheres and across V1 vertices in the relevant eccentricity range (inset) for a set of 4 Hz modulations.

In sum, our studies did not identify a reliable visual cortex response to melanopsin-directed flicker with 17% contrast at any frequency between 0.5 and 64 Hz. Attempts to replicate the small response initially observed at 4 Hz in area V1 failed. Cortical responses to these stimuli did not differ from the BOLD fMRI response to the static background spectrum and were smaller than the cortical response to a small degree of cone contrast.

Pupil response to a slow, nulled melanopsin modulation

Rapid, melanopsin-directed flicker did not evoke cortical responses for the three subjects studied using fMRI. We used measurements of the pupil to confirm that is possible to evoke a melanopsin-driven visual response in these subjects and also to compare their melanopsin-driven response with that of a larger sample of subjects. Because the melanopsin-directed stimulus was not designed to silence the penumbral cones in this experiment, a larger stimulus contrast upon melanopsin was available; it was not necessary to silence the penumbral cones because selective stimulation of these photoreceptors with a slow modulation does not produce a visible percept and does not alter the measured pupil response (Spitschan et al., 2014a). Fifteen subjects viewed slow (0.1 Hz) 32% contrast, nonpenumbral cone silent melanopsin-directed and 32% contrast L + M + S-directed stimuli during measurement of their consensual pupil response (Fig. 6a). These stimuli were perceptually nulled for each subject as described above. Across subjects, a measurable pupil response to the melanopsin modulation was observed and found to be of a different phase than that evoked by the L + M + S modulation, consistent with prior reports (Barrionuevo et al., 2014; Spitschan et al., 2014a).

Figure 6.

Pupil response to 0.1 Hz modulations. a, Polar plot of pupil responses (0.1 Hz modulation frequency) in 15 subjects. Radial eccentricity indicates pupil amplitude; angle in the polar plots indicates phase of the pupil response. Primary subjects (S01, S02, and S03) are indicated with a star symbol. b, Amplitude of pupil responses, including a light flux modulation, for the three primary subjects.

The pupillometry measures obtained on the three primary subjects also included a 32% light flux modulation. This modulation is simply a scaling of the entire background spectrum, and as such produces equal contrast on both the cones and melanopsin. Accordingly, a larger pupil response was obtained for the light flux modulation compared with the cone-only (L + M + S) modulation in these three subjects (Fig. 6b). Although a relatively small pupil response to the nulled melanopsin modulation was obtained for one subject (S01) on this occasion, this subject did have an appropriately enhanced response to the light flux stimulus and has had a measurable pupil response to un-nulled melanopsin-directed stimuli in our earlier study (Spitschan et al., 2014a; subject S01 in the present study was subject sub001 in that study) both with and without silencing penumbral cones.

These measures confirm that melanopsin-directed stimuli, even after removal of measured cone contrast, are capable of driving a pupil response despite the inability of these stimuli to evoke a reliable cortical response when modulated rapidly. We do note that the noisiness of the pupil response prevented us from measuring a reliable pupil response to our 17% contrast melanopsin-directed modulation. We return to this limitation of our work in the Discussion.

Visual cortex responses to penumbral cone flicker

The absence of a cortical response to melanopsin-directed sinusoidal flicker in our data may be contrasted with the results of a recent psychophysical study by Horiguchi et al. (2013) reporting a possible contribution of melanopsin to flicker detection in the periphery—but not the fovea—at frequencies as high as 40 Hz. Recently, we found that 8–16 Hz spectral flicker that selectively targets the penumbral cones (L* and M*, but not S*) elicits a Purkinje-tree percept in which the subject sees the spatial structure of their own retinal vasculature within the uniform stimulus field (Spitschan et al., 2015). We have also determined that inadvertent stimulation of penumbral cones in a nominally melanopsin isolating stimulus can produce the percept (Spitschan et al., 2015), suggesting a possible mechanism for the high-temporal-frequency result of Horiguchi et al. (2013).

We examined the neural correlates of this entopic perceptual phenomenon evoked with a modulation that selectively targeted the penumbral L and M cones (L* + M*) with ∼2% contrast across a range of flicker frequencies. Across visual areas, penumbral cone flicker produced a consistent pattern of temporal responses, peaking in the 8–16 Hz range (Fig. 7). This corresponds to the frequency that maximally evokes a spatially structured, Purkinje-tree percept in uniform penumbral cone flicker (Spitschan et al., 2015).

Figure 7.

Temporal transfer functions for penumbral cone (L* + M*)-directed flicker. Same format as Fig. 2.

Discussion

Our measurements provide the cortical temporal transfer functions for spatially uniform stimulation of the melanopsin and postreceptoral cone mechanisms. The stimuli separate the cones and melanopsin, while silencing (for the melanopsin measurements) the penumbral cones that experience spectral filtering from retinal blood vessels (Table 1). We made these measurements between 0.5 and 64 Hz—matching the primary operating range of human perception—for both early cortical visual areas and higher level dorsal and ventral extrastriate regions.

Postreceptoral cone mechanisms

For the high luminance, spatially uniform stimuli that we have studied, psychophysical temporal sensitivity is band pass, with a notable decline in sensitivity for slow temporal modulations (Robson, 1966; Kelly, 1979). Chromatic stimuli shift temporal sensitivity to lower frequencies (Kelly, 1983), a phenomenon that appears as early as the retinal ganglion cells (Lee et al., 2007).

Our fMRI results are consistent with these findings. Within V1, sensitivity to achromatic (L + M + S) flicker peaks at or above 16 Hz, whereas, for chromatic modulations of either L − M or S, the peak response is at 8 Hz or lower. Prior neuroimaging studies comparing luminance and L − M chromatic sensitivity did not observe such a separation of temporal sensitivity (Engel et al., 1997; Mullen et al., 2010; D'Souza et al., 2011). This difference in results may relate to our use of a bright, spatially uniform stimulus presented in the periphery compared with the stimuli used in prior studies. Our results are in agreement with prior fMRI findings of peak S-cone-driven responses at lower temporal frequencies (Engel et al., 1997; D'Souza et al., 2011).

We also studied L + M flicker. Although this modulation does not stimulate S cones, it nonetheless activates the blue-yellow postreceptoral mechanism in addition to the luminance pathway. Interestingly, cortical responses to the L + M modulation are essentially identical in shape and amplitude to those evoked by the L + M + S modulation, even in cortical regions with a substantial S-cone-driven response (but see Engel et al., 1997). Because the L + M + S response is larger than the S-driven response at all frequencies (at the 45% contrast level studied), this nonlinear combination of neural response is consistent with a winner-takes-all mechanism (Busse et al., 2009; Carandini and Heeger, 2012).

An increase in peak temporal sensitivity occurs between V1 and higher extrastriate visual areas. Within area MT, peak sensitivity shifts to 32 Hz, consistent with prior studies (Mullen et al., 2010; D'Souza et al., 2011). Within the LOC (a form sensitive visual area), the shift in temporal sensitivity is found only for stimuli activating the luminance pathway (L + M + S and L + M). Generally, sensitivity to lower temporal frequencies would be expected within the LOC (Gauthier et al., 2012). The responses to rapid L + M + S flicker in area LOC may reflect entopic visual phenomena (Brown and Gebhard, 1948; Allefeld et al., 2011). In informal observations, we experience a reticulated, geometric pattern within the uniform stimulus field for 16 and 32 Hz L + M + S flicker, but not for L − M and S cone modulations. In a prior study, we found that this entopic percept is evoked to a greater extent by L + M flicker compared with S-cone flicker (Spitschan et al., 2015).

We did not measure an S-cone-driven response in either area MT or LOC. Previous studies have demonstrated S-cone responses in area MT using high-contrast, spatially structured, moving or flickering stimuli in humans with fMRI (Wandell et al., 1999; Liu and Wandell, 2005; Mullen et al., 2010; D'Souza et al., 2011) and in monkeys with electrophysiology (Seidemann et al., 1999) and fMRI (Conway, 2014). The difference with our results might be due to the spatially structured stimuli used in these studies (compared with our spatially uniform stimulus), but we note that Jayakumar et al. (2013) found units driven by S cones in monkey MT with spatially uniform stimuli. We are unaware of a prior study of S-cone-driven responses within area LOC. Mullen et al. (2007) found responses to spatially structured, blue-yellow stimuli within ventral area VO (immediately anterior to hV4), but did not examine the more lateral, object-responsive region LOC.

Melanopsin responses and the challenge of photopigment isolation

We find no reliable visual cortex response to rapid penumbral cone-silent melanopsin-directed flicker with 17% contrast. Given the generally slow response properties of the ipRGCs, we would have predicted a low-pass response to melanopsin contrast. Our data do not support the existence of a melanopsin-driven response in visual cortex to flicker at 0.5 Hz and above. With the higher contrast (32%) that can be achieved without silencing penumbral cones, a melanopsin-directed stimulus modulated slowly (0.1 Hz) does elicit a measurable pupil response.

With few exceptions (Ecker et al., 2010; Allen et al., 2014), studies of the ipRGCs in rodents and primates have used a spatially uniform stimulus (as we have here). The ipRGCs have broad dendritic arbors and correspondingly wide receptive fields (Dacey et al., 2005), suggesting sensitivity to low spatial frequencies. Because spatially uniform flicker evokes robust responses in area V1 for all postreceptoral cone pathways, we consider it unlikely that this property of our stimulus is responsible for the absence of a melanopsin-driven response in our data.

It is of course possible that a small response to our melanopsin-directed stimuli within visual cortex went undetected in our study. The possible size of such a response is bounded by our results, with 17% melanopsin contrast producing cortical responses no larger than those produced by ∼2% cone contrast.

Melanopsin signals may also be present at different locations along the visual pathways or under different stimulus conditions. In rodents (Hannibal and Fahrenkrug, 2004; Hattar et al., 2006; Brown et al., 2010) and primates (Dacey et al., 2005; Hannibal et al., 2014), ipRGCs are known to project to the lateral geniculate nucleus, a site that was not imaged in this study. Melanopsin signals may also drive visual cortex neurons at a lower temporal frequencies (Brown et al., 2010), under different light level conditions (Davis et al., 2015), or indirectly through interaction with cone signals (Allen et al., 2014).

A particular challenge of measuring a melanopsin-driven response in humans is the need to isolate stimulation of this photopigment from incidental stimulation of the cones. The technique of silent substitution requires precise device control and specification of the effective spectral sensitivities of the photopigments, which are in turn influenced by numerous biological factors, including allelic variation, lens density, macular pigment, and photopigment bleaching. Inevitable imperfections in stimulus specification produce undesired, “splatter” contrast on the cones. Because the visual system is extremely sensitive to even small degrees of differential cone contrast (Cole et al., 1993), this splatter may be expected to produce neural (and perceptual) responses that confound a measurement of melanopsin effects.

In this study, we addressed two sources of systematic bias in photoreceptor isolation. First, subjects consistently perceived positive melanopsin contrast as reddish and containing a luminance component. Although melanopsin stimulation has been reported to produce a perception of brightness at long time scales (Brown et al., 2012), the luminance effect measured here was in 30–40 Hz flicker and thus was unlikely to be mediated by melanopsin (Berson et al., 2002; Dacey et al., 2005; Gamlin et al., 2007; but see Horiguchi et al., 2013; Spitschan et al., 2014a). We suspect (but cannot prove) that the chromatic and luminance percepts produced by the melanopsin stimulus reflect a small error in our assumed cone fundamentals (specifically, the photopigment optical density nomograms; CIE, 2006) and/or a systematic error in our spectral calibrations. To account for possible cone intrusion, we evaluated cortical response to the melanopsin-directed stimulus against the responses to low-contrast cone modulations. The cone component of the melanopsin stimulus may also be nulled (as in our second BOLD fMRI experiment), with the assumption that the chromatic and luminance effects are not themselves a manifestation of melanopsin activity.

Melanopsin stimulus specificity can also fail because of the penumbral cones. Our recent psychophysical studies show that inadvertent contrast on penumbral L and M cones can arise with modulations designed to target melanopsin, producing a Purkinje tree percept (Spitschan et al., 2015). Here, we studied a modulation that targeted melanopsin with 17% contrast while silencing both the cones and their penumbral variants. Although no reliable cortical response was found for rapid flicker of the penumbral-cone-silent melanopsin modulation, we did observe a measurable response to flicker that selectively targeted the penumbral L and M cones (L* + M*) with only 1–2% contrast. Indeed, in a meeting abstract (Spitschan et al., 2014b), we mistakenly attributed to melanopsin the cortical response to penumbral cone flicker.

We cannot rule out the possibility that a response in visual cortex would be evoked with greater melanopsin contrast (as might be achieved using a background of different chromaticity) or, for that matter, at different overall light levels or at lower temporal frequencies than the ones we studied. We can be confident, however, that our BOLD fMRI measurements of cone temporal sensitivity are uncontaminated by melanopsin stimulation.

Although cortical responses are not isomorphic to perception, we take our current results as consistent with an account that humans do not perceive rapid melanopsin flicker, but that inadvertent stimulation of the penumbral cones might be mistaken for this percept. Such an effect might explain prior reports of human perception of rapid melanopsin-directed flicker (Horiguchi et al., 2013; Cao et al., 2015).

Notes

Supplemental material for this article is available at http://dx.doi.org/10.6084/m9.figshare.1316623. We provide tabulations of the spectral modulations used in this experiment (supplemental Table S1), the sequence of modulation (Tables S2 and S3), subject demographic details (supplemental Table S4), individual subject pupil measures (supplemental Table S5), and individual subject nulling measures (supplemental Table S6); BOLD fMRI coverage maps (supplemental Fig. S1), average nulling measures (supplemental Fig. S2), V2/V3, MT, and LOC BOLD fMRI responses for our additional melanopsin modulations (supplemental Fig. S3), results from a collected but unused fMRI dataset (supplemental Fig. S4); and BOLD fMRI data files. This material has not been peer reviewed.