Neural representations of perceptual color experience in the human ventral visual pathway

Significance

There is no color in light. Color is in the perceiver, not the physical stimulus. This distinction is critical for understanding neural representations, which must transition from a representation of a physical retinal image to a mental construct for what we see. Here, we dissociated the physical stimulus from the color seen by using an approach that causes changes in color without altering the light stimulus. We found a transition from a neural representation for retinal light stimulation, in early stages of the visual pathway (V1 and V2), to a representation corresponding to the color experienced at higher levels (V4 and VO1). The distinction between these two different neural representations advances our understanding of visual neural coding.

Abstract

Color is a perceptual construct that arises from neural processing in hierarchically organized cortical visual areas. Previous research, however, often failed to distinguish between neural responses driven by stimulus chromaticity versus perceptual color experience. An unsolved question is whether the neural responses at each stage of cortical processing represent a physical stimulus or a color we see. The present study dissociated the perceptual domain of color experience from the physical domain of chromatic stimulation at each stage of cortical processing by using a switch rivalry paradigm that caused the color percept to vary over time without changing the retinal stimulation. Using functional MRI (fMRI) and a model-based encoding approach, we found that neural representations in higher visual areas, such as V4 and VO1, corresponded to the perceived color, whereas responses in early visual areas V1 and V2 were modulated by the chromatic light stimulus rather than color perception. Our findings support a transition in the ascending human ventral visual pathway, from a representation of the chromatic stimulus at the retina in early visual areas to responses that correspond to perceptually experienced colors in higher visual areas.

Vision is effortless. We recognize faces, navigate a crowded sidewalk, or judge the ripeness of strawberries with ease. These behaviors depend on the light entering the eyes, but what we experience follows from biological responses to light that result in seeing. A sharp distinction between the physical image in the eye versus the biologically rendered percept from the image is essential for understanding vision.

Historical theories of color vision failed to appreciate this distinction, leading to the mistaken assumption that color perception could be explained by the laws of physics (1). We now know that the colors we see follow from biological neural representations generated by light, but light itself carries no color. This basic principle is critical for interpreting results from both human and nonhuman primate studies that show varying the chromaticity of a light causes neural responses to change at multiple levels, from subcortical lateral geniculate nucleus (2, 3) to cortical areas including V1, V2, V3, V4v, VO1, and inferior temporal (IT) cortex (4–10). Ventral color selective regions such as V4 and VO1 have been suggested to have a role in representing colors we experience (11–15). Previous studies have reported luminance invariant chromatic representations in V4, where color constancy computations may occur (16–20). Clusters of similar color preferring neurons have been proposed to represent a perceptual hue map in V4 (21, 22), and functional maps of color have been reported also in V1 (23) and V2 (24, 25).

Neural representations corresponding to the colors we experience, however, remain a fundamental unsolved problem because previous studies often did not dissociate the chromaticity of a stimulus entering the eye, which is in the domain of physics, from the color one sees, which is in the domain of perception. This misconstrues neural representations evoked by physical-stimulus differences as a representation corresponding to color percepts. Although a given stimulus chromaticity may be strongly associated with a specific hue, a neural response to a chromaticity may not be presumed to represent a color we experience.

Isolating neural responses that correspond to colors we perceive is incomplete without dissociating physical lights entering the eye from the color percepts they evoke. The difficulty, of course, is that a given physical stimulus is strongly correlated with a specific hue, as in the schoolchild’s description of physical wavelengths in a rainbow using the perceptual acronym ROY G BIV (for red, orange, yellow, green, blue, indigo, and violet). What is critical, therefore, is to isolate neural representations that correspond specifically to various color percepts without changing chromatic stimulation in the eye. We achieved this using chromatic interocular switch rivalry (26), which presents a fixed stimulus stream perceived to fluctuate between, for example, green and magenta.

Further, to directly access the representation of perceptual color experiences in the human visual cortex, we used a model-based encoding technique (27) to reconstruct subjective color experiences. This approach enabled a comparison between two critical conditions: 1) when the color percept was dissociated from the stimulus properties (during switch rivalry), and 2) when the percept and stimulus were confounded (during “replay” stimulus presentation). Our findings tested whether the perceptual color experiences are represented at each cortical stage, including early, intermediate, and high-level visual areas in the ventral visual stream.

Results

Dissociating Subjective Color Experience and Retinal Chromatic Stimulation.

The present study used functional MRI (fMRI) to examine how subjective color experience is represented at each stage of the human ventral visual pathway. Specifically, we examined whether neural representations reflected stimulus properties (chromaticities of light entering the eye) or perceived color. Using spatially distributed patterns of the blood-oxygen-level–dependent (BOLD) response, we examined whether neural representations in each visual area corresponded to a changing perceived color while stimulus properties were held constant.

Chromatic interocular switch rivalry altered subjective color experiences without changing the stimulus presentation to either eye (26). Observers dichoptically viewed an equiluminant rotating spiral grating. The chromaticity of the spiral grating in each eye was different and rivalrous, appearing magenta in one eye and green in the other. A complementary color pair of magenta (45°) and green (225°) chromaticities was chosen from the equiluminant plane of DKL (Derrington Krauskopf Lennie) color space (3) (Fig. 1A). The two chromaticities of the spiral were swapped between the eyes at a rate of 4.25 Hz (thus, 118 ms presentation duration of one chromaticity before each swap). Unlike conventional binocular rivalry, where dissimilar stimuli are presented to the two eyes (e.g., one chromaticity steadily presented to one eye and a competing chromaticity to the other eye), in switch rivalry, the chromatic input to each eye is identical over time, except that the two eyes are in opposite temporal phases (Fig. 1 B, Top and Middle).

Fig. 1.

(A) Stimuli. Eight different chromaticites chosen from equiluminant DKL space were used for building the chromaticity encoding model. Two pairs of stimuli were tested in switch rivalry: 1) magenta (45°) and green (225°); and 2) blue (135°) and orange (315°). Each pair is indicated by an arrow. The coordinates of the same eight chromaticities were plotted in CIE xyY (45). (B) Experimental design: Switch rivalry condition. Two different chromatic stimuli were continuously presented to each eye to induce perceptual rivalry so that only one of the two colors was seen at a time. The chromatic stimuli were swapped between the two eyes at a frequency of 4.25 Hz so that identical stimuli were presented to each eye over a given time period (except for opposite temporal phase). Observers indicated their moment-to-moment perceived colors by continuous button presses.

Switch rivalry induced slow perceptual alternation between the two colors. Observers continuously reported the color they were experiencing at each moment by pressing a corresponding button. The stable duration of perceptual dominance of one color lasted on average 2.2 s (a duration longer than 18 chromaticity swaps) (SI Appendix, Fig. S1).

Representations of Subjective Color Experience.

To examine representations of subjective color experiences, an inverted encoding model (IEM) was applied to patterns of BOLD responses (27, 28). First, a chromaticity encoding model was built separately for each visual region (V1, V2, V3, V4v, and VO1) to map out voxel-wise selectivity to different chromaticities. The data for model building were acquired in a separate session, in which observers viewed a rotating spiral with one of eight chromaticities evenly spaced around the equiluminant DKL plane (Fig. 1A). While viewing the same chromaticity with both eyes (no rivalry), observers reported detecting a luminance change in the fixation point. This task aimed to exclude the involvement of higher cognitive processes, such as color naming, as it did not require any explicit judgment of the chromaticity of the stimulus. When the model for each visual region was validated with a leave-one-run-out procedure (27), the model for all visual areas, including early (V1), extrastriate (V2 and V3), and ventral visual regions (V4v and VO1), exhibited strong chromaticity selective responses (SI Appendix, Fig. S2).

Next, by using the chromaticity encoding model, color-selective responses during switch rivalry were reconstructed in each visual region. Specifically, spatial patterns of voxel-wise BOLD responses acquired during switch rivalry were transformed to reconstructed chromatic responses in the DKL space.

These reconstructed color-selective responses in early visual areas, including V1 and V2, were distinctively different from those in higher visual areas, such as V4v and VO1. In V1, the peak of the color response profile when observers indicated that they perceived magenta or green was far off from each respective color (45° or 225°; Fig. 2A, Top and SI Appendix, Fig. S3). Conversely, color responses in V4v during the moments of magenta or green color experiences closely matched the perceived colors (Top Right).

Fig. 2.

(A) Reconstructed color-selective responses. Color-selective responses during the switch rivalry and replay conditions were averaged across observers in V1 and V4v. The magenta and green color responses were estimated by using the time points of dominant magenta and green perception. Colored dashed vertical lines indicate magenta (45°) and green (225°) within DKL color space. In V1, the peaks of magenta and green color responses were far off from each respective color, whereas in V4v, color response profiles closely matched magenta and green. In the replay condition, both V1 and V4v showed similar color response profiles that were tuned to magenta and green. (B) Color-selective responses in each area were quantified by computing the average difference between the observer’s reported perceived color and the color estimated from a given color response profile. The maximum difference of 180° indicates that the estimated color was located opposite to the perceived color on the DKL hue plane, while the minimum difference of 0° indicates perfect estimation of experienced color. The dashed line at 90° is the chance level; any lower value indicates a more accurate color representation than chance. In the magenta–green switch rivalry condition, the measured difference was significantly below chance in V3 (P = 0.048), V4v (P = 0.02), and VO1 (P = 0.02). The blue–orange rivalry condition also showed a similar trend to the magenta–green rivalry condition. For the replay condition, the difference was significantly less than chance in every visual area (V1, P = 0.013; V2, P = 0.013; V3, P = 0.013; V4v, P = 0.013; and VO1, P = 0.013). The magenta–green and blue–orange switch rivalry conditions showed a significant difference across visual regions (P < 0.001 and P = 0.033, respectively), while no such difference was found in the replay condition (P = 0.57). A significant interaction effect was found between task condition (rivalry versus replay) and visual region (P = 0.007). Shaded areas in A and error bars in B represent ±1 SEM; a.u., arbitrary units; ROI, region of interest.

To further quantify color selectivity across the ventral stream, the color represented by each area was reconstructed based on the one-to-one relation between the color-selective response profile and hue. Hypothesized color-selective responses with their hues evenly covering the DKL hue plane in steps of 1° (360 hypothesized responses in total) were created and compared to each region’s color response profile. The center hue of the hypothesized color response that showed the highest correlation with a given color response profile was assigned as the color represented by the corresponding profile. Then, the difference between the color reconstructed from each area and the color reported as seen during rivalry was computed. A maximum difference of 180° would indicate a reconstructed color opposite to the perceived one (e.g., magenta reconstructed when experiencing green), while the minimum difference of 0° would reflect a perfect correspondence between the color represented in a given area and the perceived color during rivalry. The expected chance level was 90°, the median of 0° and 180°. The difference measures for perceiving only magenta or only green were averaged (Fig. 2B), as there was no significant difference between magenta and green in terms of the magnitude difference between the reconstructed and reported colors in any of the visual areas (V1, P = 0.915; V2, P = 0.915; V3, P = 0.915; V4v, P = 0.915; and VO1, P = 0.915; these and all subsequent P values were false discovery rate corrected). The mean absolute deviation of the reconstructed color from the perceived color was tested against the chance level (90°) to evaluate the significance of whether each visual region accurately represented the experienced color percept.

Across the ventral stream, the difference between the color representation and the observer’s reported color experience decreased progressively (P < 0.001) (color bars in Fig. 2B). Moreover, this difference was significantly below chance in V3 (P = 0.048), V4v (P = 0.02), and VO1 (P = 0.02), but not in early visual areas V1 and V2 (P = 0.94 and 0.95, respectively). Thus, there was a significant representation of the reported hue percept in V3, V4v, and VO1, but not in V1 or V2.

The color reconstructed from each visual area was visualized in Fig. 3, where the half of the hue plane opposite to the perceived color is shaded to indicate below-chance accuracy of the reconstructed color (>90°). The reconstructed color across observers is summarized by the vector sum (solid black line, Fig. 3). The length indicates reconstruction consistency across observers and the direction indicates the aggregated reconstructed color within the DKL space. Again, for both magenta and green (Fig. 3A), the reconstructed color of individual observers changed across the ventral visual stream to match the reported color. In early visual areas V1 and V2, reconstructed colors showed no consistency across observers and were scattered throughout the DKL hue plane. Ascending the hierarchy, reconstructed colors in V4 and VO1 closely resembled the subjective color experiences, tightly clustered around magenta or green, with less variability across the observers compared to early visual areas. The degree of individual-subject variability of reconstructed color representations in V4v and VO1 was comparable to previous results from a similar analysis (27).

Fig. 3.

(A) Reconstructed magenta and green. The color represented in each visual area during switch rivalry and replay conditions was reconstructed and plotted in DKL color space. Each dot indicates the reconstructed color for one observer. The solid black line depicts the vector sum of the reconstructed color representations across observers; the solid colored line indicates each target color (magenta or green). The half of the hue plane opposite to the perceived color is shaded to indicate below-chance accuracy of the reconstructed color. (B) Reconstructed blue and orange. A blue–orange (135° to 315°) color pair was used with switch rivalry (four observers participated).

To ensure the generalizability of the results across different colors, an additional switch rivalry experiment was conducted by using a blue (135°) and orange (315°) pair. Consistent with the results from magenta and green, reconstructed color responses for blue and orange also revealed color responses toward the perceived color, showing again a significant difference across the ventral visual stream (P = 0.033; Figs. 2B and 3B).

Further, a separate independent analysis considered the color representation during the moments of perceiving mixed color percepts during magenta and green switch rivalry. The moments of mixed perception included periods when magenta and green were 1) visible simultaneously in a superimposed or piecemeal way; 2) fused, resulting in a third and different color; or 3) alternating very rapidly (comparable to the swap rate in each eye). The reconstructed color during the mixed color experiences showed no consistency across observers in any of the visual areas (Fig. 4). These results are as expected if there is no single perceived color representation during the period of mixed perception.

Fig. 4.

Reconstructed mixed color experiences. The colors represented by each visual area during the moments of mixed color experiences in the magenta–green switch rivalry condition was reconstructed. Plotted in DKL color space, each dot indicates the representation of mixed color perception for one observer. The solid black line depicts the vector sum.

Representations of Color without Dissociation of Chromatic Stimulation from Color Perception.

Finally, reconstructed color representations in each visual area during rivalry were compared to those obtained in a condition in which retinal stimulation and color experience were intentionally confounded (“replay”). These color responses provided a baseline approximation of chromatically selective representations in earlier visual areas V1 and V2 when there was no discrepancy between the stimulus and color percept. The stimuli and procedures of the replay condition closely mimicked the switch rivalry condition, except that the chromaticities were presented without rivalry. The order and duration of the chromatic stimulus presentations for each observer followed the temporal structure of the perceived color report from the same observer during switch rivalry. As in the switch rivalry condition, observers reported their moment-to-moment color percepts; identical IEM analyses were performed to reconstruct color experiences.

The reconstructed color responses in the replay condition revealed patterns distinct from those in the switch rivalry condition. Contrary to the results with rivalry, the color-selective response profile in all of the visual areas showed significant above-chance accuracy during replay (V1, P = 0.013; V2, P = 0.013; V3, P = 0.013; V4v, P = 0.013; and VO1, P = 0.013); moreover, no significant change across visual areas was found (P = 0.57) (Fig. 2B). Crucially, the color responses in V1 and V2 were significantly different between rivalry and replay (P = 0.02), while no such difference was found in V3 (P = 0.19), V4v (P = 0.83), or VO1 (P = 0.66). The interaction effect of the presentation condition (rivalry versus replay) and visual regions was significant (P = 0.007). As visualized in Fig. 3A, the sharp distinction between rivalry and replay found in V1 and V2 suggested that these early visual areas may represent retinal chromatic stimulus information rather than perceptual color experiences. Unlike early visual areas, V4v and VO1 displayed comparable color representations during either rivalry or replay, indicating that these high-level visual areas are not exclusively modulated by the chromaticity of the stimulus, but instead represent perceptually constructed colors.

The replay condition entailed binocular fusion because both eyes received the same physical stimulation (29). In switch rivalry, however, binocular fusion was absent, so an additional control was tested. A monocular replay condition examined whether binocular fusion plays a critical role in distinguishing the above results of rivalry and replay. The procedure of the monocular replay condition was identical to binocular replay, except that only one eye was stimulated. Monocular replay yielded similar results to binocular replay. There were no significant differences between the monocular and binocular replay conditions (V1, P = 0.94; V2, P = 0.94; V3, P = 0.94; V4v, P = 0.94; and VO1, P = 0.94), as expected if differences in binocular fusion cannot explain the qualitatively dissimilar findings between rivalry versus replay.

Discussion

This study examined how color perception is represented in the visual cortex disentangled from neural responses representing only chromatic light stimulating the retina. Dissociating the retinal stimulus from the color percept one experiences is critical for understanding neural processes mediating color vision. The main finding—differential neural representations corresponding to light in the eye or to the hues perceived—fits well with the hierarchical processing of color in the ventral stream. Neural responses in earlier visual areas V1 and V2 were mainly modulated by chromatic stimulus properties. Strong chromatic selectivity was shown in these areas when the physical property of the stimulus and the color percept were correlated (so confounded experimentally), but not when the light stimulus did not correspond to the color seen.

One process that contributes to the distinct neural representations between early and higher visual cortices may be neural resolution of rivalry (for review, see refs. 30 and 31). Neural resolution of conventional rivalry, in which steady retinal stimulations are different in the two eyes, has been observed as early as V1 (32, 33), suggesting that neural responses in early visual cortex may reflect the resolution of eye-based competition (34). With switch rivalry, however, the representation of perceived color is dissociated from the retinal stimulation in each eye. The current study directly demonstrates that neural competition between different chromatic representations can be resolved by V4. A rivalry resolution mechanism analogous to one mediating stimulus rivalry (35, 36) may shift chromatic representations to correspond with perceived color experiences within higher visual areas by suppressing one of the earlier competing chromatic representations.

Functionally organized “hue maps” in V1 and V2 have been reported (23–25), but could not distinguish between neural representations of stimulus chromaticity and of color perception. Our results indicate that color maps in V1 and V2 may be stimulus-driven, chromaticity-selective clusters rather than functional maps that reflect the perceptual domain of color. Although we did not find evidence that the color percept is represented in the spatial patterns of the voxel signals of V1, there is the possibility that undetected weak signals in V1 reflect the color percept, but were unmeasurable here. Note, however, that the replay condition eliminates any concern about measuring V1 responses in general because (nonrivalrous) replay stimuli did generate V1 signals correlated with the colors seen, as expected.

Beyond the initial stages of visual processing, V4v and VO1 showed color representations that corresponded to perceived hue. V4 has been recognized for its relevance to high-level color perception in the context of color categorization (37), color illusion (11, 38), color imagery (12, 39), and color ordering (13). Moreover, responses in V4 have been observed to have an important role in color constancy (18, 40), which is posited to be a crucial milestone for conscious color perception (41). However, measuring a neural representation of color affected by surrounding chromatic context or by illumination, as in studies of color constancy, is different from measuring a neural correlate of a time varying color an observer consciously perceives, without a change in retinal stimulation, as done here. In color constancy, the perception of color is determined by both retinal stimulation of a target object and retinal stimulation by the surrounding chromatic environment. The current study, on the other hand, focuses on dissociating changes in color perception without altering the retinal stimulation.

A possible concern is that the representation of the reported hue percept in V4v and VO1 may result from the interaction of the intrinsic neuronal properties of each visual region with different temporal frequencies. The train and test experiments implicitly required different temporal dynamics of stimulus presentation, which theoretically could bias the results to favor high-level visual areas with longer temporal integration windows (42, 43). However, if differences in temporal properties of each visual region affected the IEM accuracy, these differences should also affect the results in the replay condition. Such differences were not evident in the replay condition, leading to a significant interaction effect between task conditions and visual regions (P = 0.007).

While the current study is an important initial step toward understanding computational mechanisms of a color percept, the exact mechanisms of neural computation within each visual region remain to be explored. The evidence here, which supports a transition across the ventral visual stream from the stimulus to subjective experience, can facilitate discovery of the computational mechanisms that transform visual information at each stage of cortical processing.

Materials and Methods

Observers.

Twelve observers (five female; aged 22 to 27 y) participated in a switch rivalry condition, and seven of them (three female; aged 23 to 27 y) also participated in a replay condition. Four of the 12 observers participated in another switch rivalry condition with an additional color pair. Also, five of the 12 observers participated in a monocular replay condition. Three observers from the switch rivalry condition were excluded from the analysis: two due to experiencing a sustained mixture percept of two colors and one due to perceiving only one of the two rivalrous colors. Each observer also completed a retinotopic mapping scanning session and a functional color localizer scanning session. All observers had normal or corrected-to-normal vision and provided written informed consent. All experimental procedures were approved by the Institutional Review Board of Sungkyunkwan University. All observers were monetarily compensated for their participation. Detailed materials and methods can be found in SI Appendix, SI Materials and Methods.

Stimuli.

All stimuli were generated by using Matlab and Psychtoolbox (44) and were presented by using a PROPixx projector with a circular polarizer directed onto a rear-projection screen for dichoptic projection. In all experiments, observers viewed stimuli while wearing circularly polarized three-dimensional (3D) glasses inside the scanner.

In chromaticity model building, eight chromaticities chosen from DKL color space (3) were presented as a rotating spiral grating (radius, 3°; speed of rotation, 0.134°/s) (Fig. 1A). The DKL space is a spherical cone-opponency space with three cardinal axes: L + M + S (achromatic), L − M, and S − (L + M). The eight chromaticities were obtained from the equiluminant plane of the DKL color space (elevation = 0) and were equally spaced (45° steps along the azimuth with equal radius of 0.8) around the hue angles (Fig. 1B). For each switch rivalry condition, a pair of two chromaticities diametrically opposite along the intercardinal axis was used. Two different pairs were tested: magenta (45°) and green (225°); and blue (135°) and orange (315°). Intercardinal hues were used to drive signals in the visual cortex excited by either (or both) cardinal axis (L/(L + M) and S/(L + M)). Chromaticity values in terms of Commission Internationale de l’Elcairage (CIE) color coordinates (45) were magenta: x = 0.334, y = 0.28; green: x = 0.338, y = 0.407; blue: x = 0.294, y = 0.297; and orange: x = 0.388, y = 0.376.

Building the Encoding Model for Chromaticity.

A chromaticity encoding model was built based on the approach developed by Brouwer and Heeger (27). By using an IEM, population-level color selective responses were estimated, and a chromatic representation was reconstructed. The stimulus was presented without rivalry so that both eyes received identical input (3D glasses were used to match stimulus luminance level to later rivalry and replay conditions). Each chromatic stimulus was presented for 12 s without an interblock interval and repeated three times in each run (eight runs in total). The order of the chromaticities was pseudorandomized to prevent the identical chromaticity from appearing twice in a row. While viewing stimuli, observers were instructed to press a button upon detecting a luminance change of the fixation disk (radius, 0.1°).

Interocular Switch Rivalry.

A switch rivalry paradigm was used to dissociate color perception from stimulus attributes (26). Observers viewed two equiluminant binocularly rivalrous colors, which were swapped between the eyes at a rate of 4.25 Hz (Fig. 1B) Thus, stimulus input was identical to each eye, except that the chromaticities were in opposite temporal phase to maintain chromatic rivalry. The stimuli were continuously presented for 60 s, followed by a 10 s gray blank period. A single-color pair was repeated four times in each run, which lasted 290 s, including a 10 s initial blank period. This resulted in effective perceptual rivalry with the percept of a single hue persisting, on average, for more than 2 s. To capture the moment-to-moment color percepts, observers were instructed to continuously indicate their current color perception by pressing an assigned button. They were asked to make conservative judgments and not to respond if experiencing rapid switching between the two colors, a piecemeal color spiral, or a uniformly colorless spiral. All nine observers completed eight runs of the switch rivalry condition for the magenta–green pair (45°–225°), and four of them also participated in an additional eight runs for the blue–orange (135°–315°) pair on a different day.

Replay.

All experimental procedures of the replay condition were identical to the switch rivalry condition, except that the same chromaticity was presented to both eyes without any interocular chromatic difference, thus eliciting no rivalry. Observers viewed stimuli in the same order and duration as their own color experiences during the switch rivalry condition and were instructed to report their perceived colors. Observers performed eight runs of the replay condition, and only the magenta–green (45° to 225°) pair was tested.

fMRI Acquisition.

fMRI scanning was conducted on a 3T scanner (Prisma, Siemens) with a 64-channel head coil. Functional data were acquired by using a T2*-weighted multiband gradient-echo echo-planar imaging sequence (flip angle = 90°, repetition time [TR] = 1,000 ms, echo time [TE] = 30 ms, field of view [FOV] = 96 × 96 mm, 2 mm isotropic voxel size, multiband acceleration factor = 3, generalized autocalibrating partial parallel acquisition factor = 2). A total of 32 slices parallel to the anterior commissure–posterior commissure line were acquired, leading to partial coverage selected to include temporal and occipital cortex. High-resolution anatomical scans (MPRAGE T1-weighted sequence, 256 slices, TR = 2200 ms, TE = 2.44 ms, 1 mm isotropic voxel size, FOV = 256 × 256 mm) were also obtained in each session.

fMRI Preprocessing.

All data were preprocessed by using AFNI and analyzed subsequently by using in-house Matlab and Python scripts. Upon image collection, functional images were coregistered to the anatomical data. The functional data for individual observers were motion corrected, detrended, and scaled with the mean activity of each voxel within each run. Only the color localizer data, which were used to identify the color-selective regions of interest, were smoothed with a 3-mm full-width at half-maximum (FWHM) Gaussian kernel. All other experimental data (rivalry and replay) were not smoothed in order to retain the spatial patterns of voxel signals at the finest scale.

Reconstruction of Subjectively Experienced Colors.

Neural representations of colors during rivalry were reconstructed in each visual region by using an IEM (SI Appendix, SI Materials and Methods). Eight sinusoids raised to the sixth power (FWHM = 0.94 rad) basis functions were used to evenly cover the entire DKL color space. First, data from eight chromaticity presentations with a fixation task served as a training dataset (B1) to build a chromaticity model. The training data comprised an m × n matrix, with m as a number of voxels within each visual areas and n as a number of chromaticities times number of trials within a session. Building a chromaticity model involved estimating chromaticity selective weights for individual voxels (W; m × k channels) by mapping voxel signals to channel outputs (C₁; k channels × n):

$$B_{\mathit{1}}=W{C}_{\mathit{1}}.$$ [1]

Ordinary least-squares estimation was used to solve W:

$$W=B_{\mathit{1}}{C}_{\mathit{1}}^T{\left(C_{\mathit{1}}{C}_{\mathit{1}}^T\right)}^{-\mathit{1}}.$$ [2]

Finally, switch rivalry data were used as a test dataset (B2) to compute chromaticity channel outputs (C2) that corresponded to each subjective color experience:

$$C_{\mathit{2}}={\left(W^{T}W\right)}^{-\mathit{1}}{W}^T{B}_{\mathit{2}}.$$ [3]

In order to isolate the moments of specific color experiences, only BOLD signals when observers exclusively experienced one color or the other (e.g., magenta or green) during the switch rivalry (B2) were used for the analysis.

Each color response profile from the IEM analysis consisted of eight values, each of which represented a channel output with a peak at 0; 45; 90; 135; 180; 225; 270; or 315° within the DKL color space. To further quantify and visualize the color represented in each visual region, 360 basis functions were created, which were evenly spaced over the entire range of the DKL hue plane in steps of 1° (27). Then, correlations were computed between the estimated color response profile and each of the 360 basis functions. The hue angle of the basis function that resulted in the highest correlation was selected as the color represented by a given color response profile. All rivalry and replay conditions were analyzed by using the identical analysis procedures based on the same chromaticity encoding model.

Data Availability.

The data and analysis scripts for this study are available via the Open Science Framework (OSF) database at https://osf.io/wzakb/. fMRI data are available on Zenodo (https://zenodo.org/record/3790067).