Perceptual moments of conscious visual experience inferred from oscillatory brain activity

Abstract

Transient periods of synchronized oscillating neuronal discharges in the brain have been proposed to support the discrete perceptual moments underlying conscious visual experience. However, the information content of these perceptual moments remains a critical challenge to the understanding of consciousness. We uncovered this information content in four observers who consciously perceived each interpretation of the ambiguous Dali painting Slave Market with the Disappearing Bust of Voltaire. For each individual observer, we isolated the stimulus spatial frequency (SF) features underlying their overt judgments of the input as “the nuns” and “Voltaire”. Every 2 ms between stimulus onset and overt response, we derived the sensitivity of the observer's oscillatory brain activity (in the theta, alpha, and beta bandwidths) to these SF features. Then, in each bandwidth, we estimated the moments (between stimulus onset and perceptual judgment) when perception-specific SF features were maximally integrated, corresponding to perceptual moments. We show that the centroparietal beta oscillations support perceptual moments underlying the conscious perception of the nuns, whereas theta oscillations support the perception of Voltaire. For both perceptions, we reveal the specific information content of these perceptual moments.

To study conscious visual experience in an information system such as the brain, two generic questions must be addressed. The first question is one of form: What is the nature of the brain activity supporting the perceptual moments associated with conscious visual experience; that is, what are the neural correlates of consciousness? The second question is one of content: What is the information content of the perceptual moments associated with conscious visual experience; that is, what is the information content of the neural correlates of consciousness?

The common answer to the question of form is a proposal with a long history (1–3), suggesting that the dynamics of psychological states is related to the oscillatory activity of the neural substrate, which recently has been cast in terms of binding multiple information sources (4–6). Simply stated, synchronized discharges of cortical cell assemblies with a prominent rhythm would be the neural correlates supporting discrete cognitive and perceptual conscious states (7–9). This theory is based on evidence of a consistent relationship between behavioral variables and the frequency of both the theta (4–8 Hz) and alpha rhythms (8–12 Hz) in memory (10–16) and perceptual tasks (7, 17–19). Furthermore, there is a relationship between visual perception and faster rhythms in the beta (12–25 Hz) and gamma (25–60 Hz) range, although this hypothesis is still controversial (20, 21).

In contrast, there is no common answer to the question of the information content of perceptual moments, although, as just discussed, it is supposed that conscious visual experience arises under specific parametric conditions of synchronized brain rhythms. An important task, then, is to ascribe information content to the parameters of oscillatory brain activity (22, 23). This goal is the main aim of this article.

In an experiment, we used gray-level versions of Dali's ambiguous painting The Slave Market with Disappearing Bust of Voltaire to induce conscious perceptions. This stimulus can induce two distinct phenomenal states (the perception of “the nuns” vs. “Voltaire”) from two distinct feature subsets of the same original image, creating a situation of perceptual ambiguity for the visual system to resolve (24). On each trial, we created a sparse version of the painting by randomly sampling visual information from five nonoverlapping one-octave spatial frequency (SF) bandwidths as illustrated in Fig. 1. Observers pressed response keys to indicate whether they perceived the information samples as the nuns, Voltaire, or “don't know” (24). Concurrently, we measured the electroencephalogram (EEG) activity on each trial.

Fig. 1.

Illustration of the SF information sampling procedure with bubbles. The aim of the procedure is to expose, on each trial, the visual system with a random subset of the SF information present in the original picture. (A) The original picture is a gray-level cropped version of Dali's Slave Market with Disappearing Bust of Voltaire. (B) To produce the random SF information subsets, we carried out several computations. We decomposed the Dali image into six independent SF bands of one octave each, with cutoffs at 128 (22.4), 64 (11.2), 32 (5.6), 16 (2.8), 8 (1.4), and 4 (0.7) cycles per image (cycles per degree of visual angle), from HSF to LSF. (C) For each SF band, a number of randomly located Gaussian apertures (the bubbles), with an SD of 0.13, 0.27, 0.54, 1.08, and 2.15 cycles per degree from HSF to LSF, formed SF-specific bubble masks. (D) The multiplication of the information sampled in B with the masks represented in C sampled information independently in different SF bands. (E) The sparse stimulus presented on each trial of the experiment was generated by adding together the information randomly sampled from each band in D.

After the experiment, we applied classification image techniques (i.e., “bubbles”; see refs. 24–26), independently for each observer (i) to compute the SF features correlated with behavioral judgments of the nuns and Voltaire (see Fig. 2) and (ii) to compute the SF features correlated with modulations of centroparietal EEG amplitudes (in the theta, alpha, and beta bands; see Fig. 3). Then, we compared (i) and (ii) to determine the time intervals (i.e., the perceptual moments) over which the brain processes the specific SF features underlying each observer's conscious perceptions of the nuns and Voltaire.

Fig. 2.

Behavioral classification images for the perceptual judgments of the nuns and Voltaire. For each SF band, we independently computed the statistically significant (P < 0.05; one tail) SF regions associated with each observer's perception of the nuns and Voltaire on the basis of behavioral responses (26). Together, across SF bands, the significant regions form a filter that we applied to the original picture to depict the effective stimulus of the nuns and Voltaire for each observer. Effective stimuli represent the observer-specific features used to judge the input as the nuns or as Voltaire. Differences in effective stimuli reveal observer-specific differences reflecting the subjectivity of perception.

Fig. 3.

Computation of sensitivity of EEG oscillations to the features underlying overt perceptual Judgments of the nuns in observer GB. (A) Step 1: Behavioral classification images. Illustration of the SF-specific features underlying overt perceptual judgments of the nuns for observer GB (significant features are labeled a and e; P < 0.05). (B) Step 2: EEG classification images. The movie of classification images indicated with an arrow illustrates the beta band EEG classification images for the second band of sampled SF information in the critical 190 and 248 ms after stimulus onset. At each time point, the images standing above and below the classification image (Middle) represent the sums of the bubble masks associated with EEG amplitudes 0.25 SD above (Top) and 0.25 SD below (Bottom) the mean amplitude at this time point. The two EEG classification images highlighted by a red box are expanded, thresholded for significance (P < 0.05), and interpreted by apposing their significant regions onto the ambiguous image at SF band 2. They reveal that, at these time points, beta EEG amplitudes are modulated by features c and d. (C) Step 3: Relating behavioral and EEG classification images. We computed the sensitivity of the EEG classification images to behavioral features c and d by using Pearson's correlation to correlate the latter with the former. The white box is expanded in B, where the blue curve indicates the correlation with feature c and the green curve indicates the correlation with feature d. Generalizing across all features and time points, we obtain the illustrated sensitivity matrix. Labeled features (i.e., c, d, and e) indicate periods of significant correlations (P < 0.01, Z scored with respect to prestimulus baseline correlations) for observer GB perceiving the nuns.

Results and Discussion

Evidence of Perceptual Moments.

The critical advantage of expressing a brain signal in terms of its sensitivity to visual features is that one can precisely track in time the processing of features and their integration in the brain (see Methods). To illustrate, Fig. 4depicts the average (across observers, n = 4) first- and second-order feature sensitivity measures in the theta, alpha, and beta bands for each perception. Over the time interval of 200–300 ms for the nuns (the overlaid red strip in Fig. 4) and 150–350 ms for Voltaire (the overlaid blue strip in Fig. 4), we found a main moment of feature sensitivity (i.e., a perceptual moment defined by using the maxima of first- and second-order measures). However, this feature sensitivity peaked in parietal theta oscillations for Voltaire, whereas it peaked in beta oscillations for the nuns, with an overall timing congruent with that found in related studies (21, 27); note that this timing was uncorrelated with the reaction times of observers (see Table 1; r = 0.052), ruling out an explanation in terms of motor response. We now explore the specific information contents of these perceptual moments in the beta band (for the nuns) and theta band (for Voltaire).

Fig. 4.

Average first- and second-order theta, alpha, and beta sensitivities to the specific SF features for the nuns and Voltaire perceptions. For each time point and oscillatory band, we averaged per perception (i.e., the nuns vs. Voltaire) the first- and second-order measures of sensitivity across observers (n = 4) and plotted them on a normalized scale. We also averaged across observers the first-order sensitivity measures to randomly chosen samples of generic SF information (i.e., SF information that did not intersect with the SF information of each perception) to tease apart interpretations of perception-specific vs. generic SF processing. On the x axis, a dark blue (vs. red) region indicates that beta (vs. theta) oscillations are more sensitive to SF perception features than theta (vs. beta) oscillations. Between 200 and 250 ms, the plots reveal a clear advantage of beta sensitivity over theta for the nuns for first- and second-order measurements (indicated with an overlaid red strip). The advantage is reversed for Voltaire (theta over beta, indicated with an overlaid blue strip) and occurs earlier (150–200 ms), again for both sensitivity measures. No such advantage exists for generic SF information.

Table 1.

Reaction times

Observer	Nuns, ms	Voltaire, ms
EO	585	584
GB	971	713
KB	1,985	1,926
MLS	749	721

Information Content of Perceptual Moments.

Consider observer EO in Fig. 5. Four gray-scale images depict the information correlated with her overt perceptual judgment of the nuns. The labels (a, b, c, d, e, and f) indicate the significant SF behavioral features. The modulation of beta EEG oscillations to these features (from −200 to 600 ms around stimulus onset) is shown in the two adjacent colored graphs. Significant first- and second-order measures (P < 0.01) are indicated on the y axis of the colored graphs with feature labels. From 180 to 250 ms, Fig. 5 reveals the highest first-order sensitivities in SF bands 1 and 2 representing the heads of the nuns with features a, b, and c. Highest second-order sensitivities involved integrations of SF bands 1 and 2 to represent the left head with the conjunctions of features a and c and to represent the two heads with features b and c. The overlaid red strip indicates a state of the brain of this observer corresponding to the maximal sensitivity to the SF features underlying perception of the nuns. This state is what we term a perceptual moment, and we have revealed its information content.

Fig. 5.

Perceptual moments and their information content. For observers EO and GB, the four gray-level pictures illustrate the significant labeled SF features (a–f) associated with their overt perceptual judgments of the nuns and Voltaire. The first- and second-order sensitivities of beta (for the nuns) and theta (for Voltaire) oscillations to these SF features are rendered in the two adjacent colored graphs. Labeled rows are those for which first- and second-order sensitivities reached significance (P < 0.01). The red (beta; for the nuns) and blue (theta; for Voltaire) line plots reveal the average first- and second-order sensitivities across all individual SF features (for the first-order) and SF feature combinations (for the second-order) for this EEG band. The red strip (for the nuns) and blue strip (for Voltaire) indicate the perceptual moments at which oscillatory sensitivity to SF features peaks.

Perceptual moments vary in information contents across observers. For example, compare observer EO with GB (see Fig. 5), whose perceptual moment occurred at a similar time but differed markedly in content. First- and second-order sensitivities comprised the head and the dress, involving features c, d, and e and integrations of SF features c and d within band 2 and c–e and d–e across bands 2 and 3, respectively, quantifying the subjective character of perception in brain activity (see Fig. 6, which is published as supporting information on the PNAS web site, for a description of the perceptual moments of observer KB and MLS).

Turning to the perception of Voltaire, the peaks of first- and second-order sensitivities occurred in the theta band. First- and second-order measures of sensitivities also revealed differing contents of perceptual moments across observers. For observer EO, this content included SF band 3 (the internal features of the face), although interactions involved bands 2–3 and 3–4, adding the forehead to the internal features; for observer GB perceptual content spanned more SF bands, adding external contours of Voltaire and elements of his forehead at bands 3 and 4 and in 2–3, 2–4, and 3–4 SF band interactions (see Fig. 6 for observer KB and MLS).

Content-Specific Rather than SF Generic Sensitivity.

There is a potential confound in our results: Although the perception of the nuns involved mostly (although not exclusively) high-SF (HSF) information content, the perception of Voltaire involved information represented in comparatively low-SF (LSF) bands. Thus, the reported perceptual moment of sensitivity to the specific information content of the nuns and Voltaire might be confounded with a generic sensitivity to the processing of any HSF and LSF information in the beta and theta oscillations, respectively. To tease apart these interpretations, we carried out a supplementary analysis.

We simply repeated the first-order sensitivity analysis outlined in step 3, substituting the behavioral features with information samples randomly selected from the stimulus space with the double constraint that (i) at each SF band the samples did not overlap with those features representing the nuns and Voltaire to the observer under consideration and (ii) the samples matched in size those features representing the nuns and Voltaire for this observer at each SF band. To illustrate this random sampling, if the HSF band represented the left head of the nuns (see feature a in Fig. 5), we sampled HSF band information with sample size corresponding to the size of the head anywhere in the HSF image as long as this sample did not overlap with any significant feature (i.e., the left or right nuns' face).

Specifically, for each behavioral feature, we correlated one hundred of these observer-specific randomly positioned SF templates with the EEG classification images in the theta, alpha, and beta bandwidths. This computation allowed us to measure the average generic (as opposed to perception-specific) sensitivity of beta, alpha, and theta oscillations to SF band information. Fig. 4 depicts the average (across templates and observers) perception-specific and generic SF first-order sensitivity measures. These results clearly rule out the generic SF-processing interpretation in favor of a multiplexing of specific perceptual content across the beta and theta oscillations.

Conclusions

Together, the evidence reported here supports the idea that oscillatory brain activity underlies the processing of visual information subtending the perceptual moments of conscious visual experience. Specifically, we have shown that in the context of an ambiguous input stimulus, theta oscillations subsumed the processing and integration of LSF visual features associated with the overt perceptual judgment of Voltaire and that beta oscillations supported higher SF features associated with the overt judgment of the nuns. Furthermore, we isolated for each observer the specific information content of their perceptual moments, quantifying the subjective content of perceptual experience from brain measurements.

There is now a growing body of knowledge on the brain correlates of conscious visual experience (21, 27–29). However, unique to our approach, we do not correlate important parameters of brain activity (such as rhythm, amplitude, or phase coherence) with a few experimental conditions (e.g., to perceive or not). Instead, we correlate brain activity with fine-grained samples of stimulus information (in which each information sample can be construed as an independent experimental condition) to derive a precise mapping of brain activity onto visual information. From this mapping, we relate the dynamics of information processing with the dynamics of brain states (25, 30). With this approach, we uncovered for each observer two distinct perceptual moments, corresponding to the maximal sensitivity of brain rhythms to the specific SF features subtending the perception of the nuns and Voltaire.

Perceptual moments allow new fundamental issues for perception to be addressed, ranging from the information capacity of the different bandwidths of oscillatory activity (the number of bits of information that they can encode), the mechanisms of information integration within a moment, to the relationship between discrete moments and the apparently seamless nature of perceptual awareness and perception.

Methods

A total of four observers participated in the experiment. Informed consent was obtained from all and ethical approval was obtained from the Glasgow University Faculty of Information and Mathematical Sciences Ethics Committee. We cropped the ambiguous portion of Dali's painting to retain the bust of Voltaire and the two nuns, a figure subtending 5.72° × 5.72° of visual angle on a CRT screen (for an image size of 256 × 256 pixels). On each of 6,000 trials, we randomly sampled information from the cropped image using a number of Gaussian apertures (making up “bubble masks”) to create one sparse stimulus, as detailed in Fig. 1. The sparse stimulus remained on the screen until the observer depressed one of three possible response keys, according to which image they could perceive: the nuns, Voltaire, or don't know. The total number of Gaussian apertures remained constant throughout the task, ensuring that equivalent amounts of visual information was presented on each trial, at a level of 60 bubbles found previously to maintain don't know responses at 25% of the total number of responses (26). A chin rest maintained a constant 1-m viewing distance.

Electrophysiological Recordings.

Scalp electrical activity (EEG) was recorded with 64 sintered silver/silver chloride electrodes mounted in an electrode cap (Quick-Cap; Neuromedical Supplies, Sterling, VA). Electrode positions included the standard 10–20 system positions along with intermediate positions and an additional row of low occipital electrodes. The vertical electrooculogram was bipolarly registered above and below the dominant eye and the horizontal electrooculogram at the outer canthi of the eyes. Electrodes positioned at the right and left mastoid positions served as common references, and the AFz electrode served as the ground. Electrode impedance was kept <10 kΩ during recording. EEG and electrooculogram recordings were continuously acquired at 1,024 Hz. After filtering with a narrow-band (49.5–50.5 Hz) notch filter to eliminate contamination from mains electricity, analysis epochs were generated offline starting 200 ms before stimulus onset and continuing for 1 s, aligned to a 200-ms prestimulus baseline. Artifact trials were rejected by applying a threshold criterion of ±60 μV.

Computational Analyses.

Step 1: Behavioral classification images.

Henceforth, all analyses are independently performed for each observer. We computed, independently for each of the five SF bands sampled, the subspace of information associated with the nuns and Voltaire behavioral judgments. We created a different NunsPlane per SF band, for each of the five SF bands [henceforth, NunsPlane(SFband)] and a different VoltairePlane per SF band [henceforth, VoltairePlane(SFband)]. Whenever the observer perceived the nuns (vs. Voltaire), we literally added the bubble masks (see Fig. 1) to the NunsPlane(SFband) [vs. VoltairePlane(SFband)] to integrate the information leading to these perceptions. To derive statistics on the most significant information associated with each perception, we divided the sampling frequencies in each plane by the total number of presentations and constructed a confidence interval (P < 0.05; one tail) around the mean for each of these proportions at each scale (26). We kept the areas above the confidence interval. There was no significant information in SF band 5, which was therefore removed from any further analyses. Fig. 2 depicts the selectively attended SF features combined across SF bands for each perception.

Step 2: Oscillatory EEG classification images.

We considered three main brain rhythms (theta, 4–8 Hz; alpha, 8–12 Hz; and beta, 12–25 Hz), measured on centroparietal electrode Pz^¶ in relation to the visual information represented in the four bandwidths of SF information sampled with bubbles that were relevant for behavior. For each combination of brain rhythm and SF band sampled (e.g., beta oscillations and the highest SF), we computed an independent classification image every 2 ms between −200 and 800 ms around stimulus onset. To compute the EEG classification images, we determined for each measurement time point the mean amplitude value of the filtered EEG signal^‖ across all nonartifact trials, considering only trials on which the observer perceived something (i.e., pooling together the nuns and Voltaire trials, whereas discarding don't know trials). We then assigned the bubble mask used on each of these trials to one of three bins: above, below, or around (−0.25 to 0.25 SD) the mean amplitude, depending on the EEG amplitude elicited on that trial. The EEG classification image was the sum of the bubble masks above the mean minus the sum of the bubble masks below the mean. By computing an independent classification image every 2 ms, we are able to represent at a fine temporal resolution how SF-specific information in the input modulates the amplitude of EEG oscillations.

Consider Fig. 3B, which illustrates the process for observer GB. EEG classification images are computed between 190 and 248 ms, in the beta band of EEG oscillations, for the second band of sampled SF information. The “movie” of classification images is indicated with a right-pointing arrow in Fig. 3B. Two classification images (highlighted with red boxes in Fig. 3B) are expanded for illustration, thresholded for their significant regions (P < 0.05), and interpreted by revealing the features correlated with modulations of beta amplitudes (here c is the face of the left nun, and d is the dress of the left nun). These computations were independently carried out for all oscillatory bands considered (i.e., theta, alpha, and beta) and four of the input SF bands sampled (i.e., discarding the lowest SFs).

Step 3: Relating Steps 1 and 2.

First-order sensitivity.

To determine the first-order sensitivities of the EEG activity to the features associated with behaviour, we correlated the EEG classification images with those features found to be significantly driving perceptual judgements by using Pearson's correlation. For the case of Voltaire, our behavioural templates consisted simply of the significant facial regions at each one of the SF bands. For the nuns, behavioural templates individuated the visual features present at each band. In Fig. 3A, features a to e contributed to the overt perceptual judgments of observer GB. Fig. 3B illustrates the correlations of behavioral features c (in blue) and d (in green) with the EEG classification images. Note that these correlations are cyclical (indicating that the sensitivity to the SF features is not sustained, but periodic, in accordance with refs. 7 and 9) and peak in the first sensitivity cycle at ≈208 ms and in the second cycle at ≈236 ms. Closer scrutiny of the classification images around these peaks (see the red boxes) reveal that EEG beta oscillatory amplitudes are modulated by the presence of features c and d, as is behavior. Fig. 3C presents a summary of sensitivity to all SF nun features in the beta band (the white box corresponds to the time interval expanded in Fig. 3B). Generalising this analysis to the two perceptions (i.e., the nuns and Voltaire), the three EEG oscillatory bands (theta, alpha, and beta) and the four bands of sampled SF information, we derived the first-order sensitivity measurements presented independently for each observer in .

Second-order sensitivity.

To determine the levels of information integration within each perception (for the nuns, across SF features; for Voltaire, across SF bands), we derived for each possible combination of SF features and SF bands the phase dependencies of the first-order sensitivities. We reasoned that a perceptual moment correlated with the conscious perception of a stimulus should isolate a moment in time when sensitivity to the information components (i.e., SF features or SF bands) should (i) be at its peak (ii) synchronously for the largest number of components (to illustrate, Fig. 3B indicates that the sensitivity curves to features c and d are in such synchrony). To this end, we used a standard method of phase-synchrony determination (31) and computed the convolution of the first-order sensitivity with a complex Morlet wavelet of form:

centered at the center frequency (f) of each band considered with σ_t = 6/f. The phase of this convolution (φ(t)) was then extracted for all times, t, and combinations of SF features and SF bands. The second-order phase sensitivity measure for each combination was then computed in temporal blocks of half a period at the band center frequency as follows:

where N is the number of points in each time block. In any comparison, if the phase difference varies little across the time in each half-period block, this value will be close to 1, indicating a synchronous sensitivity between two information components. If the phase of the signals is uncorrelated, however, the value will fall to 0, indicating a low synchrony between the components considered.

Note that, as computed in Eq. 2, phase relationships do not account for the magnitude of sensitivity of oscillatory processes to the compared information components (i.e., two SF features or two SF bands). Given that our goal is to isolate those phase-locked information components to which oscillatory brain processes are highly sensitive, we introduced an amplitude factor AF_t to our second-order measurement SOM_t:

In AF_t, FOM(t, f_i) is the first-order sensitivity measured for feature f_i at each time point. If the first-order value of each feature is high, then this value will be close to 1. If the EEG classification images were not sensitive to that feature, however, this value will fall to 0.

Abbreviations

SF

spatial frequency

HSF

high-SF

LSF

low-SF

EEG

electroencephalogram