The offline stream of conscious representations

Abstract

When do we become conscious of a stimulus after its presentation? We would all agree that this necessarily takes time and that it is not instantaneous. Here, I would like to propose not only that conscious access is delayed relative to the external stimulation, but also that it can flexibly desynchronize from external stimulation; it can process some information ‘offline’, if and when it becomes relevant. Thus, in contrast with initial sensory processing, conscious experience might not strictly follow the sequence of events in the environment. In this article, I will review gathering evidence in favour of this proposition. I will argue that it offers a coherent framework for explaining a great variety of observations in the domain of perception, sensory memory and working memory: the psychological refractory period, the attentional blink, post-dictive phenomena, iconic memory, latent working memory and the newly described retro-perception phenomenon. I will integrate this proposition to the global neuronal workspace model and consider possible underlying brain mechanisms. Finally, I will argue that this capacity to process information ‘offline’ might have made conscious processing evolutionarily advantageous in spite of its sluggishness and capacity limitations.

This article is part of the theme issue ‘Perceptual consciousness and cognitive access’.

1. Introduction

It has surely happened to you many times: you are absorbed in some reading or some video game, or just in your thoughts; someone asks you something and, at first, you do not react. Then they insist: ‘Did you hear what I said?’ You eventually attend to them, and you realize that you still have in your mind the message that was initially ignored ‘“Did you buy some milk?” … Ah! Yes, yes I did!’ This took a few seconds, a minute maybe. In this article, I would like to explore the proposition that this might be typical of consciousness' talent: delay some processing, and still be able to catch up later; allow for some amount of desynchronization with the unfolding of events in the external world.

This anecdote relates to a more general question: what is the temporal structure of our conscious mental life? It is both a fascinating and elusive question. Based on his introspection, the psychologist and philosopher William James compared the stream of conscious thoughts to a bird's life, made of flight and perching [1]. Through this metaphor, strong assumptions are made about the structure of the conscious stream which could be formulated as follows: (a) there is a single steam of consciousness (there is a single bird); (b) this implies that different types of representations, perceptions, memories, abstract concepts etc. are sewed together in a single stream; (c) there are stable moments of thoughts and faster transition periods: it is a succession; and (d), however, there is a continuity between thoughts.

There is no guarantee that introspective analysis, however refined it might be, reveals the true properties of how the mind works. And recently, these questions have started to be addressed using scientific methods including experimental psychology and neuroimaging. Most current research has focused not so much on the global structure of the stream, but rather on the first moments of processing following an external stimulation. From the current models of the brain mechanisms of conscious perception, we can distinguish two broad classes of hypotheses regarding its timing (figure 1). According to a first hypothesis, conscious perception arises during the initial stage of sensory integration: it has been proposed that, following a first ‘feed-forward sweep’ of sensory processing, the establishment of local recurrent loops within sensory areas constitutes conscious perception [2–4]. In this view, higher-level areas (attentional and executive systems) only have a modulating influence on this phenomenon, but do not play a key role in allowing conscious perception. According to a second hypothesis (figure 1), sensory integration is entirely ‘preconscious’; conscious perception typically arises not when the sensory cortex has converged onto a stable representation of the stimulus, but when this representation is made available brainwide, broadcasted to higher-level areas [5–8]. Regarding the timing of conscious perception, the first view states that conscious perception occurs rather early, within the first 200 ms of processing, while the second view postulates that it arises late, typically beyond 250–300 ms post-stimulus. Consequently, the debate has mostly focused on trying to date the moment when conscious perception arises, to distinguish these two models. Unfortunately, this endeavour has proved quite difficult and different experiments have yielded apparently contradictory findings [9,10]. One of the difficulties is that, when contrasting conscious and non-conscious processing of the same stimulus, we might find very early differences that actually predate the moment of conscious perception, but nevertheless correlate with it. Indeed, some experiments reveal differences relating to conscious perception of a stimulus even before the presentation of the stimulus, because the neural state that precedes stimulation can influence whether we become conscious of a stimulus or not—especially for faint stimuli at detection threshold or ambiguous stimuli [11,12].

Figure 1.

Hypotheses and predictions. The two main hypotheses about the timing of conscious processing relative to stimulus onset are illustrated here. Hypothesis 1 postulates that conscious processing occurs ‘early’, during the build-up of sensory representations, and ‘locally’ within sensory cortices; it predicts that conscious perception should always be time-locked to the moment of sensory input. Hypothesis 2 postulates that conscious processing occurs later and corresponds to broadcasting of sensory information to a global network of areas; it predicts that conscious perception is time-locked to this broadcasting event, not necessarily to sensory input.

Interestingly, there is a much stronger distinguishing feature between these two positions than the exact timing of conscious perception, one that has not received much attention so far. It relates to the question of what conscious perception is time-locked to. In the first ‘early and local’ view, conscious perception is necessarily time-locked to the stimulus. In the ‘recurrent loops’ proposition by Lamme [2], conscious perception occurs as soon as local recurrent loops are established within sensory areas. Neurophysiological studies show that these recurrent loops are tightly time-locked with the external stimulation: they arise immediately after the passage of the fast feed-forward sweep of activity, i.e. 100–150 ms after stimulus presentation in monkeys' visual cortex [4,13–17]. This proposition matches our intuition that our perception, although necessarily delayed relative to the timing of external stimuli, is nevertheless time-locked to them. On the other hand, according to the ‘late and global’ view, conscious perception is time-locked to the broadcasting event. So, in principle, it might not always be time-locked to the stimulus itself. Indeed, interactions between sensory areas and higher-level systems such as attention have been shown to be very flexible in time [18–20]. This second view therefore makes the intriguing prediction not only that conscious perception is delayed relative to the external stimulation, but also that we could find instances where it is desynchronized from it. As long as a representation of a past sensory stimulus is still present within the sensory cortex, it could, in principle, become conscious through broadcasting to a wider network, at an arbitrary time point relative to the initial timing of the stimulus.

This prediction contradicts an assumption that is widespread in studies on attention and perception: it is often considered that changes in attention occurring after stimulus presentation can no longer influence its perception per se, but can only affect ‘post-perceptual decisional stages' of processing [21–25]. One formulation of this assumption can be found, for example, in Kinchla et al. [22, p. 1146]: ‘[i]f the cue comes after the stimulus, it cannot have any effect on the perception of the stimulus. Rather, effects from a cue that appears after the stimulus must reflect non-perceptual processes’. This view is also evident in the classical interpretation of the iconic memory results, where it is assumed that ‘more is seen than what can be remembered’ [26] and that post-cueing attention after the stimulus in this protocol biases consolidation in working memory but does not affect perception per se.

In the following, I will describe the retro-perception phenomenon, which contradicts this classical assumption and provides evidence that conscious access is not necessarily time-locked to sensory processing. I will show that the temporal flexibility of conscious access is also highlighted by the well-known psychological refractory period and by recent discoveries about latent working memory. I will then detail my proposition about the contrasting temporal structure of sensory processing and conscious processing, and how they interact. Finally, I will underlie the consequences of this view in terms of re-evaluating the computational advantages of conscious processing.

2. Retro-perception: perceiving the past

In a series of experiments, my colleagues and I tested whether conscious perception could be triggered by retrospective attention [27,28]. Participants had to detect faint grids of oriented lines (Gabor patches) presented to the left or to the right of fixation, in one of two circles (figure 2a). For each participant, the target contrast was adjusted close to their perceptual threshold (80% performance in reporting the left or right tilt). Then, either before or after target presentation, attention was cued to the same or the opposite side by a brief dimming of one of the placeholders. As the location of this cue was random relative to the target location, it constituted an involuntary or ‘exogenous’ cue. At the end of each trial, less than one second after target presentation, the true target location was indicated by a central arrow and participants were asked to report its orientation and how well they had perceived it. The results revealed a ‘retro-perception’ phenomenon: cues presented several hundred milliseconds after target offset drastically improved orientation discrimination when they were valid (same side as the target) as opposed to invalid or absent (figure 2b). The same effect was observed for orientation sensitivity, detection sensitivity, subjective visibility (figure 2c) and orientation matching in a total of four replications in our laboratory, involving a total of 80 observers [27,28]. The effect was also replicated in an independent laboratory in two other experiments, involving an additional 40 observers [29].

Figure 2.

Retro-perception experiments: results and interpretation. (a) Experimental protocol. (b) Performance on orientation discrimination as a function of the stimulus-onset asynchrony (SOA) between the target and the cue, revealing both a classical pre-cueing effect (performance is better when a valid precue just precedes the presentation of the target) and a retro-perception phenomenon (performance is better when a valid retro-cue is presented after the target). Results from 18 participants. The dotted line represents the baseline performance in the absence of a cue, as achieved for each participant through a psychophysical staircase just before the experiment. Error bars represent standard error of the mean effect size. Significance: *p < 0.05; **p < 0.01; ***p < 0.005. (c) Response distributions on the subjective visibility scale for retro-cues at 100 ms show that valid retro-cues decrease the number of ‘not-seen’ trials (subjective visibility = 0) and increase the number of trials with good visibility (subjective visibility 50%). Statistics as in (b). (d) Interpretation of the results. (b) and (c) are adapted from Sergent et al. [28].

These results seem to validate our initial prediction. However, before we can conclude that this effect does indeed bear on conscious perception, it is important to explain why we can rule out non-perceptual interpretations of this phenomenon.

• (1) First, we can rule out the possibility that this is a post-perceptual decisional effect: indeed in all these experiments, the cues are not informative on the target location, and participants know that from the instructions and then from their experience with the stimuli. On top of this, any uncertainty about the target position is eliminated at the time participants are asked to make their response, so that the preceding random cue should not bias on which side the decision is made.

• (2) Second, we can rule out the possibility that this is a memory effect. Contrary to what happens in working memory or iconic memory paradigms, in this protocol, there is only one target on each trial, so there is no other item that could compete with the target for working memory resources. Furthermore, participants are asked about their perception less than 1 s after target onset (figure 2c). So, when participants report not seeing a target that was presented on its own less than a second ago (high proportion of 0% visibility in the control condition; figure 2c), it would be difficult to argue that they actually saw the target but forgot it very quickly, and that retrospective attention on the correct side only prevented this memory decay.

Beyond these general arguments, two specific analyses confirmed the interpretation that retrospective attention makes participants see targets that they would not have perceived consciously otherwise. First, visibility ratings across trials followed a bimodal distribution that could be modelled by a mixture of two types of trials (figure 2c): trials where participants have perceived the target consciously (unimodal distribution centred on high ratings, matching the distribution obtained when the target was validly pre-cued, and thus easy to see), and trials where participants had no conscious perception of the target (unimodal distribution peaking at 0, matching the distribution obtained when the target was absent) [28]. Retrospective attention did not drastically increase the visibility of seen targets; instead, its main effect was to reduce the number of ‘not-seen’ trials and increase the number of ‘seen’ trials (figure 2c). The proportion of such ‘retro-perception’ trials could be estimated at 17% of the total number of trials for retro-cues at 100 ms and 10% for retro-cues at 400 ms. This interpretation was confirmed and refined using an objective task instead [27]. On each trial, participants were asked to reproduce the target orientation as accurately as possible by adjusting the orientation of a probe with the computer mouse. The analysis of error distributions revealed that retrospective attention did not affect the precision of target representation on seen trials, as would be expected if retro-cues protected seen targets from rapid decay. Instead, retrospective attention reduced the number of random guesses. In other words, it reduced the number of trials where participants had no conscious access to the target representation.

This phenomenon thus provides evidence for the counterintuitive prediction that conscious processing need not be time-locked with initial sensory processing, and validates a critical prediction of Hypothesis 2 (figure 1). Within this framework, the following neural mechanism could be proposed: when the target is presented, visual information rapidly and automatically progresses through the hierarchy of visual areas. Initial processing is rapidly refined by recurrent loops between the different levels [4,14–17,30]. However, this initial sensory information is not conscious yet, according to Hypothesis 2. On some trials, initial conditions might be favourable, e.g. by chance initial sensory processing was slightly stronger on this trial [31,32] and/or by chance spontaneous visual attention was at the right location at the right moment [33–36]. In this case, initial sensory processing is sufficient in itself to trigger broader coupling between sensory and higher-level areas within a ‘global workspace’, corresponding to conscious access. On other trials, the initial conditions might be less favourable and should lead to a progressive decay of this information within sensory cortices, with no coupling with higher-level areas, and no conscious perception (figure 2d). It is on those trials that retro-perception can occur: by directing attention to the location of the past target while there is still a trace of this target within sensory cortices, this local information is reactivated, amplified and broadcasted within a global workspace. Through this workspace, this information can broadly affect current processes and notably the systems allowing report of this information. In other words, this information is consciously accessed.

Importantly, although top-down attention signals from the retro-cue most probably reactivate recurrent activity within sensory areas, we can argue that this alone cannot account for the effect on conscious perception. If there is some trace to be reactivated in the posterior regions, it implies that the initial sensory processing triggered by the target itself was sufficiently strong to induce local loops in the first place, before the retro-cue. Indeed, stimuli that evoke only feed-forward activation have no lasting impact, and therefore, there would be no sensory trace to reactivate [3]. If posterior recurrent signal was sufficient for conscious experience, as postulated by the ‘early and local’ view, late reactivation of this signal by retrospective attention should not determine whether the participant is conscious of the stimulus or not. There is something more happening during reactivation by the retro-cue, which did not occur during the initial processing of the target, and which allows conscious perception. This might be global broadcast.

The full temporal extent of the retro-perception effect remains to be tested empirically, but in principle retro-perception could occur as long as sensory cortices retain some trace of the stimulus. Some experiments show that even for visual stimuli at low contrast, sensory cortices can retain some trace of the stimulation in the order of seconds after stimulus offset [37]. For stimuli with higher contrast, the seminal experiments by George Sperling have shown that a very high number of items can be retained in iconic memory within a second after their presentation, and recent studies show that accessible sensory traces can remain in the visual cortex beyond 5 s after stimulus presentation [18,38–41]. Sensory buffers are also found in other sensory modalities, for example sensory memory for auditory stimuli is estimated to last between 1 and 10 s [42,43]. In principle, retro-perception could occur on any type of content that can be buffered within sensory cortices.

Another aspect of this phenomenon that remains to be tested is the perception of temporal order: is the target perceived as occurring after the retrospective cue during retro-perception? Such illusions of timing have been observed in other types of phenomena [44]. However, this issue remains open in the case of retro-perception: it might induce an illusion of timing, but it is also possible that an accurate representation of the initial timing of the target is recorded pre-consciously during the first sensory analysis of the target, and recovered at the moment of the retrospective cue, along with the other properties of the target [45].

Retro-perception could be linked to the larger family of ‘post-dictive’ phenomena. Research on perception has indeed documented many situations in which a subsequent event radically changes the perceived properties of a preceding stimulus [46–50]. For example, the perceived position of a flash is influenced by subsequent movement of a nearby stimulus [47,48]. However, in these experiments, the subsequent event does not determine whether the stimulus becomes conscious or not, which leaves open the question of when conscious access occurs, and whether post-dictive influence occurs pre-consciously or not [45,49]. By contrast, retro-perception provides direct evidence that conscious access itself can be triggered by a retrospective event.

In conclusion, retro-perception reveals a flexibility in the moment of conscious access relative to sensory input that was previously unsuspected. In the next sections, I will highlight two other lines of research that demonstrate the flexibility of conscious access relative to external input, and that will help sketch out a proposition on the temporal structure of the ‘stream of conscious representations’.

3. The psychological refractory period: catching up with the past

While retro-perception provides direct demonstration that conscious access can arise at a flexible moment following stimulus onset, more indirect but very robust evidence in this direction can be found in the classical phenomenon called ‘the psychological refractory period’, abbreviated as PRP [51,52]. This term refers to the long-standing observation that whenever we have to react to two successive targets, response to the second target is delayed by the processing of the first if the inter-stimulus interval is less than 500 ms. Both behavioural and neuroimaging evidence suggest that initial sensory processing of the second target is unaffected by the PRP, and that it is a secondary ‘central’ stage of processing that is delayed by the preceding task (figure 3a) [52–58]. In the case of a single task, this central stage is estimated to start around 300 ms following stimulus onset, but during dual-task interference, central processing of the second target is delayed until central processing of the first target is completed [54,59]. Very elegant experiments have demonstrated that this central stage relates to conscious access: indeed, inserting masks after each target transformed the PRP into an attentional blink phenomenon, where the second target actually fails to be perceived consciously [56,60].

Figure 3.

Proposition regarding the temporal structure of sensory processing versus conscious access. (a) Classical interpretation of the psychological refractory period: ‘S’ denotes the sensory processing phase, ‘C’ denotes the central phase, associated with conscious access in the present article, and ‘M’ denotes the motor planning and execution phase. The central process for target 2 (C2) is delayed until the central process for target 1 is completed, which explains the refractory period observed in response times (RT). The panel on the right shows, in the same format, an interpretation of what happens during retro-perception: the initial sensory processing of the target (S1) is not sufficient in itself to trigger conscious access. However, cueing attention to the past target with a retrospective cue (S2, C2) promotes to conscious access the initially unconscious sensory trace from the target (C1). (b) The proposed distinct temporal properties of the stream of sensory processing and the stream of conscious access. Assuming a succession of different external stimulations (V for visual and A for auditory) and spontaneous internal activations in local circuits (LTM for spontaneous activation of a representation in long-term memory), sensory processing will be time-locked to these different events and will unfold in parallel as long as the representations do not overlap. For conscious access, in contrast, there is a succession of conscious access episodes that can desynchronize from the stream of sensory processing in different ways (retro-perception, latent working memory and psychological refractory period) so that conscious access can go back in the history of events (conscious access to V4 and reactivation of latent working memory of V5) and then catch up again (conscious access to A6 and A9 for example). See the text for a more detailed description of the different scenarios.

To summarize, during the psychological refractory period, conscious access to sensory information relating to the second target is delayed for several hundred milliseconds by conscious access to the first target [54]. It is only under very constrained circumstances (masking of both targets) that this substantial delay can result in a failure to perceive the second target consciously (attentional blink) [59]. We can therefore conclude that, normally, conscious access mechanisms can catch up with lingering preconscious information after conscious processing of the preceding information is achieved.

The PRP is remarkably ubiquitous [61–63]: it is observed for all kinds of targets and tasks, with any combinations of stimulus modality (e.g. response to an auditory stimulus can delay response to a visual one and vice versa). It should thus occur frequently in everyday life, as shown in driving simulations [61]; it might be constitutive of the spontaneous stream of conscious representations. The example cited as a preamble, where you initially fail to react to what someone says and then recover the message when they manage to grab your attention, probably corresponds to a case of spontaneous psychological refractory period followed by retro-perception: the task you were engaged in (reading and surfing the web) involved several targets and several computations that delayed conscious access to auditory information from your interlocutor. Some trace of the message nonetheless remained within auditory areas, and the eventually exasperated tone of your interlocutor serves as an attentional retro-cue that disengages you from your current task and reactivates echoic memory to retrieve the message. If it were not for your companion who witnessed the delay in your response, you would probably not have noted it.

The idea that PRP might occur frequently in everyday life, given the robustness and ubiquity of the phenomenon in the laboratory, might seem very counterintuitive at first. It seems to contradict our intuition of a rather smooth flow of experience. But there are many indications that our introspection does not always faithfully reflect the way information is actually processed in our brain. And indeed, in the case of the PRP, behavioural experiments demonstrate that while we can accurately judge the time we spent consciously processing the targets, we have no introspection of the delay in conscious access induced by the PRP: this refractory period seems to correspond to an ‘introspection blindspot’ [64,65]. Flexible delays in conscious access might be both very frequent and totally unnotable, except in special cases where we have external indications that a delay has occurred, such as when we fail to answer our interlocutor like in the above example.

4. Latent working memory: recovering past information on demand

Recent discoveries about the mechanisms of short-term memory and working memory demonstrate the flexibility of conscious access at even larger time scales. Short-term memory refers to the capacity to retain information over a delay, while this information is no longer present in the external world. When this information is selected and retained for a future purpose, it is called working memory. For example, when baking a cake, the information of where you left the bowl is encoded in working memory while you are fetching the pack of flour from a cupboard. Until recently, a commonly held view postulated that order for information to be retained in working memory, this information had to be represented in an active form throughout the delay period, within high-level areas such as prefrontal cortex [66,67]. But in the last 10 years, new techniques of analysis, such as multivariate pattern analysis, have allowed new discoveries that substantially refine our view on the neurophysiology of working memory.

A large body of recent evidence suggests that memory of past sensory stimuli can stay in a ‘latent’ form within sensory areas, during an arbitrary delay period, and be reactivated at a later time, if and when this information becomes relevant. For example, in an fMRI study, participants were presented with two large patches of oriented lines, at above threshold contrast, so that both stimuli were seen [41]. After stimulus presentation, a post-cue indicated which of the two orientations had to be retained for future report. Participants fixated at a blank screen during the retention period that lasted 11 s. During this period, blood-oxygen-level dependent activity in early visual areas (V1–V4) dropped drastically, as expected in the absence of visual stimulation. However, despite this overall drop in activity, the pattern of signal across voxels was specific of the orientation that had to be memorized, a specificity that was sustained until the information could finally be compared with the probe presented at the end of the trial. Similar results were found for different types of features, such as colour [40]. Other studies have shown that such latent, low-energy memory representations within sensory cortices can be reactivated at an arbitrary delay by retrospective cues indicating the relevant stimulus or stimulus dimension [18,39,68,69]. Several authors suggest that this low-energy form of memory might be subtended by rapid rearrangements of synaptic weights within sensory cortices following initial activation [70–72]. Reactivating this reconfigured network, through retrospective attention [18] or via a bottom-up boost from a following stimulus [73], would reveal this hidden structure and recover the memory. This mechanism would explain why many recent experiments find a high susceptibility of working memory performance to manipulations of retrospective attention, even when there is a very limited number of items such as four or less [74–77], which should all fit within the classical capacity limits of working memory and thus should no further benefit from attention according to a classical view.

Interestingly, this phenomenon of ‘low energy’ or ‘activity-silent’ retention of information during working memory tasks is not only found within sensory areas: it can also be observed within the prefrontal cortex. One of the most classical experiments for testing working memory is memory-guided saccade [66]. In these tasks, a stimulus is flashed at a random location while the subject is looking at a fixation cross in the centre of a screen. After a certain delay, the fixation point disappears and the subject is allowed to make a saccade to the remembered location. Between the offset of the memory stimulus and the execution of the saccade, the representation of this location is, functionally speaking, in ‘working memory’. Neurophysiological recordings from monkey prefrontal cortex classically show sustained spiking activity specific of the memory location during this delay [66,78]. More recently, it has been shown that if a concurrent attention task is introduced, the sustained activity relating to the memorized location disappears during delay [78,79]. But this does not mean that the memory of the location is lost: indeed, as soon as the concurrent task is completed, and the subject thus presumably shifts back to the memory task, the location information is reactivated within the prefrontal cortex and the monkey can still succeed in the memory task. This presumably relies on a latent, ‘non-spiking’ form of retention during the delay.

Following these recent discoveries, our views on the neurophysiological basis of working memory and its relationship with sensory processing are rapidly evolving, and they depart from earlier views in two respects (see D'Esposito & Postle [70] for review). First, one traditional view of the transition from sensory processing to working memory postulated that a subpart of a very rich but rapidly decaying sensory information is transferred into a different format that is more stable but also more limited. But recent evidence suggests an important role of sensory cortices well beyond initial sensory processing: once a stimulus has been processed within sensory areas, this information can continue to live within the same network, sometimes over delays of several seconds as shown in the former paragraphs, and its fate for cognition is not immutable but on the contrary very flexible. Whether the stimulus is consciously perceived or not can be influenced by subsequent events, notably retrospective attention, as in retro-perception experiments [27–29]. Then, even once it has been perceived consciously and identified as potentially relevant, the latent sensory trace can be flexibly recovered depending on subsequent attentional requirements [39]. There is a second respect in which our views are evolving: while it has long been held that working memory relied on sustained delay activity, in other words that a representation in working memory had to be continuously active from encoding to final retrieval, recent discoveries moderate this proposition by suggesting that, even in prefrontal cortex, representations in working memory can take two forms: an active form and a ‘latent’ or ‘activity-silent’ form.

What does this tell us about the relationship between working memory and conscious access? Several authors have suggested that the content of conscious access could be equated with the content of working memory [6,80–83]. But these new results suggest a more complex relationship: the content of working memory could be said to correspond to all the information that remains in the form of rearranged synaptic weights following sensory processing, for future purpose. When this sensory trace is in its latent form, i.e. in the absence of active firing, this corresponds to working memory contents that are potentially available but not currently accessed, i.e. not currently conscious. Reactivation of latent working memory would correspond to current conscious access to this memory. In other words, once a stimulus has been perceived (first conscious access episode), it can leave a latent trace that can be consciously accessed again later, depending on evolving task demands. If we transpose these ideas to our initial example of an everyday working memory task, when we leave the bowl on the table, we first consciously perceive its location and encode it in parietal and prefrontal areas. We might not be continuously conscious of this location information while we are fetching the pack of flour and looking for the salt. During these concurrent tasks, the information presumably remains in a latent form within prefrontal and parietal cortices. When all the ingredients have been found, we switch to the task of returning to the bowl; information on its location is reactivated, at which point it becomes conscious again.

Working memory and conscious perception usually correspond to two distinct domains of experimental investigation. However, if we want to ultimately understand the neural basis of conscious experience in general, we must understand what is common and what is different between different types of conscious experience: the conscious experience of a percept, of a sensory memory, of a long-term memory [8]. This question becomes particularly acute with these recent discoveries showing that working memory representations might be held within the same areas where the stimulus itself is primarily processed. If both types of representation rely on the same processors, what makes the difference between a direct perceptual experience and a re-instantiation of this experience in working memory? This remains to be explored experimentally, but one important element that might distinguish direct perception from the reminiscence of prior perception is the quality of the sensory activity itself, in terms of strength and distinctiveness. Indeed, when strong distinctive sensory activations occur in the absence of external input, this is associated with hallucinations, i.e. the illusion of direct perception [84]. Another element might relate to whether the sensory activity is associated with some episodic memory or not: if it is, it might be interpreted as a ‘vivid’ memory. Interestingly, the retro-perception phenomenon provides indication that, under some circumstances, reactivating past sensory information can actually feel like direct perception: indeed, participants had no problem qualifying their experience of the target during retro-perception trials as ‘seeing’ the target [28].

In conclusion, these different lines of evidence suggest that conscious access mechanisms are decoupled from sensory processing, within the range of a few seconds, so that we consciously perceive a more or less recent past. The first occurrence of this phenomenon, following sensory processing, is experienced as perception (retro-perception and psychological refractory period). When this reactivation follows a first episode of conscious access, reactivation of latent sensory information is experienced as a reminiscence of perceived information.

5. The offline stream of conscious representations

These different lines of evidence all point to the idea that conscious access mechanisms can be, to some extent, desynchronized and thus decoupled from sensory processing (figure 1, Hypothesis 2). This idea has several implications:

1. It can help resolve apparent contradictions about the richness of conscious experience. Indeed, such flexibility confers to conscious access mechanisms a very powerful way to overcome their instantaneous capacity limits: although there is a limit to the amount of consciously accessed information at any given moment, information that is not initially accessed can still be at a later time.

2. It provides indications on the computational advantage offered by this otherwise slow and limited process of conscious access by allowing cognition to be partly decoupled from the flow of external information.

3. Finally, it opens the possibility that representations of different types, sensory and non-sensory, can be integrated within the same stream of conscious processing, or, in other words, that conscious processing of different types of representations can be accounted for by a common mechanism, as proposed notably by the global workspace model [6–8].

Before turning to theses implications, I will first detail the proposition and examine its plausibility with regard to our current knowledge of neurophysiology. In this proposition, the stream of sensory processing and the stream of conscious access interact with each other, but have essentially different temporal dynamics, schematically depicted in figure 3b.

• — The stream of sensory processing is ‘online’:

  • (i) All incoming stimulations are processed as they arrive through fast automatic routes, with very little interference between the processing of different stimuli [53]. For example, stimuli presented at different locations in the visual field are rapidly processed in parallel within visual areas. This leads to very fast and efficient analysis of a large amount of sensory information within a few hundred milliseconds [85].
  • (ii) Temporal integration can occur at different scales, depending on the time constants of each processing stage: for example, area MT will integrate visual information over a certain duration to infer motion. However, this remains an ‘online’ process in the sense that all inputs will be integrated as they arrive. Some post-dictive influences possibly occur at this stage.
  • (iii) Ongoing brain oscillations at the moment of stimulation can modulate the strength of sensory processing [31], but with little incidence on its latency. More generally, although the brain state prior to stimulation can modulate the strength of sensory processing, its latency, on the other hand, is highly reproducible for each sensory modality, in the range of a few tenths of milliseconds [15,86].
  • (iv) This unconscious stream of sensory analysis can lead to stimulus-triggered conscious perception, depending on its strength and relevance and on whether conscious access mechanisms are occupied by other representations [59].
  • (v) It can also lead to fast appropriate behaviour, when the stimulus–response association is well established.
  • (vi) It allows fast reorienting of attention when needed, through exogenous attention.
  • (vii) And finally, previously processed information leaves traces in the system that can be used ‘offline’ by the conscious access system.
• — The stream of conscious access is ‘offline’:

  • (i) Conscious processing of one stimulus can be delayed by the processing of preceding stimuli, as is the case in PRP experiments.
  • (ii) A previously missed event can be integrated in the stream if and when it becomes relevant, as in retro-perception experiments.
  • (iii) Consequently, the conscious access stream can be qualified as being ‘offline’: there is no fixed delay between an external input and the moment when the representation is consciously accessed, and, furthermore, the order in which we access different representations is flexible relative to the timing of the corresponding stimuli in the external world (figure 3b).
  • (iv) Importantly, this reordering should not necessarily lead to an erroneous perception of temporal order. Indeed, there are indications that temporal aspects of a stimulus can be extracted during sensory processing [45,49]. So, the veridical temporal label or time-stamp of a previous input could, in principle, be reactivated, in the same way as any other aspect of the stimulus can be reactivated (but see [44]).
  • (v) Constant input from the exogenous attention system prevents excessive drift of the offline system, and allows for keeping track of important events in the environment, even when the offline system is occupied by internal representations (searching long-term memories and planning future events).
  • (vi) Once a representation has been consciously accessed, it leaves a less labile trace within sensory cortices that can be integrated again at will.
  • (vii) Once a representation has been consciously accessed, it leaves a trace in prefrontal cortex and it is also part of our episodic memory: even if we do not currently have access to it, we know we have it, we know where to find it. This is why, even in their latent form, working memory representations can efficiently be recovered.
  • (viii) Many different factors compete for determining what will be accessed next: bottom-up factors such as exogenous attention to salient external events or salient internal representations (emotional memory), but also top-down factors either directed at external stimuli (endogenous attention), or directed at internal representations (abstract thoughts and inner speech). During passive perception, the bottom-up component essentially determines the flow of conscious representations, while during more controlled cognition, the central executive system (notably the prefrontal cortex) plays an important role in shaping the sequence of different conscious representations.

Figure 3b shows a schematic representation of the online and offline streams for a sequence of visual (V) and auditory (A) stimuli, mixed with some spontaneously activated internal representation (LTM, long-term memory), displaying the different types of phenomenon described in previous sections. In the case of the three initial stimulations V1, A2 and V3, conscious access naturally follows sensory processing in the same order. In the case of the visual stimulus V4, normally its representation would remain unconscious due to interference from the conscious representation of V3, but in this example, it can finally be recovered because of a retro-cue following the experience of V5. In the case of the auditory stimulus A7, on the contrary, no such cue occurs and A7 remains unconscious. While the preconscious representation of A7 is fading within sensory areas, an internal memory is activated and promoted to conscious access (LTM8). This then leads to the reactivation of the sensory trace of V5, which had remained in latent working memory form following initial conscious access. Finally, conscious access to the last stimulus A9 is delayed by the preceding conscious access episode, reflecting the psychological refractory period. As a result, we see on the Y-axis that the timing of sensory processing roughly follows the timing of external stimulation (and spontaneous internal activation, in the case of LTM8), while conscious access episodes are sometimes reordered with respect to the initial activations, not necessarily following the initial sequence.

6. Implications for debates about the richness of conscious experience

There are currently important debates concerning the ‘richness of conscious experience’ [2,3,87–94]. Indeed, during the past decades, experimental psychology has shown the stark limitations of conscious processes: it seems that conscious access mechanisms, which allow conscious report, can only handle a limited amount of information at the same time (capacity limits of working memory [95,96] and change blindness phenomena [97]) and cannot do two things at the same time (inattentional blindness [98], dual-task interference and psychological refractory period). These limitations seem to be at odds with our impression that we perceive a rich and detailed world around us [88]. Therefore, some authors have proposed that conscious experience overflows conscious access [2,3,88,94] or, in other words, that there is a form of conscious experience even outside conscious access, maybe relating to recurrent loops within sensory areas (figure 1, Hypothesis 1) [2,3,99].

A central argument for this view comes from iconic memory experiments. When several items are briefly displayed, for example an array of letters, only four to five letters can accurately be reported, irrespective of their total number. However, an attentional cue presented after the display can still flexibly determine which four to five letters can eventually be reported [26]. This has been interpreted by some authors as showing that we see more than we can remember and report: we have a conscious experience of all the letters in detail but the systems that allow maintenance and report of the information are too limited to do justice to this rich conscious experience. Importantly, this interpretation of the iconic memory phenomenon relies on the implicit assumption that conscious perception is finished and accomplished before the attentional cue, and that the attentional cue only prompts report but does not influence conscious perception itself.

Recent evidence on the temporal flexibility of conscious access reviewed in this article allows us to take a fresh look at these problems. Indeed, as argued earlier, although conscious access is a slow and capacity-limited process, temporal flexibility confers to conscious access mechanisms a very powerful way to overcome their instantaneous capacity limits: although there is a limit in the amount of consciously accessed information at any given moment, information that is not initially accessed can still be at a later time.

Within this framework, we can reinterpret what happens during iconic memory experiments. Several lines of evidence suggest that, contrary to what we would spontaneously assume, conscious perception is not a single event that occurs just after stimulus presentation, but instead relies on several successive episodes of conscious access to different levels of representation [87], following a flexible process. As illustrated in figure 4, when an array of letters is presented, the input is rapidly processed through a bottom-up stream from low-level visual areas extracting local features to object recognition areas (temporal cortex) and to areas computing the ‘gist’ or ‘summary statistics’ of the scene (figure 4a). The first representations to be consciously accessed are probably those relating to the gist of the scene (figure 4b) [100]: indeed, a recent study shows that participants can accurately report ensemble properties (for example, colour diversity) without being able to report the individual items that subtend these summary statistics [93]. Then, either through prompting by a retrospective cue, or through spontaneous exploration of the visual trace, the representation of some details, for example, some individual letters, can be consciously accessed (figure 4c). So, retrospective activation can help us note a detail that we did not actually perceive consciously in isolation when we first perceived the gist of the scene. Still, it makes sense for us to say that we saw it, if the information is there in our sensory cortices. What is important is not when we became conscious of the information, but rather if the information is available or not. Finally, by the time conscious access mechanisms are available to process other details, the sensory trace of previously ignored details might be too degraded, which limits the amount of details that can be reported.

Figure 4.

A global workspace model of iconic memory: the input (a briefly flashed array of letters) is processed through thalamo-cortical columns at different levels, from low-level vision and the extraction of very local features to object recognition and the computation of summary statistics over the whole scene (HF: summary statistics on high-frequency details; LF: summary statistics for low-frequency layout). This initial activation can reach central hubs of the global workspace (mostly located in fronto-parieto cingulate areas), which are characterized by dense reciprocal connectivity with a large number of peripheral processors (notably allowing branching peripheral processors that usually do not communicate). Following this first activation, a global workspace can form around the high-level representations: strong reciprocal loops help maintaining and broadcasting this information, notably to areas allowing report. This constitutes a first conscious access to the gist of the scene (b). Then, in response to a post-cue, the global workspace is reshaped and broadcasts a specific detail of the scene (the identity of one of the letters) that is still present as a trace within local processors (c).

So, interestingly, limitations to the amount of information that can be reported in iconic memory or working memory experiments might relate only partly to the momentary limitation of conscious access system, which can be compensated for by temporal flexibility. Instead, a large part might be accounted for by mutual inhibitions occurring within sensory buffers, as suggested by Franconeri et al. [101]. This explains why capacity limits are found to be quite different depending on the type of information that needs to be retained [102]. Another very nice demonstration of the constraints imposed by mutual interactions at low level can be found in studies of multiple object-tracking: we can double the number of moving visual objects that we can track if they are divided in the left and right hemi-fields, so that they are encoded in two independent visual maps [103].

More generally, perception is an inherently active process: if, instead of disappearing as it does in iconic memory experiments, the scene stays on the screen, we immediately start exploring the scene with eye movements, to grasp details [104,105]. The initial gist or summary statistics would guide this exploration. In the words of Kevin O'Regan, we can use the world as an outside memory: even though we are not instantly conscious of every detail, we can look for them [106]. If the scene disappears, sensory memory allows deploying a similar process, with the limitation that this memory tends to fade with time, contrary to the outside world.

So, despite the instantaneous limitations of conscious access, the richness of our perception might rely on flexible conscious access to a stable external world and internal buffers.

7. Computational advantage of decoupling conscious access from sensory processing

Evolutionarily, the possibility to desynchronize from external events is a decisive cognitive asset. While a strictly online system is compelled to ‘live in the present’, in constant reaction to the environment, the possibility for offline processing seems necessary for developing a more complex and creative cognition. It allows intermixing, within a coherent stream of operations, the perception of the environment with ‘non-perceptual’ periods of planning, simulations of the possible outcomes of an action and retrieval of long-term memories.

Intermixing can occur without losing track of interesting things in the environment thanks to sensory buffers and the possibility to integrate sensory traces a posteriori within the conscious stream. The interaction between unconscious online processes and conscious offline processes therefore achieves an ideal balance between reactivity to the environment and detachment from it.

In a review on the cellular basis of working memory, Patricia Goldman-Rakic wrote: ‘the brain's working memory function, i.e., the ability to bring to mind events in the absence of direct stimulation, may be its inherently most flexible mechanism and its evolutionarily most significant achievement’ [66, p. 483]. The present proposition echoes these considerations: it emphasizes the computational advantages of combining high capacity specialized buffers with a selective process that can flexibly access information in any of these buffers, at its own pace. Current evidence shows that this flexibility might not be restricted to specific working memory tasks: it might be the hallmark of a more general conscious access system that mixes periods of working memory tasks with periods of direct perception and periods of daydreaming within a coherent stream.

8. Conclusion

The global workspace model [8] and the global neuronal workspace model [6,7,107] have been built to account for the contrasting properties of conscious versus unconscious processing, notably the fact that conscious processing seems to show stark capacity limits. These limitations contrast with the impressive capacities of unconscious processing. Paradoxically, the thus described properties of conscious processing seem to point to its limits rather than to its computational advantage: something that has been criticized by some authors who remark that the extreme capacity limitations described for conscious access mechanisms seem to be at odds with our intimate sensation of a vivid and rich conscious experience. Here, I argue that consciousness has evolved as a system that relies on all sorts of unconscious buffers and opens the possibility to de-correlate the behaviour of an individual from immediate reaction to the environment, and allows computations and ensuing behaviour to take place, at least in part, according to a longer term, internal agenda, that mixes several elements of mental life within a stream.

Data accessibility

This article has no additional data.

Competing interests

I declare I have no competing interests.

Funding

This work was supported by a grant from the ANR (Agence Nationale de la Recherche) ANR-17-CE37-0004-01.