Time and time again: a multi-scale hierarchical framework for time-consciousness and timing of cognition

Abstract

Temporality and the feeling of ‘now’ is a fundamental property of consciousness. Different conceptualizations of time-consciousness have argued that both the content of our experiences and the representations of those experiences evolve in time, or neither have temporal extension, or only content does. Accounting for these different positions, we propose a nested hierarchical model of multiple timescales that accounts for findings on timing of cognition and phenomenology of temporal experience. This framework hierarchically combines the three major philosophical positions on time-consciousness (i.e. cinematic, extensional and retentional) and presents a common basis for temporal experience. We detail the properties of these hierarchical levels and speculate how they could coexist mechanistically. We also place several findings on timing and temporal experience at different levels in this hierarchy and show how they can be brought together. Finally, the framework is used to derive novel predictions for both timing of our experiences and time perception. The theoretical framework offers a novel dynamic space that can bring together sub-fields of cognitive science like perception, attention, action and consciousness research in understanding and describing our experiences both in and of time.

Highlights.

• The paper focuses on temporality to develop a framework for consciousness theories.

• We propose a nested hierarchical model of multiple timescales for time-consciousness and timing of cognition.

• The framework consists of three timescales, which depend on different models of time (cinematic, extensional and retentional).

• We present findings and predictions related to interactions between the different levels of the hierarchy.

Introduction

A recent proposal for a shift in explaining consciousness has been towards looking for its ‘minimal unifying model’ (Metzinger 2020; Wiese 2020). The approach towards finding a minimal model of consciousness is based on developing a model that would (i) specify necessary properties of consciousness, (ii) be descriptive of what determines certain conscious phenomena and (iii) offer@ a way to unify and integrate existing literature in consciousness by identifying common universals. The minimal model approach adds the possibility of ‘developing an idealized model of universal and repeatable features serving to gradually isolate the fundamental, explanatorily relevant, and structurally stable properties that underlie different forms of conscious experience’ (Metzinger 2020, 4). Thus, a candidate for a minimal model of consciousness would possess features of being a part of all consciously experienced content and offer a way for unifying and integrating disparate findings in consciousness research.

One candidate already postulated as a minimal unifying model of consciousness based on the criteria mentioned above is the experience of temporality Windt (2015). As many have previously argued, the feeling of temporality is a fundamental and inescapable property of our conscious experience (Callender 2017). Not only is there an apparent direction to our experiences where experiences seem to succeed each other, but also there is a special time slice between the past and the future of our experiences—the putative ‘now’ (James 1890).

Another reason for the special status of time and its candidacy as a minimal unifying model is that it is the only property that persists in both the content of our experience and the structure that realizes the experience, i.e. not only is our experience of temporality but our experience is itself also temporal (Dainton 2008; Phillips 2010; Chuard 2011). This is not true for any other property of experience; for instance, experiencing the colour red does not entail redness in its representations. However, our experiences not only represent temporal properties (like change, succession, duration etc.) but also have temporal properties themselves. This structure matching property of time makes it an excellent contender for a minimal unifying model of consciousness (for an alternate view, see Dennett and Kinsbourne, 1992).

In this paper, we focus on time to devise a novel framework that can unify various findings from temporal phenomenology and timing of cognition. We consider first previous philosophical conceptualizations of temporality in the section titled philosophy of time consciousness. We then present our framework showing how it can coexist with existing philosophical conceptualizations especially based on phenomenology and describe how this framework could be realized mechanistically. Subsequently, we argue that this framework can act as a bridge for bringing together the timing of cognition (time as an independent variable) and time-consciousness (time as a dependent variable) by presenting evidence from multiple studies that involve interactions between different temporal scales. In addition, the framework allows for the integration of vast theoretical and empirical literature in the study of consciousness. In the section after this, we discuss the philosophical basis of our framework considering the empirical evidence supporting it. Finally, we offer novel testable predictions that can be drawn from this framework in a predictions section.

Philosophy of time-consciousness

Philosophical investigations of temporal consciousness have a rich and long history in both Eastern and Western traditions (Sinha 1934; Whitrow 1980). Here, we will not attempt a historical review but focus on three conceptions of temporality of our conscious experience. The three existing conceptions of temporality can be categorized based on whether they propose temporal extensions in our experiences and their underlying presentations.

Cinematic models assume that neither our experiences nor their representations are extended in time. Cinematic models and their offshoots have given rise to other discrete models of attention, perception and consciousness (Chakravarthi and VanRullen 2012; VanRullen 2013, 2016; Herzog et al. 2016). These have been used to explain periodic sampling of attention (VanRullen 2018), apparent discreteness of perception and our visual experience (Herzog et al. 2020). These models also claim to explain various perceptual illusions where the changes in stimuli are not reliably reflected in our experience of them, for instance, wagon wheel (VanRullen et al. 2006) and silencing illusions (Watzl 2013). However, questions have been raised against the neurophysiological feasibility of computationally expensive discrete models (Fekete et al. 2018), conceptual flaws (White 2018) and their explanation of perceptual illusions (Fekete et al. 2018). A classic conceptual objection to these models has been that they consider the succession of experience to be the same as the experience of succession (James 1890).

Another way to conceptualize temporality would be to grant temporal extension to our experience but not to representations underlying experience (retentional models). Here, a duration-less representation ‘retains’ the traces of temporally extended experience. In the most famous conceptualization of this idea by Edmund Husserl, there is also a forward-looking trace for anticipation (‘protention’), which is also temporally extended but towards the future from the current duration-less moment. Grush (2016) has extended these ideas to explain perception–action coupling and how these inherently dynamic duos are realized over time. Moreover, a major strength of these models has been the ability to explain future-oriented illusions. These examples are in which a current experience is revised to accommodate upcoming stimuli. Examples of these are the cutaneous rabbit illusion and colour-phi phenomenon. Given the recent popularity of Bayesian theories of mind and brain, retentional model-based theories have been co-opted under this umbrella to explain temporality (Hohwy et al. 2016). What remains unclear though is what exactly constitutes retentions and protentions. Is it the case that they are necessarily only perceptual as conceptualized by Husserl (Wiese 2017)? It is not clear how retentions are different from memory and how protentions are different from beliefs or predictions.

Finally, another way to think of temporality is to allow both experiences and their representations to be extended in time (extensional models; naïve view of temporality). This stance has also seen empirical investigations to explain the phenomena of temporality and cognitive cycles (Fingelkurts and Fingelkurts 2006; Madl et al. 2011). A purported neural mechanism for this is the idea of ‘coherence intervals’, i.e. periods of synchrony between theta-gamma bands under which both conscious content and temporality are registered (van Leeuwen 2007). Given that extensional models propose that both representations and content are temporally extended, these theories are natural allies. These models have problems in explaining future-oriented illusions in which the temporal order of experience does not necessarily match the temporal order of stimulus presentation.

Tensions and lack of resolution

Even though these ideas have been around for a better part of 2000 years and have been formally studied for the last 100 years or more, there is little if any resolution in sight. One reason for this is that all these stances are built on findings from different timescales. For instance, the cinematic model necessarily operates in the short timescale range of 30–100 ms, whereas the extensional model postulates conscious moments to be extended on the order of 300–500 ms (Dainton 2010). Retentional models on the other hand allow retentions and protentions to extend up to 3 s or longer (Dainton 2010). It remains unclear whether all of temporality and its illusions could be entirely explained under the umbrella of a singular timescale (White 2018). Similarly, different mechanisms are proposed for sub-second and supra-second timescales even in time perception (Gibbon et al. 1997), making it difficult for a single timescale explanation of temporal consciousness. Moreover, postulating different timescales and working with them as belonging to independent and competing models means that these models best approach phenomena that appear within these ranges, thus limiting the scope of their explanation.

Another reason for the tension between these ideas is the nature of experiential content that they aim to explain. Cinematic models cannot tackle temporal phenomena that are themselves extended in nature, for instance, the experienced succession of two musical notes more than a few hundred milliseconds apart. Nor are they equipped to explain the feeling of persistence in time, say an opera singer holding the same note for an extended period the experience of which is accompanied by that of duration (Kelly 2005). Similarly, they also lack the ability to explain future-oriented illusions where events are separated by more than a few hundred milliseconds. Likewise, extensional models fail to explain phenomena in which succession is experienced within an extended block or simultaneity is experienced outside an extended block. Finally, retentional models are unable to distinguish perceptual vs. belief like protentions and retentions and hence cannot adequately explain temporal dilation/contraction that may be because of beliefs (for a full review, see Dainton 2010). Moreover, existing elaborations of these models of temporal consciousness are unidirectional, explaining either how content in the world is represented temporally in our experience or how we come to act in the world temporally. There are almost no conceptualizations that account for both bottom-up and top-down factors that influence temporal experience under a common framework (see Kon and Miller 2015).

Another limitation is that the models and proposed conceptualizations of time-consciousness while being about time do not necessarily offer explanations about time perception. One reason for this could be that time perception itself does not have a bounded timescale, and durations of estimation/production and reproduction of intervals in studies range from a few milliseconds to several minutes, while the models of temporal consciousness are bounded by the breadth of psychological ‘now’ (Pöppel 2004; Kelly 2005; Atmanspacher et al. 2008; Montemayor and Wittmann 2014). We attempt to unify the current models of time-consciousness and show how this unification could be used to make predictions about time perception, within the limits of a few seconds. In the subsequent section, we propose a framework that not only combines existing ideas on time-consciousness but also attempts to defuse some of these existing tensions between them.

A hierarchical multi-timescale framework for psychological time

It is perhaps uncontroversial to state that various cognitive processes in our minds work at different temporal scales. Given this diversity, it is perhaps unreasonable to rely on a single timescale on which to build models of time-consciousness (Varela 1999). Several contemporary approaches have tried to draw on multiple timescales to explain the timing of cognition and temporality. One prominent example is Pöppel (1972, 1997, 2004) who devised a temporal hierarchy with two levels, one based on the order threshold of perceived stimuli (30 ms) and another level to account for the breadth of a psychological now (3 s). This allowed a common hierarchical explanation of simultaneity judgements (on the order of ∼30 ms) along with continuity and subjective duration judgements (∼3 s). Around the same time, Varela (1999) proposed a dynamical tripartite split of our temporal neurophenomenology, picking apart timescales relating to basic units of neuronal events (∼10–30 ms), large-scale integration in neuron cell assemblies (30–100 ns) and descriptive-narrative timescale (∼3 s). This non-linear dynamical proposal advanced Husserl’s retentional model to link it to cognitive neuroscience and perception-motor research. Similarly, Grush (2016) advanced a retentional model with two different timescales in which a representative module with a breadth of ∼20 ms retains or predicts content over ∼200 ms from the world. Wiese (2017) has further developed Grush’s (2016) model under an explicit distributed hierarchical assumption to explain continuity and persistence in experience. Wiese (2017) proposes distributed predictive processing at sensory and interpretation stages over invariant temporal content to explain the phenomenology of ‘just past’ and ‘forward-looking’ experiences. Another, multi-scale temporal explanation of temporal experience comes from Atmanspacher et al. (2008). He advances Pöppel’s hierarchy by an additional component that operates at ∼300 ms (the approximate time for an object to reach awareness) while retaining Pöppel’s order threshold (∼30 ms) and ‘now’ units (∼3 s) to explain how mental contents unfold over time. Finally, another parsing of the present moment in a hierarchical fashion argues for three levels of the present: narrative (∼30 s), phenomenal present (∼3 s) and functional moments (∼250 ms) (Montemayor and Wittmann 2014).

These existing hierarchical models have furthered the understanding of timing in cognition and phenomenology by disambiguating timescales into multiple different levels based on the nature of experience and cognitive and ecological events. They offer new ways to explain different temporal phenomena like judgements of order, simultaneity, extension of present moments and so on. However, they have not expanded these hierarchies to cover both temporal cognition and time-consciousness. We believe these models could open avenues for a universal conceptualization of temporality of our minds, and an attempt in this direction is made here.

The current framework presented here is not of content building (such as those in visual perception or auditory perception) but of content unfolding in and over time (see Figure 1). In this section, we describe this framework consisting of three levels and interactions between the three levels based explicitly on existing models of time-consciousness. We postulate three hierarchical levels at different timescales that are nested within each other and with different nature of content unfolding at each level. We postulate the hierarchy to be nested, incorporating different temporal regularities in an overall temporal event. Similar nested hierarchies have been proposed for understanding self (Jordan 2003), sense of agency (Kumar and Srinivasan 2014, 2017), mental unity in the experience of selfhood (Fienberg, 2000, 2011) and for phenomenal unity within and between temporal events (Fingelkurts and Fingelkurts 2014). We specify how these levels interact, their updating frequencies and the nature of the content they might come to represent in our current framework.

Figure 1.

Conceptualization of our framework. On the left panel, we combine the three prominent philosophical models of time-consciousness hierarchically. At the first level, we propose a fast-updating cinematic level that updates its contents every 30–50 ms, with a slight delay with respect to the world. Next is the intermediate level, which is proposed after the extensional model unfolding both in and over time every 300–500 ms. This level is also privileged in being tied to our phenomenological experience. Finally, at the top, we have the slow-updating level that spans a breadth of 3–5 s. This level is modelled after the retentional models in time-consciousness. We situate concept and belief representations at this level. This level retains and protends onto the intermediate level. On the right panel, we bring out the nested nature of our hierarchical framework.

Level 1: fast-updating cinematic level

We propose a fast-updating cinematic level in the hierarchy, with its represented content updating every 30–50 ms, the range being similar to order thresholds in existing models (Pöppel 1997, 2004; Atmanspacher et al. 2008; Grush 2016). In considering modality-specific aspects like the temporal resolutions of audition, peripheral vision and multi-modal perception, the updating frequency of this level may have to be modified accordingly. The content and temporality present on this level is not what we are immediately or directly conscious of, so from a phenomenological point of view, this level has neither an extended felt duration nor an extended representational duration. However, we speculate that in terms of objective time, the representations at this level are updated with certain non-zero temporal breadth (30–50 ms). We conceptualize the interaction between the world and this level as ‘cinematic’ because we propose that it has no temporal extension in experience. However, strictly speaking, with the presence of interactions between this level and the next level in the hierarchy, this level is not cinematic per se. Given its representational timescale is somewhat flexible based on constraints imposed by the intermediate level within the hierarchy, perhaps the dynamic snapshot model (Prosser 2016, 2017) better captures our conceptualization of interactions at this level (for an alternate view on flexible timescales, see Herzog et al. 2020).

Level 2: intermediate extensional level

The intermediate extensional level is a structure that operates and updates in the 300–500-ms range. In our framework, the content at this level is that which is experienced and what we are conscious of. Similar to the work of Jackendoff (1987) and Prinz (2007; see also Marchi & Hohwy, 2020), we privilege the intermediate level to best represent the contents of what we experience and report. We propose that this is perhaps due to the intermediate level being extended in time and mirroring the temporality in and of our experience, i.e. it follows the structure-matching thesis of time, wherein the temporal structure of the contents of our experience mirrors the temporal structure of experience itself. The privilege of this level implies temporal mirroring (structure–matching thesis) between the contents of the fast-updating and intermediate levels; thus, these interactions are modelled after the extensional model of time-consciousness. Even though the nature of our framework is of a nested temporal hierarchy, the phenomenological experience of modality-specific content per se is still primarily tied to the intermediate level. The other levels constrain and feed into the intermediate level and are part of the general structure of temporality of the mind over and between moments of phenomenal continuity. This level is connected hierarchically both to the fast-updating level and to the slow-conceptual level in a multiplexed manner (how this happens is discussed in the next section). The intermediate level feeds forward its contents retentionally to the slow-updating conceptual level over time. It also constrains the evolution of the fast-updating level but only with the granularity of its own extension (i.e. over 2–4 Hz). Its own evolution in time is akin to overlapping extension models (see Dainton 2010). The values for temporal extension (300–500 ms) of this level are derived from empirical data investigating time taken for a stimulus to reach conscious awareness (Sergent et al. 2005; see also Atmanspacher et al. 2008) and integration cycles within conscious experiences (Herzog et al. 2016, 2020).

Level 3: Slow-updating retentional level

Finally, the framework is overarched by a slower conceptual level that interacts retentionally with the content at the intermediate level, over a span of 3–5 s. Herein, the content from the intermediate level feeds into the slow conceptual level via retentions over a proposed span of 3–5 s. The slow conceptual level could also constrain the evolution of the intermediate level through forward-looking protentions with the same granularity as the intermediate level (i.e. at ∼300–500 ms). Thus, this level ‘retains’ just past experienced content at the intermediate level (‘primal impression’ in Husserlian terms) over a span of 3–5 s, while forward-looking intentional acts (i.e. protentions) constrain the dynamics of the content at the intermediate level at its timescale. While there has been some dispute as to whether retentions are conceptualized as having perceptual or conceptual content, taking forth Wiese (2017), we see no reason for this distinction in such a framework. The representations at this level are conceptual/belief like (type) pertaining to some perceptual content (tokens). As in the original retentional models, these representations are atemporal, i.e. they have no temporal extension in our phenomenology. Similar also to Prinz (2007), these concepts are not experienced consciously and are atemporal; only their tokens or linked representations at level 2 are experienced as percepts or imagery (Kemmerer 2015). The proposed span for this level comes from previous temporal hierarchies (Atmanspacher et al. 2008) in which the extent of a now or specious present (i.e. the immediately experienced moment) is thought to lie in this range.

Possible mechanisms

This section discusses one possible way this framework might be computationally or neurally implemented, i.e. via multiplexing (see also Piper 2019). We can model the structure of temporality in and of experience in a nested hierarchy with three different rhythms (each representing a different temporal scale) that interact in six possible ways (Jirsa and Müller 2013). Two different rhythms could interact via phase–phase coupling, phase–amplitude coupling, amplitude–amplitude coupling, frequency–phase coupling, amplitude–frequency coupling and frequency–frequency coupling. With these six simple interactions, one can start to model both feedforward and feedback constraints brought about by cross-frequency coupling, which we believe can be co-opted into existing models of consciousness (see Figure 2).

Figure 2.

Possible mechanisms that could drive interactions in our proposed framework. These show the levels of our framework abstracted out as three bands of oscillations of different frequencies (representing temporal mechanisms at these scales) influencing and constraining each other through multiplexing. These interactions could be brought about by amplitude modulation (AM), frequency modulation (FM) and phase locking (PL). See also Jirsa and Müller (2013).

This is only one possible way in which our framework could be mechanistically implemented. We choose this as a starting point to draw out a graded, continuous and neurologically plausible mechanism for our framework. Moreover, with most timing studies done using Electroencephalogram (EEG) and Event-Related Potential (ERP) studies (Fingelkurts and Fingelkurts 2006; Herbst and Landau 2016), multiplexing of oscillations provides an easy accommodation of existing EEG/ERP evidence. However, the proposed framework is not dependent on this specific implementation scheme and is open to different neural implementations. Given that the role of oscillations and their effects on the brain and cognition are not without contention (Sohal 2016), the multiplexing of neural oscillations suggested here is a simplistic exploratory starting viewpoint. Empirical studies and better computational/neural models motivated by the framework would be needed to uncover the specific mechanisms that underlie our temporal experience.

Bringing together time as an independent variable and a dependent variable

Given a multi-scale temporal hierarchy, we discuss the possible ways in which several disparate phenomena and effects in cognitive science and consciousness research could be integrated (see Figure 3). To accomplish this, we first make two key assumptions. One, there are aspects of our conscious experience, which are both in and over time, i.e. not only can we experience time but our experiences themselves dynamically evolve over time. This assumption allows us to draw out temporality as a unifying common denominator between the mechanisms that are responsible for our experience and our experience itself, making it a candidate for a minimal model of consciousness. Put another way, this essentially allows us to bring together ‘time’ as a common factor from the independent and dependent variables of the experiments that have been conducted and continue to be conducted in cognitive science to unify and integrate existing literature towards drawing out a general temporal structure of our experiences.

Figure 3.

A compartmentalization of the interactions in our framework. We list empirical and phenomenological evidence from both timing of cognition and temporal experience research to elucidate the interactions at different levels in our hierarchical framework. For time-consciousness and temporal experiences, we employ prior entry and attentional scope (PE/AS; see the ‘Intermediate-level constraints on the fast-updating level: temporal experience’ section), chronostasis (CS; see the ‘Fast-updating to intermediate level: temporal experience’ section), effect of self-relation to intentional binding (Self-IB; see the ‘Slow-conceptual-level constraints on intermediate level: temporal experience’ section) and continuous flash suppression (CFS; see the ‘Intermediate to slow level: timing’ section). Complementarily from timing research, we draw on attentional blink (AB-1, see the ‘Intermediate-level constraints on the fast-updating level: timing’ section), change blindness and bistable perception (CB/BS; see the ‘Fast-updating to intermediate level: timing’ section), attentional blink again (AB-2, see the ‘Slow-conceptual-level constraints on intermediate level: timing’ section) and magnitude effects on perceived time (ME; see the ‘Intermediate to slow level: temporal experience’ section).

The second assumption that allows us to unify findings under this model is that there is an abstract structure of time, a lawlike dynamic evolution over which our experiences unfold. While there may be various explanations for different psychological phenomena, we use our framework to sketch a common temporal skeletal structure among these phenomena and not necessarily try to explain all aspects of these phenomena. This temporal skeleton can then be used as a scaffolding over and around which explanations may be built, akin to a minimal unifying model of consciousness (Metzinger 2020; Wiese 2020).

There are also limits to what kinds of phenomena this framework can accommodate. One limit is that the timescale on which a prospective finding presents itself must not exceed 3–5 s, which is the upper most timescale considered in the current framework. In the present conceptualization, mental phenomena at longer timescales (10 s or greater) cannot be accommodated by our framework. It is unclear to us whether these longer timescales would necessitate another level in our hierarchical framework or connected duplicates of ‘now’ chunks spanning ∼5 s. Some solutions for this problem are offered elsewhere (for instance, see Kent 2019 and Montemayor and Wittmann 2014). A second limitation is that we consider only those effects that occur at a specific timescale or have an influence on perceived time within the timescales of our framework. In the sections below, we parse our hierarchy into four possible interactions between the three levels and place effects from both timing and temporal experience literature to unify them under a common framework and offer support to our proposed hierarchical framework.

Interactions between fast-updating and intermediate levels

Fast-updating to intermediate level: timing

One previously reported correlate of conscious visual experience is fronto-parietal theta-gamma phase coupling (Buzsaki 2006; Doesburg et al. 2009; Cohen 2011). Several studies have shown this to hold while participants’ percepts alternate while viewing bi-stable images or in binocular rivalry settings (Kruse et al., 1996; Basar Ergolu et al., 1996; Doesburg et al. 2009; Alipour et al. 2016). These studies propose that visual conscious experience unfolds at moments of phase coupling between these two bands and is disrupted during periods of decoupling (van Leeuwen 2007; Droege 2009; Madl et al. 2011). We employ this within our framework at the interaction of the fast-updating level (∼gamma band) and the intermediate level (∼theta band) to explain how visual content unfolds over time in our experience because of phase-coupling in our nested multiplexed hierarchy (see Doesburg et al. 2009). Not only are the timescales consistent with our proposal, but we also employ the same assumptions for necessary conditions of conscious visual experience, i.e. phase coupling between these two levels. Given, we are attempting to link the temporality of this mechanism with perceived temporality, we draw out a prediction on how this phase coupling influences perceived time in the ‘Predictions and future directions’ section.

Another phenomenon associated with visual awareness that fits with this interaction is change blindness brought about by disruptions (gaps, masks and or flickers). An example of this is a ∼300-ms chunk of stimuli that is displayed repeatedly in a loop until a change is detected. Each 300-ms chunk consists of frame 1 presented for 100 ms, a blank interval for 80 ms and frame 2 presented for 100 ms. Frame 2 here has a change, which participants fail to notice immediately. Participants take a few seconds to report the change, presumably from remembering each aspect of the image one by one (Simons and Rensink 2005). We propose that the integration of content over time at the intermediate level is perturbed by the gap/mask that is interleaved between the two frames for ∼80 ms. Herein, the dissimilar content (gap/mask) unfolding at the fast-updating level individuates the two frames as separate events, breaking the succession between the two frames and thus making it harder to recognize the change. There is preliminary evidence from EEG/ERP studies supporting interactions at these timescales of our framework (Koivisto and Revonsuo 2003; Pourtois et al. 2006).

Several temporal illusions could also be placed at the interaction of these levels. These include the phi illusion, waterfall illusion, wagon-wheel illusion, Michotte launching effect and kappa illusions (see Eagleman 2008; Grondin 2010). While they have been explained using a discrete cinematic conceptualization of perception at less than 100-ms timescales (Herzog et al. 2016), more recent arguments against these explanations instead call for discrete perceptions nested within an underlying continuous structure (Kon and Miller 2015; Fekete et al. 2018). Finally, this interaction could also fit with findings on the timing of actions, which have an inherent expectation of temporal contingency between action and outcomes (Hughes et al. 2013). Acting on the environment as agents ourselves, we are entrained to expect outcomes with a relatively fixed delay. Alterations in these delays can make us feel like we did not cause the outcome, even when we did (Eagleman 2008).

Fast-updating to intermediate level: temporal experience

Concurrently, we view the effects of perceived time that are complementary to the timing effects along the same timescale and are accounted for through this interaction. One of these are the effects of actions slowing down perceived time (chronostasis). An action can appear to delay the onset of a stimulus just after it has been performed, for instance, switching ears while listening to a phone receiver makes it seem like the periodic trilling of the line takes longer or moving one’s eyes to an analog clock makes it seem like the second-hand took a little longer to tick (Grondin 2010). These effects are apparent in our experience and occur generally at sub-second timescales, with a subjective compression or dilation of time equivalent to half-width of the intermediate level (∼120–150 ms); thus, we place them at the intermediate level of our framework. Note that while the content of our visual experience may appear to update cinematically (delayed snapshots) just after an action in the case of chronostasis, the delay is only apparent because of a continuous extended frame of our experiences. If such effects were only because of a frame-like mechanism for consciousness, the content of our experience would still be delayed (with respect to objective time) but not apparent in our experience. This is because there would be no continuous reference with respect to which we would experience delay; thus, there is still an aspect of our experience that is extended (here the intermediate level), which captures this delay. Such an interaction can only be explained with an underlying extended structure in which frame-like updating of content comes about (see also Fekete et al. 2018).

Like for the bi-stable percepts (see the ‘Fast-updating to intermediate level: timing’ section), a reset of the phase between these levels at the onset of an action temporarily delays the content being multiplexed with the intermediate level, until they are phase locked again (Doesburg et al. 2009). We propose this phase-lag reset as being responsible for the apparent delay perceived in period stimuli after an action is performed. However, the overall conscious experience at the intermediate level updates continuously nevertheless with only the visual content lagging, bringing about the feeling of delay for the ticking of the clock in the experience overall.

Effects that alter felt time and are proposed to lie at this interaction also include bigger and faster stimuli prolonging perceived time, when these are displayed for sub-second durations and dilate felt time by 100–150 ms. These effects could be accounted for at the interaction of these levels by amplitude modulation for larger stimuli where larger stimuli lead to increased neuronal activity in early visual processing and by frequency modulation for faster stimuli that lead to increased firing rates in early visual cortex. Similar mechanisms acting this interaction allow integrating effects of compressed felt time in intentional binding by frequency modulation at the scale of ∼300 ms (Vastano et al. 2020).

Intermediate-level constraints on the fast-updating level: timing

The proposed framework allows phenomena at the intermediate level to constrain the content at the lower fast-updating level. One example of this is based on the priority in visual experience for one property over another. Consider the ‘silencing illusion’ (Suchow and Alvarez 2011). The illusion uses a stimulus set, which has a doughnut-shaped ring made up of several multi-coloured small circles. Here, changes in the colours of small individual circles are visible if the larger global ring they make up to does not move. As soon as the global ring starts rotating, the colour changes within the individual circles are no longer easily seen. Priority for unfolding whole/global content here takes priority over local changes, altering the way in which temporal relations are integrated and represented. Some have cited this as evidence against the ‘structure matching thesis’ of temporal representation (Watzl 2013). However, we argue that the prioritized property of the visual stimuli (global circle rotation) still follows temporal-relation matching, i.e. global motion is faithfully represented in time, but not local colour changes. This is true especially when the global changes (i.e. the speed of rotation of the ring in this case) are limited to less than 1 rotation per 300–500 ms. Rotation speeds slower than this would not lead to a silencing illusion. In the ‘Predictions and future directions’ section, we draw out experimental ideas based on these claims.

Another way the intermediate level constrains the unfolding of content at the fastest-updating level is seen in attentional blink, when the stimuli in a rapid serial visual presentation (RSVP) are pieces of a shape that make a complete figure (Akyürek et al. 2012). Herein, the RSVP has frames of stimuli presented at a fast timescale (∼50 ms). Participants often report seeing an integrated T1 and T2 if they meaningfully join to form a complete image. However, order information is often lost (Akyürek et al., 2016). The intermediate level extends over to perceptually complete or integrate a figure, at the cost of temporal order. Similar results from visual integration experiments from these timescales (Ronconi et al. 2017; Ronconi and Melcher 2017) could also be accounted by the proposed framework.

Intermediate-level constraints on the fast-updating level: temporal experience

Similarly, intermediate level may constrain the temporality of the content unfolding at the fast-updating level. An example of this comes from temporal attention sampling information at 4 Hz (see Nobre and Van Ede 2018), bringing about prior entry of attended stimuli in simultaneity judgement and Temporal Order Judgment (TOJ) tasks (Shore and Spence 2005; Spence and Parise 2010). Here, temporal attention sampling occurring at the intermediate level could lead to prior entry of content at the fast-updating level by increasing the amplitude of the representations of the contents (Vibbel et al., 2007) or by improving temporal precision through frequency modulation (Yeshurun and Levy 2003). Similarly, scope of attention at the intermediate experiential level may also influence temporal resolution of the cinematic level, through a top-down-driven frequency modulation (see predictions in the ‘Predictions and future directions’ section).

Interactions of the intermediate and slow levels

Intermediate to slow level: timing

An existing paradigm that is best set up to explore the interactions between intermediate and slow levels is continuous flash suppression (CFS). In this paradigm, a stimulus is presented to one eye and a flickering noise mask to another. Initially, participants see an intermixed percept dominated by the mask and slowly (∼3 s) the stimulus breaks into awareness. A key factor that drives these breakthrough times is the frequency of the flickering noise mask. Previous studies have reported this to be around ∼6 Hz when the stimulus must be identified (Zhu et al. 2016; Drewes et al. 2018; Han et al. 2018). In our framework, these results act as a ‘temporal lesion’ bringing to light the timescale at which objects in our experience reach awareness and are identified. Here, the flicker perturbs the interaction between the intermediate level and its slow-updating content representations at the conceptual level. The flicker here at the half-width of the intermediate level ensures that this phase locking continues over one cycle of the slow conceptual level, hence obscuring the process of identification of the stimulus for around 2–3 s. Only after one cycle of the slow retentional level is completed (∼3 s) does the stimulus break into awareness.

Similar interactions of visual experiences to object recognition and meaning can also be found in attention research (Meijs et al. 2018). For instance, this happens through ‘perceptual episodes’, which happen when temporal attention operates in selecting stimuli that are presented in an RSVP or in a multiple set (see Snir and Yeshurun 2017).

Intermediate to slow level: temporal experience

How would the content of our experience lead to estimates of longer durations? We account for some of these effects (Eagleman 2008; Grondin 2010) as arising out of the interaction between the intermediate and slow-conceptual levels. Herein, magnitude representation of our content may inflate estimates of time for supra-second timescales, without changing the temporal resolution of our experiences themselves. Several such effects exist in the magnitude-representation literature of time perception (Walsh 2003). One such example is the effect of ‘bigger equals longer’. Here, stimulus properties of numerosity tap into a common magnitude representation of numerical quantities (space, time, number, etc.) and are reported as longer. These can be best accounted as biasing our judgements of time based on magnitude representations of what we see. For instance, a circle with number 9 written on it may be reported as lasting longer than a circle with number 1 written on it (but they are not likely to have different flicker frequency thresholds, given here the interaction excludes the fast-updating level).

Similar accounts can also be drawn from the effects of emotion on time perception literature. Accounts of fear dilating judged time (Eagleman 2008) is one example of this. One way this can be brought about is how we parse events into meaningful units. Emotional events receive greater attention and see more fine-grained narrative parsing of events. This could be brought in through both frequency modulation of the slow conceptual level, which alters the extent of event boundaries, and amplitude modulation, which alters the fuzziness of the event boundary representations. Herein again, emotional events would lead to judgements of longer passage of time without necessarily slowing down perceptual frames or altering flicker frequency thresholds (Eagleman 2008).

Slow-conceptual-level constraints on intermediate level: timing

In an earlier section (‘Intermediate-level constraints on the fast-updating level: timing’ section), we discussed an Attentional Blink (AB) effect to demonstrate the constraints imposed by the intermediate level on the fast-updating level. Here, we use another AB effect to demonstrate the constraints that the slow-conceptual level places on the content unfolding in our experiences at the intermediate level. This AB effect is of the kind where T2 is experienced but not reported (see Block 1995; Vogel et al. 1998). Consider again the ‘perceptual episodes’ over which selective attention operates (Snir and Yeshurun 2017). The concepts employed at the slow-updating level can then constrain what is selected (here read retention) in an RSVP unfolding at the intermediate level. An example of this is the increased number of blink trials when T1 and T2 are faces belonging to the same emotion category although being individually different (Ray et al. 2020). Here, the conceptual task-relevant category of emotion retains the emotional category of T1 leading to participants missing the T2 when they belong to the same emotion category and appear in successive lags in an RSVP task by forming a single perceptual episode. If the slow-updating level constrains the content at the intermediate level, it should also be able to facilitate recognition of T2 when T2 completes a perceptual episode. Supporting evidence for this can be found in Meijs et al. (2018), where the predictive contingency of T2 on T1 reduces the number of trials where an attentional blink occurs (see also Alilovic, et al. 2020). These form examples of the slow-updating level constraining what is recognized (T2; ∼300 ms) over an RSVP (1–2 s).

Slow-conceptual-level constraints on intermediate level: temporal experience

The previous section showed how the slow-conceptual level constrains what is ‘picked out’ from an RSVP stream. This section explores a similar possible constraint from the slow-conceptual level to the temporality of experiences at the intermediate level. For instance, IB is modulated by action–outcome contingency (Moore et al. 2009) and conceptual relations to self (Makwana and Srinivasan 2019); an initial speculation is that the slow retentional level downwardly constraints the temporal evolution of the contents at the intermediate level. Thus, contents that match the predictability of the action and or are conceptually related to oneself appear in our awareness faster, bringing about intentional binding.

Revisiting philosophical conceptualizations of our framework in the light of empirical evidence

Overall, our attempt here has been to draw out certain properties of time-consciousness and possibly link them with findings from the timing of cognition under a common framework. These properties include succession, change, temporal order, persistence, and duration perception and estimation. However, our current proposal leaves out two key properties from time-consciousness, namely flow and saddle-back nature of the ‘now’, i.e. the specious present housing both just-past events and forward-looking events (James 1890). We do not propose explicit mechanisms to explain these phenomenological facets of temporal experience. Our speculation is that the nature of interactions in our nested framework gives rise to these phenomena as an emergent property, i.e. the multiplexing of content simultaneously at various nested levels is enough to bring about the feeling of flow and an extended present.

Another open question in the time-consciousness literature has been about whether our experiences are extended in time or not, put another way whether they are discrete or continuous. Similar extensions of this debate follow in visual awareness research, where there are opposing views on whether awareness is graded or all-or-none (Windey & Cleermans, 2015). Our framework aims to neutralize this debate with a common ground between discrete or continuous positions, given our framework proposes that certain visual phenomena can appear to update discretely in our experience (interactions from fast-updating level to intermediate level; see the ‘Fast-updating to intermediate level: temporal experience’ section) although overall the vehicle (propensity) for content is continuously flowing at the intermediate level. We do employ oscillations and interactions between different oscillations as a putative mechanism for our framework, which would imply that we take the graded stance for visual awareness. However, from the perspective of the general framework we propose, there is nothing in principle that stops from reworking our framework based on an all-or-none stance.

Where does our framework fit in with existing philosophical literature in time-consciousness? One of the aims of the framework was to resolve the existing debates between philosophical conceptions in time-consciousness, by allowing them to coexist (for another similar attempt, see Dorato and Wittmann 2020). We extend these philosophical conceptions to empirical work in both timing and temporal experience. Unlike previous multi-scale models (Pöppel 1997, 2004; Atmanspacher et al. 2008; Montemayor and Wittmann 2014; Kon and Miller 2015; Grush 2016; Wiese 2017), not only do we construct a bidirectional putative scaffolding over which our experiences unfold, but we also show how empirical predictions can be drawn from the same framework for a myriad of sub-fields in consciousness and cognitive science research. The key being two aspects of these predictions: (i) the timescales at which these effects may unfold and (ii) the phenomenological nature of these effects. Remember that not only do we think that the temporal structure of our experiences consists of multiple timescales, but we also privilege the intermediate level to represent first-person subjective experience of modal content.

The empirical evidence we present to elucidate the nature of interactions between the different levels of our framework can also be employed to explain the rationale for our choices of philosophical models for each interaction. As we discussed in ‘Fast-updating to intermediate level: temporal experience’ and ‘Intermediate-level constraints on the fast-updating level: timing’ sections, the design of our framework allows us to incorporate and reconcile claims from certain temporal illusions arguing that our temporal consciousness follows a frame-like snapshot structure. The interactions we discuss in these sections show the duality of our experience in being both frame-like and continuous. If the nature of content updating in our phenomenology was only frame-like, the delay in content-updating would not be self-evident to us. On the other hand, if the nature was only of temporal extension, there would be no delay in the first place. It is this interaction between cinematic and extensional structures that brings about our experience in our nested framework, where the fast-updating level unfolds and contained within the intermediate level.

Along the same lines, the slow-conceptual level interacts with the intermediate level similar to that of a retentional model. The just-past retention of content updating at the intermediate level is picked up by the slow conceptual level as our experiences unfold. It should be noted that the representations themselves at this level are atemporal, complementing both the intermediate-level hypothesis (Prinz 2007) and the conceptualization in retentional models (Wiese 2017). Put together, this postulation explains the empirical evidence we discuss in ‘Intermediate to slow level: timing’ and ‘Slow-conceptual-level constraints on intermediate level: timing’ sections. In the former, we provide evidence for the span of the slow level being over a few seconds in a CFS paradigm, where the breakthrough times are indicative of a temporal event. In the latter, we discuss how protentions from the slow level could modulate whether targets in an RSVP are reported or not.

Finally, the overall nested hierarchical structure of our framework and its philosophical conceptualizations are perhaps best captured by the change blindness and CFS phenomena (see ‘Fast-updating to intermediate level: timing’ and ‘Intermediate to slow level: timing’ sections). By ‘change blindness’, here, we refer to paradigms in which participants report missing changes in and around brief disruptions (gaps, masks and flickers) that take place within the timescales discussed in our framework. In such paradigms, it is the fast-updating- and intermediate-level interactions that prevent the succession of the two scene frames with a gap/mask interleaved within ∼300-ms chunks. The intermediate level evolving horizontally does not overlap with the ‘gap’, breaking the succession between the frames with the scene leaving the slow-conceptual level’s retentions with no information about the change in every alternation of the frame-gap-frame loop. The participant in such a design is left to piece together aspects of both the frames slowly over each loop, taking around ∼3 s to pick out the change. CFS is another paradigm that brings out all three levels of our framework. With all three levels of the hierarchy phase locked initially to the more prominent flickering Mondrian, it takes the duration of an entire temporal event (∼2–3 s) before a participant can identify the stimulus.

Predictions and future directions

Our framework can offer robust and immediately empirically testable predictions and further the research of timing of cognition and time-consciousness. We assume that temporality is fundamental to conscious experience. This leads to a prediction that in the absence of consciously experienced content, there should be no feeling of time. One way to test this claim is via bi-stable images. It is speculated that conscious experience is correlated with the fronto-parietal theta-gamma phase locking (Buzsaki 2006; Doesburg et al. 2009; Cohen 2011). This coupling is absent for brief periods of time during the period of perceptual switches (van Leeuwen 2007; Madl et al. 2011), and there is possibly no conscious visual experience during a perceptual switch. If perceived time is a function of consciously experienced content, then one would predict a duration in which there is a perceptual switch should be experienced as shorter than the same duration without a perceptual switch.

To test this assumption, we asked participants to view a Necker cube and report when they experience a perceptual switch and also make duration (Singhal and Srinivasan, in preparation). In separate experiments, these switches were either exogenously induced through a rod-like object moving through the Necker cube or endogenously produced under free viewing. We consistently find that participants report the same intervals containing perceptual switches as shorter than those that do not contain perceptual switches. To show phenomenologically that this is because of a lack of consciously experienced content around the moment of a perceptual switch, we construct a simple video where viewers can demonstrate for themselves missed visual content at the moments of perceptual switches. The demo presents a number-sequence that loops (for eg. 1–2–3–4) at the centre of a Necker cube, here, in our perception a number appears to go missing at the time of a perceptual switch (Singhal and Srinivasan, 2021). These results from our experiments and the phenomenological demo offer support to our assumption linking consciously experienced content and felt time. Moreover, the results add to the existing evidence of the complementarity between time-consciousness, timing of cognition and time perception.

Another fundamental tenet of our framework is that there are multiple temporal scales at which mental content unfolds, and each of these levels has a different mental content. While the idea of using multiple timescales to explain time-consciousness is not new (Pöppel 1997; Atmanspacher et al. 2008; Montemayor and Wittmann 2014), we hypothesize an empirical test for this claim. This central assumption can be tested by what we call using ‘CFS as TMS’. If there are multiple levels unfolding on different timescales with differential content properties, they should be prone to selective disruption, i.e. different frequencies of noise masks in CFS should affect different properties of stimuli during their breakthrough or identification. There is some preliminary evidence to support this (Zhu et al. 2016; Drewes et al. 2018; Han et al. 2018), but a study targeting these claims specifically could be employed to test the validity of our framework.

There are several kinds of stimuli that distort our perception of time. A central issue about such distortions is whether these effects are perceptual (how things appear phenomenologically) or these effects are due to memory or judgement biases. The proposed model weighs on this debate by classifying these distortions as occurring through interactions between the fast and intermediate levels or between the intermediate and conceptual levels. The privilege we grant to the intermediate level of being tied to phenomenological experience allows for this distinction. The interactions of the fast-updating level and the intermediate level are proposed to be extensional and to follow the structure matching thesis. The alterations in content representations at this level bringing about changes in perceived time would then be part of subjective experience and could be investigated by changes in TOJ sensitivities and flicker fusion frequency thresholds. In the case of interactions between the intermediate and slow-conceptual levels, these effects would not translate to differences in TOJ slope differences or differences in flicker frequency. To elaborate on this, consider the example of the findings from Eagleman (2008). Here, to test whether frightened emotional states slow down time in perception, they made people jump off a height while they watched a flickering display. If time really did slow down in perception, they would be able to see the overlapping stimuli in the flickering display (25–30 Hz). While the participants did judge their (∼3 s long) fall as lasting for longer than it did, they did not see any perceptual difference in the flickering display. The study concluded that the reported dilation of felt time was most likely brought about by memory and judgement biases and not perceptual slowing down of time.

In our framework, similar effects that can be placed at the interaction between intermediate and conceptual levels are odd-balls, self-related stimuli dilating felt time and conceptual knowledge of stimulus altering its judged duration (Grondin 2010). Consider odd-ball effects (Pariyadath and Eagleman 2007) where the odd-ball stimulus is an odd-ball by virtue of being categorically different from the chain of stimuli previously present over a few seconds. Here then, going by our previous predictions for placing this effect at the interaction of the intermediate and slow-conceptual level, one would not expect odd-balls to show differences in temporal acuity. There is some preliminary support for this prediction (Wehrmann 2020). Same can be said about the effect of self-related stimuli on time (Makwana and Srinivasan 2019), semantic information about speed, and size affecting judged time (Mioni et al. 2015) and so on. These effects could be classified as being brought about by judgement biases, if it can be shown that these same stimuli do not show differences in temporal sensitivities (say in a TOJ or simultaneity judgement task) and/or do not show differences in critical flicker fusion thresholds. For instance, a shape associated with the self may be judged to last longer, but it may not appear to flicker at different rates compared to a neutral shape. Although stimuli that are predictable, of importance to self and or receive temporal attention may show prior entry or priority effects in TOJ tasks through threshold shifts without changes in temporal discrimination.

On the other hand, distortions of felt time that are driven by the interactions between the fast and intermediate levels would hypothetically show differences in temporal resolution when tested through TOJs and flicker frequency threshold paradigms. A set of these illusions could possibly be the result of the disparity in temporal resolutions between the magnocellular and parvocellular pathways (see Piper 2019). Even though we do not make this distinction in our framework yet, it is something that could be incorporated in the future to better model these effects. Examples of these that would probably qualify are those that show bigger stimuli last for longer (Rammsayer and Verner 2015), speed of stimuli altering judged time (Kaneko and Murukami 2009), differences in felt time based on attentional scope (Lawrence et al. 2020) and differences in dynamics of figure-ground segregation, to name a few. Based on these findings, using our framework, we can predict differences in temporal resolution (TOJ slopes, CFT) for bigger stimuli compared to smaller ones, wide scope of attention compared to a narrow scope (for preliminary evidence, see Mudumba and Srinivasan, 2021) and differences in temporal sensitivity for objects appearing in figure vs. ground.

Not only do these differences offer support to both the privilege of the intermediate level being tied to phenomenological experience and being extended in time, but it also allows our framework to propose a putative test for phenomenological differences (between judgements and perception) in distortions of felt time.

Continuing with the distinction between perception and judgement, there are similar debates about failure to perceive vs. failure to report that persist in consciousness literature. A question along these lines can be asked of attentional blink; is it the case that participants do not perceive T2 (Sergent et al. 2005) or is the case that they fail to report T2 (Vogel et al. 1998)? There has been very little phenomenological analysis to answer this question. Within our framework, we predict that both could occur, depending on the nature of stimuli and the speed at which they are presented in an RSVP. We postulate two kinds of attentional blinks, one that occurs at the intersection of the fast-updating level and the intermediate level (AB Type 1; see the ‘Intermediate-level constraints on the fast-updating level: timing’ section) and another that occurs at the intersection of intermediate and conceptual level. The latter is what we believe is the traditional AB effect (see the ‘Slow-conceptual-level constraints on intermediate level: timing’ section), i.e. a failure to report a stimulus rather than perceive it (we call this AB Type 2 here). Thus, we postulate two kinds of AB in our framework, one occurring at a sort of temporal bottleneck for perceptual features, while the other occurring at a temporal bottleneck for concepts. There is support for such temporal bottlenecks in extensions of global workspace theory (for instance, see Raffone et al. 2014).

In addition to making predictions based on interactions between different levels in the temporal hierarchy, the proposed framework also provides a base for other theories of consciousness. The proposal that the intermediate level is extensional could provide a rationale not only for intermediate-level theories of consciousness (Jackendoff 1987; Prinz 2007) but also for global workspace theory of consciousness (Baars 2005). Global workspace theories of consciousness argue that the contents of the workspace are conscious but that does not allow a direct explanation of other aspects of consciousness like temporal extension and continuity. If the contents at the intermediate level constitute the contents of the global workspace, the extensional nature of the intermediate scale would provide the foundation of temporal properties of consciousness, not just for the contents of consciousness.

Conclusion

The current paper proposes a multi-scale temporal hierarchy framework to explain findings of time-consciousness and timing in cognition. We discuss different ways in which temporality is incorporated and modelled at different levels (timescales) providing a pathway to integrate divergent models of time-consciousness. In addition to providing empirical evidence for the proposed hierarchy and interactions between different levels in the hierarchy, we also provide a set of empirical predictions that could take research on time-consciousness and timing. This paper allows us to extend the candidature of ‘time’ as a minimal unifying model for consciousness. The proposed hierarchical framework would complement content-based or state-based theories of consciousness. Further work would be needed to flesh out the relationship between the proposed framework with content-based theories of consciousness.

Data Availability

Not applicable, theoretical paper.

Funding

Mr. Ishan Singhal was supported by a Prime Minister’s Research Fellowship (Government of India) grant.

Conflict of interest statement

None declared.