Neural Encoding and Representation of Time for Sensorimotor Control and Learning

Abstract

The ability to perceive and produce movements in the real world with precise timing is critical for survival in animals, including humans. However, research on sensorimotor timing has rarely considered the tight interrelation between perception, action, and cognition. In this review, we present new evidence from behavioral, computational, and neural studies in humans and nonhuman primates, suggesting a pivotal link between sensorimotor control and temporal processing, as well as describing new theoretical frameworks regarding timing in perception and action. We first discuss the link between movement coordination and interval-based timing by addressing how motor training develops accurate spatiotemporal patterns in behavior and influences the perception of temporal intervals. We then discuss how motor expertise results from establishing task-relevant neural manifolds in sensorimotor cortical areas and how the geometry and dynamics of these manifolds help reduce timing variability. We also highlight how neural dynamics in sensorimotor areas are involved in beat-based timing. These lines of research aim to extend our understanding of how timing arises from and contributes to perceptual-motor behaviors in complex environments to seamlessly interact with other cognitive processes.

Introduction

In real-life behaviors, motor circuits operate smoothly in concert with perceptual and cognitive systems (Song, 2017; Cisek, 2019). Timing plays a central role in all these domains: we coordinate and adapt movements to the environment with strikingly accurate timing; we perceive and predict dynamic events; we communicate by generating and interpreting temporally patterned sounds and speech. The coordination of perceptual, motor, and cognitive processes is a critical part of many adaptive behaviors; time is the “glue” that integrates our perceptions, actions, experiences, memories, and emotions.

A traditional focus in the study of timing has been how we measure time intervals (Ivry and Schlerf, 2008). With this emphasis, a plethora of studies on the mechanisms of timing has operated under the assumption that higher-order brain areas control the timing of our actions and that the sensorimotor system produces an output in response to these timed brain commands. For example, a long-standing view is that a putative internal “clock” in the CNS times sensorimotor behavior (Buhusi and Meck, 2005; Meck et al., 2008; Harrington et al., 2010). This viewpoint sidesteps any influence and contributions from the sensorimotor system's dynamic properties to timing, as it assumes that the temporal parameters of a movement stem solely from centralized representations. However, recent research has shown that the sensorimotor system plays a more critical role and may also be fundamentally necessary for the perception of time (Wiener et al., 2019). From a phylogenetic perspective, it is clear that organisms could move long before they could think (Mendoza and Merchant, 2014; Cisek, 2019). Hence, assuming that timing was important in motor systems, even before cognitive functions developed, it seems likely that the intrinsic timing ability of the motor system has shaped the development of timing in cognitive systems.

In this review, we address how the timing of natural movements can affect the perception of time. We present behavioral, computational, and neural evidence for the link between the mechanisms of sensorimotor control and those underlying two primary aspects of time perception: interval-based (absolute) timing and beat-based (relative) timing (Grube et al., 2010; Iversen and Balasubramaniam, 2016; Ross et al., 2016). Interval-based timing refers to the ability to estimate, measure, and discriminate between isolated time intervals. In contrast, beat-based timing refers to the ability to measure the duration of time intervals in the context of temporal regularities, such as rhythmic beats (Teki et al., 2011; Merchant et al., 2015a). We discuss how the sensorimotor system relates to interval- and beat-based timing with new perspectives challenging the theoretical framework of a centralized clocking mechanism. This collection of studies shows how temporal processing contributes to sensorimotor behaviors in complex environments by coordinating perceptual, motor, and cognitive processes. Based on these studies, we argue that a dynamic systems perspective is the appropriate framework to understand the interrelation of perception, action, and cognition for sensorimotor timing (Fig. 1).

Figure 1.

A dynamic systems view on brain and behavior in the context of perception, action, and cognition. This perspective challenges the theoretical framework of a centralized clocking mechanism by showing how temporal processing in perception and sensorimotor actions is achieved by coordinating perceptual, motor, and cognitive processes.

Influence of dynamic motor primitives on the perception of time

Over the past decades, research in motor neuroscience has converged to recognize that control of our high-dimensional motor system is unlikely to be an exclusively top-down process. As already envisaged by Sherrington (1906), coordination is hierarchical and relies on multiple interacting modules within the complex neuromechanical system (Bernshteĭn, 1967; Turvey, 1990). Extending from Sherrington's vision, a present-day computational proposal is that these modules are lower-dimensional subsystems that can be viewed as dynamic systems or primitives of the movement (Schaal et al., 2000; Sternad et al., 2000; Hogan and Sternad, 2012). These dynamic primitives unfold over time and, with practice, can attain a stable spatiotemporal structure. For example, when walking, the multisegment movement apparatus establishes a stable pattern, and each person exhibits only a relatively narrow range of preferred periods. More complex acquired skills, such as a tennis serve, also develop into a stable spatiotemporal pattern with practice. Further, when performing a sequence of similar actions, these individual actions merge into a repetitive or rhythmic pattern with dynamic stability. Dynamic stability implies that an action is robust to perturbations that can obviate explicit corrections. Such a stable rhythmic pattern has been shown in the example of repeated throwing actions, where a preferred interthrow period evolved that exhibited dynamic stability (Zhang and Sternad, 2019). Figure 2A–C overviews the virtual throwing task with exemplary trials illustrating the evolving periodicity and stability. Can dynamic primitives developed in the context of motor actions also influence perceptual timing?

Figure 2.

Simplified virtual throwing task. A, In the virtual task, the participant performs forearm movements via a manipulandum and throws a virtual ball to hit a target on the screen. The error is defined and calculated as the shortest distance that the ball trajectory achieves to the target. The time at which this closest distance occurs can differ between trials. B, The task has redundancy as infinitely many different ball releases (angle and velocity at ball release) can achieve a given error. The combinations of angle and velocity that achieve zero error define the solution manifold (green band). Orange line indicates an exemplary arm trajectory plotted in the same space. It closely aligns with the solution manifold where a “timing window” can be defined. Any ball release within this window achieves a zero-error target hit (Zhang et al., 2018). C, Continuous arm movements plotted in phase space spanned by position and velocity, display a closed orbit, indicating periodicity. With practice, successive throws develop a stable periodic pattern (from day 1 to day 4). Black dots indicate the ball releases. The variability of these trajectories significantly decreases from day 1 to day 4. Red line indicates a Poincare section, where the intersections of the arm trajectory are analyzed to test for stability (Zhang and Sternad, 2019). D, Velocity profile of two successive throwing movements illustrates different temporal intervals defined by kinematic landmarks. Red dot indicates the ball release time. The interval between the start of the movement to the ball release (∼300 ms) is most critical and positively affects interval perception.

When throwing a ball to hit a target, a high degree of skill and accurate timing of the ball release appears crucial. Several studies examined the degree of temporal precision required for throwing a ball to hit a target and proposed timing precision to be in the order of 1-2 ms (Chowdhary and Challis, 1999; Timmann et al., 1999; Smeets et al., 2002). Not only do these estimates challenge the known temporal precision of neural firing, which is more in the order of 10 ms, but closer examination of throwing also showed that successful timing of ball release can be achieved in many different ways: an important aspect of target-oriented throwing is that an infinitude of combinations of position and velocity at ball release can attain zero error and thereby circumvent the demands on temporal precision (Müller and Sternad, 2004). Indeed, the subspace of all successful ball releases can be characterized in terms of a solution manifold (i.e., a lower-dimensional set in the high-dimensional space of all ball releases that achieve zero error) (Fig. 2B). Such a manifold can significantly simplify the timing problem as movement trajectories can exploit such manifolds and create tolerance, or timing windows within which the ball may be released and hit the target (Fig. 2B).

A recent study by Zhang et al. (2018) demonstrated that, with practice, humans develop hand trajectories that align with this manifold such that they create a temporal window where each ball release generates a ball trajectory that hits the target with zero error. Using a virtual environment as a testbed, participants practiced throwing a ball to hit a target for several days. As expected, humans improved their hitting accuracy, but not only through reducing variability in the timing of ball release. Participants also modified their hand trajectory to create “windows” for ball release timing by aligning the hand trajectory with the solution manifold (Fig. 2B). With practice, participants shaped their hand trajectories such that they contained segments up to 20–40 ms long within which a ball release would hit the target. Therefore, the exquisite skill that humans have acquired over evolution relies not only on improving the timing of ball release but also on mitigating demands on timing by developing “smart” movement strategies that are less sensitive to temporal variability arising from neuromotor noise (Sternad, 2018).

Given the intimate relationship between timing and dynamics, one intriguing question is whether such stable patterns or dynamic primitives developed in the context of motor actions can also impact the perception of temporal intervals. Song (2017) and Sternad (2018) addressed this question using the same virtual throwing task. A preliminary study by Guo et al. (2019) asked whether practicing a sequence of throwing movements with no explicit periodicity could enhance the sensitivity of auditory interval discrimination. If so, is this enhanced sensitivity selectively linked to the timing of the trained movement? Participants practiced throwing a ball to hit a target over several daily sessions, during which they stabilized a close-to-periodic spatiotemporal pattern of the hand trajectory, as shown in Figure 2C, D. Following each throwing session, participants performed an auditory time-interval discrimination task. The results showed that, with throwing practice, time discrimination was selectively enhanced for the interval that was close to the release time, defined between the start of the movement to the ball release. This was the most salient interval for an accurate throw (Fig. 2D). Other intervals of the movement profile did not affect interval perception. Notably, the amount of stabilization of the spatiotemporal profile predicted the enhancement of time-interval discrimination within each individual. In contrast, the control group, who did not practice throwing, did not show any improvement of time discrimination.

Together, these results demonstrate that humans can overcome the ever-present noise in motor timing through developing dynamically stable performance strategies that create timing-insensitive solutions. These learned spatiotemporal patterns, dynamic primitives, then interact with perceptual timing. In the next section, we discuss how the brain deals with the variability and noise inherent to timing mechanisms and achieve these behavioral results.

Reducing timing variability using neural manifolds in the sensorimotor cortex

Timing mechanisms in the brain are inherently unreliable and cause variability in all temporal aspects of behavior. As noted in the previous section, behavioral studies suggest that practice reduces variability in part through exploiting manifolds that support a multitude of solutions within the space of behaviorally relevant variables. However, the mechanism by which the nervous system implements this computational strategy is not understood. Specifically, we know very little about how the sensorimotor cortex establishes neural manifolds and how those manifolds reduce timing variability.

Jazayeri and colleagues addressed this question using a neurophysiology experiment in rhesus macaque monkeys (Sohn et al., 2019). Monkeys, like humans, have an unreliable sense of time, and their timing behavior exhibits a great deal of variability. To reduce this variability, humans and animals exploit temporal regularities in the environment. Sohn et al. (2019) trained monkeys to perform a time-interval-reproduction task, known as the Ready-Set-Go task (Fig. 3A, top). In Ready-Set-Go, animals have to estimate a sample interval between the first two beats of an isochronous rhythm (Ready and Set) and initiate a movement at the expected time of the omitted third beat (Go). Across trials, the sample intervals were drawn from a fixed probability distribution. With practice, humans and monkeys learn that certain intervals have a higher probability and bias their responses in that direction. This strategy, which is known as Bayesian integration, reduces variability and improves performance.

Figure 3.

Representation of time during time interval reproduction and rhythmic timing. A, Top, Time interval production task. Monkeys were required to estimate a sample interval demarcated by Ready and Set, and reproduce that interval by a delayed motor response (Go). Sample intervals were drawn from one of two prior distributions: Short or Long. Bottom, A schematic showing the curved neural trajectory during the Ready-Set epoch for the Short prior condition. Linear readout of time intervals from the curved neural trajectory (left) generates biased internal estimates of the sample interval (middle) and reduces variability near the extrema of the prior distribution (right). B, Top, Synchronization task. Monkeys were required to tap (circles) synchronously three intervals (SO1-SO3) to an external metronome (arrows). The interstimulus interval was either Short or Long. Bottom, Neural trajectories during the synchronization task. The trajectory starts from a tapping manifold (black line), completes a cycle during every intertap interval, and returns to the tapping manifold. The tapping manifold is invariant across durations and serial order elements of the task. The metronome's tempo modulates the amplitude of the trajectories and the serial order element as the third axes in the state population.

To understand the neural basis of this strategy, Sohn et al. (2019) recorded neural activity in the dorsal region of the medial frontal cortex that is thought to play a central role in higher-level control of sensorimotor behaviors. Results provided unequivocal evidence that neural signals during the estimation phase of the task (between Ready and Set) evolved over a curved manifold that was temporally tuned to the distribution of the sample intervals (Fig. 3A, bottom).

The next key question was whether this curved manifold was indeed responsible for reducing the animals' timing variability. Further analysis of the neural data revealed how this curved manifold led to reduced variability: the curvature warped the internal representation of time and created a bias toward the high-probability intervals, thereby reducing variability (Fig. 3A, bottom). To substantiate this finding, Sohn et al. (2019) trained recurrent neural network models to perform the same task. Through learning, the model also established a curved manifold that was highly similar to the manifold observed in the monkeys' frontal cortex. Reverse-engineering the network models revealed that the curvature emerged as a result of experience-dependent modification of the coupling between neurons. Ongoing learning experiments in monkeys suggest that the curvature of the manifold plays a causal role in the control of timing variability. These findings provide direct neural evidence for the hypothesis that the brain controls the variability of its timing behavior by creating suitably curved manifolds in the sensorimotor cortex.

Neural dynamics in the sensorimotor systems during the processing of rhythmic timing

Mechanisms of timing have been studied not only via the perception of discrete intervals but also of rhythmic beat sequences. Beat- or rhythm-based time perception relies on the formation of internal predictive models that gradually develop in the primate order (Merchant and Honing, 2014). Humans are remarkably adept in recognizing the beat within a wide range of complex rhythms and show a spontaneous tendency to predict and synchronize to the beat (e.g., by tapping our fingers or feet to the beat). Notably, these actions tend to precede the beat by a few milliseconds. There is growing neurophysiological evidence that the motor system is significantly involved in this beat processing, even in the absence of actual movement (Grahn and Brett, 2007; Chen et al., 2008; Ross et al., 2018).

Neural ensembles show pronounced oscillations, alternating between high and low excitability states. Previous work has shown that, in sensory areas, the magnitude of the neural response to sensory input and the subsequent perceptual performance (e.g., the probability of near-threshold stimulus detection) correlates with the phase of intrinsic low-frequency oscillations (Busch et al., 2009; Mathewson et al., 2009; Ai and Ro, 2014). Based on these findings, it has been suggested that oscillations provide a substrate for sampling sensory inputs (VanRullen et al., 2011; VanRullen, 2016) and can also support rhythmic performance (Fiebelkorn et al., 2011; Landau and Fries, 2012).

Going beyond spontaneous oscillations, the neural entrainment hypothesis proposes that rhythmic sensory streams can reset the phase of ongoing neural oscillations, and thereby amplify the neural response to sensory events by synchronizing the internal phase with the external rhythm. In addition, entrainment represents a mechanism through which the brain may instantiate temporal predictions. Indeed, there is a remarkable match between the rhythms of many natural, behaviorally relevant events (e.g., speech, music) and rhythms in the brain (Zion Golumbic et al., 2013). It has been proposed that, if the high-excitability phase of intrinsic oscillations becomes aligned with task-relevant external events, sensory processing is optimized (Schroeder and Lakatos, 2009). This intuitively appealing notion of entrainment inspired many recent studies examining the involvement of neural oscillations in perceptual processing (for discussion, see Schroeder and Lakatos, 2009; Thut et al., 2011; Herbst and Landau, 2016; VanRullen, 2016). However, the influence of task demands and temporal context on oscillatory neural dynamics still lacks a thorough empirical foundation (Haegens and Zion Golumbic, 2018; Saskia Haegens, 2020).

In a recent study, Wilsch et al. (2020) recorded brain activity using MEG in healthy humans performing an auditory and visual target-discrimination task. Cross-modal cues provided both temporal and spatial information concerning the upcoming stimulus presentation. A rhythmic cue provided temporal information in the δ band (1-3 Hz), such that target stimuli were more likely to occur in-phase with the cued rhythm (80% of trials). Overall, temporal expectation effects were more prominent in the auditory than the visual domain. Providing a rhythmic temporal cue led to increased postcue synchronization of low-frequency oscillations, as assessed by intertrial phase coherence, compared with a condition with random cues. Moreover, increased intertrial phase coherence correlated positively with performance measures. However, contrary to the entrainment hypothesis, this effect was not restricted to the cued rhythm but was more broadband in nature. This result suggests a general phase reset rather than frequency-specific neural entrainment. These findings call into question the notion of neural entrainment in sensory systems as it is currently formulated in the literature (Saskia Haegens, 2020).

Recently, the neural processes that perceive and entrain to the simplest form of beat, an isochronous beat sequence, have been identified in nonhuman primates. Beat perception has been measured with mismatch negativity, an auditory event-related EEG potential that serves as an index for violations of temporal expectation. Notably, whereas mismatch negativity is sensitive to violations in both simple and complex rhythms in humans, monkeys only show this mismatch in isochronous rhythms (Honing et al., 2012, 2018). In addition, psychophysical experiments showed that, when the duration of the metronome's stimuli varied according to the stimulus–movement asynchronies, monkeys can predictively entrain to an isochronous beat, generating tapping movements with anticipation of the metronome (Gámez et al., 2018). These findings support the gradual audio-motor hypothesis, which suggests that beat-based timing emerged in primates and further developed in humans because of a more sophisticated audio-motor circuit (Merchant and Honing, 2014). Indeed, the privileged access of the humans' superior temporal areas of the dorsal auditory stream to the premotor areas of the frontal lobe and the neostriatum, the input of the basal ganglia, seem to have emerged gradually from precursors of the great ape lineage in the course of evolution (Rilling et al., 2008; Rauschecker and Scott, 2009; Balezeau et al., 2020). Entrainment might be present in macaques because of the close interaction between the medial premotor areas (supplementary motor area [SMA] and pre-SMA), the basal ganglia, and the auditory cortex. Indeed, in both humans and macaques, it has been shown that SMA and pre-SMA play a critical role in beat extraction and entrainment (Merchant et al., 2015a).

A fundamental property evident in neurons in the macaque medial premotor areas during rhythmic tapping is their relative representation of beat timing. Cells that encode elapsed time or remaining time-to-tap show ramping profiles that span the interbeat interval, scaling in speed as a function of tempo (Merchant et al., 2011; Merchant and Averbeck, 2017). These cells are recruited in rapid succession producing a progressive activation pattern that gradually fills the interval. Notably, the speed of this recruitment depends on the tempo, and the instantaneous pattern of activity provides a representation of how far an interval has evolved (Crowe et al., 2014).

Another critical aspect of the medial premotor beat-based clock is that it resets on every interval. Thus, the progressive pattern of activation starts with a group of cells, migrates to other cells during the timed interval, stops with the last group of cells, and simultaneously initializes the initial set of cells for the next interval (Merchant et al., 2015b). The cyclic evolution and resetting of neurons can be powerfully visualized by a state-space analysis of population dynamics (Fig. 3B) (Gámez et al., 2019). The population trajectories show the following properties: First, they have circular dynamics that form a regenerating loop for every produced interval. Second, they converge to similar locations in state space at tap events and reset the beat-based clock. Finally, the amplitude of the periodic trajectories increases with longer intervals of the isochronous beat (Fig. 3B) (Gámez et al., 2019).

While progress has been made in identifying neural correlates of rhythmic timing, insights about the neural contributions to beat-based timing remain scarce. Additional work is needed to connect current motor theories of beat perception to their neural underpinnings. Specifically, there is growing interest in whether the motor system plays a predictive role in rhythm and time perception. Transcranial magnetic stimulation in humans provides a powerful methodology to explore causal relations. Specifically, transcranial magnetic stimulation can temporarily excite or suppress cortical activity in focal motor and premotor regions to test the motor system's predictive role for beat-based timing.

For example, R.B. and colleagues showed that continuous theta-burst transcranial magnetic stimulation downregulated the left posterior parietal cortex and interfered with beat-based timing ability. It selectively affected the ability to detect shifts in beat-phase but did not affect the tempo (Ross et al., 2016, 2018). In contrast, the downregulation of the left SMA did not interfere with beat-based timing. This suggests that disruption of the internal model/simulation mechanisms for beat perception has a deleterious effect on accurate rhythm perception and auditory-motor synchronization (Patel and Iversen, 2014; Iversen and Balasubramaniam, 2016). Overall, these results support the idea that the sensorimotor system is not only active during the perception of rhythms (Grahn and Brett, 2007), but may even play a causal role in shaping the perception of rhythm (Comstock et al., 2018; Gordon et al., 2018).

Conclusion: a dynamic systems approach to sensorimotor timing

Temporal processing at the scale of tens or hundreds of milliseconds is a requirement for a wide range of behaviors, spanning basic skills such as interception of a moving target, advanced motor skills such as throwing, and complex social interactions, including language and music cognition. In any scenario, timing has two components: when an event will occur and how long an event lasts. The former implies the capacity to measure the time until the onset of an event; the latter implies the quantification of elapsed time from an input signal.

Many cognitive scientists have adopted a dynamic systems perspective to describe elapsed or predictive temporal processing (e.g., Pressing, 1999). This approach is concerned with continuous sensory and motor events, represented as trajectories in state space; for rhythmic timing, this approach has used coupled oscillators as the mathematical description of recurring events (Large and Jones, 1999). Similarly, rhythmic motor control has been successfully described by the dynamics of coupled oscillator models (Schmidt et al., 1993; Kelso, 1995; Sternad et al., 1999). Recent theoretical and neurophysiological studies have reached a similar conclusion: that timing behavior can be captured in terms of dynamic patterns of activity that emerge from interactions between populations of neurons (Laje and Buonomano, 2013; Remington et al., 2018; Wang et al., 2018). These interactions constrain the patterns of neural activity in the network and create low-dimensional neural trajectories. The structure and dynamics of these trajectories can be formalized using the same tools adopted by dynamic systems approaches to motor control. Given the importance of dynamics in theory, behavioral studies, and neurophysiology of timing, we propose a “computation through dynamics” perspective in which temporal computations emerge from and are tightly linked to the dynamics in the sensorimotor system.

Analysis of timing through the lens of dynamics could also explain other aspects of temporal control of behavior. For example, it is thought that timing circuits operate predictively in a vast repertoire of behaviors involving anticipation of sensory inputs of different modalities and movement planning with different effectors. This predictive processing is thought to involve the motor system, with the cortico-thalamic-basal ganglia circuit at its core (Wiener et al., 2010; Coull et al., 2011). The medial frontal cortex, including the SMA, is also thought to play a critical role (Grahn and Brett, 2007; Chen et al., 2008). Our work in animal models suggests that the mechanisms of predictive processing in both interval- and beat-based timing may be understood by their distinct underlying dynamics. For example, neural activity in the medial frontal cortex during the Ready-Set-Go task shows a form of temporal scaling that can be directly explained in terms of adjusting the speed of dynamics (Wang et al., 2018; Egger et al., 2019). In contrast, state trajectories during rhythmic timing display regenerating loops or orbits that can be readily explained in terms of attractor dynamics (Gámez et al., 2019). Importantly, in both interval- and beat-based timing, the dynamics operate predictively such that changes in the patterns of neural activity are proportional to the desired interval (relative timing) instead of tracking absolute elapsed time (Fig. 3).

These findings show a close link between dynamic processes in sensorimotor circuits and timing control, which may also be consistent with the broadly construed framework of embodied cognition. This perspective on cognition gives credence to the motor system and how it may play a critical role in higher-level perceptual and cognitive functions. The tenet is that our perceptual and cognitive processes are fundamentally grounded in our physical interactions with the environment (O'Regan and Noe, 2001; Wilson, 2002; Andersen, 2007; Barsalou, 2008; Adolph and Hoch, 2019). The dynamic systems view of motor coordination also consistently postulates that as individuals gradually acquire skill throughout practice, timing is an emergent property of movement patterns (Saltzman and Kelso, 1987; Zhang and Sternad, 2019). Therefore, as movement skills develop a stable spatiotemporal pattern, this motor timing arises implicitly and may contribute to shaping the timing of other domains such as perception and cognition.

Why do the dynamic processes in the motor system play such a prominent role in timing? One speculative possibility is that the neural circuits for movement are phylogenetically older and present a “legacy code,” that subsequent development of temporal structures in the perceptual and cognitive systems use and build on (Cisek, 2019). Further research is required to better understand the close connection between these domains of behavior. Investigating temporal control at the intersection of perception, cognition, and action may require experiments beyond highly controlled laboratory setups and adopt a more integrative approach to studying naturalistic behaviors. We hope that our work on the link between timing and the dynamics in the sensorimotor system will help pave the way in that direction.