Predictive motor control of sensory dynamics in Auditory Active Sensing

Abstract

Neuronal oscillations present potential physiological substrates for brain operations that require temporal prediction. We review this idea in the context of auditory perception. Using speech as an exemplar, we illustrate how hierarchically organized oscillations can be used to parse and encode complex input streams. We then consider the motor system as a major source of rhythms (temporal priors) in auditory processing, that act in concert with attention to sharpen sensory representations and link them across areas. We discuss the anatomo-functional pathways that could mediate this audio-motor interaction, and notably the potential role of the somatosensory cortex. Finally, we reposition temporal predictions in the context of internal models, discussing how they interact with feature-based or spatial predictions. We argue that complementary predictions interact synergistically according to the organizational principles of each sensory system, forming multidimensional filters crucial to perception.

Neuronal oscillations as substrates of temporal predictions

Sensory processing can be viewed as an inference problem: given the noisy sensory and neural signals, the brain estimates which stimuli caused these observations [1]. Accordingly, neuronal representations may be viewed as continuous top-down prediction/expectation signals, that is, internal models of causal dynamics in the world shaped by one’s perceptual, behavioral and emotional experience [2]. Such theories have two far-reaching corollaries: 1- predictions are reflected in the brain’s intrinsic activity [3,4], and 2- sensory evoked neuronal activity corresponds to the modulation of ongoing circuit dynamics by input signals, initiating the generation of an updated representation of the world [5**,6].

How are predictions encoded in the brain? Dynamic attending theory suggests that during perception attention is modulated dynamically to optimize sensory processing at expected points in time [7]. Crucially, this framework capitalizes on the fact that many stimuli and actions are rhythmically organized (e.g. speech, music, walking, among others) [8]. By extracting such temporal regularities, the brain is able to predict the occurrence of subsequent events of interest and optimize their processing.

Behaviorally, temporal predictions optimize perception by dynamically modulating the gain of sensory information [9*,10*]. Improved performance is associated with a reduction of the sensory evoked neuronal response [11-13]. Neurophysiologically, low-frequency neuronal oscillations are the hypothesized substrate of temporal predictions [4,5**,14]. Neuroelectric oscillations correspond to temporal fluctuations of the excitability state of neuronal ensembles (Figure 1C) [4]. Using these as ‘instruments’ [15,16] the brain can selectively amplify neuronal responses to events occurring at predicted moments, while suppressing responses to events that do not adhere to the prediction [17,18,19**]. This mechanism results in a selective neuronal tracking of attended sensory streams, as exemplified by the cocktail party effect, where listeners have to ‘tune in’ to one conversation in a noisy scene [20].

Figure 1. Multiple time scales of speech and auditory brain rhythms.

(A) Time-frequency decomposition of a sentence envelope (a 1/f detrending on the signal’s amplitude was applied for visualization purpose). (Upper inset) Sentence waveform and its corresponding envelope (black thick line). (Right inset) Modulation spectrum. (B) Time-frequency decomposition of the spontaneous activity recorded in the supragranular primary auditory cortex of an awake macaque monkey. (Upper inset) Current source density (CSD) raw data. (Right inset) Modulation spectrum. Adapted from [69]. (C) Relationship between excitability, as indexed by the action potential firing rate (red), and the phase of oscillation, as indexed by a local field potential (blue); neuronal oscillations have optimal (high excitability) and non-optimal (low excitability) phases. (D) Schematic highlighting the typical ‘nested’ nature of oscillations. The (top) green trace illustrates a typical recording of oscillations. The traces below illustrate the individual oscillatory components in the delta (1.5Hz), theta (7Hz) and low gamma (35Hz) bands that comprise the composite waveform. Hierarchically organized phase-amplitude coupling between frequencies is present. (C-D) Adapted from [14]. (E) From top to bottom: a. Low-frequency LFP, with phase angle color-coded. b. Binned LFP phase during 30 trials. c. Corresponding spike raster plot. d. Mean firing rate across trials: The colored bars indicate the typical phase at each peak. Gray lines mark pairs of peaks with similar mean firing rate but different phase angle. The two instances in each pair (e.g. a1, a2) can be distinguished based on LFP phase but not based on the firing rate. Adapted form [32].

A prototype: the hierarchy of time scales in speech processing

Speech perception arises from the dynamic sampling of acoustic information at multiple time-scales. Specifically, entrainment of cortical oscillatory dynamics to the acoustic envelope has been described for delta (1-4 Hz), theta (4-8 Hz), and low-gamma (25-45 Hz) frequencies (Figure 1 A-B) [14,21*,22*]. And top-down predictive influences on speech perception are amply supported by multiple lines of evidence [14,22*,23-25].

The most basic level of the proposed speech processing hierarchy is delta-band mediated parsing of the prosodic phrasal rate of speech [14,21*]. A right-lateralized delta entrainment is observed during speech processing, co-occurring with theta entrainment, but carrying complementary information [22*]. Left-lateralized involvement of delta oscillations is on contrary hypothesized to reflect motor-driven top-down modulation [26,27]. The idea that motor influences contribute to sensory speech processing places it squarely in the domain of ‘active sensing’ [28,29], as outlined below.

Although phrasal and syllabic modulations (1-7 Hz) are crucial to speech comprehension [30], acoustic modulations above 12 Hz also contribute to speech identification, as they are associated with some phonetic feature dimensions [31]. The speech processing hierarchy, as it stands now, does not explicitly deal with the question of how the auditory system encodes the higher frequency information, densely packed in consonant sounds. An obvious possibility is that the neuronal representation of speech utilizes a spike phase code, like that described for naturalistic auditory scene processing [32]. In such a scheme, entrainment of a delta-theta-gamma hierarchy in auditory cortex to the nested low, middle and higher frequency rhythms inherent in speech provides a dynamical structure against which neuronal firing in response to speech sounds can be indexed (i.e. a reference frame). As was shown by Kayser and colleagues (Figure 1E) [32], concurrent implementation of complementary stimulus coding schemes presents several benefits that would be particularly useful in speech comprehension. In particular, such a scheme would increase the amount of information that can be packed into a neuronal firing pattern and make the aggregate representation more robust to noise [32]. In order for higher order neuron assemblies to act as efficient readers they would have to share the reference frame, likely by entraining at the same set of frequencies as the neuron assemblies performing the encoding step in auditory cortex. Interestingly in this regard, widespread delta and theta band entrainment in frontal and prefrontal regions complements that occurring in auditory cortical areas during speech tracking [20].

The proposition of a hierarchical system of brain dynamics that can parse and decode the information that is embedded within at least four nested temporal scales in speech has wide-ranging implications for non-auditory, as well as auditory processing. The fastest scale in this scheme corresponds best to neuronal firing. Firing is essential for the transmission of information through the system, but decoding that information by central readers depends on the fact that sensory evidence is parsed by a nested hierarchy of oscillations and information is systematically encoded according to oscillatory phase within each scale.

Motor origin of temporal predictions in active sensing

Capitalizing on predictive coding, and other Bayesian inference frameworks, recent papers suggest that in addition to generating movements, the motor system may be essential in generating the temporal predictions that shape perception (Figure 2A and 4A) [5**,29,33,34]. The motor system has been shown to be a key structure in the precise estimation of short durations [35,36], and in beat and rhythm processing [33,37,38]. The motor system is likely involved in a wide variety of temporal-predictive functions, and may be a major causal agent in both ambient and task-related rhythms in sensory pathways. As discussed above, these rhythms would imbue the motor system’s predictions.

Figure 2. Regional connections of auditory cortex in the macaque brain.

(A) Topographic connections of belt/parabelt auditory areas with areas of prefrontal, posterior parietal, superior temporal, and visual cortex (red: rostral areas; blue: caudal areas). All connections are reciprocal. Line thickness denotes relative strength. Primary (core) areas (not shown) have extremely limited connections outside of auditory cortex. (B) Schematic diagram summarizing connections of belt/parabelt areas with sensory, multisensory, and motor areas. Note that connections between auditory and premotor/motor areas have not been established in primates, and are either weak or absent. (C) Schematic diagram of principal routes of information flow between major regions of cortex, basal ganglia, and thalamus. Corticostriatal projections from most areas are topographically organized and not reciprocal. (D) Summary of topographic projections from belt/parabelt auditory cortex to caudate and putamen. Topographical relationships indicated by color gradient. Primary (core) auditory cortex does not project to the striatum in primates, or projections are limited. Abbreviations: CB/CPB, caudal belt and parabelt; RB/RPB, rostral belt and parabelt; M1, primary motor cortex; S2, second somatosensory area; PV, parietoventral somatosensory area; Ri, retroinsular area; STGr, rostral superior temporal gyrus; TPO, temporal polysensory area; Tpt, temporal parietotemporal area; VIP, ventral intraparietal area; V1/V2, visual areas 1 and 2; ProS, area prostriata (visual); VA, ventral anterior nucleus; VL, ventral lateral nucleus; MD, medial dorsal nucleus; GP, globus pallidus; NB, nucleus basalis.

Figure 4. Auditory active sensing.

(A) A forward model (corollary discharge/efference copy) predicts the sensory consequences of a movement based on the motor command. When a movement is self-produced, its sensory consequences can be accurately predicted and this prediction can be used to attenuate the sensory effects of the movement. Adapted from [71]. (B) Example of auditory active sensing paradigm. Participants listen to a sequence of pure tones and then estimate the average pitch of targets, while ignoring distractors. First row: rhythmic motor tracking in phase with the reference beat throughout the sequence. Second row: references indicating the beat. Third (fourth) rows: targets (distractors) presented in phase (antiphase) with the reference beat. Dark (light) grey lines indicate the temporal distance between the motor act and the onset of the target (distractor). Fifth row: sensory gains assigned to successive targets and distractors. The shorter the temporal distance between a tone and a motor act, the stronger the temporal prediction, and the higher the sensory gain assigned to the tone. Adapted from [10*].

Active sensing [29] mainly refers to acquiring sensory inputs by overt motor sampling behaviors, such as hand movements [39*,40] or eye movements [41,42], and whisking or sniffing [28,43,44]. These sampling movements are generally rhythmic, but can be aperiodic, as in an isolated hand-reach [39*]. In either case, attention is an essential component of the process, and its top-down influence helps to impose the motor sampling pattern on the relevant sensory stream; in addition to driving of activity in sensory areas by volleys of ascending input, there is top-down (corollary discharge and/or attentional) modulation [45], all yoked to the movements [29]. ‘Covert’ attentional sampling is a part of active sensing that takes over where overt motoric sampling leaves off. When holding still and attending covertly to one of several rhythmic sensory streams (e.g. one conversation at a cocktail party), the brain ‘selects’ by enforcing oscillatory entrainment of sensory activity to one task-relevant stream at the expense of all others. This is true whether attention is selecting between modalities [17,46], within a modality [19**,47], or integrating across modalities [20]. Motor systems appear integral to covert-attentional, as well as overt-motoric sampling of sensory input. Motor and premotor cortices consistently emerge as the most active regions in both scalp EEG studies [48] and intracranial studies of low frequency rhythmic attentional selection [20,49]. Low frequency (delta) rhythms in M1 predict the timing of informative cues in a motor planning task [50] and facilitate coherence between M1/M2 and parietal regions [51]. These observations reinforce the more general conclusion that, while the brain’s time-keeping networks likely involve cerebellum, basal ganglia, insular cortex and thalamus [48,52], motor and premotor cortices are critical nodes in these networks [53,54], especially when the task is difficult [55], and rhythms are used to anticipate and predict [26,49,50]. According to the Pre-motor Theory of Attention [56], covert shifts of spatial attention are governed by the same FEF-centric network that underlies saccades of similar dimensions. Although one can argue specific tenets of the Theory [57], it clearly underscores a fundamental relationship between motor systems and active/attentive sensory processing, one which comes to the fore when event rhythms allow predictions [10*,29].

Active sensing in the auditory domain

The active sensing framework proposes that perception is shaped by the motor system in two distinct ways. First, the motor system directly causes sensory inflow as a consequence of the motor acts it directs (e.g. finger squeezes cause somatosensory stimulation, ocular saccades cause specific visual stimulation, sniffs cause stimulation of the olfactory epithelium), thereby structuring both the specific content and the temporal/rhythmic context of bottom-up sensory information inflow. Second, it modulates the processing of sensory information via top-down attentional control and/or corollary discharge signals, that is, copies of movement commands sent to associated sensory structures [58]. Top-down signals predictively modulates sensory processing according to the temporal (and spatial) patterns of attentional and motor sampling plans, thus providing ‘when’ (and ‘where’) predictions at a minimum [45].

Although motor influences over auditory cortices have been reported [59*], active sensing has not been described in the auditory domain, because, in contrast to the other modalities, bottom-up auditory processing is generally disconnected from movement. The bulk of auditory inputs are like the subsets of somatosensory inputs that fall on a still hand, or visual events that occur during fixation. Even in species that can move their ears flexibly, such as rabbits, cats and monkeys, these movements function like gross head movements that are used to maximize binaural cues. Crucially, motor acts do not cause inflow of auditory input, excepting when movements make noise. Nonetheless, temporal sensitivity is at its best in audition. The striking relationship between rhythm in the auditory and motor systems [37,60] in music perception points to a more general role that the motor system could play in exerting top-down predictive influences on auditory processing. In the speech example, top-down motor influences would structure both the parsing of complex input streams and the construction of the reference frame for the phase coding of specific acoustic elements in spiking.

Although direct projections from secondary motor cortex to auditory cortex have been observed in mice [59*], no such projections have been as yet reported in primates. There are, however, a number of routes by which activity in the motor system as broadly construed to include primary motor, premotor and motor-related prefrontal regions, basal ganglia, and surprisingly the somatosensory system, may modulate the gain of auditory processing and perception (Fig 2).

Notably absent are projections from motor (Area 4) and premotor (Area 6) regions although there are some projections from Areas 45 and 8 to Temporo-parietal (Tpt), caudal belt and parabelt areas. Prefrontal cortical connections with rostral belt/parabelt areas are reciprocal, but would be an exceedingly indirect relay of motor corollary discharge. There are topographic projections to the striatum from both rostral and caudal belt/parabelt regions of auditory cortex, which could inform motor planning and movement generation, but the return connection by which motor rhythms would be injected into the auditory processing stream is, again, indirect at best.

One interesting possibility outlined here is that motor influences may be conveyed to auditory cortex through adjacent somatosensory areas in the Lateral Sulcus. Movement related signals clearly would access belt regions of auditory cortex through these connections. In this regard, it is intriguing to note that in a task entailing rhythmic presentation of attended audiovisual stimuli, multiunit firing and beta band power in second somatosensory (S2) ensembles (Figure 3) display systematic fluctuations, reflecting modulation of local neuronal excitability in anticipation/prediction of potential target stimulus appearance. The predictive component of the activity is like that shown in motor cortex itself [50]. The Beta power drop and associated excitability changes are like those that attend movement onset in motor cortex [61]. These dynamics reflect a complex interdependence of motor and somatosensory physiology surrounding goal directed movement and likely occur in S2 and other somatosensory cortices by virtue of the somatosensory system’s extensive connections with the motor system. Auditory belt cortices in turn, would have access to these signals due to their extensive interconnections with the somatosensory system [62]. It is not yet clear whether these S2 effects manifest in auditory cortex.

Figure 3.

Summary of the average temporal patterns of MUA (blue) and the CSD-derived beta band power (red) in the supragranular layer of the second somatosensory area (S2) recorded from one monkey during performance of an audio-visual (AV) oddball task [70]. Although the monkey held a lever, a series of audiovisual stimuli were presented. Data are aligned to the onset (0 sec) of AV stimuli during the task. Stimuli were repeating 500 ms vocal movie clips (SOA = 1.4 sec; static-to-movie face) with random oddballs (20%) in which face or voice differed from the standard. Upon detection of an oddball, monkey released the lever for a reward. The black dashed line indicates the mean reaction times on the adjoining target trials. For each signal, mean and 95% confidence intervals are shown (n=16, bootstrap). Each signal was normalized by the standard deviation estimated from periods of −2 to 2 sec from the onset of stimuli. MUA was sampled at 2 kHz, CSD power was sampled at 0.5 kHz.

Motor contributions to the temporal precision of auditory attention

We recently developed a mechanistic behavioral account of auditory active sensing [10*]. We measured participants’ ability to extract relevant auditory information interleaved with distracting information and embedded in rhythmic streams; thus, performance depended crucially on temporal predictions. We used a perceptual decision-making task, to study the dynamics of evidence accumulation and its modulation by temporal predictions. During the task, participants were either allowed to use their motor system, by noiselessly pressing in rhythm with the relevant auditory information, or prevented from doing so.

Findings (Figure 4B) show that rhythmic movements engage a top-down modulation that sharpens the temporal selection of auditory information [10*]. It improves the segregation between relevant and distracting information, facilitating perception of relevant items and suppressing perception of irrelevant items. The impact of overt motor tracking on perception parametrically scales with the temporal predictability of the auditory sequence, and depends on the temporal alignment between motor and attention fluctuations.

Beyond the sensory level: hierarchical organization of predictive filters in auditory perception

Ideally, one would study temporal predictions independently from other dimensions (e.g. location or feature). However, in most paradigms these other attributes are held constant and predictable across trials, and thus may constrain perception. Indeed, in primary auditory cortex the combination of temporal and spectral predictions results in both the amplification and sharpening of expected neuronal responses, acting as a spectrotemporal filter mechanism [19**]: The top-down modulation of neuronal excitability is dynamically and simultaneously adjusted at the temporal and spectral level. Such combination is not restricted to what and when dimensions in audition, but can also integrate spatial [63] and intensity [64] information at higher levels. Thus predictions seem to constantly fuse into a dynamically evolving model of upcoming events, which enhances the representation of predicted stimuli.

However, predictions do not seem to combine linearly. For example, temporal and spatial predictions interact to improve the quality of visual information, but the benefit of temporal predictions may disappear in lieu of valid spatial predictions [65,66*]. Thus, the impact temporal predictions have on perception may be conditional upon the co-occurrence of spatial predictions. The lower visual areas being retinotopically (i.e. spatially) organized, temporal predictions might not be able to modulate the first cortical stages of visual processing unless they are part of a spatiotemporal predictive filter that taps onto retinotopic organization. The same logic applied to the lower levels of the auditory system would argue that the most effective predictions would address tonotopic as well as temporal dimensions of stimulation [19**].

It is likely that the hierarchical progression of processing starts with the most fundamental attributes at the lower sensory levels, rhythm and sensory primitives (e.g. frequency in auditory cortex). Salient stimuli can drive rhythmic entrainment in A1 in a bottom-up fashion without the subject’s attention [67], in which case, the very tendency of local pyramidal/interneuron ensembles to oscillate and nest hierarchically in delta, theta and gamma frequencies at rest, provides a system of predictions. As discussed above, the entrained oscillations provide initial parsing of the input stream and a reference framework for phase coding of information carried in spike trains across the population of A1 neurons. The untested assumption here is that at the level of A1, oscillatory parsing with spike-phase coding creates ‘packets’ of information that are ‘multiplexed’, by combining coding across different scales in the hierarchical oscillatory structure. As these packets progress to higher order areas, additional predictions/filters are applied with increasingly longer time constants [3], promoting integration over sensory input stream segments of increasing length. Though speech is an appealing illustration, in principle this would apply to any complex, naturalistic sensory stream.

In active sensing, top-down control becomes determinative, and the system’s evolved capabilities combine with the subject’s experience to allow stronger predictions. In the ‘Motor Theory’ of speech perception for example [68], the motor system provide the filter. In the ‘Premotor Theory’ of attention [56], the premotor regions would do the same for covert visual active sensing. When actively engaged, motor and premotor regions would reinforce the natural tendency for hierarchical entrainment of A1 neuron ensembles to the rhythmic structure of the input stream. These effects are the basis for additional transformation of the input representation in belt areas and beyond. Successively higher-order premotor/prefrontal areas modulate motor cortex to impose hierarchical (predictive) oscillatory structure like that we suppose is operating in language, and this helps to group the processing of primitives like stimulus pitch/frequency and Interaural time difference into higher order structures (‘a language-like’ stimulus streams) and also combines across primitive code categories to form auditory objects. This thinking is consistent with the idea (e.g. [3]) that the time constant of activity increases at higher order regions. Such progressive abstraction sets the stage for covert (mental) processes that are largely disconnected from the temporal patterns of the ongoing sensorium.