Continuous speech processing

Abstract

Speech processing in the human brain is grounded in non-specific auditory processing in the general mammalian brain, but relies on human-specific adaptations for processing speech and language. For this reason, many recent neurophysiological investigations of speech processing have turned to the human brain, with an emphasis on continuous speech. Substantial progress has been made using the phenomenon of “neural speech tracking”, in which neurophysiological responses time-lock to the rhythm of auditory (and other) features in continuous speech. One broad category of investigations concerns the extent to which speech tracking measures are related to speech intelligibility, which has clinical applications in addition to its scientific importance. Recent investigations have also focused on disentangling different neural processes that contribute to speech tracking. The two lines of research are closely related, since processing stages throughout auditory cortex contribute to speech comprehension, in addition to subcortical processing and higher order and attentional processes.

Introduction

Speech is inherently a dynamic and non-repetitive acoustic stimulus. For human listeners in particular, repeated presentation fundamentally alters how speech is perceived and experienced. As a consequence, there are limitations to the extent to which the neural basis of human speech perception can be studied with traditional, trial based experimental designs. Advances in the reverse correlation technique for electroencephalography (EEG), magnetoencephalography (MEG), and electrocorticography (ECoG) have opened the possibility of studying brain responses to long duration, non-repetitive stimuli [1–5], such as audiobooks, and have the potential to be applied to even more naturalistic stimulus materials. Additionally, continuous speech drives robust neural responses all along the auditory pathway, opening up the possibility of investigating multiple speech processing mechanisms, at different hierarchical levels, with the same speech stimuli.

Cortical speech tracking

Low frequency (< 10 Hz) cortical responses to continuous speech show a consistent phase relationship with the acoustic speech envelope [6], and the same holds for the slow envelope of the response at higher frequencies (gamma) [7]. This phenomenon is often referred to as “neural speech tracking” [8], and conceived of as an ongoing brain response related to moment-to-moment slow fluctuations in the amplitude of the speech envelope [1]. A simple quantitative measure of speech tracking can be obtained from stimulus reconstruction (Figure 1–A). A somewhat more differentiated model conceives of the neural responses as a continually evoked response to fluctuations in the speech envelope (Figure 1–B). The functions estimated to predict brain responses can be interpreted analogous to evoked responses and are often called the Temporal Response Functions (TRFs) [3]. In essence, both methods quantify the extent to which the brain response is linearly dependent on the speech envelope.

Figure 1.

Models for analyzing speech tracking. A) Stimulus reconstruction (backward model): a decoder is trained to reconstruct the stimulus envelope from the neural response, and speech tracking is quantified by how well the reconstructed envelope matches the actual envelope. A typical decoder uses a linear combination of the neural responses in a window following the envelope by 0 – 500 ms. B) Temporal response functions (TRFs) (forward model): a TRF is trained to predict the neural response from the speech envelope, and speech tracking is quantified by how well the predicted response matches the actual response. A typical TRF uses various delayed versions of the envelope from 0 – 500 ms. Responses originating from different brain areas are each characterized by their own TRF.

Because speech envelope tracking is so robust, it is a good candidate for use in clinical settings, especially as it employs an ecologically valid stimulus [9]. Envelope tracking, by definition, is a brain response closely related to an acoustic signal. Accordingly, for clean speech, presented in quiet, speech tracking often does not differ drastically between the native language and a language not spoken by the listeners [10]. However, because the envelope of speech is an important cue for speech intelligibility [11], speech tracking is used as a measure to assess whether pre-conditions for speech intelligibility are satisfied. For instance, for cochlear implant (CI) users it is of primary interest how well the modified speech signal is transmitted to the level of the cortex [12]. Speech tracking indeed may be causally related to processes required for successful comprehension, as transcranial alternating current stimulation (tACS) targeting the envelope of speech can negatively impact speech understating [13–15].

Yet, there is clearly no one-to-one correspondence between speech understanding and envelope tracking. For instance, even though older adults frequently complain of speech comprehension difficulty, cortical envelope tracking actually increases with advancing age [16–18]. An early observation was that speech tracking strength may correspond more to the perceived speech than simply reflecting the bottom up acoustic input. In responses to two talkers, the attended talker is often tracked more reliably than the ignored talker [19], and this modulation is robust enough to allow for detecting changes in the focus of attention in relatively short segments of data [20,21]. Here, envelope tracking thus measures how well the to-be attended speech is represented despite the fact that it is different from the actual acoustic input signal. Similarly, tracking even of clean speech is increased during periods in which attentional focus is high [22]. Such trial-by-trial variation in clean speech tracking has also been shown to reflect task performance, with better memory for words that occurred in sentences with higher speech tracking [23].

This raises the possibility that envelope tracking may reflect a sort of cleaned-up and attended-to representation of the acoustic input, which might form the basis for comprehension. For speech presented with different kinds of background noise, increased tracking of the attended envelope is associated with better speech understanding even after controlling for the objective background noise level [17]. Consistent with a strong top-down influence, tracking of the attended speech can actually be higher for speech in noise than for clean speech [24] and, for a well-known stimulus, tracking can even persist during short gaps in which the stimulus is replaced with pure noise [25]. In addition to this attentional enhancement, tracking of attended speech in noise differs qualitatively depending on whether the language is known to the listener [10,26], suggesting that speech tracking includes a language-specific component in addition to acoustic processing.

Envelope tracking thus likely reflects an interaction of the bottom-up input to the auditory cortex with resource-dependent, higher order processes. This is demonstrated by varying the amount of cognitive resources devoted to the speech [27]: At high signal to noise ratios (SNRs), speech tracking is similar, whether participants attend to the speech, or whether they ignore it and watch a silent movie instead. At lower SNRs, however, when more attentional resources would be required to recover the speech signal, speech tracking decreases much more in the movie condition. When subjects were playing a video game, speech tracking was even lower, decreasing even for clean speech. This suggests that speech tracking even of clean speech has a resource-dependent component, with increasing demands for speech in noise.

Components of speech tracking

The results summarized above suggest that, while the speech envelope is by definition an acoustic property of speech, considering speech tracking as a measure of basic acoustic processing is an oversimplification. A better understanding of speech processing requires disentangling representations of different properties of speech. One such dissociation can be gained by analyzing the temporal relationship between stimuli and responses using TRFs (Figure 1–B). For instance, when listening to two concurrent talkers, early (~50 ms) responses reflect the stimulus heard at the periphery, the acoustic mixture of the competing speech signals, whereas later (~100 ms) responses are dominated by a segregated version of the attended speaker [19,28–30]. These response components might also have a degree of task-dependency. For instance, under some challenging conditions, late representations may even specifically track the speech of the ignored talker [31]. Furthermore, not only is the speech envelope a collection of related acoustic speech features [32], but it is further modulated as carrier for linguistic units at different time scales from phonemes to phrases (as seen in Figure 2) [23]. Speech tracking thus likely reflects a family of representations at different hierarchical levels. Which of those representations exactly contribute to speech tracking probably differs across different stimuli and tasks. Tellingly, tACS at envelope frequencies selectively modulates neural activity in speech-specific, rather than general auditory brain regions [33], and brain responses that track the speech envelope are also found in areas outside of auditory cortex proper [4,34].

Figure 2.

Example of speech as a carrier for linguistic units at different time scales from phonemes to phrases. From [23], used with permission.

Auditory processing

For all these reasons, there is increasing interest in further disentangling the specific acoustic and linguistic features that drive the neural response through hypothesis-driven models. While the envelope is a useful summary variable, perhaps unsurprisingly cortical responses can be predicted more accurately when also considering other acoustic features. For example, a spectrogram, corresponding to the envelope of the acoustic signal computed separately for different frequency bands, reliably predicts brain responses better than the envelope alone [35,36]. A common variation on speech tracking uses a transformation of the envelope or spectrogram that emphasizes acoustic onsets [37], consistent with the observation that onsets are particularly important for intelligibility [38,39]. Used by itself, an acoustic onset spectrogram is indeed a better predictor than the envelope-based spectrogram [40], but both envelopes and onsets explain unique variability in the brain responses not explained by the other [35,41]. In addition, an anatomically localized region represents onsets not of local acoustic elements, but of larger acoustic groupings such as sentences and phrases [42]. Finally, a further level of complexity might come from non-linearities in responses to simple acoustic features, such as a modulation of the response to the envelope with absolute intensity [43].

Speech comprehension requires transformations of acoustic representations into speech-specific dimensions. For instance, brain responses are also modulated by the pitch contour of speech, which is an important component of prosody [44,45]. Because speakers differ in their fundamental voice pitch, the same acoustic pitch can have different linguistic implications for different speakers. Pitch thus needs to be normalized relative to the speaker to be interpreted linguistically. ECoG studies have shown such a speaker-dependent shift in response characteristics both in representations of vowels [46] and prosodic contours [44].

Linguistic processing

Eventually, acoustic representations are transformed into linguistic representations that are abstractions built upon the specific speech signal [47]. One approach to studying such representations is to predict brain responses from times-series of experimenter-coded linguistic features. A challenge for this approach is that, statistically, linguistic features can be highly correlated with the acoustic features used to communicate them. For example, each phoneme is defined as an equivalency class of related acoustic patterns. What might look like brain responses to categorical representations of phonemes, e.g., consonants, might actually be explained just by responses to acoustic onsets [35]. Few studies to date include a detailed acoustic model to control for acoustic representations; thus, results purporting to demonstrate sensitivity to linguistic features should be interpreted with care. One alternative approach to mitigate this issue is to decouple linguistic features from acoustic features by using fixed-rhythm speech [48,49], although this often still leaves ambiguity as to the specific linguistic features responsible for a certain response [50].

The approaches discussed so far are largely based on representations of specific stimulus features. Another approach that has been successful in linking brain activity to linguistic representations is though the predictive coding framework [51]. EEG studies of language processing have long used the N400, one of the most well-studies ERP components, as an index of how surprising a word is in its linguistic context [52]. A similar response is found in continuous speech based on word-by-word measures of how surprising each word is in its context [53–55]. An advantage of jointly modeling responses to acoustic and semantic properties is that it makes it possible to assess interactions between the two [56]. Furthermore, while the N400 literature might some times imply that there is only a single kind of surprisal, estimates of surprisal associated with different aspects of language may affect different brain areas, suggesting potentially separable underlying mechanisms [57]. This opens up the possibility of distinguishing different neural processes by comparing the predictiveness of surprisal computed from different language models, in particular when taking advantage of more advanced language models developed in linguistics and computer science [58,59].

Phoneme processing

The predictive coding framework also applies at the level of phonemes [60]: Phonemes can be described as acoustic patterns, but they are also information carriers. Speech perception can be cast as information transmission, where the information carrying units are phonemes, and the goal is to identify words [61]. Measures of the informativeness of phonemes thus provide an index of lexical processing of speech, and such measures are predictive of brain responses to continuous speech, even when controlling for a complex model of acoustic processing [41,62]. Furthermore, these measures show a striking dissociation in responses to two concurrent talkers: while acoustic features from both talkers are represented neurally to some degree (see Cortical speech tracking above), time-locked lexical processing is strictly associated with the attended talker only [41].

Cortical speech tracking as “entrainment”

Another possible source of modulation for speech tracking comes from the brain’s own internal rhythms. Given that speech is rhythmic, speech tracking reflects the brain matching certain external rhythms. These rhythmic responses may be more than just time-locked neural responses to rhythmic features, but rather endogenous rhythms that phase shift to match and predict speech rhythms [8,63]. Phase-locked responses and entrainment are often hard to dissociate, since the main predictions of both are synchronization between speech and brain rhythms. A clear signature of entrainment, distinct from phase-locked responses, may require showing a dissociation of the neural rhythms from the speech rhythms. This might come in the form of neural oscillations that out-live the stimulus [64], particularly if localized to the narrow frequency band in which the neural oscillators operate [65].

Subcortical speech tracking

Subcortical signals have a much lower amplitude than cortical signals, and are traditionally assessed through averaging thousands of repetitions of identical stimuli [66]. More recent work found that reverse correlation can also recover the brainstem response from a non-repetitive stimulus, such as an audiobook, without any repetitions [67,68]. Besides making the measurement of brainstem responses more entertaining for participants, the ability to measure subcortical responses with naturalistic stimuli also creates new opportunities to study brainstem responses in more ecological tasks. For example, some research suggests that selective attention to one out of two speakers modifies even brainstem representations of that speaker [68–70]. Such an effect is plausible given cortico-fugal connections [71], although it has not yet been replicated by other labs. Generally, this approach makes it possible to investigate cortical and subcortical responses to an ecologically valid stimulus concurrently in the same experiment.

Final thoughts

While there are many reasons for studying the neural processing of continuous, unrepeated speech, this is complicated by the fact that the acoustic speech signal is correlated in complex ways with the linguistic information it conveys. An advantage of the TRF approach is that it can model responses to acoustic and linguistic features jointly [4], and thus has some potential to decompose speech tracking into component neural mechanisms related to different aspects of speech processing. In addition, recent advances make it possible to study representations at multiple levels, from subcortical to semantic representations in the same dataset. This opens up new possibilities for studying different processing stages not just in isolation, but also for establishing connections and dependencies between processing at different stages. Understanding such connections might prove essential for clinical applications with a concern for speech comprehension “in the wild”.