The neuroanatomic and neurophysiological infrastructure for speech and language

Abstract

New tools and new ideas have changed how we think about the neurobiological foundations of speech and language processing. This perspective focuses on two areas of progress. First, focusing on spatial organization in the human brain, the revised functional anatomy for speech and language is discussed. The complexity of the network organization undermines the well-regarded classical model and suggests looking for more granular computational primitives, motivated both by linguistic theory and neural circuitry. Second, focusing on recent work on temporal organization, a potential role of cortical oscillations for speech processing is outlined. Such an implementational-level mechanism suggests one way to deal with the computational challenge of segmenting natural speech.

Introduction

Experimental research on the neurobiological foundations of speech and language processing has taken considerable strides in the last decade, due in part to advances in the methods available to study the human brain (improved resolution of recording techniques) and in part to more theoretically motivated research that builds on critical distinctions provided by the results of linguistics, cognitive psychology and computer science (improved ‘conceptual resolution’). As the neurobiology of language matures, the units of analysis continue to change and become increasingly refined: from (i) broad (and somewhat pre-theoretical) categories such as ‘production’ versus ‘perception/comprehension’ to (ii) subroutines of language processing such as phonology, lexical processing, syntax, semantics, and so on, to (iii) ever more fine-grained representations and computational primitives argued to underpin the different subroutines of language, such as concatenation, linearization, etc.

In all areas of language processing, noteworthy new perspectives have been developed (reviewed, among many others, e.g., in [1–3], with special emphasis on speech, linguistic structure-building, and the sensorimotor basis of speech/language, respectively). Notwithstanding the novel approaches, many of the substantive challenges are only now becoming clear. The number and arrangement of the cortical and subcortical regions underpinning speech and language processing demonstrate that the system is considerably more complex and distributed; the age of Broca’s and Wernicke’s areas and the era of left-hemisphere imperialism are over. Here I focus on a two issues that are redefining the research agenda, pointing towards a computational neurobiology of language [4], a research direction that emphasizes the representational and computational primitives that form the basis of speech and language.

There are, of course, many ways to illustrate the progress that has been made, highlighting new ideas and directions. One approach would be to review the different aspects or levels of language processing that have been examined in new neuroscientific experimentation, i.e. phonetics, phonology [5,6], lexical access [7–10], lexical semantics [11], syntax [12,13], compositional semantics [14,15], discourse representation [16,17]; moreover, the interaction of the linguistic computational system with other domains has been investigated in interesting ways, including how language processing interfaces with attention [18], memory [19], emotion [20], cognitive control [21], predictive coding [22–24], and even aesthetics [25]. A different approach is taken here, focusing first on the revised spatial map of brain and language; then, narrowing to one functional problem, a new ‘temporal view’ is discussed to illustrate a linking hypothesis between the computational requirements of speech perception and the neurobiological infrastructure that may provide a neural substrate.

The new functional anatomy: maps of regions, streams, and hemispheres

Our understanding of the anatomic foundations of language processing has changed dramatically in the last 10 years, ranging from the biggest to the most local levels. One might call this the maps problem [26], i.e. the challenge to define the best possible spatial map that describes the anatomic substrate [27–29]. The older, ‘classical’ view and its limitations are discussed further in Hagoort, this volume, where a contrasting dynamic network view of local function is described.

a. Starting at the most coarse level, consider the role of hemispheric asymmetry. Historically, the lateralization of language processing to the ‘dominant hemisphere’ has been one of the principal defining features. It was uncontroversial that language processing is strongly lateralized. However, a more nuanced and theoretically informed view of language processing, breaking down the processes into constituent operations, has revealed that lateralization patterns are complex and subtle – and that not all language processing components are lateralized. For example, when examining the cortical regions mediating speech perception and lexical level comprehension, lesion [30,31], imaging [32–34], and electrophysiological data [35,36] demonstrate convincingly that both left and right superior temporal cortical regions are implicated. Indeed, the operations mapping from input signals (e.g. sound) to lexical-level meaning, argued to be part of ventral stream processing (see b, below) appear to be robustly bilateral, as illustrated in Figure 1a (bottom panel).

Figure 1.

a. Dual stream model [1]. b. Note the bilateral arrangement of the ventral stream network (mapping from sound to meaning) and the lateralized dorsal stream projections (mapping to articulation). b. Bilateral processing of sensory-motor transformations for speech, from [37]. c. Spectrograms for three tasks: A’ Listen-Speak’ task: subjects heard a word and after a short delay had to repeat it; A ‘Listen-Mime’ task: subjects heard a word and after the same short delay had to move their articulators without vocalizing; ‘Listen’ task: subjects listened passively to a word. Sensory-motor (S-M) responses are seen in example electrodes in both the left (top row) and right (bottom row) hemispheres as demonstrated by a high gamma neural response (70–90+ Hz) present both when the subject listened to a word and when they repeated/mimed it. d. Population average brains with active electrodes demonstrate that S-M processes occur bilaterally (red). Electrodes that responded to the passive listen condition are also noted with green outlines (red with green outlines).

In contrast, it is typically argued that the structures and operations underlying production, for example, are lateralized. As illustrated in Figure 1a, one of the dorsal stream projections, suggested to underpin the sensory-motor mapping necessary for perception-production alignment, is depicted as fully lateralized. However, new data acquired in pre-surgical epilepsy patients using electrocorticography (ECog) seriously challenge even this generalization [37]. It is shown based on a range of tasks requiring sensory (sound)-to-motor (articulatory) transformation that the dorsal stream structures that provide the basis for this mapping are clearly bilateral as well (Figure 1b). Other, non-speech dorsal stream functions, for example operations that are part of grammatical relations, may be supported by other dorsal stream projections, and their lateralization pattern has not been fully established, although there appears to be a fair degree of lateralization to the dominant hemisphere (see [2] and Hagoort, this volume).

Various other imaging and physiology experiments on other aspects of language [38,23] also invite the interpretation that lateralization patterns are more complex than anticipated. Cumulatively, in other words, the language-ready brain (to use a phrase of Hagoort, this volume) appears to execute many of its subroutines bilaterally, unlike the 150-year-old dogma. And yet … There remain compelling reasons why an explanation for lateralization of function is required. Lesions to the left versus the right peri-Sylvian regions do not lead to the same linguistic deficits, so a new approach to this classical issue is necessary.

Which operations are functionally lateralized (and why) remains controversial. It has been debated for the speech perception case, for instance [39–41], but the nature of the questions is at least fine-grained, that is to say at the level of circuits that execute specific (possibly network-dependent) computations [42]. One hypothesis that merits exploration is as follows: the operations that comprise the processing of input and output systems (the interfaces) are carried out bilaterally; the linguistic operations per se, beyond the initial mappings, and specifically those that require combinatorics and composition (COM) as well as linguistically (structurally-) based prediction (PRE) are lateralized. A COM-PRE view predicts that the neural circuits mediating those types of operations on linguistic data structures are asymmetrically distributed; but how such circuits might look remains pure speculation.

b. The organization of language regions within a hemisphere has also seen a major shift in perspective. The classical model – still the prevailing view in most textbooks – identifies a few critical areas (Broca’s and Wernicke’s regions, often the angular gyrus, connected by the arcuate fasciculus) and attributes entire chunks of linguistic cognition to these large brain regions. (See Hagoort, this volume, for critique.) There now exists consensus that a more likely organization involves processing streams organized along dorsal and ventral routes. This is argued to be the case for speech [1,43,44], lexical level processing [45], syntactic analysis [2], and semantics [46].

The existence of concurrent ‘what,’ ‘where,’ and ‘how’ streams in vision highlights how large-scale computational challenges (localization, identification, sensorimotor transformation/action– perception linking) can be implemented in parallel by streams consisting of hierarchically organized cortical regions. This key idea from visual neuroscience was adapted for speech and language in the past 10 years [1,2,43,44]. Figure 1a illustrates one such model, constructed to account for a range of phenomena in speech. The bilateral ventral streams are considered responsible for supporting the transformation from auditory input to lexical representations. The dorsal stream (one of at least two parallel dorsal streams, see [2]) is primarily critical for sensorimotor transformations.

c. Critical advances on local anatomy. Until about 2000, the interpretation of experimental data implicating Broca’s area were made at a level of analysis referring to left inferior frontal gyrus and, at best, about Brodmann areas 44 versus 45. There exist some interesting connectivity studies, as well [47], but by and large the functional anatomic characterization has been coarse. (Hagoort, this volume, provides a more extensive discussion of the functional role of Broca’s area.)

New anatomic techniques (imaging, immunocytochemistry) have now been applied. Data in an influential paper [48] show that the organization of this one cortical region is much more complex, incorporating (depending how one counts) 5 – 10 local fields that differ in their cellular properties. Figure 2a [48] illustrates the currently hypothesized organization of Broca’s region, highlighting anterior and posterior as well as dorsal and ventral subdivisions of known fields and pointing to additional, separate opercular and sulcal regions that are anatomically identifiable.

Figure 2.

A. Broca’s region, from [48]. Note the numerous subdivisions in the region. B. Major pathways associated with left MTG, from [50]. Tractography data from two subjects are show. Each row depicts the individual subject’s ROI, represented in their native space (left, yellow). Sagittal, axial, and coronal slices of the fiber bundles involved are shown. As per DTI mapping, left MTG is massively connected to critical language regions in a hub-like manner.

It is highly likely that each sub-region performs at least one different computation — after all, they are anatomically distinct. Suppose then that there are merely five anatomic subdomains in Broca’s region, and suppose that each anatomic subdomain supports, say, two computations. These are conservative numbers, but we must then still identify 10 computations (underestimating the actual complexity). Some of the computations will apply to any input data structure (structured linguistic representation, speech signal, action plan, rhythmic sequence, etc.), since it is established that Broca’s region is engaged by numerous cognitive tasks [49]. Other computations may be dedicated to data structures native to the linguistic cognitive system. We know little about precisely what kind of conditions need to be met to trigger selectively the many areas that constitute this part of the inferior frontal cortex.

d. A final point about anatomy: the vast majority of research on brain and language has focused on the traditional peri-Sylvian language areas; but both cortical and subcortical regions that have not been investigated before as much play a critical role. (This point is amplified, as well, in Hagoort, this volume.) Two regions, in particular, deserve emphasis with respect to linguistic computation: the left anterior temporal lobe (ATL), and the posterior middle temporal gyrus (pMTG). The left ATL has been implicated in a variety of experiments on elementary structure assembly, i.e. the primitive combinatorics (COM) that underlie syntactic and semantic structure building. fMRI data show quite consistently that combination into phrasal units, perhaps in the service of subsequent semantic interpretation, is mediated there [2]. Recent MEG data [14,15] provide evidence about the dynamics; the experiments demonstrate that between 200–300 ms after the onset of a critical word that can be combined with a previous item, this region reflects the execution of basic combinatoric operations.

The pMTG appears to be a crucial lexical node in the larger network of language areas – and heavily connected to lATL. This region is, in general, very richly connected to other language areas (see Figure 2b, [50] and is driven by many lexical tasks in imaging studies, including lexical access, lexical ambiguity resolution, and other processes. Lesions to the MTG lead to severe word comprehension difficulties [51] and profound aphasia, if paired with STG lesions. Recent MEG data on the dynamics in pMTG during lexical access show that, after initial activation of lexical targets, unambiguously lexical attributes such as surface frequency and neighborhood density elicit MTG modulation [52] no later than 350 ms after word onset.

In sum, the functional anatomy of speech and language looks quite different now than the classical model has taught us. Locally highly structured areas are connected into networks that are themselves organized into processing streams that support broader computational goals (e.g. sound to meaning mapping, sound to articulation mapping) and that are, in turn, bilaterally supported for many linguistic functions. The research program for the next years will ask about which basic computations are executed in the most local circuits and which computations then group together to generate the linguistic functions that constitute language, say, syntax, phonology, lexical access, or semantics. Structure building – be it for sound or meaning [14,53,54] – requires the assembly of elementary representations by elementary computations, and it is ultimately these basic operations we are seeking to identify in a new neurobiology of language.

The temporal view: a linking hypothesis for the neurobiology of speech perception

For nearly a century, speech perception has been studied primarily from the perspective of the acoustic signal. Spectral properties were thought to provide the principal cues for decoding phonemes, and from these building blocks words and phrases would be derived. In cognitive neuroscience studies, the basic intuitions of such a model are investigated regularly, testing which acoustic-phonetic primitives are essential and invariant, acoustic attributes [6], distinctive features [5,6,55,56], or phonemes [57]. Despite the successes of this research program, many issues have remained unanswered. One major issue concerns how the brain extracts the relevant units for analysis to begin with.

Speech and other dynamically changing auditory signals (as well as visual stimuli, including sign) contain critical information required for successful decoding that is carried at multiple temporal scales (e.g. intonation-level information at the scale of 500–2000ms, syllabic information that is closely correlated to the acoustic envelope of speech, ~150–300 ms, and rapidly changing featural information, ~20–80 ms). These different aspects of signals (slow and fast temporal modulation, frequency composition) must be analyzed for successful recognition. Recent research has aimed to identify the basis for the required multi-time resolution analysis [58,72]. A series of recent experiments suggests that intrinsic neuronal oscillations in cortex at ‘privileged’ frequencies (delta 1–3 Hz, theta 4–8 Hz, low gamma 30–50 Hz) may provide some of the relevant mechanisms to parse continuous speech into the necessary chunks for decoding [59– 63]. To achieve parsing of a naturalistic input signal (e.g. speech signal) into elementary pieces, one ‘mesoscopic-level’ mechanism is suggested to be the sliding and resetting of temporal windows, implemented as phase locking of low-frequency (delta, theta) activity to the envelope of speech and phase resetting of the intrinsic oscillations on privileged time scales [58,64]. The successful resetting of neuronal activity, triggered in part by stimulus-driven spikes, provides time constants (or temporal integration windows) for parsing and decoding speech signals. Recent studies link the infrastructure provided by neural oscillations (which reflect neuronal excitability cycles) to basic perceptual challenges in speech recognition, such as breaking the continuous input stream into chunks suitable for subsequent analysis [59,63,65].

Two recent studies serve to illustrate the logic of the research program. Figure 3a [65] shows the data from a magnetoencephalography (MEG) recording in which subjects listened to continuous regular or manipulated speech. The data show the close alignment between the stimulus envelope and the response from auditory cortex, as has also been shown in other recent studies [59,62,63]. How this alignment between speech acoustics and neural oscillations might underpin intelligibility has been unclear. This study tested the hypothesis that the ‘sharpness’ of temporal fluctuations in the critical band envelope was a temporal cue to syllabic rate, driving the intrinsic delta or theta rhythms to track the stimulus and thereby facilitating intelligibility. It was observed that ‘sharp’ events in the stimulus (i.e. auditory edges) cause cortical rhythms to re-align and parse the stimulus into syllable-sized chunks for further decoding. Using MEG recordings it was shown that by removing temporal fluctuations that occur at the syllabic rate, envelope-tracking activity is compromised. By artificially reinstating such temporal fluctuations, envelope-tracking activity is regained. Critically, changes in tracking correlate with stimulus intelligibility. These results suggest that the sharpness of stimulus edges, as reflected in the cochlear output, drive oscillatory activity to track and entrain to the stimulus, at its syllabic rate. This process facilitates parsing of the stimulus into meaningful chunks appropriate for subsequent decoding.

Figure 3.

A. Entrainment of cortical activity to speech envelope, from [65]. B. Spectrograms of stimulus and neural activity illustrate correspondence in frequency domain. C. Contour maps of two conditions show typical auditory distribution. D. Attention dependent processing of a complex speech signal with two simultaneous speakers [69]. Stimulus is same mixture of two voices. Upper trace: attend to same speaker/stream on different trials. Lower trace: attend to different speaker/stream on different trials.

If neuronal activity locks or entrains to stimulus features, is this process subject to the vagaries of naturalistic communication, with wandering attention, for example? Several recent studies have tested this [66–69]; one result is shown in Figure 3b [69]. This ECog study investigated the manner in which speech streams are represented in brain activity and the way that selective attention governs the representation of speech using the ‘‘Cocktail Party’’ paradigm. It is shown that brain activity dynamically tracks speech streams using both low-frequency (delta, theta) phase and high-frequency (high gamma) amplitude fluctuations, and it is argued that optimal encoding likely combines the two – in the spirit of multi-time scale processing. Moreover, in and near low-level auditory cortices, attention modulates the representation by enhancing cortical tracking of attended speech streams (Figure 3b). Yet ignored speech remains represented. Jointly, these studies demonstrate the potential role that neuronal oscillations may play in the parsing, decoding, and attending to naturalistic speech [18].

Conclusion

There has, of course, been exciting progress in other more neuro-computational studies of language, including speech production [70], lexical representation and processing [(10)], and predictive coding [71]. This perspective is necessarily brief and selective. It is fair to say, however, that in the last 10–15 years, the model of how language is processed in the brain has changed dramatically from the classical perspective developed between 1861 (Broca), 1874 (Wernicke), 1885 (Lichtheim), and the 1970s (Geschwind). These changes are a consequence of a more mature linguistics, psychology, and neuroscience. It is important to acknowledge the immense contribution to basic and clinical neuroscience that the classical model has made – but to acknowledge as well that it cannot be carried forth as a realistic view on language and the brain.

Highlights.

• The anatomic organization for language is less lateralized than assumed.

• Speech and language processing are mediated by segregated concurrent processing streams.

• The anatomic and functional complexity of local regions needs to be acknowledged.

• Neural oscillations provide a possible mechanism to parse speech.