Structured neuronal encoding and decoding of human speech features

Abstract

Human speech sounds are produced through a coordinated movement of structures along the vocal tract. Here we show highly structured neuronal encoding of vowel articulation. In medial-frontal neurons, we observe highly specific tuning to individual vowels, whereas superior temporal gyrus neurons have non-specific, sinusoidally-modulated tuning (analogous to motor cortical directional tuning). At the neuronal population level, a decoding analysis reveals that the underlying structure of vowel encoding reflects the anatomical basis of articulatory movements. This structured encoding enables accurate decoding of volitional speech segments and could be applied in the development of Brain-Machine Interfaces for restoring speech in paralyzed individuals.

Introduction

The articulatory features that distinguish different vowel sounds are conventionally described along a two-dimensional coordinate system that naturally represents the position of the highest point of the tongue during articulation, e.g., in the International Phonetic Alphabet (IPA) chart¹. The two axes of this system are height (tongue vertical position relative to the roof of the mouth or the aperture of the jaw) and backness (tongue position relative to the back of the mouth). How is the structured articulatory production encoded and controlled in brain circuitry? The gross functional neuroanatomy of speech production was described by multiple imaging, lesion and stimulation studies^2,3 and includes primary, supplementary and pre-motor areas, Broca’s area, superior temporal gyrus, anterior cingulate cortex and other medial frontal regions^2-4. The temporal dynamics of collective neural activity was studied in Broca’s area using local field potentials^2,5. However, the basic encoding of speech features in the firing patterns of neuronal populations remains unknown.

Here, we study the neuronal encoding of vowel articulation in the human cerebrum at both the single-unit and the neuronal population levels. At the single neuron level, we find signatures of two structured coding strategies: highly specific sharp tuning to individual vowels (in medial-frontal neurons) and non-specific, sinusoidally-modulated tuning (in the superior temporal gyrus). At the neural population level, we find that the encoding of vowels reflects the underlying articulatory movement structure. These findings may have important implications for the development of high-accuracy brain-machine interfaces for the restoration of speech in paralyzed individuals.

Results

Speech-related neurons

Neuronal responses in human temporal and frontal lobes were recorded from eleven patients with intractable epilepsy monitored with intracranial depth electrodes to identify seizure foci for potential surgical treatment (see Methods). Following an auditory cue, subjects uttered one of five vowels (a /α/, e /ε/, i /i/, o /o/, u /u/), or simple syllables containing these vowels (consonant+vowel: da /dα/, de /dε/, di /di/, do /do/, du /du/…). We recorded the activity of 716 temporal and frontal lobe units. As this study focuses on speech and due to the inherent difficulty to distinguish between auditory and speech related neuronal activations, we analyzed only the 606 units that did not respond to auditory stimuli. A unit was considered speech-related if its firing rate during speech differed significantly from the pre-cue baseline period (see Methods). Overall, 8% of the analyzed units (49) were speech-related, of which, more than a half (25) were vowel-tuned, showing significantly different activation for the 5 vowels (see Supplementary Figure S1).

Sharp and broad vowel tuning

Two areas commonly activated in speech studies², the superior temporal gyrus (STG) and a medial-frontal region overlying the rostral anterior cingulate and the adjacent medial orbitofrontal cortex (rAC/MOF; Brodmann areas 11 and 12; See Supplementary Figure S2 for anatomical locations of the electrodes), had the highest proportions of speech-related (75% and 11%, respectively) and vowel-tuned units (58% and 77% of these units). In imaging and electrocorticography studies, the rostral anterior cingulate cortex was shown to participate in speech control^2,6 the orbitofrontal cortex in speech comprehension and reading⁷ and the STG in speech production at the phoneme level⁸. Involvement of STG neurons in speech production was also observed in earlier single unit recordings in humans⁹. We analyzed neuronal tuning in these two areas and found that it had divergent characteristics: broadly tuned units that responded to all vowels, with a gradual modulation in the firing rate between vowels, comprised 93% of tuned units in STG (13/14) but were not found in rAC/MOF (0/10), whereas sharply tuned units that had significant activation exclusively for one or two vowels comprised 100% of the tuned rAC/MOF units (10/10) were but were rare in STG (1/14).

Figure 1 displays responses of five sharply tuned units in rAC/MOF, each exhibiting strong, robust increases in their firing rate specifically for one or two vowel sounds, whereas for the other vowels firing remains at the baseline rate. For example, a single unit in the right rAC (Fig. 1, top row), elevated its firing rate to an average of 97 spikes/s when the patient said “a”, compared with 6 spikes/s for “i”, “e”, “o” and “u” (p < 10^-13, one-sided two-sample t-test). Anecdotally, in the first two trials of this example (red arrow) the firing rate remained at the baseline level, unlike the rest of the “a” trials; in these two trials the patient wrongly said “ah” rather than “a” (confirmed by the sound recordings).

Figure 1.

Sharply-tuned medial-frontal (rAC/MOF) units. Raster plots and peri-stimulus time histograms (PSTH) of five units during the utterance of the five vowels: a, e, i, u, and o. For each of the units, significant change in firing rate from the baseline occurred for one or two vowels only (see Methods). Red vertical dashed lines indicate speech onset. All vertical scale bars correspond to firing rates of 20 spikes/s.

A completely different encoding of vowels was found in the STG, where the vast majority of tuned units exhibited broad variation of the response over the vowel space, both during the articulation of vowels (Fig. 2a), and simple syllables containing these vowels (see Supplementary Figure S3a). This structured variation is well approximated by sinusoidal tuning curves (Fig. 2b and Supplementary Figure S3b) analogous to the directional tuning curves commonly observed in motor cortical neurons¹⁰. Units shown in Fig. 2 had maximal responses (“preferred vowel”; in analogy to “preferred direction”) to the vowels “i” and “u” which correspond to a closed articulation where the tongue is maximally raised, and minimal (“anti-preferred”) response to “a” and “o” where it is lowered.

Figure 2.

Broadly-tuned STG units. a, Raster plots and PSTHs during the utterance of the five vowels: a, e, i, u, and o. Significant change in firing rate from the baseline occurred for all or most vowels, with modulated firing rate (see Methods). Red vertical dashed lines indicate speech onset; vertical bars, 10 spikes/s. b, Tuning curves of the respective units in a over the vowel space, showing orderly variation in the firing rate of STG units with the articulated vowel.

Population-level decoding and structure

Unlike directional tuning curves, where angles are naturally ordered, vowels can have different orderings. In the tuning curves of Fig. 2 we ordered the vowels according to their place and manner of articulation as expressed by their location in the IPA chart¹, but is this ordering natural to the neural representation? Instead of assuming a certain ordering, we could try and deduce the natural organization of speech features represented in the population-level neural code. That is, we can try to infer a neighborhood structure (or order) of the vowels where similar (“neighboring”) neuronal representations are used for neighboring vowels. We reasoned that this neighborhood structure could be extracted from the error structure of neuronal classifiers: when a decoder, such as the population vector¹¹ errs, it is more likely to prefer a value which is a neighbor of the correct value than a more distant one. Thus, classification error rates are expected to be higher between neighbors than between distant features when feature ordering accurately reflects the neural representation neighborhood structure. In this case, classification error rates expressed by the classifier’s confusion matrix will have a band-diagonal structure.

To apply this strategy, we decoded the population firing patterns using multivariate linear classifiers with a sparsity constraint to infer the uttered vowel (see Methods). The five vowels were decoded with a high average (cross-validated) accuracy of 93% (significantly above the 20% chance level, p < 10^-5, one-sided one-sample t-test, n = 6 cross-validation runs; see Supplementary Table S1), and up to 100% when decoding pairs of vowels (Fig. 3a). Next, we selected the vowel ordering that leads to band diagonal confusion matrices (Fig. 3b). Interestingly, this ordering is consistent across different neuronal subpopulations (Fig. 3b and Supplementary Figure S4) and exactly matches the organization of vowels according to their place and manner of articulation as reflected by the IPA chart (Fig. 3c). Because the vowel chart represents the position of the highest point of the tongue during articulation, the natural organization of speech features by neuronal encoding reflects a functional spatial-anatomical axis in the mouth.

Figure 3.

Inferring the organization of vowel representation by decoding. a, Average decoding accuracy (±standard error) vs. the number of decoded vowel classes. Red dashed lines represent the chance level. b, Confusion matrix for decoding population activity of all analyzed units to infer the uttered vowels. Band diagonal structure indicates adjacency of vowels in the order: a,e,i,u,o in the neural representation. High confusion in the corner (between u and i) implies cyclicity. c, The vowels IPA chart, representing the highest point of the tongue during articulation, on top of a vocal tract diagram. The inferred connections (blue lines), demonstrate neuronal representation of articulatory physiology.

Discussion

These results suggest that speech related rAC/MOF neurons use sparse coding for vowels, in analogy to the sparse bursts in songbirds’ area HVc¹² and to the sparse, highly selective responses observed in the human medial temporal lobe¹³. In contradistinction, the gradually modulated speech encoding in STG implies previously unrecognized correlates with a hallmark of motor cortical control - broad, sinusoidal tuning, implying a role in motor control of speech production⁹. Interestingly, speech encoding in these anatomical areas is opposite in nature to that of other modalities: broad tuning for motor control is common in the frontal lobe¹⁰ (vs. STG in the temporal lobe) whereas, sparse tuning to high-level concepts is common in the temporal lobe¹³ (vs. rAC/MOF in the frontal lobe). Analogous to the recently-found sub-pathway between the visual dorsal and ventral streams¹⁴, our findings may lend support to a speech-related “dorsal stream” where sensorimotor prediction supports speech production by a state feedback control³. The sparse rAC/MOF representation may serve as predictor state, in line with anterior cingulate¹⁵ and orbitofrontal¹⁶ roles in reward prediction. The broad STG tuning may support evidence that the motor system is capable of modulating the perception system to some degree^3,17,18.

Our finding of sharply-tuned neurons in rAC/MOF agrees with the DIVA model of the human speech system¹⁹, which suggests that higher-level prefrontal cortex regions involved in phonological encoding of an intended utterance, sequentially activate speech sound map neurons that correspond to the syllables to be produced. Activation of these neurons leads to the readout of feedforward motor commands to the primary motor cortex. Due to orbitofrontal connections to both STG²⁰ and ventral pre-motor cortex (vPMC)²¹, rAC/MOF neurons may participate also in the feedback control map, where sharply-tuned neurons may provide a high-level “discrete” representation of the sound to utter, based on STG input from the auditory error map, before low-level commands are sent to the articulator velocity and position maps via vPMC. Our broadly-tuned STG units may also be part of the transition from the auditory error map to the feedback control map, providing a lower-level “continuous” population representation of the sound to utter.

Our results further demonstrate that the neuronal population encoding of vowel generation appears to be organized according to a functional representation of a spatial-anatomical axis: tongue height. This axis was shown to have a significant main effect on decoding from speech motor cortex units²². Whether these structured multi-level encoding schemes also exist in other speech areas like Broca’s² and speech motor cortex, and how they contribute to the coordinated production of speech are important open questions. Notwithstanding, the structured encoding observed naturally facilitates high fidelity decoding of volitional speech segments and may have implications for restoring speech faculties in individuals who are completely paralyzed or “locked-in”^23-28.

Methods

Patients and Electophysiology

Eleven patients with pharmacologically resistant epilepsy undergoing invasive monitoring with intracranial depth electrodes to identify the seizure focus for potential surgical treatment²⁹ (9 right handed, 7 females, ages 19-53) participated in a total of 14 recording sessions, each on a different day. Based exclusively on clinical criteria, each patient had 8-12 electrodes for 1-2 weeks, each of which terminated with a set of nine 40-μm platinum–iridium microwires. Their locations were verified by MRI or by computer tomography coregistered to preoperative MRI. Bandpass filtered signals (0.3–3kHz) from these microwires and the sound track were synchronously recorded at 30kHz using a 128-channel acquisition system (Neuroport, Blackrock). Sorted units (WaveClus³⁰, SUMU³¹) recorded in different sessions are treated as different in this study. All studies conformed with the guidelines of the Medical Institutional Review Board at the University of California Los Angeles.

Experimental Paradigms

Patients first listened to isolated auditory cues (beeps) and to another individual uttering the vowel sounds and three syllables (me/lu/ha) following beeps (auditory controls). Then, following an oral instruction, patients uttered the instructed syllable multiple times, each following a randomly-spaced (2-3sec.) beep. Syllables consisted of either monophthongal vowels (a /α/, e /ε/, i /i/, o /o/, u /u/) or a consonant (of: d /d/, g /g/, h /h/, j /d3/, l /l/, m /m/, n /n/, r /ɹ/, s /s/, v /v/) and one of the aforementioned vowels (e.g., da /dα/, de /dε/, di /di/, do /do/, du /du/)¹. For simplicity, this paper employs the English rather than IPA transcription as described above. All sessions were conducted at the patient’s quiet bed-side.

Data Analysis

Of the 716 recorded units, we analyzed 606 that were not responsive during any auditory control (rostral anterior cingulate and adjacent medial orbitofrontal cortex (rAC/MOF): 123 of 156; dorsal and subcollosal anterior cingulate cortex (dsACC): 68/72; entorhinal cortex: 124/138; hippocampus: 103/114; amygdala: 92/106; parahippocampal gyrus: 64/66; superior temporal gyrus: 32/64). The anatomical sub-divisions of the ACC are according to McCormick et al.³² Due to clinical considerations²⁹, no electrode was placed in the primary or pre- motor cortex in this patient population. Each brain region was recorded in at least three subjects. A unit is considered speech-related when the firing rate differs significantly between the baseline ([-1000,0]ms relative to the beep) and the response ([0,200]ms relative to speech onset; paired t-test, p < 0.05, adjusted for False Discovery Rate (FDR)³³ control for multiple units and vowels, q < 0.05, n ranges between 6 and 12 trials depending on the session). For these units, we found the maximal response among the four 100ms bins starting 100ms before speech onset, and computed mean firing rates in a 300ms window around this bin. Tuned units are speech-related units for which the mean firing rate is significantly different between the 5 vowel groups (analysis of variance (ANOVA); F-test, p < 0.05, FDR³³ adjusted, q < 0.05, n between 6 and 12 for each group). Broad, sinusoidally-tuned units are tuned units whose firing rate correlates with: a+bcos(c+i·2π/5) (where i=0,…,4 is the index of the vowel in a,e,i,u,o) with coefficient of determination R² > 0.7 ^[10]. Sharply-tuned units are tuned units for which the mean firing rate in the 3 vowel groups of lowest mean firing rate is the same with high probability (ANOVA; F-test,p > 0.1, n between 6 and 12 for each group). The vowel decoder is a regularized multivariate linear solver, which minimizes ∥x∥_L1 subject to ∥Ax-b∥ _L2 ≤ σ (Basis Pursuit Denoising problem³⁴). It has superior decoding performance and speed relative to neuron-dropping decoders (unpublished results). A contains the feature inputs to the decoder: spike counts of all units in a baseline bin ([-1000,0]ms relative to the beep) and in two 100ms response bins that followed speech onset; b are 5-element binary vectors coding the individual vowels uniquely. All decoding results were 6-fold cross-validated using trials that were not used for decoder training. The decoder was trained on all of the aforementioned features from the training set only, with no selection of the input neurons or their features. Instead, the sparse decoder described above automatically selects task-relevant features by higher weights it allocates to them using the minimal ∥x∥_L1 constraint; task-unrelated features are thus diminished by low weights. Due to the high accuracy in decoding, we randomly dropped 20% of the units (in each cross-validation training) when computing confusion matrices, to increase the amount of confusions and allow the extraction of a meaningful band diagonal structure (except for the STG-only training, Supplementary Figure S4).

The vowels in Fig. 3c were placed on the IPA chart according to the locations previously calculated for American speakers (Ref¹, page 42) and the overlaid connections (blue lines) were inferred by the maximal non-diagonal element for each row and each column of the confusion matrix.