Direct neural coding of speech: Reconsideration of Whalen et al. (2006) (L)

Abstract

Previous brain imaging results indicated that speech perception proceeded independently of the auditory primitives that are the product of primary auditory cortex [Whalen, Benson, Richardson, Swainson, Clark, Lai, Mencl, Fulbright, Constable, and Liberman (2006). J. Acoust. Soc. Am. 119, 575–581]. Recent evidence using electrocorticography [Hamilton, Oganian, Hall, and Chang (2021). Cell 184, 4626–4639] indicates that there is a more direct connection from subcortical regions to cortical speech regions than previous studies had shown. Although the mechanism differs, the Hamilton, Oganian, Hall, and Chang result supports the original conclusion even more strongly: Speech perception does not rely on the analysis of primitives from auditory analysis. Rather, the speech signal is processed as speech from the beginning.

I. INTRODUCTION

Perception of the acoustic speech signal is an important component of human communication, subject to many decades of research. Because it is an acoustic signal, it has been taken as obvious that the objects of speech perception are acoustic (Diehl and Kluender, 1989; Ohala, 1996; Fox et al., 2020). In such an approach, auditory processing in the primary auditory cortex (PAC) is necessary (Obleser et al., 2006; Mesgarani et al., 2008; Steinschneider et al., 2013).

Many behavioral studies challenge the assumption of the primacy of acoustics, indicating instead that perception extracts the articulatory gestures that could have generated the acoustic pattern (Liberman et al., 1967; Fowler, 2006; Whalen, 2019). One extensive study shows that none of the Gestalt principles of perceptual organization that have been proposed for acoustic signals apply to speech perception (Remez et al., 1994).

These theories, and the assumptions about the contribution of PAC to speech perception, have been put to the test in a variety of studies. Two studies are the focus of this Letter to the Editor: one relying on the assumed neural pathway for acoustic information, and the other showing previously undetected pathways. Both indicate that the processing of speech does not rely on primary auditory cues but, instead, on speech as a separate, specialized system.

II. WHALEN ET AL. (2006)

A functional magnetic resonance imaging (fMRI) study was undertaken to determine whether auditory primitives from PAC were necessary for speech perception or not (Whalen et al., 2006). The stimuli were designed to test a range of complexity in both speech and nonspeech signals. Although such correspondences are challenging to equate, the speech signals were, if anything, more complex than the nonspeech ones. Therefore, if the first stage of processing were to take place in PAC, both the speech and nonspeech should have shown greater activation there with increasing complexity.

The results indicated that the speech complexity did not correlate with PAC activation at all; instead, portions of superior temporal gyrus (STG) showed a correlation. This was interpreted as showing that speech is treated in a specialized way from the beginning of the analysis of the acoustic signal. No use of primary auditory features appeared to be used in constructing a speech percept.

Examination of neural pathways available at the time did not indicate any significant direct connections from subcortical regions to STG (e.g., Scott et al., 2000), and such views continue to be published (e.g., Peelle and Wingfield, 2022). It was therefore posited that a form of “post-emption” (Whalen et al., 2006, p. 580) would inform PAC that analysis was not necessary. That is, the speech system would recognize that a part of the acoustic signal was speech and would then “post-empt” that energy from the PAC's analysis. This model was consistent with several behavioral studies showing that after about 50 ms, a nonspeech percept could be established that would then pre-empt its usage as speech (Whalen and Liberman, 1996).

III. HAMILTON ET AL. (2021)

A recent study employed intracranial recordings and stimulation of the human brain to explore auditory dependencies much more directly (Hamilton et al., 2021). PAC is located deep within the lateral sulcus, and it is thus challenging to access. However, in rare cases, high-resolution information functional mapping of this region is required for neurosurgical intervention. Such cases provide an opportunity to record neural responses in PAC to auditory stimuli of various sorts using electrocorticography (ECoG) and to perturb and/or suppress activity in PAC and other auditory regions using direct focal electrocortical stimulation (ECS). The ECoG recordings reported by this study showed that both PAC and regions of STG were rapidly activated during audition, suggesting two pathways process acoustic information in parallel. Furthermore, similarly to the earlier studies of Penfield (Penfield and Roberts, 1959), it is possible to induce localized, reversible (in)activation of neural regions by stimulating one or more electrodes with a small amount of electricity. The researchers did so with PAC and STG. When PAC was stimulated, patients reported hearing strange nonspeech sounds. At the same time, if they were presented with speech stimuli acoustically, they showed no interference at all. When STG was stimulated, speech perception was marginal.

The authors conclude that there are strong indications for direct connections from subcortical auditory regions to STG. While further work is needed to elaborate on the pathways, such a specialization would be far more efficient than the post-emption proposed by Whalen et al. (2006), but as stated earlier, there was little previous evidence for such a pathway. The fact that the speech was not impaired at all when PAC was not involved indicates that this direct pathway contains sufficient information for speech perception without contributions from auditory primitives.

IV. SPECIALIZATION AND COMPETITION

The timing of the processing of speech must be, to some extent, simultaneous with that of nonspeech. That is due to the fact that there is no overt marker of what is speech and what is not. If a signal can be parsed as a speech signal, it is speech; otherwise, not (see Mattingly and Liberman, 1988, p. 785). This is true even when the signal does not appear to come from a human but nonetheless has the same kind of coherence, namely, in sinewave speech (Remez et al., 1981). Therefore, it is likely that all sounds project to both PAC and STG. Exactly how the perceptual system resolves the resulting competition is only partly understood, but the results from the two target papers of this Letter indicate that further effort at tracking the coordination is justified, perhaps with a different perspective than has been taken in the past.

One concrete example of how the competition unfolds is found in duplex perception (Rand, 1974), in which a signal that is heard as nonspeech also contributes to the speech on the opposite ear. When two percepts are attributed to the same signal, it is clear that two systems are active. With dichotic presentation, such doubling of the percept seems rather intuitive, but duplex perception can occur with monaural presentation as well (Whalen and Liberman, 1987). When a small portion of the signal that determines the place of articulation of the perceived stop (in this case, the F3 transition) is low in intensity, only the speech is heard. At higher intensities, the nonspeech aspect of the F3 transition becomes apparent, and its loudness increases as the intensity increases (Xu et al., 1997). The nonspeech system can “capture” the F3 transition if it has a 50 ms head start (Whalen and Liberman, 1996). Thus, the competition between the systems is immediate and intricate.

V. CONCLUSION

Our knowledge of brain organization continues to expand, as new techniques and situations are explored. The results of Hamilton et al. (2021) were not attainable until recently, and thus the evidence for a direct auditory connection to STG was quite difficult to discover. Post-mortem analysis of brain connections does not allow for a determination of the utility of any neurons that might connect different regions, so only a combination of imaging and behavioral results can provide the relevant data. Whalen et al. (2006) was based on the best evidence at the time, but the recent results from Hamilton et al. (2021) undermine those assumptions.

The conclusion of Whalen et al. (2006), however, is made even stronger by the results of Hamilton et al. (2021): Acoustic analysis of speech occurs largely in STG. The analysis of auditory primitives in PAC is not necessary and would, indeed, be counterproductive in many cases. It is still possible that an acoustic representation of speech occurs entirely within STG, rather than depending on the gestures proposed in Whalen et al. Such a duplication of the products of PAC is possible, but there is not an immediately apparent reason for it. In either case, speech signals are processed, at least for the most part, as speech from the first connection with the cortex.

AUTHOR DECLARATIONS

Conflict of Interest

The author reports no conflict of interest.

DATA AVAILABILITY

The data that support the findings of this study are available from the corresponding author upon reasonable request.