Acoustic noise and vision differentially warp the auditory categorization of speech

Abstract

Speech perception requires grouping acoustic information into meaningful linguistic-phonetic units via categorical perception (CP). Beyond shrinking observers' perceptual space, CP might aid degraded speech perception if categories are more resistant to noise than surface acoustic features. Combining audiovisual (AV) cues also enhances speech recognition, particularly in noisy environments. This study investigated the degree to which visual cues from a talker (i.e., mouth movements) aid speech categorization amidst noise interference by measuring participants' identification of clear and noisy speech (0 dB signal-to-noise ratio) presented in auditory-only or combined AV modalities (i.e., A, A+noise, AV, AV+noise conditions). Auditory noise expectedly weakened (i.e., shallower identification slopes) and slowed speech categorization. Interestingly, additional viseme cues largely counteracted noise-related decrements in performance and stabilized classification speeds in both clear and noise conditions suggesting more precise acoustic-phonetic representations with multisensory information. Results are parsimoniously described under a signal detection theory framework and by a reduction (visual cues) and increase (noise) in the precision of perceptual object representation, which were not due to lapses of attention or guessing. Collectively, findings show that (i) mapping sounds to categories aids speech perception in “cocktail party” environments; (ii) visual cues help lattice formation of auditory-phonetic categories to enhance and refine speech identification.

I. INTRODUCTION

In everyday life, we effortlessly combine information from multiple sensory systems to derive a robust unified percept of events. This ability is essential in the context of speech comprehension, whose success requires the many-to-one mapping of continuously varying acoustic signals onto discrete phonetic sound categories (Pisoni, 1973; Harnad, 1987a; Pisoni and Luce, 1987; Liberman and Mattingly, 1989). In the context of speech, this type of categorical perception (CP) is indicated when listeners hear gradually morphed speech sounds as one of only a few discrete phonetic classes, with an abrupt shift in perception near the midpoint of a stimulus continuum (Liberman et al., 1967; Pisoni, 1973; Harnad, 1987b; Pisoni and Luce, 1987; Kewley-Port et al., 1988). Given the rapid rate of speech transmission (∼200 words per minute; Miller et al., 1984), successful comprehension demands that observers process the incoming acoustic signal with maximal efficiency. CP facilitates speech perception by grouping unimportant differences within categories and boosting discriminability between categories, thereby providing the listener a more constrained, manageable perceptual space. Presumably, this “downsampling” process of CP also generates needed perceptual constancy in the face of individual variation along multiple acoustic dimensions (e.g., talker variability) (Prather et al., 2009) or, as tested here, signal degradation (e.g., perceiving speech in noise).

CP also manifests in the visual domain, including the perception of faces (Beale and Keil, 1995), colors (Franklin et al., 2008), and visual speech (O'Sullivan et al., 2017). While there is a substantial literature on CP for single-cue (auditory or visual) contexts, less is known about its role in multisensory conditions such as audiovisual (AV) contexts. Multi-cue integration is necessary in face-to-face communication in which visual articulatory information from a talker's face provides a critical complement to what was said. In these AV contexts, dynamic speech features in auditory and visual channels reflect discrete representations of phonetic-linguistic units (phonemes) and corresponding representations of mouth shapes (visemes) (Peelle and Sommers, 2015). Such integration creates a “visual gain” compared to auditory-only speech, especially when the acoustic signal is degraded (Sumby and Pollack, 1954; MacLeod and Summerfield, 1987; Vatikiotis-Bateson et al., 1998; Ross et al., 2007, Golumbic et al., 2013; Xie et al., 2014).

Visual cues aid comprehension of speech in several ways. Synchronous visible mouth movements can guide comprehension by providing cues to both the timing and content of the acoustic signal (Peelle and Sommers, 2015). Spatial cues such as mouth shape help disambiguate less salient speech sounds, while dynamic articulatory cues help predict upcoming elements in the speech stream (Peelle and Davis, 2012) and allocate attention to target sounds (Carlyon et al., 2001). To varying degrees, facial kinematics and acoustic envelopes are correlated and this cross-modality comodulation can enhance target speech information and improve segregation amidst competing signals (Grant and Bernstein, 2019). In addition, lipreading and coherent visual cues enhance the mere detection of speech in noise (Grant and Seitz, 2000; Grant, 2001; Bernstein et al., 2004; Schwartz et al., 2004).

Sentence-level recognition depends on a variety of cues such as semantic context, lexical frequency, and other indexical cues that can aid speech perception, especially in noise (e.g., Boothroyd and Nittrouer, 1988; Helfer and Freyman, 2009). Moreover, prominent theories of multisensory facilitation typically explain AV speech benefits as an increase in redundancy (e.g., RACE models; Miller and Ulrich, 2003; Colonius and Diederich, 2006) or decrease in cognitive demand (Peelle and Sommers, 2015) offered by combining information from multiple modalities. We explore here an alternate, perhaps more fundamental mechanism to account for AV speech benefits that would not depend on such high-level context, lexical, or cognitive effects. Namely, that vision might alter the underlying categorical (acoustic-phonetic) representations of speech.

Our scientific premise was based on the notion (heretofore untested) that visual cues might “sharpen” the perceptual (category) units of speech itself rather than modulating listening effort or signal information/redundancy, per se. Supporting this hypothesis, visual cues have been shown to influence (alter) speech percepts even when the auditory signal is perfectly clear (McGurk and MacDonald, 1976). Such effects suggest that visual cues shape speech categories and that informational content of the visual signal systematically influences the perceptual identity of speech objects themselves (Massaro and Cohen, 1983; van Wassenhove et al., 2005). Green and Kuhl (1989) theorized that visual and auditory cues of speech contribute complementary phonetic information about place and manner of articulation. They compared participants' CP of a nonvisible voicing feature of auditory stimuli presented in AV and auditory-only conditions and found that AV items yielded perception of a longer voicing boundary relative to the auditory-only speech, suggesting that AV cues for speech are processed together as a global percept.

Presumably, the inherent process of categorizing can be further beneficial to speech perception in degraded (noisy) listening conditions. Phonetic categories (a higher-level code) are thought to be more robust to noise than physical surface features of a stimulus (lower-level sensory code) (Gifford et al., 2014; Helie, 2017; Bidelman et al., 2019). That is, the construction of a perceptual object and natural filtering process of CP might enable category members to “pop out” among a noisy feature space (e.g., Nothdurft, 1991; Perez-Gay et al., 2018; Bidelman et al., 2019). Thus, from a theoretical perspective, the mere process of grouping speech sounds into categories may aid perception of speech in noise. While the benefits of visual cues on degraded speech recognition are well documented (Sumby and Pollack, 1954; MacLeod and Summerfield, 1987; Vatikiotis-Bateson et al., 1998; Ross et al., 2007; Xie et al., 2014), we are unaware of any studies directly assessing how the operation of speech categorization itself—a fundamental mode of perception—is influenced by visual cues of a talker (e.g., phoneme-viseme interactions), particularly as a function of sensory uncertainty (e.g., noise) (cf. Bejjanki et al., 2011). This is surprising given the robust CP observed in the visual domain, including perception of faces (Beale and Keil, 1995) and colors (Franklin et al., 2008). Thus, a novel aspect of the current study is to characterize the extent to which visual information (and noise) interact during the core process of categorization (i.e., the acoustic-phonetic conversion).

Additionally, a large portion of speech research on AV integration examines “conflict situations” that involve testing performance for incongruent speech cues in the auditory and visual modalities, as in McGurk experiments (McGurk and MacDonald, 1976). Such paradigms can produce intersensory biasing, in which listeners develop a propensity to categorize speech based on cues from the other modality (Walden et al., 1990; Bertelson et al., 2003). Indeed, studies have shown that normal listeners categorizing incongruent AV speech rely more on auditory cues whereas cochlear implant users rely more on visual cues (see also Schorr et al., 2005; Desai et al., 2008). Thus, it is difficult to draw conclusions about AV benefits from studies employing such conflict situations that require congruency resolution.

In the present study, we tested the hypothesis that visual cues aid speech perception by sharpening categorical representations for speech, particularly under noise-degraded listening conditions. We measured participants' behavioral categorization as they identified speech stimuli along an acoustic-phonetic /da/ to /ga/ continuum presented in clear or noise-degraded listening conditions. This paradigm is particularly advantageous for assessing multisensory processing because stimulus features change orthogonal to the behavioral percept (Bidelman et al., 2013), and therefore might be more veridical for studying perceptual “binding” vs mere “integration” in speech perception (for review, see Bizley et al., 2016). To assess acoustic-visual interactions in CP, continua were also presented with or without visual cues (i.e., visemes) of the talker. We predicted that mouth movements accompanying the auditory input should further warp (bias) participants' auditory perceptual space to one or the other end of the speech continuum. Comparing the slopes of listeners' psychometric functions across conditions assessed the degree to which noise and visual cues influenced the strength/precision of CP. Previous studies have shown dissociations in the accuracy (%) and speed (i.e., response times, RTs) of listeners' categorization (Binder et al., 2004). These dual properties of behavior might also be supported by different brain regions (e.g., %-correct: auditory cortex; RTs: inferior frontal cortex; Binder et al., 2004; Chang et al., 2010; Bidelman and Lee, 2015), suggesting categorization can be parsed into sensory-perceptual and decision processes (cf. “early- vs late-stage” or “pre- vs post-labelling” models of AV integration; Braida, 1991; Peelle and Sommers, 2015). Thus, we measured both the accuracy and speed of listeners' speech categorization to assess visual and noise-related modulations in behavior and to tease apart these factors in relation to CP. We hypothesized that noise would weaken the categorical representations of speech but that visual cues would partially counteract noise-related decrements in CP. While some work suggests visually presented syllables (visemes) are categorically perceived (Weinholtz and Dias, 2016), visual contributions are limited under normal circumstances. Thus, we predicted visual phonetic cues might become more effective for ambiguous or degraded speech classification (Massaro and Cohen, 1983).

II. METHOD

A. Participants

Fifteen young adults were recruited to the experiment. One participant's data were lost due to a technical error in data logging. Thus, the final sample consisted of fourteen participants [six males, eight females; age: mean = 26.9, standard deviation = 3.0 years]. All exhibited normal hearing sensitivity [i.e., <25 dB hearing level (HL) thresholds, audiometric frequencies]. Each was strongly right-handed (82.4 ± 17.3% laterality index; Oldfield, 1971), had obtained a collegiate level of education, and had normal or corrected-to-normal vision. Musical training is known to modulate categorical processing and speech-in-noise listening abilities (Parbery-Clark et al., 2009; Bidelman et al., 2014; Smayda et al., 2015; Mankel and Bidelman, 2018). Consequently, we required that participants have minimal music training throughout their lifetime (1.9 ± 2.9 years). All received payment for their time and gave written informed consent in compliance with a protocol approved by the University of Memphis Institutional Review Board.

B. Stimuli: AV speech continua

We used a 7-step, stop-consonant /da/ to /ga/ sound continuum (varying in place of articulation) to assess CP for speech [Fig. 1(A)]. Each sound token (Tk) was separated by equidistant steps acoustically yet was perceived categorically from /da/ to /ga/. Stimulus morphing was achieved by altering the F2 formant region in a stepwise fashion using the STRAIGHT software package (Kawahara et al., 2008). We chose a consonant-vowel (CV) continuum because compared to other speech sounds (e.g., vowels), CVs are perceived more categorically (Pisoni, 1973; Altmann et al., 2014) and carry more salient articulatory gestures and visual cues for perception (Moradi et al., 2017). Original video material consisted of a single talker (Talker #6) from the “congruent” set of AV CVs described in Nath and Beauchamp (2012) and Mallick et al. (2015).¹ The total length of each video clip ranged from 1.5 to 2.0 s to start and end each speaker in a neutral, mouth-closed position. The acoustic portion of each video (∼350 ms corresponding to where the talker was opening/closing her mouth) came from the same talker in the video and was temporally centered within each clip.

FIG. 1.

(Color online) Stimuli and task design. (A) Spectrogram of the /da/-/ga/ continuum. Each of the seven acoustic tokens of the morphed speech continuum was overlaid onto a video of either a prototypical /da/ (Tk 1–3) or /ga/ (Tk 4–7) (Nath and Beauchamp, 2012; Mallick et al., 2015). We intended to further warp (bias) participants' auditory perceptual space to one or the other end of the speech continuum with visual cues of the talker. (B) Single trial time course. After a brief orienting screen (+), participants rapidly identified whether they perceived each audio(visual) token as a “da” or “ga” via computer keypress. Speech stimuli were presented in four different blocks which varied in the number of sensory cues from the talker and the presence/absence of acoustic noise interference: A, A+noise, AV, AV+noise.

Though it would have been a desirable, creating a morphed video channel from /da/ to /ga/ is technically challenging given the time-varying nature of images on the screen and integration of the sound channel. Therefore, to investigate if phonetic visual cues (visemes²) enhance the salience of speech CP, we superimposed the morphed acoustic continuum (each of the seven steps) on a prototypical “da” or “ga” video production. For the AV conditions, this resulted in an overlay of audio tokens 1–3 onto the “da” video and tokens 4–7 onto the “ga” video [Fig. 1(A)]. Thus, each half of the acoustic-phonetic /da/-/ga/continuum was latticed with either “da” (Tk 1–3) or “ga” (Tk 4–7) visual cues.³ That is, we intended to further warp participants' auditory perceptual space to one or the other end of the continuum via biasing visual cues of the talker. Mixing of the auditory and visual channels was achieved using FFmpeg software and custom routines coded in matlab 2013 b (The MathWorks, Inc.).

In addition to these AV conditions, we examined participants' speech categorization for identical tokens where the video channel was absent from the screen [i.e., auditory-only (A) condition]. To further investigate the impact of visual cues on speech categorization, we also constructed a similar AV continuum by partially masking the sound channel with multi-talker babble noise [signal-to-noise ratio (SNR) = 0 dB SNR] (Killion et al., 2004). This SNR was chosen on the basis of previous studies showing behavioral performance is most dynamic under challenging (0 dB or negative) SNRs (e.g., Sumby and Pollack, 1954; Xie et al., 2014; Reetzke et al., 2016).

In total, the stimulus set consisted of four /da/-/ga/ continua that varied in AV cues and noise degradation to the sound channel: A, A+noise, AV, AV+noise. N = 5 of the participants also took part in a V-only condition where the speech continuum was presented with a muted sound channel. This pilot control allowed us to assess whether participants could categorize speech sounds based solely on visual cues of the talker. While listeners could identify V-only speech above chance, performance was highly variable. Given that V-only speech did not produce consistent/reliable identification and previous findings that visual speech offers impoverished phonetic detail in isolation (Walden et al., 1977; Kuhl and Meltzoff, 1988; Bernstein and Liebenthal, 2014), we did not consider this condition further.

C. Task procedure

Participants sat in a double-walled sound attenuating chamber (Industrial Acoustics, Inc.) ∼90 cm from a computer monitor. Where necessary, participants wore corrective lenses/contacts for the experiment. All confirmed the screen and visual stimuli were clearly visible before initiating the experiment. matlab was used as a driving engine to present AV stimuli via the VLC media player as well as collect response data. Stimuli appeared at the center of the screen on a black background, subtending a 7.0° visual angle (Samsung SyncMaster S24B350HL; nominal 75 Hz refresh rate). High-fidelity circumaural headphones (Sennheiser HD 280 Pro) delivered the auditory channel binaurally at a comfortable level [80 dB sound pressure level (SPL)].

Participants heard 210 trials of each individual CV (30/token) presented in the different AV stimulus conditions (four separate blocks). Blocking was used to minimize adding unnecessary cognitive effort or distractibility to the task that would have occurred from trial-to-trial switching between modalities. Block order was counterbalanced across participants according to a Latin square sequence (Bradley, 1958). On each trial, participants labeled the perceived speech token with a binary response via the computer keyboard (“da” or “ga”). They were encouraged to respond as quickly and accurately as possible. Both percent identification and response times (RTs) were logged. Breaks were allowed between blocks to avoid fatigue.

D. Data analyses

For each stimulus condition, we measured the steepness of participants' psychometric function as the slope of each curve where it straddled the CP boundary (i.e., slope = [PC_Tk3 – PC_Tk5]/2, where PC_n is the identification score at token n). Larger slopes reflect steeper psychometric functions and hence, stronger CP (Xu et al., 2006; Bidelman and Lee, 2015; Bidelman and Walker, 2017).We measured the location of the CP boundary as the point (token number) along the continuum where the psychometric functions crossed, measured via the matlab function InterX.⁴ Comparing these metrics between AV and noise conditions assessed possible differences in the location and “steepness” (i.e., rate of change) of the perceptual boundary as a function of AV context and noise interference.

Behavioral RTs for speech labeling speeds were computed as participants' median response latency across trials for a given condition. We excluded outliers (RTs outside 250–6000 ms) from further analysis since these reflect fast guesses and lapses of attention (e.g., Bidelman and Walker, 2017).

Psychometric slopes were analyzed using a mixed-model analysis of variance (ANOVA) (subject = random factor) with fixed effects of SNR (two levels: clean, noise) and modality (two levels: audio, audiovisual) (PROC GLIMMIX, sas^® 9.4). RT data were analyzed with fixed effects SNR, modality, and token (seven levels: Tk 1–7). False discovery rate (FDR) was used to adjust for multiple comparisons (Benjamini and Hochberg, 1995). Effects sizes are reported as partial-eta squares (${\eta }_p^2$) and mean differences for omnibus ANOVAs and post hoc contrasts, respectively.

III. RESULTS

A. Psychometric identification functions

Figure 2 shows psychometric identification functions for the different AV conditions. An ANOVA revealed a significant SNR × modality interaction on psychometric slopes [F_{1, 39} = 7.50, p = 0.0092; ${\eta }_p^2=$ 0.16]. FDR-corrected paired contrasts revealed that noise weakened CP for auditory stimuli (A vs A+noise: t₁₃ = 4.78, p = 0.00145; mean difference = 19.7%) [Fig. 2(B)]. AV speech was also perceived more categorically than A speech [t₁₃ = 2.51, p = 0.035; mean difference = 5.2%], suggesting that multisensory cues enhance CP. The AV and AV+noise did not differ [t₁₃ = 1.44, p = 0.17; mean difference = 4.7%], implying that CP is robust for multisensory speech even in the presence of noise. Lastly, psychometric slopes were sharper for the AV+noise compared to the A+noise condition [t₁₃ = 4.17, p = 0.0011; mean difference = 20.1%], suggesting that the addition of visual cues helped counteract the negative effects of noise on auditory speech categorization.

FIG. 2.

(Color online) Perceptual identification for AV speech with and without noise interference. (A) Psychometric functions show an abrupt shift in perception when classifying speech indicative of discrete perception (i.e., CP). Degree of categorization varies depending on the sensory modality and quality of cues that are available. Note the two curves within each panel are mirror images since the categorization task is a binary decision. (B) Psychometric function slopes. Sharper identification curves are observed for clean A and AV speech. Acoustic noise weakens CP for speech (A vs A+noise), but this decrement is counteracted by the aid of visual cues of the talker (AV vs AV+noise). (C) Location (cross-over) of the CP boundary. Participants show slight bias to respond “ga” in the A+noise condition but otherwise location varies little across stimulus conditions. Shading and error bars = ±1 standard error of the mean (s.e.m.).

Figure 3 shows an alternate presentation of these data, plotting each participant's psychometric slope for one condition vs another for the four major stimulus contrasts of interest. This visualization highlights the relative improvement or decrement in CP with added visual cues and noise, respectively. Points in the upper left half of each plot (above the diagonal) show an improvement in categorization for the ordinate relative to the abscissa condition. We found that a majority of participants showed stronger CP when categorizing clean compared to noisy speech (panel A), again confirming that noise weakened phonetic representations of speech. Similarly, stronger CP (steeper identification functions) for AV compared to A speech (panel B) confirms that visual cues helped strengthen phonetic categories. Last, visual cues enhanced categorization for degraded speech compared to the degraded auditory channel alone (panel D).

FIG. 3.

(Color online) Multisensory cues enhance the CP of clear and especially noise-degraded speech. Individual points show each participant's psychometric slope for two different stimulus continua plotted against one another. Errorbars (±1 s.e.m.) show variance around the group average centroid. Points in the upper left half of each plot (above the diagonal) show an improvement in CP for that ordinate relative to the abscissa condition. A majority of participants show stronger CP when categorizing (A) clean compared to noisy speech sounds, (B) AV compared to A speech, and (D) AV+noise compared to A+noise speech. (C) AV and AV+noise speech yeild similar slopes.

While the location of the perceptual boundary was largely centered across conditions [Fig. 2(C)], it did shift depending on both noise and multisensory cues [SNR × modality interaction: F_{1, 39} = 4.87, p = 0.033 $;\hspace{0.17em}{\eta }_p^2=$ 0.11]. In particular, clean AV speech produced a leftward shift in identification curves (cross-over = 3.5; t-test against perceptual boundary at Tk 4; t₁₃ =−2.43, p = 0.029), indicating a small but measurable bias toward producing more frequent “ga” responses across a wider range of the continuum. However, this might be expected given that “ga” visemes were more prevalent across the auditory perceptual continuum than “da” videos. Contrastively, A+noise yielded a rightward shift in the psychometric function (cross-over location = 4.7; t₁₃ = 2.50, p = 0.026) suggesting a bias to more frequently respond “da” in noisy listening conditions. We note that this noise-related effect occurred even though “ga” videos were overall more prevalent across continua [see Fig. 1(A)]. This latter bias effect may reflect top-down influences of lexical knowledge because /da/ has a higher frequency of occurrence than /ga/ in spoken language (Denes, 1963). All other stimulus conditions produced symmetric psychometric functions (ps > 0.09).

B. RTs

Behavioral RTs, reflecting the speed of participants' categorization, are shown in Fig. 4. RTs are plotted relative to the average RT across all conditions to highlight differential changes in categorization speed with noise and visual cues. RTs showed main effects of noise [F_1,351 = 137.96, p < 0.0001; $\hspace{0.17em}{\eta }_p^2=$ 0.28] and modality [F_1,351 = 5.55, p = 0.019$;\hspace{0.17em}{\eta }_p^2=$ 0.016]. On average, RTs were faster when classifying clean compared to noise-degraded speech (A vs A+noise: p < 0.0001; mean difference = 318 ms). Similarly, clean AV speech elicited faster RTs than degraded AV speech (AV vs AV + noise: p < 0.0001; mean difference = 403 ms). Participants were equally fast at categorizing clear A and AV speech (p = 0.49; mean difference = 30 ms). However, in noise, they were faster at classifying A+noise vs AV+noise speech (p = 0.0086; mean diff. = 114 ms). While these findings reveal a prominent effect of noise on the speed of categorical processing, the relative pattern of RTs is often more meaningful: CP is characterized by a slowing in response speed near the ambiguous midpoint of the continuum (Pisoni and Tash, 1974; Bidelman et al., 2013; Bidelman et al., 2014; Bidelman and Walker, 2017; Reetzke et al., 2018). To assess this effect, we conducted a priori contrasts of RTs on the perceptual boundary (Tk 4) vs others along the continuum [mean (Tk 1,2,3,5,6,7)]. This confirmed CP for A speech [t₁₃ = −2.51, p = 0.026; mean diff. (Tk₁₂₃₅₆₇ vs Tk₄) = 81.5 ms]. RTs to AV speech similarly showed the categorical (inverted-V) pattern [t₁₃ = −3.16, p = 0.0075; mean difference = 81.3 ms]. A categorical RT effect was not observed for A+noise [t₁₃ = 0.45, p = 0.66] nor AV+noise [t₁₃ = −0.42, p = 0.68; mean diff. = 12.3 ms], further suggesting that noise weakened categorical decisions in these degraded stimulus conditions.

FIG. 4.

(Color online) Speech classification speeds [response times (RTs)] vary with sensory modality and clarity of the speech signal. RTs are plotted relative to the mean RT across conditions (dotted line) to highlight the differential in speed of categorization judgments with noise and visual cues. Clean A and AV speech produce the fastest labeling speeds. Participants are 200–300 ms slower at categorizing A and AV speech amidst noise. Only clear A and AV speech produce a categorical pattern of RTs (*p < 0.05) where ambiguous speech tokens elicit slower decisions than tokens with clear phonetic categories [i.e., contrast: Tk 4 vs mean (Tk 1,2,3,5,6,7)] (Pisoni and Tash, 1974; Bidelman and Walker, 2017). errorbars = ±1 s.e.m.

Even in the absence of significant differences in central tendency, RTs can differ in terms of intrasubject variability (Bernstein et al., 2014; Bidelman et al., 2017). Reduced RT variability in certain conditions might also reflect improved cognitive processing (cf. Strauss et al., 2002). Visual inspection of RT variance suggested more restricted, less variable response speeds for multisensory AV stimuli (cf. error bars for A vs AV). Pooling across tokens, formal tests of equal variance (two-sample F-test) revealed that RT dispersion was indeed smaller when categorizing AV compared to the A speech for both the clean [F_13,13 = 7.72, p = 0.0008] and noise-degraded [F_13,13 = 10.46, p = 0.0002] conditions.

C. Signal detection theory (SDT) modeling

To better understand multisensory and noise effects on CP [Fig. 2(A)], we modeled our empirical data using SDT (e.g., Rozsypal et al., 1985; Braida, 1991; Getz et al., 2017) [Fig. 5(A)]. We estimated different properties of each participant's psychometric functions using Bayesian inference via the psignifit toolbox (Schütt et al., 2016). This allowed us to measure individual lapse (λ) and guess (γ) rates from their identification data. Lapse rate (λ) is computed as the difference between the upper asymptote of the psychometric function and 100% and reflects the probability of an “incorrect” response at infinitely high stimulus levels [i.e., responding “da” for Tk 7; see Fig. 2(A)]. Guess rate (γ) is defined as the difference between the lower asymptote and 0% and reflects the probability of a “correct” response at infinitely low stimulus levels [i.e., responding “ga” for Tk 1; see Fig. 2(A)]. For an ideal observer λ = 0 and γ = 0. λ and γ were measured from each participant's individual psychometric function per AV stimulus condition [see Fig. 5(A)].⁵

FIG. 5.

(Color online) Signal detection model of AV and noise effects in speech CP. (A) Schematic identification curves illustrating definitions of lapse and guess rates from the psychometric function. (B) Lapse and (C) guess rates during CP across stimulus conditions. (D) SDT framework for understanding noise and AV effects in CP. (left) Observers' responses are modeled along a perceptual decision axis. In a binary classification task, the probability of responding one or another stimulus class (i.e., /da/ or /ga/) is modeled as two Gaussians. μ_da and μ_ga represent the means of the /da/ and /ga/ distributions; σ is their widths, reflecting response variability. An observer responds “ga” if the signal “energy” falls above the decision criterion (dotted line) and “da” below. (right) Integrating either probability curve results in a cumulative density function, modeling observers' psychometric functions. In an SDT framework, changes in the slope of observers' psychometric functions with multisensory cues and noise [Fig. 2(A)] are well modeled as changes in response variance σ². Reduced height of the psychometric function in the AV+noise condition can be attributed to lapses of attention (see panel B, “AV+noise”), which prevent full unity at the asymptotic end of the curve (Schütt et al., 2016). errorbars = ± 1 s.e.m., **p < 0.01.

An ANOVA revealed that lapse rates depended on stimulus condition [F_3,39 = 5.42, p = 0.0032;$\hspace{0.17em}{\eta }_p^2=$ 0.29] [Fig. 5(B)]. However, this effect was solely attributable to more lapses in the AV+noise compared to A+noise condition (p = 0.0016), paralleling the RT effect between these conditions (Fig. 4). No other pairwise comparisons differed in lapse rate (ps > 0.09–1.0). Similarly, guess rates were invariant across stimuli [F_3,39 = 1.85, p = 0.154;$\hspace{0.17em}{\eta }_p^2=$ 0.12] [Fig. 5(C)]. Given that lapse and guess rate parameters were stable across stimuli, these data indicate that while visual cues and noise modulated the degree of CP for speech (Fig. 2), those effects were not driven by obvious differences in lapses of attention or guessing, per se (Shen and Richards, 2012; Schütt et al., 2016).

In an SDT framework, observers' responses can be modeled along a perceptual decision axis where the probability of responding one or another stimulus class (“da” or “ga”) follows Gaussian normal distributions of the form $y=e^{-{(x-\mu )}^2/2{\sigma }^2}$, where μ_da and μ_ga represent the means of the /da/ and /ga/ normal curves and σ, their width, reflecting variance due to the probabilistic nature of decision. An observer responds “ga” if the signal energy falls above the decision criterion (dotted line) and “da” below. The degree of categorization can then be expressed in terms of sensitivity, d′ = (μ_da – μ_ga)/σ (Geschneider, 1997; p. 118). Theoretically, d′ represents the strength of perceptual difference between /da/ and /ga/ classes. If we assume that the distance between internal representations (μ_da, μ_ga) for each phonetic class are fixed (e.g., because CV phonemes are overlearned sounds), more or less precise classification across conditions must result from changes in response variance σ². Integrating the Gaussian functions results in a cumulative density function, mirroring observers' psychometric functions (Fig. 5, right). Our empirical findings across AV conditions [Fig. 2(A)] are well-modeled as changes in response variance σ². We attribute this reduction in σ² as a sharpening of speech categories.

IV. DISCUSSION

A key aspect of speech comprehension is the ability to categorize variable acoustic input into discrete phonetic units. How different sensory modalities influence and potentially enhance this ability is important for understanding human speech comprehension. By measuring participants' CP for speech, we were able to assess whether the availability of additional sensory cues and the quality of the auditory signal influenced the degree of categorical speech processing. A primary finding was that noise and visual cues exerted opposite effects on CP, with noise eliciting weaker CP and visual cues eliciting stronger CP. Thus, noise appears to blur categorization whereas visual cues help compensate for the compromised auditory modality. Our results support the notion that observers integrate information from multiple sensory domains to categorize signals, especially those which lack sensory precision as in the case of noise degradation (Hélie, 2017; Bidelman et al., 2019). Additionally, we extend previous findings demonstrating that visual cues enhance speech comprehension in noise (Sumby and Pollack, 1954; MacLeod and Summerfield, 1987; Vatikiotis-Bateson et al., 1998; Ross et al., 2007; Golumbic et al., 2013; Xie et al., 2014) by showing these multisensory benefits extend to the level of individual/isolated phonetic speech units and the fundamental process of CP (cf. Massaro and Cohen, 1983).

We found participants exhibited weaker CP for noisy relative to clear speech, as evidenced by flatter identification curves and slower RTs. While clear AV and A speech slowed responses to more ambiguous syllables near the midpoint of the continuum, this hallmark of CP was not elicited for the two degraded continua, implying weaker categorical processing in those latter conditions. We also found an enhancement in CP for AV compared to A speech [e.g., Fig. 3(B)]. This suggests visual cues helped lattice internalized acoustic-phonetic representations, allowing the formation of more well-formed (sharper) speech categories. Moreover, while the overall speed of access to phonetic labels (i.e., average RTs) was similar with (AV) and without (A) viseme information of the talker, visual cues reduced the overall variability in participants' response speeds for both clean and noise-degraded speech (Fig. 4). This reduction in decision variance is consistent with the idea that multisensory cues provide more precise access to speech categories post perceptually.

Overall, we found that AV benefits in CP were more prominent for perceptual identification (%) compared to RT data. This suggests a differential effect of multisensory cues on “early vs late” (Peelle and Sommers, 2015) or “pre- vs post-labeling” (Braida, 1991) stages of categorization. Similar dissociations in behavioral identification and response timing have been noted in previous neuroimaging studies examining speeded speech labeling tasks (Binder et al., 2004). For example, in conditions with highly experienced listeners (Bidelman and Lee, 2015) or overlearned stimuli (Binder et al., 2004; Chang et al., 2010), early auditory cortex is sufficient to generate discrete neural representations that code discrete acoustic-phonetic categories. However, the decision process, as indexed by the speed of listeners' categorical judgments (i.e., RTs) are largely determined by activation in inferior frontal brain regions (Binder et al., 2004). Other animal (Bizley and Cohen, 2013) and human studies (Bidelman and Howell, 2016; Bidelman et al., 2018) have shown that functional interplay between frontal and superior temporal areas is necessary for robust speech recognition and neural coding within this network varies with signal clarity (SNR), intelligibility, and linguistic experience (Adank et al., 2012; Scott and McGettigan, 2013; Bidelman and Dexter, 2015; Bidelman and Howell, 2016; Alain et al., 2018; Bidelman et al., 2018). While the purely behavioral nature of our data cannot adjudicate brain mechanisms, future neuroimaging experiments could address the neural underpinnings, temporal dynamics, and multisensory benefits in CP.

Noise manipulations revealed that acoustic interference weakened speech identification to the point where participants heard continua in a near continuous rather than categorical manner. This suggests a robust (perhaps expected) effect of signal clarity on the formation of categorical percepts; noise in the sensory input blurs acoustic-phonetic mapping inhibiting a strong match between the external signals and internalized memory templates. Nevertheless, while noise compromised categorical representations for speech, we found that visual cues counteract these behavioral disadvantages (at least for the moderate SNR tested here). Interestingly, we find that the influence of visemes on auditory categorical processing is also larger when the speech signal is degraded. That is, the benefits of AV integration on CP appear stronger for degraded relative to clear speech [cf. Figs. 3(D) vs 3(B)]. Thus, in addition to providing useful groupings and perceptual constancy of sensory space (Prather et al., 2009), our findings reveal another important benefit of CP: building phonetic categories (a higher-level discrete code) is more robust to noise than the physical surface features of a stimulus (lower-level sensory code) (cf. Gifford et al., 2014; Hélie, 2017; Bidelman et al., 2019). Consequently, our data imply that the mere process of grouping speech sounds into categories seems to aid speech comprehension in adverse listening conditions. Future studies should test whether AV effects on CP observed here for CVs also applies more broadly to CP for other speech (and nonspeech) stimuli (e.g., vowels which carry more obvious visual cues).

Our empirical multisensory and noise effects on CP are parsimoniously described via concepts of SDT (e.g., Rozsypal et al., 1985; Braida, 1991; Getz et al., 2017). Under an SDT framework, changes to internal response variability of the observer account for the flattening of the psychometric function with additive noise (A+noise) and conversely, (re-) sharpening with visual cues (AV speech): external noise increases σ, leading to wider spread identification curves whereas visual cues reduce σ and steepen the psychometric function (see also Gifford et al., 2014). The subtle reduction in height of the psychometric function in AV+noise (see Fig. 2) can be attributed to attentional lapses in this condition [see Fig. 5(B)], which prevent full unity at the asymptotic end of the identification curve (Shen and Richards, 2012; Schütt et al., 2016). Although accuracy and decision speed (RT) are dissociable in categorization tasks (Binder et al., 2004), our RT data further support these notions. Delayed response speeds in the AV+noise condition also suggest attentional lapses, at least in that condition [cf. Figs. 4 and 5(B)]. Indeed, RTs were correlated with attentional lapses but only for AV+noise speech (r = 0.71, p = 0.0043). Still, the fact that lapse effects were only limited to the difficult AV+noise condition—requiring parsing of speech from noise and the integration of phoneme and viseme information—suggests effects in the other conditions are probably not attributable to attention or post-labeling decision stages, per se (cf. Braida, 1991), but a sharpening (de-sharpening) of internalized categories. While SDT does not explicitly account for the speed of an observer's decision (only % identification/accuracy), the reduction in RT variability with visual cues (and increased variability with noise absent any V cues) suggests that these stimulus factors modulate observers' response variability (σ) during CP in opposite directions—a reflection of changes in the precision of the underlying perceptual object(s).

We interpret multisensory effects on CP to reflect a sharpening of internalized speech categories, consistent with notions that auditory and visual components of speech are fused into a single global percept (Green and Kuhl, 1989). Still, an alternate interpretation of our data relates to a reduction in stimulus uncertainty (Gifford et al., 2014); visual cues add another source of information that might reduce uncertainty in making categorical judgments. Similarly, vision might provide an added “gain” of sensory information relative to A-only speech (Sumby and Pollack, 1954; MacLeod and Summerfield, 1987; Vatikiotis-Bateson et al., 1998; Ross et al., 2007; Xie et al., 2014). Under this interpretation, the observed sharpening of the CP boundary (Fig. 2) might be explained by a reduction in uncertainty due to AV facilitation.

In this vein, comparing AV to AV+noise responses for Tk 4 revealed that classification was more reliably “ga” in the presence of noise. This implies a Bayesian-like integration (Deneve and Pouget, 2004). When noise is absent, A cues still dominate the behavioral decision. However, when noise is added, the reliability of A cues is severely diminished and so V cues take over as they offer more reliable inference on which to form AV speech percepts (Bidelman et al., in press). Indeed, studies have shown that in situations where visual cues are deemed unreliable (e.g., noise, sensory impairments), sound can trump vision to maintain robust perception (Alais and Burr, 2004; Narinesingh et al., 2015; Myers et al., 2017). Similar cue (re)weighting has been observed with analogous degradation to the auditory channel (Bidelman et al., 2019a).

Still, evidence against a strict cue weighting explanation is the fact that RTs for clean A and AV speech were identical (Fig. 4), in contrast to the faster response speeds that would be expected by multisensory facilitation (e.g., RACE and redundant signal models; Miller and Ulrich, 2003; Colonius and Diederich, 2006). Similarly, the slower RTs for AV+noise vs A+noise speech (Fig. 4) also runs counter to a strict RACE framework, where multisensory speech would be expected to facilitate response speeds, even in noise. These data reveal a functional distinction in multisensory processing for isolated speech categorization (present study) that is perhaps fundamentally distinct from sentence-level recognition, where semantic context, lexical frequency, and other indexical cues can aid speech-in-noise perception (e.g., Boothroyd and Nittrouer, 1988; Helfer and Freyman, 2009). Instead, we attribute the overall slower RTs for AV+noise to reflect cognitive interference in attempting to reconcile the minimal categorical cues supplied by our CV visemes with the more salient ones from the auditory-phonetic input (Files et al., 2015) coupled with the inherent listening effort associated with degraded-speech perception tasks (Picou et al., 2016; Bidelman and Yellamsetty, 2017; Bidelman et al., in press). Under this interpretation, the relative invariance in CP slope but counterintuitive slower RT in AV+noise could reflect a dissociable effect in pre- vs post-labeling aspects of CP (Braida, 1991); V cues help lattice (i.e., sharpen) the categorical object at a pre-perceptual stage resulting in robust identification [Fig. 2(A)] but noise impairs the speed of access to this representation post-perceptually, as reflect in the delayed RTs (Fig. 4).

The notion of AV integration itself has been questioned since animal work has shown visual stimuli modulate cortical responses in auditory cortical fields independently of visual stimulus category (Kayser et al., 2008). Calvert et al. (1997) even suggested that activation of primary auditory cortex during lip reading implies visual cues influence perception even before speech sounds are categorized into phonemes (for review, see Bernstein and Liebenthal, 2014). Visual cues precede the corresponding auditory signal, so they may also serve a predictive role in facilitating speech processing, particularly in noisy situations (Golumbic et al., 2013). Unfortunately, behavior alone cannot delineate accounts of our data based on reduction in stimulus uncertainly due to additional (AV) information or true sharpening of the category—although these explanations need not be mutually exclusive. Ongoing neuroimaging experiments are currently underway in our laboratory to adjudicate these competing mechanisms (e.g., Bidelman et al., 2019).

V. CONCLUSIONS

In sum, our findings support a view of multimodal integration in which observers use available cues from multiple sensory modalities to perceptually categorize speech, especially when the acoustic signal is impoverished. While the present study was not designed to provide a mechanistic account of how visual and auditory cues may combine or interact in CP, our results provide important evidence that the perceptual and categorical organization of speech is not a unimodal process. Visual cues reduce the precision of categorical representations leading to sharper phonetic identification whereas noise exerts the opposite pattern, increasing variability in CP and leading to less precise speech categories. Though controversial, dyslexia has been linked to poorer CP (Messaoud-Galusi et al., 2011; Noordenbos and Serniclaes, 2015; Hakvoort et al., 2016; Zoubrinetzky et al., 2016)—a deficit which may be exacerbated in noise (Calcus et al., 2016). Future work could focus on identifying possible contributions that information from visual and additional sensory modalities make to auditory speech comprehension in normal and disordered populations and in other complex perceptual scenarios (e.g., reverberation, visual noise interference).