Audiovisual synchrony in left-hemisphere brain-lesioned individuals with aphasia

Abstract

We investigated the ability of 40 left-hemisphere brain-lesioned individuals with various diagnoses of aphasia to temporally synchronize the audio of a spoken word to its congruent video using a maximum-likelihood adaptive psychophysical procedure. We found a statistically significant effect of aphasia type, not explained by lesion volume, on measures of AV synchrony. Brain-lesioned individuals with no symptoms of aphasia, and those with conduction aphasia performed on the synchrony task more similarly to age-matched neurotypical controls, whereas those with anomic aphasia performed the poorest. In addition, we examined the correlation between this ability and AV integration (fusion) and observed a significant correlation between measures of AV synchrony and fusion. A region-of-interest analysis of stroke lesion maps showed that damage to the left posterior temporal regions adversely affected audiovisual processing, although whole-brain univariate lesion-symptom mapping analyses did not yield any significant results. These findings contribute to a better understanding of the functional relationship between different audiovisual processes in multimodal integration and their underlying cortical networks in the human brain.

Introduction:

The perception of audiovisual (AV) synchrony has been extensively studied in psychophysics. Several studies have reported that, on average, the audio signal (e.g., speech) must be delayed relative to the visual (lip movements) by approximately 50 ms for the two to be perceived synchronous. Across studies this delay has been reported to range from a few to over 100 ms depending on experimental paradigm and the nature, complexity, and ecological validity of stimuli (Dinnerstein and Zlotogura, 1968; Dixon and Spitz, 1980; Jaskowski et al., 1990; Keetels and Vroomen, 2005; Spence et al., 2003; Vatakis and Spence, 2006a,b; Zampini et al., 2003; van Eijk et al., 2008; Venezia et al., 2016). Explanations for this phenomenon range from a need to account for the physical transmission time of sound (which travels approximately 1 ft per ms), to faster neural transduction of auditory signals from the sensory periphery to the cortex relative to visual stimuli (10 vs 50ms), necessitating a visual “head start” (King and Palmer, 1985; Pöppel, 1988; Arrighi et al., 2006; van Eijk et al., 2008; Vroomen and Keetels, 2010).

More recent studies have implicated critical contributions from higher-order cognitive factors, including attentional mechanisms, adaptation, object formation, predictive processes, and error reduction by the brain (Vroomen and Keetels, 2010; Venezia at al., 2016; Mattes and Ulrich, 1998; Schneider and Bavelier, 2003; Shore and Spence, 2005; Shore et al., 2001; Stelmach and Herdman, 1991; Zampini et al., 2005; Stekelenburg and Vroomen, 2007; van Eijk et al., 2008; Friston and Kiebel, 2009). Venezia et al. (2016) for example, using a paradigm that investigated the joint effects of AV synchrony and fusion (integration), showed that earlier arriving visual speech information may generate implicit predictive cues to both the content and timing of later-arriving auditory speech signals, not unlike how the visual trajectory of a bouncing ball (a classic paradigm in studies of audiovisual synchrony) predicts the timing of the accompanying sound at the moment of impact. Other factors include stimulus intensity (faster processing of more intense stimuli; Sanford, 1971; Jaśkowski and Verleger, 2000) and individual differences (Mollon and Perkins, 1996; Stone et al., 2001). For reviews see Vroomen and Keetels (2010), Venezia et al. (2016), and Zhou et al. (2020).

Fewer studies have investigated the neural bases of AV synchrony for speech, but neuroimaging and stroke-induced lesion-symptom mapping studies have provided important information on the cortical mechanisms involved. The primary finding from these studies is that the superior temporal cortex is critical both to AV synchrony processing (Merchant et al., 2012; Zhou et al., 2023) and to multisensory integration (Nath and Beauchamp, 2012; Erickson et al., 2014; Hickok et al., 2018) and, importantly, that not all AV processes are correlated with one another, either psychophysically or neurally (Hickok et al., 2018; Nath and Beauchamp, 2012; Erickson et al., 2014; Stevenson et al., 2011; Zhou et al., 2020). In a large-scale study of 100 brain-lesioned individuals, our research group, in collaboration with six other institutions conducted an extensive and detailed investigation of psychophysical and cortical audiovisual processing and found that different AV processes engage functionally and anatomically different cortical regions (Hickok et al., 2018). We found no correlation between two behavioral measures of AV processing: AV advantage (benefits in intelligibility from adding visual to auditory speech) and AV integration (changes in speech-token identity resulting from AV fusion in a McGurk-MacDonald (1976) task). Furthermore, our study found that the posterior superior temporal regions of the brain are involved in AV integration, though not AV advantage. We concluded that: 1) not all AV integration tasks tap into the same generalized AV neural processes, and 2) that different AV measures are detecting different combinations of abilities.

The current study expands on these findings from our group in two ways. First, in a new population of 40 left-hemisphere brain-lesioned individuals we investigated the relationship between two other AV variables, synchrony and fusion (as contrasted to advantage vs. fusion). The results were informative and distinct from our earlier findings. Second, we examined how aphasia diagnosis resulting from stroke-induced brain lesions affects AV synchrony and fusion. We found that there was a statistically significant effect of aphasia type, not explained by lesion volume, on measures of AV synchrony. Brain-lesioned individuals with no symptoms of aphasia, and those with conduction aphasia performed more similarly to age-matched controls, whereas those with anomic aphasia performed the poorest (or most divergent from the control group). Furthermore, we found a statistically significant correlation between AV fusion rate and the ability to perform the AV synchrony task (unlike the orthogonal relation between AV advantage and fusion). Finally, consistent with our earlier work (Hickok et al., 2018) we found that damage to superior temporal areas adversely affects AV processing (especially fusion), with the additional novel finding that posterior and ventral temporal lesions may be disruptive to AV synchrony.

Materials & Methods:

Participants

Two categories of participants across four institutions took part in this experiment. The participant categories comprised 40 individuals with stroke-induced left-hemisphere brain lesions (mean age 56.1 years, σ=12.2, age range 29-76) and 15 older adults with no neurological disease (mean age 60.1 years, σ=7.5, age range 48-72). The brain-lesioned population was recruited by and participated in the experiment at Arizona State University (n=12), the University of South Carolina (n=20), and the Medical College of Wisconsin (n=8). The older adult control group were recruited by and participated in the experiment at the University of California, Irvine. All lesion and control participants completed audiometric tests at their respective institutions. In general, these participants showed some characteristic hearing loss at higher frequencies above 2 kHz.¹ All participants had normal or corrected-to-normal vision. Participants signed written informed-consent forms approved at their respective institutions’ IRB.

Stroke participants were included in the present study based on the following criteria: (i) a chronic focal (6 months or more post-onset) lesion due to a stroke in the cerebrum (ii) no significant anatomical abnormalities other than the signature lesion of their vascular event (iii) an absence of a history of psychological or neurological disease other than stroke, (iv) native English speaker, (v) right handed pre-stroke, and (vi) ability to follow task instructions. Stroke participants underwent 3T or 1.5T MRI scanning at their respective testing site, including T1 and T2 MRIs with 1mm³ resolution. Lesion mapping was performed using trained manual lesion demarcation and then enantiomorphic normalization as described in Rogalsky et al. (2022). Scans confirmed damage to the left cerebral hemisphere in all stroke participants. Figure 1 shows overlap maps (axial slices) of damaged areas in the stroke group participants, following transformation of the areas of lesion into standardized space using standard procedures (Rogalsky et al., 2022). The following aphasia types were identified via site-specific protocols that included the Western Aphasia Battery-Revised (WAB), Boston Diagnostic Examination (BDAE), and clinical observations: Broca’s (n = 13), conduction (n= 8), anomic (n=9), Wernicke’s (n=1), global (n=1), transcortical sensory (n=2), transcortical motor (n=1), and no aphasia (n=5).

Figure 1.

Representative axial slices (z= −22 to 70 Montreal Neurological Institute coordinates) depicting the areas of left hemisphere damage in the stroke group participants (N=40). The areas of maximum overlap, N=22, include the left inferior frontal, superior temporal, and insular cortex.

Stimuli & Procedures

AV Synchrony task:

Stimuli were prerecorded videos of a female native speaker of English who spoke one of 8 monosyllabic words on each trial (doubt, give, knot, loan, pail, read, theme, voice). The timing of the audio was adjusted to one of 201 delays (or advances) from 500ms before the video onset to 500ms after in steps of 5ms, resulting in 1608 prerecorded videos (0 ms represents perfectly synchronized audio and video). These AV stimuli were then used in a 2-alternative forced-choice (2IFC) maximum-likelihood psychophysical procedure (Green, 1993; Saberi and Green, 1997; Hautus et al., 2021) to adaptively track the delay at which the participant perceived synchronicity. On each of 64 trials of a run, the participant had to determine if the audio and video were synchronous or not (a yes/no task). The maximum-likelihood procedure tracked the sweetpoint (Green, 1993) of the psychometric function (i.e., a Gaussian probability density function) that best fit the cumulative response pattern of the participant. At the end of the run, the point of subjective synchrony (PSS) was determined from the psychometric function that optimally fit the participant’s pattern of responses using maximum-likelihood rules (Green, 1993; Saberi and Green, 1997).² Each participant in the control group completed approximately 5 runs.³ Each brain-lesioned participant completed a single run because of secondary fatigue and task difficulty. No statistically significant difference was observed between threshold estimates for the “first run” and averaged runs of the control group (t(28)=0.42, p=0.68) or first and last run of this group (t(28)=0.87, p=0.39) and therefore the averaged estimates were used in data analysis. This protocol is identical to that used in our earlier work with other auditory tasks on brain-lesioned individuals (Farahbod et al., 2023).

AV fusion task:

The stimuli employed for the AV fusion task were adopted from an earlier study from our laboratory on the McGurk-MacDonald effect (Hickok et al., 2018). Details of the stimulus generation are fully described in that study and will be briefly outlined here. The AV fusion stimuli were of two types, matched (congruent) or mismatched (incongruent) audio-video. Videos were recorded of a male native speaker of English producing the following single syllables in isolation: /pa/ and /ka/. Separately, high quality auditory recordings of these same syllables from the same talker were made in an anechoic chamber. The incongruent AV stimulus was created by aligning the onset of the consonant burst for /pa/ with that for the replaced audio of the natural /ka/ associated with the video of /ka/, resulting in the /pa/-/ka/ incongruent AV stimulus. These /pa/-/ka/ incongruent stimuli reliably generate a perception of /ta/ in control participants as reported by Hickok et al. (2018). To create the congruent AV /pa/ and /ka/ videos, we performed the same procedure but with congruent auditory and visual stimuli. The AV fusion task comprised 30 trials in which participants were asked to indicate which of three auditory stimuli was presented: /pa/, /ka/, or /ta/. Participants were instructed to pay close attention to both the face in the video and the auditory stimulus. The 30 trials consisted of 20 congruent trials (ten AV stimuli /pa/ and ten AV stimuli /ka/), and 10 incongruent trials (auditory stimulus was /pa/, visual stimulus was /ka/). After each stimulus presentation, three printed response options were displayed horizontally across the computer screen, “Pa Ta Ka”, with the serial positions of the three options presented in a fixed random order across trials for each participant. Participants were instructed to determine which of the three response options corresponded to the sound that they had heard on that trial. Fusion rate was defined as the proportion of incongruent trials on which the participant responded “Ta” (fusing the auditory /pa/ with the visual /ka/ resulting in the McGurk-MacDonald illusion of Ta). An auditory-only task was also employed in the same manner as the AV task, but no visual speech stimuli were presented. The auditory-only task consisted of 30 trials. Participants were given the same instructions as in the AV task, i.e., to indicate which sound they heard. As in the AV task, three response options (“Pa, Ta, Ka”) were presented on the screen. In 20 trials the auditory stimulus /pa/ was presented, and in 10 trials /ka/ was presented (see Hickok et al., 2018 for additional details).

Stimuli for both tasks were presented using a Dell Latitude E5450 computer. The audio portion was presented binaurally through digital-to-analog converters and Sennheiser headphones (HD360 Pro). All four research sites used identical computers, stimuli, and headphones calibrated and tested at UCI and shipped to the other 3 research sites. The level of the audio signal was adjusted individually to what was reported by the participant to be a comfortable listening level (Farahbod et al., 2023). Before data collection began, all participants were given instructions and listened to a few practice trials of each task until it was clear to the experimenter that they fully understood the task.

Lesion-Symptom Mapping:

To further investigate the neural correlates of AV synchrony, lesion-symptom mapping was conducted using the NiiStat toolbox for Matlab (http://www.nitrc.org/projects/niistat) on the binary lesion maps in standard Montreal Neurological Institute (MNI) space. Two univariate analyses fitting a general linear model (GLM) were computed within each of the 94 left hemisphere regions of interest (ROIs) in the JHU atlas (Mori et al. 2008; Faria et al. 2012), as described in previous studies (e.g. den Ouden et al. 2019; Rogalsky et al. 2022). A GLM was fit for (1) PSS (synchrony) values below 45 ms (i.e. lower than the control group’s mean performance of 45 msec), and (2) PSS (synchrony) values above 45 msec, to identify regions in which the percentage of lesioned voxels was associated with PSS values below or above those of the control group. Overall lesion volume and data collection site were included as covariates. To correct for multiple comparisons (p< .05 controlled for family wise error), permutation thresholding was used with 4,000 permutations.

Results:

Figure 2 shows results of the AV synchrony experiment for the two participant groups. Panel A shows PSS values individually for each participant arranged in ascending order. The abscissa shows participant number and the ordinate the PSS, with positive values representing the amount of audio delay relative to the video. While the PSS for the control participants is nearly always positive (audio must be delayed relative to the video to be perceived as synchronous with the video) the lesion group’s results are symmetric about the “zero delay” point. Panel B shows mean PSS value for each participant group. For the control group, mean PSS was 44.6 ms, consistent with findings from prior studies (van Eijk et al., 2008; Vroomen and Keetels, 2010). Mean PSS for the lesion group was close to zero (−6.0 ms) with nearly equal distribution of PSS thresholds above and below zero (19 of 40 participants had negative PSS values). The mean PSS for the control group was statistically significantly higher than that for the lesion group (t(53)=2.07, p=0.043).⁴

Figure 2.

Results of the AV synchrony experiment. A) PSS for individual participants. Positive values represent the amount of audio delay relative to video. B) Averaged PSS for the two participant groups. Error bars are one standard error of the mean. N=40 (lesion); N=15 (control).

Figure 3 shows PSS values individually for all 40 brain-lesioned participants plotted in ascending order and labeled by aphasia type which is color coded as shown in the legend. A cursory examination suggests that brain-lesioned individuals with no aphasia (magenta symbols) and those with conduction aphasia (yellow) nearly always produced PSS values above zero, similar to the control group.

Figure 3.

PSS values for 40 brain-lesioned participants with aphasia diagnosis color coded and shown along the abscissa.

Figure 4 shows averaged PSS for four subgroups of brain-lesioned participants categorized by aphasia type: conduction, Broca’s, anomic, and no aphasia. We excluded from this analysis aphasia types for which we did not have a sufficient number of participants (see methods). The dashed horizontal line represents the average PSS for the control group. As noted earlier, the no-aphasia and conduction aphasia groups produced averaged positive PSS values, close to the control group (dashed line). The anomic and Broca’s aphasia groups produce negative mean PSS values. A one-way between-groups analysis of variance (ANOVA) on the data of Fig. 4A showed a significant effect of aphasia condition (F(3,31)=3.55, p=0.025, η²=0.26). Levene’s test showed no violation of homogeneity of variance (L(3,31)=1.589, p=0.21). Post-hoc t-tests (two-sided) showed a significant difference between the no-aphasia and anomic group (t(12)=2.34, p=0.037; Cohen’s d=1.31) as well as between the conduction and anomic groups (t(15)=2.13, p=0.05; Cohen’s d=1.03). Given the counterintuitive finding of a negative mean PSS value for the anomic group (−66.4 ms) we conducted a one-sided post-hoc t-test of the hypothesis that this negative PSS is significantly below a referent point of zero. The result showed that the mean negative PSS for the anomic group is not significantly below zero, though it did approach significance (t(8)=−1.79, p=0.055, Cohen’s d=−0.539).

Figure 4.

A) Averaged PSS values for four subgroups categorized by aphasia diagnosis. N=35. Error bars are one standard error. B) PSS as a function of brain lesion volume for each individual participant. N=40.

Inspection of the data shown in Fig. 3 shows that two participants with anomic aphasia produced the lowest PSS values (at −202 and −291 ms). We considered whether these two points may be extreme, and if yes, how they may have affected our statistical analyses especially for the anomic participant group which had a mean negative PSS (Fig. 4A). Based on a 3IQR criterion (3 times the interquartile range) which identifies extreme points in a univariate population (IBM SPSS v. 29; Hoaglin and Iglewicz, 1987) we found that neither point fell into this category. Note that the PSS value of −202 ms (the second largest negative PSS) is close in magnitude to PSS values of the two participants who produced the largest positive points (181 and 183 ms, Fig. 3), and therefore, we do not consider this single point at −202 ms to be aberrant (i.e., all three points also fall well within 3 z-score units of the population mean). These findings were confirmed with a Dixon Q test for outliers (Dixon, 1950; Rorabacher, 1991) which verified that the largest negative PSS (−291 ms) was not a significant outlier at a 95% confidence level (Q(n=40)=0.19, p>0.05, n.s.). However, because this negative PSS value (−291 ms) is relatively large in magnitude, we repeated the ANOVA test with this data point excluded and still found a significant effect of aphasia type (F(3,30)=3.00, p=0.046, η² =0.23). Post-hoc t-test analysis showed that with this point excluded, there also remained a significant difference between the mean PSS values of brain-lesioned participants with no aphasia and the anomic group (t(11)=2.35, p=0.038 two sided, p=0.019 one sided, Cohen’s d=1.34), and a near-significant difference between the conduction and anomic groups (t(14)=1.93, p=0.074 two-sided, significant with a one-tailed criterion p=0.037, Cohen’s d=0.965). If we restrict our analysis only to the anomic subpopulation, then the large negative PSS values are even less of an issue as the lowest PSS of −291 ms is only about 2 z-units away from the anomic mean PSS (−66 ms). We should add that of the 9 participants with an anomic diagnosis, 3 produced positive PSS values and 6 negative. Even with the PSS of −291 ms included in the analysis, the mean PSS for the anomic group is not significantly different than zero, though as noted earlier it approached significance (p=0.055). While this negative mean PSS for the anomic group is possibly statistical noise, its interesting counterintuitive direction and near-significance, in addition to findings from several prior studies that some listeners consistently adjust the audio to negative PSS values (van Eijk et al., 2008) make this an interesting case for at least some consideration (we briefly revisit this point in the discussion section).

Panel B of Fig. 4 shows PSS as a function of brain lesion volume. We found no statistically significant correlation between lesion volume and the ability of participants to synchronize the audio and video of the stimuli (r(38)=0.17; p=0.30), nor did we find significant lesion volume effects as a function of aphasia type (F(3,31)=2.09, p=0.12, η² =0.17). In fact, the mean lesion volume for the conduction aphasia group was larger than that for the anomic group (though not significantly: t(11)=0.36, p=0.73) in spite of the fact that the conduction group performed on the synchrony task more similarly to controls than other aphasia groups (Anomic or Broca’s; Fig. 4A). There was also no statistically significant difference in lesion volume between the no-aphasia (‘None’ in Fig. 4A) and the anomic group (t(12)=0.75, p=0.47), i.e., the two groups that produced the most divergent PSS values relative to each other.

Figure 5 shows results of the AV fusion experiment and its relation to AV synchrony. Because AV fusion requires processing of linguistic and phonemic content, and not just perceptual as in the synchrony task, we restricted data analysis for the AV fusion task to a subset of participants who were able to perform the auditory only syllable identification task with greater than 80% accuracy for both syllables (see methods: two auditory stimuli /pa/ and /ka/ and 3 response options that included /ta/). This procedure yielded a subgroup of 24 participants in the lesion group who on average performed the auditory only syllable identification task with 95% accuracy (96% for /pa/ and 94% for /ka/).

Figure 5.

A) AV fusion rate plotted as a function of AV synchrony (PSS) individually for each of 24 participants. B) AV fusion rate as a function of aphasia diagnosis. N=20.

Panel A of Fig. 5 shows AV fusion rate (i.e., percentage of “Ta” responses in the incongruent /pa/-/ka/ condition) plotted as a function of PSS (synchrony task) for each of the 24 lesion participants with 95% accuracy on the syllable identification task. The AV fusion rate represents the percentage of trials on which the McGurk-MacDonald illusion was perceived, with the participant reporting hearing a syllable that neither matched the auditory nor visual stimuli. We found a significant correlation between AV synchrony and fusion (r(22)=0.59, p=0.0013).⁵ This correlation cannot be explained by lesion volume as there was no significant correlation between lesion volume and fusion rate (r(22)=0.03, n.s.). The higher the PSS values, the more likely it was for the same participant to display the McGurk-MacDonald illusion. Participants with PSS values that departed most from controls (negative PSS values in Fig. 5A) were least likely to display the McGurk-MacDonald illusion although they performed quite well in their ability to identify auditory syllables in isolation (no visual cue). In fact, the participant who produced the lowest PSS value (near −200 ms) in the synchrony task showed no McGurk-MacDonald illusion at all (0% Ta responses) even though this same participant performed perfectly (100% correct) in identifying auditory alone syllables (/pa/ or /ka/).⁶

Panel B of Fig. 5 shows AV fusion rate as a function of aphasia diagnosis. We included only those categories of aphasia for which there were at least 5 participants. To determine if there was an effect of aphasia type on AV fusion, we first performed a Levene’s test for the homogeneity of variance, the result of which was highly significant (L(2,17)=7.04, p=0.006). This suggests that the variances of the fusion-rate measure across aphasia types are not homogeneous, and therefore, we used Welch’s F-test to determine if there was a significant difference in AV fusion rate across aphasia diagnoses. We found a significant effect of aphasia type (Welch’s F(2,11.17)= 4.37, p-0.04) which as observed in Fig. 5B was a result of the higher fusion rate of the “no-aphasia” brain-lesioned group. Furthermore, as before, this effect could not be attributed to differences in lesion volume across aphasia groups: There was no significant lesion volume difference between the no-aphasia and anomic groups (t(11)=0.87, p=0.404), nor between the no-aphasia and conduction aphasia groups (t(10)=0.26, p=0.801).

Several prior neuroimaging and lesion studies have implicated superior temporal areas, especially the posterior superior temporal gyrus (pSTG) as critical to AV processing (Merchant et al., 2012; Zhou et al., 2023; Nath and Beauchamp, 2012; Erickson et al., 2014; Krason et al., 2023; Hickok et al., 2018). We tested the hypothesis that cortical damage to pSTG adversely affects AV processing via a directional one-tailed t-test comparing performance of individuals with any damage to the left pSTG to those without any pSTG damage. Left pSTG damage was defined as any lesion map including any left superior temporal gyrus gray or white matter posterior to the posterior edge of Heschl’s gyrus. The JHU atlas (i.e. the same atlas as used in the lesion-symptom mapping analyses; Mori et al. 2008; Faria et al. 2012) was used to define Heschl’s gyrus and the superior temporal gyrus (both gray and white matter). PSTG was defined as any portion of the superior temporal gyrus (as delineated by the JHU atlas) that was posterior to the most posterior point of the JHU atlas’ Heschl’s gyrus region. We found a statistically significant effect on AV fusion at the full population level of 40 brain-lesioned participants (t(38)= 2.12, p=0.017) and a near-significant effect on the subpopulation of 24 participants (t(22)=1.33, p=0.079). The lesion-symptom mapping analyses did not identify any regions in which damage was significantly associated with PSS values above or below 45 msec (i.e. control group PSS values), using appropriate multiple correction procedures. For exploratory purposes, the un-thresholded z-scores for each region of interest were inspected. Damage to left posterior superior, middle, and inferior temporal gyri regions of interest corresponded to the highest z-scores for PSS (synchrony) values below 45 ms, with z-scores ranging from 2.24 to 2.61. Damage to frontal and insula regions yielded z-scores below 1. For PSS (synchrony) values above 45 ms, there were no regions in which the lesion-symptom mapping analyses yielded z-scores > 1.

Discussion:

An important finding of the current study is that “aphasia type” selectively affects AV synchrony. Setting aside the no-aphasia brain-lesioned group who performed most similarly to the control group, the largest contrast in performance of aphasic populations was between those with conduction and anomic aphasia. Individuals with conduction aphasia adjusted the audio to be delayed by approximately 25 ms on average relative to the video for perceived synchrony. This value is close to (or at least in the same direction as) that of control participants (45 ms). Conversely, those with anomic aphasia performed most differently than controls, with an average adjustment of −66 ms, i.e., with the video delayed relative to the audio (opposite to that of controls).

Anomic aphasia is characterized by difficulty in recalling words or finding the appropriate words to identify a person or object or numbers. Conduction aphasia, in contrast, is primarily a speech production impairment characterized by frequent phonemic speech errors and deficiency in verbatim repetition but with no difficulty in speech perception (Baldo, Klostermann, & Dronkers, 2008; Benson et al., 1973; Damasio & Damasio, 1980; Goodglass, 1992). While individuals with conduction aphasia performed more similarly to the control group, this is not to say that they were unaffected in AV tasks but rather that the adverse effect of lesions in this population is less pronounced. Figure 6 shows lesion overlap maps for the conduction and anomic aphasic participants. Most of the conduction aphasia lesions in our participants do not affect the ventral part of the pSTG, in the superior temporal sulcus, which is most often implicated in AV perception (Hickok et al., 2018). Anomic aphasia, however, is related to more significant lesions in the pSTS (sulcus) compared to conduction aphasia which was associated with more superior lesions (pSTG but not STS). It is interesting that most participants with conduction aphasia have damage to pSTG and surrounding regions, yet still perform more similarly to the control group in the AV synchrony task. Perhaps the more posterior and ventral temporal lesions seen in the participants with anomic aphasia are more disruptive to AV synchrony.

Figure 6.

Representative orthogonal slices of the overlap of the lesions of individuals with (A) anomic aphasia range of n=1 to n=6 with warmer colors indicating greater overlap; (B) conduction aphasia with range of n=1 to n= 5 with cooler colors indicating greater overlap, and (C) the areas of maximum overlap for each group.

It is unclear why the anomic group, on average, adjusted the video to lag the audio for perceived synchrony (why not simply a zero delay if there’s a general systemic failure?). Even though this effect (negative PSS) was not statistically significant when compared to a referent point of zero, it was near-significant (p=0.055) and therefore some consideration of its potential origins may be warranted here. Perhaps, in the anomic group, disruption of the neural pathways that process the ascending auditory information may have delayed the arrival of the auditory signal to higher cortical centers that encode synchrony perception. While this is speculative, a more circuitous distal route to higher centers caused by anomic-specific lesions may delay the arrival of the auditory neural signal and thus require a concomitant compensatory delay of the physical visual stimulus (to reproduce the original expected time difference). This would be the case if such cortical decision centers are calibrated to (and expecting a) fixed time difference between the arrival of auditory and visual information. Consistent with this interpretation, there is evidence from EEG and MEG studies that aphasia may in fact cause a delay in neural transmission of speech stimuli (Giaquinto and Ranghi, 2009; Kielar et al., 2019). We should also clarify that while, on average, most normal-hearing listeners adjust the audio to lag that of its corresponding video for perceived synchrony, several reports have shown that some listeners consistently adjust the audio to lead the video, i.e., an ecologically invalid adjustment since an audio signal never naturally leads its corresponding visual component. The reason for this is unknown but it is observed in different experimental designs including AV temporal-order judgements and synchrony decisions (van Eijk et al., 2008).

The few prior studies that have examined the relationship between aphasia type and AV processing have been individual case studies. Youse et al. (2004) for example investigated AV fusion rates (McGurk-MacDonald illusion) in an individual with anomic aphasia and found that compared to an age-matched control there was a significant decline in the illusion which they attributed to a perseverative response pattern. They suggested that this result may potentially be caused by the difficulty that aphasics have with nonsense syllable combinations (Duffy and Coelho, 2001). In another case study, Ramachandran et al. (1999) found a significant decline in performance of an individual with Broca’s aphasia in an AV fusion task and concluded that Broca’s area is critical to AV integration of speech. Andersen and Starrfelt (2015), however, in another case study of an individual with Broca’s aphasia found, contrary to Ramachandran et al., that their participant did in fact experience the McGurk-MacDonald illusion, indicating that an intact Broca’s area is not required for AV integration of speech. Several other studies of AV processing by aphasics have reported group-level results for aphasia (in general) versus control groups but not selectively by aphasia type (Michaelis et al., 2020; Karson et al., 2023).

One caveat when interpreting our aphasia type-specific results in the context of previous findings is that aphasia classification may vary based on the clinical assessment used to determine aphasia type (Crary et al. 1992; Swindell et al. 1984). For example, there is some evidence that the WAB and BDAE may have inconsistent agreement for aphasia type, including anomic aphasia potentially being under-identified by the BDAE compared to the WAB (Crary et al. 1992). Within the present study, our site-specific assessment procedures included (but were not limited to) either the WAB or BDAE. Each site has their own protocol for monitoring and maintaining the reliability of their diagnostic tests, employing teams of neurologists, speech-language pathologists, and experienced research assistants in classifying aphasia subtype (with significant data-sharing and interaction across sites). Since all three sites contributed both anomic and conduction aphasics to this dataset, if different sites on occasion classify a subject differently (adding noise to the data and reducing statistical power) then our finding is still compelling in that we observed a statistically significant difference between the anomic and conduction groups in perception of AV synchrony in spite of the additional statistical noise. It is also notable that the lesion overlap maps for our anomic and conduction aphasia groups (Fig. 6) align with a variety of previous studies localizing lesions associated with these aphasia subtypes and their defining impairments, further corroborating our diagnosis of the subtypes (Dronkers & Baldo 2009; Baldo et al. 2008, 2013; Yourganov et al. 2016; Akinina et al. 2019). Nonetheless, because this potential variability in assessment protocol may be adding noise to our data, the aphasia type results should be viewed with this potential limitation in mind. Our main finding is however still empirically noteworthy and clinically relevant: there are two groups of aphasia participants who perceive AV synchrony in very different ways.

Another significant finding of the current study is the positive correlation between AV synchrony and fusion in the lesion population. Participants who performed most similar to controls in the AV synchrony task also showed the strongest AV fusion rate even when we controlled for lesion volume and for their ability to accurately perceive “auditory-only” syllables in isolation. This finding adds to results of Hickok et al. (2018) who found no correlation between AV fusion and AV advantage. They concluded that the absence of a correlation suggests that these two metrics do not necessarily measure the same underlying process. They further noted that AV advantage likely engages lip-reading as an orthogonal source of information to compensate for degraded speech (i.e., an additive effect instead of an interactive process). They observed that in the AV advantage task, unless speech perception is degraded to begin with, there is, by definition, no room for improvement by adding a video to the audio signal (i.e., a ceiling effect). This requirement does not exist for AV synchrony which (in this sense) is a more proximate metric to AV fusion in measurement of perceptual integration. Further support for this view is provided by Michaelis et al. (2020) who demonstrated an inherent link between AV fusion and synchrony by desynchronizing the audio and video of an incongruent (McGurk-MacDonald) stimulus.⁷ They found that if the audio lags the video there is no significant reduction in AV fusion rate. However, if the audio leads the video there is a significant decline in fusion (i.e., the illusion is much weaker). This temporal asymmetry may be explained by the fact that a delayed audio is expected for naturally occurring sounds (due to acoustic propagation time) but the reverse, the audio leading the visual, never occurs in nature. Their results suggest that temporally desynchronizing the audio and video is not a simple decoupling of orthogonal cues to fusion, but rather, its directional nature implies a robust integrative relationship between AV synchrony and fusion.

Concluding remarks:

Our findings have important implications for the cortical bases of multisensory integration and how different audiovisual processes interact in encoding incoming stimuli in complex natural environments. Some AV processes may be interactive, as our findings on synchrony and fusion show, or additive, as the orthogonal cues reported for audiovisual advantage and fusion (Hickok et al., 2018). Our findings additionally demonstrate that different audiovisual processes engage functionally and anatomically different cortical regions, with superior temporal areas critical to audiovisual fusion and the posterior and ventral temporal regions important in processing synchronous audiovisual information. Finally, we would like to conclude by noting that beyond what it can reveal about the foundations of cortical organization in AV processing, a better understanding of how various metrics of AV integration are related has practical implications for clinical strategies in treatment of aphasia and other language dysfunctions (Geschwind, 1974; de Freitas, 2012; Prigatano et al., 1985). Treatment approaches to aphasia rely heavily on bimodal interventions that combine auditory and visual stimulus components (Duffy and Coelho, 2001). Insight into these AV interactions can aid speech pathologists in developing more appropriate treatment protocols, facilitate communication between aphasia patients and clinicians, and develop more effective therapeutic goals based on aphasia-specific diagnoses (Youse et al., 2004). The current findings may therefore additionally contribute to improvements in clinical interventions and rehabilitation strategies for treatment of language disorders.

Data Availability:

Raw data and analysis programs associated with the current study are available on request.