Links between temporal acuity and multisensory integration across lifespan

Abstract

The temporal relationship between individual pieces of information from the different sensory modalities is one of the strongest cues to integrate such information into a unified perceptual gestalt, conveying numerous perceptual and behavioral advantages. Temporal acuity, however, varies greatly over the lifespan. It has previously been hypothesized that changes in temporal acuity in both development and healthy aging may thus play a key role in integrative abilities. This study tested the temporal acuity of 138 individuals ranging in age from 5 to 80. Temporal acuity and multisensory integration abilities were tested both within and across modalities (audition and vision) with simultaneity judgment and temporal order judgment tasks. We observed that temporal acuity, both within and across modalities, improved throughout development into adulthood, and then subsequently declined with healthy aging, as did the ability to integrate multisensory speech information. Importantly, throughout development, temporal acuity of simple stimuli (i.e., flashes and beeps) predicted individuals’ abilities to integrate more complex speech information. However, in the aging population, while temporal acuity declined with healthy aging and was accompanied by declines in integrative abilities, temporal acuity was not able to predict integration at the individual level. Together, these results suggest that the impact of temporal acuity on multisensory integration varies throughout the lifespan. Whereas the maturation of temporal acuity drives the rise of multisensory integrative abilities during development, it is unable to account for changes in integrative abilities in healthy aging. The differential relationships between age, temporal acuity, and multisensory integration suggests an important role for experience in these processes.

Introduction

The ability to integrate multiple pieces of sensory information into a unified and coherent percept is fundamental to our ability to perceive the world around us. One of the best examples of the seamless integration that can take place across the senses is in the perception of speech. Thus, in a conversation we do not individually perceive the speaker’s visual aural articulations separately from the auditory speech information, but instead perceive the mouth movements and auditory signals as a single unified event (McGurk & MacDonald, 1976). Integrating information in this way is fundamental in the construction of our perceptual Gestalt, and has been long known to convey myriad perceptual and behavior benefits, from the speeding of processing as measured by response times to increases in perceptual accuracy (Altieri, Stevenson, Wallace, & Wenger, 2015; Hershenson, 1962; Murray & Wallace, 2011; B. Stein & Meredith, 1993; Sumby & Pollack, 1954).

Given the wealth of information impinging on our sensory apparatus at any given moment, only certain pieces of information are associated with specific objects or events. Whereas it is critical that this related information is integrated, it is equally important that the unrelated information be segregated. Collectively, such integration and segregation can be captured within the cognitive construct of the “binding problem” (Treisman, 2006).

One of the strongest cues in the evaluation of common cause (and thus integration versus segregation) is the temporal relationship of the stimuli to one another (Vroomen & Keetels, 2010). In short, the closer two incoming sensory signals are in time, the more likely they are to be integrated. This has been seen across a wide array of measures including: single unit recordings (Meredith, Nemitz, & Stein, 1987; B. Stein & Meredith, 1993; Wallace, Wilkinson, & Stein, 1996), measures of neural populations (Miller & D’Esposito, 2005; Stevenson, Altieri, Kim, Pisoni, & James, 2010; Stevenson, VanDerKlok, Pisoni, & James, 2011), behavioral measures such as stimulus detection (Fister, Stevenson, Nidiffer, Barnett, & Wallace, 2016; Lovelace, Stein, & Wallace, 2003; B. E. Stein & Wallace, 1996), and perceptual measures such as gains in accuracy (Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Stevenson, Bushmakin, et al., 2012; Sumby & Pollack, 1954). Given the critical role that relative timing cues play in multisensory integration and binding, the acuity of one’s temporal perception has been hypothesized to play a major role in the ability to effectively integrate sensory information across the senses. Indeed, in healthy adults, it has previously been shown that multisensory temporal acuity, even as measured with very simple stimuli such as flashes of light and noise bursts, is strongly correlated with multisensory integration of complex social stimuli, such as audiovisual speech (Stevenson, Zemtsov, & Wallace, 2012), though this relationship is less apparent in developing populations (Stevenson, Siemann, Schneider, et al., 2014).

Temporal acuity, and temporal processing more generally defined, changes across lifespan due to the development of peripheral organs, the refinement of cerebral functioning, and experiential factors. This includes unisensory changes in the timing acuity of sensory evoked responses within the individual visual (Saul & Feidler, 2002; Wang, Zhou, Ma, & Leventhal, 2005; Yang, Liang, Li, Wang, & Zhou, 2009) and auditory (Gelfand, Hoffman, Waltzman, & Piper, 1980; Ozmeral, Eddins, Frisina, & Eddins, 2016; Robin & Royer, 1989; Strouse, Ashmead, Ohde, & Grantham, 1998) systems as well as the precision with which sensory information is encoded (Anderson, Parbery-Clark, White-Schwoch, & Kraus, 2012; Guest, Howard, Brown, & Gleeson, 2015). In addition to the changes within the individual senses, temporal acuity in integration information across the different senses also changes during development. Audiovisual temporal perception in particular has been shown to have quite an extended window of development. As early as four months after birth, amodal temporal cues can be detected, including synchrony (L. Bahrick, 1987; L. E. Bahrick, 1983, 1988; Lewkowicz, 1986, 1992, 1996; Spelke, 1979) and temporal microsctructure (L. Bahrick, 1987). The ability to detect these cues, particularly temporal synchrony, increase through infancy and childhood (L. Bahrick, 1987; L. E. Bahrick, 1983, 1988; Lewkowicz, 1996; Spelke, 1979). Quite surprisingly, the capacity to detect audiovisual synchrony continues to improve well into adolescence, indicating a prolonged window of maturation (Hillock-Dunn & Wallace, 2012; Hillock, Powers, & Wallace, 2011; Kaganovich, 2016; Lewkowicz & Flom, 2014). Following adulthood, multisensory temporal fidelity begins to decline again, where even healthy older adults show decreased sensitivity for detecting asynchrony in audiovisual stimuli (Chan, Pianta, & McKendrick, 2014a, 2014b; Setti et al., 2011). Thus, across the lifespan, the ability to accurately perceive audiovisual temporal relationships across the lifespan seems follow an inverted U-shaped function, with performance peaking in late adolescence and then waning in middle and old age. Furthermore, these declines in audiovisual temporal processing cannot be solely accounted for by declines in unisensory acuity (Chan et al., 2014b).

In addition to these changes in audiovisual temporal acuity across lifespan, the ability to integrate or fuse information across the different senses also changes across the lifespan. One canonical measure of multisensory fusion or “binding” that has been extensively studied is the McGurk Effect (McGurk & MacDonald, 1976). This effect has been studied in infants, children and adolescents (Burnham & Dodd, 2004; Hockley & Polka, 1994; Massaro, 1984; McGurk & MacDonald, 1976; Stevenson, Siemann, Schneider, et al., 2014; Stevenson, Siemann, Woynaroski, et al., 2014a; Tremblay et al., 2007) and in adults (Baart & Vroomen, 2010; Beauchamp, Nath, & Pasalar, 2010; Nath & Beauchamp, 2012; Saint-Amour, De Sanctis, Molholm, Ritter, & Foxe, 2007; Soto-Faraco & Alsius, 2009; Stevenson, Zemtsov, et al., 2012), and preliminarily studied in older adults (Cienkowski & Carney, 2002; Sekiyama, Soshi, & Sakamoto, 2015). In the most common form of this illusion, an individual is shown a video in which a speaker is visually articulating the syllable “ga” while concurrently presenting the auditory syllable “ba.” When these two streams of sensory information are integrated, an illusory percept of “da” or “tha” is reported.

As with temporal processing abilities, the ability to integrate speech information as indexed by the McGurk effect increases throughout development (Massaro, 1984; McGurk & MacDonald, 1976; Tremblay et al., 2007) and is relatively stable during adulthood (Mallick, Magnotti, & Beauchamp, 2015). Results in aging populations, however, reveal mixed findings in perception of the McGurk Effect, though fewer studies have been conducted in this population (Cienkowski & Carney, 2002; Sekiyama et al., 2015). The integration of more realistic auditory stimuli like words in speech seems to show the same tendency (Ross et al., 2011), but with mixed results in healthy aging (Gordon & Allen, 2009; Hugenschmidt, Peiffer, McCoy, Hayasaka, & Laurienti, 2009; Peiffer, Mozolic, Hugenschmidt, & Laurienti, 2007; Sommers, Tye-Murray, & Spehar, 2005; Ryan A Stevenson et al., 2015; Tye-Murray, Sommers, Spehar, Myerson, & Hale, 2010).

It has been hypothesized that changes in temporal processing abilities may drive, or at least contribute to, the changes in multisensory integration (Gordon-Salant & Fitzgibbons, 1993; Stevenson, Segers, Ferber, Barense, & Wallace, 2014; Stevenson, Zemtsov, et al., 2012). This hypothesis is based on evidence that of temporal acuity in sensory processing (both within and across modalities) and integrative abilities across the lifespan share similar developmental trends, and that temporal acuity in adults is highly predictive of integrative abilities. However, this proposal has not, to our knowledge, been empirically tested in developing populations or aging populations. The present study aims to address this important knowledge gap. We tested 138 individuals ranging in age from 5 to 80 years old on a battery of behavioral tasks measuring both temporal processing and multisensory integration. More specifically, each individual’s auditory, visual, and audiovisual temporal acuity were tested through standard measures (simultaneity judgment and temporal order judgment tasks). Audiovisual integration was tested through the McGurk Effect, with auditory, visual, and audiovisual perception of concordant speech tokens included as control measures. We measured the ability of temporal processing abilities to predict integrative abilities and, of greatest import, measured how these predictions change throughout the lifespan beginning with children (5–12 years old), and progressing through adolescents (13–19 years old), younger adults (20–39 years old), middle age (40–59 years old), and in older adults (60–80 years old). We hypothesized that, through development temporal acuity would be predictive of multisensory integration through at least adolescence, and possibly into adulthood. As temporal acuity declines with healthy aging, however, it is currently unclear if this is associated with decreases in integrative abilities, as previous studies have reported mixed findings. Thus, it is possible that temporal acuity and multisensory integration will be more weakly coupled in aging that in development.

Materials and Methods

Participants

Participants included 138 individuals ranging in age from 5 to 80 years old (mean = 37.1 ± 22.7). Participants reported no history of neurologic illness, and normal or corrected-to-normal vision and hearing. Participants were divided into 5 age groups, children (5–12 years old, N = 24, mean age = 8.9 ± 1.8), adolescents (13–19 years old, N = 22, mean age = 14.2 ± 1.6), younger adults (20–39 years old, N = 26, mean age = 28.3 ± 6.5), middle aged (40–59 years old, N = 31, mean age = 49.4 ± 5.7), and older adults (60–80 years old, N = 35, mean age = 66.5 ± 5.1).

General procedures

Participants were asked to complete 4 perceptual tasks, an auditory temporal order judgement task, a visual temporal order judgment task, an audiovisual simultaneity judgement task (SJ), and a phoneme perception/McGurk illusion task. All experiments took place in a dimly lit, sound-controlled room. All visual stimuli were presented on a NEC MultiSync FE992 monitor at 100 Hz at a distance of approximately 60 cm from the participants at an average luminance of 55.8 cd/m². All auditory stimuli were presented binaurally via Phillips noise-cancelling SBC HN-110 headphones at 72 dB SPL. The timing and duration of all visual and auditory stimuli was confirmed using a Hameg 507 oscilloscope with a photovoltaic cell and microphone. Participants were actively monitored for compliance by a researcher sitting next to the participant. All methods and procedures were approved by the Vanderbilt University institutional review board.

Unisensory timing tasks

TOJ tasks were run with auditory and visual stimuli to measure unisensory temporal acuity. SJ tasks were not used, as the high acuity of unisensory simultaneity judgments often leads to a ceiling effect (Stevenson, Siemann, Schneider, et al., 2014). Participants were presented with auditory or visual stimulus pairs in separate runs. In the visual TOJ task, stimuli consisted of two white circles on a black background, above and below the fixation cross (duration = 10ms), presented at parametrically varied stimulus onset asynchronies (SOAs; 10, 20, 30, 60, 60, 80, 100, and 150 ms) with 20 trials at each SOA presented in random order. In the auditory TOJ task, stimuli consisted of a high- and low-pitch (1000 and 500Hz, respectively, duration = 10ms) pair of beeps presented at a range of SOAs (10, 20, 35, 50, 75, 100, 150, 200, and 250 ms) with 20 trials at each SOA presented in random order. In both tasks, participants reported which came first, the high or low stimulus. SOAs were chosen in accordance with previously conducted studies using equivalent tasks and stimuli, which are know to cover the psychophysical range (Stevenson, Siemann, Schneider, et al., 2014) in each modality.

Audiovisual timing task

The audiovisual SJ task consisted of paired presentations of a single flash (white ring on a black background with a duration of 10 ms) and a single beep (500 Hz pure tone with a duration of 10 ms). Stimulus pairs were presented at parametrically varied SOAs including 0, ±10, ±20, ±50, ±80, and ±100 to 300ms in 50ms intervals, with positive values indicating a visual lead, and negative values indicating an auditory lead. Participants reported if stimuli were temporally aligned and completed 20 trials per SOA in random order.

Audiovisual speech perception and integration

Participants were presented with auditory, visual, and audiovisual single-phoneme speech tokens. All stimuli consisted of the syllables “ba” and “ga.” Unisensory control trials consisted of audio-only or visual-only presentations of either “ba” or “ga,” and multisensory control trials consisted of congruent audiovisual presentations of either “ba” or “ga.” McGurk stimuli, which were used to measure multisensory integration, consisted of pairing temporally aligned visual “ga” with the auditory “ba”, a pairing known to induce the illusory perception of “da” or “tha.” This stimulus set was originally described in full detail previously (Quinto, Thompson, Russo, & Trehub, 2010), and has been successfully used in multiple previous studies of audiovisual integration (Stevenson et al., Under Review; Stevenson, Siemann, Schneider, et al., 2014; Stevenson, Siemann, Woynaroski, et al., 2014a; Stevenson & Wallace, 2013a; Stevenson, Zemtsov, et al., 2012). Participants reported what the speaker said in each condition by pressing one of four keys, “b,” “g,” “d,” or “t”. Twenty trials of each condition were presented in random order.

Analysis

Mean responses for each individual at each SOA were used to calculate a threshold at which individuals were able to correctly identify the temporal order of presentations at a 75% rate. This threshold was found by fitting a single sigmoid function using the MatLab glmfit function with the parameters ‘binomial’ and ‘logit’. The offset at which individuals obtained a 75% accuracy rate through the MatLab glmval function.

Mean responses from the SJ task were used to calculate a temporal binding window (TBW) for each subject and stimulus category. Two psychometric sigmoid functions were fit to rates of perceived synchrony across SOAs; one to the audio-first (left) presentations and a second to the visual-first presentations (right). To account for non-zero points of subjective simultaneity, the SOA at which these two sigmoid functions crossed was extracted. If this point was greater or less than the next closest SOA value, two new sigmoid functions were fit splitting the data at the SOA at which the original sigmoid functions crossed. This process was continued in an iterative manner until the SOA at which best-fit sigmoid functions crossed fell between the two data points at which the data were split (Fister et al., 2016; Stevenson, Fister, Barnett, Nidiffer, & Wallace, 2012; Stevenson, Siemann, Schneider, et al., 2014; Stevenson & Wallace, 2013b).

Individuals’ susceptibility to the McGurk effect was calculated as the percentage of McGurk trials in which he/she reported the fused (i.e. McGurk) percept (i.e., “da” or “tha”).

Results

Visual temporal perception

A one-way ANOVA was conducted comparing visual temporal thresholds using a temporal order judgment (TOJ) task across age groups (Figure 1A), revealing a significant effect of age group (p < 0.001, F_(4,129) = 10.73, η_p² = 0.25). Follow up, pairwise comparisons were conducted, revealing a sharp improvement in visual temporal acuity between children and adolescents (p = 0.0003, t₍₄₃₎ = 3.87, d = 1.18), and a smaller, non-significant increase in temporal acuity into younger adults, where visual temporal acuity peaked (p = 0.15, t₍₅₂₎ = 1.47 d = 0.42). Visual acuity significantly improved from children to younger adults (p = 9.02e⁻⁷, t₍₄₉₎ = 5.62, d = 1.59) and middle age (p = 2.27e⁻⁵, t₍₅₈₎ = 4.61, d = 1.23), but exhibited only an improvement trending towards significance when compared with older adults (p = 0.10, t₍₆₂₎ = 1.67, d = 0.45).

Figure 1.

Temporal acuity measured with simple flash-beep stimuli across the lifespan. Panel A depicts visual (red) temporal acuity, which increases in development into adulthood, and then declines with healthy aging. Panel B depicts auditory (blue) temporal acuity, which increases in development into adulthood, and then declines with healthy aging. Panel C depicts audiovisual (purple) temporal acuity, which increases in development into adulthood and through to middle age, and then declines with healthy aging. It should be noted that lower values on the y-axis represent more acute temporal acuity. Error bars represent standard error. Ticked lines indicate significance at the p < 0.05 level, # indicated p < 0.06.

A significant decline in temporal acuity was observed from younger adults into middle age (p = 0.05, t₍₆₅₎ = 1.98, d = 0.53), which declined further still into older adults (p = 0.0006, t₍₄₉₎ = 3.63, d = 0.91). Older adults also exhibited a significant decline relative to adolescents (p = 0.003, t₍₅₆₎ = 3.09, d = 0.87) and younger adults. No difference was observed between the middle-age and adolescent groups (p = 0.91, t₍₅₂₎ = 0.12, d = 0.03).

Auditory temporal perception

A one-way ANOVA was conducted comparing auditory temporal thresholds using a temporal order judgment (TOJ) task across age groups (Figure 1B), revealing a significant effect of age group (p < 0.001, F_(4,115) = 7.17, η_p² = 0.20). Follow up, pairwise comparisons were conducted, revealing a sharp increase in auditory temporal acuity between children and adolescents (p = 7.96e⁻⁷, t₍₄₅₎ = 5.73, d = ), and relative to the children, all other groups showed improvements, including younger adults (p = 5.48e⁻⁸, t₍₄₉₎ = 6.41, d = 2.08), middle age (p = 1.82e⁻⁶, t₍₅₄₎ = 5.35, d = 1.89), and older adults (p = 9.75e⁻⁴, t₍₅₈₎ = 3.47, d = 1.12).

A significant decline in acuity was observed from the younger adults to the older adults (p = 0.006, t₍₅₆₎ = 2.83, d = 0.77), though the individual steps from younger adults into middle age (p = 0.41, t₍₅₆₎ = 0.82, d = 0.22) and from middle age into older adults (p = 0.059, t₍₆₅₎ = 1.92, d = 0.51) failed to reach significance. Significant declines in auditory temporal acuity were also observed between adolescents and older adults (p = 0.03, t₍₅₆₎ = 2.24, d = 0.66), and between younger adults and older adults (p = 0.03, t₍₅₆₎ = 2.24, d = 0.66). No differences were seen between adolescents and younger adults (p = 0.71, t₍₄₇₎ = 0.38, d = 0.11) or adolescents and middle age (p = 0.67, t₍₅₂₎ = 0.42, d = 0.12).

Audiovisual temporal perception

The audiovisual SJ task was used to derive the TBW, which was used as a proxy measure of audiovisual temporal acuity (Figure 1C). A one-way ANOVA was conducted comparing the TBW across age groups, and showed a significant effect of age group (p < 0.004, F_(4,116) = 4.04, η_p² = 0.13). Follow up, pairwise comparisons were conducted, revealing a similar, yet more gradual pattern than that observed for the unisensory stimuli - a developmental increase in acuity followed by a decrease in acuity with age. A small but not significant improvement in acuity between children and adolescents was observed, (p = 0.43, t₍₄₉₎ = 0.80, d = 0.05), a difference that became significant when comparing children to younger adults (p = 0.03, t₍₄₉₎ = 2.18, d = 0.86) and to the middle aged group where temporal acuity peaked (p = 0.004, t₍₅₈₎ = 2.99, d = 1.05), and trended towards significance relative to the older adults (p = 0.08, t₍₆₂₎ = 1.78, d = 0.27). Additionally, relative to the adolescents, younger adults (p = 0.05, t₍₄₃₎ = 2.01, d = 0.62) and the middle-aged group (p = 0.004, t₍₅₂₎ = 3.02, d = 0.73) showed significant improvements in temporal acuity.

The older adults showed significant declines in temporal acuity relative to middle age (p = 0.03, t₍₆₅₎ = 2.27, d = 0.79). Older adults did not show any difference from adolescents (p = 0.26, t₍₅₆₎ = 1.14, d = 0.23) or younger adults (p = 0.20, t₍₅₆₎ = 1.30, d = 0.61), and no difference was observed between the younger adults and middle-age adults (p = 0.57, t₍₅₂₎ = 0.57, d = 0.18).

Audiovisual integration: The McGurk Effect

Multisensory integration of speech information was indexed via the McGurk effect, where participants were presented with an auditory “ba” and a visual “ga.” In this paradigm, integration (i.e., fusion) is indexed via reports of “da” or “tha,” as the perceived phoneme is not present in either of the sensory signals (Figure 2A). A multivariate ANOVA was conducted measuring the 3 possible responses (“ba”, “ga”, or a “da”/”tha” fusion) across the 5 age groups. Significant effects were found between age groups for reporting the fused percept (p < 0.001, F_(4,116) = 5.41, η_p² = 0.16), as well as for reporting the visual token “ga” (p < 0.002, F_(4,116) = 4.39, η_p² = 0.13). No difference was observed for reporting the auditory token “ba” (p < 0.14, F_(4,116) = 1.77, η_p² = 0.06), though the effect size was in the medium range.

Figure 2.

The McGurk Effect and Phoneme perception throughout the lifespan. Panel A depicts responses to McGurk stimuli in which a visual “ga” was presented with an auditory “ba.” Reports of perceptions indicative of visual capture are presented in red (“ga”), auditory capture in blue (“ba”), and audiovisual integration in purple (“da”). Panel B depicts accuracy of phoneme perception with unisensory visual presentations (red), unisensory auditory presentations (blue), and congruent audiovisual presentations (purple). Ticked lines indicate significance at the p < 0.05 level, with colors matching depicted bars.

Follow-up analysis of the fusion responses showed a numerical but not significant increase in the perception of the McGurk effect from children into adolescents (p = 0.17, t₍₄₉₎ = 1.38, d = 0.40) and from adolescents into younger adults (p = 0.71, t₍₃₀₎ = 0.37, d = 0.16). Following that, however, a significant drop in reports of the illusory percept was seen between younger adults and the middle-age group (p = 0.01, t₍₃₇₎ = 2.69, d = 1.20), which remain, unchanged, between the middle-age and older adults (p = 0.87, t₍₆₁₎ = 0.16, d = 0.04).

Detailed age-group analysis confirmed this trend of a marginal increase of fusion responses into adulthood with a dramatic decrease during middle age. Relative to the group of children, the younger adults did not significantly differ (p = 0.17, t₍₃₆₎ = 1.39, d = 0.60), but did differ relative to middle-age (p = 0.03, t₍₅₆₎ = 2.24, d = 0.60) and older adults (p < 0.03, t₍₆₀₎ = 2.21, d = 0.57). Furthermore, adolescents showed significantly more fused percepts than middle-age (p < 0.002, t₍₅₀₎ = 3.34, d = 0.97) and older-adults (p = 0.001, t₍₅₄₎ = 3.38, d = 0.96). The younger adult group, where fused percepts peaked, showed significantly higher fusion rates than older adults (p = 0.001, t₍₄₁₎ = 2.73, d = 1.21).

Follow-up pairwise analyses of the unisensory responses revealed that, in general, older adults showed an increased rate of unisensory capture. In the case of reporting the visual token, older adults showed significantly higher rates than children (p = 0.005, t₍₆₀₎ = 2.89, d = 0.76), adolescents (p = 0.003, t₍₅₄₎ = 3.06, d = 0.89), and younger adults (p < 0.04, t₍₄₁₎ = 2.15, d = 0.99), while showing numerically but not significantly higher rates than middle-aged adults (p = 0.13, t₍₆₁₎ = 1.52, d = 0.39). No age-related differences were seen in reports of the auditory token.

Control trials for the McGurk effect were also analyzed (Figure 2B). A repeated-measure ANOVA was conducted with modality (audio, visual, and audiovisual) as a within-subjects contrast, and age-group as a between-subjects contrast. A main effect of modality (p < 0.001, F_(2,232) = 131.94, η_p² = 0.53), age group (p < 0.001, F_(4,116) = 11.65, η_p² = 0.29), and a significant interaction between the two was observed (p < 0.001, F_(8,232) = 10.31, η_p² = 0.26).

Follow-up pairwise tests were conducted to explore this interaction. No pairwise comparisons of audiovisual accuracy significantly differed between any age group (all p’s > 0.05). Auditory accuracy did not differ in any pairwise comparisons between the children, adolescents, or younger adult. However significant drops in auditory accuracy were seen in middle-age relative to children (p = 0.0006, t₍₅₆₎ = 3.65, d = 0.97) and adolescents (p = 0.00003, t₍₅₀₎ = 4.68, d = 1.38), and between older adults and children (p = 2.30e⁻¹⁰, t₍₆₀₎ = 7.60, d = 1.98), adolescents (p = 2.80e⁻¹², t₍₅₄₎ = 8.97, d = 2.59), younger adults (p = 0.0004, t₍₄₁₎ = 3.83, d = 1.32), and middle-age (p = 0.01, t₍₆₁₎ = 2.63, d = 0.66). Visual accuracy did not differ between any age groups younger than middle age. Older adults showed significantly reduced visual accuracy relative to children (p = 0.004, t₍₆₀₎ = 3.02, d = 0.77), adolescents (p = 0.0001, t₍₅₄₎ = 4.16, d = 1.17), younger adults (p = 0.002, t₍₄₁₎ = 3.38, d = 1.25), and middle age (p = 0.004, t₍₆₁₎ = 2.98, d = 0.76).

Temporal processing abilities predict multisensory integration

Two hierarchical linear regression models were used to explore the predictive power of measures of temporal acuity for multisensory integration across the lifespan, with data split into age-groups that showed increases in multisensory temporal processing with development and those that showed decreases in multisensory temporal processing with healthy aging. The first was thus conducted in the children, adolescents, and younger adults where multisensory integration increased with development. In the first step of the regression, age was entered in as a predictor to control for the possibility that temporal acuity and multisensory integration are related to age itself and not related to each other. Age did not significantly predict multisensory integration (Model 1: p = 0.47, r² = 0.01, F_(1,39) = 0.53). In the second stage of the regression, visual thresholds, auditory thresholds, and audiovisual TBWs were entered as variables in an effort to predict individual rates of multisensory integration above and beyond what age was able to predict. This model was significantly able to predict McGurk performance (Model 2: p = 0.001, r² = 0.38, F Change_(3,36) = 6.93), with individual variables all predictive in the same direction. Thus, the more acute an individual’s temporal acuity (i.e., lower unisensory thresholds and narrower TBWs), the more the fused percept (i.e., integration) was observed (Visual β = −0.38; Auditory β = −0.29; Audiovisual β = −0.20).

The second hierarchical regression was conducted in the middle-aged and older-adult groups where multisensory integration declined with age. Regression stages were identical to those with younger participants. Age did not significantly predict multisensory integration (Model 1: p = 0.47, r² = 0.003, F_(1,45) = 0.74). The inclusion of visual, auditory, and audiovisual temporal acuity did not significantly improve the prediction of multisensory integration (Model 2: p = 0.83, r² = 0.02, F Change_(3,42) = 0.30; Visual β = 0.04; Auditory β = 0.10; Audiovisual β = −0.16). While temporal processing appears to significantly contribute to the development of multisensory integration, age-related declines in integration are not necessarily driven by declines in temporal perception.

Given the inability of temporal processing to account for these age-related declines in integration, post hoc hierarchical regressions were conducted to explore the possibility that the significant decreases in unisensory speech perception abilities in the middle-age and older-adult groups may contribute, while controlling for changes in temporal acuity. The first model included visual thresholds, auditory thresholds, and audiovisual TBWs, as reported in the regressions above (and thus statistics were identical to those reported above). In the second-level model, auditory and visual speech perception accuracies from McGurk control trials were entered as predictive variables. In the younger groups showing age-related increases in integrative abilities, this additional model did not significantly contribute to the predictive ability of the model (p = 0.58, F-change = 0.55, R²-change = 0.02). Likewise, in the older groups showing age-related declines in integration, this additional model did not significantly contribute to the predictive ability of the model (p = 0.25, F-change = 1.46, R²-change = 0.07).

Discussion

This study explored the relationship between age-related changes in auditory, visual and audiovisual temporal acuity and audiovisual integration as indexed via a common speech-related illusion (i.e., the McGurk effect). The work was predicated on the hypothesis that temporal acuity, as a critical factor for the integration and binding of multisensory signals, would play an instrumental role in shaping the strength of multisensory binding across lifespan (Gordon-Salant & Fitzgibbons, 1993; Stevenson, Segers, et al., 2014; Stevenson, Zemtsov, et al., 2012). Four observations were readily apparent. First, temporal processing, both within and across modalities, improved throughout development and then subsequently declined with healthy aging. Second, the ability to integrate multisensory speech information (i.e., perceive the McGurk illusion) also increased throughout development, and subsequently declined with healthy aging. Third, in development from childhood into adulthood, individual temporal processing abilities can successfully predict the ability to integrate speech information across modalities – a finding that has previously hypothesized but not empirically tested. Finally, and contrary to prediction, while integrative abilities and temporal processing both decline in healthy aging on the group level, temporal processing was not able to predict integration at the individual level during this portion of the lifespan. Together, these results suggest that the impact of temporal processing on multisensory integration varies throughout the lifespan, with the decline in multisensory integration of speech signals in healthy aging likely being driven more by cognitive processes and less by lower-level temporal abilities. Importantly, our data do not provide evidence that either unisensory auditory nor visual temporal acuity seem to show a larger contribution to multisensory integration in speech signals. Future work should examine these possible contributions of unisensory temporal acuity with both simple and more complex speech signals.

Temporal processing and multisensory integration in development

The development of temporal processing of multisensory information has been shown previously to extend well into development (Ernst, 2008; Hillock-Dunn & Wallace, 2012; Hillock et al., 2011). Most germane to the current findings, and using audiovisual multisensory timing tasks similar to those used in this study, reports have show a dramatic increase in temporal acuity between childhood, through adolescence, and into adulthood (Hillock-Dunn & Wallace, 2012; Hillock et al., 2011). These studies used audiovisual SJ tasks in children ranging from 6 to 23 years of age and showed that the ability to accurately assess the temporal relationship of paired auditory and visual cues did not reach their full developmental peak until well into late adolescence. These current data support this finding, and extend them to suggest that the peak for audiovisual temporal acuity is not reached until middle age (40–59 years old). Studies using alternate methods and across different sensory modalities also provide insight and support to these finding (Gori, Del Viva, Sandini, & Burr, 2008; Nardini, Jones, Bedford, & Braddick, 2008). These studies suggest that this late maturation may be due to a prolonged development of optimal integration of multisensory cues (Ernst, 2008), and the similarities in findings across different tasks and sensory modalities suggest such this finding may represent a general aspect of multisensory perception.

The temporal alignment of incoming sensory signals can be utilized as a strong cue to bind these signals into a single, unified percept (Conrey & Pisoni, 2006; Dixon & Spitz, 1980; Meredith et al., 1987; Senkowski, Talsma, Grigutsch, Herrmann, & Woldorff, 2007; B. Stein & Meredith, 1993; Stevenson, Zemtsov, et al., 2012). It has thus been hypothesized that the previously described developmental increase in temporal acuity could contribute to the developmental increase in multisensory integration. We have shown that, in addition to showing that temporal perceptual acuity and multisensory integration do follow a similar time course, that temporal fidelity is significantly predictive of an individual’s ability to integrate, supporting this hypothesis.

The conclusion that temporal acuity and multisensory abilities are interrelated in a developmental context is further buttressed by previous studies of populations with developmental disabilities presenting with impaired temporal processing abilities and whom also exhibit difficulties with multisensory integration (for review, see Wallace & Stevenson, 2014). One example of this is dyslexia. Individuals with dyslexia show decreases in fidelity in temporal processing (Baum, Stevenson, & Wallace, 2015; V. Blau, van Atteveldt, Ekkebus, Goebel, & Blomert, 2009; Farmer & Klein, 1995; Hairston, Burdette, Flowers, Wood, & Wallace, 2005; Van Ingelghem et al., 2001; Wallace & Stevenson, 2014), and likewise exhibit reduced benefit from audiovisual integration (Vera Blau et al., 2010; V. Blau et al., 2009; Hahn, Foxe, & Molholm, 2014; Hairston et al., 2005; Harrar et al., 2014; Barry E Stein, 2012). Additionally, children on the autism spectrum, exhibit decreased temporal acuity relative to their typically developing peers (Bebko, Weiss, Demark, & Gomez, 2006; de Boer-Schellekens, Eussen, & Vroomen, 2013; R. B. Grossman, Schneps, & Tager-Flusberg, 2009; Ruth B Grossman, Steinhart, Mitchell, & McIlvane, 2015; Kwakye, Foss-Feig, Cascio, Stone, & Wallace, 2011; Patten, Watson, & Baranek, 2014; for review, see R. A. Stevenson, M. Segers, et al., 2015; Stevenson, Siemann, Schneider, et al., 2014; Woynaroski et al., 2013), and this same population shows decreased multisensory integration, as measured through the McGurk Effect (Stevenson, Siemann, Woynaroski, et al., 2014b), as well as through other measures of multisensory integration including the sound-induced flash illusion (Stevenson, Siemann, Woynaroski, et al., 2014b) and the pip-pop effect (Collignon et al., 2013). Furthermore, deficits in temporal processing in this population were shown to be predictive of audiovisual integrative abilities (Stevenson, Siemann, Schneider, et al., 2014), though this has not been explored across different age groups, but was limited to children and adolescents.

The current findings suggest that audiovisual temporal processing develops on a concurrent trajectory with multisensory integration, and indeed is a predictive factor for the strength or magnitude of multisensory integration. Taken together with studies using similar methodologies, sensory modalities, and populations, as well as with studies using different tasks, modalities, and populations, these novel data provide support for the hypothesis that during development and well into adulthood, the magnitude of multisensory integration is strongly dependent upon temporal acuity both within and across the auditory and visual modalities.

Temporal processing and multisensory integration in healthy aging

Temporal processing abilities, specifically temporal acuity on unisensory TOJ and audiovisual SJ tasks, decline with aging. This finding is concordant with the work of others for both unisensory (Gelfand et al., 1980; Robin & Royer, 1989; Strouse et al., 1998) and audiovisual (Barsz, Ison, Snell, & Walton, 2002; Chan, Pianta, & McKendrick, 2014c; Fiacconi, Harvey, Sekuler, & Bennett, 2013; Gordon-Salant & Fitzgibbons, 1999; Setti et al., 2011) temporal processing. In the current study we found that auditory and visual acuity peaked in the 20–39 year-old groups, whereas multisensory temporal acuity peaked even later, in the 40–59-year-old group. The late maturation of multisensory temporal processing relative to unisensory is logical, given that multisensory acuity is dependent upon unisensory acuity.

In contrast with the general consensus on age-related changes in audiovisual temporal acuity, the current literature on age-related changes in multisensory integrative abilities is controversial. Our results illustrate an age-related decline in the ability to integrate auditory and visual speech information, as seen by a substantial decrease in the perception of the McGurk Effect. Multiple other studies, however have shown intact or even enhanced multisensory integration in older adults (Hugenschmidt et al., 2009; Peiffer et al., 2007). Indeed, while not the primary focus of this study, examination of the decline in unisensory phoneme perception relative to the apparent retention of perception in congruent audiovisual phonemes suggests that this difference in findings may be task dependent. A recent study of speech-in-noise perception of older adults showed an intact ability to integrate speech signals at the level of individual phonemes but a deficit in integration at the level of whole words (R. A. Stevenson, C. E. Nelms, et al., 2015). One possibility that could reconcile these findings is that whereas the integration of congruent sensory information may remain intact in aging, the integration of incongruent sensory information (as is needed to perceived the McGurk percept) may decline. Such a result may be due to the differential cognitive demands of integrating concordant versus discordant information.

This difference in the association between temporal function and multisensory integration between development and aging was also seen in the predictive ability of temporal processing on multisensory integration. Whereas younger participants’ temporal processing abilities were highly predictive of their multisensory integrative abilities, this was not observed in the two age groups that showed a reduction in multisensory integration with age (middle- and older-age groups). Furthermore, the decline in multisensory integration of speech signals was not predicted by the unisensory decline in speech perception. Taken together, these results provide novel evidence that, unlike what is seen in their younger counterparts, changes in multisensory integration in middle-aged and older adults are not driven by declines in temporal processing.

Conclusions

These results provide a number of novel insights as well as confirming multiple previous findings. Improvements in temporal acuity from childhood to adulthood are able to predict perceptual changes in multisensory integration, while deterioration of temporal acuity throughout the lifespan from middle age to late adulthood cannot predict the concurrent deterioration of multisensory abilities. As has been previously established, temporal processing improved throughout development and then subsequently declined with healthy aging both within and across modalities. These data also suggest that audiovisual temporal processing continues to develop later in life than unisensory temporal processing, peaking in middle age across modalities, while peaking in earlier adulthood within modality. Additionally, multisensory speech integration, as measured by the McGurk Effect, improved throughout development into adulthood, supporting previous research. We found that this same ability declined with healthy aging. Perhaps of greatest importance was the finding that multisensory temporal processing was highly predictive of the increase in multisensory integration of speech in development, but not of the decline in integration seen with healthy aging, suggesting that the role and importance of temporal processing on multisensory integration itself varies throughout the lifespan.

Public Significance Statement.

How we combine information across the senses, known as multisensory integration, greatly influences how we perceive the world around us. One driving factor in multisensory integration is the relative timing of this information – when we sense auditory and visual information at the same time it is a strong indicator that we should perceive these two inputs as a single object or event. This study shows that throughout development, from childhood, through adolescence, and into adulthood, our ability to perceive the timing of sensory inputs accurately drives our multisensory integrative abilities. Contrasting that, as we age, and our ability to perceive the timing of sensory inputs declines, this does not lead to a decrease in our ability to combine sensory information across modalities. Taken together, this suggests that the impact of temporal perception on multisensory integration changes across the lifespan.