Effects of multimodal synchrony on infant attention and heart rate during events with social and nonsocial stimuli

Abstract

Attention is a state of readiness or alertness, associated with behavioral and psychophysiological responses, that facilitates learning and memory. Multisensory and dynamic events have been shown to elicit more attention and produce greater sustained attention in infants than auditory or visual events alone. Such redundant and often temporally synchronous information guides selectivity and facilitates perception, learning, and memory of properties of events specified by redundancy. In addition, events involving faces or other social stimuli provide an extraordinary amount of redundant information that attracts and sustains attention. In the current study, 4- and 8-month-old infants were shown 2-min multimodal videos featuring social or nonsocial stimuli to determine the relative roles of synchrony and stimulus category in inducing attention. Behavioral measures included average looking time and peak look duration, and convergent measurement of heart rate (HR) allowed for the calculation of HR-defined phases of attention: Orienting (OR), sustained attention (SA), and attention termination (AT). The synchronous condition produced an earlier onset of SA (less time in OR) and a deeper state of SA than the asynchronous condition. Social stimuli attracted and held attention (longer duration of peak looks and lower HR than nonsocial stimuli). Effects of synchrony and the social nature of stimuli were additive, suggesting independence of their influence on attention. These findings are the first to demonstrate different HR-defined phases of attention as a function of intersensory redundancy, suggesting greater salience and deeper processing of naturalistic synchronous audiovisual events compared with asynchronous ones.

Introduction

One feature of sensory stimulation that has received growing appreciation for its role in guiding attentional allocation in early development is intersensory redundancy (Bahrick & Lickliter, 2000, 2012; Bremner, Lewkowicz, & Spence, 2012). Intersensory redundancy refers to the simultaneous availability and temporal synchronization of the same information across two or more sensory systems. For example, when the rhythm and tempo of speech can be perceived by looking and listening, they are redundantly specified. Only amodal properties (properties not specific to a particular sensory system such as tempo, rhythm, duration, and intensity) can be redundantly specified across the senses. Natural multimodal events typically provide both redundant amodal information and nonredundant modality-specific information such as color, pitch, and timbre. According to the intersensory redundancy hypothesis (IRH; Bahrick & Lickliter, 2000, 2002, 2014), redundant temporally synchronous information guides selective attention at the expense of nonredundant information in early development, facilitating the perception, learning, and memory of amodal properties (e.g., rhythm, tempo, intensity) of events at the expense of unimodally specified properties. In addition, events with social stimuli, when compared with events with nonsocial stimuli, provide a greater amount of redundant information (across face and voice) that attracts and sustains attention (Bahrick, 2010; Bahrick, Todd, Castellanos, & Sorondo, 2016). The intersensory redundancy provided by events with social stimuli is particularly useful for guiding attention during the first year of life, as infant attention shifts from being exogenous or event driven during the first 6 months to being more endogenous or internally controlled toward the end of the first year (Colombo & Cheatham, 2006; Ruff & Rothbart, 1996).

Behavioral findings in humans and nonhuman animals support the principles of the IRH (see Bahrick & Lickliter, 2012, 2014). However, less is known about how intersensory redundancy affects infant neural responses (e.g., event-related potentials) and physiology (e.g., heart rate). Despite the established benefits of employing converging methods (e.g., Brez & Colombo, 2012; Reynolds, Courage, & Richards, 2010; Reynolds & Richards, 2008), only a few studies of synchrony have attempted to integrate infant event-related potentials (see Hyde, Flom, & Porter, 2016, for a review) and infant heart activity (Pizur-Barnekow, Kraemer, & Winters, 2008) with behavioral measures. These psychophysiological measures of infant attention are necessary for advancing our understanding of the role that redundant multimodal stimulation plays in attention, perception, and memory of events with nonsocial and social stimuli in early development. The current study assessed the effectiveness of synchronous (redundant) and asynchronous (nonredundant) events with nonsocial and social stimuli in attracting and sustaining 4- and 8-month-old infants’ attention using behavioral and physiological indices.

Multimodal synchrony (e.g., simultaneous auditory and visual stimulation) is a fundamental attribute used to coordinate multisensory information in early development (Bahrick & Lickliter, 2002, 2012; Lewkowicz, 2000). For example, 2-month-old infants can integrate auditory and visual information based on synchrony (Bahrick, 1988; Lewkowicz, 1986; Spelke, 1979) and detect synchrony changes in audiovisual events (Bahrick & Lickliter, 2014; Lewkowicz & Kraebel, 2004). Research has consistently demonstrated that infants prefer to attend to events presented synchronously compared with events presented asynchronously (Bahrick, 1988; Dodd, 1979; Spelke, 1979; Spelke, Born, & Chu, 1983). Synchrony has also been shown to facilitate discrimination of rhythm and tempo (Bahrick & Lickliter, 2000; Bahrick, Flom, & Lickliter, 2002). Infants presented with a plastic toy hammer hitting a surface could discriminate changes in rhythm (at 5 months of age) and tempo (at 3 months) when the information was audiovisual and synchronous (redundant), but not when the stimulation was asynchronous or unimodal (Bahrick & Lickliter, 2000). Investigations of intersensory facilitation have been extended to events with social stimuli (see Bahrick, 2010). For example, Flom and Bahrick (2007) showed that 4-month-old infants were able to discriminate affective facial expressions when they were audiovisual and presented in synchrony, but not when they were asynchronous or unimodal (voice or face alone). Only later in development could infants discriminate affective expressions when they were unimodal. In addition to providing further support for the IRH, this work highlights the important role of intersensory redundancy, across voice and face, for infant perception of social stimuli.

Social stimuli, defined by the presence of people or animate objects, are essential sources of stimulation that guide perceptual, cognitive, social, and linguistic development. It is proposed that social stimuli provide a greater amount of intersensory redundancy (across rapid changes in face, voice, and gesture) compared with nonsocial events (Bahrick & Lickliter, 2014). Events with social stimuli have been shown to have an attentional advantage over events with nonsocial stimuli during infancy. For example, Courage, Reynolds, and Richards (2006) found greater behavioral attention (longer looks) and physiological attention (greater changes in heart rate) to silent dynamic events that were social (Sesame Street scenes) compared with nonsocial (geometric patterns). Recently, Bahrick et al. (2016) extended this work to multisensory naturalistic events by examining attention to audiovisual faces and voices, audiovisual objects, silent dynamic faces, and silent dynamic objects. They found greater attentional maintenance to speaking faces compared with other event types for 4- and 5-month-olds and 6- to 8-month-olds. One likely explanation for this attentional advantage is the intersensory redundancy provided by face–voice synchrony.

Only a few studies have examined the effect of synchronous and asynchronous events on infant neural responses (see Hyde et al., 2016, for a review; Hyde, Jones, Flom, & Porter, 2011; Kopp, 2014; Kopp & Dietrich, 2013; Reynolds, Bahrick, Lickliter, & Guy, 2014). Hyde et al. (2011) investigated the neural basis of face–voice synchrony in 5-month-old infants. They presented infants with a static face and a voice saying “hi” in synchrony or with a 400-ms delay in the appearance of the face (asynchrony). In a second experiment, Hyde et al. presented dynamic faces paired with the soundtrack “Oh, hi baby.” In the synchronous condition the face matched the soundtrack, but in the asynchronous condition the face mouthed the phrase “You’re such a beautiful baby.” In a similar study, Reynolds et al. (2014) presented infants with dynamic videos of a woman saying “Come over here by me!” or “Where’s the baby going?” in three conditions: unimodal visual, audiovisual synchronous (matching soundtrack), and audiovisual asynchronous (mismatched soundtrack). Both studies found a difference in the amplitude of the negative central (Nc) component (indicative of attentional engagement) for synchrony versus asynchrony—but in different directions. Hyde et al. (2011) found a greater amplitude response to asynchrony, whereas Reynolds et al. (2014) found a greater amplitude response to synchrony, likely due to procedural differences across the studies. Furthermore, both studies found similar effects of the late positive slow wave (PSW; indicative of recognition memory), with dynamic synchronous faces and voice processed more deeply than asynchronous faces and voice. Together, these findings confirm behavioral results; multimodal events with social stimuli are highly salient and capable of attracting and sustaining infant attention when presented in synchrony (e.g., temporally coordinated faces and voices). Furthermore, these studies emphasize the value of using psychophysiological measures to further our understanding of the attentional processes underlying the effects of intersensory redundancy.

Infant attentional processes might not be directly reflected in behavioral data alone because a single measure (i.e., look duration) is often used to represent multiple processes (e.g., Brez & Colombo, 2012; Richards, 1985, 1989, 1997, 2003). Supplementing behavioral data with psychophysiological measures, such as neural responses and heart rate (HR), allows for a better understanding of attentional processes. For example, Richards and Casey (1991, 1992) identified three HR-defined phases of looking that correspond to different levels of information processing that occur over time when an infant attends to an event. At the onset of an event, the infant begins orienting (OR) toward potentially important sources of information. If the infant selects a novel or salient event for further exploration and learning, OR is followed by sustained attention (SA), which is marked by infant looking accompanied by a deceleration of HR. The SA phase often reflects active processing and can indicate that the infant has reached an attentional state (Graham & Clifton, 1966; Richards, 1985). This phase of decelerated HR is maintained until the infant is no longer in an engaged attentional state. The attention termination (AT) phase, defined by the return of HR to prestimulus levels even though looking may continue, signifies the disengagement of attention.

Simultaneous measurement of visual behavior and HR is noninvasive, is relatively inexpensive, and has been validated across a range of event types (Reynolds & Richards, 2008). However, to date no studies have explored the effects of synchrony on infant attention with HR-defined phases of attention. Pizur-Barnekow et al. (2008) conducted a pilot study examining the relationship between infants’ visual attention and cardiac vagal tone (i.e., respiratory sinus arrhythmia [RSA]) to synchronous versus asynchronous auditory and visual nonsocial stimuli. They presented 5-month-old infants with shapes moving across a screen from top to bottom, bottom to top, and left to right. The shapes were each arbitrarily paired with a sound (circle–chime, rectangle–camera click, triangle–typewriter click). In the synchronous condition there was onset and offset synchrony between the visual image and the sound, and in the asynchronous condition there was a 1-s delay between the onset and offset of the shape and the sound. Although looking behavior did not differ for synchronous and asynchronous stimuli, RSA was significantly higher during the synchronous condition compared with the asynchronous condition. These findings suggest that auditory and visual images presented in synchrony and out of synchrony with a sound elicit physiological changes that may be associated with differing levels of attention in infants.

However, it is not known how findings from studies presenting arbitrarily paired images and sounds generalize to naturalistic dynamic audiovisual events. Naturalistic audiovisual events provide multiple levels of temporal structure common across sights and sounds (e.g., temporal macrostructure and microstructure; Bahrick, 2001; Bahrick & Lickliter, 2012) unavailable in images paired with sounds. For example, audiovisual speech not only provides synchrony between the onset and offset of the movement of the face and sound of the voice (macrostructure) but also creates a more complex nested level of temporal synchrony between the specific movements of the lips and sounds of speech (microstructure). This temporal microstructure makes common rhythm, tempo, and intensity patterns available across the movements of the face and sounds of the voice, amodal properties not available in images paired with sounds. Thus, it is not known whether intersensory redundancy (i.e., temporal synchrony) in naturalistic dynamic events elicits different HR-defined phases of attention compared with nonredundant (asynchronous) stimulation.

The current study was designed to examine the effects of synchrony on behavioral and physiological indices of infant attention of 4- and 8-month-olds. These ages are ideal for exploring the effects of stimulus events before and after the transition from exogenous attention to endogenous attention. The goal was to determine whether dynamic events providing synchronous and asynchronous audiovisual stimulation differentially affect infant looking, HR (beats per minute [bpm]) changes, and HR-defined phases of attention. We were particularly interested in the sustained attention (SA) phase, specifically the proportions of time spent in SA and the amount of change in HR from baseline. The deceleration in HR during SA was expected to change depending on the level of engagement with each condition and stimulus type (Richards & Casey, 1992). We predicted that the intersensory redundancy provided by synchronous audiovisual events compared with asynchronous ones would produce longer look durations, greater proportions of time spent in a state of SA, and greater changes in HR during SA. A second goal of the current study was to compare attentional indices for events with social stimuli (e.g., women speaking) and nonsocial stimuli (e.g., objects striking a surface). We expected that the saliency of faces and voices would provide an attentional advantage for social stimuli over nonsocial stimuli, indicated by longer looking, a larger percentage of time spent in SA, and a larger decline in HR during SA. Finally, because the factors of synchrony and stimulus type were manipulated simultaneously, we thought that we would be able to determine whether the effects of these factors, if they did exist, were independent of one another or interacted with one another in some way.

Method

Participants

A total of 80 4-month-old (n = 38) and 8-month-old (n = 42) infants were recruited from the greater Kansas City metropolitan area in the midwestern United States. This sample was drawn from a predominantly upper-middle-class population; the sample was 89% White non-Hispanic, 6% Asian, 4% Hispanic, and 1% American Indian. Of the original sample, 16 infants were excluded from the final analyses because of fussiness (n = 7), equipment failure (n = 3), or gestation length less than 37 weeks (n = 6). After these exclusions, 64 healthy infants with no medical history involving auditory or visual problems comprised the final sample. The mean age of the 4-month-olds (20 girls and 12 boys) was 117.97 days (SD = 10.07). The mean age of the 8-month-olds (15 girls and 17 boys) was 145.72 days (SD = 14.51).

Apparatus

During testing, infants were seated in a car seat approximately 112 cm away from a 30-inch (76 cm) monitor in a room with black walls and ceiling. All stimuli were presented using Windows Media Player at infants’ midline and at approximately 65 dB. A video camera placed at the base of the monitor recorded and transmitted an image of infants’ face to an adjacent room, where a trained observer coded the direction and duration of infants’ looks to the stimulus by pressing a button that timed looks, recorded accumulated time, and interfaced with the HR data acquisition system (see below). All sessions were recorded on DVD.

Infants’ HR was measured with shielded silver–silver chloride (Ag–AgCl) electrodes placed on either side of the chest and grounded with an unshielded electrode just above the navel. The electrocardiogram (ECG) was digitized using a data acquisition interface and a second computer running software from a commercial data acquisition system (BioPac, Santa Barbara, CA, USA) configured for psychophysiological recording, with a sampling rate of 250 Hz. The data acquisition interface also received input from the button used to record looking data and mark stimulus onset, so that the HR file was synchronized with stimulus events and the coding of visual fixations.

Procedures

Procedures were conducted in keeping with American Psychological Association (APA) standards of ethical treatment of human research participants and were approved by the University of Kansas institutional review board. On participants’ arrival at the lab, the experimental procedures were explained to parents and informed consent was obtained. Parents then completed a demographic and health questionnaire. Electrodes were placed on the chest and abdomen of infants to obtain measures of HR prior to and during the session. The length of the presession baseline period varied somewhat, but its average was 33.52 s (SE = 2.33). Infants were secured in the car seat to reduce motion, the lights were dimmed, and the session began. Parents remained with the infants in the testing room but were instructed to stand behind the car seat and avoid distracting the infants. Each infant was presented with a 2-min multimodal synchronous or asynchronous video involving events that featured social or nonsocial stimuli. Condition (synchronous vs. asynchronous) and stimulus type (social vs. nonsocial) were randomly assigned between participants within each age group.

Stimulus events

The social stimulus event depicted an adult female actress (recorded from the shoulders up) reciting three phrases with positive affect (“Look at you!”; “Come over here by me!”; and “Where’s the baby going?”) in a continuous loop of infant-directed speech (see Flom & Bahrick, 2007). The nonsocial stimulus event was also used in prior studies (Bahrick & Lickliter, 2000; Bahrick et al., 2002; Pickens & Bahrick, 1995), and depicted a red hammer moving up and down, tapping on a wooden surface at 240 beats per minute. The rhythm was x o xx x, where x is a whole-beat impact, o is a whole-beat rest, and xx is two half-beat impacts (Pickens & Bahrick, 1995). The synchronous condition consisted of dynamic videos with temporally matching soundtracks, whereas the asynchronous condition consisted of dynamic videos with mismatching soundtracks. In the asynchronous condition, the soundtracks were delayed with respect to the videos. For the social events, they were delayed by approximately 7 s such that the video depicting one phrase was heard with the soundtrack from a different phrase. For the nonsocial events, the soundtrack was delayed by approximately 0.5 s such that the visual impacts of the hammer did not coincide with the auditory impacts. Thus, the synchronous videos provided amodal temporal macrostructure and microstructure, whereas the asynchronous videos provided neither.

Behavioral and heart rate measures of attention

Infant attention, often characterized by state of readiness or alertness, is associated with behavioral and psychophysiological responses (Colombo, 2001; Colombo, Richman, Shaddy, Greenhoot, & Maikranz, 2001; Richards & Cameron, 1989). Attention was evaluated across the 2-min viewing session using behavioral and HR measures as described. Behavioral data, look onsets and offsets, were coded in real time during the session by a trained observer, not blind to condition and stimulus type, and integrated into the HR files. A second observer, blind to condition and stimulus type, rescored 25% of the sessions from recordings of the live sessions. Interobserver reliability was calculated by a Pearson product–moment correlation. We analyzed two dependent variables of behavior: average look duration and peak look duration. Average look duration was the average or mean length of looks during the 2-min viewing session, whereas peak look duration was the duration of the longest look during the session. The interobserver reliability for duration of average look was .96 and for duration of peak look was .99.

Infants’ ECG was converted from graphical representation into a numerical data file for analysis using BioPac software that stored the time code of the R waves from the digitized ECG. The time codes from stimulus event onsets and infant behaviors (look onsets and offsets) were interleaved among the R-wave time stamps to provide a complete sequential record of infants’ viewing session. Using custom software, infants’ looking was parsed into three HR-defined phases of attention: Orienting (OR), sustained attention (SA), and attention termination (AT) (Richards, 1985). Orienting was defined as the period of looking before significant HR decelerations. Sustained attention was defined as looking accompanied by HR decelerations of at least 5 consecutive beats below the median HR obtained during presession or preattention baseline (Richards, 1997). Each time infants looked away from the screen, preattention baseline HR was recalculated using median HR during the entire look away; HR-defined phases of attention for each look were calculated using the preattention baseline HR established during the previous look away. Attention termination was defined as looking that continued after SA but during which HR returned to at least the median preattention baseline level. Given that behavior might not be directly related to attention, we expected changes in HR and the proportion of time spent in the HR-defined phases to provide measures of attentional engagement with each condition and stimulus type. Dependent HR measures included (a) median HR (in bpm) before and during the viewing session, (b) proportion of time spent in the three HR-defined phases of attention across the viewing session, and (c) mean HR decelerations during SA, defined as mean change in HR from preattention baseline.

Results

Behavioral measures of attention

Behavioral measures of attention, average and peak look duration during the viewing session, were analyzed to examine the effects of condition and stimulus type. Analyses were 2 (Age: 4 vs. 8 months) × 2 (Condition: synchronous vs. asynchronous) × 2 (Stimulus Type: social vs. nonsocial) univariate analyses of variance (ANOVAs). Looking time data were positively skewed; therefore, data were log transformed prior to the analyses (means reported are based on raw data). ANOVAs based on raw data revealed the same patterns of significance. There was a main effect of stimulus type for average look duration, F(1, 56) = 17.10, p < .001, ${\eta }_{\text{p}}^2=.23$; infants spent more time looking, on average, during events with social stimuli (M = 16.84, SD = 18.50) than during events with nonsocial stimuli (M = 5.62, SD = 4.50). There were no main effects or interactions involving age or condition for average look duration. There was also a significant main effect of stimulus type for peak look duration, F(1, 56) = 12.43, p = .001, ${\eta }_{\text{p}}^2=.18$. Consistent with the main effect of stimulus type for total look duration, infants had longer peak looks to events with social stimuli (M = 39.16, SD = 27.88) compared with events with nonsocial stimuli (M = 21.99, SD = 18.84). No other significant main effects or interactions emerged for peak look duration. The lack of age effects was unexpected; however, previous research has demonstrated that developmental effects on attention vary depending on stimulus characteristics (Reynolds, Zhang, & Guy, 2013; Shaddy & Colombo, 2004).

Heart rate measures

We examined the effects of age, condition, and stimulus type on median HR during the viewing session, relative to baseline, with a 2 (Age) × 2 (Condition) × 2 (Stimulus Type) × 2 (HR: presession vs. during session) mixed-design ANOVA, with repeated measures on the final variable. Mauchly’s test of the violation of the assumption of sphericity was conducted for all mixed ANOVAs. Where applicable, the Huynh–Feldt correction was employed. There were significant main effects of age, F(1, 55) = 21.21, p < .001, ${\eta }_{\text{p}}^2=.28$, and stimulus type, F(1, 55) = 5.43, p = .023, ${\eta }_{\text{p}}^2=.09$. Older infants had lower HR (M = 134.60, SD = 10.46) than younger infants (M = 146.07, SD = 12.64). In addition, HR was lower for events with social stimuli (M = 136.55, SD = 12.25) compared with events with nonsocial stimuli (M = 144.31, SD = 12.47). This main effect was qualified by a significant Stimulus × Type HR interaction, F(1, 55) = 10.25, p = .002, ${\eta }_{\text{p}}^2=.16$ (see Fig. 1). Subsequent pairwise comparisons using Bonferroni-corrected t tests revealed that although HR was similar during baseline (p = .23), it was significantly lower for infants who viewed social stimuli compared with those who viewed nonsocial stimuli (p = .003). In addition, whereas HR decreased marginally during the task for the social stimulus event (p = .071), it increased during the nonsocial stimulus event (p = .01). There were no other significant main effects or interactions.

Fig. 1.

Heart rate with standard error bars as a function of data collection period for events with social and nonsocial stimuli.

Heart-rate-defined measures of attention

Looking behavior across the viewing session was parsed into HR-defined phases of attention (i.e., OR, SA, and AT). Proportion of time spent in the HR phases was analyzed with a 2 (Age) × 2 (Condition) × 2 (Stimulus Type) × 3 (HR-Defined Phase: OR vs. SA vs. AT) mixed-design ANOVA, with repeated measures on the final variable. There were no main effects of age, condition, or stimulus type. There was a significant main effect of HR-defined phase, F(1.44, 80.49) = 90.67, p < .001, ${\eta }_{\text{p}}^2=.62$. All infants spent significantly more time in SA compared with OR and AT (ps < .01). This effect was qualified by a significant Condition × HR-Defined Phase interaction, F(1.44, 80.49) = 5.14, p = .015, ${\eta }_{\text{p}}^2=.08$ (see Fig. 2). Subsequent pairwise comparisons using Bonferroni-corrected t tests revealed that infants spent less time in OR during the synchronous condition compared with the asynchronous condition (p = .009). In addition, infants spent marginally more time in an SA and AT in the synchronous condition than in the asynchronous condition (ps = .073 and .051, respectively). Because AT data were positively skewed, we conducted an additional t test using 20% trimmed means (robust to violations of normality; Wilcox, 2017). Results indicated that infants in the synchronous condition did spend significantly more time in AT compared with those in the asynchronous condition (p = .009). There were no significant interactions with regard to stimulus type.

Fig. 2.

Proportion of looking time with standard error bars as a function of heart rate (HR)-defined phase of attention—Orienting (OR), Sustained Attention (SA), or Attention Termination (AT)—for the synchronous and asynchronous conditions.

A 2 (Age) × 2 (Condition) × 2 (Stimulus Type) ANOVA was conducted on mean HR change, from baseline, during SA to determine the effects of condition and stimulus type on the amplitude of SA HR decelerations. There was a significant main effect of condition, F(1, 56) = 4.37, p = .041, ${\eta }_{\text{p}}^2=.07$; the synchronous condition resulted in deeper HR decelerations during SA (M = 9.11, SD = 4.32) than the asynchronous condition (M = 6.79, SD = 4.10). Mean HR during preattention baseline, OR, SA, and AT is presented in Fig. 3.

Fig. 3.

Heart rate as a function of data collection period—Preattention Baseline, Orienting (OR), Sustained Attention (SA), or Attention Termination (AT)—for the synchronous and asynchronous conditions.

Discussion

The current study investigated the effects intersensory redundancy on infant attention. Our primary goal was to examine the effects of synchronous (redundant) and asynchronous (nonredundant) events on behavioral and physiological indices of infant attention. We hypothesized that the presentation of a multimodal synchronous event would attract and hold infants’ attention and induce an attentional state more effectively than a temporally asynchronous event. Our physiological results lent support to this hypothesis. Infants in the synchronous condition spent less time in OR (HR decelerated to SA more quickly) and more time in AT compared with infants in the asynchronous condition. Although the difference in the proportion of time spent in SA for the synchronous condition compared with the asynchronous condition did not reach significance, the synchronous condition did elicit more attentional engagement than the asynchronous condition, as indicated by greater HR decelerations. Given the saliency of speaking faces and infants’ preference for social stimuli compared with nonsocial stimuli (e.g., Courage et al., 2006; Reynolds et al., 2014), our second goal was to compare behavioral and physiological indices of attention for events with social and nonsocial stimuli. We found the predicted attentional advantage for social stimuli; events with social stimuli produced longer duration of average and peak looks and lower HR compared with events with nonsocial stimuli. Interestingly, the influence of synchrony and the social nature of stimuli on attention were additive, suggesting independent underlying processes.

Effects of multimodal synchrony

Behavioral measures did not support our hypothesis that intersensory redundancy would facilitate attention to a greater extent than nonredundant stimulation. Infant look durations were not different during the synchronous and asynchronous conditions. One explanation for this finding is that the viewing session used in the current study was not as sensitive to changes in synchrony as the habituation or selective attention procedures used in previous studies that reported behavioral differences (e.g., Bahrick & Lickliter, 2000; Flom & Bahrick, 2007). Indeed, null behavioral results were obtained in similar studies using accumulated looking procedures (Reynolds et al., 2014) and slideshow presentations (2-min viewing sessions; Pizur-Barnekow et al., 2008). Reynolds et al. (2013) found no differences in look duration (i.e., average and peak) to synchronous and asynchronous stimuli during 20 s of accumulated looking. Furthermore, Pizur-Barnekow et al. (2008) found that look duration (i.e., total and frequency) did not differ during synchronous and asynchronous slideshows (2-min viewing sessions). Despite finding no behavioral differences, Pizur-Barnekow et al. (2008) reported physiological differences; cardiac vagal tone (i.e., RSA) was greater during the synchronous session compared with the asynchronous session. In addition, a number of studies report neural differences in the effects of synchrony on infant attention (Hyde et al., 2011; Kopp & Dietrich, 2013; Kopp, 2014; Reynolds et al., 2014). Thus, psychophysiological measures appear to be more sensitive to the effects of synchrony than behavioral measures. In particular, parsing looking behavior into HR-defined phases of attention provides a more sensitive measure of attentional engagement than using behavior alone (Brez & Colombo, 2012).

Findings from the current study are the first to provide support for the IRH at the physiological level using HR-defined phases of attentional engagement. The proportions of time that infants spent in the HR-defined phases varied as a function of condition regardless of age and stimulus type. The primary phase of interest was SA. Although the proportions of time spent in SA during the synchronous and asynchronous conditions were only marginally different, they were accompanied by a significantly greater decline in HR during SA in the synchronous condition. Infants in the synchronous condition spent less time in OR, indicating that they began to process and engage with the events more quickly than infants in the asynchronous condition. In addition, when events were shown synchronously as opposed to asynchronously, infants spent more time in AT, suggesting that they had more difficulty in disengaging from the synchronous stimulus. These results mirror those of behavioral and neural studies; redundant synchronous events elicited more attentional engagement and active stimulus processing compared with events that are asynchronous (see Bahrick & Lickliter, 2012, 2014; Hyde et al., 2016). These findings are also consistent with the view that the enhancing effect of intersensory redundancy on infant perception, learning, and memory for amodal properties of stimulation is in part based on attentional salience of amodal information.

Regarding psychophysiological findings, Reynolds et al. (2014) found a greater amplitude of the Nc component, indicative of attentional engagement, following synchronous audiovisual presentations compared with asynchronous ones. In addition, Hyde et al. (2011) and Reynolds et al. (2014) obtained similar effects of face–voice synchrony on late PSW associated with greater recognition memory, suggesting that synchronous stimuli were processed more deeply than asynchronous stimuli. The lack of convergence between behavioral and physiological responses in the current study and others (Brez & Colombo, 2012; Pizur-Barnekow et al., 2008) highlight the importance of evaluating infant attention with multiple indices (e.g., behavior and psychophysiology).

Effects of social stimuli

In the current study, we found an attentional advantage for events with social stimuli relative to those with nonsocial stimuli; this was consistent with our predictions. Infants’ average and peak look durations were longer to the dynamic multimodal speaking face compared with the tapping hammer regardless of age or condition. These results support the well-established social preference that infants develop by 4 or 5 months of age (Bahrick et al., 2016; Bahrick, 2010; Courage et al., 2006). In addition to behavioral differences, infants’ HR was significantly lower during the session for the social stimulus compared with the nonsocial stimulus; lower HR is often associated with more active engagement and stimulus processing. Although HRs were similar prior to the session, we found an interesting pattern of HR changes during the session—the social stimulus elicited a decrease in HR that was only marginally significant, whereas the nonsocial stimulus produced a significant increase in HR. The differences in the direction of HR changes following the presession may indicate varying levels of engagement and sustained attention during the session. Accelerations in HR, much like those observed for infants who were presented with the nonsocial stimulus, are usually associated with increased arousal. Although the exact interpretation of this increase is unclear, it certainly is not consistent with improved levels of processing or engagement. Our hypothesis that social and nonsocial stimuli would differentially influence HR-defined phases of attention was not supported. Results showed that the proportion of time spent in each of the phases did not vary as a function of stimulus type. Courage et al. (2006) found that when infants viewed silent dynamic events that were social compared with nonsocial, they spent a larger proportion of time in SA. Although we did not find the same increases in SA during the social stimulus or any other differences in the pattern of HR-defined phases based on stimulus type, the overall HR results indicate that infants were more engaged during the social stimulus compared with the nonsocial stimulus. One important difference between the studies is that Courage et al.’s (2006) stimuli were silent, whereas the stimuli used in the current study were multimodal.

Independent effects of multimodal synchrony and social stimuli

In the current study, there were no interactions between condition and stimulus type, indicating that the influences of synchrony and the social nature of the stimulus event were additive. Synchrony had a similar impact on HR-defined phases and HR decelerations during SA regardless of stimulus type (social or nonsocial). In addition, the advantage for events with social stimuli, longer look duration, and lower HR, were not affected by synchrony. The finding of parallel effects of synchrony on attention to social and nonsocial events is not surprising given that stimulation from social events is similar to that from nonsocial events (providing amodal temporal macrostructure and microstructure); however, social events may be more engaging because they provide a greater amount of stimulation and variability (Bahrick & Lickliter, 2012, 2014).

Summary and conclusions

Findings from the current study provide new information about the effects of intersensory redundancy at the physiological level using HR-defined phases of infant attention. When stimuli were redundant, with auditory and visual stimulation presented synchronously compared with asynchronously, infants had faster (less OR) and deeper (during SA) HR decelerations and more difficulty in disengaging their attention (more AT). These findings reveal the attentional salience of intersensory redundancy at the physiological level and are consistent with predictions of the IRH—temporally synchronous (redundant) information guides selectivity at the expense of nonredundant information and facilitates perception, learning, and memory of amodal properties of events. We also assessed the influence of naturalistic social (face–voice) stimuli compared with nonsocial (objects) stimuli on behavioral and physiological indices of infant attention. The findings were consistent with the hypothesis that social stimuli provide infants with an attentional advantage, longer average and peak look durations, and lower HR compared with nonsocial stimuli. In conclusion, this study shows that synchronous and social stimuli that are dynamic and multimodal independently enhance infant attention.