Sound support: Intermodal information facilitates infants’ perception of an occluded trajectory

Abstract

In a visual occlusion task, 4-month-olds were given a dynamic sound cue (following the trajectory of an object), or a static cue (sound remained stationary). Infants’ oculomotor anticipations were greater in the Dynamic condition, suggesting that representations of visual occlusion were supported by auditory information.

Given the dynamic nature of the natural environment, infants must be able to learn from correlations within and across sensory modalities in order to perceive the world as predictable and intelligible. Infants’ skill at detecting correlations across multiple sources is well documented (e.g., Lewkowicz, 2000; Morrongiello, Fenwick, & Chance, 1998; Richardson & Kirkham, 2004; Spelke, 1981; Wu & Kirkham, 2010; Wu, Gopnik, Richardson, & Kirkham, 2011), and intermodal information has been shown to facilitate learning in infancy (Bahrick & Lickliter, 2000; Gogate & Bahrick, 1998; Slater, Quinn, Brown, & Hayes, 1999). Sensitivity to intermodal information clearly develops early, and contributes to the development of attention. Here, we asked specifically whether intermodal information facilitates the anticipation of object movement.

The occlusion of trajectories is ubiquitous as objects routinely go in and out of view. Typically in visual occlusion paradigms, infants are habituated to silent (i.e., unimodal) displays, and then tested with full and interrupted trajectories. Results suggest that by 4 months of age infants can perceive a center-occluded trajectory as continuous (Bremner et al., 2005). Infants’ sensitivity to audio-visual correlations within a habituation paradigm suggests that performance is supported by the addition of auditory information (Bremner, Slater, Johnson, Mason, & Spring, 2011); however, whether or not auditory information allows infants to represent an object’s trajectory is not currently known. Perhaps using a method that allows one to measure saccadic anticipations could broaden our understanding of infants’ object representations, and add to the current knowledge.

To examine this question, we targeted an age group (4 months) for which perception of trajectory completion appears to be especially challenging, and for which attempts to facilitate trajectory completion appear to be especially fruitful (Bremner et al., 2005). Within a visual occluded trajectory eye tracking paradigm, we presented infants with either supportive (Dynamic) or less-informative (Static) auditory accompaniment, and reasoned that representation of the trajectory as continuous despite occlusion would be reflected in a consistent pattern of anticipatory eye movements toward the place of reemergence (cf. Johnson, Amso, & Slemmer, 2003; Rosander & von Hofsten, 2002).

Participants were 22 full term 4-month-old infants (11 females; M age = 4.1 months, SD = .2 months). Three additional infants were observed but not included in the analyses due to fussiness and/or poor calibration of point of gaze (POG). Eleven infants were tested in the Static condition, and11 were tested in the Dynamic condition.

Stimuli were presented on a Macintosh G4 computer and 76 cm color monitor, and infants’ eye movements were recorded by an Applied Science Laboratories Model 504 corneal reflection eye tracking system. Three infants were tested using a Macintosh G5 and 152 cm rear projection screen; the size of the display was identical and otherwise there were no differences in presentation. Two computer speakers were placed 122 cm apart behind the front surface of the monitor. Each stimulus consisted of a 30-s animation depicting an approximately 7 cm (4.0° visual angle at the infant’s viewing distance) object translating laterally across 45.4 cm (25.5°) at 18.2 cm/s (10.4°/s). The object changed direction (left-right) every 2.5 s. The center of the trajectory was occluded by a 21.5 cm × 17.7 cm (12.3 × 10.1°) blue box. Ball and occluder were presented against a textured background (a 20 × 12 grid of white dots on black) measuring 48.8 cm × 33.0 cm (27.4 × 18.7°).

In each of the eight trials, consisting of six complete cycles of the object trajectory, a sound was played in tandem with the visual stimulus. Each trial presented a different object and a different sound, and order of both was randomized for each infant (see Fig. 1). In the Static condition, the sound was presented from two side-positioned speakers at the same time and the same volume. In the Dynamic condition when the object was at the far left of the display, the sound emanated entirely from the left-located speaker; as the object moved across the display, the sound became softer in the left-located speaker and louder in the right-located speaker until both object and sound were entirely at the right side of the display. When piloted with adults, this produced the impression that the sound traveled with the object across the display.

Fig. 1.

Examples of two of the eight displays shown to the infants.

Infants were tested individually while seated in a caregiver’s lap 100 cm from the stimulus monitor or screen. Prior to testing, the infant’s eyes were calibrated with a 2-point calibration system. Eye movements were coded for instances of “perceptual contact:” In each of the 96 excursions presented to the infant, an eye movement was entered into the data set if the infant’s point of gaze (POG) was directed toward a region of the display within 1.5° (horizontal) and 3° (vertical) of the moving object trajectory as it was visible on either side of the occluder, after a starting position of the POG outside this region. (Trials in which the POG did not leave the “anticipation region” across object excursions were not counted, as when infants remained fixated on one or the other side of the display.) Eye movements that were initiated less than 150 ms subsequent to object emergence were coded as anticipations, and those initiated later than 150 ms subsequent to object emergence were coded as reactions. The 150-ms criterion was derived from past reports of predictive and reactive eye movements in infants, and is thought to reflect the time it takes for the infant to access an expectancy and then program a saccade (cf. Canfield, Smith, Brezsnyak, & Snow, 1997; Johnson, Amso, et al., 2003; Johnson, Bremner, Slater, Mason, & Cheshire, 2003).

The experiment yielded 984 “codable” eye movements meeting the criteria described previously. Since the number of codable eye movements varied between conditions, following previous research (Johnson, Amso, et al., 2003; Johnson, Bremner, et al., 2003) proportions of anticipations and responses were used. Fig. 2 shows the frequency of anticipatory and reactive eye movements in the Static and Dynamic conditions. The infants in the Dynamic condition produced a higher proportion of anticipations as a function of all codable eye movements (M across infants = 50.3%, SD = 13.6) relative to infants in the Static condition (M = 29.1%, SD = 9.6), t(20) = 4.23, p < .001. Seven of the 11 infants in the Dynamic condition produced more anticipations than reactions, compared to 0 of the 11 infants in the Static condition, χ² = 4.89, p < .05. It is worth noting that our strict coding system possibly underestimates anticipation (e.g., an infant might look over to the side of reemergence, but not get into the allowable area, retreat and then search again, and be coded as reactive).

Fig. 2.

Histograms of the proportions of infants’ eye movement response times relative to the reemergence of the object from behind the occluder (RT = 0). Average proportions, by participant, are reported on the graph. The object was fully occluded at −700 ms, was again fully visible at 400 ms, and remained visible until 1800 ms had elapsed. Anticipations are shown in light grey, and reactions in dark grey. Top panel: Dynamic sounds condition. Bottom panel: Static sounds condition. Infants in the Dynamic condition produced a reliably greater proportion of anticipations relative to infants in the Static condition.

A 2 (condition) × 8 (trial) × 2 (anticipation or reaction) mixed ANOVA revealed a significant effect of trial, F(7, 140) = 3.05, p < .01, due to a decline in codable eye movements across trials, and a significant effect of eye movement, F(1, 20) = 19.23, p < .001, due to a greater number of reactive eye movements overall. There was also a reliable interaction between condition and eye movement, F(1, 20) = 19.21, p < .001, and no other significant effects. Simple effects tests revealed significantly more reactions vs. anticipations by infants in the Static condition, F(1, 20) = 38.44, p < .0001, but no reliable difference in reactions vs. anticipations by infants in the Dynamic condition, F(1, 20) < 1, ns.

We posit that a continuous path was highlighted in the Dynamic condition, providing meaningful auditory information (e.g., Bremner et al., 2011). This additional information supported infants’ ability to predict the trajectory of the object. In the Static condition, without a meaningful auditory cue, infants showed less consistent anticipation, replicating previous findings (Johnson, Amso, et al., 2003; Johnson, Bremner, et al., 2003). Our hypothesis that infant anticipations in the Dynamic condition would be helped by the added cue of sound motion, was supported. It is important to stress that infants were not simply looking in the direction of the sound since anticipations were coded only if their termination occurred within the anticipation region. Orientation toward the speakers would have led to eye movements that terminated far outside the anticipation region. Anticipations were, by definition, directed ahead of both the object and sound location. The proportion of anticipation seen in 4-month-olds in Johnson, Amso, et al. (2003) and Johnson, Bremner, et al. (2003) study (29.5%) is almost identical to that obtained in the current static sound condition (29.1%). This stands in slight contrast to Bremner et al. (2011) who found that a static sound during habituation trials supported perception of trajectory continuity. The effect, however, was not particularly strong, showing up after the first test trial. It is possible that the improved performance in Bremner et al. (2011) condition would not be enough to support anticipatory saccades, and again supports the idea that convergent methods provide even more understanding of infants’ abilities in this domain.¹

The results of the present report suggest that 4-month-olds’ sensitivity to intermodal correlations facilitates their representation of an object’s trajectory. This fits nicely with findings showing that intermodal correspondence can aid in other types of object perception, such as perception of the serial order of moving objects (Lewkowicz, 2004), and that by 3 months of age infants detect amodal temporal synchrony in object events (Bahrick, 2002). In the present study, the synchrony between the moving objects and the moving sounds may have served as an important amodal relation, guiding the infants’ learning about the event. The current research investigated one solution to the problem of interpretation of input, grounded in the fact that infants exist in a stable environment characterized by a wealth of perceptual redundancies. If developing perceptual systems orient toward synchronized multimodal sensory inputs, then infants will necessarily attend to reliable information. Thus, the skill of integrating information across multiple sources becomes a requisite for the developing infant. We suggest that this skill not only points the system in the appropriate direction, but also gives the infant a scaffold upon which to build increasingly complex representations.