Relational congruence facilitates neural mapping of spatial and temporal magnitudes in preverbal infants

Highlights

• •Infants show a relational congruence bias in learning spatial and temporal pairings.
• •Relational congruence bias is not dependent on redundancy within or across the senses.
• •Representations of space and time interact in infancy, before schooling or language.

Abstract

Mental representations of space, time, and number are fundamental to our understanding of the world around us. It should come as no surprise that representations of each are functional early in human development, appear to share a common format, and may be maintained by overlapping cortical structures. The consequences of these similarities for early learning and behavior are poorly understood. We investigated this issue by assessing neurophysiological processing of audio-visual temporal and spatial magnitude pairs using event-related potentials (ERPs) with young infants. We observed differential early processing and later enhanced attentional processing for pairings of spatial and temporal magnitudes that were relationally congruent (short visual character paired with a short auditory tone or long visual character paired with a long auditory tone) compared to the same stimuli paired in a relationally incongruent manner (short visual character with the long auditory tone or long visual character with a short tone). Unlike previous studies, these results were not dependent on a redundancy of information between the senses or an alignment of congruent magnitude properties within a single sense modality. Rather, these results demonstrate that mental representations of space and time interact to bias learning before formal instruction or the acquisition of spatial language.

1. Introduction

Immanuel Kant (1781/1998) proposed that a primitive understanding of space, time, and number is innate and forms the basis of later learning and experience. Kant's philosophical assertion has garnered support in light of research showing that preverbal human infants, as well as many non-human animals, have the cognitive capacity and underlying neural structure to support mental representation of space, time, and number (Dehaene and Brannon, 2010, Feigenson, 2007, Lourenco and Longo, 2011). Furthermore, research with human adults demonstrates that mental representations of space, time, and number interact with each other (e.g. Casasanto and Boroditsky, 2008, Cohen Kadosh et al., 2007, Pinel et al., 2004, Walsh, 2003). For instance, Casasanto and Boroditsky (2008) found that judgments made regarding the temporal duration of visual stimuli are influenced by the physical length of the stimuli. Likewise, past work (e.g. Algom et al., 1996, Cohen Kadosh et al., 2007, Tzelgov et al., 1992) with the “numerical Stroop task” has shown that adults’ judgments of numerical size (e.g. which is numerically larger?) are influenced by the physical size of the Arabic digits (i.e. 2 vs. 7). These interactions suggest that representations of one dimension (e.g. size) activate representations in the other (e.g. number). Unfortunately, the origin of these interactions remains hotly debated and largely unknown. One possibility is that interactions between distinct representations of space, time, and number arise as a result of repeated associations between particular pairings of space, time, and/or number in the context of sensory-motor experience, language, or culture (Boroditsky, 2000, Boroditsky, 2011, Gelman and Williams, 1998, Johnson, 2003, Piaget, 1927). Alternatively, interactions between representations of space, time, and number might arise because of an innate link, heretofore referred to as a functional overlap, between these dimensions (e.g. Cantlon et al., 2009, Pinker, 1997, Srinivasan and Carey, 2010, Walsh, 2003). Some have hypothesized that innate functional overlap could be realized as an initially privileged relationship between distinct representations of space, time, and number (e.g. van Marle and Wynn, 2006), while others hypothesize that innate functional overlap could be realized as an generalized magnitude system, which may later become differentiated over development (e.g. Lourenco and Longo, 2010; or see Lourenco and Longo, 2011 for a review). In either case, the current study tests for the presence of functional overlap in 5-month old infants, before associations between space, time, and number could likely be culturally, linguistically, or educationally constructed.

Three lines of evidence suggest a similarity, or overlap, in the way space, time, and number are mentally represented. First, these representations appear to share a common analog format, where quantities are represented as approximate physical magnitudes proportional to that being represented (see Meck and Church, 1983, Walsh, 2003). Second, representations of space, time, and number have common or signature behavioral limits (see Dehaene and Brannon, 2010 for a review). For example, the ability of human infants, human adults, and many non-human animals to compare two magnitudes, whether they be spatial, temporal, or numerical, is limited by the ratio of the two magnitudes being compared rather than the absolute difference between them (e.g. Brannon et al., 2006, Brannon et al., 2008, Lipton and Spelke, 2003, van Marle and Wynn, 2006). Infants of the same age make comparisons at about the same level of precision, or ratio, regardless of the magnitude domain (e.g. Brannon et al., 2006, Brannon et al., 2007, Feigenson et al., 2004). While the particular precision of each domain diverges in later childhood, all domains show continued gains in precision into adulthood (e.g. Droit-Volet et al., 2008, Halberda and Feigenson, 2008). Third, representations of space, time, and number have also been hypothesized to share a common neural substrate (Fias et al., 2003, Izard et al., 2009, Meck and Church, 1983, Walsh, 2003). For example, neuroimaging and neurophysiological work suggests overlap in regions of the intraparietal sulcus (IPS) involved in the representation and/or comparisons of number and physical size (i.e. area or length) (e.g. Pinel et al., 2004, Cohen Kadosh et al., 2007, Cohen Kadosh and Henik, 2006). In sum, while evidence suggests behavioral and neural similarities in the way space, time, and number are represented, the functional significance of this similarity for learning remains unclear. In the current study, we ask if similarity between representations of space and time biases early learning. Specifically, can young 5- to 6-month-olds learn magnitude relationships between dimensions of space and time more readily when they are relationally congruent compared to when they are relationally incongruent?

Several recent studies suggest that the overlap between representations of space, time, and number plays a functional role in learning, especially early in development (deHevia and Spelke, 2010, Lourenco and Longo, 2010, Mix et al., 1999, Mix et al., 2002, Srinivasan and Carey, 2010). For example, work with pre-school and elementary school children suggests that early numerical calculation abilities may be aided by corresponding cues to spatial magnitude, or extent (Mix et al., 2002). Similarly, understanding of fractions may be facilitated by overlap in the way in which numerical and spatial magnitudes are represented (e.g. Mix et al., 1999). Again, like with human adults, explicit instruction, linguistic metaphor, or cultural experience may explain the facilitation of learning of children in these contexts.

Recent work with pre-verbal infants suggests the learning may be facilitated between dimensions before such experience. For example, Lourenco and Longo (2010) demonstrated that infants are able to form interchangeable associations between number and space and time in learning which stimuli go together. Additional work shows that infants learn associations between relationally congruent line lengths and numerical quantities more readily (smaller numbers of objects associated with shorter line lengths/larger numbers of items associated with longer lines) than they learn relationally incongruent pairings of the same stimuli (shorter lines with larger numbers of items/longer lines with smaller number of items) (deHevia and Spelke, 2010). In both of these experiments (i.e. deHevia and Spelke, 2010, Lourenco and Longo, 2010), however, pairings were made only over visual dimensions. Thus, it is unclear whether infants made the associations because of functional overlap between abstract representations of space, time, and number, an abstract generalized magnitude, or whether the results were due to common visual-specific properties of the stimuli.

In an attempt to discern whether the ability to learn associations between space and time was a product of functional overlap between abstract mental representations or whether previous results are due to common visual properties, Srinivasan and Carey (2010) required such associations between space and time to be made across two different sense modalities. Srinivasan and Carey (2010) found that infants are better at learning associations between visually presented length (space) and temporal duration of an auditory tone (time) when they are relationally congruent (e.g. relatively long duration tone with long line, short duration tone with short line) than when they are relationally incongruent. Moreover, Srinivasan and Carey (2010) argue that the functional overlap between abstract representations of space and time allow the correspondences between them to be spontaneously aligned, facilitating the learning for relationally congruent pairings but not for incongruent pairings.

While the results of Srinivasan and Carey (2010) support their position, it should be noted that temporal magnitude information was conveyed both aurally and visually. That is, the duration of stimulus was redundant across the senses, or could be obtained both visually and aurally. Spatial and temporal information was also presented in perfect temporal synchrony from stimulus onset to stimulus offset. This is significant because related work has shown that, within the domain of number as well as other domains, redundant and temporally synchronous information conveyed across multiple senses enhances learning compared to bimodal presentation without redundancy (e.g. Bahrick and Lickliter, 2000, Jordan et al., 2008, Flom and Bahrick, 2010). Therefore, it is unclear if the ability to learn congruent pairings in Srinivasan and Carey (2010) was dependent on the redundancy of temporal magnitude information across both sense modalities (see Bahrick and Lickliter, 2000, Bahrick et al., 2004 for reviews). Alternatively, despite using audio-visual stimuli, differential learning between congruent and incongruent pairs in Srinivasan and Carey (2010) could also have taken place purely within the visual domain. Visual length and visual duration changed as a function of condition. It is plausible that the duration of visual stimulation was mapped to the spatial extent of the visual stimulation and this mapping was easier when the temporal and spatial information was congruent compared to when it was incongruent. In this case, auditory information could be completely ignored and still produce the observed results. If this is so, it is still unclear whether differential learning occurs as a result of functional overlap in representations of space and time abstracted away from basic sensory properties or whether differential learning can be explained based on basic sensory correspondences, or lack thereof, between temporal and spatial properties of primary visual stimulation.

The current study was designed to overcome the limitations of previous work by using audio-visual stimuli with asynchronous offsets, thereby presenting non-redundant audio and visual magnitude information for length and time in distinct sense modalities (see Fig. 1). In this way, the learning of space–time magnitude pairings could not be explained by temporal synchrony of information or redundancy of spatial and temporal information across the senses. In addition, differential learning in the current study could not be explained by temporal and spatial magnitude correspondences within a single sense modality because visual–temporal information was held constant across all test conditions. Finally, while behavioral measures used within previous studies (e.g. deHevia and Spelke, 2010, Lourenco and Longo, 2010, Srinivasan and Carey, 2010) have revealed interesting patterns of results, they provide limited, if any, insight into the level of processing at which differential processing or learning may occur. It is possible, for example, that enhanced learning of congruent spatial–temporal pairings is due to facilitated early processing, more attention toward, and/or better memory for congruent pairings compared to incongruent pairings. The current study allowed for an assessment at each of these levels by employing event-related potentials (ERPs) to measure the brain response.

Fig. 1.

Audiovisual stimulus pairings used for training and test phases of the experiment. All visual stimuli were presented for 1000 ms and auditory stimuli were presented for 250 or 750 ms. The audiovisual onset was always synchronous and offset was always asynchronous. (A) Short caterpillar paired with a short duration tone. (B) Long caterpillar paired with a long duration tone. (C) Short caterpillar paired with a long duration tone. (D) Long caterpillar paired with a short duration tone.

Neurophysiological work with infants employing ERPs has documented early and mid-latency posterior activity related to processing of high-level properties of complex stimuli such as the configuration or structure of faces and objects (e.g. N290 and P400), mid-latency anterior activity related to attentional orienting (Nc), and late anterior slow wave activity related to memory (e.g. PSW, NSW, or LPC) (e.g. Halit et al., 2004, Scott and Nelson, 2006, Scott, 2011, Carver et al., 2003, de Haan and Nelson, 1997, Nelson, 1994, Richards, 2003, Grossmann et al., 2009, Quinn et al., 2006). Given the established functional properties of these distinct ERP components, measuring the infants’ neurophysiological response to relationally congruent and incongruent pairings may not only provide further evidence of functional overlap between representations of space and time, but may also help determine at what level differences arise.

In order to investigate the neural basis of cross-modal magnitude learning, we recorded event-related potentials as 5-month old infants viewed asynchronous, non-redundant, audiovisual space–time pairings. Separate groups of infants were familiarized to audio-visual length-time pairings that were either relationally congruent (a long visual image was paired with a long auditory tone, or a short image was paired with a short tone) or relationally incongruent (short tone with long image or vice versa) (Fig. 1). The same auditory tones and visual cartoon characters were used in the training of both groups, but their relative pairings were different. This training phase was necessary to familiarize children with the range of magnitudes and particular pairings, as previous work has shown that young children may understand magnitudes better when defined in contrast to other magnitudes (rather than in isolation) (see Duffy et al., 2005). After a training phase, all participants were shown equal numbers of familiar and novel test pairings of the same visual–spatial lengths and auditory durations. Importantly, both groups of infants viewed the exact same test pairings of stimuli. Whether a given test pairing was familiar or novel was dependent on the training condition they had previously viewed (e.g. if trained with relationally congruent pairs, then relationally incongruent pairings were novel and vice versa). Thus, differences in brain responses to test trials between groups could not be readily explained by basic sensory differences between the test stimuli as all infants saw the same set of test stimuli. Instead, differences in neural processing between training conditions would likely to be due to the type of training they previously received (i.e. congruent or incongruent). Specifically, we examined differential learning in the training conditions by comparing event-related brain signatures of early high-level processing, mid-latency attentional orienting, and later memory updating. If infants are able to quickly and reliably learn both relationally congruent and incongruent pairings, then the brain processes reflecting early high level processing, attention, and/or memory should discriminate between familiar and novel stimulus pairings regardless of the training condition. However, if learning is facilitated by relationally congruent pairings compared to incongruent pairings, then brain processes reflecting high-level processing, attention, and/or memory should discriminate familiar and novel test pairings in those infants who were exposed to the congruent pairings during the training phase but not in those infants exposed to the incongruent pairings during the training phase.

2. Materials and methods

2.1. Participants

Thirty-two five-month old infants (18 females; M age = 152 days) were randomly assigned to a relationally congruent (n = 16) or relationally incongruent training condition (n = 16). An additional 62 infants participated but were excluded from the analyses: 31 for early termination of experiment due to fussiness, crying, or inattentiveness and 31 for retaining too few good data segments after ERP artifact rejection. This attrition rate is within the range of attrition rates in other ERP studies of similar-aged infants (e.g. Hyde et al., 2010a, Hyde et al., 2010b, Hyde and Spelke, 2011, Izard et al., 2008, Quinn et al., 2006).

2.2. Stimuli and procedure

Infants were initially exposed to 20 relationally congruent (10 spatially long image-temporally long tone/10 spatially short image-temporally short tone) or 20 relationally incongruent (10 spatially long image—temporally short tone/10 spatially short image—temporally long tone) audio-visual length-time pairings. Stimuli were presented in a pseudorandom order with the constraint that each (of the two) space–time pairings had to be presented before re-randomization occurred. After the training period, all infants were shown the same 44 test images of familiar (22 trials) or novel (22 trials) pairings of the same stimuli using the same pseudo-random procedure described above (each of the four pairings presented in a random order before re-randomization occurred). Length stimuli consisted of cartoon images of a yellow caterpillar of differing lengths (short and long) presented on a white background (Fig. 1). Stimuli were adapted with permission from Srinivasan and Carey (2010). Specifically, we adapted two of the more extreme versions of “long” and “short” visual and auditory stimuli that differed in magnitude by a 1:3 ratio, as previous infant studies have shown this difference to be sufficient for behavioral discrimination of spatial and temporal information at 5-months of age (Brannon et al., 2006, Brannon et al., 2008). The short caterpillar consisted of three overlapping concentric body segments 2.5 in. in height and 2.5 in. in length. The long caterpillar consisted of 11 overlapping concentric body segments 2.5 in. in height but 7.75 in. in length. Both caterpillars had two eyes and antennas on the right-most segment to look like a head. Time stimuli consisted of the auditory tones lasting 250 ms (short temporal duration) and 750 ms (long temporal duration) played at approximately 70 decibels (Fig. 1). Congruent familiarization pairings consisted of equal numbers of trials with the longer caterpillar paired with the temporally longer tone and the shorter caterpillar paired with the temporally shorter tone. Incongruent familiarization pairings consisted of equal numbers of trials containing the shorter tone paired with the longer caterpillar and the longer tone paired with the shorter caterpillar. Two exemplars were used for each relationship (congruent or incongruent) to make it less likely that the average ERP response to familiar and novel pairings would be due to idiosyncratic sensory properties of a particular stimulus pairing used and more likely due to the overarching relational congruence of stimuli in relation to training.

Critically, the temporal and spatial information were not redundant across the senses. Visual stimuli were always presented for 1000 ms and, while onset of the visual and auditory tones were synchronous, offset of the visual and auditory stimulus was never synchronous (determined by the length of the auditory stimulus: either 250 ms or 750 ms). All trials were followed by a blank-screen inter-stimulus interval that varied randomly in length from 900 to 1500 ms (1000 ms stimulation + 900–1500 ms ISI = 1900–2500 ms trial duration).

Stimuli were presented from a 17-in. computer screen and computer speakers approximately 35–50 cm from the infant seated in a parent's lap. The parent was instructed to look at the back of the infant's head instead of the display in order to blind them from the experimental condition that was being presented. Parents were also instructed not to speak to the infant in an attempt to reduce environmental interference. Looking behavior was monitored online, both training and test trials were only started when the infant was looking at the screen, and the experiment was paused when infants looked away for more than 1 s. If the baby became fussy, began to cry, or continually looked away, data collection was terminated. Halfway through the familiarization period and two times during the testing phase a short break was taken by showing the infant a picture of an animal with an accompanying animal sound (e.g. picture of a dog with a barking sound, picture of duck with quacking sound, etc.). This was done in an effort to reduce boredom and recapture infants’ attention to the video screen. This rest/break stimulus was presented at the same point in the experiment for all infants.

2.3. Data acquisition and reduction

Infants’ heads were first measured and then, while the vertex was being located and marked on the baby's scalp, the appropriate size sensor net was soaked in a potassium chloride solution. While the infant sat in a parent's lap, the sensor net was systematically placed on the head with reference to the identified vertex and left and right mastoid bones. Before data collection, impedances were checked and maintained below 50 kΩ. The ongoing EEG was then recorded from scalp locations using a 64 channel EGI HydroCel Sensor Net (Electrical Geodesics Inc., Eugene, OR) as infants were presented with stimuli in a dimly lit room. Data were recorded at 250 samples per second and digitally filtered online at 0.1–100 Hz, referenced to the vertex.

Data from infants that completed the experiment were further processed in three steps. First, raw data were lowpass filtered at 30 Hz, segmented into epochs from 200 ms before to 1450 ms after stimulus onset, and baseline corrected to the 200 ms before the stimulus was presented. Second, two experienced staff members independently visually examined each epoch for artifacts. Any epoch containing an eye blink, eye movement, excessive noise, or more than 12 bad channels was rejected from further analysis. Questionable epochs were discussed between staff members until consensus could be reached. Any infant retaining less than 10 good epochs in either test type (familiar or novel test pairings) after artifact rejection was eliminated from the final analysis (see Section 2.1 for exact numbers). Remaining acceptable epochs were further processed by running an automated bad channel replacement algorithm based on spherical spline interpolation from surrounding good channels for trials containing less than 13 bad channels, then creating averages for each test type (familiar and novel test pairings) for each subject, re-referencing the data to the average reference, and again baseline correcting to 200 ms before stimulus onset to correct for any absolute amplitude differences created by processing. In addition to averages for each test type, a grand average for both test types (collapsed across familiar and novel pairings) was created for each training condition separately for visual inspection and peak latency analysis purposes. Individual participants included in the final analysis contributed no less than 10 trials per condition, and, on average, 12.328 trials per condition. There were no significant differences or interactions in the number of trials retained between Training Conditions or Test Types (all p's > .18) (congruent training: M familiar test trials = 12.50, M novel test trials = 11.50; incongruent training: M familiar test trials = 12.75, novel test trials = 12.56).

2.4. Data analysis

Early posterior processing was characterized as the average response over left, central and right posterior parietal electrode groups (EGI Hydrocel GSN electrodes: 31, 33-left; 36, 37-central; and 38, 40-right). Mid-to-late anterior processing was characterized as the average response over a fronto-central electrode group (sites 3, 6, 9). Electrode groups were chosen to characterize early posterior and mid-to-late anterior activity based on previous literature (see de Haan, 2007 or Reynolds and Richards, 2005 for reviews). We temporally focused our analysis from 250 ms post stimulus onset, given that none of the conditions could be distinguished before then (“short” auditory stimulus was 250 ms), to the end of the segment (1450 ms). Our analysis of the early posterior activity was restricted to rising second half of the first major posterior negativity (250–450 ms). Time windows of interest for the mid-to late latency anterior components (Nc, PSW) were chosen by using broad time windows to first acquire the peak latency of the grand mean (collapsed across familiar and unfamiliar test trial pairings) for each training condition (Nc = 450–950 ms; PSW = 950–1450 ms). Statistical analyses (see Section 3) revealed significant differences in peak Nc latency but not PSW latencies between training conditions (see Section 3). As a result, distinct, symmetrical time windows (−100/+100 ms) around the corresponding peak latencies for each training condition (congruent training: 599 ms; incongruent training: 711 ms) were used to characterize the Nc (congruent training: 499–699 ms; incongruent training: 611–811 ms) and a single, fixed, symmetrical time window (−100/+100) around the average peak latency (1131 ms) was used to characterize both training conditions for the PSW (average amplitude between 1031 and 1231 ms).

Independent samples t-tests were used to test for latency differences between training conditions (collapsed across test type). ANOVAs with the between-subjects factor of Training Condition (relationally congruent vs. relationally incongruent) and the repeated within-subject factor of Test Type (familiar vs. novel pairing) statistically assessed mean component amplitude. Scalp Region was also a factor in the analysis of posterior processing (left, central, right electrode groups). Significant interactions were further examined post hoc using paired sample t-tests.

3. Results

3.1. Early posterior activity

Test stimuli evoked a large negativity followed by a rising positivity over widespread posterior parietal scalp locations between about 175–700 ms after stimulus onset (Fig. 2). Timing and topography were consistent with the previously identified infant N290/P400 complex (e.g. de Haan, 2007). Waveform morphology, however, was slightly different (see Fig. 3) from that observed in previous studies using only visual stimulation (see de Haan, 2007 for review of infant visual-evoked posterior activity). Given that congruent and incongruent stimuli could only be differentiated after 250 ms and mid-latency anterior activity started around 450 ms, we were restricted to analyzing the rising (second) half of this early posterior negative component using the mean amplitude of a fixed time window (250–450 ms) averaged over left, midline, and right posterior sites.

Fig. 2.

Topographical scalp maps for components of interest as viewed from the top of the head (oriented with back of the head facing the bottom of the figure). Black dots represent electrode sites used to compute mean response.

Fig. 3.

Mean posterior event-related potentials. (A) Average waveform over posterior channels between −200 and 1450 ms to familiar and novel test pairings for each training condition. The shaded region represents approximate time window of analysis. (B) Mean amplitude for the early posterior negativity to familiar and novel test pairings for each training condition. The “*” indicates a significant difference between conditions.

The amplitude analysis revealed a main effect of Scalp Region (F (2, 60) = 4.384, p < .05, ${\eta }_p^2=.127$), a main effect of Training Condition (F (1, 30) = 6.160, p < .05, ${\eta }_p^2=.170$), and an interaction between Training Condition and Test Type (F (1, 30) = 4.253, p < .05, ${\eta }_p^2=.124$) (all other p's > .14). The left posterior parietal scalp group showed the most negative mean amplitudes during this time frame. Further post hoc analysis revealed that the interaction between Training Condition and Test Type on the second half of the early posterior negativity could be explained by a main effect of Test Type (F (1, 15) = 5.462, p < .05, ${\eta }_p^2=.267$) for those infants that were familiarized to the congruent pairing, with more negative amplitudes observed for familiar pairings compared to unfamiliar pairings, and no significant differences in Test Type for those infants who were familiarized to the incongruent pairings (all p's > .35) (Fig. 3).

3.2. Mid-latency anterior negativity

Test stimuli also evoked a mid-latency anterior negativity between 450 and 950 ms after stimulus onset (Fig. 2). Scalp topography and timing were consistent with the previously identified Nc component (e.g. de Haan and Nelson, 1997, Nelson, 1994, Reynolds and Richards, 2005). Again, however, morphology was somewhat different (Fig. 4) than that observed in purely visual studies and more consistent with Nc morphology seen in studies with multisensory stimuli (Hyde et al., 2010b, Hyde et al., 2011).

Fig. 4.

Mean anterior event-related potentials. (A) Average waveform over anterior channels between −200 and 1450 ms to familiar and novel test pairings for those subjects in the congruent training condition. The shaded region represents approximate time window of analyses. (B) Mean amplitude for the mid-latency Nc over anterior sites to familiar and novel test pairings for each training condition. The “*” indicates a significant difference between conditions. (C) Mean amplitude for the later positive slow wave (PSW) over anterior sites to familiar and novel test pairings for each training condition. The “*” indicates a significant response above baseline. (D) Average waveform over anterior channels between −200 and 1450 ms to familiar and novel test parings for those subjects in the incongruent training condition. The shaded region represents approximate time window of analyses.

An analysis of the peak latency of the Nc between 450 and 950 ms revealed a main effect of Training Condition (t (30) = −2.445, p < .05), with the Nc peaking significantly earlier (601 ms) for those infants presented with the congruent pairing during the training period compared to those infants presented with the incongruent pairing during the training phase (708 ms). Given the significant differences in Nc timing between groups, distinct time windows were used to characterize the mean amplitude of the Nc for each training group. Specifically, Nc was defined as a symmetrical 200-ms time window (−100 to +100 ms) surrounding the peak for each Training Condition over the fronto-central scalp group1 (congruent training 599–799 ms; incongruent training: 611–811 ms).

An analysis of the mean Nc amplitude revealed a significant interaction between Training Condition and Test Type over anterior scalp (F (1, 30) = 4.183, p < .05, ${\eta }_p^2=.122$) (other p values > .43) (Fig. 4). Post hoc comparisons revealed that infants presented the congruent pairing during the training phase showed more negative frontal Nc responses to the familiar (congruent) pairings than to the novel (incongruent) pairings during the test phase (t (15) = −2.701, p < .05) (see Fig. 4). In contrast, infants presented incongruent pairings during the training phase showed no difference in mean Nc amplitudes to congruent and incongruent pairings at test (t (15) = .739, p > .47) (Fig. 4).

3.3. Late anterior positive component

Some test stimuli evoked an anterior, late-going slow wave, which emerged once stimuli were no longer present (around 1000 ms) and returned to baseline around 1400 ms (see Fig. 2). Scalp timing, anterior topography, and wave morphology (see Fig. 2, Fig. 4) were consistent with characterizations of the PSW or LPC in previous ERP studies with infants of this age (see de Haan, 2007 or Reynolds and Richards, 2005 for reviews). An analysis of peak latency over a broad time window (950–1450 ms), collapsed across test trial types, revealed no differences in timing between training conditions. As a result, a single symmetrical time frame (−100/+100 ms) around the average peak latency (1130 ms) was used to analyze the anterior positive slow wave for all conditions (1030–1230 ms).

An analysis of the mean positive slow wave amplitude over anterior scalp revealed a significant interaction between Training Condition and Test Type (F (1, 30) = 5.631, p < .05, ${\eta }_p^2=.158$) (all other p's > .82). Post hoc analysis revealed only a marginal difference between test types for those infants trained on the congruent test pairs (F (1, 15) = 3.393, p = .085, ${\eta }_p^2=.184$), with unfamiliar test trials eliciting a more positive amplitude than familiar test pairs, and no difference between test types for those infants trained on incongruent test pairs (F (1, 15) = 2.42, p > .14, ${\eta }_p^2=.139$).

Given that the late slow wave activity is associated with memory updating for partially encoded stimuli and that a return to baseline is observed for full-encoded items and also for items that are not encoded at all (Nelson, 1994, Nelson and Collins, 1992, Nelson and de Regnier, 1992, Reynolds and Richards, 2005, Richards, 2003), we further analyzed, in a post hoc manner, the PSW relative to baseline using one-sample t-tests (against zero). This analysis revealed only the familiar test stimuli for those trained on incongruent pairings produced a PSW significantly greater than baseline (t (15) = 2.780, p < .05) (congruent training/familiar test, p = .79; congruent training/novel test: p = .063; incongruent training/novel test: p = .973).

4. Discussion

We observed that the infant brain discriminates relationally congruent from relationally incongruent cross-modal space–time pairings when briefly familiarized with the relationally congruent pairings but not when familiarized with the relationally incongruent space–time pairings. These results are not due to low-level sensory differences between conditions, as infants in both training groups were presented with the same test stimuli. Furthermore, the design of our stimuli allowed us to rule out that differential processing was due to common spatio-temporal cues across a single sense modality, was dependent on redundant cues across multiple sense modalities, or was based on lower-level similarities between dimensions in the visual domain alone; instead, it appears that associations were made over amodal representations of space and time, abstracted away from non-redundant and asynchronous auditory and visual information. Therefore, we interpret the asymmetry in processing (i.e. congruent compared to incongruent pairings) between training groups to a functional overlap between mental representations of space and time, or a relational congruence bias, where structural similarities in the neural representation facilitate the automatic aligning and mapping between dimensions for relationally congruent pairings over relationally incongruent pairings, as suggested by Srinivasan and Carey (2010).

By measuring the neurophysiological response, we observed that the asymmetry in processing begins, over posterior scalp sites, almost immediately after stimulus differences emerge. Neural processing was further differentiated both in timing and magnitude between training groups during mid-to-late latency time periods over anterior sites. Our functional interpretation is that, after a brief familiarization period to define relatively long and short magnitude pairings of auditory durations (tones) visual lengths (cartoon characters), relationally congruent pairings showed enhanced processing and attention orienting relative to incongruent pairings in those infants first familiarized to congruent pairings (long duration-long character/short duration-short character). In contrast, infants familiarized to incongruent pairings (short-long/long-short) showed no differences in early processing or attentional orienting between congruent and incongruent pairings in the test phase.

Early posterior activity was temporally and topologically consistent with the infant N290/P400 complex typically observed for visual processing of faces and objects (e.g. Halit et al., 2004, Scott and Nelson, 2006, Scott, 2011). However, we are not committed to the idea that the observed early posterior differences have the same hypothesized ventral origin as those observed typically for visual processing of faces and objects (e.g. Halit et al., 2004, Scott and Nelson, 2006, Scott, 2011). Studies of numerical, spatial, and temporal processing, in contrast to face processing, typically activate dorsal cortical areas in the parietal cortex (Izard et al., 2008, Hyde et al., 2010a, Brannon et al., 2008). The spatial nature of the variables of interest, the audio-visual nature of the stimuli, and the gross scalp topography of the effects are at least equally consistent with a parietal rather than a temporal–ventral brain origin. This speculation, however, should be followed up using imaging techniques with better spatial resolution if our claim is to be empirically substantiated.

Although the waveform morphology of the mid-latency anterior processing was different, timing and topography were consistent with the infant Nc component identified by other as a marker of attentional orienting (see de Haan et al., 2003 or Reynolds and Richards, 2005 for reviews). Morphological differences were likely due to the multimodal nature of our stimuli compared to previous visual work. Previous work has shown that the infant Nc response is related to behavioral attention (e.g. Richards, 2003; also see Reynolds et al., 2010). Unfortunately, behavioral looking times could not be obtained in our study to compare directly with previous behavioral work, given the necessity for presenting many trials of short duration for the ERP paradigm. We were also unable to obtain enough artifact-free trials per stimulus type to look at differences in learning between specific pairings of “long” and “short” auditory and visual stimuli. Nevertheless, “long” and “short” visual and auditory stimuli differed by a 1:3 ratio and previous work suggests a 1:3 difference is well within the 5-month old infants’ ability to discriminate (limit of about a 1:2 ratio; Brannon et al., 2007, Lipton and Spelke, 2003, Wood and Spelke, 2005). Furthermore, the patterns of Nc modulation observed in our study accord with previous behavioral work showing discrimination for space–time pairs when first familiarized to congruent pairings but not when first familiarized to incongruent pairings (Srinivasan and Carey, 2010).

We also observed an interaction between Training Condition and Test Type in the late positive slow wave component. Post hoc analysis revealed that only familiar test stimuli for those infants training on incongruent spatial–temporal pairings elicited a late PSW significantly different from baseline. In contrast, neither the familiar or novel test conditions for infants who trained on congruent pairings nor the novel condition for infants trained on incongruent pairings were significantly different from baseline. Previous studies suggest that late positive slow wave (PSW) activity is related to memory updating for partially encoded stimuli, and that stimuli that are fully encoded in memory and stimuli that are not encoded at all do not evoke late slow wave activity above baseline (e.g. de Haan, 2007 or Reynolds and Richards, 2005 for reviews). Although speculative, the return to baseline for the familiar stimuli of those trained with congruent pairings may be a result of the cross-modal spatio-temporal relations being fully encoded into memory while the lack of a significant PSW for the other conditions may be an indicative of a lack of higher-level encoding at all. In contrast, an increase in amplitude for the familiar test stimuli of those in the incongruent training group may indicate updating of a partially encoded stimulus in memory.

More generally, in multisensory contexts, redundancy of information across the senses has been shown to facilitate learning and early development, particularly in young infants around the same age as we test here (see Bahrick and Lickliter, 2000). While onset synchrony may have equally queued training groups into the events, intersensory redundancy cannot explain the differential learning patterns between the training groups of our study because the test stimuli were the same for both groups and because the relational information was not redundant across the senses. While redundancy of temporal information would likely facilitate learning in this context, as it has shown to do in others (e.g. Jordan et al., 2008) and may have in Srinivasan and Carey (2010), our study shows that it is not necessary for learning pairings of audio-visual magnitude information. A relational congruence bias, in addition to intersensory redundancy, should now be considered a cue that facilitates processing of complex audio-visual information.

5. Conclusions

In sum, our results suggest that young infants encode, attend to, and potentially remember spatial and temporal relations that are congruent more readily than they do relations that are incongruent. The age of our participants, 5-months, makes it unlikely that explicit instruction or language experience can account for the results. Rather, these results suggest the structure of the mind biases us to learn certain magnitude pairings over others. A remaining issue is if this interaction can be explained by a privileged connection between domain-specific representations of space and time or by generalized representations of magnitude (see Lourenco and Longo, 2011 or Walsh, 2003 for reviews). Another remaining issue is why this bias is present. Is this a feature or a bug of the brain? Like intersensory redundancy (Bahrick and Lickliter, 2000), this bias may further highlight and facilitate learning of important magnitude correspondences that transcend the primary properties of sound and sight early in development. Alternatively, relational congruence bias may result from a recycling of ancient mechanisms originally evolved for other purposes (Gould and Vrba, 1982, Dehaene, 2005). Whatever the answer, it seems as if this organization has important implications for human nature and culture as pervasive as guiding the way we learn about, understand, and choose to describe the world around us (Clark, 1973, Gruber, 1965, Jackendoff, 1983, Lakoff and Johnson, 1980, Langacker, 1987, Srinivasan and Carey, 2010, Talmy, 1988).

Conflicts of interest

We declare no conflicts of interest in conducting or publishing this work.