Learning to Attend Selectively: The Dual Role of Intersensory Redundancy

Abstract

Selective attention is the gateway to perceptual processing, learning, and memory, and is a skill honed through extensive experience. However, little research has focused on how selective attention develops. Here we synthesize established and new findings assessing the central role of redundancy across the senses in guiding and constraining this process in infancy and early childhood. We highlight research demonstrating the dual role of intersensory redundancy -- its facilitating and interfering effects-- on detection and perceptual processing of various properties of objects and events.

The environment provides a flux of changing, concurrent stimulation to all our senses, far more than can be attended at any given moment in time. Consequently, we must selectively attend to some aspects of objects and events while ignoring others. Adults are highly skilled at directing selective attention to information that is relevant to their needs, goals, and interests, while ignoring a vast array of irrelevant stimulation. For example, we easily pick out a friend in a crowd, follow the flow of action in a ball game, and attend to the face and voice of a single speaker in the context of competing conversations. These attention skills, however, must be learned and honed through experience and practice. Much of this learning takes place in early development. Infants quickly learn to intercoordinate their patterns of looking and listening to determine which sights and sounds belong together and which do not. They learn to parse the visual array into coherent objects and speech into meaningful words by attending to invariant patterns across variation in input. Such selective attention is widely recognized as the gateway to successful information pick-up and processing (Neisser, 1976).

The Dynamics of Selective Attention

An obvious but important insight is that selective attention to stimulation generated from exploratory activity provides the basis for what is perceived, learned, and remembered. In turn, what is perceived, learned, and remembered influences what is attended to in subsequent bouts of exploration, in continuous cycles of attention → perception → learning → memory → attention, and so on. Figure 1 illustrates this dynamic system of influences and the often overlooked but fundamental role of selective attention for perception, learning and memory. Moreover, action is tightly coupled with these processes, providing new stimulation for attention, perception, learning, and memory across continuous bidirectional feedback loops (Figure 1; see also Adolph & Berger, 2005; Gibson & Pick, 2000). This system of dynamic, interactive influences evolves over time, with concurrent changes in neurodevelopment that go hand in hand with changes in perception and action. Simply put, we create our effective environment by what we attend to.

Figure 1.

The central role of selective attention in perception, action, learning, and memory is depicted in two interrelated, concurrent feedback loops: (a) the attention – perception – learning – memory—attention system, and (b) the attention – perception – action—attention system. The arrows represent the primary direction of the flow of information but each component process (and each system) is involved in continuous, bidirectional feedback loops with the other components (and systems). Stimulation available for exploration is generated through action/exploratory activities (e.g., eye movements, reaching, posture changes) which in turn produce more stimulation for exploration, in continuous cycles. Selective attention to this stream of stimulation provides the basis for what is perceived, and thus what can be learned and remembered, and this affects what is attended to next and in subsequent encounters with similar stimulation. (reprinted from Bahrick, 2010 with permission from Wiley/Blackwell Publishers)

Infants face a particularly daunting challenge. They must learn to attend selectively to the vast array of changing multimodal stimulation with limited attentional resources and limited experience with objects and events to guide them. Selective attention here refers to a focus on particular aspects of sensory stimulation (external or internal) at the expense of other aspects, leading to enhanced neural activity and readiness for information pick-up. The control of attention can be overt or covert, conscious or unconscious, endogenous or exogenous, bottom-up or top-down. Selective attention develops with experience and becomes increasingly more economical (E.J. Gibson 1969; Gibson & Pick, 2000).

What guides selective attention to relevant aspects of stimulation in early infancy? Despite its obvious importance for perceptual, cognitive, social, and linguistic development, the degree and nature of attentional honing required for typical perception is underappreciated, and little is known about the principles that govern these important processes (but see Courage, Reynolds, & Richards, 2006; Richards, Reynolds, & Courage, 2010). For experienced perceivers, top-down processes such as prior knowledge, categories, goals, and expectations primarily guide information pick-up (e.g., Neisser, 1976; Schank & Ableson, 1977). In contrast, early attention development is more influenced by bottom-up processes, including sensitivity to salient properties of stimulation such as contrast, movement, intensity, statistical regularities, and redundancy across the senses (Bahrick & Lickliter, 2000, 2012; Kellman & Arterberry, 1998; Lewkowicz & Turkewitz, 1980). With experience, selective attention gradually becomes more adult-like, endogenous and modulated by top-down processes (Plude, Enns, & Grodeur, 1994; Colombo, 2001; Ruff & Rothbart, 1996).

The Salience of Intersensory Redundancy

One feature of stimulation which has received growing appreciation for its role in guiding attentional allocation during early development is intersensory redundancy (Bahrick & Lickliter, 2000, 2012; Bremner, Spence, & Lewkowicz, 2012). Intersensory redundancy, provided by most naturalistic events, refers to the same information simultaneously available and temporally synchronized across two or more sensory systems. For example, when the rhythm and tempo of speech can be perceived by looking and listening, the rhythm and tempo are redundantly specified. By definition, only amodal properties (information not specific to a particular sensory system; e.g., tempo, rhythm, duration, intensity) can be redundantly specified across the senses. Young infants readily perceive amodal information (Bahrick & Pickens, 1994; Lewkowicz, 2000). By detecting amodal information there is no need to learn to integrate stimulation across the senses to perceive unified objects and events (e.g., a person speaking, a ball bouncing), as originally proposed by constructivist accounts of early cognitive development (Piaget, 1952). Instead, as argued by James and Eleanor Gibson (E.J., 1969; J.J., 1966), sensory stimulation is already united in these events and we detect this amodal information through a unified perceptual system.

Perceiving redundant amodal information, combined with an increasing sensitivity to the statistical regularities of the environment, ensures that inexperienced perceivers selectively attend to unified multimodal events, such as people speaking or keys jingling (as opposed to looking to one event while listening to another). In fact, multimodal redundancy is so effective in directing selective attention and unitizing audiovisual stimulation that it can “tell” infants which of two superimposed video events to watch and which to ignore. The sound-synchronized event appears to “pop out” from the background of the silent superimposed visual event and directs attentional selectivity (Bahrick, Walker, & Neisser, 1980). Infant sensitivity to the salience of intersensory redundancy plays a key role in the early development of a number of cognitive/perceptual skills, including operant learning (Kraebel, 2012), emotion discrimination (Flom & Bahrick, 2007), rhythm and tempo discrimination (Bahrick et al., 2010), numerical discrimination (Jordan et al., 2008), sequence detection (Lewkowicz, 2004), abstract rule learning (Frank, Slemmer, Marcus, & Johnson, 2009), and word comprehension and segmentation (Gogate & Bahrick, 2001; Hollich et al., 2005).

The Intersensory Redundancy Hypothesis

Detecting intersensory redundancy gives rise to attentional salience hierarchies. The Intersensory Redundancy Hypothesis, (IRH, Bahrick & Lickliter, 2000, 2012) is a theory of selective attention that addresses how attentional salience hierarchies organize and guide early selective attention and perceptual processing. According to the IRH, events provide far more information than can be attended at a given time and attention and perceptual processing are thus directed to the most salient aspects of stimulation first, and later in exploratory time, to less salient aspects. All multimodal events provide both redundant, amodal information as well as nonredundant, modality-specific information such as color, pattern, pitch or timbre, aspects available to only a particular sense. The IRH describes the conditions under which we attend and process amodal versus modality-specific information and how this changes across development. According to the IRH, when exploring multimodal events with more than one sense, amodal properties such as synchrony, rhythm, and tempo are most salient and processed first (intersensory facilitation) but when exploring events unimodally (e.g., talking on the phone, viewing a silent person), modality-specific properties are most salient and processed first (unimodal facilitation, Bahrick 2010, Bahrick & Lickliter, 2012). Thus, the nature of exploratory behavior (multimodal, unimodal) and the stimulation provided (multimodal, unimodal) dictate the type of information that is most salient.

The principles of intersensory and unimodal facilitation were established and documented across species (human and avian) a decade ago (Bahrick, Lickliter, & Flom, 2004). They have recently been extended from nonsocial to social events and to new domains. Below we briefly review these principles, illustrated with more recent examples from social events, and then focus on findings from new domains including the roles of developmental change, task difficulty, educating attention, and neural evidence for intersensory redundancy and attention allocation.

Intersensory Facilitation of Amodal Properties

Intersensory facilitation is the principle that amodal properties are detected more readily and earlier in development when they are redundantly specified in multimodal stimulation than when the same amodal properties are detected in unimodal stimulation. This was originally demonstrated for the amodal property of rhythm. At 5-months, infants detect the rhythm of a toy hammer tapping in audiovisual synchronous, but not unimodal visual, auditory or asynchronous stimulation (Bahrick & Lickliter, 2000). This principle was subsequently extended to social events. For example, by 4-months, infants discriminate affect (specified by a combination of amodal properties) in synchronous audiovisual speech but not in unimodal auditory, visual, or asynchronous audiovisual speech (Flom & Bahrick, 2007). Similarly, quail embryos learn and remember the rhythm and tempo of a maternal call following synchronous prenatal audiovisual exposure, but not following the equivalent amount of unimodal auditory or asynchronous audiovisual exposure (Lickliter, Bahrick, & Honeycutt, 2002).

The importance of intersensory redundancy for infant perceptual processing has recently been demonstrated at the neural level using measures of event-related potentials. Five-month-old infants showed heightened attentional salience (greater amplitude Nc) and longer and deeper processing (reduction in late slow wave) for synchronous audiovisual speech than asynchronous or unimodal visual speech (Reynolds, Bahrick, Lickliter, & Guy, 2014). This new finding broadens the conceptual frame of the IRH by revealing that intersensory redundancy not only promotes selective attention to certain (amodal) event properties, but also promotes longer engagement and deeper processing.

Unimodal Facilitation of Modality-Specific Properties

During unimodal stimulation, such as watching a silent event or hearing a person speak over the phone, attention is not captured by salient intersensory redundancy and is free to focus on modality-specific properties, making the pitch and timber of a voice, or the appearance and features of a person’s face most salient. This principle of the IRH, unimodal facilitation, holds that modality-specific properties (e.g., color, pattern, pitch, timbre) are detected more readily and earlier in development when they are explored through only one sense, than when the same information is detected in multimodal, synchronous stimulation.

This principle was first documented for infant perception of spatial orientation for nonsocial events (Bahrick, Lickliter, & Flom 2006). More recently, it has provided new information about early face perception. Bahrick, Lickliter, and Castellanos (2013) demonstrated that two-month-old infants discriminate between the faces of two women best when the women are speaking silently as compared with speaking audibly in synchrony with their voices. Even more striking, face discrimination is enhanced during asynchronous as compared with synchronous audiovisual speech, highlighting the interfering role of intersensory redundancy for detection of modality-specific information such as facial configuration. During audiovisual speech, intersensory redundancy captures attention, directing it to amodal properties of speech. In contrast, in our asynchronous control (in which intersensory redundancy was eliminated but the amount and type of stimulation were preserved), infants discriminated between the two faces. This dual role of intersensory redundancy (both facilitating and interfering) is often overlooked and instead it is assumed that intersensory redundancy enhances attention to all aspects of an event. Recent data from quail chicks provide parallel findings in the auditory modality. Chicks learned the pitch of a maternal call in unimodal auditory stimulation or when the call was asynchronous with a flashing light, but not when intersensory redundancy was provided by synchronizing the notes of the call with the flashing light (Vaillant-Mekras, Bahrick, & Lickliter, submitted).

Developmental Change and the Role of Task Difficulty: New Findings

Research generated by the IRH initially focused on selective attention in early development because during this period attentional resources are most limited, attention progresses slowly along the salience hierarchy, and resulting attentional trade-offs and hierarchies are most evident. However, the principles of the IRH also likely apply across the life-span whenever attentional resources are limited (e.g., difficult tasks, high cognitive load). Recent research indicates that in later development, as attention becomes more efficient and flexible, it progresses along the salience hierarchy more quickly and infants can detect both amodal and modality-specific aspects of stimulation in unimodal and multimodal stimulation within a single bout of exploration (Bahrick, 2010, Bahrick & Lickliter, 2004). For example, Bahrick and colleagues (2013) demonstrated that although 2-month-olds showed unimodal facilitation for face discrimination, by 3-months infants discriminated faces under all conditions, including in the presence of intersensory redundancy.

However, salience hierarchies are evident in later development if a task is difficult or attentional resources are taxed. For example, although 5-month-olds showed no intersensory facilitation for discriminating tempo changes of low difficulty, they did for tempo changes of moderate and high difficulty (Bahrick, et al., 2010). Further, preschoolers show unimodal facilitation for face discrimination in a difficult task (short familiarization times and memory load), paralleling performance of infants in a task with lengthy familiarization times and no memory load (Bahrick, Krogh-Jesperson, Argumosa, & Lopez, 2013). Newer evidence indicates that even adults show intersensory facilitation for discrimination of tempo when the contrasts are difficult, but not when they are easier (Bahrick, Todd, & Martin, 2013).

Educating Attention: New Findings

Recent findings from both human and non-human animals indicate that intersensory redundancy can educate selective attention and provide a mechanism for promoting developmental change (Bahrick & Lickliter, 2012). Once amodal properties (i.e., synchrony, rhythm, tempo) “pop out” in multimodal stimulation as a result of intersensory redundancy, infants can then detect these same amodal properties in subsequent unimodal stimulation, at younger ages and under exposure conditions that would otherwise not support detection of amodal properties. This is similar to a priming effect, but longer lasting. Lickliter, Bahrick, and Markham (2006) found that quail chicks showed no preference for a familiar maternal call after a brief prenatal unimodal auditory familiarization. In contrast, by first exposing embryos to a redundant audiovisual presentation (call synchronized with flashing light) followed by the unimodal auditory presentation (i.e., bimodal → unimodal), chicks preferred the familiar auditory maternal call 2 days after hatching. Embryos who received the reverse sequence (unimodal → bimodal) showed no preference. This education of attention was effective even after delays of 2 or 4 hours between initial bimodal exposure and subsequent unimodal exposure, and continued to affect learning and memory days later (Lickliter, et al., 2006).

Studies of human infants have shown parallel findings. Four month-olds detect a change in the tempo of a toy hammer tapping in unimodal visual stimulation only if they received a brief pre-exposure to the tempo in redundant (synchronous audiovisual) stimulation, educating their attention to the tempo information. Infants failed to detect the tempo change following non-redundant (unimodal visual or asynchronous audiovisual) pre-exposure (Castellanos, Vaillant-Molina, Lickliter, & Bahrick, 2006). By educating attention to amodal properties, animal and human infants can continue to detect these amodal properties in the same events, even when redundancy is no longer available. This expands the conceptual frame of the IRH, suggesting that education of attention can foster flexible processing and serves as a mechanism for promoting developmental change in attentional selectivity, from detection of amodal properties in multimodal stimulation to amodal properties in all types of stimulation.

Putting it All Together: The Dual Role of Intersensory Redundancy

Taken together, studies generated by the IRH reveal a bidirectional or dual role (both facilitating and interfering effects) of intersensory redundancy on attention and perceptual processing of event properties. Consequently, multimodal and unimodal stimulation have opposite effects: multimodal events facilitate detection of amodal properties at the expense of modality-specific properties, whereas unimodal stimulation facilitates detection of modality-specific properties at the expense of amodal properties. Because competition for attentional and processing resources underlies these effects, they are most evident in early development, but are also at play in later development for difficult tasks or conditions of high cognitive load. The convergence of data across species, developmental periods, event types, and methods provides strong evidence for these conclusions. Alternative models or hypotheses, including the proposal that the greater amount or complexity of stimulation from multimodal than unimodal events can account for findings can be discounted, as they do not explain both the facilitating and interfering roles of multimodal stimulation. Such arguments are also discounted by data from our asynchronous control groups.

Real-time infant exploration of events illustrates the dual role of intersensory redundancy and its application to naturalistic settings. For example, when an infant looks and listens to a novel person speaking, she would first attend and process amodal properties such as rhythm, tempo, affect, or prosody of audiovisual speech at the expense of modality-specific details. In contrast, when the person is silent, the infant may shift attention to the appearance of her face, clothing, and hair, and when the woman or infant turn away, attention would shift to the particular sound of her voice. Attention may thus shift across exploratory time as a function of the changing context, nature of exploration, and type of stimulation available. Further, with increasing experience, infants show more flexible, rapid shifting along the salience hierarchy, resulting in attention to multiple properties of events (including both amodal and modalityspecific) in a single bout of exploration. Of course, factors such as complexity, novelty, difficulty, the length of exploratory time, and expertise also affect the speed of progression through this salience hierarchy. Future research is needed to explore how changes in selective attention to event properties progress along the salience hierarchy in real-time for individual infants.

New Directions

Infant sensitivity to intersensory redundancy has thus far been assessed at the group level, limiting our understanding of the nature and basis of developmental change and application to real-world learning settings. We are currently focusing on establishing individual difference measures of intersensory functioning (Bahrick et. al., 2013). This approach will allow systematic explorations of early developmental trajectories and their relations with cognitive, social and language outcomes in both typical and atypical populations. Moreover, this grain of analysis will advance theories of attention and perception by revealing the pathways through which simple attentional skills and trade-offs cascade into complex cognitive, social, and language skills.