The Integrated Development of Sensory Organization

Synopsis

The natural environment provides a flux of concurrent stimulation to all our senses, and the integration of information from different sensory systems is a fundamental feature of perception and cognition. How information from the different senses is integrated has long been of concern to several scientific disciplines, including psychology, cognitive science, and the neurosciences, each with different questions and methodologies. In recent years, a growing body of evidence drawn from these various disciplines suggests that the development of early sensory organization is much more plastic and experience-dependent than was previously realized. In this article, I briefly explore some of these recent advances in our understanding of the development of sensory integration and organization and discuss implications of these advances for the care and management of the preterm infant.

Sensory Integration and Organization

Most objects and events present a complex mix of visual, auditory, tactile, and olfactory stimulation to the senses. How do young infants determine which patterns of sensory stimulation belong together and which ones are unrelated? For much of the twentieth century, the majority of developmental scientists assumed that infants must gradually learn to coordinate and integrate information obtained by the separate sensory systems ^[1,2,3]. From this view, information had to be integrated across the separate senses through a gradual process of association in order for infants to perceive unified objects and events. This integration was thought to occur by the infant interacting with objects, experiencing concurrent feedback from different senses, and associating, assimilating, or calibrating one sense to another. For example, the pioneering developmental psychologist Jean Piaget ^[3,4] proposed that it was not until well into the first half year following birth that vision and touch begin to be integrated. Through acting on objects, tactile feedback was thought to gradually endow the two dimensional visual image of an object with three dimensionality. The attainment of perceptual abilities such as size and shape constancy, visually guided reaching, and object permanence were thought by Piaget and his colleagues ^[5] to be slow to emerge and to depend on the gradual development of sensory integration. Prior to this integration, the visual world of the infant was thought to consist of images shrinking, expanding, changing shape, and disappearing and then reappearing. Until the gradual achievement of sensory integration, infants were thought to perceive unrelated patterns of visual, acoustic, or tactile stimulation, expressed by the well-known description of the world of the newborn infant by William James as a “blooming, buzzing confusion”.

Infant-based research performed over the last several decades has seriously challenged this traditional view of early sensory organization and perceptual development. We now know that the senses function in concert even in very early infancy and that young brains are organized to use the information they derive from the various sensory systems to enhance the likelihood that objects and events will be detected rapidly, identified correctly, and responded to appropriately, even during very early development ^[6]. Infants are sensitive to audio-visual synchrony from birth. For example, even newborns can match visual with auditory information ^[7] and orient visually towards a sound ^[8]. By four months of age, infants presented with two superimposed films and an audio track that corresponds to only one of the films will attend to the film that is in synchrony with the soundtrack^[9]. Such abilities are likely based on young infants' sensitivity to relatively low levels of intersensory relations, including intensity and temporal synchrony ^[10,11].

Evidence obtained from neurophysiological research over the last decade indicates that the brain is remarkably skilled at integrating input from the different sensory systems to maximize the information available for perception and action ^[6,12,13,14]. Further, the ability to integrate information from different senses is not limited to any particular brain structure. Multisensory integration has been found in neurons at many locations in the nervous system, including sub-cortical areas like the superior colloculus, early cortical areas like the primary visual and auditory cortices, and higher cortical levels like the superior temporal sulcus and intraparietal areas ^{[13,15,16,17]}. Available evidence from human brain imaging studies also indicate that cortical pathways once thought to be sensory specific can be modulated by signals from other sensory modalities ^{[18,19,20,21,22]}.

This more “integrated” view of sensory organization can be traced in part to the ground-breaking work of the perceptual psychologists James J. Gibson^{[23, 24]} and Eleanor Gibson ^[25]. In a sharp break from the traditional association views of perceptual development described above, the Gibsons recognized that the existence of different forms of sensory stimulation was not a problem for the perception of unitary events, but instead provided an important basis for it. They argued that all senses should be considered as a “perceptual system” that interact and work together to pick up invariant aspects of stimulation. One important type of invariant information is amodal information that is common across the senses. Amodal information is not specific to a particular sensory modality but can be conveyed redundantly across multiple senses. For example, the rhythm or tempo of a ball bouncing can be conveyed visually or acoustically and is completely redundant across the two senses. One can detect the same rhythm and tempo by watching the ball's motion or by listening to its impact sounds. The sight and sound of hands clapping likewise share temporal synchrony, a common tempo of action, and a common rhythm.

We know from developmental research conducted over the past 30 years, inspired in large part by the Gibsons' innovative approach to perception, that young infants are adept perceivers of amodal information ^{[10,26,27,28]}. Infants readily detect the temporal aspects of stimulation such as synchrony, rhythm, tempo, and prosody that unite visual and acoustic stimulation from objects and events, as well as spatial co-location of objects and their sound sources and changes in intensity across the senses during the first 6 months following birth ^[29,30]. Such demonstrations of infants' detection of amodal information seriously question the notion that young perceivers have to learn to coordinate and somehow put together separate and distinct sources of information. By detecting higher-order amodal information common to more than one sense modality, even relatively naïve perceivers can explore a unitary multimodal event in a coordinated manner. The major task of perceptual development then becomes to differentiate increasingly more specific information through detecting invariant patterns across both multimodal and unimodal sensory stimulation ^[24,25,31]. Importantly, during perinatal development selective attention appears to be readily biased toward stimulus properties that are common or redundant across sensory modalities ^[32,33].

The Salience of Intersensory Redundancy During Early Development

To provide an organizing conceptual framework for defining the conditions that facilitate selective attention and perceptual learning during early development, Bahrick and Lickliter ^{[34,35, 36]} have proposed and provided converging evidence across species (human and quail), developmental periods (prenatal and postnatal), and skill domains (discrimination, learning, memory) in support of the Intersensory Redundancy Hypothesis (IRH). The IRH is a framework that describes how selective attention is allocated to different properties of objects and events in multimodal and unimodal stimulation. The IRH was derived from the application of a convergent-operations approach ^[27] that designs studies that can pursue parallel research questions across human and non-human animal subjects to identify developmental principles involved in early intersensory perception. In brief, the IRH addresses how the detection of amodal information (not specific to any one sense modality, such as rhythm, tempo, duration, and intensity) can guide selective attention and learning during early infancy and how this process is coordinated with perception of modality-specific information (specific to the individual sensory systems, such as color or pitch). Findings from both animal-based and human-based research consistently indicate that intersensory redundancy (the same information simultaneously available and temporally synchronized across two or more senses) promotes attention and perceptual processing of amodal properties of stimulation at the expense of other stimulus properties, particularly when attentional resources are most limited, such as during early development ^[35,36].

The IRH has proven to be a useful framework for advancing our understanding of the emergence and maintenance of a number of perceptual and cognitive skills observed during infancy, including the development of affect discrimination ^[37], rhythm and tempo discrimination ^[38], numerical discrimination ^{[39, 40]}, sequence detection ^[41], abstract rule learning ^[42], and word comprehension and segmentation ^[43,44]. These studies have all shown that intersensory redundancy can facilitate earlier and better detection of amodal information available in multimodal as compared to unimodal stimulation.

Prenatal Intersensory Stimulation

The prenatal environment provides the fetus a variety of tactile, vestibular, chemical, and auditory sensory information ^{[45,46,47,48,49,50]}. Although little research has directly focused on this issue, the human fetus likely experiences a great deal of integrated multimodal stimulation across the auditory, vestibular, and tactile senses in utero. For example, when the mother walks, the sounds of her footsteps can be coordinated with tactile feedback as the fetus experiences changing pressure corresponding with the temporal patterning and shifting intensity of her movements, as well as accompanying and coordinated vestibular changes. In addition, the mother's speech sounds, laughter, heart beat, or sounds of breathing may create tactile stimulation that shares the temporal patterning of the sounds as a result of changes in the musculature involved in producing the sounds.

Fetuses also engage in spontaneous motor activity of limbs and body ^[51], providing themselves temporally organized cyclic stimulation. When the fetus moves in the uterus, the movement generates both proprioceptive feedback as well as temporally coordinated tactile consequences of the motion, such as changes in pressure on the skin. Additionally, the mother also responds with temporally coordinated movements to externally generated sounds. For example, she may dance or exercise to music, startle to a loud noise, engage in conversation which has a distinctive turn-taking contingent structure, all of which produce movements that have tactile and/or vestibular correlates that share intensity and temporal patterning with the sounds. Thus, the fetus likely has ample opportunity to become familiar with and detect redundant stimulation across the various senses during the late stages of prenatal development. The role of this prenatal intersensory experience in the normal development of sensory integration and organization is currently not well understood.

It is certainly the case that during prenatal and postnatal development, fetuses and infants are ongoingly exposed to self-generated and externally generated multisensory stimulation. Evidence from research with both non-human animals and human fetuses and infants indicates that the specific stimulation histories of the sensory systems during prenatal and early postnatal development plays a key role in the development of selective attention, as well as early perceptual and cognitive development ^[27,32,35]. Of course, experiential manipulations of human fetuses and neonates are necessarily limited in scope and duration, and the traditional experimental manipulations used with animal subjects, such as sensory deprivation or sensory augmentation, are generally prohibited. As a result of these necessary restrictions, animal-based research has provided most of our advances in the understanding of the emergence of intersensory organization, including the importance of the timing of sensory experience during perinatal development ^{[52,53,54,55,56]}, the strong intermodal linkages of the sensory modalities during perinatal development ^{[57,58,59,60]}, and the critical role of intersensory redundancy in guiding and shaping early selective attention, and in turn, perception, learning, and memory ^{[32, 61,62]}.

One obvious advantage of the use of animal subjects to study sensory organization and perceptual development in the perinatal period is the ability to readily alter both the timing and amount of particular sensory experience available to the developing fetus. Animal-based research employing sensory deprivation or sensory augmentation during the perinatal period have yielded a useful body of information regarding the experiential conditions necessary for the normal development of early sensory and perceptual organization in animal infants ^{[52,63,64,65,66,67,68,69]}. This body of research has demonstrated that patterns of sensory stimulation available during the prenatal period actively shape emerging perceptual and cognitive capabilities. More specifically, this research indicates that the specific effects that sensory experience have on early perceptual development and sensory integration depend on a number of interrelated factors, including (a) the timing of sensory experience, (b) the amount of sensory experience, and (c) the type of sensory experience encountered by the fetus or the newborn ^[70].

Structure/Function Dynamics Across the Sensory Systems

It is important to keep in mind that the various sensory systems do not start out at birth on equal footing. This is the case because the sensory systems of birds and mammals, including humans, do not become functional at the same time in prenatal development. Rather, the sensory systems become functional in a specific and invariant sequence across early development: tactile > vestibular > chemical > auditory > visual ^[46,71,72]. As a result, because of the timing of their onset of function, the various sensory modalities have markedly different developmental histories at the time of birth. For example, the earlier developing tactile and vestibular systems have had much more experience during the late stages of gestation than has the later developing auditory system. These temporal dynamics likely have significant consequences for the course of early postnatal perceptual development ^[49] and much remains to be learned about links between the order and timing of prenatal sensory experience and subsequent postnatal perceptual processing.

Turkewitz and Kenny ^[73] proposed that the differential timing of sensory system onset provides a context in which earlier developing sensory systems can develop without competition or interference from later developing sensory systems. One approach to examining the importance of asynchronous sensory development is to alter the time when particular sensory input would normally be present during the perinatal period. Using this approach, Lickliter ^[53] found that the introduction of unusually early prenatal visual experience interfered with species-typical auditory responsiveness in bobwhite quail chicks following hatching. Chicks that experienced patterned light prior to hatching did not exhibit a naïve preference for their species-specific maternal call, a reliable phenomenon in chicks not receiving prenatal visual stimulation. Related research demonstrated that increasing the amount of tactile and vestibular stimulation availability prenatally likewise altered postnatal auditory and visual responsiveness in quail chicks ^[74]. Importantly, differences in the timing of augmented prenatal stimulation led to different patterns of auditory and visual responsiveness following hatching. No effect on normal visual responsiveness to maternal cues was found when exposure to tactile and vestibular stimulation coincided with the emergence of visual function, but when exposure took place after the onset of visual functioning, chicks displayed enhanced responsiveness to the same maternal visual cues. When augmented tactile and vestibular stimulation coincided with the onset of auditory function, embryos subsequently failed to learn a species-typical maternal call prior to hatching. However, when given exposure to the same type and amount of augmented stimulation following the onset of auditory function, embryos did successfully learn the individual maternal call ^[66]. These findings indicate that augmented stimulation to earlier-emerging sensory modalities can either facilitate or interfere with perceptual responsiveness in later-developing modalities, depending on when the modified prenatal stimulation takes place.

Changes in Sensory Organization Associated with Changes in Sensory Experience

The limited sensory capacities of the embryo and fetus (as a result of the sequential onset of sensory system function prenatally) and the constrained and buffered developmental context of the uterus combine to effectively limit and regulate the relative amount, type, and timing of sensory stimulation available during the prenatal period. These limited and regulated patterns of sensory stimulation associated with prenatal development are profoundly disrupted by preterm birth. Infants born weeks or even months before term receive dramatically altered amounts, types, and timing of sensory stimulation when compared with full-term infants. These include significant modifications in normal patterns of somesthetic, vestibular, proprioceptive, olfactory, auditory, and visual stimulation ^[70]. For example, the preterm infant in the NICU receives decreased amounts of some types of sensory stimulation normally available in utero (tactile and vestibular stimulation from maternal motion) and substantially increased amounts of other types of stimulation not present in the interuterine environment (unfiltered auditory stimulation and patterned visual stimulation). The perceptual and cognitive consequences of these alterations in light, sound, and movement are currently not well understood, but studies have suggested that the atypical sensory environment provided the high risk preterm infant in the NICU can have enduring effects on the developing premature brain ^[75,76].

Although little is known at present about how infants integrate multisensory information at the neural level ^[77,78], research from animal based research suggests that modifications of normal patterns of perinatal sensory experience can have significant effects on early brain growth and development. For example, Markham, Shimizu, and Lickliter ^[79] presented augmented amounts of auditory stimulation to bobwhite quail embryos during early, middle, or late prenatal development and then tested postnatal responsiveness to species typical auditory and visual cues. Embyos receiving auditory stimulation during middle or late stages of prenatal development showed altered postnatal visual responsiveness when compared to controls. Prenatally stimulated birds also showed a greater number of cells per unit volume of brain tissue in deep optic tectum, a midbrain region implicated in multisensory function. These results indicate that modified sensory experience delivered during prenatal development can have effects on postnatal mutimodal perception as well as on the developmental trajectory of brain growth and development. Importantly, these effects were temporally constrained – when the sensory modification occurred mattered.

Working at the neurophysiological level of analysis, Wallace and Stein ^[69] provided a striking example of the neural consequences of being reared in a modified, species-atypical environment. In this study domestic cats were raised from birth to adulthood in highly controlled sensory environments that allowed the systematic manipulation of the temporal and spatial features of audio-visual experience. Cats reared in this modified sensory environment, in which visual and auditory stimuli were paired to be temporally synchronous, but originated from different locations (spatially disparate), showed significant changes in the neural activity evoked by multisensory events. In particular, neurons located in superior colliculus developed a form of multisensory integration in which spatially disparate audio-visual stimuli were integrated in the same way that neurons in normally reared cats integrate audio-visual stimuli from the same location. Similarly, King and Carlile ^[80] found that ferrets deprived of visual experience during early development show abnormal topography and precision of spatial tuning of individual neurons in their superior colliculus, resulting in the misalignment of their auditory and visual spatial maps.

Similar results have also been reported in human based research. Le Grand and colleagues ^[81] compared face processing in normal individuals with those for whom visual input had been restricted to one hemisphere from birth until 2-6 months of age due to congenital cataracts. They found that even after more than nine years of recovery, early deprivation of visual input to the right hemisphere severely impaired configural face processing. Early deprivation to the left hemisphere did not. These results are particularly striking, in that when visual stimulation was delayed by as little as two months, permanent deficits were observed.

Sensory deprivation or augmentation in one sensory modality can also have effects on the development of the other senses ^[52,53,82]. Studies of deaf and blind humans have provided a wealth of evidence of increased capabilities and compensatory expansion in their remaining modalities ^[83]. For example, individuals who become blind early in life can process sounds faster, localize sounds more accurately, and have sharper auditory spatial tuning than sighted individuals ^[84,85]. Deafness likewise leads to a change in the spatial distribution of visual attention, with an enhancement of visual attention towards the peripheral visual field ^[86,87]. Putzar and colleagues ^{[88, 89]} recently documented that human adults deprived of visual experience during the first 5-24 months following birth as a result of congenital cataracts show reduced audio-visual interactions as adults. Individuals who received early visual deprivation were impaired in both face recognition and in integrating auditory and visual speech signals when compared to controls. In this study, multisensory capacities had not fully recovered in adulthood, even after at least 14 years of visual experience following cataract removal.

Taken together, these animal and human-based studies of sensory augmentation and sensory deprivation suggest that neural plasticity in early development is considerable. This plasticity is developmentally determined and allows neural systems to adjust to perturbations in the internal or external environment. It appears that neural plasticity allows sensory experience during early development to leave lasting structural and functional changes in the brain that can influence the nature and course of intersensory interactions. Importantly for concerns with care of the preterm infant, plastic changes across brain systems and related behavior vary as a function of the timing and the nature of changes in experience ^[83].

Implications for Care of the High-Risk Preterm

Our growing appreciation of the plasticity and experience-dependent nature of early sensory organization underscores the complexity of the challenge of identifying optimal care and management of the high-risk preterm infant. What will be effective or optimal for a preterm infant is a function of many interrelated factors, including at the very least their sensory and perceptual capacities, the maturity and integrity of their nervous system, and the particular characteristics of the sensory stimulation provided or denied.

In light of the remarkable plasticity of sensory organization during early development, the significant modifications of sensory experience that come with preterm birth are likely to have a range of effects on the normal course of the development of sensory organization. That being said, we are a long way from understanding the particulars. Given that auditory experience is typically available prenatally and that visual experience is not normally available until after birth, is there some necessary period or level of auditory experience in the period before birth for the emergence of normal patterns of postnatal perception? Does the unusually early visual experience associated with preterm birth and the resulting dramatic increase in the intensity and amount of auditory and visual stimulation interfere with normal auditory or visual development? What kinds of sensory stimulation is the fetus, preterm, and full-term infant particularly sensitive to? These important questions remain mostly unanswered at present. Further, little conclusive evidence is currently available about when, how much, and what type of sensory stimulation regimes are best suited to promote optimal outcomes during the various developmental stages associated with the perinatal period.

As briefly reviewed above, we do know that the sensory systems are strongly linked in the fetus and the neonate, such that alterations in sensory stimulation presented to one sense can result in changes in responsiveness not only in that modality but also in other sensory systems as well. We also know that detection of amodal stimulus properties, such as synchrony, intensity, tempo, and rhythm, is promoted by redundancy across sensory modalities and is involved in the emergence of normal patterns of perceptual organization. Young infants must learn to selectively attend to relevant information, screen out irrelevant information, and efficiently detect which patterns of sensory stimulation constitute unitary multimodal events (for example, the face and voice of a person speaking) and which patterns are unrelated. These emerging skills are facilitated by intersensory processing and the detection of redundant amodal information, including temporal synchrony, rhythm, tempo, and intensity ^[36,36].

The nature of delivery of the stimuli that preterm infants are exposed to in the NICU may, however, reduce the amount or availability of intersensory redundancy and, in turn, be detrimental to the development of early sensory integration ^[70,90]. For example, conditions in the NICU often do not allow preterm infants to experience stimulation in one modality concurrent with stimulation in other sensory modalities. In the full term newborn, auditory stimulation typically results in an orienting response, a turning of the eyes in the direction of the sound source. This allows the infant to perceive the auditory and visual characteristics of the object or event from which the sound originates. In the NICU, sound sources are often not visible to the infant, even if the infant is able to turn toward them. Sounds (such as respiratory and monitoring equipment) typically occur independent of stimulation to other sensory modalities, and provide little if any opportunity for the infant to match a particular sound with its visual and tactile referents. The short term and possible long term consequences of this reduced opportunity for intersensory redundancy on the preterm infant's emerging patterns of selective attention, perceptual processing, and learning are at present unknown. Importantly, social events provide high amounts of sensory redundancy relative to most non-social events. Parents and other caretakers can provide social stimulation to the high-risk infant that contains a great deal of amodal redundancy across tactile, auditory, and visual sensory systems. For example, audiovisual speech is rich with intersensory redundancy uniting the tempo, rhythm, and intensity shifts across faces and voices. This multimodal and redundant stimulation fosters the emergence of social orienting in early development by attracting and maintaining selective attention to faces, voices and audiovisual speech. This can in turn promote early social development, as well as related perceptual and cognitive development.

It is interesting to note that recent research has indicated that multisensory integration skills are associated with the development of intellectual abilities in school-age children ^[91]. In particular, children with enhanced multisensory integration in quiet and noisy conditions were more likely to score above average on the Wechsler Intelligence Scale for Children. This finding underscores the need for additional studies on the availability and effective use of intersensory experiences in the NICU care environment. Shifting the focus of study from whether experience contributes to intersensory development to how particular experiences at particular times influence intersensory development can enhance progress in the design of care and intervention programs for infants born at different levels of prematurity. We still have a long way to go to achieve this challenging goal, and more studies that include the biology, behavior, and environment of the preterm in the experimental design are needed. Theoretical frameworks and statistical and modeling tools that effectively address the interactive effects that occur across these levels of analysis are also needed.

Brief Summary

The last two decades have seen a dramatic increase in research activity on multisensory integration. Information drawn from a range of organisms, including humans, has advanced our knowledge of the developmental dynamics involved in early sensory organization. Data indicate that the sensory systems do not develop in isolation. Rather, they develop and function in concert with other sensory systems, even during the prenatal period. Converging evidence from behavioral, neurophysiological, and neuroimaging studies are providing a new way of thinking about the development of sensory organization and multisensory perception. This new framework recognizes that the young brain is able to integrate input from the various sensory systems to maximize the information available for perception and action. This framework also highlights the remarkable degree of neural plasticity present during early development, raising important and challenging questions about how to best manage the sensory environment of the preterm infant. More than a decade of research suggests that the nature of the delivery of sensory experience that preterms receive in the NICU can over-stimulate later developing sensory systems (i.e., auditory and visual) and under-stimulate earlier developing systems (i.e., tactile and vestibular), while also reducing the amount and availability of intersensory redundancy, which has been shown to be important to early selective attention, multisensory processing, and the emergence of normal patterns of early perceptual organization. We are still a long way from understanding the specific pathways and processes by which this unique sensory ecology of the NICU influences perceptual, behavioral, and cognitive development and additional research will be required to make informed decisions regarding how to best support the optimal development of the preterm infant.