Auditory object perception: A neurobiological model and prospective review

Abstract

Interaction with the world is a multisensory experience, but most of what is known about the neural correlates of perception comes from studying vision. Auditory inputs enter cortex with its own set of unique qualities, and leads to use in oral communication, speech, music, and the understanding of emotional and intentional states of others, all of which are central to the human experience. To better understand how the auditory system develops, recovers after injury, and how it may have transitioned in its functions over the course of hominin evolution, advances are needed in models of how the human brain is organized to process real-world natural sounds and “auditory objects”. This review presents a simple fundamental neurobiological model of hearing perception at a category level that incorporates principles of bottom-up signal processing together with top-down constraints of grounded cognition theories of knowledge representation. Though mostly derived from human neuroimaging literature, this theoretical framework highlights rudimentary principles of real-world sound processing that may apply to most if not all mammalian species with hearing and acoustic communication abilities. The model encompasses three basic categories of sound-source: (1) action sounds (non-vocalizations) produced by ‘living things’, with human (conspecific) and non-human animal sources representing two subcategories; (2) action sounds produced by ‘non-living things’, including environmental sources and human-made machinery; and (3) vocalizations (‘living things’), with human versus non-human animals as two subcategories therein. The model is presented in the context of cognitive architectures relating to multisensory, sensory-motor, and spoken language organizations. The models’ predictive values are further discussed in the context of anthropological theories of oral communication evolution and the neurodevelopment of spoken language proto-networks in infants/toddlers. These phylogenetic and ontogenetic frameworks both entail cortical network maturations that are proposed to at least in part be organized around a number of universal acoustic-semantic signal attributes of natural sounds, which are addressed herein.

1. Introduction: A new model for auditory perception, and why we need one

Categorization and recognition of objects are crucial to survival, as it allows us to interact with the world as well as predict what might happen and what actions we may immediately need to take. What defines an ‘object’ (visual, auditory, or haptic; see glossary) is not static, and changes due to experience, circumstances, and faculties of the perceiver. Nonetheless, we can gain understanding of the mechanistic underpinnings of cognition in humans in part by studying how we process and interact with the objects that we perceive (Varela et al., 1991). Because human and non-human primates are highly visiondominated species, and visual object processing in humans has been relatively easy to study in neuroimaging environments (such as with functional magnetic resonance imaging; fMRI), neurobiological models of human perception have been more thoroughly developed by studies in the realm of vision (Bar et al., 2001; Kaas and Collins, 2001; GrillSpector, 2003). Moreover, the visual acuity and ability to discriminate visual objects at a basic level appears to be reasonably similar in monkeys, great apes, and humans (Schmitt et al., 2013). Thus, models of visual object processing in the brain have been well formulated and refined through cross-species comparisons, with the presumption that the basic neuronal architectures for parallel, hierarchical processing in visual cortices have been relatively stable over the time course of primate evolution. However, hearing is different, both in terms of the physical signal attribute differences that need to be reconstructed by the brain, but more so the level of spectro-temporal discrimination that evolved in hominins (see glossary). Out of survival necessity, and now unique to humans, the auditory-vocal system had evolved the ability to convey and interpret increasingly subtle socially relevant emotional states of others (Donald, 1991; Munoz-Lopez et al., 2015; Fritz et al., 2016). Moreover, the formalization and unification of thought and knowledge in cortex is thought to have set the stage for spoken language systems to develop together with vastly greater degrees of auditory working memory abilities. In other words, humans may evoke a number of different sound processing strategies relative to other animals with dependence on listening task demands, intentional or unintentional influences by language systems, and/or our ability to decode a variety of high order signal attributes that may not be behaviorally relevant to other species. Notwithstanding, many models of sensory perception acknowledge that object representations are quintessentially multisensory in global organization (Calvert and Lewis, 2004; Lewis, 2010; Murray et al., 2016) and we further assert that the auditory system in humans is also shaped by sensory-motor organizations in the brain that relate to acoustic communication. In the current review, we present a fundamental neurobiological model of hearing perception for natural sounds that addresses the question of what an “auditory object” is and the degree to which this is a useful concept for thinking about the cortical mechanisms that mediate hearing perception.

While many signal processing models have been developed for early stages of visual and auditory systems in human and non-human primates, such bottom-up models often fail to fully address the questions of “why” or to what end are the stimuli being processed. Many top-down cognitive models of sensory perception, even if heavily vision-dominated, do capture principles of knowledge representation that apply across sensory modalities (Tranel et al., 1997; Caramazza and Shelton, 1998; Caramazza and Mahon, 2003; Damasio et al., 2004). However, tests of such models often use language as either a stimulus, part of the task, or in other ways may inadvertently incorporate language system recruitment, which, from a hominid evolution perspective, may mask more fundamental processing principles or organizations of the auditory system. Thus, there remain significant gaps in our understanding of non-linguistic natural sound recognition mechanisms by the human brain at rudimentary perceptual levels.

Advances in models of hearing perception for everyday real-world natural sounds should be able to account for (i) previously established models of how “bottom-up” acoustic signal attributes at the cochlea become reconstructed along subcortical pathways and feed into appropriately specialized cortical processing pathways; (ii) apply to basic models in other mammals with hearing and oral communication ability; (iii) help account for the neuropsychology of auditory agnosias observed in some individuals after specific brain lesions, and thus incorporate “top-down” cognitive models of knowledge representations; and (iv) address how uniquely human qualities for comprehending spoken language, music forms, and interpreting subtle emotional cues may be interrelated with more rudimentary acoustic signal processing systems. Adhering to these considerations, the purpose of this review is to present a simple, formalized neurobiological model of hearing perception at an acoustic-semantic level, which has largely been refined based on fMRI neuroimaging findings over the past decade.

First, the model is introduced briefly below. Then, after addressing considerations of both bottom-up visual and auditory system models (Section 2) and top-down cognitive models (Section 3), we further elaborate on the general tenants of the hearing perception model at a more theoretical level (Section 4). This is followed by a discussion of testable predictions based on the model both in human and non-human primates (Section 5), and of limitations and future directions (Section 6). We believe that cortical models of auditory perception, in concert with visual perception models (and haptic perception, though this falls outside the scope of this review), will make unique contributions to understanding the nature of the human brain/mind and perceptual awareness on a number of fronts. This includes (1) fundamental advances in understanding how cognition and perception may function more globally, (2) understanding spoken language development in children, (3) understanding central hearing deficits and recovery after brain injury, (4) advances in biomimetic hearing aid algorithm designs, and (5) advances in anthropological models of oral communication during hominin evolution, by revealing potential vestiges in cortical networks that antedated modern spoken language systems.

In its simplest form, the proposed neurobiological model of hearing perception asserts that the cortical processing of meaningfulness associated with all natural sounds can be characterized by one of three major acoustic-semantic categories (Fig. 1, bold text) depicted effectively as a 2×2 design with subcategories therein. The first major category distinction is living (animate, biological) versus non-living. This boundary was derived from nearly a century of neuropsychology literature (Martin et al., 1996; Moore and Price, 1999; Grossman et al., 2002; Damasio et al., 2004), and further incorporates theories of embodiment mechanisms that may function to convey a sense of meaning or intent behind biological sounds to the listener (Barsalou et al., 2003; Barsalou, 2008). A second major category distinction is vocalizations versus non-vocalizations, wherein harmonic content (specific frequency combinations) reflects a primary acoustic signal attribute that is probabilistically more characteristic of vocalizations relative to most other behaviorally relevant action sounds (defined here as being natural sounds devoid of vocal content). There are no non-living vocalizations from an ethological perspective of this otherwise 2×2 model, though an interesting possibility is that this conceptual niche may in part be filled by sounds of musical instruments at some higher cognitive levels (addressed in Section 6).

Fig. 1. A neurobiological model of hearing perception for different categories of real-world, natural sounds.

This model was refined largely from recent human neuroimaging studies, but should apply to most mammalian species with hearing and sound production capabilities.

The three major acoustic-semantic categories encompass other subcategories of natural sounds (Fig. 1, plain text). For the subdivision of “living things”, this includes actions sounds and vocalizations made by conspecific versus non-conspecific animals (i.e. human versus non-human in this review). Cortical representations of conspecific sounds are generally presumed to develop through experience, mediated by attentional systems that relegate greater processing to socially and behaviorally relevant nuances contained in conspecific action events and their resulting sounds (e.g. skilled tool use sounds) and in conspecific vocalizations (e.g. native speech sounds and singing talent). These subcategories are also likely to develop richer cortical representations due a listener’s physical experiences with producing the sounds themselves, and thus encoding a sense of a sound’s meaningfulness through audio-motor associations and ‘embodiment’ mechanisms. The generalizability of this model to all mammals would, of course, need to be adapted to account for a given species neurobiological constraints for sound transduction at the cochlea and capabilities for sound production and acoustic communication.

The category of non-living environmental action sounds (e.g. rain, water flow, fire, wind) includes the evolutionarily more recent subcategory of sounds produced by human-made automated machinery (mechanical action sounds devoid of an agent immediately instigating the action). Both of these non-living subcategories contain a number of distinguishing acoustic-semantic attributes that allow them to be characterized along an object-vs-scene signal dimension (addressed in Section 2), and thus perhaps best associated with the concept of an “auditory object”.

Because the three major category boundaries of this model have foundations in acoustic signal attribute processing that are probabilistically related to semantic features of their sound sources, we argue that these natural sound categories (and subcategories) are founded upon acoustic-semantic universals, for which the brain evolved to efficiently process, and serve as a foundation for other higher cognitive architectures, being processed by cortical networks that ultimately mediate hearing perception in non-deaf individuals.

2. Bottom-up perspectives of vision and hearing models

Over the past few decades, neuroimaging studies in human and non-human primates have revealed a panoply of parallel, hierarchical pathways for the processing of visual information (Farah and Aguirre, 1999; Bar, 2004; Palmeri and Gauthier, 2004; Bi et al., 2016), which has been compared and contrasted with audition (Kaas and Hackett, 2000b; Adams and Janata, 2002; Calvert and Lewis, 2004; Poremba and Mishkin, 2007; Lewis, 2010). This section aims to give the reader basic knowledge of these hierarchical pathways and networks of the mammalian brain that underlie perception of “what” the sound is, namely hearing for perception, recognition, and identification, which will be compared with the hierarchical pathways for visual object processing. Below we briefly summarize visual and auditory system models from a bottom-up unisensory signal processing perspective.

Visual inputs that define objects arrive as light rays along the two-dimensional array of the retina in the form of changes in luminance and frequency over time, and remains organized in a retinotopic (visuotopic) manner that is propagated in the neural code along various parallel, hierarchical pathways in visual cortices – a set of features that allows very high precision in spatial representation and the ability to discriminate both moving and static stimuli (Box 1). Visual objects can be segmented from the background at a fine spatial resolution, within roughly 100–250 ms (Thorpe et al., 1996). Object processing primarily, though not exclusively, involves ventrally directed streams from occipital to inferior temporal cortices (Ungerleider et al., 1982; Goodale and Milner, 1992; Freud et al., 2016). Visual object representations are built up in the brain through collections of neurons with lower-level visual receptive field properties to higher-level stages with receptive fields having greater specificity for different combinations of lower level visual features (Felleman and Van Essen, 1991; DeYoe et al., 1994). This includes increases in size and complexity of receptive fields, ranging from center-surround cells in the lateral geniculate nucleus, to representations of edges, lines/curves in primary visual cortices, to “object-like” features in the lateral occipital cortices (LOC). This information reaches different portions of the inferior temporal cortices that show selective, or at least preferential, processing for specific categories of visual entities such as for faces, distinct categories of objects, places, and scenes (Sergent et al., 1992; Kanwisher et al., 1997; Epstein et al., 1999; Pietrini et al., 2004). Lesions to various brain regions can lead to specific visual agnosias, including deficits in face recognition (prosopagnosia), an inability to read written words (pure alexia), or for general misperception of objects (visual object agnosias), among others (Geschwind, 1965; Damasio et al., 1982; Liepmann, 2001; Karnath et al., 2009).

Box 1. Comparison of real-world visual and auditory objects.

In this box, we see a visual and auditory landscape, as experienced by the woman in blue as a viewer (A), and a listener (B). Note differences in the richness of spatial detail, number and types of objects perceived in the visual vs. auditory world. The visual world is more spatially precise and detailed, such that details of the grass, clouds and dogs’ coats can be ascertained in high resolution due to retina properties. In contrast, only objects in motion make vibrations that activate the cochlea, but these sounds can travel even through a visual occlusion like a fence, and if loud enough can be detected even if looking away or asleep. Objects like the silent dogs are invisible to the blindfolded listener, while the hammering neighbor behind the fence would be undetected by a deaf viewer. All categories proposed in the model (Fig. 1) are present. This scenario will parallel discussion of the model in the text in Boxes 2 and 3.

Visual motion processing is another attribute of visual perception that impacts object recognition, ranging from structure-from-motion to gaining social cues from complex biological motion. Since sound production (by ‘auditory objects’) necessarily implies physical motion of some form, a consideration of visual motion processing systems (i.e. beyond static visual image processing) are imperative when making comparisons with auditory object processing systems. Ostensibly, a listener’s ability to recognize real-world auditory objects and acoustic events often requires processing of sound wave signal changes over considerably longer periods of time relative to vision, in some cases on the order of several seconds to recognize temporal patterns, and assert a sense of accurate recognition or identification.

The auditory system also has a parallel, hierarchical organization but there are key differences compared to vision. The everyday sounds we hear are initially represented at the cochlea, which is organized around tonotopy—a map of high to low frequencies of sound waves (organized roughly like a piano keyboard). Neural representations of any sound can be precisely defined physically by three attributes: the specific frequencies present, their relative intensities, and duration. The auditory system is far more sensitive than the visual system to timing and fast changes in signal energy, with temporal resolution of roughly 10’s msec (Phillips, 1999). Information is sent up from the cochlea to the cortex via roughly half a dozen brainstem areas (Kandel et al., 2000), incorporating receptive fields with monaural or binaural representations conveying spatial location and increasing selectivity for various combinations of spectral and temporal signal information. The receptive field properties of auditory neurons similarly build-up in complexity and specificity along the cortical mantle to represent objects and action events (Rauschecker et al., 1995, 1997; Kaas and Hackett, 1998). Representations of sound at the level of primary auditory cortex proper maintains at least a roughly tonotopic organization in humans (Wessinger et al., 1997; Formisano et al., 2003), and neuropsychological studies indicate that information must reach this stage for normal bottom-up conscious awareness of sound and auditory object perception (Engelien et al., 2000). Auditory “core” regions of cortex receive strong thalamic inputs, are tonotopically organized, and thus generally regarded as primary auditory cortices (Kaas and Hackett, 1998; Sweet et al., 2005). These primary auditory regions send information on to higher hierarchical stages, which in macaques are termed “belt” and “parabelt” regions that surround the core (Rauschecker et al., 1995; Kaas and Hackett, 1998, 2000a; Rauschecker, 1998). In humans, primary auditory cortices (two or three auditory fields) are located along the medial two-thirds of the transverse temporal gyrus, or Heschl’s gyrus (Rademacher et al., 2001; Formisano et al., 2003; Talavage et al., 2004; Lewis et al., 2009), which are depicted as functional landmarks in Figs. 2 and 3.

Fig. 2.

A bottom-up model for vocalization signal processing. (a–d) Three-dimensional spectrograms of example sound stimuli and a pure tone. Note the prominent stacks of energy along frequency bands of the vocalizations (b–c). (e) Chart illustrating harmonic content ranges (in dB_HNR) of various types or classes of communicative vocalizations. The y-axis depicts actual or relative degrees of fMRI activation in the left middle superior temporal gyrus (mSTG) region, with the blue curve depicting a response profile to animal vocalizations. Blue dots on curve correspond to sounds depicted in spectrograms. Ovals and boxes hovering above the curve depict dB_HNR values of different categories of animal vocalizations (blue hues) and human vocalizations (violet hues). (f) Progression of cortical pathways for processing harmonic content and information content of vocalizations. The “rainbow arrows” depict two prominent processing pathways showing increasing specificity for human vocalizations. Intermediate colors depict regions of overlap (refer to key for color codes). Data are illustrated on slightly inflated renderings of averaged cortical surface models (all at p_corrected < 0.01). Adapted and reprinted from Lewis et al. (2009) with permission from the publisher. Refer to text for other details.

Fig. 3. Meta-analysis of brain regions preferential for each of the three major categories of natural sound in the model.

The data from select published studies were adapted to be color coded and overlaid in transparent layers, revealing the major cortical regions and networks associated with natural sound processing. Red colored cortices correspond to vocalizations, yellow to biological action sounds, and blue to non-living environmental and mechanical action sounds. Studies include selected results restricted to the defined sound category boundaries from Lewis et al., (2004, 2005, 2006, 2009); Engel et al. (2009); Lewis et al. (2011a), (2011b), (2012); and Webster et al. (2017)). Refer to color key and text for other details.

The tonotopic organizations of primary auditory cortices (PAC) give way to other organizations, including broadly defined “what” versus “where” divisions, which respectively include ventral versus dorsal pathways as major divisions of labor (Rauschecker, 1998; Kaas and Hackett, 1999; Rauschecker and Tian, 2000). Akin to the visual system, a dorsally directed processing stream (relative to the location of PAC) is critical for processing the location of sound (“where is it”) relative to the listener’s body for purposes of potentially interacting with the sound-source, while a ventral stream is thought to be involved more in processing for perception (“what is it”). This basic division reported in macaque monkeys is also supported by human neuroimaging studies using fMRI (Belin and Zatorre, 2000; Clarke et al., 2002; Arnott et al., 2004, 2008; Barrett and Hall, 2006; Wang et al., 2008; Recanzone and Cohen, 2009), neuropsychological brain lesion studies (Goodale and Milner, 1992), and electroencephalography (EEG) imaging (Ahveninen et al., 2006; Murray et al., 2006), and is proposed to further include a possible stream for defining “how” a sound is produced (Belin and Zatorre, 2000; Johnson-Frey, 2003; Kellenbach et al., 2003; Lewis et al., 2005). An extension of this dual-stream theoretical framework has also been applied to models of spoken language perception (Obleser et al., 2006; Dick et al., 2007; Rauschecker and Scott, 2009), including the idea that dorsal systems may be involved in the framing and temporal dynamics of word processing while ventral systems are more involved in representing content (MacNeilage, 1998). We further address spoken language processing from the perspective of top-down influences in Section 3, while in this section we address more fundamental natural sound processing that is more likely to be relevant to mammalian auditory systems more generally.

The comparisons thus far of bottom-up signal processing in visual and auditory systems for object representation have led us to a stage for addressing details of neuroimaging evidence that helped establish the model as a parsimonious account of the neurobiological data to date. This includes (2.1) vocalizations (and speech signals), which entail the processing of quantifiably differing degrees of harmonic content, (2.2) action sounds produced by living things, which can often be ‘embodied’ by sensory-motor systems, and (2.3) action sounds produced by non-living things, which are less likely to be embodied by sensorimotor systems but can be associated with high level representations of visual objects and scenes (in sighted individuals). The acoustic signal attributes that distinguish these three major categories of natural sounds may seem complex mathematically from a signal processing perspective, but we propose that they can at least be probabilistically modeled and thus represent acoustic-semantic universals that serve to bridge bottom-up processing with top-down multisensory, sensory-motor, and cognitive representations (Section 3) that help address the end(s) to which natural sounds are being processed in a listener’s brain.

2.1. Vocalization processing pathways and harmonic content

Unique to visual object processing are attributes such as color, relative brightness, and figure-ground segregation of static non-moving images or scenes. Conversely, unique to auditory processing are attributes such as pitch and harmonicity, which are prevalent in vocal sounds that convey communicative intent and/or emotional states in many species. We argue that harmonic content represents a universal acoustic-semantic signal attribute that effectively shapes (or helps shape) the organization of cortical networks for vocalization perception, and thus ultimately for spoken language perception. Harmonic content, or harmonicity, can be quantified by a harmonics-to-noise ratio (HNR) value over discrete time periods of a sound (Boersma, 1993; Riede et al., 2001; Ferrand, 2002). For instance, a wolf howl has much stronger harmonic content than a snake hiss, which is readily evident by strong frequency bands (“stacks”) in their spectrograms (cf. Figs. 2a and c). Moreover, various subcategories of behaviorally relevant vocalizations (Fig. 2e, colored ovals and boxes) can at least roughly be organized along a continuum of harmonic structure (Lewis et al., 2009). The lower end of the harmonic content scale includes hisses, growls, grunts, and groans, which are mostly associated with threat warnings and phatic utterances expressing negative emotional valence (Austin, 1975; Lewis et al., 2009; Talkington et al., 2013). At the other extreme, whistling, howls, and vocal singing are characterized by relatively higher harmonic content. Adult-to-adult conversational speech lies in a range between these extremes, and interestingly, adult-to-infant speech by the same individual speakers shows higher degrees of harmonicity in addition to the higher pitch ranges typically associated with “motherese” (Cooper and Aslin, 1990; Mastropieri and Turkewitz, 1999; Falk, 2004a). With regard to speech signals, the stress phonemes of onomatopoetic descriptors of several English words symbolically representing the different subcategories of vocalizations (such as the ‘ss’ in hissing, the ‘gr’ in growling, and the ‘oo’ in mooing), also correlated with the relative harmonic content ranges of their respective vocal call subcategories.

To highlight the essence of the cortical pathways for vocal sound processing of the model (Fig. 1), Fig. 2 shows results from an fMRI paradigm that used the above-mentioned subcategories of vocalizations as sound stimuli (Lewis et al., 2009). This study revealed a sound processing hierarchy emanating out from PAC along the left and right hemisphere auditory cortices. The progression was based along dimensions of parametric sensitivity to harmonic content as well as increasing levels of information content or meaningfulness, and included three processing tiers (Fig. 2f, rainbow arrows). Outside of tonotopically organized core regions (Fig. 2f, yellow cortices), portions of the superior temporal plane showed parametric sensitivity to the harmonic content of artificially constructed sounds (green hues). Further lateral, the superior temporal gyri (STG) were regions parametrically sensitive to the harmonic content of animal vocalizations (dark blue hues). The processing of human speech sounds, which contained a greater degree of specific frequency combinations characteristic of human (conspecific) voice, resulted in activation located further antero-lateral and postero-lateral along the STG and superior temporal sulcus (STS) in the left hemisphere (purple hues). Similarly, human non-verbal vocalizations (e.g. sighs, moans, crying, laughter), which are conspecific signals that also contain specific combinations of co-modulating frequencies and presumably other higher order acoustic signal attributes that are learned to convey subtleties in meaning, recruited cortices along the STS, but mostly in the right hemisphere (pink hues).

Though the specific receptive field properties and processing mechanisms of vocalization signal preferring pathways remain to be more fully characterized, elements of harmonically structured signals inherent to vocalizations are clearly being utilized during vocal sound reconstruction, and are a feature unique to the hearing modality. Thus, in the model, harmonic content is proposed to represent an acoustic-semantic universal that the auditory system uses at a fundamental level as part of an organization to efficiently process natural sounds for determining meaningfulness.

This pathway for processing vocalizations as one of the major categories of natural sound is also illustrated as part of a meta-analysis (Fig. 2f data re-color coded in Fig. 3, contributing to red hues). This meta-analysis figure includes select data from several fMRI studies by our group to highlight the fundamental processing pathways and networks adhering to the acoustic-semantic categories defined in the proposed model. Neuroimaging results that further contributed to this meta-analysis are described briefly below and further in Section 3 (also see figure legend), followed by a formalization of the general tenants of the model.

Continuing with vocalization processing, human vocalizations seem to be special for humans (conspecifics). In general, conspecific vocalizations for a given species may recruit greater expanses of cortical hierarchies for signal processing due to life-long familiarity and experience, and the relatively greater degrees of attention devoted to extracting behaviorally relevant information. This notably includes the processing of harmonic-vocal sounds produced by caretakers that may even begin in utero (Lee et al., 2007; Webb et al., 2015). Neuroimaging studies of non-human animals have identified brain regions along auditory cortices showing specificity, or at least preferential activation, in response to that species conspecific calls and vocalizations, including dogs (Andics et al., 2014), macaque monkeys (Rauschecker, 1998; Petkov et al., 2008; Ortiz-Rios et al., 2015), and chimpanzees (Taglialatela et al., 2009). Based on behavioral studies, acute sensitivity to conspecific calls further seems to be a feature of other mammals including ungulates (e.g. deer, elk) (Lingle and Riede, 2014) and even aquatic mammals such as pinnipeds (e.g. walruses, seals) (Cunningham et al., 2014; Reichmuth and Casey, 2014). Note, however, that some emotional calls, especially distress vocalizations, appear to reflect evolutionary conserved mechanisms underlying both production and behavioral responses (Lingle et al., 2012). Thus, responses to some categories of emotional calls may reflect innate or ‘reflexive’ circuit mechanisms that could dissociate from the proposed hearing perception model, which emphasizes processing of natural sounds that are learned to have meaning to the listener through experience. Notwithstanding, in human neuroimaging studies, spoken language sounds lead to activation along specific brain regions, with comprehensible language being more lateralized to the left hemisphere (Binder et al., 1997, 2000; Belin et al., 2000; Belin and Zatorre, 2003; Obleser et al., 2006), and pitch, prosody, and emotional qualities in human vocals more heavily involving the right hemisphere (Zatorre et al., 1992; Buchanan et al., 2000; Gandour et al., 2004). Human voice sounds, in contrast to instrumental sounds, are also reported to show distinct responses in electroencephalography (EEG) studies (Levy et al., 2001). In infants, human voice leads to preferential activations (Fifer and Moon, 1989; Dehaene-Lambertz et al., 2002; Pena et al., 2003; Grossmann et al., 2010; Sato et al., 2012), attesting to their special status as a conspecific (or at least familiar) acoustic subcategory of vocalization that may be learned very early in life.

To identify which brain regions might show specificity for processing non-linguistic human vocalizations, Talkington et al. (2012) used fMRI to examine cortical responses to the processing of sounds produced by humans mimicking animal calls in contrast to hearing the original animal calls themselves as a critical control. Interestingly, hearing the human mimic sounds more strongly activated auditory belt/parabelt regions in the left cortical hemisphere (contributing to Fig. 3, red hues). These results suggested that a left lateralization bias may exist not only for speech sound processing but perhaps more generally for processing the spectro-temporal attributes characteristic of human (conspecific) vocal tract sounds, though this finding and its relation to lateralizations for spoken language processing remain to be further elucidated.

Conversely, hearing animal vocalizations, in contrast to humans mimicking the corresponding sounds in the above study, led to greater activation along a number of right lateralized brain regions (Fig. 3, contributing to red hues). These higher processing tiers were consistent with earlier described networks for processing affective prosodic cues of vocalization stimuli, such as slow pitch-contour modulations (Zatorre and Belin, 2001; Kotz et al., 2003; Friederici and Alter, 2004; Ethofer et al., 2006; Ross and Monnot, 2008; Grossmann et al., 2010). Thus, some of the right lateralized regions may have been related to processing the prosodic cues and other signal features in the original animal calls (Wilden et al., 1998; Farago et al., 2014) that were not adequately, or at least differently, conveyed by the human actors mimicking them. These findings were consistent with neuropsychological studies indicating that lesions to the right inferior frontal gyrus (IFG) and anterior insula region can lead to sensory aprosodia, an inability to interpret emotion in voice (Heilman et al., 1975).

In sum, pathways for non-linguistic vocalization processing are robustly present in both cortical hemispheres out to the STG/STS regions (Fig. 3, red hues). The left hemisphere appears to show relatively greater sensitivity to the processing of human (conspecific) vocal tract sounds and for linguistic content, while the right hemisphere shows processing biases for prosody and other emotional attributes. This lateralization effect for higher level conspecific communication signals is argued to be related to hemisphere specializations originally related to handedness, praxis skills, and gestural origins of language/communication, as addressed in later Sections. In contrast to vocalization sounds, however, we next consider how the auditory system deals with non-vocal “action sounds”.

2.2. Action events produced by living things

The next two categories, living and non-living action sounds, are defined here as representing all other types of natural sound signals other than vocalizations, and share many features that correlate in time across visual and/or sensorimotor modalities (e.g. cross-modal correlated changes in stimulus intensities, such as impact sounds). The intermodal invariant features of action sounds produced by non-living things, such as wind blowing through trees, often include complex visual motions that correlate (for sighted listeners) with sound signal textures and amplitudes (addressed in Section 2.3 below). Action sounds produced by living things, such as when hearing and viewing an individual dribbling a basketball, entail changes in signal energy that correlate in time across visual, auditory, and often with sensorimotor systems. Thus, a listener’s experience with producing similar sounds themselves are likely to establish associations (“embodiments”) that further convey potential intention or meaning behind the sounds when heard in isolation. These multisensory and audio-motor association distinctions, we argue, help establish other sets of acoustic-semantic universals used for organizing the mammalian auditory system to process action sounds produced by living things, as addressed below.

A recent fMRI study by our group directly tested where auditory pathways for processing action sounds by living things versus vocalizations might diverge (Webster et al., 2017). Non-human animal vocalizations and non-human action sounds were used to minimize confounds associated with potentially greater semantic processing of conspecific sounds. Relative to primary auditory cortices, vocalizations activated the STG regions bilaterally as expected (Fig. 3, contribution to red regions), while animal action sounds preferentially activated the posterior insulae bilaterally (Fig. 3, contributing to yellow hues). These regions were presumed to be associated with audio-tactile or audio-sensorimotor cortices, similar to the organization reported in macaques (Schroeder et al., 2001; Fu et al., 2003; Hackett et al., 2007). Thus, the divergence in processing for this major acoustic-semantic boundary was along intermediate auditory cortical stages, overlapping classically defined parabelt regions. However, at lower threshold settings, additional left-lateralized cortical regions were also preferentially activated by the action sounds, including frontal, parietal and posterior temporal regions that have previously been associated with mirror neuron systems (MNS) in human (Rizzolatti and Craighero, 2007; Molenberghs et al., 2012) and non-human primates (Rizzolatti and Craighero, 2004). While the implications of the MNS-like representations will be addressed further in Section 3 from a top-down perspective, these results support the notion that the audio-motor associations of biological action sounds reflect a form of acoustic-semantic attribute that may serve as universal signals for organizing or refining the mammalian “auditory” system to mediate hearing perception.

In dual stream vision models, motion processing has a strong dorsally directed component, involving a number of occipito-parietal and parietal cortices plus the lateral temporal visual motion area hMT—terminology derived from the macaque monkey area MT (Van Essen et al., 1981; Tootell et al., 1995). The visual processing of articulated biological motions preferentially occurs just anterior to hMT along the posterior middle temporal gyri (pMTG) and posterior superior temporal gyri (pSTS) regions (Grossman et al., 2000; Beauchamp et al., 2002; Grossman and Blake, 2002; Thompson et al., 2005; Han et al., 2013), and are reported to represent primary loci for complex natural motion processing (Martin, 2007; Lewis, 2010), most notably including human (conspecific) actions. Activation in the pSTS/pMTG complexes (Fig. 3, labeled yellow region) shows interaction or integration effects when corresponding sounds are also present (Calvert et al., 2000; Beauchamp et al., 2004b, 2004a; Taylor et al., 2006, 2009; Campanella and Belin, 2007; Campbell, 2008), and are generally activated by human action sounds in the absence of visual input (Lewis et al., 2004, 2006; Bidet-Caulet et al., 2005; Gazzola et al., 2006; Galati et al., 2008; Engel et al., 2009). These regions were further shown to be more strongly activated by human action sounds relative to non-human animal action sounds, and lesser still by non-living action sounds (Engel et al., 2009) or vocalizations (Webster et al., 2017). Thus, from a bottom-up signal processing perspective, these complexes appear to play a prominent perceptual role in transforming the spatially and temporally dynamic features of natural auditory (and visual) action information into a common neural code, conveying symbolic associations of physically matched audio-visual features. The pSTS/pMTG regions are also activated in association with hearing tool-use sounds and with manipulating virtual tools (Lewis et al., 2005, 2006). Hence, multisensory and sensorimotor association processing appears to be central to the supramodal processing functions of the pSTS/pMTG complexes.

Human action sound processing at a categorical level appears to develop prior to spoken language abilities, as assessed from a study of prelingual infants (Geangu et al., 2015). Using sound stimuli consistent with the category boundaries defined in the proposed neurobiological model, they examined event related potentials (electroencephalography methodology) of infants at 7 months of age, as they listened to human action sounds, human vocalizations, house-hold mechanical action sounds and sounds of the natural environment. Their results indicated that human action sounds were being differentially processed as a distinct category of socially relevant sound. This category-specific sound processing was occurring prior to the development of top-down language system influences, and also prior to formation of audio-motor associations with human action sounds, although they likely have visually observed other human conspecifics in their world using tools, with some level of pantomime or play tool-use movements.

In sum, a complex interplay of audio-visual associations and/or audio-motor associations can help shape the differential responsiveness of brain regions for processing different categories of natural sound (though also see Section 3), and namely action sounds produced by living things. However, associations with non-living sound sources are qualitatively different, leading to the third major category of natural sounds.

2.3. Action events produced by non-living things

Non-living action sounds, such as rain, ocean waves, fire and wind, produce sound in a manner that cannot be fully or meaningfully emulated by a listener’s own motor system (which makes the X-Men character “Storm”, who can control weather, a particularly interesting fictional superhero from the scientific perspective of this review). Consequently, the human brain must rely more heavily on learning and associating the acoustic signal attributes of non-living action sounds with visual motion cues (if sighted) and with tactile inputs. However, early blind individuals readily learn to recognize non-living action sounds (see Section 3), indicating that visual associations are not absolutely required.

In another line of reasoning, natural sound events, living or non-living, are usually not heard in complete isolation in real-world settings. Hence, the auditory system was proposed to have evolved to focus on streaming sound components that likely belong to a given sound-source (Bregman, 1990; Teki et al., 2011). This processing strategy shares analogies with segmenting of figure-versus-ground and structure-from-motion in the visual system (Parks, 1984; Yantis and Jonides, 1990; Rubin, 2001). Thus, one might similarly expect a model of hearing perception to show a dichotomy, or at least a gradation, in cortical organization for representing auditory objects versus auditory scenes (acoustic soundscapes and/or distracting ambient noises). In this regard, an important role of the auditory system is not only to alert and attend to potential auditory objects of interest, but also to perform dynamic acoustic accommodation by simultaneously “filtering out” the drone of less relevant background acoustic noise (Bregman, 1990), thereby freeing up attentional resources for other sensory or cognitive processes. In ferrets, directing attention to particular types of sound as either foreground versus background were shown to modulate primary auditory cortices (Fritz et al., 2007b, 2007a), and in humans detecting and recognizing animal calls amidst acoustically complex background scenes revealed activated foci along the left and right angular gyri (Maeder et al., 2001). Thus, some aspects of auditory object processing (in sighted and blind) may be addressed in terms of an object-vs-scene dimension.

Because the auditory system, we argue, places a premium on assessing whether or not a sound-producing action is being caused by a living agent (addressed in Sections 2.2 and 3), notably by engaging audio-motor embodiment mechanisms to ascertain possible intentful actions, a comparison between visual and auditory object-vs-scene processing systems might be more straightforward when examining sound-sources that fall within the category of non-living action sounds (Fig. 1, rightmost box). With this rationale in mind, another study by our group examined cortical networks showing sensitivity to auditory objects versus auditory scenes using ‘non-living’ environmental and mechanical sounds (Lewis et al., 2012), adhering to the category boundary definitions in the proposed model. Sounds were assessed psychophysically, revealing a continuum from object-like to scene-like exemplars that avoided perceptual processing issues related to embodiment and intention. This included environmental sounds rated as more object-like (e.g. water dripping in a cave) versus more scene-like (e.g. wind blowing through trees), plus mechanical sounds rated as more object-like (e.g. a ticking watch) versus more scene-like (e.g. industrial laundering machinery). Neuroimaging revealed a double dissociation of cortical processing regions on this object-vs-scene dimension, with foci along the left and right STG showing preferential activation to object-like sounds (Fig. 3, light blue hues) versus various cortical midline regions preferential for scene-like sounds (dark blue hues). What bottom-up acoustic or acoustic-semantic signals might have been contributing to this perceptual dichotomy of auditory objects versus scenes?

Prominent low-level acoustic attributes, such as loud sounds or salient three-dimensional spatial cues, can cause a sound-source to suddenly pop out as an “auditory object”. This includes signal processing along reflexive circuits, involving brainstem regions such as the inferior colliculi (Belenkov and Goreva, 1969), though this does not lead to recognition per se. Sound-production necessarily implies some form of motion, which often includes fairly robust first order motion cues such as interaural intensity differences (IID), interaural time differences (ITD), amplitude changes (e.g. looming), and Doppler effects (e.g. changing frequency of sound of a train speeding by on a track). These acoustic cues, and even illusory spatial motion, are reported to activate primary auditory cortices (Griffiths et al., 1994; Mäkelä and McEvoy, 1996; Murray et al., 1998; Baumgart et al., 1999; Lewis et al., 2000; Warren et al., 2002). However, hearing stationary non-living sound sources, such as a watch ticking or listening in a noisy room for your computer muffin fan to determine if it might be overheating, may be largely devoid of distinctive first order motion cues, with only subtle or no visual motion cues.

A number of higher order signal attributes can capture the temporally homogeneous signals that represent scene-like sounds, and be used to help segment auditory objects from otherwise uninteresting acoustic scenes or soundscapes. This includes, for instance, “sound textures” (McDermott and Simoncelli, 2011). In vision, textures represent intermediate-level feature attributes that the system can use to define salient object boundaries or to fill in surfaces of perceived visual objects (Reppas et al., 1997; Kastner et al., 2000). Another form of higher order acoustic signal attribute related to textures includes spectral structure variation (SSV) measures, which quantifies changes in signal entropy over time (Reddy et al., 2009). Auditory object-like sounds tend to show relatively greater measures of SSV and lower mean entropy levels than scene-like sounds (Lewis et al., 2012), and an fMRI study indicated that the activation along the anterior STG regions for object-like sounds (Fig. 3, light blue) shows parametric sensitivity to SSV measures. Thus, the processing of sound textures, entropy, and SSV measures may be reflective of prägnanz computations in cortex used to probabilistically simplify the likely segmentation of sounds and auditory objects from acoustic background scenes (Cusack and Carlyon, 2003; McDermott and Oxenham, 2008).

Many scene-like sounds that have relatively homogeneous acoustic temporal structure over time (e.g. Box 1, rainfall) also have a power spectrum that can at least be grossly characterized by smoother 1/f^α spectral shape: where f = sound wave frequency and α ranges from 1 to 2 (Lewis et al., 2012). In other words, the physics of sound propagation is such that as the distance between an observer and a given sound source increases, the higher frequency sound pressure waves undergo disproportionately greater losses in intensity. Thus, distant sound-sources may effectively be filtered along cortical pathways as object-versus scene-like based on the learned relative shapes of their power spectra. In experienced adult listeners, many sound producing events that are located far away may tend to be less immediately relevant (though with dependence on survival/task demands and settings) and thus more likely relegated as sensory background noise or ambience rather than attention demanding closer range object-like status. Physically experiencing and interacting with up close sound-sources (living and non-living) presumably help develop representations for object-like acoustic signal attributes (through vision and touch)—and involve cortical regions such as the STG (Fig. 3, light blue hues). In this sense, the term “auditory object” seems to have its greatest relevance as a concept when comparing multisensory features to visual objects and haptic objects.

Scene-like action sounds, however, which are not tangible but may develop visual associations (in sighted individuals), appear to develop by establishing representations outside of auditory cortex proper (Fig. 3, dark blue hues) relating to “non-self” and/or episodic representations (Ries et al., 2007; Burianova et al., 2010). Probabilistically learned scene-like acoustic signatures based on relative loudness, 1/f^α spectral shape, acoustic textures, SSV, and presumably other higher order signal attributes, are proposed here to reflect acoustic dimensionality reductions that could define acoustic-semantic universals that help shape the organization of the mammalian brain for natural sound processing. Notwithstanding, the auditory system may be prone to accommodate or calibrate to acoustic signal structures that are statistically more likely to represent a background scene, representing a form of acoustic accommodation. While this may be especially pertinent to representing environmental sounds of nature, this same mechanism may also apply to the processing of crowded social scenes of people talking as background (e.g. at a market place). The learned acoustic features of scene-like sounds may thus probabilistically lead to the differential processing in the brain, contributing to the perception of acoustic scenes that may apply to any of the major categories of natural sound, though may be most pertinent to non-living categories from an ethological perspective (Box 2).

Box 2. Auditory objects versus auditory scenes.

Returning to the example scenario from Box 1, bottom up cues may direct processing of the perceived stimuli. For example, the ‘Ahh’ utterance depicted in Box 1 has high harmonic content that would engage signal processing along lateral aspect of medial temporal cortices, as depicted by the rainbow arrows in Fig. 2. Being a human utterance, the sound would also share other particular low level attributes that may further direct its processing to speech related regions. The acoustic signal changing loudness over time as the hammer clinks and the dog runs towards her is another specific spectro-temporal attribute characteristic of action sounds. Soundscapes characterized by the relatively constant drone (constancy) of the rain and air conditioner, along with relatively flatter 1/f^α spectral power, would more likely be relegated as background status and thus less likely to require attentional processing priority.

Relaxing acoustic scenes

Incidentally, for relaxation purposes many people like to listen to soundscapes of nature (most commonly non-living things), such as the soothing sounds of a babbling brook in the woods, the tranquil crackling from a Yule Log fireplace video, music that emulates the above mentioned acoustic features (such as “alpha wave” music), and soundscapes of crickets or street traffic. Because these sounds represent actions that are generally well out of a person’s ability to directly physically control or influence, they are less readily represented in cortical networks relating to purposeful or intentful behaviors. Thus, they less readily activate motor repertoires affiliated with a listener’s sense of “self” or the embodiment of intentful actions of others, and instead engage other cognitive processes not related to self-representations. Though speculative, the processing of environmental sounds outside of motor-related networks (Fig. 3, dark blue hues) may thus help explain why subdued “sounds of nature” aid in relaxation and stress reduction, as they can help take one’s mind off of their self-ruminating thoughts.

In sum, research approaches using bottom-up acoustic signal processing similar to those described above, have led to tremendous strides in our understanding of how sound processing may be organized in cortex beyond tonotopic organizations. However, to more comprehensively address the neurobiology of perception other major lines of research have been the development of a variety of top-down models of brain organization that may mediate perception and cognition, as addressed next.

3. Top-down perspectives of vision and hearing models

Relying on bottom-up theories alone does not provide an adequate answer as to ‘why’ the brain is organized the way it is. A main category boundary of our model is living (action sounds and/or vocalizations) versus non-living action sounds. When one considers what may be at the root of this difference, network representations conveying meaning and intent form a solid explanation. Animate sources are much more likely to be imbued with intent. Thus, animate sounds, with a generally greater sense of meaning to the subject, are more likely to require complex behavioral responses such as social engagement, fleeing, approach, or engagement as predator or prey. But how do these sounds become cognitively interpreted so that correct behavioral action can be affected?

Some theories of cognition have posited that high level object and action knowledge resides in a central semantic memory system (a central store) separate from the brain’s sensory, or modal, systems for perception (Pylyshyn, 1984; Fodor, 2001). However, grounded (or embodied) cognition theories (Barsalou, 2008) together with neuroimaging studies of semantic systems (Martin, 2007; Binder et al., 2009) espouse another extreme, that knowledge representations emerge from weighted activity within property-based brain regions. The past century of neuropsychological research, including human brain lesion studies, has demonstrated that word form knowledge of different object categories is represented in part along distinct brain regions (Damasio et al., 1996; Martin et al., 1996; Moore and Price, 1999; Grossman et al., 2002), notably for representing living versus non-living things (Warrington and Shallice, 1984; Hillis and Caramazza, 1991; Silveri et al., 1997). This living versus non-living category boundary in neuropsychological models further inspired research regarding other semantic category representations in the brain. In the realm of visual object recognition, specific brain lesions were found to disproportionately impair, or spare, semantic knowledge of various visual object categories, including animals, tools and artifacts, famous people, and fruits & vegetables (Tranel et al., 1997; Caramazza and Mahon, 2003). Questions remained, however, as to how all these visual object and action categories might relate to representations of potentially distinct categories of “auditory objects”.

From the perspective of grounded cognition theories, a couple studies sought to assess whether a cortical organization respecting a living versus non-living boundary would hold true of real-world sounds (Engel et al., 2009; Lewis et al., 2011b). However, because vocalizations are acoustically characterized and distinguished by strong harmonic content, to facilitate data interpretation (addressed in Section 2.1) the sound stimuli used were restricted to those devoid of any vocal content. The category of living things included human action sounds (readily recognized as a human agent instigating the action) versus non-human action sounds, which were not as easily emulated or mimicked by humans (e.g. horse galloping). The non-living category included sounds of the natural environment (e.g. wind blowing through trees) versus mechanical sounds of automated machinery (e.g. watch ticking or laundry machine tumbling). Importantly, the machinery and mechanical action sounds were judged as not being instigated by a human or living agent. In short, brain regions responsive to correctly categorized living (biological) action sounds (Fig. 3, contributions to yellow hues) were strikingly distinct from those for non-living (non-biological) sounds (contributions to blue hues). The perception of biological action sounds was correlated with activation in numerous motor-related and audio-motor association regions, plus portions of the thalamus, basal ganglia and cerebellum. This was consistent with the idea that the listener was “embodying” the sounds (especially human and to a lesser extent animal actions). Presumably, participants were effectively comparing or probabilistically matching the sound stimuli (incoming sound streams) to their own repertoire of sound-producing motor actions to attain a sense of recognition. The findings that human action sound processing recruits premotor and motor-related cortices has also received support from other studies using fMRI (Bidet-Caulet et al., 2005; Gazzola et al., 2006) and EEG (Pizzamiglio et al., 2005; De Lucia et al., 2009).

In contrast to living things, the processing of sounds produced by non-living things, which are not as readily or easily embodied, preferentially recruit brain regions commonly associated with high level visual processing, episodic memory, and other network processes that remain to be fully resolved (Engel et al., 2009). Sounds produced by non-living things preferentially activated occipito-parietal cortical regions typically regarded as visual areas, plus the parahippocampal gyri and posterior cingulate cortices. In a follow-up study the same participants were subsequently highly familiarized with all the sound stimuli and re-tested in the fMRI scanner (Lewis et al., 2011a), which also resulted in preferentially activated regions of cortex (though with some variations after perceptual learning) that further supported the four-fold dissociation of action sound subcategories in the proposed model. Thus, the living versus non-living double-dissociation for cognitive level processing organization (i.e. for word form and visual object processing) also persisted in the realm of hearing perception.

Conspecific versus non-conspecific action sound processing differences thus far appear to be mostly reflected as a matter of degree of activation of regions in a network rather than the recruitment of any unique brain regions outright. For instance, the left and right posterior insulae were more strongly activated by action sounds that were clearly perceived as being caused by an animal versus a human agent under different listening tasks or with different listening experiences, and were presumed to have functions related to audio-tactile or audio-sensorimotor associations (Engel et al., 2009; Lewis et al., 2011a; Webster et al., 2017). The human action sounds, relative to animal actions, more prominently activated a left-lateralized fronto-parietal and pSTS/pMTG network, overlapping mirror neuron systems (MNS), classically defined as involving the inferior parietal lobule, inferior frontal gyrus plus ventral premotor cortex (Rizzolatti and Craighero, 2004; Molenberghs et al., 2012). This supported the notion that action sounds produced by living things, human conspecifics, and to some extent non-human animals, were effectively being ‘embodied’ for purposes of sound categorization. As mentioned in Section 2.2, the pSTS/pMTG complexes have roles in dynamic temporal processing of object actions, whether viewed, heard, or manipulated, and thus have metamodal or supramodal functions. They are also regions reported to have prominent roles in social cognition (Pelphrey et al., 2004; Jellema and Perrett, 2006; Zilbovicius et al., 2006). Thus, reading subtleties of human expressions and body language together with associated sounds produced by biological actions may in part be represented there, conveying information that helps guide social interactions.

3.1. Listener extremes

To further probe the possible function(s) of the different brain regions and boundaries identified for processing action sounds in the proposed model, we next examine evidence from the brain organization in listeners who effectively grew up in different multisensory environments, including left-handers and early blind listeners.

3.1.1. Influence of handedness on action sound processing

Language and music processing in the human cortical hemispheres show strong lateralizations that are proposed to have their origins in gestural networks (Johannesson, 1950; Hewes, 1973; Grigor’eva and Deriagina, 1987; Corballis, 1999; Li et al., 2000; Zatorre et al., 2002; McNamara et al., 2008; Morillon et al., 2010), which in turn are thought to have evolved from adaptations leading to handedness (Lausberg et al., 2003; Meguerditchian et al., 2013). In right handed listeners, hearing and categorizing uni-manual tool-use sounds (a subcategory of human action sounds) led to network activation that overlapped with regions independently activated during pantomime of tool use with the dominant hand (Lewis et al., 2005). This included activation of the left hemisphere somatosensory- and motor-related networks, most notably including the left inferior parietal lobule (IPL; Fig. 3, green solid outline). This was consistent with reported left-lateralized MNS-like networks (Kohler et al., 2002; Rizzolatti and Craighero, 2004; Molenberghs et al., 2012), and thus consistent with brain organizations typical of right-handed individuals.

To examine the effects of handedness on these hearing perception networks, strongly left-handed participants were recruited to perform the exact same listening task involving uni-manual tool-use sounds, and performing and the virtual tool manipulation task with their dominant left hand (Lewis et al., 2006). The left-handers, who grew up associating the sounds of uni-manual tool use predominantly with their dominant left hand, recruited strong activation along the right hemisphere IPL (cf. Fig. 3, dotted versus solid green outlines)—this IPL activation focus thus effectively flipped sides! This difference in activation pattern was concluded to be reflecting learned audio-motor associations, linking hearing perception with hand and arm movements (motor action schemas) associated with the dominant hand. This lent strong support for embodied cognition accounts of knowledge representation relating to audition. Knowledge of tool making and tool use, and thus of tool use sound processing, is arguably more highly developed in humans than any other species (Paillard, 1993). Thus, the proposed model regards tool-use sound processing as an extension of human action sound representations (Fig. 1, plain text), wherein body schemas may become adapted to perceptually assimilate objects as extensions of one’s body (Paillard, 1993; Iriki et al., 1996). This effect of handedness on sound processing pathways illustrates a compelling instance of how top-down or cross-sensory processing influences the organization of cortical networks mediating hearing perception.

Interestingly, in some ancient Indian philosophical systems, such as the Abhidharma, classifications of sound objects originating some 2500 years ago expressed main categories similar to some of those in the proposed model (Fig. 1), including: (1) sounds that originate from conscious elemental causes such as the voice of a sentient being or a finger snap; (2) sounds that originate from unconscious elemental causes such as the sounds of a river and the wind; and (3) sounds that originate from both conscious and unconscious elements such as a drum beat (Mipham, 2000). The latter category is similar to tool use sounds which we classified in our model as an action sound by a ‘living thing’. This is in accord with the clarification that animate sounds can “express meaning” while inanimate sounds do not, and is generally supported by neuroimaging data, which ancient philosophers would not have had access to. The high correspondence of our model with how philosophers thought about sound object categories in earlier millennia supports the universality of the conceptual qualities of the model.

3.1.2. Influence of blindness on action sound processing

As addressed earlier, the pSTS/pMTG complexes overlapped with regions involved in visual biological motion processing (e.g. lip reading, running), clearly encroaching on cortices regarded as visual areas. However, individuals who have never had visual experience (early blind listeners) are clearly fully capable of recognizing and identifying action sounds. Examining brain responses of congenitally blind listeners (Lewis et al., 2011b), human action sounds deemed as recognized, in contrast to backward played version that were not recognized, also recruited the pSTS/pMTG complex in addition to portions of occipital cortices associated with visual processing (Fig. 3, yellow outlines). Thus, bilateral pSTS/pMTG regions appear to be recruited even in the absence of visual motion experience with the action sounds. Incidentally, when testing for brain activation in response to the four action sound subcategories described earlier, using human, animal, mechanical, and environmental sound stimuli (Engel et al., 2009), a four-fold dissociation of cortical networks was similarly observed in some early blind individuals (n = 2; unpublished data). Thus, while visual experience influences cortical network pattern recruitment, categorical boundaries for hearing perception nonetheless persist in the absence of vision. How might one reconcile the audio-visual representations of objects given that categorical sound perception organizations appear to develop in cortex even in the absence of visual experience?

The functional roles of the pSTS/pMTG in hearing perception are perhaps best interpreted in the context of ‘metamodal operators’ (Pascual-Leone and Hamilton, 2001). In this theoretical framework, different brain regions may be genetically “pre-wired”, developing microcircuitry that happens to be efficient for conducting certain types of operations. Thus, for instance, if an individual has visual input, then regions such as the pSTS/pMTG complex will compete to perform relevant processing to establish representations related to the meaningfulness of the sensory event, typically being recruited for biological visual motion processing functions in sighted individuals. However, in the absence of vision, these cortices still compete to perform certain operations germane to biological action processing. Thus, the pSTS/pMTG complexes appear to play a functional role in transforming the spatially and temporally dynamic features of natural action event information into a common neural code (for audition, vision or touch), and may form a reference frame for probabilistically comparing the predicted or expected incoming auditory (and/or visual and haptic) information based on what actions have already occurred in real time (Lewis, 2010). These regions appear to be ideally suited for processing action sound sequences, especially for sounds produced by living (biological) things, less so for non-embodiable non-living things, and lesser still for vocalizations that cannot be directly viewed (at the vocal cords/laryngeal source)–the latter two of which have fewer intermodal invariant attributes in general. Thus, features of biological motion sequence processing appear to generally relate to operations of the pSTS/pMTG regions.

Interestingly, our meta-analysis revealed regions activated by human action sound processing in the early blind listeners that overlapped with regions activated by non-living action sound processing by the sighted group (Fig. 3, yellow outlines overlapping blue cortex). Thus, a number of cortical territories commonly allotted to the visual system may be more accurately regarded as metamodal operators that compete to process signals from whatever sensory input happens to be available, which in sighted individuals would typically be bottom-up visual signal inputs.

Another study with congenitally blind listeners examined brain regions activated upon hearing hand-made human action sounds, which revealed activation of motor-related networks defined as MNS networks (Ricciardi et al., 2009). However, in the congenitally blind study addressed early (Lewis et al., 2011b) only the sighted control group recruited significant activation in MNS-like networks relative to the early blind group. The activation pattern differences between groups appeared to differ in degree of activation, suggesting that blind participants may have effectively opted to use a different cortical processing strategy for “recognizing” the action sounds given the two-alternative forced choice task (recognized or not recognized). In particular, blind listeners recruited network structures implicated in episodic memory, rather than MNS-like networks implicated in embodiment or procedural memory, as their default strategy (Wagner et al., 2001; Yonelinas et al., 2005). Hence, the brains of blind individuals may adapt to the lack of visual motion input by preferentially using different encoding/decoding strategies to more efficiently represent auditory objects (or action events) for purposes of recognition. Thus, a sound-source may be deemed as “recognized” by different individuals by using completely different strategies (an issue of qualia). This further complicates what is meant by auditory object “recognition” in the human mind (Box 3).

Box 3. Top-down influences on the hearing model.

To examine the influence of top down processing, using the picture example in Box 1, one can infer why the sounds from living things may be more relevant, why they have meaning, and how they are embodied to help process these meanings. When the listener embodies the human sounds, she will realize that when she has made such an utterance and loud clap it was usually to grab someone’s attention. Such sounds would typically be accompanied by a physical state of arousal, apprehension or anticipation. Thus, she could match the activation pattern to earlier learned states and applying those to her friend (theory of mind). The growl would likely also warrant preferred processing, being a signal with relatively low harmonic content that is often associated with threat or negative valence. Humans make similar low harmonic utterances, which often indicate anger and aggression, and our listener in Box 1 likely has memories and other experiences with dogs and other animals making growls, perhaps associated with fear or even a painful bite. The acoustic signal changes evolving over time as the dog runs towards her direction is also relevant here (e.g. changes in loudness and other specific spectro-temporal attributes). Motor regions in the brain may be activated that embody running, and the increase in loudness over time may help recall earlier multisensory experiences when animals may have been running towards one. What the above sound-source examples have in common is intent, the kind of intent and meaning that comes from animate objects and makes them an important stimulus to process when planning action. Finally, although the sound of the rain may be relevant for whether she wants to go inside for shelter, it is inanimate and temporally homogeneous in quality, and will not respond to her actions - this requires a different kind of processing. The air conditioner whirr and rain are difficult to embody into a motor schema, and thus associative learning would rely more heavily on other processing mechanisms in the brain. However, rain, if taken as a deliberate object, may conjure bodily sensations associated with memories of rain (e.g. wet, cold). Both the rain and whirr sounds are more likely to be processed as background, until the situation involving the ‘figure’ animate object sounds is properly assessed.

In sum, the data pertaining to the brains of blind listeners to date buttress the idea that acoustic-semantic universals are driving, or are being utilized by, the cortical organization for auditory processing that respects the three major categories of natural sound (Fig. 1). Moreover, this organization is not critically dependent on visual experience. Furthermore, handedness influences which networks are ultimately adapted as extended portions of the “auditory system”. Thus, the underlying cortical network architecture genetically established at birth (prior to audio-visual and audio-motor associative learning) appears to include metamodal operator network architectures that are well suited for processing acoustic-semantic signals of natural sounds, as outlined in the proposed model, which is formalized next.

4. General tenants of the acoustic-semantic model for hearing perception

Now having reviewed much of the literature leading to the development and constraints of the proposed model of hearing perception for natural sounds, we next formalize some of the general tenants (4.1–4.4) that will have broader implications for the study of sensory perception, neurolinguistics, spoken language evolution, auditory cognition, and potentially other fields of neurocognition.

4.1. Parallel hierarchies process increasing degrees of information content

Our model presumes that, through experience, the brain further organizes beyond nascent networks in order to optimize the representation and processing of sensory information (leading to memory formation) that may occur through classic Hebbian-like mechanisms or other network-level mechanisms of sensory encoding (Dosher and Lu, 2009). The neural representations of sound events thus likely propagate simultaneously along many of the cortical pathways, reaching various intermediate processing stages or tiers (Fig. 3, all colored cortices). Network activations may continue until leading to a “stable” activation pattern that matches a ‘memory trace’ (episodic, procedural, semantic), reflecting a local minimum state (Hopfield and Tank, 1985) that mediates or confers a sense of successful recognition.

Conspecific, versus non-conspecific, action sounds and vocalizations in general would arguably be more familiar and behaviorally relevant for each given species. Thus, hearing conspecific sounds in isolation would likely tap into the greater depths of how they had been encoded over life-long experiences by the individual, which in turn often translates to more expansive network activations in neuroimaging paradigms. However, there are a few caveats with this simplistic interpretation. Perceptual learning studies using visual, tactile, or auditory processing have revealed a number of cortical network mechanisms for memory encoding and retrieval. One is that perceptual learning can lead to greater activation as the stimulus takes on greater degrees of behavioral relevance, newly engaging regions previously not recruited or not to as great an extent (Gauthier et al., 1999). A second is that greater familiarity can lead to less, rather than more, network activation over time due to the networks becoming more efficient and/or faster (sharpening and facilitation models) at processing specific stimuli (Wiggs and Martin, 1998). A third is that network patterns may undergo outright changes in which different brain regions become recruited (scaffolding mechanisms) with experience (Petersen et al., 1998; Lewis et al., 2011a). Another consideration is that when a sound is heard and it is unclear exactly what the sound is (but not confabulated as belonging to a wrong category) it can lead to greater degrees of activation in a given network as it effectively continues to “try” to settle on a probabilistic solution (Lewis et al., 2005). Consequently, the neural correlates of sound perception may be obscured in a given study depending on the specific stimuli used, the nature of the task demands (Bracci et al., 2017), and listener biases, as well as the spatial and temporal resolution of the imaging modality (Santoro et al., 2014). Thus, when exploring the processing of different semantic categories, one may be activating or revealing a cross section through several parallel hierarchical stages of sound representation, which can easily lead to complicated interpretations of the specific functional roles of different brain regions.

4.2. Metamodal operators guide sound processing network organizations

Various cortical regions may be genetically established and interconnected so as to excel at performing specific types of operations, regardless of what sensory input they happen to receive (or fail to receive), and have been termed “metamodal operators” (Pascual-Leone and Hamilton, 2001). For the studies of hearing perception in the blind, this concept of metamodal operator mechanisms is particularly elegant for explaining patterns of activation in what are traditionally regarded as “visual cortices” (Section 3; Fig. 3, yellow outlines). Such processing regions may compete to perform operations with certain types of acoustic attributes, or universal acoustic-semantic attributes, which consequently shape the organization(s) of networks that ultimately mediate hearing perception. The natural sound categories in the proposed model may thus be reflective of whole-brain level strategies optimized for encoding meaningfulness to learned sound stimuli.

4.3. Natural sounds are embodied when possible

Outside of innately pre-wired acoustic reflex circuits (e.g. startle reflexes, emotional communication sounds with newborn infants), a primary cortical mechanism behind encoding and recognizing natural sounds appears to be to “embody the sound if possible”. In this regard, when a sound is heard in isolation the brain attempts to match acoustic inputs with learned audio-motor or audio-sensorimotor association representations, which appear to be lateralized to the left hemisphere. This notion is consistent with the idea that a number of cognitive functions become more lateralized to one hemisphere (e.g. spatial skills, handedness) to minimize interhemispheric “wiring” and thereby being more efficient (Preuss, 2011). Evidence of embodied representations for natural sound processing was perhaps most striking when comparing left-handed versus right-handed listeners upon hearing uni-manual tool-use sound (Section 3), in that certain functional loci flipped sides (especially in inferior parietal cortex), consistent with dominant hand audio-motor association embodiment.

Even music can be embodied in perceptual systems. For example, one study found that individuals listening to piano pieces after (versus before) training in how to play the piece with their own hands newly led to activation in motor-related networks (Lahav et al., 2007). Thus, while the complexities of music appreciation may be represented among widespread brain regions (Koelsch et al., 2004; Zatorre et al., 2007), experience with musical sound production can influence which brain networks are recruited when hearing a musical piece—“feeling” the music emotionally, and/or embodying it technically in motor systems (also see Section 6).

4.4. Categorical perception emerges in neurotypical listeners

The model presumes that neuronal organizations or representations of certain fundamental acoustic-semantic categories develop both in neurotypical human listeners and presumably other mammalian species with hearing ability (and perhaps necessarily with auditory communication ability as well). We further presume that other sub-categories may develop with listening expertise (e.g. birders, hunters, musicians), which may develop with dependence on visual, sensory-motor, and other multisensory inputs or contexts. Conversely, categorical perception may fail to fully develop, such as for some individuals on the autism spectrum who have difficulty with generalizing objects into categories (e.g. difficulty conceiving all dog barks as belonging to one specific subcategory; “dog barks”) (Grandin, 2008), which presumably renders some of the model boundaries as less distinct for those listeners. Thus, targeted interventions may focus on training young individuals to hear different natural sounds as belonging to distinct semantic categories in an effort to tap in to nascent cortical circuitries that may ultimately develop to become more efficient for representing acoustic-semantic knowledge or meaning at a categorical level, as addressed with other issues of potential clinical significance in the following sections.

5. Model implications and predictions

As mentioned in the introduction, the proposed model of hearing perception should have impact on multiple lines of thinking across fields of sensory, multisensory, and cognitive neurosciences in both human and non-human animals, as addressed below.

5.1. Fundamental advances in understanding how perception functions

As our model focuses on a lesser studied modality, audition, and incorporates both bottom-up and top-down influences on object perception, we hope to spur thinking about how different modalities can inform perception, and how their unique properties can shape how that modality is used by the individual and why. Because sound, in contrast to vision, only comes from motion (changes in energy to produce sound pressure waves), the auditory system is likely to be more heavily dominated by representations for agent intention and meaning, potentially as a vestige of sound processing crucial for survival. Stationary visual objects may become of interest to us from a top-down perspective, so although meaning is still key for object perception in any modality, the visual world can accommodate many more immediately meaningful perceptions of segmented objects compared to audition, where auditory object perception may require several seconds of sound events to unfold to accurately convey a sense of recognition. Thus, this may make audition an ideal model system for advancing fundamental constraints of multimodal models of object perception.

We further assert that our model may contribute to the study of object representation in general, as well as how factors such as intention, learning, and action influence perception. Section 4 elucidated multiple tenants set forth by the model that are likely applicable across other sensory modalities. As the auditory faculty has attributes that are either absent or novel compared to vision, one can utilize audition to further tease apart what qualities of object perception are truly modality invariant versus dependent as it relates to object perception in brain network representations.

5.2. Understanding spoken language development in children

Vocal imitation and mimicry, together with sound symbolism, are known to play a crucial role in a child’s spoken language neurodevelopment (Kuhl and Meltzoff, 1982; Rhoades, 2007; Imai et al., 2008, 2015; Ozturk et al., 2013). A number of theories suggest that aspects of vocal communication should show a resemblance to properties of sensory referents, as formalized in theories of sound symbolism and iconicity (Imai and Kita, 2014; Perniss and Vigliocco, 2014).

Humans infants are known to be sensitive to voices and speech sounds shortly after birth. Two-day-olds prefer their native language (Moon et al., 1993; Beauchemin et al., 2011; Sato et al., 2012), and are thought to be learning the significance of maternal voices as early as during fetal development, showing responsiveness (e.g. heart rate changes) to speech produced in the child’s mother tongue (DeCasper et al., 1994), which may be influencing the development of hearing perception proto-networks in utero (DeCasper and Fifer, 1980; DeCasper et al., 1994; Kisilevsky et al., 2003, 2009). Thus, one future direction would be to determine if acoustic-semantic universals of vocalizations are the first to start driving auditory system development (perhaps in utero), followed by later stages that are driven by processing of other acoustic-semantic universals that may emerge later in post-natal life. In this regard, some aspects of the proto-language networks that will develop to mediate spoken language processing and perception may depend on the development of rudimentary semantic networks (for categorical perception) before symbolic linguistic representations can be properly formed and organized. This neurodevelopmental mechanism might prove to help explain some etiologies of spoken language delays in children (Sheridan, 1959; Stothard et al., 1998).

5.3. Understanding central hearing deficits and recovery after brain injury

Outside of peripheral hearing loss, we know from examples of central auditory disorders that damage to specific brain regions can lead to a variety of different hearing deficits, including agnosias that may be specific for environmental sounds, timbre, rhythm, words, melody in music, and in rare cases for more specialized sound object categories (Goll et al., 2011; Trumpp et al., 2013). This indicates that there are separable functions and processes mediating hearing perception in the human brain, similar to the visual system, which has led to the idea that brain networks mediating categorical perception are also a likely hallmark feature of the human auditory system (e.g. Section 4.4). However, pure associative agnosias for specific categories of sound other than voice and melodies are rare (Saygin et al., 2003). This may be due to the nature of how natural sounds, which may require relatively longer stimulus durations to unambiguously identify, and can be confabulated and completely mis-categorized to the satisfaction of the listener. For instance, animal vocalizations that were misperceived by individual listeners as representing tool sounds (in a two alternative forced choice task) were correlated with cortical activation of “tool sound processing” networks (Lewis et al., 2005). Thus, sound confabulation represents a feature of the auditory system that may make auditory object processing deficits trickier to assess neuropsychologically relative to visual object processing deficits. Further advances in defining the processing mechanisms associated with the different category boundaries for hearing perception relative to other modalities will likely help develop clearer taxonomies for describing, diagnosing and developing interventions for individuals recovering from auditory cognitive deficits, and guide targeted interventions at perhaps more rudimentary semantic levels as a key to neurorehabilitation.

5.4. Advances in biomimetic hearing aid designs

In an effort to help people who suffer from hearing loss, models of how the brain processes sound have led to ideas for engineering biologically-inspired (“biomimetic”) hearing aid algorithm designs (Wang and Shamma, 1994; Smith and Fraser, 2004; Coath et al., 2008; Shannon, 2012). A need still persists for the continued development of both smaller devices and more “intelligent” designs (NIDCD, 2009) that are effective for different listening environments. Selectively amplifying frequency bandwidths characteristic of human speech represents one strategy (Harkins and Tucker, 2007; Takahashi et al., 2007), though this does not always allow a listener to segregate, for instance, the sounds of one person speaking in the presence of a crowd of people, or to tolerate the noises heard while chewing food. Further efforts have and will continue to require not only considerable improvements to the hardware implementing them, such as small size, ultralow power consumption, field programmability (Rumberg and Graham, 2015), but also to the simultaneous development of more sophisticated algorithms that appropriately suppress background acoustic noise based on probabilistic sound signal profiles to better enhance signals of interest (Takahashi et al., 2007; Chung and McKibben, 2011; Lowery and Plyler, 2013)—potentially capitalizing on some of the putative acoustic-semantic universal attributes proposed herein. For instance, biomimetic designs may be able to capitalize on capturing temporally correlated signals of harmonic profiles and power spectra profiles to effectively enhance sounds that are characteristic of a single natural sound-source category, and filter out acoustic signatures characteristic of background acoustic noise, thereby performing acoustic accommodation filtering on the front end.

5.5. Advances in anthropological models of oral communication in hominins

The proposed model has implications regarding literature debating the acoustic versus gestural origins of spoken language systems (Darwin, 1871/1981; Hewes, 1973; Liberman and Mattingly, 1985). Mimesis, the ability to produce self-initiated representational acts (also see glossary), is thought to represent one of the earliest forms of cognitive-motor abilities that distinguished hominins (e.g. homo erectus and possibly homo habilis) from the great apes (Hewes, 1973; Grigor’eva and Deriagina, 1987; Donald, 1991), and the above theory purports gesture movements as an early form of communication that predated vocal language.

Mimicking the events and sounds of the natural world, and conveying propositional communications, may have constituted a form of pre-linguistic communication. This includes big game pre-hunt organizations, teaching complex skills to other troop members, interpretive dance, and numerous other socializing events that would help stabilize larger groups of individuals in a community. Some oral communication advancements presumably enabled hominins to detect and interpret increasing degrees in nuances of emotional states and intentions of conspecifics through an ability to produce and perceive non-stereotyped oral communications (e.g. acoustically conveying jealousy, love, triumph) (Donald, 1991), which in extant humans are largely processed in cortical networks lateralized to the right hemisphere. Over roughly the last 100,000 years, the auditory and semantic systems of humans are thought to have evolved further to accommodate high speed articulated speech perception, likely through exapting circuits used for gestural communication planning and generative praxis (MacNeilage, 1998; Corballis, 1999; Rizzolatti and Craighero, 2004; Arbib, 2005; Corina and Knapp, 2008; Stout et al., 2008), in that in most individuals language is left lateralized. Eventually, vocal sounds largely supplanted manual gestures as the main form of communication, permitting substantially faster communication of ideas, communication in total darkness, and communication over greater distances through visual barriers like dense forests. Recent theories posit that some of the earliest categories of natural sounds that needed to be orally mimicked would likely have included incidental sounds of locomotion, tool-use sounds, and vocal calls of other animals (Falk, 2004b; Larsson, 2014, 2015), which are consistent with the boundaries of the proposed model.

The brain regions related to language processing may be rooted in evolutionarily earlier systems when gestural mimesis prevailed. This ‘default’ gestural origin theory is supported by the proposed model and meta-analysis data, which shows that living action sounds, and not conspecific vocalizations per se, predominantly activate fronto-parietal (motor-related) regions in the left hemisphere (Fig. 3, yellow): This includes regions commonly associated with language reception (e.g. Wernicke’s area) and production (e.g. Broca’s area). Thus, the evolution of hearing perception systems in modern humans, based on vestiges of how the brain appears to be organized for natural sound processing (Fig. 3), appears to have been closely tied to the ability to produce and interpret communicative action sounds produced by living things as a semantic category.

With regard to human and primate communication, the proposed model thus leads to a number of predictions or questions. One is that conspecific action sounds for other species, including sound producing body-action asymmetries of great ape species (Cashmore et al., 2008), might also show evidence of lateralized networks for processing and conveying a sense of meaning (and perhaps related to the degree of primate handedness). Another prediction is that the organizational principles for sound processing at a categorical level should also be respected in organizations for sound production or mimesis at a categorical level. For instance, oral production and orchestration of non-linguistic natural sound events (imitation, mimicry and/or mimesis) might entail motor planning networks that also respect the category boundaries of the proposed model.

Regarding language, a third prediction is that cognitive architectures for phrase-level language comprehension should at some level also respect the major category boundaries. Models of language and cognition suggest that parallel hierarchies entail perceptual-semantic links ranging from lower sensory signal features, to auditory/visual object representations, to situations/events, and to abstract ideas (Perlovsky, 2011). Onomatopoeia represents one level of linkage (Hashimoto et al., 2006) though this does not adequately explain the more highly symbolic levels of language (Imai and Kita, 2014). However, natural sounds at a category level more generally may prove to correlate with the linguistic concept representations associated with short spoken phrases or utterances. For instance, grounded cognition models (Barsalou, 2008) would predict that the comprehension of short spoken phrases describing sound-producing events (e.g. “wind storm”) should engage at least some of the same brain regions that demonstrated category-specificity to perceptual-level processing of those corresponding sound events (i.e. perceiving the sounds of a wind storm), independent of the language(s) used. If verified, this would lead to new lines of research for characterizing the nature of perceptual-linguistic links in the brain; with the idea that linguistic-semantic systems may largely be grounded in perceptual-semantic systems. Acoustic-semantic universals may thus represent one form of natural sensory signal attribute to help bootstrap cortical networks for cognition. These avenues of research may thus shed light on the theories behind both the phylogeny and ontology of spoken language systems from more of a “bottom-up” perspective.

6. Limitations of the model and issues for future research

While our neurobiological model accounts for much of the neuroimaging data from humans and other mammals to date, there are a number of limitations of the model and need for future research, as addressed below.

6.1. Category refinement and task dependency

The chosen categories and subcategories of sound-producing events illustrated in the model accounted for most if not all of the observed brain processing organizations for non-linguistic hearing perception. However, when considering a global model that extends to other sensory perception domains, these may not necessarily be the most informative category division or subdivision definitions. In order to make direct comparisons across modalities, one must ask if these categories fully correlate with those proposed in the visual and haptic modalities, and in what semantic contexts. Studies that compare objects of our included categories that can be recognized and separately presented as auditory, visual and ideally haptic stimuli may help address the cross-modality applicability of this model. Because audition is different, certain unique attributes may not extend to other modalities and it is worth exploring these similarities and differences. The weighted degree of activation of a specific brain region in a given network may vary based on experience, as mentioned earlier, and studies examining expertise versus novice observers, as well as behaviorally manipulating context and task demands may help refine this model.

6.2. Processing of music, emotional and threatening sounds

The top-down influences of the affective and reward systems have only been superficially addressed in this model. Sounds from any of the categories can take on a highly positive or negative valence, and any of the categories may be used to elicit musical forms (rhythm and melodies), wherein music appreciation appears to be relatively unique to humans. Conceivably, the missing ‘fourth’ category of sound of the model, non-living vocalizations, may be reflective of music as a sound category at a conceptual level that can be found or learned to be appealing by different listeners. In the context of this review, both speech processing and music appreciation (enjoyable or disagreeable) are regarded as tapping into higher forms of acoustic communication that utilize brain systems that extend beyond the more rudimentary levels of natural sound recognition in the proposed model. This corresponds reasonably well with brain imaging studies of music in general (Zatorre et al., 2002; Salmi et al., 2016). Understanding how lower and higher level acoustic regions interact in the context of music has implications for evolutionary theories, as complex forms of music production and appreciation have also appeared to evolve in hominins dating back 2–3 million years ago (Donald, 1991).

Of course, music is an important part of human culture and experience, and even has therapeutic implications (Raglio et al., 2016). Testing the boundaries of the proposed model with different instruments from voice and voice-like (“non-living”) wind instruments, versus background percussion or artificially created sounds with manipulated attributes, could have interesting implications for testing category fluidity and figure/ground distinctions and music therapies. In addition, auditory processing of music may interact with speech vocal networks at both low and high levels. It may also have a salient link with affective processing, such as with song being used as a tool for memorization. As to how lower level auditory cortical processing serves as a bridge to these higher functions will be an important area of future research.

Further research is also needed to understand how the limbic system (for affect in general) and auditory perception systems interrelate beyond reflexive circuits, which should have important clinical implications. For example, in post-traumatic stress disorder (PTSD) specific acoustic-limbic circuits may become overly interactive (Schechter et al., 2012; Suo et al., 2015). Conversely, in conditions such as autism spectrum disorder (ASD) there may be a relative lack of acoustic-limbic interconnections that may lead to inappropriate (or a lack of) responses to threat or social/emotional sounds (Tecchio et al., 2003; Baranek et al., 2007; Linke et al., 2017; Lortie et al., 2017). The proposed model may provide a framework for understanding how learned acoustic signals affect, and are affected by, interactions with the limbic system, thereby leading to evidence-based research on targeted intervention for clinicians.

6.3. Cross-species comparisons of the model

A comparative cross-species study of brain organization for processing conspecific versus non-conspecific action sounds or vocalizations, perhaps testing both relative to non-living environmental sounds, would provide a robust method for further testing of whether this theoretical framework truly reflects a fundamental model underlying perception. For example, studying vocalization versus action sound processing in species who rely more on one or the other sound category as a form of communication or environmental interaction, or are more visual versus auditory dominant, would help refine the model.

6.4. Category and object-scene fluidity

The category of sound and how it is processed in the brain for a given auditory event may have some “fluidity”. As an example, the sound of an unseen cricket (a living, non-conspecific animal) in a closed environment, such as a tent or yurt, could be readily identified as an object. However, even though it is an action sound (with strong harmonic content), the general background of the cricket(s) or other nature sounds for most people might become a background acoustic scene, as addressed in Section 2.2. Even human conspecific speech sounds can become part of the cacophony background noise of a restaurant, evidenced perhaps by the fact background restaurant babble is a sound type that is rated as emotionally neutral in the International Affective Directory of Sounds (IADS) (Fernandez-Abascal et al., 2008). However, if in that restaurant one is attempting to covertly overhear a conversation, one’s relationship to the processing stream would change and one may carefully attempt to parse the frequency, intensity and duration characteristics of the individual voice they are attempting to decipher.

The potential for fluidity is likely similar with all categories in our model, and it is currently unclear whether manipulating low level signal attributes within a category (such as harmonicity in vocalizations, constancy, or 1/f^α in environmental sounds) can lead to category manipulation. Exploring cortical activation to the same set of natural sound stimuli from different categories when they are attentively or contextually cued as being either more ‘object-like’ versus more ‘ground/scene-like’ might also be an important test of the fluidity of the bottom-up versus top-down drivers of auditory cortical processing across categories.

6.5. Limitations of neuroimaging technology

Thus far, neuroimaging of auditory systems has been limited by the constraints of the methodologies available. Whole brain imaging currently can only be achieved when the participant is extremely still. Also, the environment of the fMRI is very noisy, requiring creative methodologies, such as event-related sparse sampling in which short clips of auditory stimuli are presented in quiet periods between brain slice acquisitions. Any action of the participant is limited to a short button press or utterance. Studying participants making natural vocalizations or action sounds, or naturally interacting with sound stimuli in conventional neuroimaging devices is thus rather difficult given the artificial constraints of such environments. Novel neuroimaging technologies currently being developed, including a wearable, upright positron emission tomography (PET) helmet with greater head motion tolerance that our group is developing (Bauer et al., 2016), will allow one to probe deeper into predictions generated by the model, accommodating whole brain imaging during natural movement such as speech, gestures, and many types of tool use. Virtual reality (VR) further lends promise for more immersive studies that can be well controlled, and one could use VR to study how the model functions in relation to action, motivation, emotion and attention in more natural settings. Wearable brain imaging technologies will allow for imaging with robust natural movements, and compatibility with headphones, VR, as well as EEG, and with neurotransmitter targeted ligands to examine different systems (Fig. 4). Advances in wearable brain imaging technology that allows greater spatial resolution and depth penetration (Boto et al., 2017), may mean exciting developments for future investigation of auditory processing in natural contexts.

Fig. 4. Wearable PET technology that could be used for interactive auditory perception studies.

The top panel shows the potential designs from seated to standing and walking, that could be utilized for behavioral studies of perception and action, and would allow for gestures, vocalizations and avoid/approach behaviors. Lower left shows our proof-of-concept device with virtual reality goggles which could be a mechanism for studying auditory and multimodal perception in an interactive environment (30 s - several minute temporal resolution) that could be combined with EEG to obtain higher temporal resolution (cite abstracts?). Lower right shows human patient data (one participant actively turning head from side to side) from our limited brain coverage prototype, but still demonstrating medial brain structures such as basal ganglia and thalamus.

7. Concluding remarks

We have proposed a simple, neurobiological acoustic-semantic model of hearing perception in which bottom-up and top-down influences interact, resulting in perception of auditory objects. The model takes into account bottom-up acoustic properties that are either instinctual or learned as belonging to either of three main types of semantically meaningful categories of natural sound: living action sounds, non-living action sounds, or vocalizations (living sources). Such principles may underlie neuronal processing mechanisms to efficiently direct sound signal processing, based probabilistically on a number of acoustic-semantic universals, to cortices or networks best adapted for conveying a sense of meaningfulness to the listener. Our model further accounts for top-down influences that reflect grounded (“embodied”) cognition principles for both vocalization and action sound processing. We additionally explored multiple fields of research that could benefit from the model’s framework. Auditory processing should ultimately be studied in the context of how a listener interacts with the sound-sources, and thus advances in interactive auditory stimulus delivery and neuroimaging technologies, such as wearable PET neuroimaging systems, will likely be crucial. The proposed model could serve as a test bed for predictions made by cross-species comparisons, for developing or refining taxonomies for cognitive deficits, for advancing models of spoken language evolution, and for refining models of auditory processing neurodevelopment trajectories in children.

Glossary of terms

Acoustic-semantic universals

A quantifiable acoustic parameter or set of parameters inherent to sounds of the natural world that probabilistically assigns a sound-source to membership of a distinct semantic category. In principle, the auditory system of all mammals has evolved intrinsic cortical micro-circuitry that efficiently develops to extract or segment sound-sources based on a number of universal signal attributes

Agnosia

The loss of ability to recognize the import of sensory stimuli or impressions. Different varieties of agnosia (e.g. pure auditory agnosia) are distinguished by different sensory modalities of functions

Auditory object

A collection of acoustic data bound in a common perceptual representation and disambiguated from other events in an auditory scene. This term has variations in definition in different fields of study that are addressed in this review, and thus is an operationally defined term

Episodic memory

The system that allows one to remember (consciously recollect) past experience of autobiographical events, reflecting concrete or time-bound memory

Event perception

The ability to perceive complex, usually moving, clusters and patterns of stimuli as a unit. This contrasts with characteristics of object perception in that it further takes into account motion and context

Hominin

Humans (Homo sapiens sapiens) and their closest non-extant relatives (e.g. homo habilis, homo erectus). The term hominid includes the great apes

Mimesis

The ability to produce conscious, self-initiated, representational acts that are intentional but not linguistic (e.g. charades, pantomime, ritual dance). Mimicry is different from mimesis in that it is more literal as an attempt to render an exact duplicate of an observed act. Imitation, found especially in monkeys and apes, is also different, wherein mimesis adds the element of invention of intentional representations

Object

An object is loosely defined as “a thing, person, or matter to which thought or action is directed” (Random House dictionary). All objects (visual, auditory or haptic), however are seemingly defined by ‘meaning’ and therefore, what constitutes an object implies some degree of fluidity as meaning can be different to different organisms and may even change over the lifespan of a given organism. The status of an object (auditory or otherwise) may depend on the perceivers a) experience, b) faculties and capabilities, and c) circumstances