The singular nature of auditory and visual scene analysis in autism

Abstract

Individuals with autism spectrum disorder often have difficulty acquiring relevant auditory and visual information in daily environments, despite not being diagnosed as hearing impaired or having low vision. Resent psychophysical and neurophysiological studies have shown that autistic individuals have highly specific individual differences at various levels of information processing, including feature extraction, automatic grouping and top-down modulation in auditory and visual scene analysis. Comparison of the characteristics of scene analysis between auditory and visual modalities reveals some essential commonalities, which could provide clues about the underlying neural mechanisms. Further progress in this line of research may suggest effective methods for diagnosing and supporting autistic individuals.

This article is part of the themed issue ‘Auditory and visual scene analysis'.

1. Introduction

According to the Centers for Disease Control and Prevention of the USA, about 1 in 68 American children has been identified with autism spectrum disorder (ASD) [1]. ASD is a complex neurodevelopmental disorder, which is diagnosed based on key symptoms including impaired social communication and restricted interests and repetitive behaviours [2]. ASD is often accompanied by sensory atypicalities [3]. The new diagnostic criteria (DSM-5) include ‘Hyper- or hypo-reactivity to sensory input or unusual interests in sensory aspects of the environment (e.g. apparent indifference to pain/temperature, adverse response to specific sounds or textures, excessive smelling or touching of objects, visual fascination with lights or movement)’ [2]. In addition, many autistic individuals show atypical characteristics in auditory and/or visual scene analysis. Scene analysis involves neural computation that makes sense of sensory inputs in terms of events and objects and their spatio-temporal configuration in the external world, taking account of their relevance and significance. Essential subjective attributes that form our perceptual experience, such as the pitch, loudness, timbre and perceived location of sounds, or the perceived shape, distance, colour and motion of objects, are derived from auditory and visual scene analyses. Auditory and visual scene analysis is important for appropriate actions and flexible communication in daily life.

Autistic individuals may have very specific sensory atypicalities that vary across individuals and that may be associated with either better or poorer performance than for neurotypical (NT) individuals. Descriptions illustrating the singular nature of auditory and visual scene analysis can be found in autobiographies written by autistic people. For example,

In a busy airport […] I can't screen out the airport background noise and listen to the phone at the same time. My hearing acuity is normal, but I prefer to use the special amplified phones for the deaf at the airport. [4]

This is a common example of complaints by autistic individuals, namely, difficulty in selective listening in the presence of competing sounds, despite normal pure tone thresholds, as measured by the audiogram. On the other hand, some autistic individuals show extraordinary performance in scene analysis. For instance, an autistic author described a childhood experience as follows:

My father had a puzzle with several thousand pieces, and he used to ask me to help. My sharp eyes meant that I would find the right bit of puzzle among the thousands of others. Father used to show me the place where the bit should go, and I would find it. Mostly, I was allowed to help for only a little while because I spoilt the pleasure for him when it was so easy for me. [5]

Such specific abilities in scene analysis for autistic individuals, no matter whether or not they are beneficial in daily situations, may provide valuable clues about the neural mechanisms underlying ASD. They may also shed light on the mechanisms of auditory and visual scene analysis in NT individuals.

In this paper, we review recent studies, including ours, that examine the singular nature of auditory and visual scene analysis in autistic individuals. Of interest here are which sub-processes, or which neural sites, are responsible for the specific profile of scene analysis in autistic individuals, and whether the specific pattern of atypicalities is common between the two sensory modalities. Finally, we suggest directions for future research.

2. Background: scene analysis in the auditory and visual modalities

From the viewpoint of information processing, scene analysis is not an easy task. First, sensory data are inherently ambiguous. In vision, to recover the three-dimensional structure of external objects from two-dimensional images projected onto the two retinae, the inverse problem of determining the optical process that has produced the images must be solved. This inverse problem generally has no unique solution [6]. There is a similar difficulty in audition, where each ear receives a mixture of sound waves from all active sound sources. To decompose the mixed signal into independent signals, each corresponding to a sound source, is also an inverse problem that generally has no unique solution [7]. It is debatable whether our perceptual systems try to solve such inverse problems in a strict sense, but in any case, achieving a consistent and reasonable interpretation of the external scene surely requires complex information processing. Second, the process of scene analysis has to be robust against various types of variability and noise, such as differences in illumination and viewpoint in vision, and masking by background sounds and reverberation in audition. Third, the process of scene analysis needs to handle the vast amount of sensory data efficiently to generate timely actions. It is also required to select relevant targets in the current scene and to adjust the level of analysis according to relevance. A central challenge in cognitive neuroscience has been to uncover the computational and neural mechanisms that achieve scene analysis in the face of the above-mentioned difficulties.

Diverse lines of research on the auditory and visual systems suggest that scene analysis is achieved through a series of sub-processes, which are organized hierarchically. In audition, the hair cells in the cochlea convert sound into neural signals. In vision, the photoreceptor cells in the retina convert light into neural signals. The converted neural signals are sent to sensory areas of the brain via the medial (audition) and lateral (vision) nuclei in the thalamus. Despite the fact that sound and light have qualitatively different physical properties, there are fundamental similarities in the processes of scene analysis for the two modalities. In the early stages of the auditory and visual systems, basic features such as sound frequency and the orientation of edges are extracted by neurons with frequency or orientation preferences (also called selectivity). As processing proceeds, those basic features are integrated into global patterns, partly based on automatic processes, i.e. not requiring attention or conscious awareness. Eventually, by means of selective attention and prior knowledge, an observer constructs the higher order properties of objects or events in the environment that are relevant to the current goal or task.

(a). Auditory scene analysis

Natural sounds, including speech, usually exhibit characteristic spectro-temporal patterns extending over a wide frequency range. In the early stages of auditory processing, the basilar membrane in the cochlea acts as a bank of bandpass filters and the hair cells along the basilar membrane convert the outputs of the filters (i.e. the local vibration patterns of the basilar membrane) into neural signals [8]. Neurons are ordered topographically according to their preferred frequency (also called the characteristic frequency or best frequency). Hence, there is a frequency-to-place mapping (called tonotopic mapping) throughout the ascending auditory system, from the auditory nerve to the auditory cortex, via the brainstem nuclei and the medial geniculate body of the thalamus [8].

Following the initial bandpass filtering in the auditory periphery, various types of acoustic features are extracted in each frequency channel. Such features include sound level, periodicity, amplitude and frequency modulation (AM and FM), and interaural time and level differences (ITD and ILD). It has been shown that some of these features are extracted in parallel by specialized neural mechanisms in the brainstem. For example, in many mammals, ITD and ILD, two major cues for sound localization in the horizontal plane, are extracted in the medial superior olive and the lateral superior olive, respectively [9].

To derive perceptual attributes such as pitch, loudness, timbre and subjective location, it is necessary to integrate relevant features across frequency channels, and also across different types of features for a perceptual attribute. In the case of sound localization, the ITD in a given frequency channel provides ambiguous information. For example, for the channel ‘tuned’ to 1 kHz, an ITD of 0.5 ms (one half of the period at 1 kHz) could be associated with a sound either to the left or the right. For broadband sounds, the ambiguity can be resolved by comparing the ITD across frequency channels; the ITD that is in common across channels represents the ‘true’ ITD. The inferior colliculus in the brainstem plays an important role in such across-channel integration of ITD information to resolve phase ambiguity and in across-feature integration of ITD and ILD cues to determine sound localization [9].

It should be noted that when there are multiple sound sources, as is often the case in everyday situations, features likely to be related to a single source need to be integrated selectively. Such integration occurs pre-attentively and automatically, based on several acoustic cues [7]. Cues for integrating concurrent features include harmonic structure and temporal coherence; frequency components that form a harmonic series or that start and stop together tend to be grouped together [10,11]. Cues for integrating successive features include proximity in a feature domain [12]. As a consequence of feature integration, the internal representation of each sound source is generated. Such a representation is called an ‘auditory object’ or ‘auditory stream’ (the latter is used to put emphasis on the inherently temporal nature of the representations). Recent neuroimaging and physiological studies have shown that auditory stream formation involves widely distributed neural sites in, below and above the auditory cortex [13], including the brainstem [14,15], the thalamus [16] and the intraparietal sulcus [17].

So far, we have considered auditory information processing as a bottom-up process, but the auditory system has bidirectional connections at almost every stage from the cochlea to the cortex, which provide the basis for top-down modulation [8]. Such top-down modulation may contribute to the appropriateness, robustness and efficiency of scene analysis. Auditory attention allows a listener to direct the focus of processing toward a sound of interest and filter out irrelevant sounds. It is known that attention can be guided both by bottom-up novelty detection [18] and by top-down modulation based on the use of extracted features and depending on task demands [19–21]. Auditory attention modulates neural responses in widely distributed areas from the association cortex to the cochlea [22–25]. In addition to modulating attention, prior knowledge or expectations influence scene analysis per se. For example, prior knowledge of speech content enhances perceptual clarity [26]. A recent study using electroencephalography and magnetoencephalography (MEG) demonstrated that the enhanced perception of degraded speech produced by prior knowledge is correlated with top-down feedback from the inferior frontal gyrus (for more abstract linguistic processing) to the superior temporal gyrus (for lower level acoustic processing) [27]. Interpreting inherently ambiguous sensory data appropriately requires prior knowledge or internal models, as many researchers have pointed out [6,7,28,29]. This is consistent with the framework of predictive coding, which has been widely used in cognitive neuroscience [30].

(b). Visual scene analysis

In the visual system, basic features of multiple stimuli, such as colour, edge orientation and motion direction, are analysed simultaneously across the visual field [31,32]. Physiological studies show retinotopic mapping of visual space onto several divisions of the visual cortex, allowing the parallel processing of visual features of multiple objects [33,34]. The basic features, which are extracted across the visual field, are integrated via automatic grouping processes, based on properties such as continuity of contours, leading to visual representations of object forms [35].

In contrast to the parallel processing of multiple objects, serial inspection accompanied by selective attention plays a pivotal role in the natural environment, where objects have various combinations of basic visual features and compete for attention. Visual attention makes use of both spatial and visual features in order to prioritize relevant information and to ignore objects or events unrelated to current goals [36]. Spatial attention in the visual system is thought to play a role in the binding of visual features into a single object [31]. Based on both behavioural and physiological evidence, the allocation of visual attention appears to be implemented via recurrent top-down control signals from higher cortical areas, including the posterior parietal cortex and the frontal eye field to the visual cortex [37]. Spatial attention can be focused on an accurate position in the visual field, thanks to superior spatial resolution in the visual cortex. In addition, results using a visual masking method called object substitution have shown that feature binding is dependent on recurrent processing [38]. In object substitution, a brief display containing a target and mask is replaced by the mask alone, which prevents repeated visual search for the target. This method impairs performance in a feature binding task [38]. Recurrent processing also plays a significant role in the interpretation of visual scenes. To represent the external world accurately, the brain needs to use prior knowledge, or internal models about the environment, just as for the auditory system. For example, the inference of three-dimensional shape from shading patterns, in which most depth cues are ambiguous, depends on the assumption that the illuminant is over the observer's head, as is usually the case [39].

(c). Similarities and dissimilarities between auditory and visual modalities

In addition to similarities in the flow of information processing, some neural mechanisms are shared by the auditory and visual systems. For example, recent neuroimaging studies demonstrated the presence of a common frontoparietal neural network for visual and auditory spatial attention [19–21]. However, the two systems differ substantially in some respects. As mentioned earlier, owing to the retinotopic mapping of visual space onto the visual cortices, the spatial relationships of objects are explicitly represented in the visual system. By contrast, the auditory system has to infer the locations of sound sources and their spatial relationships based on non-spatial features such as ITD and ILD. Moreover, most visual objects are spatially localized, so that an object excites a contiguous region on the retina, and an object in front occludes an object behind it. On the other hand, sounds from different sources are mixed when they arrive at the eardrum. Once decomposed into frequency components in the cochlea, it is often the case that components from a single source are distributed over a wide frequency range and those from different sources overlap that range. The auditory system decomposes acoustic signals into separate time-varying frequency components via the auditory filters. Higher in the auditory system, the temporal fluctuations in each frequency channel may be processed using an array of modulation filters, and spectro-temporal patterns may be analysed via neurons with specific spectro-temporal receptive fields. The visual system analyses visual images using spatial-frequency channels. Low spatial frequencies convey the global pattern of the visual scene whereas high spatial frequencies carry the fine structure of the scene [40].

It should be also noted that functionally similar sub-processes in the two modalities are probably implemented at anatomically different levels. For example, the contribution of subcortical sites to scene analysis differs between the auditory and visual systems. Perceptual organization in vision is thought to emerge only at the level of the visual cortices [35]. On the other hand, physiological and brain imaging studies have shown that the brainstem plays a role in auditory stream formation [14,15].

We consider next which aspects of the information processing are different between ASD and NT individuals, and how they differ. In other words, we review the singular nature of auditory and visual scene analysis in individuals with ASD.

3. Auditory scene analysis in autism

Several studies have investigated whether there are specific atypicalities in basic aspects of perception for autistic individuals, but the results have been rather inconsistent, due, at least partly, to the diversity of stimuli, tasks, type of participants (e.g. children versus adults, with versus without language-related impairments, musical savants and so on), and individual differences. For example, only a portion of autistic participants may show a specific type of atypicality [41]. Fundamental-frequency discrimination and categorization of musical or sentence stimuli have been shown to be better than normal for autistic children [41]. The frequency discrimination of pure tones by autistic individuals has been reported to be better than for NT individuals in some studies [42–44] and worse in other studies [45,46]. For intensity discrimination, the performance of autistic individuals has been reported to be either similar to that for NT individuals [43,44] or worse [46]. Autistic individuals show higher gap-detection thresholds than NT individuals [45]. Sound localization in the horizontal plane was not adversely affected in autistic individuals [47].

When trying to understand one talker in the presence of one or more competing talkers, listeners may have difficulty in knowing which parts of the signal emanated from which talker, especially when the talkers appear to have the same location in space. This is called ‘informational masking’. Autistic individuals, as well as NT individuals, can take advantage of a difference in apparent location of the target and interfering sounds to reduce informational masking [48]. In a typical room, the sound coming directly from the source reaches the ears slightly earlier in time than the reflections (echoes) from the room surfaces. Usually, the echoes are perceptually suppressed and the entire sound is perceived as coming from the direction of the leading sound. This is called the precedence effect. The effect breaks down if the time delay between the leading sound and the echo is too long. This breakdown occurs at a shorter time for autistic individuals than for NT individuals [47]. In summary, at a group-level autistic individuals may show worse, better or similar performance to NT individuals. The inconsistencies across studies may partly reflect large individual differences among autistic people.

To further examine the extent to which basic auditory functions are altered in autistic individuals in a manner that is specific to each individual, we conducted a set of psychophysical experiments using ASD and NT participants [49]. Twenty-one high-functioning adults with ASD (six females) and 19 NT adults (10 females) were matched in age (mean ± s.d.: ASD group, 30.0 ± 7.5, NT group, 27.2 ± 6.0) and IQ (ASD group, 105 ± 15, NT group, 117 ± 12) for both verbal IQ (ASD group, 111 ± 16, NT group, 120 ± 12) and performance IQ (ASD group, 97 ± 14, NT group, 109 ± 11). All participants were evaluated regarding their autistic traits using the autism spectrum quotient (a measure of the strength of autistic traits; ASD group, 36.2 ± 7.1, NT group, 18.5 ± 5.4). The measures were auditory filter bandwidth (a measure of frequency selectivity), the reception threshold for speech in noise (the speech-to-noise ratio at which speech can just be understood), discrimination thresholds for ITD and ILD, binaural masking level difference (the advantage for signal detection when the ILD and/or ITD of a signal are different from that of a masker), gap-detection threshold, threshold for detecting a change in level, threshold for detecting a change in frequency, threshold for discrimination of a pitch created by binaural interaction (Huggin's pitch; a white noise is presented to each ear and is identical at the two ears except for a phase difference over a restricted frequency range; a pitch is heard corresponding to the frequency where the phase shift occurs), and threshold for discriminating a change in pitch associated with a frequency shift of the components in a harmonic complex tone, keeping the frequency spacing of the components constant (for example, components with frequencies of 2000, 2200 and 2400 Hz might be shifted to 2030, 2230 and 2430 Hz, respectively) [49]. We found that the ASD group performed significantly more poorly than the NT group in the discrimination of ITD and ILD, and the ASD group had higher (worse) reception thresholds for speech in noise. Of special interest here are the results for discrimination of the pitch of frequency-shifted complex tones. This task essentially measures sensitivity to the temporal fine structure (TFS) of the waveform [50]. We found no significant difference in mean thresholds between the two groups. However, the distribution of individual thresholds differed significantly across groups. In the ASD group, the smallest detectable frequency shift was greater than 0.5 times the fundamental frequency (essentially meaning that the task could not be performed) for eight out of 17 participants. In the NT group only one out of 19 participants were unable to perform the task. This finding suggests that nearly half of autistic individuals have very low sensitivity to TFS. Given that ITD, ILD and TFS cues play a significant role in the perceptual separation of multiple sound sources, it would seem reasonable to conclude that autistic individuals who have low sensitivity to these cues will have trouble in selective listening. The results are consistent with anatomical findings showing that some neurons in the brainstem of autistic individuals are disordered [51,52], because the brainstem plays significant roles in the processing of ITD, ILD and TFS cues. The auditory system in the brainstem has multiple pathways, each specialized for the processing of specific cues [8,9]. Therefore, it is possible that the processing of some cues is impaired while the processing of others is intact. However, it is not clear at this point whether the impairments of specific auditory functions are solely due to disorders in brainstem structures. Top-down modulation from the cortex may also play a role. Further research is necessary to clarify this point.

Temporal processing can be assessed by measuring the response to a brief sound (the target) presented shortly after a masking sound, i.e. by measuring forward masking. Autistic individuals show smaller auditory brainstem responses (ABRs) to forward-masked targets but not to unmasked targets relative to NT individuals [53]. In other words, they show more forward masking. In response to spectro-temporally complex auditory stimuli, autistic individuals show diminished activity in the non-primary auditory cortex but increased activity in the primary auditory cortex [54]. Autistic individuals show reduced ABRs synchronized to the fundamental frequency of speech syllables with descending or ascending fundamental-frequency contours. One possible explanation for these observations is inefficient temporal processing in autistic individuals [55].

In auditory scene analysis, sequences of sounds tend to be heard as one stream (as if they emanated from a single source) when they have similar spectral properties and perceived locations but to split into more than one stream if these properties change markedly across successive sounds. For example, if the frequencies of successive tones in a rapid sequence alternate between two very different values, NT individuals tend to hear two streams. This effect can be studied indirectly by measuring the mismatch negativity, which is an electrical response of the brain to ‘oddball’ stimuli (stimuli that diverge from an expected pattern). Autistic individuals showed a smaller effect of the frequency separation of successive tones than NT individuals in a mismatch negativity study [56]. Another study showed that autistic individuals have less sharp ‘spatial filters’ (quantified using judgements of whether two consecutive tones come from the same or different locations in space) than NT individuals [57]. While spatial separation of two sound sources usually helps in perceptual segregation of the sounds, autistic individuals show less benefit than NT individuals.

In everyday life, background sounds often fluctuate in amplitude over time. NT individuals can take advantage of this for speech perception by obtaining ‘glimpses’ of the target speech during dips in the background. This is often called ‘listening in the dips’. To dip-listen effectively, the information from the glimpses must be combined over time. Autistic individuals show poorer performance than NT individuals when trying to understand speech in noise with temporal modulations [58,59], i.e. autistic individuals show a deficit in dip listening. This may happen because of problems in extracting information from the dips or because of problems in combining the glimpses over time [55].

Although the results of several of the studies described above suggest that autistic individuals have deficits in processing the cues that are used for scene analysis, our study [48] showed that autistic individuals have superior performance in hearing out a target auditory sequence embedded in masker auditory sequences with similar auditory texture [60]. In one condition of this experiment, the pure tone bursts in the target had randomized frequencies across bursts and the pure tone bursts in the masker had the same frequencies across bursts. In this condition, both autistic and NT individuals found it easy to hear the target. In a second condition, both the target and the masker had randomized frequencies across bursts. NT individuals found that this made the target much harder to perceptually segregate from the masker. However, autistic individuals did not find this condition difficult; indeed their performance in this condition was similar to that in the condition where the masker had the same frequencies across bursts. We argue that the superior performance of the ASD group in the condition with high masker uncertainty indicates a lack of automatic grouping of the target and masker based on similar spectral uncertainty. Furthermore, we argue that the lack of automatic grouping might lead to worse performance in hearing speech in a noisy environment, because speech is a complex, broadband stimulus and our auditory system must rely on automatic grouping to process speech stimuli efficiently.

Auditory scene analysis also relies on top-down modulation based on templates formed by previous experience [61]. Evidence for this top-down process comes from studies of the perception of local features when there is global interference. Autistic individuals are less affected than NT individuals by global contours (defined by their spectro-temporal patterns) when they are asked to pay attention to the local features [62]. The reduced top-down influence and reduced use of global information may adversely affect speech perception by autistic individuals.

Autistic individuals tend to show atypical responses to social auditory stimuli, namely, voices. Autistic children have less preference for their mothers’ speech [63], and they show smaller responses than NT individuals when other people call their names [64]. Moreover, autistic adults have difficulty perceiving prosody in speech [65]. Autistic individuals have reduced voice-specific cortical responses to human voices [66,67] and to speech or speech-like complex sounds [67–69]. Autistic individuals have similar event-evoked brain potentials to NT individuals for speech stimuli but show less involuntary attentional orientation to speech [70,71]. A brain functional connectivity study also revealed a weaker connection between voice-specific auditory cortex and the reward system in autistic individuals [72].

In summary, several studies have revealed diverse differences in basic auditory processing and auditory scene analysis between autistic individuals and NT individuals. The behavioural and neurophysiological differences observed in experiments investigating auditory scene analysis in autistic individuals indicate abnormal processing not only in the auditory cortex but also in subcortical parts of the auditory pathway.

4. Visual scene analysis in autism

There is a growing body of evidence that atypical processing occurs at many levels of the visual system [73,74]. For low-level visual processing, Berton et al. [75] reported lower (better) orientation-discrimination thresholds for luminance-defined (first-order) gratings in autistic individuals than in NT individuals. Atypical low-level visual processing is further supported by the finding of altered occipital visual evoked potentials (VEPs) associated with sine-wave gratings of medium and high spatial frequency [76–78]. However, psychophysical measures of contrast sensitivity to several spatial frequencies showed that autistic individuals do not have different spatial frequency processing to NT individuals ([79], see also [80,81]). In contrast to enhanced or intact sensitivity to luminance-defined (first-order) stimuli, autistic individuals show inferior sensitivity to texture-defined (second-order) stimuli, which involve mid-level visual processing [75,82]. It is known that first- and second-order information analyses rely on distinct neural mechanisms; the first-order attributes are processed in the striate cortex, whereas second-order attributes are processed in additional brain areas including extrastriate cortices [82,83]. Regarding second-order information analysis, Rivest et al. [82] reported that, unlike NT children, autistic children did not show pronounced neural activity in extrastriate visual areas in response to texture as compared to luminance gratings. Contour integration processes need to integrate local edges, which may correspond to the outline of a visual shape. Pei et al. [84] were unable to identify the neural correlate of contour integration in autistic children. Those findings imply that autistic individuals have difficulties in mid-level visual processes (texture segregation and contour integration).

It has been found consistently that autistic individuals demonstrate superior performance on the embedded figure task (EFT). In this task, participants are asked to find a simple ‘hidden’ figure embedded in a larger figure with complicated patterns [85–88]. An fMRI study with NT adults showed activation specific to the EFT in the left inferior and superior parietal cortex and left ventral premotor cortex [89]. Activation of those brain areas has been associated with spatial attention shifts [90,91]. However, such activation was not found for an ASD group [89,92]. These findings suggest that impaired attentional control in autistic individuals results in better performance of some visuospatial tasks.

There is also extensive research on visual search in autistic individuals. There are two types of visual search task: feature search and conjunction search. In feature search, participants are asked to detect a target item that differs from all simultaneously presented distractor items along a single feature dimension (e.g. searching for a red S target among green S distractors). In conjunction search, the target shares one feature with some distractors and another feature with other distractors (e.g. searching for a red S target among green S and red T distractors). Most studies have shown that autistic individuals excel in feature and conjunction search tasks (see [74,93] for review) with some exceptions [94–96]. The superior performance in visual search is found at different developmental stages from two-month-old infants to adults [97–104].

There are two classical models for visual search: the feature integration model and the guided search model [31,32], but they are not applicable to the ASD advantage in visual search in a straightforward way. These two models are based on the assumption that basic features of multiple stimuli such as colour, edge orientation and motion direction are analysed simultaneously across the visual field in parallel processing stages called maps (one for each feature). For feature search, the target can be identified using a single map. In this case, the target subjectively ‘pops out’ from the distractors. For conjunction search, more than one map must be used, and serial inspection directed by attention is necessary to find the target [31,32]. This suggests that serial shifts of spatial attention are necessary for finding a target, that is, observers cannot discriminate stimuli simultaneously in conjunction search. However, the results of several studies suggest that the superior search skills of autistic individuals are associated with better simultaneous discrimination of multiple complex stimuli [97,99,104]. Eye-tracking studies demonstrated that the superior skills are associated with decreased frequency or duration of fixations during visual search [97,99]. In order to interfere with serial search processes, we conducted a study using a backward masking technique in which a brief display containing a target and distractors was followed by a noise mask [104]. The results revealed that autistic individuals excel in processes that do not involve serial search, especially in the instantaneous discrimination of multiple visual stimuli, for example, during a single fixation [104]. Also, Remington et al. [105] reported that, in a dual-task experiment, the ability to detect a target stimulus deteriorated with increasing perceptual load for NT individuals but not for autistic individuals [105].

Autistic individuals show impairments in attentional networks, including the alerting, orienting and executive control networks [106]. Some researchers have suggested that the superior performance of autistic individuals in visual search is related to their atypical attention system. Blaser et al. [107] found that better search performance was correlated with larger phasic pupil response; the latter reflects activation of the locus coeruleus–norepinephrine system, which in turn facilitates performance [108]. The results suggest that greater attentional focus might be the basis of the ASD advantage in visual search. By contrast, individuals with better visual search are less efficient at filtering out irrelevant stimuli by employing feature-based attention [109], and individuals with a higher autism quotient also have a worse ability to filter out irrelevant stimuli [110].

In contrast to their superior search ability, autistic individuals are worse in the extraction of global information from natural images [111]. They are slower in tasks where participants are asked to describe the gist of the presented scene when looking at briefly present real-life photographs [111]. Some researchers have pointed out that although autistic individuals have a reduced tendency to report global features of a stimulus when offered a choice of what features to report, they perform similarly to NT individuals when explicitly instructed to attend to global features [112].

Regarding the top-down modulation of visual perception, several lines of evidence support the notion that perceptual experience is less influenced by prior knowledge in autistic than in NT individuals [113,114]. Autistic children showed a failure of shape constancy perception, namely the ability to correctly judge the shape of an object like a coin despite changes in the angle of viewing. Prior knowledge of perspective cues surrounding a shape influences perception for NT children, whereas autistic children perceive the shape more veridically [114]. Visual illusions can also be used to assess the influence of prior knowledge on perception. Researchers have been debating whether autistic individuals are less susceptible than NT individuals to visual illusions, such as the Kanizsa triangle, the Ehrenstein illusion, and the Poggendorf and Shepard table illusion [73,74,113] (figure 1), but the evidence at present is mixed. For tasks involving complex geometrical reasoning, such as Raven's Progressive Matrices, autistic individuals exhibit better performance than NT individuals [115]. Neuroimaging studies revealed that the process of solving such tasks involves a spatially extended brain network including prefrontal and parietal regions in NT individuals [115,116]. In autistic individuals, on the other hand, stronger recruitment of occipital and temporal regions but less recruitment of frontal and parietal regions was found [115,116].

Figure 1.

(a) The Kanizsa triangle: a bright white triangle, occluding three black circles and a black-outlined triangle, can be perceived even though there are no explicit lines or enclosed spaces to indicate such a triangle. (b) The Ehrenstein illusion: a configuration of four line segments induces the perception of an illusory figure at the centre of the configuration. (c) The Poggendorff illusion: colinear diagonal lines behind an intervening rectangle appear to be misaligned. (d) The Shepard table illusion: the right-hand table top seems longer and thinner, even though it is actually identical to the other table top.

5. Discussion

(a). Autistic scene analysis: comparison between the auditory and visual systems

For both auditory and visual modalities, psychophysical and electrophysiological studies have shown specific effects of ASD in the low-level processing of certain features. These include enhanced or poorer sensitivity to frequency [41], poorer sensitivity to TFS, ITD and ILD [49], and weaker ABRs during temporal masking and pitch tracking tasks [53,55] in audition, and enhanced sensitivity to the first-order orientation [75] and poorer sensitivity or weaker VEPs to the second-order features [75,82] in vision. Although these findings show some similarities between the two modalities, the neural sites responsible for the anomalies are not necessarily similar. In vision, the first- and second-order features are mainly processed in the striate and extrastriate cortices, respectively [83]. In audition, subcortical structures play essential roles in the processing of features for pitch and localization, and contribute actively to auditory scene analysis [14,15]. Consistent with this, autistic individuals have anatomical and functional anomalies in the brainstem [52,117]. To the best of our knowledge, no study has demonstrated a link between atypical visual perception and brainstem function in autistic individuals.

For both auditory and visual modalities, autistic individuals are less proficient than NT individuals in the automatic grouping of local features, which depends on mid-level processing. Autistic individuals perform more poorly than NT individuals in sound segregation tasks [56] and a visual contour integration task [84]. Autistic individuals also show relatively poor grouping over time. In audition, they have difficulty hearing out target speech in background sounds, probably partly because of difficulty in integrating fragmentary information about the target obtained during temporal dips in the background [58,59]. In vision, autistic individuals perform more poorly than NT individuals in recognizing familiar objects moving behind a narrow slit [118]. However, inefficiency in automatic grouping processes does not degrade performance in all tasks. For example, autistic individuals performed relatively well in detecting a target tone sequence embedded in distractor tones, presumably because the distractor tones were not grouped with the target tones [48]. Furthermore, there is considerable evidence for an ASD advantage in the EFT and visual search tasks [73,74]. Although it is still controversial, some researchers regard dysfunctions in space- or feature-based attention as the underlying cause of the ASD advantage [92,109].

For both auditory and visual modalities, autistic individuals are less influenced by top-down modulation, which reflects high-level processing [62,114,119]. They show superior performance when processing local elements embedded in a melody, suggesting reduced global-to-local interference [62,119]. Likewise, the influence of prior knowledge about linear perspective on visual three-dimensional perception is different between individuals with and without ASD [114].

(b). Hypotheses and neural mechanisms

As described above, a number of studies have indicated that, in ASD, diverse atypicalities can be found at various levels of auditory and visual information processing, namely, the extraction of local features, the automatic integration of local features and top-down modulation by attention and prior knowledge. In the light of these findings, we next examine three influential theories about the specific profile of perception in ASD.

The weak central coherence theory [120] is based on the idea that ASD is a consequence of a reduced tendency to integrate local information into a coherent or ‘global’ whole, combined with increased attention to detail. The enhanced perceptual functioning theory suggests that ASD results from enhanced perception of simple, low-level sensory information without an impairment of global processing [121]. The neural complexity theory [75,122], proposed to explain auditory processing in ASD, posits that ASD is a result of enhanced perception of simple, low-level auditory stimuli, together with impaired perception of more complex auditory information. The theories differ in their assumptions as to whether atypical processing in ASD occurs only for local or global processing or applies to both. The weak central coherence theory posits that local processing is intact and global processing is impaired. The enhanced perceptual functioning theory posits that local processing is enhanced and global processing is intact. The neural complexity theory posits that local processing is enhanced and global processing is impaired. Although all three of the theories can explain some aspects of perception by autistic individuals, none is perfectly consistent with all of the data. As reviewed above, in ASD, local processing can be enhanced or impaired, and global processing such as automatic grouping and top-down modulation is impaired. Impaired global processing would result in the final percept being strongly affected by the characteristics of local processing, and the characteristics of local processing in ASD can be atypical. The combination of these factors can make scene analysis for autistic individuals quite different from that for NT individuals. For the latter, the representations of objects or events in the brain gradually become fully formed as processing proceeds. For autistic individuals, on the other hand, this process is incomplete, and complete representations of objects or events may not be achieved. In this sense, scene analysis in ASD is ‘truncated’. The concept of truncated scene analysis may help in understanding the seemingly mixed experimental findings regarding unusual perception in autistic individuals.

We consider next why truncated scene analysis in autistic individuals does not always lead to poorer performance than for NT individuals. There is a trade-off between representations formed at earlier levels and those formed at higher levels in terms of quality and quantity. For NT individuals, basic features are integrated into perceptual wholes and categorized as objects such as phonemes in speech or letters in visual analysis. Higher level sensory information processing guided by attention and based on stored knowledge can increase the speed and accuracy of object formation, but at the expense of a decrease in the number of stimuli that can be processed. For example, observers frequently miss objects outside the focus of attention [123]. Also, it is difficult to access basic representations of sensory features after perceptual organization has been completed [124]. Impaired higher level processing in autistic individuals may reduce or slow down the formation of objects but improve access to basic representations of stimulus features. Therefore, truncated scene analysis in autistic individuals does not always lead to poorer performance than for NT individuals. NT individuals have an advantage when the task demands integration of local features into a unitary object and an abstract category, as in many situations in daily life. However, autistic individuals may excel in auditory and visual tasks for which the formation of unitary objects makes it harder to access basic stimulus features [48,85,103].

The concept of truncated scene analysis appears to be consistent with the underconnectivity hypothesis, which posits that long-distance connectivity in the brain is reduced in autistic individuals. There are two aspects of brain connectivity: structural connectivity and functional connectivity. In diffusion tensor imaging, there are consistent findings of decreased fractional anisotropy (FA) in white matter tracts in regions such as the corpus callosum, cingulum and arcuate fasciculus in autistic individuals [125]. White matter connectivity from tractography is usually measured by diffusion anisotropy, and the most widely used measure of anisotropy is FA. Decreased FA indicates less directional diffusion along the white matter. On the other hand, there is debate about whether there is decreased functional connectivity in autistic individuals [126]. A recent study used the Autism Brain Imaging Data Exchange to analyse data for a large number of individuals with or without ASD. The results showed a predominance of hypoconnectivity in ASD [127]. In addition, an MEG study showed that it was possible to identify autistic individuals based on measures of long-range and local functional connectivity obtained while participants were viewing faces [128]. The underconnectivity hypothesis, is consistent with the idea of truncated scene analysis in ASD, such as deficient feature integration and a weaker influence of top-down modulation, both of which require connections across distant neural sites. The hypothesis is also consistent with neuroimaging data. For example, the lower activation in the frontal cortex of autistic individuals during the EFT may be due to lower functional connectivity between higher order working memory/executive areas and visuospatial areas [129].

Another neurophysiological hypothesis relating to ASD is excitatory/inhibitory (E/I) imbalance, which refers to increased glutamatergic (excitatory) signalling and decreased GABAergic (inhibitory) signalling, as observed in animal models [130,131]. The E/I imbalance could affect the processing of local features [132], because it might result in reduced neural dynamic range, inappropriate gain control, and spatially and temporally imprecise inhibition. Moreover, some researchers have proposed that the E/I imbalance may disrupt feature integration in scene analysis. Binocular rivalry has been used as a tool to test the E/I imbalance hypothesis in humans. When different stimuli are presented to the two eyes, a mixture is sometimes perceived, but often the percept switches irregularly between the stimulus at one eye and the stimulus at the other eye. Computational models of binocular rivalry predict lengthened durations of mixed percepts if the E/I ratio is not balanced. Two studies have investigated binocular rivalry in autistic individuals. The outcomes were inconsistent: one study showed no group difference [133] while another showed the predicted lengthened durations of mixed percepts for the ASD group relative to the NT group [134]. Clearly, further research is required.

An important issue is the relationship between specific sensory effects and the core symptoms of ASD. One possibility is that they coexist without direct causality. Neurophysiological or neuroanatomical characteristics of the brain implicated in ASD, such as E/I imbalance and underconnectivity, would be expected to affect the general operation of the brain. In this case, mutually independent functions might be similarly affected, but this would not imply that abnormalities in one function caused abnormalities in another. Another possibility is that specific sensory effects are one of the causes of the core symptoms. As discussed earlier, scene analysis usually cannot be achieved based only on sensory data, which are inherently ambiguous. Efficient and effective scene analysis requires prior knowledge about and internal models of the events and objects in the external world. If we assume that knowledge is acquired through learning in the course of development, the nature of the acquired knowledge would be critically dependent on the early processing of the sensory inputs. As a consequence, the acquisition of knowledge and the formation of internal models may be atypical in autistic individuals. Social cognition and interpersonal processes require prior knowledge in order to interpret non-verbal information such as vocal or facial expression. If atypical scene analysis in individuals with ASD influences the acquisition of knowledge about cues to social and emotional aspects of voices and faces, this could account for their difficulties in social communication and interaction.

(c). Future directions

In this paper, we have reviewed the singular nature of scene analysis in autistic individuals, and its similarity between the auditory and visual modalities. At present, the computational and neural mechanisms underlying the specific atypicalities and their variation across individuals remain unclear. There are still inconsistencies in research findings in some areas. In resolving the inconsistencies, researchers should be aware of the great diversity of autistic individuals. There may be several sub-types of ASD, which do not necessarily share the same underlying mechanisms. Also, researchers should be careful about the selection of experimental stimuli and methods. Consider pitch processing as an example. Different types of stimuli and tasks involve quite different levels of processing and neural sites: for example, pure tone frequency discrimination requires local processing, discriminating the fundamental frequency of a harmonic complex tone requires across-frequency grouping, judging speech prosody involves speech-specific processes, and musical-interval judgements depend on learned categories in a musical context. Researchers should be aware of which stages of processing might affect performance for their specific stimuli and methods. In addition, some findings in one modality have not been extensively studied in another modality. For example, atypicalities in top-down modulation of visual perception for autistic individuals have been studied extensively, but there are few similar studies in the auditory domain.

Advances in imaging methods, such as functional connectivity analysis, tractography and magnetic resonance spectroscopy, have made it possible to explore the neural mechanisms underlying scene analysis, but great care is needed concerning methodological details. In addition, present technology is limited in its ability to detect abnormalities in subcortical areas. For example, subcortical areas such the superior olivary complex and inferior colliculus play critical roles in auditory processing, but imaging resolution for the brainstem and midbrain is not yet sufficient to detect minor structural atypicalities.

No matter what the mechanism is, the specific profile of auditory and visual scene analysis in autistic individuals could contribute to their communication difficulties. To appropriately support autistic individuals in their daily lives, we need a detailed understanding of the specific sensory abnormalities of each individual. If reliable correlations are found between specific aspects of sensory processing in scene analysis and the severity of (a sub-type of) ASD, these could be exploited for the early detection of ASD. Also, the correlations could provide important clues bridging the phenomenology and biological mechanisms of ASD.

Authors' contributions

All authors wrote the manuscript.

Competing interests

We have no competing interests.

Funding

M.K. is partly supported by JST CREST and A.S. is supported by JSPS.