Designing sensory-substitution devices: Principles, pitfalls and potential

Abstract

An exciting possibility for compensating for loss of sensory function is to augment deficient senses by conveying missing information through an intact sense. Here we present an overview of techniques that have been developed for sensory substitution (SS) for the blind, through both touch and audition, with special emphasis on the importance of training for the use of such devices, while highlighting potential pitfalls in their design. One example of a pitfall is how conveying extra information about the environment risks sensory overload. Related to this, the limits of attentional capacity make it important to focus on key information and avoid redundancies. Also, differences in processing characteristics and bandwidth between sensory systems severely constrain the information that can be conveyed. Furthermore, perception is a continuous process and does not involve a snapshot of the environment. Design of sensory substitution devices therefore requires assessment of the nature of spatiotemporal continuity for the different senses. Basic psychophysical and neuroscientific research into representations of the environment and the most effective ways of conveying information should lead to better design of sensory substitution systems. Sensory substitution devices should emphasize usability, and should not interfere with other inter- or intramodal perceptual function. Devices should be task-focused since in many cases it may be impractical to convey too many aspects of the environment. Evidence for multisensory integration in the representation of the environment suggests that researchers should not limit themselves to a single modality in their design. Finally, we recommend active training on devices, especially since it allows for externalization, where proximal sensory stimulation is attributed to a distinct exterior object.

1. Introduction

In his seminal epistemological work, De Anima, the Greek philosopher Aristotle made an important distinction between the individual senses and the common sense (koinē aesthesis, see Ackrill, 1987). The individual senses (seeing, hearing, touch and smell) have their own domains but many perceptual functions do not involve simple operations of individual senses, but a central holistic representation of the environment. Aristotle realized that perception of the environment is a multifaceted operation where all available senses come into play. Aristotle's insight highlights not only that our representation of the world is holistic, but also that this representation is multimodal since it involves information from all the senses. Borrowing Aristotle’s term, the common sense builds a representation of the environment that encompasses information from allsenses. There are various examples of how information from one modality influences, or even alters, our perception of stimuli in another modality (McGurk & McDonald, 1976; Shams, Kamitani, & Shimojo, 2000; Hötting & Röder, 2004), showing how important integration from different sensory modalities can be. Perhaps most importantly, this holistic representation of the environment is influenced by perceptual interpretation (Hoffmann, 1998; Palmer, 1999; Rock, 1983). The stimulation from the external world (light, air pressure, tactile stimulation) is only part of the story in perception. From the stimulation, interpretive mechanisms of the nervous system create a useful representation of the external environment.

When individuals lose the function of one sense, such as through damage to sense organs, mechanisms concerned with perceptual interpretation are not necessarily affected. It should in principle be possible to find a way of conveying the missing information, if this information can still be processed. As in the case of blindness that can be traced to damage to the retinae, there is still a lot of intact neural hardware devoted to visual function. As Bach-y-Rita and Kercel (2003) put it: “Persons who become blind do not lose the capacity to see” (p. 541). Information may be fed to these neural mechanisms via other channels, utilizing the fact that the main organ of perception is the nervous system. The brain is not just a sensory-machine that responds separately to specific stimuli or sensory modalities but rather a complex “task-machine”, that can partially restore function with input from other senses (Maidenbaum et al., 2014a; Murray et al., 2016). This is the domain of sensory substitution (SS), where touch or audition, for example, convey information that is otherwise not available, such as vision. Sensory substitution devices (SSDs) have been available for a long time. The white-cane for the blind translates environmental structure into haptic and proprioceptive feedback and sign language translates visual stimuli into language. In Braille, “verbal” information is conveyed through haptic stimulation. Such devices have become increasingly sophisticated with advances in technology.

Here we present an overview of research on ways of conveying information to the blind about the external environment through other intact senses with the aim of improving perceptual function and mobility. While we limit our focus to haptic and auditory feedback approaches for the blind, many issues mentioned here undoubtedly apply to other forms of sensory substitution. Our review provides a number of novel perspectives pertinent to sensory substitution that have not been addressed systematically in the literature. We discuss research on neural activity during sensory substitution. Additionally we highlight the importance of training on sensory substitution devices, the best ways of implementing such training and constraints stemming from inherent characteristics of sensory mechanisms. We also point out some pitfalls that any such enterprise must avoid.

2. Key general considerations for sensory substitution

The neural mechanisms of vision in humans are more complex than for the other senses. The auditory nerve has around 30,000 fibers while the optic nerve contains over 1 million fibers (Wurtz & Kandel, 2000), and psychophysical measurements show that the information capacity of vision is considerably higher than of audition (Ash, 1951). Brown, Simpson and Proulx (2014) found that there is an upper limit to representation of auditory objects due to intrinsic limits of transmission in auditory cortical areas. Investigations of the capacity of the visual system reveal that it has four orders of magnitude greater bandwidth than haptic perception (Kokjer, 1987; Schmidt, 1981) while the information capacity of the human ear falls between these two estimates (Bialek et al., 1993; Dudel, 1986; Proulx et al., 2016; Wurtz & Kandel, 2000). Loomis, Klatzky and Giudice (2012) argue that the bandwidth of touch equals blurred vision through filtering of information in the cutaneous system.

Vision loss leads to lessened mobility. Various attempts have been made at generating compensatory strategies where information about the environment is conveyed to the blind through audition and/or touch. The goal has been to implement this feedback in real time, to enable obstacle avoidance, scene analysis or object recognition through interactions with the environment. One of our central points is that any sensory substitution device (SSD) and related training must be based on a thorough understanding of the key aspects that need to be conveyed, and – no less importantly – a thorough understanding of the psychophysics of the perceptual channels involved. Notably, although the available evidence suggests that there is great potential in sensory substitution methods (see Bach-y-Rita & Kercel, 2003; Merabet & Pascual-Leone, 2010; Proulx et al., 2016; Shull & Damian, 2015; Nagel et al., 2005, for reviews), such devices are still not in widespread use (Elli, Benetti & Collignon 2014).

2.1. The need for basic research

Although many technologically sophisticated methods of sensory substitution have been developed, basic research on cognition and perception necessary for effective generation of these technologies is often neglected (see e.g. Loomis, et al., 2012). Important questions, such as how sounds and tactile stimulation are interpreted, and what sort of stimulation is most effective for conveying particular information about the environment to a particular sense, are not always considered in enough detail. We revisit this issue at various points throughout this review, as many issues are related to this general one. In the end, this boils down to the obvious point that information conveyed through sensory substitution needs to match the capabilities of the human nervous system to be useful. The most straightforward way of answering such questions is through psychophysical experimentation.

Devices have been developed that may exceed the capabilities of perceptual channels, such as haptic stimulation devices that exceed 2-point thresholds for touch of the stimulated part of the body (Weinstein, 1968; Gardner & Kandel, 2000). Note however that even if two-point thresholds are exceeded it is possible that perceptual interpretation can compensate for this (see e.g. Novich & Eagleman 2015; and discussion in Bach-Y Rita, 2003). The pattern of stimulation may be detected and perceptual interpretation can functionally result in better resolution than 2-point thresholds dictate (Bach-Y-Rita & Kercel, 2003). Findings on fundamental limits imposed by attentional capacity and potential sensory overload (section 2e) demonstrate how basic perceptual and cognitive science can provide vital information for SSD design.

Loomis et al. (2012) argue that two main steps are needed to generate effective auditory or haptic substitution devices for the blind. The first is to determine what information is most critical for enabling a particular function. During design it is important to focus on the information that is most relevant for the task at hand, and this involves determining which aspects of the visual environment are functionally most important. If the aim is mobility and obstacle avoidance, the emphasis should be on identifying obstacles and conveying their location and size, rather than their color or shape. Secondly it is necessary to determine how to effectively convey this information to the haptic and auditory sensory systems. The basic processing properties of the perceptual channels in question need to be considered and effective models of encoding the information into sound and haptics must be developed (see e.g. Brown et al., 2014), for instance, encoding obstacle direction to sound through spatial audio (Blauert, 2013; Geronazzo et al., 2016) and/or appropriate haptic rendering techniques (van Erp et al., 2005). Neglect of this important point may have prevented the usability of previous SSDs (see e.g. Elli et al., 2014).

2.2. Comfort and ease of use

Ease of use, comfort level, mobility and appearance need to be seriously considered if the blind are to be expected to use a particular device (Dakopolous & Bourbakis, 2010; Elli et al., 2014; Shull & Damian, 2015). Dakopolous and Bourbakis (2010) suggest the following principles for any sensory substitution system for vision:

• i)Users’ hands should be free. Interesting substitution devices have been developed that are handheld (Hugues et al., 2015), but an optimal system would probably allow the hands to be free or at least minimize their use for operation.
• ii)Any auditory device should not interfere with the users’ ability to sense the environment directly (i.e. hear environmental sounds). There are at least three reasons why this is important: general sensory overload, limited attentional resources (points that we discuss in more detail in in section 2e), and thirdly, the intended users are typically highly accustomed to using environmental sounds.
• iii)The system needs to be easily wearable. Equipment that is too bulky or restricts mobility defeats its own purpose and will probably be rejected by intended users.
• iv)The system should be simple and easy to use. Its operation and the user interface should not be loaded with unnecessary features but only the bare minimum required for successful operation and usability.
• v)The equipment should not require extensive training. This last point made by Dakopolous and Bourbakis (2010) may be contentious, though, and if equipment that works, but does require training, is developed, this should certainly not be held against it. Successful approaches that require considerable training have been developed (Kärcher et al., 2012). Notably, training may be especially important for successful externalization (see section 2.3), which is, in all likelihood, beneficial for successful sensory substitution.

2.3. Externalization

Consider the case of the widely used white cane. The cane provides auditory, tactile and proprioceptive information about the environment. While the cane, however useful, has many drawbacks, it hammers home the important point that sensory substitution devices can become externalized (also called distal attribution; Auvray et al., 2005; Hartcher-O’Brien & Auvray, 2014), so that the haptic pressure picked up by mechanical receptors on the hand is experienced as occurring at the tip of the cane, making use of the “out-of-body” illusion (Maravita et al., 2003). Maravita et al. showed that through training, tools can become externalized so that visual-tactile interaction extends into far space (Maravita et al., 2001) and representations can change following active tool-use, which can extend the participants’ representation of peripersonal space (Maravita, Spence & Driver, 2003). Indeed, the cane can be experienced as an extension to the body (Serino et al., 2007). The ultimate aim with sensory substitution systems is to allow the user to build an effective mental representation of the environment, and experience from tactile or auditory stimulation that is externalized would help with this (see e.g. Loomis et al., 2012; Väljamäe & Kleiner, 2006).

Bach-y-Rita et al. (1969; see also White et al., 1970) developed a haptic stimulator array that conveyed information about the environment to blindfolded or blind observers. Following training with vibration applied to the observers’ back, the sensation changed from the vibration itself being the perceived stimulation site, to being felt as coming from the environment. White et al. (1970) referred to this as “seeing with the skin”, an example of externalization. Note importantly, that this only occurred for observers who actively trained with the device, consistent with the findings of Maravita et al. (2001, 2003; see also discussion in Proulx et al., 2008; Ward & Meijer, 2010; Bach-Y-Rita & Kercel, 2003; Stiles, Zheng & Shimojo, 2015). Similarly, participants in Bach-Y-Rita et al. (1972) reported externalized experiences following considerable training. It is only at this externalization stage that we can literally speak of sensory substitution. Ward and Wright (2014) argue that sensory substitution with good externalization might even be considered a form of acquired synesthesia (see also Farina, 2013).

Ward and Meijer (2010) reported that a user of the vOICe SSD exclaimed “It is sight!” about their experiences with auditory feedback. But note importantly that this participant (and others reporting similar experiences) previously had intact vision for a number of years and that the recruited cortical sites were likely previously used for vision. This may therefore not necessarily be the case for the congenitally blind. Another example of externalization comes from Maidenbaum et al. (2014b), where a blind participant described their experience from the use of the EyeCane: “I could feel the world stretching out before me. It was as if my hand could reach much further. As if the silent objects on the other side of the room were suddenly there” (p. 821). Another example of externalization comes from Nagel et al. (2005) who presented orientation information from a magnetic compass through a haptic belt. Following training, participants began to experience the magnetic feedback as an extra sense. Nagel et al. called this a sixth sense, since this involved qualitative changes of sensory experience.

2.4. Relation between sensory substitution and perceptual illusions

Studies of perceptual illusions have long given insight into the nature of perceptual processing (Gregory, 1998). This is no different in the context of sensory substitution. Hötting and Röder (2004) found that when a single tactile stimulus was accompanied by more than one auditory sound, this led to the perception of more than one touch. Interestingly this was less pronounced for congenitally blind observers. This illusion is a haptic-auditory analogue of a visual-auditory illusion introduced by Shams, Kamitani and Shimojo (2000) who found that when a single visual flash was accompanied by a number of auditory beeps, observers perceived the single flash incorrectly as many flashes. This demonstrates the operation of interpretative mechanisms in perception (section 1). It is also of great interest that congenitally blind participants were not as prone to the auditory-tactile illusion as sighted participants, suggesting that multisensory integration may differ in the congenitally blind, perhaps because they have better auditory discrimination abilities than sighted people or because vision plays a role in crossmodal calibration (Gori et al., 2010; Occelli, Spence & Zampini, 2013). The McGurk effect is another example of a multimodal illusion where lip movements and heard speech interact (McGurk & MacDonald, 1976). Interestingly, for the current context, Fowler and Dekle (1991) found a tactile/auditory McGurk effect when auditory syllables were presented along with a haptic representation of the syllables.

The cutaneous rabbit illusion (Geldard & Sherrick, 1972) involves a rapid repeated stimulation sequence at several locations on the skin that causes the perception of stimulation at intervening spaces between the locations that are actually stimulated. The stimulation is felt as if a small animal (e.g. a rabbit) hopped along successive stimulation sites. Blankenburg et al. (2006), using fMRI, showed how somatotopic brain sites corresponding to the experienced illusory percept were activated although they were not directly stimulated. While such haptic perceptual interpretation raises challenges for vision-to-haptic substitution devices, this may also provide cues regarding the nature of haptic processing, which can be utilized for sensory substitution. All this highlights the need for a deeper understanding of the psychophysical properties of the perceptual channels in question, and raises questions such as whether such illusions may be affected with training.

2.5. Attentional capacity and potential sensory overload

Humans can only process a limited amount of the information hitting their senses at a given time, since attentional capacity is severely limited (Broadbent, 1958; Treisman, 1960; Neisser, 1967; Most et al., 2005; Kristjánsson, 2006; Schneider & Shiffrin, 1977). A lot of information goes unnoticed because selective attention filters it out (Broadbent, 1958; see e.g. Driver, 2001 for review). Observers making eye movements across a visual scene miss large changes in the scene during eye movements (Grimes, 1996; McConkie & Zola, 1979) and are surprisingly inept at noticing large changes to otherwise identical visual scenes if the changed scene is alternatively presented with the original scene at a rapid rate with another visual event in between (O’Regan, Rensink, & Clark, 1999; Rensink, O’Regan, & Clark, 1997; see also Mack & Rock, 1998; Simons, 1996). Such attentional limitations also occur for haptic perception (Gallace, Tan & Spence, 2006) and audition (Vitevitch, 2003), showing how this is a universal principle of attention and perception.

An informative example of how attentional focus can have profound implications for technological design comes from a study where information from flight instruments in the cockpit of a passenger-jet was projected onto the windshield of the cockpit (Haines, 1991). The intention was that pilots would be able to simultaneously keep their eyes on the instruments and the visual scene outside the cockpit. Nevertheless, during flight simulation test pilots frequently missed large obstacles as they came in for landing, such as jets right in the landing path on the runway. Their attention was so focused on the instruments that they could not pay attention to vital information right in front of their eyes. This result has clear implications for sensory substitution. Information conveyed through headphones as a substitute for vision can be prioritized over other auditory information and little attention will be paid to real-world sounds. Kärcher et al. (2012) argue that even though sensory substitution devices can be beneficial for navigation and orientation in the environment, they place a strain on attentional resources, which may detract from attention paid to other aspects of the environment. Note also that within-modality interference is probably larger than between modality interference (e.g. Liu, 2001). This must be taken into account during design of devices and training. Implementation that places minimum demands on attentional resources is essential. Processing of information may, however, become increasingly automatic so normal attentional filtering may ensue.

Sensory overload must be avoided in the design of sensory substitution devices (White et al., 1970; see discussion in Elli et al., 2014). Auditory substitution must not interfere with the natural auditory environment (Härmä et al., 2004). To avoid sensory overload only vital information should be conveyed, to guarantee that the SSD does not obstruct perception of the natural soundscape, nor contradict it (Collins, 1985). Loomis et al. (2012) go so far as saying: “The failure to recognize the processing constraints of perception and cognition has resulted in limited utility of sensory substitution devices for supporting real-world behaviors”. Again, this highlights the importance of basic research into sensory and attentional mechanisms for sensory substitution.

2.6. Issues regarding training

The literature does not provide clear and precise training guidelines regarding SSDs, although good evidence suggests that training is important for successful sensory substitution (Stronks et al., 2015). Externalization, for example, seems to be achieved with considerable training only. Sensory substitution studies typically involve training regimes and the general finding is that performance improves with training (but see Stiles & Shimojo, 2015), but no studies have yet addressed this question systematically.

Increasing the capacity of the human mind has long been a goal. An example of this is the recent trend for brain-training games (Green & Bavelier, 2008). Another example is the literature on video-game training, where it has been claimed that playing first-person action video-games can enhance attentional abilities and that this learning transfers readily to other tasks (Bavelier et al., 2012, Spence & Feng, 2010). Although promising, on closer look, the video-game literature is fraught with confounds, ambiguities and overclaims regarding causal influences of such training (Boot, Blakely, & Simons, 2011; Kristjánsson, 2013), and benefits from brain-training games appear to be task specific (Owen et al., 2010).

Solid evidence for enhanced brain capacity through training is therefore scarce. At the same time, however, the human nervous system shows impressive flexibility (Merabet & Pascual-Leone, 2010). Disabilities from brain damage can be lessened through training (Singh-Curry & Husain, 2008; Saevarsson, Halsband, & Kristjánsson, 2011). But in such cases deficits from the loss of neural function are overcome, while many brain and video-game training approaches involve attempts to increase functionality of intact neural mechanisms. (See Shull, & Damian, 2015 for discussion of SSDs for intact senses and Wright, Margolis & Ward, 2015, for discussion of SS for low-vision). So even if brain training evidence is scarce in normal populations, the implications may not be as severe for situations involving dysfunctional senses. And there is indeed promising evidence regarding the use of virtual training environments such as virtual mazes or video-games in sensory substitution (Lahav, Schloerb, & Srinivasan; 2012; Merabet et al., 2012; Levy-Tzedek et al., 2016; Maidenbaum et al., 2014c; Maidenbaum & Amedi, 2015).

The brain is capable of considerable reorganization of neural function (Held et al., 2011; Rauschecker, 1995), while younger brains are typically more flexible (Hubel & Wiesel, 1970; Merabet & Pascual-Leone, 2010). Cortical areas responsive to audition expand in visually deprived cats (Rauschecker & Korte, 1993) and their neurons become more sharply tuned to location on the azimuth (Korte & Rauschecker, 1993; see also Rauschecker & Kniepert, 1994; King & Parsons, 1999). Extensive practice on particular dimensions can lead to structural changes in neural mechanisms (Pascual-Leone et al., 2005), such as Braille (Hamilton & Pascual-Leone, 1998; Pascual-Leone & Torres, 1993) and music (Elbert et al., 1995) and cortical representations for musical tones are enlarged in experienced musicians (Pantev et al., 1998; Schlaug et al., 1997). Note that sensory substitution may not require training if crossmodal mappings are strategically utilized (Stiles & Shimojo, 2015). Intrinsic crossmodal mappings may play an equally important role in sensory substitution pattern recognition as crossmodal plasticity and training based on crossmodal mappings may improve performance and shorten trainingtimes.

3. Neural evidence regarding sensory substitution

Evidence from neuroimaging and transmagnetic stimulation (TMS) highlights the general point that the different senses do not operate in isolation, again reminding us of Aristotle’s considerations concerning multimodal representations of the environment. Furthermore, recruitment of unused visual cortex can help with compensation for the loss of vision (Pasqualotto & Proulx, 2012; Merabet & Pascual-Leone, 2010). The main point that has emerged from this research is that brain regions often considered ‘visual’, or ‘auditory’ should not necessarily always be considered as such, but that they may serve supramodal functions (Ricciardi, Handjaras, & Pietrini, 2014; Ricciardi, Bonino, Pellegrini & Pietrini, 2014; see Proulx et al., 2016 for review).

Tactile stimulation can cause activity in neural mechanisms that are devoted to other senses. Poirier et al. (2007; see also Renier et al., 2005) show how tactile stimulation can cause activity change in visual areas. Blind participants show activation of primary and secondary visual cortex from tactile stimulation during Braille reading (Sadato et al., 1996). Ortiz et al. (2011) argue that the magnitude of activity in visual cortex with tactile stimulation correlates with observers’ experiences of illusory flashing lights (phosphenes). Interestingly, Striem-Amit et al. (2015) show that even without visual input functional organization of visual cortex follows retinotopic organizational principles (see also Wang et al., 2015).

In Amedi et al. (2001, 2002) the lateral occipital complex (LOC), an area involved in visual object perception, was also activated during haptic object perception in sighted individuals This was not the case for participants who simply learned associations between objects and soundscapes. Amedi et al. (2010) found that tactile object identification activated visual object-recognition regions within LOC in blind individuals similarly as for sighted controls. Furthermore, Reich et al. (2011) found that the so-called visual word form area in the fusiform gyrus (VWFA; Cohen et al., 2000; Sigurdardottir et al., 2015) was activated during Braille reading. Activity has been found in the so-called Fusiform Face Area (FFA; Kanwisher, McDermott & Chun, 1997) for haptic face recognition in both blind (Goyal, Hansen & Blakemore, 2006) and sighted participants (Kilgour et al., 2004). The FFA has been thought to be primarily visual, but representations of the environment probably involve integration of information from other senses, and activity in ‘visual’ areas through stimulation of other modalities may therefore be expected. Occipital visual regions that are heavily involved with judging orientation are also activated by discrimination of tactile orientation applied to the finger (Sathian et al., 1997; de Volder et al., 1999; Kupers et al., 2010). Similarly, Stevens and Weaver (2009) found that congenitally blind participants showed alterations in auditory cortex consistent with more efficient processing of auditory stimuli. Also of interest is hippocampal activity during tactile maze solving in congenitally blind individuals (Gagnon et al., 2012), since such activity is typically associated with visual information processing. Furthermore, using visual-to-music sensory substitution, Abboud et al. (2015) show that when congenitally blind participants manipulate symbols as if they were numbers, there is activity in the right inferior temporal gyrus, a region involved in visual number processing. Touch can even modulate visual imagery (Lacey et al., 2010).

Sound source discrimination, auditory motion discrimination, auditory change detection and sound localization all result in occipital activations in blind subjects (Kujala et al., 2005; Poirier et al., 2006; Voss et al., 2008; Weeks et al., 2000). This has been shown to occur early in development (by 4 years; Bedny, Richardson & Saxe, 2015). An intriguing possibility is that precise localization provided by retinotopic visual cortex may allow more accurate auditory (or haptic) localization if visual areas are recruited by audition (Striem-Amit et al., 2015; Wang et al., 2015).

Kupers et al. (2010) measured neuronal activity as blind participants used an SSD involving stimulation of the tongue finding that the parahippocampus and visual cortex were highly activated. These areas play a fundamental role in visual spatial representation and topographical learning in sighted individuals performing comparable tasks. The visual and the parietal cortices are also activated during Braille reading (Büchel et al., 1998) in the congenitally blind, together with the parietal association areas, the superior visual association cortex and the cerebellum. For late-blind individuals, unlike the congenitally blind, the primary visual cortex and the right medial occipital gyrus were activated. The results for both groups show plastic crossmodal responses in extrastriate cortex. Additionally, tactile motion discrimination activates dorsal visual stream areas in the congenitally blind (Ptito et al., 2009).

Ptito et al. (2005) also studied cross-modal plasticity in congenitally blind participants receiving information about the environment through tongue stimulation. They found that cross-modal plasticity developed rapidly and that the occipital cortex was involved in tactile discrimination. De Volder et al. (1999) conveyed feedback through an ultrasonic echolocation device finding high metabolism rates in occipital visual areas. Also, echolocation has been found to lead to activity in higher order visual areas, for various tasks such as material processing (Milne et al., 2015), and motion (Thaler et al., 2014). Arno et al. (2001), found that visual to auditory encoding of camera-based images resulted in occipital activation in blind subjects. Calvert et al. (1999) also found that the auditory cortex of normal observers was activated as they attempted, in silence, to read speakers’ lips. Importantly, active interaction with auditory sensory substitution stimuli seems to cause more neural activity in the visual cortex than passive processing (Murphy et al., 2016).

Disrupting function with transmagnetic stimulation (TMS) can answer questions regarding neural mechanisms involved in sensory substitution. Zangaladze et al. (1999) found that TMS to regions typically considered primarily visual disrupted tactile discrimination. Additionally, neural disruption of visual cortex with TMS interferes with Braille reading (Cohen et al., 1997; Kupers et al., 2007). Braille reading has also been found to be impaired following lesions to occipital areas (Hamilton et al., 2000). Moreover, TMS of occipital visual areas can induce tactile sensations in Braille readers, similarly to visual phosphenes in the sighted (Cowey & Walsh, 2000; Penfield & Boldrey, 1937; Tehovnik et al.,2005).

Studies of sensory deprivation in animals are also of interest in this context. Visual deprivation in kittens from birth sharpens auditory tuning (Rauschecker, 1995). Such processes may underlie sensory substitution in humans as unused cortex may be taken over by other functions (Blakemore & Cooper, 1970; Wiesel, 1982). Such evidence suggests that the loss of one perceptual channel can be compensated for by higher resolution activity in another (Gougoux et al., 2004; Stevens & Weaver, 2009). It is controversial, however, whether blind individuals have better tactile discrimination acuity than the sighted, and this may depend on task. For example, Alary et al. (2009) found no difference in tactile grating orientation discrimination or vibration frequency judgment between blind and sighted participants, while blind participants outperformed the sighted on a texture discrimination task. In Alary et al. (2008), the blind outperformed the sighted on haptic-angle discrimination. Alary et al. (2009) speculate that this may be due to practice (e.g. with Braille reading). Wan et al. (2010) tested congenitally, early and late-blind participants finding enhanced auditory capacities only for those who became blind early. Participants who became blind at 14 years or later showed no benefits over sighted participants. Finally, we note that certain “visual” functions are heavily influenced by at what time the visual deprivation occurs and by the date of sight restoration (Collignon et al., 2015; McKyton et al., 2015). Such findings have strong implications for sensory substitution.

4. Research on similarities and differences between perceptual processing in different modalities

In an elegant experiment, Gibson (1962) demonstrated the difference between passive haptic perception and haptic perception of a moving stimulus. When a cookie-cutter was pushed onto participants’ palms but remained otherwise stationary, identification rates of its pattern were just under 50% percent, while if the cutter was pushed around in the observers’ palm, recognition became about 95% correct (see also Novich & Eagleman, 2015). This cleverly demonstrated how perception is a continuous process and does not involve a snapshot of the environment. Design of sensory substitution devices therefore requires assessment of the nature ofspatiotemporal continuity for different senses. There is no reason to think that this issue of continuity differs fundamentally between the senses (see e.g. Gallace & Spence, 2008), although particular details maydiffer. In fact, as argued above, our representation of the environment is multimodal (see e.g. Driver & Spence, 2000), raising the question of the relative roles of unimodal versus multimodal processing. The literature on multimodal interactions (see e.g. Deroy, Chen & Spence, 2014; Driver & Spence, 2000; Koelewijn, Bronkhorst & Theeuwes, 2010; Kristjánsson, Thorvaldsson, & Kristjánsson, 2014; Stein & Meredith, 1993) makes clear that a holistic picture of perceptual representation cannot be constructed without understanding multisensory interactions. Neural activity related to multisensory interactions can be seen surprisingly close to cortical areas typically considered unimodal (Driver & Noesselt, 2008). In fact, studies of multisensory interplay make the case that representations are inherently multimodal, if they involve input from more than one sense in the first place. In the current context, this raises the very interesting question of the nature of such representations for someone who lacks a sense.

Humans use various forms of depth information for navigation. The sensory channels differ both in bandwidth and in what information they most effectively transmit. The conveyed information should therefore be chosen in light of the specific aims of a device. Taking the case of sensory substitution for the blind, it is probably infeasible to convey the rich detail of visual experience.

Can the resolution of e.g. hearing as compared to vision be estimated? In a pioneering study, Kellogg (1962) found that blind particpants could use echolocation to detect size differences (see also Ammons, Worchel & Dallenbach, 1953). These abilities vary with practice, however. Rice, Feinstein and Schusterman (1965) tested the abilities of blind observers to echolocate metal disks of various sizes at varied distances. Performance varied considerably but the average detectable size of the disk was 4.6 cm at 67 cm distance. Teng and Whitney (2011) found that sighted novices quickly learned to discriminate size and position through echolocation with notable precision (see also Kolarik et al., 2014). In Lakatos (1991) participants were able to recognize letters traced out with specific auditory patterns in a speaker array, indicative of considerable localization abilities. Cronly-Dillon, Persaud and Blore (2000) trained observers to recognize sounds through tonotopic representation (Väljamäe & Kleiner, 2006) finding that observers were able to recognize basic shapes such as buildings and cars. Other psychophysical research shows that humans can recover physical properties from objects causing sound themselves, such as dimensions (Carello, Anderson, & Kunkler-Peck, 1998; Lakatos, McAdams, & Caussá, 1997; Houben, Kohlrausch, & Hermes, 2004), shape (Kunkler-Peck & Turvey, 2000; Grassi, 2005), or material (Klatzky, Pai, & Krotkov, 2000; Giordano & McAdams, 2006). These results show that, in principle, discrimination routinely performed through vision can be performed with audition up to a certain, undeniably limited, degree, particularly following training.

What does research tell us about how to convey information from vision into touch? Apkarian-Stielau and Loomis (1975) presented alphabetic letters haptically and visually, finding that recognition across modalities became similar only when the visual presentation was considerably blurred, as if a low-pass filter were applied to the tactile stimuli. In another study, Ernst and Banks (2002) observed that while both haptics and vision conveyed accurate information about the height of a bar this was much less reliable for touch than vision. Brown et al. (2014) make the case that there may be upper resolution limits for sonification of visual information. But they also emphasize that effective resolution can be increased through patterns of stimulation (Gibson, 1962; Loomis et al., 2012; Novich & Eagleman, 2015).

Spatial localization accuracy is at least an order of magnitude poorer for haptics than for vision (Apkarian-Stielau & Loomis, 1975; Loomis & Apkarian-Stielau, 1976). Two point thresholds (the ability to discriminate two taps), vary across different parts of the body from ∼3-4 mm (on the fingers) to over 40 mm on the thighs and calves (Weinstein, 1968). This may set an upper limit for SSDs that use haptics, and may furthermore only apply to low-noise implementations. Sampaio, Maris and Bach-y-Rita (2001) found that acuity, measured with a haptic rendering of a Snellen visual acuity chart with a tongue-based SSD was 20/860 on average but doubled after 9 h of training. Interestingly, their participants reported that they located the objects in space rather than at the stimulation site, an example of externalization.

In sum, what these studies highlight is that there are large differences in bandwidth between the senses. As it is only feasible to convey a limited amount of information, this needs to be taken into account during SSD design and at least during initial design the focus should be on what is needed for the intended task (see also Elli et al., 2014). And again, the findings highlight the importance of psychophysical experimentation for insights regarding sensorysubstitution.

5. Sensory substitution devices for vision through haptics

Using haptics to convey information to the blind about the external unreachable environment is strongly constrained by the intricacies of haptic processing. Important considerations follow from any attempt to convey information that the system has not evolved to process. With such devices, information is applied through stimulation that is projected to some surface of the body, often in an isomorphic way, such that a particular location on the body corresponds to a particular location in space, and the closest stimulation sites correspond to locations close to the projected one.

In a pioneering study, Bach-Y-Rita et al. (1969) developed a chair that conveyed tactile stimulation based on a 20 by 20 stimulation grid applied to participants’ backs. The conversion algorithm translated the pixel brightness into tactile stimulation. Following 5 to 15 hours of training, participants were able to recognize objects and experience them as if an image appeared in front of them (White et al., 1970). Some highly trained observers were able to report remarkable detail, such as whether people wore glasses and even peoples identity. This was an important proof of concept, although the use in practical settings may be limited, since the device was immobile.

The acuity of the sense with the lower limit places an upper limit on what can be conveyed in sensory substitution. Research on tactile sensitivity would, at first glance, seem to indicate that the hands, and especially the palms, would be ideal since spatial acuity is high. But for usability, the hands should be free for other purposes and the haptic stimulation applied to passive parts of the body. Koo et al. (2008) developed a vibration device for the fingertips (see also Shah et al., 2006) but such methods may fail the usability test (Dakopolous & Bourbakis, 2010). Handheld devices have been developed (Hugues et al., 2015), but an optimal system should free the hands for other uses, especially if increased mobility and use in daily functioning is the goal. There is a challenging tradeoff here, since it is logical to haptically stimulate the most sensitive parts of the body.

The way that information about the environment is encoded through haptics must be chosen with care, in other words, and stimulating areas that will be the least impeding should be preferred. One promising way involves conveying vibration through a belt to the participants based on information from video cameras (McDaniel et al., 2008). Van Erp et al. (2005) used a haptic belt, coding distance with vibration rhythm while direction was translated into vibration location. While cueing direction by location worked well, conveying distance in this way was not successful (for other implementations see Cosgun, Sisbot & Christensen, 2014; Johnson & Higgins, 2006; Segond et al., 2013).

Devices have been developed where haptic stimulation is applied to the tongue, because of its sensitivity. The Tongue Display Unit (TDU; Bach-Y-Rita et al., 1998; see also Chebat et al., 2011; Tang & Beebe, 2006) is a tactile-to-vision SSD that translates images into tongue stimulation. Kupers et al. (2010; see also Nau et al., 2015) trained and tested the virtual navigational skills of ten congenitally blind and ten blindfolded sighted controls in an active (route navigation) and passive (route recognition) task using the TDU. By the end of each training day, participants were asked to draw the routes in order to evaluate learning. Following training, participants showed considerable knowledge of the routes that they navigated.

6. Sensory substitution devices for vision though audition

Many attempts have been made at conveying information about the environment to the blind through auditory stimulation. While echolocation approaches are one option, yielding promising results (e.g. Ifukube, Sasaki, & Peng, 1991; Kolarik et al., 2014), the sound emission that is required makes this option in many ways impractical. Noise pollution would likely prevent many blind people from using the device. Camera-based auditory feedback devices may therefore be more practical. We note that promising GPS-based sensory substitution devices exist (Loomis et al., 1998, 2005; Marston et al., 2006, 2007; van Erp et al., 2005) but fall beyond the current scope.

One important consideration for auditory sensory substitution is whether to convey information with sounds or speech. Loomis, Golledge and Klatzky (1998) showed how virtual sounds resulted in better navigation performance than verbal commands. Väljamäe and Kleiner (2006) suggest that one practical way of conveying information may be to use a sequential way of generating sounds that correspond to another sequential dimension. Brightness could be denoted with loudness, size with pitch, and direction with timbre, and so on (a loud high pitched stimulus of a certain timbre would indicate a large bright object, to the right etc.). The vOICe sonic imaging system (Meijer, 1992; OIC: “Oh, I see”), transforms elements of a digital image of the environment into a sound pattern. The vertical position of a particular pixel is mapped to frequency and the brightness of an element in the scene to sound intensity. A recent study (Haigh et al., 2013) indicates that the accuracy of visual-to-auditory sensory substitution (with the vOICe) is promising.

Striem-Amit et al. (2012a) designed a visual-to-auditory version of the Snellen visual acuity E-chart, testing it on eight congenitally and one early-onset blind subjects who received training with the vOICe system for 73 hours, in weekly training sessions. The training program involved 2 strategies: 2-dimensional training, where participants were taught how to visualize static images through sound (geometrical shapes, digital numbers, objects, faces) and live-view training aimed at providing visual depth perception through a mobile version of the SSD. During testing, observers were presented with E’s of different sizes and orientations. The sound was played until participants indicated the direction of the sound by keypress. Participants performed better than in experiments with tactile SSDs (Chebat et al., 2007, Sampaio et al., 2001), retinal prostheses (Zrenner et al., 2010) and 55% showed acuity higher than the threshold for blindness (see also Levy-Tzedek, Riemer, & Amedi, 2014).

Ward and Meijer (2010) found that two blind users of the vOICe reported visual-like perception (externalization) acquired over several months of use. At first, both users reported that visual details provided by the device were weak, flat and monochrome. Later, following training for several months or even years, they experienced smooth movement, and depth, although the SSD did not convey this type of information. One participant even reported seeing color, but this probably reflected the participants’ memory of color (the participant became blind at 21). Ward and Meijer argued that even though the vOICe system enabled visualization through training, users’ experiences depend on previous visual experience. This may be due to cross-modal plasticity and experience-driven multi-sensory associations, reflecting an interaction between auditory mapping triggered by use of the SSD and previous sight experience.

Auvrey, Hanneton and O’Regan (2007) found that following 15 hours of training with the vOICe, participants could identify objects on a table, including new objects (see Proulx et al., 2008 for similar conclusions), while Kim and Zatorre (2008) found that their participants were able to associate objects and soundscapes without knowing the corresponding matching algorithm, suggesting that participants learned abstract matching rules.

Amedi et al. (2007) demonstrated that a congenitally blind and a late-blind subject could identify objects by recognizing their shape from visual-to-auditory encodings provided by the vOICe. Moreover, the lateral-occipital tactile visual area (LOtv) was activated only for participants who received extensive training, but not for those who simply learned associations between objects and soundscapes. Striem-Amit et al. (2012b) trained eight congenitally blind and seven sighted control observers on object-identification and classification on the vOICe. The Visual Form Word Area (VWFA; Cohen et al., 2000; Sigurdardottir et al., 2015) was activated in the blind when they processed letter soundscapes, following a 2 h training session (see also Reich et al., 2011).

The PSVA sensory substitution device (Arno et al., 2001; Capelle et al., 1998), involves images captured by a miniature head-mounted TV camera that are simplified at high-resolution centrally but lower resolution peripherally (analogous to the human retina). Each pixel is associated with a particular sound frequency, from left to right and from top to bottom. Arno et al. (2001) studied activations of neural structures in early blind and sighted controls when the PSVA was used for pattern recognition. The visual cortex of the early blind was more activated during pattern recognition both before and after training, where participants learned to move a pen on a graphic tablet to scan patterns presented on a monitor, than during two auditory processing tasks: a detection task with noise stimuli and a detection task with familiar sounds.

In the image-to-sound conversion algorithm of EyeMusic (Abboud et al., 2014) higher musical notes represent pixels located higher on the y-axis, the delay of sound presentation corresponds to the location of a pixel on the x-axis (a larger delay for the sounds from the right and a smaller one for the sounds from the left), while color is encoded by the timbre of musical instruments. In Levy-Tzedek et al. (2012), blindfolded participants were trained on EyeMusic in order to improve their ability to perform fast and accurate movements towards targets. After a 25-minute familiarization session and brief training, participants were able to reach the targets (using the SSD) with high accuracy and efficiency, similar to that achieved during a visually-assisted task (see also Levy-Tzedek, Riemer & Amedi, 2014). Finally, we note an interesting new avenue that involves “zooming-in” with the EyeMusic. Even users with no visual experience can integrate information across successive zooms (Buchs et al., 2015).

The EyeCane (Maidenbaum et al., 2014b) provides distance estimation and obstacle detection through narrow-beam infrared sensors. The user receives spatial information by sweeping the device across the visual scene. Distance is converted into pulsating sounds (delivered via headphones) and vibrations (perceived in the palm of the hand). In Chebat, Maidenbaum and Amedi (2015) fifty-six participants were divided into four groups: early blind, low vision and late blind, blindfolded and sighted. They performed navigation tasks in real and virtual environments. All participants improved their navigational and spatial perception skills, reflected in fewer errors and collisions in the maze as they learned to use EyeCane.

7. The possibility of combining haptics and audition for sensory substitution

Both haptics and audition show promise for use in sensory substitution. If the aim is to convey qualities provided by vision for sighted people, the effective resolution of both these perceptual channels is a drawback, however (see discussion above). Although this presents many challenges, a feedback system that codes the environment through haptics and audition in conjunction may compensate for the drawbacks of using only each individual sense if strengths are strategically utilized and weaknesses avoided (Shull & Damian, 2015). We may, for example, speculate that tactile information may convey direction accurately (Van Erp et al., 2005), while audition may be more conducive to conveying information about the nature of the stimuli, and their distance. A sensory device that uses both channels could strategically utilize strengths and weaknesses.

There is now good evidence that brain regions considered primarily devoted to a particular sense also respond to stimulation from other senses (Proulx et al., 2016; Ricciardi et al., 2014a; see section 3). The common sensible (using Aristotle’s terminology), involves integrated input from all modalities, and this can be utilized in SSD design. While evidence of multi-modal processing (Driver & Spence, 2000) may present challenges for sensory substitution, the converse might also be argued: this may bode well for any sensory substitution device – and may in many ways be a prerequisite for a successful device, as long as these principles are considered during design. For example, evidence of multisensory perceptual learning (Proulx et al., 2014) is encouraging for the idea of multisensory sensory substitution. But it is also important to note some caveats.

Multisensory integration can in some cases be worse for those with impaired senses. Gori et al. (2014) tested the ability of blind participants to judge the relative position of a sound source in a sequence of three spatially separated sounds. They found impaired auditory localization in congenitally blind participants (∼4 times the normal threshold). What this may mean is that visual experience is important for the development of a representation of the environment even for another sense, consistent with the idea of crossmodal calibration (Gori et al., 2010; Occelli et al., 2013). Crossmodal connections might strengthen unimodal representations. Also, some aspects of haptic perception may actually be worse in the blind, which has implications for the use of haptic feedback. Vision may be necessary for successful cross-calibration (Hötting & Röder, 2009; Occelli et al., 2013; Wallace et al., 2004). On the other hand, the early blind outperform the sighted in pitch discrimination (Gougoux et al., 2004) and also show better temporal resolution for auditory stimuli (Muchnick et al., 1991; Stevens & Weaver, 2009). Occelli et al. (2013) highlight that loss of vision may affect the establishment of peripersonal spatial coordinates, that then affects haptic and auditory processing. This raises the crucial question of how well we actually understand representation of space in the blind, reminding us of the need for basic psychophysical research with the aim of understanding perceptual processing for SSD design and that principles discovered in one scenario may not apply to others. The necessity of experience for calibration, and the evidence of considerable neural reorganization (such as following late-onset blindness), presents important challenges for sensory substitution (see discussion in Pasqualotto & Proulx,2012).

8. Conclusions

We highlight the following main points: regarding the design of sensory subtitution devices (SSDs).

• 1.Conveying unnecessary information about the environment risks sensory overload and limited attentional capacity raises challenges regarding SSD design. In many cases only critical information should be conveyed.
• 2.Differences in bandwidth between sensory systems severely constrain the nature and amount of information that can be conveyed. SSDs should therefore be task-focused.
• 3.Basic psychophysical research into representations of the environment and the most effective ways of conveying information will lead to better design of SSDs.
• 4.SSDs must not interfere with other perceptual function.
• 5.Strong evidence for multisensory integration suggests that design should not necessarily be confined to one feedback modality.
• 6.Active training appears to be crucial, especially since following training, proximal stimulation is often attributed to exterior objects. Such externalization may be an important test of the efficacy of SSDs.
• 7.Perception is a continuous process and does not involve a snapshot of the environment. Design of sensory substitution devices therefore requires assessment of the nature of spatiotemporal continuity for different senses.