Use of sensory substitution devices as a model system for investigating cross-modal neuroplasticity in humans

Blindness provides an unparalleled opportunity to study plasticity of the nervous system in humans. Seminal work in this area examined the often dramatic modifications to the visual cortex that result when visual input is completely absent from birth or very early in life (Kupers and Ptito, 2014). More recent studies explored what happens to the visual pathways in the context of acquired blindness. This is particularly relevant as the majority of diseases that cause vision loss occur in the elderly. Our lab and others have demonstrated compromised visual pathway integrity in those with peri-natal and acquired blindness (Schoth et al., 2006; Chan et al., 2012; Li et al., 2013; Lee et al., 2014; Dietrich et al., 2015; Ho et al., 2015; Reislev et al., 2015). Additional studies have begun to examine the changes occurring with certain disease states: patients suffering from retinitis pigmentosa, optic neuritis, and glaucoma, all so far demonstrate deterioration of the white matter tract architecture as a function of disease severity (Garaci et al., 2009; Gabilondo et al., 2014; Ohno et al., 2015). This evidence indicates that the visual system as a whole is profoundly susceptible to degeneration even with small amounts of vision loss. On the surface, these investigations appear to have negative implications for vision restoration efforts. Yet, parallel studies which examine the phenomenon of cross-modal plasticity suggest that a remodeling of the central nervous system is possible, such that areas of the brain which have been deprived of normal afferent input are able to reconstitute themselves to be receptive to alternative sensory channels (Merabet and Pascual-Leone, 2010; Kupers and Ptito, 2014). The literature includes several examples of investigations which show that the visual cortex will react to tactile and auditory stimuli in the blind but will be less readily recruited in sighted patients (Merabet and Pascual-Leone, 2010). Moreover, cross-modal interactions have been demonstrated well beyond the traditional “critical period” and into late adulthood, albeit perhaps in a less robust fashion (Sadato et al., 2002; Bedny et al., 2012; Collignon et al., 2013). The notion that the adult brain is still capable of significant structural and functional remodeling after vision loss provides opportunities to restore vision through mechanical or biological means.

Sensory substitution is a non-invasive method for restoring a sense of the environment. The first sensory substitution device was the white cane which is still widely used by the blind community. Braille is another example of tactile sensory substitution, and software programs such as JAWS substitute are auditory based. More modern attempts at sensory substitution aim to translate visual, camera-based stimuli into a non-visual tactile or auditory stimuli. Sensory substitution in this context was described by Bach-y-Rita in the 1960's (Bach-y-Rita et al., 1969) and has gained traction in the last few years as a potentially legitimate means of providing functionality to the blind that is “beyond the reach of the cane’. The most recent devices continue to exploit tactile and auditory afferent channels. The BrainPort (Wicab, Inc.) and the AuxDeco (EyePlus-Plus, Inc.) are tactile based sensory substitution devices using the tongue and forehead, respectively. The vOICe (Metamodal, Inc.), and the spinoff software program known as EyeMusic (Abboud et al., 2014) both translate visual stimuli into soundscapes. The EyeCane (Maidenbaum et al., 2014b) attempts to improve the functionality of the white cane to include positional information. Sensory substitution devices can be relatively difficult to master and the current resolution provided may remain relatively limited. Nevertheless, sensory substitution devices have been shown to result in improvements over baseline in a number of outcomes metrics (Lee et al., 2014; Maidenbaum et al., 2014a; Nau et al., 2015). Researchers are working on improving this nascent technology to be more user-friendly as well as developing the rehabilitation protocols necessary to properly retrain the brain to accept more complex alternative sensory input.

Beyond the potential for improving the mobility and independence for the blind, sensory substitution devices also provide an excellent tool to study cross-modal plasticity of the visual system in living humans. The primary advantage is their relatively inexpensive and non-invasive nature, which allows for large numbers of subjects with different etiologies and durations of blindness to be followed. Modern sensory substitution devices are trying to enable activities of daily living such as object recognition, navigation beyond the reach of the white cane and non-text sign identification (Lee et al., 2014; Maidenbaum et al., 2014a; Nau et al., 2015). These attributes are starting to allow researchers to study the cross-modal interactions of the brain in tasks that can more accurately represent daily activities than early studies which were relegated to testing very rudimentary stimuli. Moreover, sensory substitution devices exploit multiple afferent streams including tactile (hand, tongue, back) and auditory channels. This unique attribute can provide information about normal interactions between sensory subsystems and how they can be remodeled in the setting of blindness. In addition, it is possible to compare plasticity in the same subject over time, or compare training effects between age groups, disease severity and etiology of vision loss. It is possible that the way a brain reacts to using a sensory substitution device could be a biomarker for success with other, more invasive vision restoration technologies such as retinal implants (Sadato et al., 2002). Using sensory substitution devices as a model system for studying the mechanisms and effects of cross-modal interactions can improve our understanding of the plasticity of the nervous system that may be generalizable to conditions other than blindness.

In the past few years, positron emission tomography (PET) and functional magnetic resonance imaging (fMRI) have been used in combination with sensory substitution devices to investigate the mechanisms of cross-modal neuroplasticity in the blind. Early PET studies by our group and others showed that the visual cortex is metabolically active in the blind during sensory substitution tasks (Lee et al., 2014), and that the strength of this activity may be dependent on task difficulty, the duration of blindness, or the etiology of visual impairment (Kawashima et al., 1995; Gougoux et al., 2005). However, PET offers relatively low spatial resolution and is expensive to administer. In addition, PET studies require injection of radioactive isotopes, and limited tasks are allowed in any given experimental session. In contrast, fMRI, especially blood-oxygenation-level-dependent (BOLD) fMRI utilizes the intrinsic properties of paramagnetic deoxyhemoglobin in the brain and the hemodynamic response triggered by neuronal activities for non-invasive functional imaging. It provides a cheaper and safer alternative neuroimaging tool that allows multiple or repeated tasks for examining brain activities in the same subject under different experimental conditions in the same scanning session and across time. Our recent fMRI studies using the BrainPort and The vOICe helped reveal the nature of brain responses under different types of sensory substitution tasks.

As shown in Figure 1, when a typical checkerboard visual stimulus was presented to the eyes of normally sighted subjects, a positive BOLD response to the visual cortex was observed. This occurred primarily because of a corresponding increase in brain activity and oxygen-rich blood contents entering the visual brain region. Similarly, when BrainPort (tactile-vision) and The vOICe (sound-vision) sensory substitution tasks were administered to the sighted subjects and both peri-natally blind and acquired blind subjects in the fMRI scanner, positive BOLD responses were found in the somatosensory cortex and auditory cortex, respectively in both sighted and blind subject groups. This was expected because the somatosensory cortex mediates tactile stimuli on the tongue, and the auditory cortex mediates sound information. In the visual cortex, a negative BOLD response was found in the sighted subjects with the use of either BrainPort or The vOICe, suggestive of relatively decreased blood flow to this area. The negative BOLD response in the visual cortex of sighted subjects was likely due to cross-modal inhibitions which naturally occur because the visual part of the brain in sighted subjects is not necessarily required for interpreting non-visual sensory stimuli (Kawashima et al., 1995; Laurienti et al., 2002; Hairston et al., 2008). In contrast, a positive BOLD response was found in the visual cortex of both peri-natal and acquired blind subjects with both BrainPort and The vOICe input, suggesting that this brain region, deprived of afferent visual input because of blindness, is being recruited to assist with the interpretation of both tactile and auditory information. The different BOLD responses exhibited between blind and sighted subjects in the visual cortex demonstrated direct evidence of functional brain reorganization as a function of blindness. Not only were the brain responses different between blind and sighted subjects, but the brains of the same blind subjects were also able to secondarily activate the visual cortex from two completely different primary alternative senses (touch and sound). This finding supports the flexibility of the visual brain to adapt to multi-sensory cross-modal inputs, at least in the context of visual deprivation.

Figure 1.

BOLD functional MRI of multimodal sensory substitution.

Task-based BOLD functional MRI (BOLD-fMRI) of the brain during (a) direct visual input to the eyes of sighted subjects via 8 Hz checkerboard visual presentation, and (b) visual sensory substitution to the tongue (left column) and ears (right column) of the same sighted and blind subjects via BrainPort tactile-vision (left column) and The vOICe sound-vision tasks (right column). BOLD percent change represents the amount of brain BOLD signal changes during task periods relative to resting periods. Note the positive BOLD responses in the visual cortex of sighted subjects in (a), and in the somatosensory cortex (left column) and auditory cortex (right column) of both sighted and blind subjects in (b) reflective of the subjects’ perception of vision, touch and auditory sensations respectively. In (b), the visual cortex of sighted subjects showed negative BOLD responses upon both tactile and auditory sensory substitution tasks, suggestive of cross-modal deactivations or inhibitions, whereas the visual cortex of blind subjects showed positive BOLD responses to both tactile and auditory vision substitution tasks similar to sighted subjects’ brain reactions to visual presentations in (a), indicative of the flexibility of the visual cortex to multi-sensory cross-modal plasticity upon visual deprivation. BOLD: Blood-oxygenation-level-dependent; MRI: magnetic resonance imaging.

To evaluate the possible causes of visual cortex activation with sensory substitution, a separate experiment was performed to test the minimum duration of training needed to increase activity of the visual cortex in the blind. Our initial findings suggested that this can occur shortly. In naive subjects with no prior exposure to The vOICe, active interpretation of the vOICe stimulus after only 10 minutes of rudimentary training increased the BOLD signal in the visual cortex of the blind subjects compared to passive presentation of the same stimulus at baseline (Murphy et al., 2014). Notably, sighted subjects did not show apparent changes in BOLD activation in any visual cortical regions before and after the short training session.

In addition to active task-based fMRI, fMRI technology has the capability to evaluate functional connectivity between brain regions when the subjects are at rest. Our initial findings suggested that in the passive, resting state when the subjects were not instructed to perform any tasks, functional connectivity of the visual cortex becomes weaker within sensory networks but stronger in those brain regions responsible for higher-order cognitive functions such as task-positive networks, where activity increases during cognitive tasks, and salience networks (Chan et al., 2014a). These findings support the recent model that the visual brain is not limited to a single sensory modality but rather is highly plastic and task-flexible (Reich et al., 2012). We speculate that an adaptive brain reorganization which shifted the visual cortex from a predominantly sensory network to that of a higher-order, top-down task-positive cognitive network was already extant in the blind group before task-based fMRI experiments, and that this adaptation enabled BOLD activation of the visual cortex during our experiments.

In addition to functional imaging, magnetic resonance enables multiparametric assessments of the metabolism and structure of the central nervous system in both humans and animal models. Future studies are envisioned that utilize magnetic resonance spectroscopy to determine the neurochemical changes in visually deprived brains such as the balance between excitatory and inhibitory neurotransmissions (Chow et al., 2011; Weaver et al., 2013; Wu et al., 2013). Advanced diffusion MRI techniques and MRI tracers can also be used to determine which neural pathways are altered in different types of blindness (Schoth et al., 2006; Chan et al., 2012, 2014b; Li et al., 2013; Lee et al., 2014; Dietrich et al., 2015; Ho et al., 2015; Reislev et al., 2015), and which pathways are predominantly responsible for cross-modal brain activation by probing training-induced white matter plastic changes during sensory substitution rehabilitation over time. This data set could be co-registered with fMRI to study the structure-function relationships between white matter integrity and brain activities as well as sensory substitution performance across different life fractions of blindness.

An interesting observation is that even though cross-modal plasticity is a well-established phenomenon in the blind, and that this effect is measurable using various methods of neuroimaging when the blind use sensory substitution devices, whether these neuronal changes confer superior abilities in the functional domain remains controversial. For example, in our laboratory, we have yet to find a correlation between increased activation of the visual cortex and performance improvements on many different types of outcomes measures (Lee et al., 2014). This finding is at odds with the generally accepted notion that the blind should be better able to discriminate auditory and tactile stimuli than their sighted counterparts. It is also inconsistent with experiments that suggest brain structure and function can predict performance with spatial hearing (Roder et al., 2002; Gougoux et al., 2005) and pitch discrimination (Voss and Zatorre, 2012; Voss et al., 2014). Future experiments with greater numbers of subjects followed for longer periods of time will be needed to shed light on this question, but are not likely to occur until sensory substitution devices are more widely available and are easier for subjects to use throughout the day.

In summary, sensory substitution devices for blindness should be considered a valuable method for studying plasticity of the central nervous system. In order for vision restoration efforts to move forward, a better understanding of the brain changes as a result of vision loss is urgently needed. Neuroimaging in combination with sensory substitution devices offers tremendous versatility to provide the answers needed when deciding on the appropriate candidates for vision restoration.

This work was supported by National Institutes of Health Contracts P30-EY008098 and T32-EY017271-06 (Bethesda, MD); United States Department of Defense DM090217 (Arlington, VA); Alcon Research Institute Young Investigator Grant (Fort Worth, TX); Eye and Ear Foundation (Pittsburgh, PA); Research to Prevent Blindness (New York, NY); Aging Institute Pilot Seed Grant, University of Pittsburgh (Pittsburgh, PA); and Postdoctoral Fellowship Program in Ocular Tissue Engineering and Regenerative Ophthalmology, Louis J. Fox Center for Vision Restoration, University of Pittsburgh and UPMC (Pittsburgh, PA). We thank all collaborators who contributed to our research papers upon which the present commentary is based.