Preparing Participants for the Use of the Tongue Visual Sensory Substitution Device

Abstract

Visual sensory substitution devices (SSDs) convey visual information to a blind person through another sensory modality. Using a visual SSD in various daily activities requires training prior to use the device independently. Yet, there is limited literature about procedures and outcomes of the training conducted for preparing users for the practical use of SSDs in daily activities. We trained 29 blind adults (9 with congenital and 20 with acquired blindness) in the use of a commercially available electro-tactile SSD, BrainPort. We describe a structured training protocol adapted from the previous studies, responses of participants, and present retrospective qualitative data on the progress of participants during the training. Information on the training would be of interest to other researchers in preparing studies, and eye care professionals, who may advise patients to use SSDs.

Introduction

Visual sensory substitution devices (SSDs) provide a non-invasive assistive technology for the blind/visually impaired. General-purpose SSDs aim at improving blind people’s functional performance in a variety of everyday tasks such as light detection, object recognition and mobility, and consequently enhancing independence of blind individuals [1,2].

SSDs convey visual information through other sensory modalities such as touch or audition. Understanding the information normally processed by one modality when presented in another may not be an intuitive and automatic task [3], but rather a learned skill like reading or language that requires training and practice [4,5] (but see Stiles and Shimojo [6] for results indicating that perception is intuitive, to some extent, with some auditory sensory substitution).

One example of a general-purpose SSD that requires extensive training prior to practical use of the device for daily activities is the BrainPort (Wicab Inc., Madison, WI), a commercially available SSD that conveys video camera-based visual information through electro-tactile stimulation on the tongue. The Wicab’s website [7] (and the user manual) states that a typical 10-hour supervised training is necessary before using the BrainPort independently. Studies conducted to evaluate the functional performance of the BrainPort also pointed out the necessity of training by a professional and following the training home self-practice by user in activities of daily living, and they described a structured training protocol which is carried out over a few 3-hour sessions and usually lasts for a total of 10-15 hours [e.g., 3,5,8]. The training protocol starts with familiarizing participants with the device, its components, purpose, and limitations, how it converts visual information to tactile stimulation on the tongue, and how to interpret the stimulation. The protocol involves several tasks with increasing levels of complexity, such as understanding the spatial relations between the stimulation on the tongue and surrounding visual environment, recognition of high-contrast basic shapes, letter recognition, and recognition of common wall signs such as a restroom sign.

Although participants were trained by following the structured training protocol in the aforementioned studies [3,5,8], details related to their individual performance during the training have not been reported. For example, Grant et al. [5] reported that 57 participants who completed their study were trained by an experienced BrainPort trainer prior to functional testing with the device. However, they did not report whether all participants progress similarly during the training or performed at an expected success level in all the steps included in the training protocol. Similarly, 100 participants were trained by Nau et al. [3] but functional testing was carried out only with 18 participants. They did not explain why more than 80 percent of participants that they had trained were not included in the following experiments. As a result, the following questions usually remained unanswered in these papers [3,5,8]: Are participants expected to reach a certain expertise level in order to consider that they completed the training’s objective of preparing users for daily activities, or is the completion of the training defined just by time (e.g. 10 hours)? How does performance vary among participants? Does it take a similar time for each participant to complete a particular task in the training? Are there any participants who were not able to complete all training tasks? Are there any predictive participant characteristics for better or poorer training outcome? Such details about the performance and the progress during the structured training applied for rehabilitation purposes might be useful for evaluating the potential of the adoption of SSDs by users.

There are also studies that train participants only on a single targeted task in order to evaluate the performance in that particular task, such as basic shape discrimination [9], or motion direction discrimination [10]. In these studies, training is usually tailored for the requirements of the study and the SSD is used, not as visual rehabilitation aid but a research tool, to answer specific questions, such as the spatial or temporal resolution of the tactile system [11,12]. In such studies, the progression in performance is usually reported during the training, and performance is usually expected to asymptote before the final evaluation is conducted. This type of training is outside the scope of the current paper.

Over 2 years, we trained 29 blind participants (20 acquired and 9 congenital blind) following a structured training protocol. We adapted the protocol described by Nau et al. [3], with modifications as described in the method section. Participants who completed the training participated in our studies where they were trained again especially for the targeted task tested in the study [e.g., 13]. Our aim was not to systematically investigating the performance in the training protocol [3,5,8], but merely to prepare our participants for the use of the device in various studies. However, during the training, we observed some strategies developed intuitively by participants and experienced difficulties in training some of our participants in some tasks. To the best of our knowledge, in addition to the unanswered questions mentioned above, such observations on the training procedure have not been reported in the literature before. In order to fill this gap, we retrospectively analyzed our training records and present here a qualitative summary of participants’ performance and reactions. We also provide a detailed description of the training protocol. We followed a structured training protocol with pre-defined stages. Yet, since the training was not the aim of the study, the exposure of each participant to certain tasks differed based on the progress and interest of the participant, and the acceptable level of performance at each task was decided by the trainer subjectively, matching what might happen in clinical practice. Therefore, no statistical comparisons were performed, and this paper should be considered as a retrospective observational report. Information in this paper should be of value to researchers planning future studies, to eye care professionals who may advise patients regarding the use of this device, and to clinicians who may train people in the use of the device for daily activities.

Method

Participants

Thirty functionally blind adults (10 with congenital and 20 with acquired blindness) with the visual ability of light perception, hand motion or worse bilaterally, from a variety of etiologies (see Table 1) were recruited. One congenitally blind participant was excluded without any training because of damage to the frontal lobe and inability to differentiate his right from his left, which is important to be able to use the BrainPort effectively. Exclusion criteria included cortical blindness from any cause, oral lesions or piercings, perceptual abnormalities in the tongue or skin based on personal report of participants, and low scores (below 23 out of 27) on the Mini Mental State Examination [14]. Participants (21 men) ranged in age from 31 to 86 (M=60, SD=12) years.

Table 1.

Demographic characteristics of participants

Etiology	Congenital	Acquired
	Number of participants	Number of participants	Onset before training (years)
Retinopathy of Prematurity	4	1	37
Diabetic Retinopathy		4	10, 13, 23, 39
Retinitis Pigmentosa	1	3	20, 34, 48
Leber's Optic Neuropathy		2	11, 12
Glaucoma		2	33, 43
Trauma		2	2, 13
Leber’s Congenital Amaurosis	1		-
Microphthalmia	1		-
Uveitis		1	45
Detached Retina and Band Keratopathy		1	53
Cone Dystrophy		1	15
Immune Deficiency		1	4
Retinal Blastoma		1	57
Optic Nerve Damage		1	6
Not Known	2		-
Total	9	20

Demographic information, cause of blindness and onset, and any abnormalities in the tactile system or any cortical damage were recorded based on personal reports. Training protocol adhered to the tenets of the Declaration of Helsinki, and the protocol and informed consent were approved by the Massachusetts Eye and Ear Institutional Review Board.

Apparatus and Stimuli

BrainPort

The BrainPort is an electro-tactile tongue SSD. It delivers the visual information acquired through a head-mounted camera to a 20 × 20 grid of electrodes on a 25.8 × 25.8 mm intra-oral device (IOD) placed on the tongue (for more information on the device see Nau et al. [3]). Three versions of the device, V100, V200, and Vision Pro were used during the training. The dimension and resolution of IODs are similar in all three versions. In BrainPort V100, the camera is located on sunglasses and there is a separate handheld controller unit, whereas in the other two versions both the camera and controller are placed on a headset. For the purpose of the training, these two devices are identical. The controller mainly enables the user to turn on and off the device, adjust stimulation intensity, and field of view (zoom), select different viewing modes (e.g., inverting light and dark options). In the default configuration, visual stimuli are captured by the camera and then sent to the IOD, which can be monitored by sighted observers on a computer display through a Wi-Fi interface.

Training Environment

Stimuli are presented on a black trifold cardboard and the center board (91 × 60 cm) was covered in black felt cloth. The trifold board was placed vertically open on a table that was covered in black, non-reflective fabric. This setup allowed for an all-black, non-cluttered background area for object viewing. Stimuli were cut from a white felt cloth that could be flexibly attached to the felt-covered trifold (Fig 1).

Figure 1.

BrainPort testing environment. (a) Black trifold cardboard covered centrally in black felt standing on a black-clothed table. Sample stimuli cut out of white felt (A, C, circle, and arrow) are shown on the black surface to be viewed with BrainPort. The monitoring screen is seen to the left. (b) The BrainPort monitoring view. The left window shows the camera field of view together with the yellow cropping box indicating the visual field of the tongue display (default 24° × 24°, controlled by the zoom). The right window illustrates the electrical stimulation sent to the IOD at the resolution of 20 × 20 pixels. Note that the default field of view presented on the IOD is much smaller than the felt-covered trifold (about 10% of the area).

Procedure

We adapted the first four levels of the training protocol described in Nau et al. [3] and created a nine-stage training protocol by adding new tasks (see Stages 2, 3, 4, and 6 below).

The pace through each training session was adjusted by the trainer based on individual participant’s progress and interest. Some participants wished to spend more time on some tasks although they were already performing satisfactorily at that stage. Others got bored and wanted to move on to other tasks although they were not yet successful at given stage. In the latter case, the trainer moved back to the previous incomplete task after spending some time on a new task.

Each session lasted 2 to 3 hours depending on each participant’s fatigue and interest. Trainer subjectively decided on the acceptable level of performance at each stage required to move forward. The purpose of the training was to advance participants’ usage of the device. The trainer prioritized keeping participants motivated and if the trainer noted that the participant was getting bored with one task, the trainer moved to the next stage after successfully completing fewer trials than were required and completed by other participants. Although some participants completed all the stages in the training protocol, they were asked to participate in more sessions to determine if the performance will improve further, such as recognizing letters or objects in a shorter time with more practice on already completed tasks.

The nine stages in the training protocol are described below.

1. Device Familiarization and Mapping of the Tongue Stimulation to the Visual/Spatial World

Participants were introduced to the operation of the device and were taught to use the controller; turning the device on and off and adjusting the intensity level. A white circle or square target (approximately 8° × 8°) was placed on the board centrally so that it would likely appear within the centre of IOD field of view (24° × 24°). Participants were asked to put the IOD onto their tongue and adjust the intensity to a comfortable level.

Next, while viewing the monitor, the trainer presented a handheld white Styrofoam bar (about 2.5 × 2.5 × 30 cm, approximately 2° × 2°× 24° but varies slightly based on where the trainer holds the bar) to different edges of the field of view of the IOD (Yellow frames in Fig. 1) stimulating one edge at a time. The trainer explained which side of the IOD/tongue stimulation corresponds to what part of the visual field (i.e., right to right, left to left, an object at the upper visual field stimulates the back of the tongue, and an object on the lower visual field stimulates the tip of the tongue). When participants could correctly locate the stimulation on the tongue, they were asked to hold the Styrofoam bar in their hand and practice stimulation of the four edges of the IOD visual field on their own. The main purpose of this task was to introduce participants to the IOD “field of view” and its relation to the real-world directions.

2. Static Bar Orientation

Trainer attached a white bar (approximately 2° × 30°) to the black vertical board at one of 4 different orientations (horizontal, vertical, left- or right-tilted) and participants were required to report the orientation of the bar. Tracing strategies with the head or tip of the tongue helped participants having difficulty in recognizing orientation. In head tracing, they moved the head-mounted camera and noted the direction of head movement necessary to keep the stimulation generated by the bar stimulus essentially in the same position on the IOD during the movement. In tip tracing, they traced the stimulation on the IOD with the tip of their tongue while keeping the head static. Some of the participants used the tracing strategies intuitively. For others having difficulty in this task, the trainer introduced to both strategies. All participants reported benefiting from using head or tip tracing.

3. Localization

Trainer attached a circle (about 8° diameter) at a random location on the board (about 85°×60°) and instructed the participants to locate it within the 24°×24° default field of the device using head scanning movements. The purpose of this task was to familiarize participants with the relationship between head movements and the external world (e.g., when they move their head towards left in relation to their body, they would detect a stimulus on their left). No specific scanning strategy was suggested. Participants typically used random movements in searching for the stimuli.

4. Counting Targets

Trainer placed multiple circles (2, 3, or 4) of different sizes at different locations on the board and asked participants to count the number of objects on the board. The purpose of this task was to strengthen the localization skill of Stage 3. Keeping track of each found object’s location strengthens their abilities to integrate head movement/spatial location relationship; without keeping track of each object’s spatial location, they might count the same object more than once. After the participant was able to count correctly the objects on the board, the trainer brought the objects closer together and encouraged the participant to differentiate the objects even when there was only a small gap between them. To make the task more interesting, sometimes the trainer asked participants to locate the biggest stimulus on the board.

5. Using Zoom-In and -Out

Trainer introduced participants to the zoom function of the system controller and explained to the participant that zooming-out is an efficient method for detection and localization (as it increases the field of view), while zooming-in is helpful for recognizing details of the visual stimulus (as it increases the apparent resolution). The trainer asked participants to locate a stimulus on the board by first zooming-out, head scanning to locate the stimulus, and then zoom-in to try to recognize the details of the stimulus. Participants were asked to locate and examine their own hand or trainer’s hand using the zoom functions. In the default zoom setting, the whole hand (at arm length) is visible within the field of view but the gaps between fingers cannot be appreciated on the IOD (see Fig. 2 for illustration). Participants were taught to zoom-in and explore the fingers and spaces between them at a higher resolution. Hand exploration task in the default zoom setting also helped participants to comprehend the field of view of the camera by moving the hand through the field and exploring the extent of stimulation on the IOD caused by their own hand from the arm-length distance.

Figure 2.

Use of the zoom function. On the left the camera view, with the yellow frame marking the portion of the image sent to the IOD. On the right is a simulation of the signal sent to the electrode. Note that the sensation on the tongue may not have the same resolution and certainly does not have the same number of grey levels. (a). Trainer’s hand seen within the default 24° field of view. The gaps between fingers may not be easily recognizable with the tongue. Participants report that this stimulation feels like a single large blob. (b). Zoomed-in onto 2 fingers makes it possible to distinguish the two fingers and the gaps between the fingers.

6. Discriminating the Direction of Tumbling E and Landolt C stimuli

To become more familiar with the zooming function, participants were trained to identify the direction of the opening in a tumbling E and Landolt C targets placed on the board in one of the four different directions (up, down, left, or right).

7. Recognizing Basic Shapes

Shapes cut out of bright velvet cloth such as triangle, square, circle, hexagon, diamond, arrow, and heart (spanning approximately 10°) were placed on the black board. Participants were asked to identify the shape by utilizing both zooming-in and tracing with the tip of the tongue or head and name it. During the pilot testing of the device and training of the first few participants, both “filled” and “outlined” shapes were used [15]. Our participants reported that recognizing outlined shapes was easier because the outline was easier to trace with the head or the tip of the tongue. Therefore, only the outlined shapes were used in most of the training sessions. Note that creating contours with image processing may also convert filled shapes to outlined shapes [16]. BrainPort provides two edge enhancement modes: In the first mode, image is converted to just edges. In the second mode, enhanced edges are enhanced and presented on top of the image. Neither of these was used in our training.

8. Recognizing Letters and Numbers

Participants (specifically congenitally blind participants) were first asked if they are familiar with the letters and numbers. If they were not confident, (capital) letters and numbers cut from cloth were put on the table one by one and they were asked to identify it by tracing with their fingers. All the participants were able to identify most of the letters and numbers. Only letters and numbers that participants could identify by touch were used for further training. Participants were asked to name letters and numbers (approximately 10° height) by utilizing zooming-in and tracing the IOD with the tip of the tongue or by using head tracing. For some participants, only a single letter at a time was presented on the board. For others who could easily recognize a single letter on the board, three letters, which would form a word such as DOG, were presented. For a few highly performing participants, ten letters were placed randomly on the board, and participants were asked to locate a target letter among the others.

9. Recognizing Real Objects on a table

An actual object (i.e., a mug with a handle, a paper coffee cup, a funnel, a hand sanitizer bottle) was placed on the table in front of the participant. Participants were asked to name the object using the BrainPort without prior knowledge of objects in their first exposure to each object. After they explored each object using the BrainPort, they could touch the object. Next, multiple objects (any combination of the four objects mentioned above) were placed on the table, and participants were asked to name the objects and verbally report the location of each object relative to others. Participants were also introduced to occlusion and perspective in this stage, as they were encouraged to look at the objects from different angles (multiple viewpoints) and different physical distances (perspective) while they were sitting on a chair or standing up. Seeing objects from multiple viewpoints did not provide much additional information for circular symmetric objects such as a paper coffee cup, but it did help to detect distinct properties of asymmetrical objects such as the handle of a mug. Also, in order to determine the size of the objects, we suggested to participants to use their thumb or index finger at arm’s length as a reference of size, but keeping the finger and an object within the same IOD field of view was difficult because of the occlusion from the finger.

Results and Discussion

Each participant attended different number of sessions (minimum 1, maximum 11 sessions; See Fig 3 for each participant’s progress in the first 5 sessions. Out of nine participants with congenital blindness, six (66%) reached the last stage in the training protocol (i.e., successfully recognized letters and numbers); 4 at the end of the 1^st session, 1 at the end of the 2^nd session and 1 at the end of the 3^rd session. Three out of the six were successful in recognizing the real objects satisfactorily. Some participants were tested on all letters that they are familiar with and some with just a few of these letters depending on time restrictions in the session and participant’s interest. Some of them were able to name the letter in the second or third attempt, and those were considered as a correct response. Two congenitally blind participants could not progress further than Stage 1 (one reached Stage 3 in the 1^st session but the performance was worse in the following sessions) even after 4 training sessions (approximately 12 hours of training and they were not trained further). One was only able to tell if stimulation is on or off at the end of the 1^st session (he was not trained further). Two out of nine participants attended only one training session lasted approximately 3 hours (One could only tell if the stimulus is on/off and the other completed Stage 8).

Figure 3.

Training stages completed in the first 5 training sessions for all 29 participants. The data shows that if participants can complete Stage 4 in the first session, they can complete all the stages (or reach the last Stage) in the training protocol within the first five sessions. Each symbol represents one participant. Participants who attended only one training session were indicated by grey circles. Participants who attended more than 5 sessions are indicated by three dots at the end of the fifth session.

Out of 20 participants with acquired blindness, 10 (50%) reached the last stage in the training protocol (5 at the end of the 1^st session, 5 at the end of the 2^nd session). Five out of the 10 (who reached the last stage) were successful in recognizing the real objects satisfactorily. Four out of 20 could not progress further than Stage 1 (i.e., spatial mapping; 1 at the end of the 4^th, 1 at the end of the 3^rd, and 1 at the end of the 1^st session. One reached Stage 4 at the 4^th session but performance dropped at the 5^th session). Five could progress only up to Stage 2 (N=2), Stage 3 (N=1), or Stage 4 (N=2) at the end of the 1^st session (they were not trained further). Nine out of 20 participants attended only 1 training session (see the grey circles in Fig. 3, right plot).

Out of 29 blind participants, 12 either dropped out or did not want to continue to participate in the training. Two dropped out of the study because they were not able to keep the device in the mouth for long durations without gagging. Two showed no progress beyond Stage 3 even after at least 4 sessions, and two more showed no progress beyond Stage 1 even after at least 3 sessions. Therefore, these four participants were not called back for further training. Six reported that they were not interested in participating in the training anymore because of disinterest in the device (4 at Stage 8, 1 at Stage 4 and 1 at Stage 2).

Performance in the first two sessions seems to be a good indicator of participants’ ability to progress in the training protocol. Participants who could complete at least the fourth stage within the 1^st session were at least able to recognize letters and numbers in the following sessions. Participants who were stuck at one of the first 4 stages in the first session did not show any further progress, for some of them, even after 4 training sessions. Therefore, the stage completed in the first session seems to be a good indicator of future progress. As can be seen in the Fig. 3, most (15 out of 16 participants) who can recognize at least letters and numbers reached this performance level within the first two sessions. Also, although some participants completed all stages, they were asked to attend more training sessions to see if further practice affects the performance at least in terms of recognition time. We haven’t noticed such an improvement.

Observations on the Training Stages

Device Familiarization and Spatial Mapping of the Tongue in Relation to the Visual World

During the spatial mapping task, most of the participants were not able to detect stimulation at the back of the tongue with the adjusted comfortable intensity level. They reported feeling the stimulation when stimulation is moved toward the tip of the tongue (lower visual field, the center of the IOD) which is consistent with reports in the literature about lower sensitivity at the back of the tongue [11,13,17,18]. Two participants were not able to feel stimulation on the right side of the tongue. Another participant was not able to feel stimulation on any part of the tongue except the tip. Yet, all three participants were able to advance in the training using tip tracing and completed the Stage 8.

It was difficult to explain the lower and upper visual field especially to participants with congenital blindness. Therefore, we tried to give them real life examples to clarify the spatial configurations in a visual environment and its representation on the IOD by the device. For instance, we told participants that if they are looking at a person, the head is in the upper visual field, and the feet are in the lower visual field. Hence, the head should be presented on the back of the tongue (on the upper rows of the IOD) and feet should be presented on the tip of the tongue (on the lower rows of the IOD closer to the cable). Few participants stated that the spatial configuration of the device was not intuitive for them. They thought that it would make more sense for them to present the upper visual field on the tip of the tongue. This observation is consistent with previous studies which reported individual differences in natural spatial perspective for spatial patterns presented on the body [19-21]. Yet, there was not such reversal in the left/right relationship.

Static Bar Orientation

Most of the participants were not able to perceive the orientation of the bar when they were first exposed to the stimulation. They developed different strategies to succeed in this task. For example, many participants developed a strategy to move their head horizontally and vertically. If the bar is vertical, the stimulation does not change when they slightly move their head up and down, and thus they concluded that the bar is vertical. Similarly, when the bar is horizontal, stimulation does not change when they move their head from left to right. This strategy suggests that they did not actually perceive the orientation but used a cognitive approach to discriminate the change in sensation with the head movements [22,23]. Another strategy they used was tracing the stimulus with the head or tip of the tongue (See “2. Static Bar Orientation” under the method section for more details). Note that head tracing is different than head movement strategy. In the latter, they did not trace the stimulus; they moved their head along different directions and noted the changes in stimulation. They usually cognitively assembled the percept using the smaller parts they could locate. For instance, instead of perceiving a smooth diagonal bar, they reported detecting stimulation “on the top right and bottom left of the IOD”, and then interpreted that “it should be a diagonal/left-tilted bar”. Thus they were able to perform in the multiple-choice task without actually perceiving bar or line orientation at any instant. Similar cognitive strategies have been recently reported to be used by blind persons implanted retinal prosthesis in orientation and motion discrimination tasks [23].

Localization & Target Counting

Participants were successful in localizing objects by head scanning and bringing the detected stimulation roughly to the centre of the IOD (trainer tracked the position in the BrainPort monitoring view which can be seen in Fig 1). However, especially participants with congenital blindness had difficulty in relating the spatial location of the object to their body and on the blackboard. For instance, even if they scan their head far to the right, if the stimulation falls onto the left side of the IOD, they tended to ignore the position of their head and reported that the object is on their left side relative to their body. A similar problem was reported for patients with tunnel vision using a head-mounted display system to search for objects outside their residual central field [24]. In that situation, the patients responded with eye and head movement based on the position of the image of the target relative to the fovea without regards to the head position [25]. Therefore, an extra time was required to explain the relationship between head position and observed visual field to congenitally blind participants. In addition, some participants assumed that the camera was located on the controller held in the hand in the BrainPort V100 version of the device. Therefore, with this version of the device, it took even more time for participants to learn the head position/spatial location relationship. The later versions of the device appeared to solve this problem by moving the controller to the headset. However, with that heavier headset, some participants reported that the new design was not comfortable causing headache or skin irritation.

Introducing Zoom & Discriminating Tumbling C and E Direction

The concept of zooming-in and -out was hard to grasp for most of the congenitally blind participants. They usually mistook it for a function showing distance between the stimulus and the camera or varying intensity. Participants with acquired blindness grasped the concept of zooming easily.

Participants mainly used one of two strategies to detect the direction of the gap in Tumbling C or E. Some of them zoomed-in all the way so that only a small portion of the test target was presented on the IOD and traced the stimulus with their head. Participants were allowed to use similar head tracing movements (or head-like movements via a computer mouse) in previous acuity studies with Tumbling E or T [26,27]. Others zoomed-in only until all parts of the object were presented on the IOD and then traced the pattern with the tip of the tongue. Some participants had difficulty using gentle head movements. Therefore, they frequently lost the stimulus when zoomed-in. For those participants, tip tracing was more effective. Participants who used the zoom effectively progressed better in the subsequent stages.

Recognizing Shapes & Letters and Numbers

Although Vincent et al. [15] showed that recognition performance was not affected by the shape being filled or outlined while using the BrainPort, our participants preferred the outlined shapes because they were easier to trace with the head or tip of the tongue. As in the bar orientation task, participants first reported the components of shapes, letters, and numbers using the zoom-in function with either head or tip tracing. They then deduced the response from the components they identified. For instance, one participant stated that “There are two legs, and a round top. Therefore, it should be R”. It seems that top-down processing is frequently involved in these forced-choice tasks. Participants combine the information that they could detect from the stimulation with the possible responses based on the assigned task and infer the response. Therefore, these tasks required a lot of focus and cognitive processing, which is fatiguing [28]. One participant who was successful at recognizing letters and numbers stated that “It takes a lot of focus and time so I cannot imagine wanting to use it regularly although I enjoy the in-lab tasks”.

Eight blind and eight normally-sighted participants who trained under this protocol participated in a study where we tested different presentation modes with a letter recognition task [13]. All of them completed at least Stage 8, letter and number recognition, and an additional targeted training specific to the task tested. Although the task in the experiment was similar to the training task at Stage 8 (except time limitation), percent correct performance varied from 39% to 69%. This indicates that an acceptable (above chance) performance in the training does not necessarily lead to a nearly perfect performance even in a relatively similar task. It should be noted that normally-sighted participants’ performance was not included in this report because the training protocol was not strictly followed when training them, and most of them received a shorter training for an hour. Despite the differences in training between blind and normally-sighted participants, both groups performed similarly in the experiment (See Pamir et al. [13] for further details).

Recognizing Real Objects

When moved from the board to the tabletop, due to two-dimensional to three-dimensional expansion of the environment, more variables were inevitably added to the task including distance, perspective, and occlusion, all of which created some confusion for the participants. For instance, many participants perceived objects located at different distances from the participant as if they were located at different heights, rather than different distances. This interpretation makes sense because when a three-dimensional scene is converted into a static two-dimensional low-resolution image, most depth cues are unavailable without reference of size and distance, and the outcome image is consistent with participants’ reports (See Fig 4 for an example).

Figure 4.

A screenshot taken from BrainPort monitoring software with real objects on the table. Left panel shows the camera view and the yellow square indicates the IOD’s field of view. Right panel shows the IOD simulated view. Although limited, there are some depth cues in the camera view such as shadows of the objects and the boundary between the table and the wall enabling interpretation of the depth with normal sight. Although shadows may be seen in the IOD simulated view, at least at a minified view, they do not seem to be perceivable on the IOD, perhaps because of the limited dynamic range on the tongue [17]. Stimulation on the tongue seems mostly binary [13].

Another major challenge for recognizing real objects in space with the BrainPort is the size and distance relationship. When both are unknown to the user, it is almost impossible to decide if the target is a closer small object or a large object located far away, which makes bottom-up recognition very difficult. We observed that most of our participants reported the size of objects relative to the other objects in the scene. For instance, when mug, sanitizer bottle, and funnel are placed side by side on the table, participants reported which one is the tallest and which one is the widest based on the image position on the IOD. We had great difficulty in explaining this relationship to congenitally blind participants. Because this is not an intuitive concept for them, even if they comprehend the logic behind it, they frequently tended to forget about it while concentrating on recognition. As reported above attempts to scale object size with subject hand was not successful due to interference between the hand image and the other object.

General Observations

Our experience with BrainPort training suggests that the length of the training is not a critical factor in reaching an advanced stage with the current training protocol. There are large individual differences in how far and how fast each participant can progress in the training protocol [29].

Recognition with the BrainPort is mainly achieved through top-down processing with prior knowledge of the training stimulus and environment. Participants usually recognize the parts of the stimulus and assemble it in their mind based on the possible responses. Therefore, even though they are successful in the training, the generalizability to everyday tasks is still questionable.

Usually participants could successfully progress in the protocol to the last stage if they were able to use head or tip tracing and zooming function effectively. The limited spatial resolution of the tongue might be the reason why using these strategies are essential in order to be successful in recognizing stimuli with SSDs [13].

Implications for Rehabilitation.

• There are large individual differences in how far and how fast each participant can learn to use a visual-to-tactile sensory substitution device for a variety of tasks.

• Recognition is mainly achieved through top-down processing with prior knowledge about the possible responses. Therefore, the generalizability is still questionable.

• Users develop different strategies in order to succeed in training tasks.

Biography

ZP: Zahide Pamir received her BA degree in Psychology from Bilkent University, Turkey in 2011 and her MS degree in in Cognitive Science from the Middle East Technical University, Turkey in 2014. She received her PhD degree in Neuroscience from Bilkent in 2017. She investigated how fundamental visual features such as contrast and motion are processed in the brain using psychophysics and fMRI methodologies during her graduate studies. Dr. Pamir is currently continuing her academic studies as a postdoctoral research fellow at the Schepens Eye Research Institute of Mass Eye and Ear, Harvard Medical School, where she has been working on low vision rehabilitation.

JHJ: Jae-Hyun Jung is an Instructor at Department of Ophthalmology, Harvard Medical School and an Investigator at Schepens Eye Research Institute of Mass Eye and Ear. His current research interests focus on imaging systems for vision rehabilitation, visual fatigue of human vision related to three-dimensional display, image processing including computational photography and depth extraction. He received his Ph.D. in Electrical Engineering from Seoul National University, Korea, in 2012. He is a Fellow of American Academy of Optometry (FAAO) since 2019. He received the Alice Adler Fellowship Award for Junior Faculty in 2017, best paper of the year award in Schepens Eye Research Institute, Harvard Medical School in 2015, the best researcher award in School of Electrical Engineering, Seoul National University in 2011, and Merck Young Scientists Award in 2009. He received outstanding paper awards in several optical/electrical engineering conferences in 2008, 2010, 2011, and 2012.

EP: Dr. Eli Peli is a Professor of Ophthalmology at Harvard Medical School and the director of the Vision Rehabilitation Service at Tufts Medical Center Hospitals. Dr. Peli is a Fellow of the American Academy of Optometry, a Fellow of the Optical Society of America, a Fellow of the SID, and of the SPIE. He was presented the 2001 Glenn A. Fry Lecture Award, the 2009 William Feinbloom Award, and the 2017 Charles Prentice Medal Award by the American Academy of Optometry, the 2004 Alfred W. Bressler Prize in Vision Science, the 2006 Pisart Vision Award, the 2009 Alcon Research Institute award, the 2010 Otto Schade Prize from the SID, and the 2010 Edwin H Land Medal. Dr. Peli's principal research interests are image processing in relation to visual function and clinical psychophysics in low vision rehabilitation. Dr. Peli has been a consultant to more than 60 companies in the ophthalmic instrumentation area and head mounted displays (HMD). Dr. Peli published more than 220 peer reviewed scientific papers and was awarded 16 US Patents. He edited a book entitled Visual Models for Target Detection and co-authored a book entitled Driving with Confidence: A Practical Guide to Driving with Low Vision.

Data availability statement

The authors confirm that the data supporting the findings of this study are available within the article.