Distal attribution and distance perception in sensory substitution

Abstract

In sensory substitution, the user may be directly aware of distal objects, as in everyday perception, or make explicit cognitive inferences based on an awareness of the proximal stimulation. Anecdotal evidence supports the experience of distal attribution, but so far there have been few rigorous experimental tests of the claim. In this study, blindfolded participants observed a target light using a device consisting of a finger-mounted photodiode that drives tactile vibration on the back. With the blindfold off and the target removed, participants moved a reference object to match the perceived egocentric distance of the target. Participants who were instructed to attend to the distal target improved significantly during two hours of practice, whereas those instructed to attend to proximal variables showed no improvement. Unsigned error increased with ratings of proximal attention but decreased with ratings of target object solidity, consistent with distal attribution. Performance transferred to the non-dominant arm and to a rotated body orientation, demonstrating that learning did not depend on a joint-specific sensorimotor relationship between target distance and arm configuration. The results experimentally confirm that distal attribution can occur in sensory substitution, based on a perceptual strategy rather than an explicit cognitive strategy. Moreover, they suggest that the informational basis for distal attribution is not a joint-specific sensorimotor relation, but a more abstract spatial invariant.

Introduction

In ordinary perception, we are directly aware of objects in the environment, rather than the proximal stimulation on the receptor surface (Harman 1990). This phenomenon of distal attribution, or the perceptual experience of objects in external space (Epstein et al 1986; Loomis 1992), is so ubiquitous and present so early in life that it is extremely difficult to study. To investigate how it arises would require manipulating the usual sensory interface in order to present novel proximal stimulation and analyze the emergence of distal awareness.

Such an opportunity is afforded by the technique of sensory substitution. By inserting an artificial boundary between the environment and the observer, sensory substitution devices create a unique situation in which the user initially experiences the proximal pattern of stimulation. There are numerous reports that distal attribution can subsequently occur, such that the user comes to experience an object in the environment rather than the proximal stimulation (Bach-y-Rita 1972; Lenay et al 2001; Segond et al 2005). However, it remains unclear whether these reports reflect a perceptual experience of distal objects or a cognitive inference based on perceived patterns in the proximal stimulation.

The purpose of the present study is to develop an experimental test for distal attribution, and to use it to investigate the informational basis for this phenomenon. We distinguish between two fundamentally different strategies for using a sensory substitution device: cognitive and perceptual. In a cognitive strategy, one is only directly aware of the proximal stimulation, and the presence of the distal object is derived from high-level cognitive inferences and associations based solely on perceived patterns in the proximal stimulation. This may allow the user to infer that a distal object is present, without experiencing it as such. In a perceptual strategy, on the other hand, one is aware of the distal object itself, without explicitly attending to the proximal stimulation. In this situation, distal attribution takes place, and an object in the environment is perceived. We believe this distinction has not been sufficiently emphasized in the sensory substitution literature.

Previous studies of sensory substitution have offered anecdotal reports suggesting that distal attribution may occur. A classic example is the use of the long cane by the blind, in which the perceiver experiences the distal objects contacted by the cane, rather than the proximal pressure of the handle in the hand (Bach-y-Rita and Kercel 2003). Other early evidence comes from auditory-to-tactile substitution studies, in which vibrations on the chest are used to localize a sound source in space (von Békésy 1967).

In visual-to-tactile devices such as the Tactile Visual Substitution System (TVSS), the image from a video camera is used to drive a vibrating tactile matrix placed on the back of a blind subject (Bach-y-Rita 1972). An experimenter in one such study reported: “When observers were given control of the camera […] the nature of their reports changed significantly. Where before they had reported in terms of feelings on their backs, they now began to give reports in terms of objects that had definite external localization” (White 1970, p 54). After sufficient training, participants in these studies claimed to ignore the stimulation on their backs while becoming directly aware of objects detected by the camera (Bach-y-Rita 1972). Devices such as these have reportedly enabled blind people to navigate without the use of a cane, intercept moving objects, and recognize objects they could not touch (Bach-y-Rita 1969; Bach-y-Rita and Kercel 2003; Jansson 1983). Similar results were reported for visual-to-auditory substitution (Auvray et al 2007) using the vOICe system (Meijer 1992). There have also been recent reports of subjects using sensory substitution to experience objects in space (Segond et al 2005) and experience optical illusions (Renier et al 2005, 2006). Do these functional similarities between sensory substitution and normal visual perception provide sufficient evidence to conclude that participants are directly aware of distal objects?

Although reports of distal attribution in sensory substitution are some of the most exciting findings in this field of research, it is not easy to dismiss the skeptical claim that such reports could actually be based on a cognitive strategy. In the case of visual-to-tactile sensory substitution, for example, participants may be doing nothing more than feeling tactile vibrations and learning to use them to make explicit inferences about distal objects. Consider the case of the TVSS. It is possible that the abilities gained by subjects after many hours of training with feedback were based on cognitive inferences and learned associations. In principle, the presence, location, and identity of environmental objects might be inferred from tactile stimulation without experiencing distal objects in space. As far as we can tell, no controls were included to rule out this scenario. Even if some participants did have a compelling experience of distal objects, there was no attempt to distinguish them experimentally from those who did not.

In a recent study, researchers allowed blindfolded participants to use a visual-to-auditory substitution device about which they were given no prior information (Auvray et al 2005). Participants subsequently rated the likelihood that various descriptions of the device and their surroundings were correct. When a scenario involving the presence of a distal object was rated highly, the experimenters took this to be an indication of distal attribution. We believe this conclusion is not justified, because the ability to infer the presence of an object does not imply the subjective experience of a distal object. In conditions in which the participant could actively occlude the distal object by intermittently blocking it with another solid object, they were more likely to report its presence. The occlusion might have been a cognitive cue, enabling participants to infer the presence of the distal object without actually perceiving it. Although we believe it is extreme, the skeptical claim that distal attribution does not occur in sensory substitution has never been rigorously refuted.

We believe that previous interpretations of sensory substitution experiments suffer from two problems: (1) they take distal attribution for granted, assuming that awareness of objects in sensory substitution is analogous to normal perception without providing sufficient evidence to support this claim, or (2) they fail to distinguish between cognitive and perceptual strategies, making some sensory substitution results appear quite surprising.

In the present experiment, we sought to bias participants toward a cognitive or perceptual strategy by manipulating instructions to attend to proximal or distal variables. We measured the effect of instructions on judgments of egocentric distance and on ratings of subjective perceptual experience. The primary goal of the experiment was to determine which strategy, if either, yields more accurate distance judgments and reports indicative of distal attribution.

Following the work of Lenay (1997) and Lenay et al (2001), we used a highly simplified visual-to-tactile interface. A target light was detected with a single photodiode mounted on the index finger, while tactile stimulation was presented through a single vibrating motor on the participant’s back (Figure 2). Although the path the optical information took to reach the brain was radically different from that of information arriving through the eyes, we hypothesized that blindfolded participants would nonetheless be able to judge the distance of the target in their environment.

Figure 2.

(a) The sensory substitution device used in the experiment. Photodiode on the index finger transduces light from the target and drives a circular vibrating motor on the subject’s back. (b) Pattern of arm movements participants were instructed to make when exploring the target.

Importantly, the signal delivered by the motor was binary. The motor was active when the photodiode was pointed directly toward a light source, and inactive when it was not. Thus, there was no distance information in the stimulation on the skin, making it impossible to localize the target from a single arm position. However, arm movements made sufficient proximal information available over time: proprioception for pointing direction when the tactile stimulus was active allowed a form of implicit triangulation (Cabe et al 2003). A cognitive strategy would be predicated on an awareness of the proximal variables of felt arm configuration and tactile vibration, followed by an explicit triangulation estimate to infer the target distance. With a perceptual strategy, awareness of a distal object would derive from the family of pointing directions producing tactile vibration, which converge on a constant location in space. The participant would not need to attend to the arm configurations or make an explicit triangulation estimate, just as we do not need to attend to cues such as vergence or disparity when making visual depth judgments. Instead, distal attribution would result from the perceptual system abstracting the relation among propriceptively specified pointing directions and tactile vibration that specifies a constant spatial location. By correlating distance judgments with ratings of experience and attention to proximal variables, we can examine whether cognitive or perceptual strategies are more strongly associated with indicators of distal attribution.

A secondary goal of the experiment was to begin investigating the informational basis for distal attribution. Why do we perceive objects as existing out in the world, rather than on the sensory surface? A typical explanation might be that perceiving an object in depth requires activation of neural substrates involved in processing visual depth cues. However, if distal attribution occurs in sensory substitution, this would imply a more general or abstract basis for distal attribution that is not tied to a particular sensory modality.

One current hypothesis proposes that distal awareness is mediated by some form of sensorimotor knowledge. Since the first sensory substitution experiments, it has been observed that the artificial transducer must be actively moved by the participant for distal attribution to be reported (White 1970). Recent proponents of the sensorimotor hypothesis (O’Regan and Noë 2001; Hurley and Noë 2003; Noë 2005) claim that the visual experience of distal objects is constituted by learned sensorimotor expectancies or contingencies, a variant of the motor theory of perception (Weimer 1977). They argue that visual-to-tactile sensory substitution involves similar sensorimotor contingencies as ordinary vision, and consequently results in a similar experience of distal objects despite the different sensory modality.

O’Regan and Noë (2001) interpret Lenay’s (1997) experiment with the finger-mounted photodiode as a concrete example of sensorimotor contingencies. A given target distance along a known egocentric axis (γ) is related to a set of shoulder (α) and elbow (β) angles that “fixate” the target and hence produce tactile vibration (Figure 1a):

$$\beta =arctan\frac{D+c·sin\gamma -a·sin(\alpha -\gamma )}{c·cos\gamma -a·cos(\alpha -\gamma )}-\alpha -\gamma +\pi$$

where a is the length of the upper arm and c is half the shoulder width. This set of arm configurations can be represented as a unique “sensorimotor manifold” in joint space (Figure 1b), with a different manifold for each egocentric axis (γ). Distal awareness of a given target location is thus constituted by knowledge of these sensorimotor contingencies: if the arm is moved into a particular configuration then tactile vibration will result. On this account, distal attribution derives from learning a joint-specific sensorimotor manifold. Thus, it predicts that changing the relationship between target distance and arm configuration would require that a new sensorimotor manifold be learned, temporarily disrupting distance judgments. For example, rotating the trunk by 90° with respect to the target axis would require learning the top manifold (γ = 0°) in place of the bottom manifold (γ = 90°) in Figure 1b. We sought to test whether distal attribution depends on particular arm configurations or a more abstract spatial invariant.

Figure 1.

(a) The relationship between target distance D, trunk angle with respect to the target axis γ, shoulder angle α , and elbow angle β, where upper arm length is constant a and the distance from the head to the shoulder is constant c. (b) The sensorimotor manifold for pointing to a target, computed for three trunk angles with respect to the target (γ = 0, top surface; γ = 45°, middle surface; and γ = 90°, bottom surface) (Modified from Lenay et al., 2001).

In the present study, participants used the visual-to-tactile photodiode device to localize a target in depth, and made an egocentric distance judgment using a matching task, in a version of the method of adjustment. On each trial, participants knew only that the target would be positioned somewhere along the depth axis directly in front of them. Thus, their judgments had to be based on the perceived location of the target, not a set of previously learned target distances (cf. Lenay 1997). Participants never saw the target, nor were they given feedback about its location or the accuracy of their performance.

To investigate the differences between perceptual and cognitive strategies, we manipulated the instructions given to two groups of subjects. In the distal attention condition, participants were told to attend to the target and to ignore their arm position when making distance judgments. In the proximal attention condition, participants were told to attend to their arm position and the tactile vibration and to explicitly triangulate the target location. As far as we know, this is the first sensory substitution experiment to include such a manipulation.

In addition to measuring distance judgments, we also sought to quantify the subjective experience of a distal object. After each of three experimental sessions, participants were asked to rate, on a numerical scale, their experiences of (a) the amount of attention paid to their arms, (b) how much the target felt like a solid object, and (c) their confidence in their distance judgments. The solid object rating provided an indirect measure of the experience of an external object, consistent with the view that the felt resistance to applied force that is provided by solid objects is one basis for the capacity of distal attribution (Smith, 2002). This allowed us to correlate variation in experience with variation in behavior.

Finally, to explore the informational basis of distal attribution, we included a transfer phase. In this portion of the experiment, the testing conditions were altered in order to disrupt any learned sensorimotor relationship between target distance and arm configuration. In one subgroup, the photodiode was moved from the participant’s dominant to non-dominant hand, and in another subgroup the participant’s chair was rotated 90° clockwise. It has been reported that changing the effector that moves an artificial transducer has little or no effect on participants’ abilities to use a sensory substitution device (Bach-y-Rita 1984). We wanted to test whether the ability to judge distance transfers immediately when using our simplified device.

The overarching goal of the experiment was to assess whether judgments of target location in sensory substitution are based on an awareness of proximal variables or distal objects. The ability of users to achieve distal attribution has been anecdotally reported, but not rigorously tested, in previous studies. We hypothesized that distal instructions would promote a perceptual strategy, leading to greater improvement in distance judgments as well as to higher ratings of a solid object than proximal instructions, consistent with distal attribution. In contrast, if we found no differences between the two instruction conditions, we could not reject the skeptical argument that sensory substitution performance is based on a cognitive strategy. Our second goal was to investigate the informational basis of distal attribution by testing whether performance depends on a joint-specific sensorimotor configuration or a more abstract spatial relationship. If it can be shown that distal attribution occurs in sensory substitution, then we can determine the conditions under which it arises, potentially shedding light on the basis of distal awareness in normal perception.

Methods

Participants

There were a total of 31 sighted participants, 13 male and 18 female. Apart from one participant who was 56, the ages of participants ranged between 18 and 36 years old, with a mean age of 22.0 ± 4.3 years. All participants described themselves as strongly right-handed and had normal or corrected-to-normal vision. They were paid at a rate of $8 per hour.

Participants were divided into two groups based on the instructions they received, with 11 participants in the Proximal Attention (PA) group and 20 in the Distal Attention (DA) group. In the transfer phase, the DA group was further divided into two subgroups, with 11 participants in the Arm Transfer condition and 9 in the Rotation Transfer condition. The PA group also performed the Arm Transfer condition.

Another 6 participants were tested as a visual control group (mean age of 21.6 ± 1.2 years). These participants performed the same distance judgment task as the PA and DA groups, but observed the target visually, rather than with the sensory substitution device.

Apparatus

The sensory substitution device was designed to detect the presence or absence of a target light in a given direction (see Figure 2a). It consisted of a single photodiode 0.95 cm in diameter, placed behind a 0.5 cm aperture to increase resolution, and secured to a finger mount. The finger mount was designed to prevent any flexion of the finger joints while wearing the photodiode.

The output of the photodiode, which was proportional to the light intensity it detected, ran through the microphone input of a desktop computer (Dell Dimension 8100 running Windows XP, 1.68 GHz Pentium Processor). The analog signal was digitized at 44100 Hz using Matlab 7.1 (Mathworks, Inc., Natick, MA). The target was a vertical 60-cm-long fluorescent lightbulb, 3.8 cm in diameter, which flickered at a rate of 120 Hz. The digitized signal was Fourier-transformed, and the resulting power at 120-Hz was used to determine the presence or absence of the target, based on a threshold of 1/10 the maximum power.

On each trial, the target could be placed anywhere along a 193-cm track, which stood 72 cm off the ground. Whenever the photodiode was pointed directly at the target, a vibrating pager motor 1.3 cm in diameter (Solarbotics Ltd., Calgary, AB) was activated using the output channel of a data acquisition box (Measurement Computing, Norton, MA). The motor took 240 ms to reach full power, but could be felt by participants immediately after activation. The motor remained active as long as the 120-Hz power in the photodiode’s signal was above threshold. The motor was mounted in the center of the participants’ seat back. It contacted the midline of the participant’s back approximately 36 cm above the horizontal cushion. The back-mounted location was chosen because stimulation of this region of the body is very rarely correlated with movements of the arm and was opposite the direction of the target. Thus, no meaningful spatial information can be gleaned from the motor alone. The entire device cycled at an average rate of 19.2 ± 1.18 Hz, meaning the status of the motor was refreshed every 0.05 seconds.

The response device was a wheeled robot (TAB Electronics, 11.5 cm wide × 16.5 cm long × 4.5 cm high) positioned in the same track and marked on the front with a horizontal red line (0.6 cm wide). The participant used a hand-held remote control to adjust the position of the robot along the track until the red line matched the remembered target location.

Procedure

Before the experiment began, the photodiode was securely mounted on the index finger, and participants were given a verbal description of how the device functioned and instructions on how to use it. Participants in the Proximal Attention (PA) group were instructed to attend to the location of their arm when the vibrating motor was active, and to consciously triangulate the location of the target by imagining their finger extending out into space. Participants in the Distal Attention (DA) group were explicitly told to not attend to their arms during the experiment, but to get an intuitive sense of the target’s location and report how far away it felt. These instructions were based on reports from participants in a pilot study, who were not given attention instructions (J.H.S. and W.H.W., unpublished results). They tended to report either explicitly triangulating the location of the target by attending to their arms, or obtaining an intuitive sense of the target’s external location.

Other than the instructions, the procedure was identical for both groups. Both groups were told to sweep their arm back and forth, then bend their elbow and sweep the arm again (see Figure 2b). Participants had to keep their arm moving continuously while it passed over the target, and to keep their arm parallel to the table and at a constant height. They were instructed to bend their arm only at the shoulder and the elbow, unless their arm crossed the midline of their body, in which case they could bend at the wrist.

Following the learning phase (first two sessions), the DA group was divided into two subgroups for the transfer phase (third session). Participants in the Arm Transfer condition (N = 11) switched the photodiode from their dominant to their non-dominant hand. Participants in the Rotation Transfer condition (N = 9) rotated their chair 90° with respect to the target, in the direction of their dominant hand. A subset of participants in the PA group (N = 9) also completed the Arm Transfer condition.

The general procedure for each trial was as follows. First, the participant placed the blindfold over his or her eyes, put on a set of headphones, and the room lights were turned off. While white noise played through the headphones, the experimenter placed the target on the track. The white noise was turned off, the photodiode was activated, and the blindfolded participant was instructed to explore with the device. Once the participant gave a verbal signal or a 2-minute time limit was reached, the photodiode was deactivated and the participant was instructed to rest his or her arm on the table. White noise again played through the headphones while the experimenter removed the target and placed it out of view. The track was then illuminated with an auxiliary lamp (60-Watt incandescent bulb), the white noise was turned off, and the participant was instructed to remove the blindfold and make a response. The participant adjusted the position of the robot so that the probe line was aligned as closely as possible with the remembered location of the target. The experimenter recorded the location of the probe, removed the robot, and began the next trial. Throughout the experiment, participants never actually saw the target light.

Participants completed 30 trials in each of three sessions, for a total of 90 trials. The sessions took place on separate days, usually with 48 hours between sessions. In each session, 10 target distances (approximately 40, 50, 60, …, 130 cm) were presented 3 times in a pseudorandom order. There were no markings on the track to indicate distance, and at no point did the participant receive feedback about the actual location of the target during a given trial, or about the general accuracy of their judgments.

At the end of each session, participants completed a verbal questionnaire. First, they responded to three questions: (1) “How much were you attending to your arms as the basis for your judgment?” (1 = not attending to arms at all, 10 = only attending to arms), (2) “What would you expect the target to feel like if you reached out and touched it?” (1 = nothing at all, it is imaginary; 10 = solid object) and (3) “How confident were you in your distance judgments?” (1 = least confident, 10 = most confident)? Next, participants were asked to describe, in their own words, their technique for judging the distance of the target. These questions were based on the self-reports collected during the pilot study (J.H.S. and W.H.W., unpublished results). Participants in the pilot study who reported an intuitive sense of the target’s external location also frequently reported that the target itself felt like a “solid” object.

Results

Learning phase

Three error measures for distance judgments are presented in Figure 3 for the first 20 trials (1–20) and the last 20 trials (40–60) of the learning phase. To assess improvement in distal attribution, we decided at the outset to compare data from the first and last 20 learning trials to estimate the initial and final state of distal awareness. This minimized learning effects during the first session and avoided transients at the beginning of the second session as participants became reacquainted with the device.

Figure 3.

(a) Mean unsigned error, (b) mean variable error (standard deviation of signed error), and (c) mean constant error (signed error), for the first 20 trials and last 20 trials of the learning phase. Error bars represent between-subject SE. DA = Distal Attention instructions, PA = Proximal Attention instructions.

The results reveal a significant effect of instructions on participants’ abilities to learn to judge distances with the device. First, consider the unsigned error between judged and actual target distances, a measure of overall performance (Figure 3a). In the Distal Attention (DA) group, the mean unsigned error decreased by 25% between the first and last 20 learning trials, but it remained unchanged in the Proximal Attention (PA) group. A 2×2 mixed-design ANOVA revealed a marginal main effect of instructions, (F_1,29 = 3.341, p = 0.0779), a main effect of learning (F_1,29 = 8.935, p = 0.0057), and, most importantly, a significant interaction (F_1,29 = 5.044, p = 0.0325). Post-hoc tests revealed a significant decrease in mean unsigned error between the first and last 20 trials for the DA group, but not for PA group (Bonferroni correction, α = 0.05). This confirms a greater overall improvement with distal instructions than proximal instructions.

Next, consider the mean variable error (within-subject standard deviation of signed error over 20 trials), a measure of judgment reliability (Figure 3b). In the Distal Attention (DA) group, this measure decreased by 24% between the first and last 20 trials, but it remained essentially unchanged in the Proximal Attention (PA) group. A 2×2 mixed ANOVA revealed no main effect of instructions, (F_1,29 = 0.657, p = 0.4243), a main effect of learning (F_1,29 = 12.716, p = 0.0013), and a significant interaction (F_1,29 = 5.576, p = 0.0251). Post-hoc tests revealed a significant decrease in variable error between the first and last 20 trials for the DA group, but none for the PA group (Bonferroni correction, α = 0.05). Although the variable error on the first 20 trials appears higher in the DA group than the PA group, this difference was not significant (p = 0.11, uncorrected t-test). These results indicate that the distal instructions yielded a greater reduction in variability than the proximal instructions, consistent with the perception of a more definite object location. The distal instructions may have led DA participants to attend to an initially vague impression of an external target that became more definite over trials, yielding an improvement in reliability during learning. In contrast, the proximal instructions may have led PA participants to use an explicit arm triangulation strategy that was reasonably reliable at the start, but did not improve over trials. This pattern of results is thus consistent with the emergence of an increasingly definite target location in the DA group, but not the PA group.

Next, consider the mean constant error (signed error), a measure of judgment bias (Figure 3c); positive errors indicate matching responses that overestimated the target distance, and negative errors those that underestimated target distance. In the Distal Attention (DA) group, the constant error decreased by 20% between the first and last 20 trials, compared to 12% for the Proximal Attention (PA) group. A 2×2 mixed ANOVA found a marginal effect of instructions, (F_1,29 = 3.675, p = 0.0651), no effect of learning (F_1,29 = 2.320, p = 0.1386), and no interaction (F_1,29 = 0.023, p = 0.8793). Taken together, the results indicate a greater improvement with distal than proximal instructions.

For purposes of comparison, the errors in the visual control group were generally much lower than in the sensory substitution groups. The mean unsigned error of visual distance judgments was 9.7 ± 3.2 cm, the mean variable error was 8.8 ± 2.8 cm, and the mean constant error was – 4.8 ± 7.7 cm. This confirms that most of the error in the DA and PA groups was due to the haptic localization task rather than the visual response.

Mean distance judgments are plotted as a function of actual target distance for the first and last 20 learning trials in Figure 4. On the first 20 trials (Figure 4a), the regression equation for the DA group (solid curve), y = 0.41 x + 75.3 (r = 0.40), was nearly equal to that for the PA group (dashed curve), y = 0.40 x + 84.1 (r = 0.46). In contrast, on the last 20 trials (Figure 4b), the regression equation for the DA group (solid curve) was y = 0.59 x + 53.6 (r = 0.59), whereas that for the PA group (dashed curve) it was y = 0.36 x + 82.2 (r = 0.34). Thus, both groups overestimated target distance, but the DA group was more accurate, had a slope much closer to 1.0, and a higher correlation coefficient than the PA group. During the learning phase, DA participants improved their ability to localize near targets, which brought down their overall bias.

Figure 4.

Mean judged distance as a function of actual target distance for the DA group (circles) and PA group (squares) in (a) the first 20 learning trials and (b) the last 20 learning trials. Plotted with best-fit regression lines. Error bars represent between-subject SE.

To examine learning effects across all 60 learning trials, we performed a regression analysis of unsigned error as a function of trial number for each participant. The mean slope of the best-fit line for all DA participants was −0.1087, which was significantly negative (p < 0.0001), indicating a reduction in error over trials. The mean slope for all PA participants over the same period is −0.0271, which was not significantly different from zero (p > 0.25), indicating no improvement. An ANOVA on all 60 learning trials that treats trial number as a continuous predictor reveals no effect of condition (F(1,1854) = 0.5, p = 0.4814), a significant effect of trial number (F(1,1854) = 14.81, p = 0.0001), and a significant interaction (F(1,1854) = 11.47, p = 0.0007). This result is consistent with our conclusion that practice has significantly different effects for participants in the PA and DA conditions.

Individual participants

The unsigned error on each trial over the course of the entire experiment (including the transfer phase) is shown for three participants in Figure 5 (light curve); these data are plotted as a function of time to reflect the amount of experience with the device, for some participants responded faster than others. Participant YPR, an exemplary participant in the DA condition, began the experiment with a mean unsigned error of approximately 50 cm. After about 30 minutes of learning, the mean unsigned error for consecutive blocks of 5 trials (heavy curve) dropped suddenly to around 10 cm, the same level of performance as the visual control group. YPR’s mean variable error of 8.8 ± 2.8 was lower than the visual controls, and the mean constant error was only −0.68 cm..

Figure 5.

Sample time series of unsigned error during learning. (a) An exceptional DA participant (YPR), (b) a typical DA participant (GTF), (c) a typical PA participant (RKF). Thin curves represent unsigned error on individual trials. Heavy curves represent mean unsigned error for consecutive blocks of five trials. Data is plotted as a function of the time the photodiode was turned on and being actively manipulated by the subject. Horizontal line represents the mean unsigned error for visual control group (9.71 ± 3.18 cm).

Participant GTF, a more typical DA subject, also began the experiment with an unsigned error around 50 cm, and reached a minimum unsigned error (mean for blocks of 5 trials) of about 25 cm after 20 minutes of learning. Once this minimum was reached, unsigned errors for individual trials rarely deviated above 60 cm (one-third of the length of the track).

Participant RKF, a typical PA participant, showed little or no improvement over the course of the experiment. Mean unsigned error for blocks of 5 trials hovered between 30 and 60 cm over all 60 trials. The standard deviation of the mean signed error remained high until the end of the experiment, indicating that this participant’s localization abilities with the device are quite inconsistent.

Correlations between ratings and distance judgments

The mean questionnaire ratings for each group appear in Figure 6a, along with their mean unsigned errors, for both the learning and transfer phases. There was a trend for the DA group to report lower attention to arms, greater solid object ratings, and higher confidence in their distance judgments. A mixed ANOVA with a between-subjects variable of instruction condition and a within-subjects variable of experimental phase revealed a significant main effect of instructions only for confidence ratings (F_1,88 = 4.42, p = 0.044).

Figure 6.

(a) Mean unsigned error (Err) and mean subjective ratings during the learning phase. “Arms”: 1 = not attending to arms at all, 10 = only attending to arms. “Solid”: 1 = feels like there is no object in front of me; 10 = feels like there is a solid object in front of me. “Conf”: 1 = not at all confident, 10 = very confident. Error bars represent between-subject SE. Note: The bar plot for “Err” uses the left y-axis, while the bar plots for “Arms,” “Solid,” and “Conf” use the right y-axis. (b) The judgment surface: mean unsigned error as a function of subjective ratings of attention to arms and target solidity, shown with best-fit multiple linear regression surface. Lightness of surface corresponds to mean unsigned error.

We performed a correlational analysis on this data, pooled across instruction groups and learning sessions. A participant’s mean unsigned error in a given session (30 trials) was correlated with their questionnaire ratings at the end of that session. The key results are represented as a “Judgment Surface” in Figure 6b, which plots mean unsigned error as a function of attention to arms and solid object rating. First, mean error is positively correlated with attention to the arms (r = 0.2594, p < 0.05), implying that the more a participant attended to proximal variables, the higher their error. Conversely, mean error is negatively correlated with the perceived solidity of the target (r = −0.3790, p < 0.001), that is, the more a participant felt the target to be a solid object, the lower their error. Mean error was also negatively correlated with confidence ratings (r = −0.2915, p < 0.05), confirming that a participant’s uncertainty increased with the error in their judgments. In addition, confidence ratings were positively correlated with solid object ratings (r = 0.3821, p < 0.001), indicating that a more compelling experience of a distal object went hand-in-hand with a higher confidence in judging its distance. These correlations, though modest, suggest that participants who attended to distal properties came to experience a more solid target object, improved their distance judgments, and had greater confidence in those judgments.

Transfer phase

In order to provide a strong test of transfer, we decided a priori to analyze only the first 20 transfer trials (out of 30), thereby minimizing any learning effects during the transfer phase. The first nine participants in the PA group had a mean unsigned error of 33.52 ± 5.94 cm in the transfer phase, comparable to their performance in the learning phase. However, because the PA group did not demonstrate a significant improvement during learning, these results do not bear on hypotheses about the basis of distal attribution. We thus discontinued the transfer test for the remaining two PA participants.

For the DA group, the mean unsigned error in each transfer condition appears in Figure 7, together with that for the first and last 20 trials of the learning phase. In both conditions, performance in the transfer phase was not significantly different from the last 20 learning trials. Ratings of the impression of a solid object, attention to arms, and confidence also remained the same after the transfer trials (two-tailed t-test, p > 0.5). In contrast, mean error in the arm transfer condition remained significantly lower than the first 20 learning trials (t(10) = 3.7051, p < 0.005, two-tailed paired t-test). Similarly, mean error in the rotation transfer condition also remained significantly lower (t(8) = 3.0096, p < 0.05, two-tailed paired t-test). Furthermore, mean unsigned error for DA participants in the transfer phase remained significantly lower than that of PA participants completing the transfer phase (t(26) = 2.2371, p < 0.05, two-tailed t-test). These results indicate that performance is not affected by transferring the photodiode to the opposite arm or rotating the body by 90°. Thus, the ability to localize the target with the sensory substitution device is not specific to learned joint configurations.

Figure 7.

Mean unsigned error for the DA group in the transfer phase. Bars represent the first and last 20 learning trials and the first 20 transfer trials in the Arm Transfer and Rotation Transfer conditions. Error bars represent between-subject SE.

Discussion

The present study was designed to investigate whether the use of a simple sensory substitution device leads to perceptual awareness of distal objects, as opposed to cognitive inferences about the environment based on an awareness of proximal variables. A secondary aim was to explore the informational basis for such a distal awareness. We found that instructions to attend to distal properties during learning resulted in improved performance and more precise judgments of target distance, whereas instructions to attend to proximal arm position yielded no improvement in distance judgments. In addition, improved distance judgments were significantly correlated with increased perception of a solid object and inversely related to attention to the arms. Finally, improved distance judgments immediately transferred to the opposite arm and to a 90° body rotation. This pattern of results leads to three main conclusions.

First, the data provide moderate experimental support for the emergence of distal attribution in sensory substitution. In as few as 60 trials (about 2 hours of experience) without feedback, the Distal Attention group exhibited a significant reduction in the unsigned error and variable error of their distance judgments. This is consistent with perceiving the target at a more definite distal location. Moreover, lower errors were correlated with solid object ratings, consistent with perceiving a more concrete physical object at the target location. In contrast, higher errors were correlated with participants’ ratings of attention to their arms, that is, to the relevant proximal variables. Although these effects were modest, they were statistically significant. Distal attribution may have been limited by the somewhat sluggish response of the vibrating motor, which took 240 ms to reach peak activation, and may have yielded the impression of a somewhat indistinct object with fuzzy edges at best. Despite this limitation, the results cohere as evidence of the emergence of distal attribution in some participants: those who reported greater distal attention experienced a more solid object, better performance on distance judgments, and higher confidence in those judgments. This pattern is consistent with an emerging distal awareness of a physical object at a definite spatial location.

Second, the results indicate that improved distance judgments reflect a distal perceptual awareness rather than an explicit cognitive strategy. It is possible that reports consistent with distal attribution in sensory substitution actually derive from explicit inferences about distal objects based on an awareness of proximal variables. In the present case, participants might be explicitly triangulating the location of the target based on a proprioceptive awareness of their arm positions together with the tactile vibration. However, our results indicate that the PA group—who were instructed to attend to their arm positions and use a conscious triangulation strategy—did not improve in the distance-judgment task. In contrast, participants in the DA group improved their performance and became significantly more precise. The particular aspect of the instructions that facilitated improvement cannot be determined from this study, because DA participants were instructed both to attend to the distal object and to ignore their arms. Nevertheless, although both strategies yield above-chance performance, the results support the view that a cognitive strategy actually limits performance whereas a putatively perceptual one results in improvement, consistent with the emergence of distal awareness.

Of course, there may be other training procedures in which cognitive strategies can also yield improved performance. For example, if feedback about distal properties is provided during training, participants may learn to make better inferences from proximal variables without distal awareness. In sensory substitution devices that do not reproduce “natural” patterns of stimulation, participants may even learn arbitrary (symbolic) associations between proximal and distal properties (Bach-y-Rita and Kercel 2003; Kaczmarek 1995). Previous researchers have often failed to clearly distinguish cognitive and perceptual strategies, with the consequence that they do not control for the possible use of cognitive strategies and may misinterpret the results of cognitive strategies as reflecting perceptual mechanisms. Distinguishing these strategies is necessary to characterize the conditions under which distal attribution occurs, as well as for successful rehabilitative and augmentative applications of sensory substitution techniques.

Third, given the present evidence for distal attribution, the results from the transfer phase allow us to draw some preliminary conclusions about its informational basis. The reduction in unsigned error observed in the DA group was maintained during transfer from the dominant to the non-dominant arm, and when the body was rotated 90° with respect to the target. These data clearly indicate that the emergence of distal awareness did not depend on a particular set of arm configurations at a muscle- and joint-specific level. This finding is consistent with studies that have shown that action calibration is organized functionally, rather than anatomically, such that learning with one set of muscle groups or limbs often transfers to other actions in service of the same goal (Withagen and Michaels 2002; Withagen and Michaels 2004).

In the present task, distance judgments depend on proprioception for arm positions that produce tactile vibration, together with an implicit or explicit triangulation process. This pattern of relations can be described at various levels of abstraction. At a joint-specific level, a given target distance corresponds to a particular set of elbow and shoulder angles, represented as a manifold in joint space (Figure 1b). O’Regan and Noë’s (2001) version of the sensorimotor hypothesis states that distal awareness of target location is constituted by knowledge of the sensorimotor contingencies represented by this manifold—that is, knowledge that if the arm is moved into certain configurations, tactile vibration will result. However, if the body is rotated by 90°, the set of joint angles that yield vibration changes completely; thus, a new manifold would have to be learned before distal awareness is recovered and distance is reliably perceived. Nonetheless, DA participants in the rotation condition exhibited complete transfer to the new body orientation. More radically, transfer to the opposite arm changed not only the joint angles but the joints and muscles involved, yet DA participants in this condition also exhibited complete transfer. These data undermine a muscle- or joint-specific version of the sensorimotor hypothesis, and indeed any other hypothesis defined at the level of limb configurations.

An alternative hypothesis places less importance on specific sensorimotor relationships and more on the active detection of abstract spatial invariants as the basis for distal attribution. On this view, movement is necessary to reveal invariant information about the distal layout and to dissociate it from varying stimulation that depends on self-movement (Gibson 1979). In the present case, multiple pointing directions that elicit tactile vibration are required to triangulate a constant target location, and these must be distinguished from other pointing directions that do not yield vibration. In addition, sensing the arm’s pointing direction may be enhanced by active movement (Feldman and Latash 1982; Pagano and Turvey 1995).

The invariance hypothesis accounts for distal attribution by claiming that patterns of stimulation that are invariant under self-produced movement are due to distal environmental entities. Thus, the family of pointing directions that elicit vibration converge on a constant location in space, in a form of implicit triangulation, and yields awareness of a distal object. This view assumes that proprioceptive information is available about the position and orientation of the hand relative to the body, specifying a pointing direction. The family of pointing directions can be characterized as a spatial invariant that specifies target location (Gibson 1979), or perhaps as a higher-level sensorimotor contingency that is not joint-specific (O’Regan and Noë 2001). Either way, the abstract relation among pointing directions converges on a constant spatial location despite body rotation and arm transfer, consistent with the present data. While the specifics of this hypothesis remain to be tested empirically, the present results support a fairly abstract characterization of the information that supports the emergence of distal awareness.

Conclusion

We have argued that a sensory substitution device can be used in two fundamentally different ways: as a basis for cognitive inferences, or as a means of perceiving the distal environment. Future studies of sensory substitution should take this distinction into account and control for possible cognitive strategies. The present results offer preliminary experimental support for the emergence of distal attribution—i.e., distal perceptual awareness—in at least some participants using a sensory substitution device. Moreover, the results show that the informational support for distal awareness is not muscle- and joint-specific, but should be characterized at a fairly abstract level.

These findings have implications for the practical use of sensory substitution in perceptual rehabilitation and augmentation. Although the present study focused on visual-to-tactile sensory substitution, the experimental paradigm can be easily adapted for other sensory substitution devices. All types of sensory substitution are vulnerable to the claim that subjects are merely using some feature of the proximal stimulus to make cognitive inferences, and not experiencing genuine distal attribution. Further studies of distal attribution intended to examine its informational basis, the role of a stationary or moving observer, and the contribution of passive or active movement, may inform the design of interfaces that can be more effectively integrated into human perceptual-motor systems.

On a theoretical level, we believe that sensory substitution has the potential to help us understand the basis of distal awareness in ordinary perception. Although the present study focused on distal awareness in visual-to-tactile sensory substitution, the experimental paradigm can be easily adapted for use with any type of sensory substitution device. All forms of sensory substitution are vulnerable to the claim that a perceptual strategy is being used when, in fact, subjects are merely exploiting some feature of the proximal stimulus to make their judgments, and not experiencing genuine distal attribution. Distal attribution is a pervasive feature of everyday experience, but to study how it might arise requires manipulating the usual sensory interface between environment and perceiver. The present findings suggest that distal awareness can emerge rather rapidly with such a novel interface, without explicit cognitive inference. This implies that distal attribution does not depend on dedicated neural pathways, but rather on appropriate relations between observer movement and invariant patterns of stimulation. Experiments with sensory substitution devices thus offer a promising route to understand the basis of distal attribution in normal, or “unsubstituted,” perception.