Visual-Haptic Mapping and the Origin of Crossmodal Identity

Abstract

We found that the congenitally blind person who gains sight initially fails to identify seen objects with their felt versions: a negative answer to the Molyneux question. However, s(he) succeeds in doing so after a few days of sight. We argue that this rapid learning resembles that of adaptation to rearrangement in which the experimentally produced separations of seen and felt perceptions of objects are rapidly reunited by the process called capture. Moreover, the original ability to identify objects across modalities by the neonate may be assured by the same process.

The Question

In 1688 William Molyneux sent a message¹ to the philosopher John Locke asking: “Suppose a man born blind, and now adult, and taught by his touch to distinguish between a cube and a sphere of the same metal … Suppose then the cube and sphere placed on a table, and the blind man be made to see: query, whether by his sight, before he touched them he could now distinguish and tell which is the globe, which the cube …?”

Taken literally in contemporary terms, will the newly sighted, formerly blind patient identify by sight objects s(he) has experienced only by touch and feel (haptics) ? Although the history of cases of recovery from blindness goes back to antiquity² and continues in recent times^3,4, Molyneux's question has not yet been convincingly answered. The reason for this lack is failure to meet the stringent requirements for definitively answering the question. First of all, the blindness must have been verifiably congenital and continuous. Otherwise there may have been an opportunity for acquisition of the crossmodal transfer through visual experience whose exclusion is the purpose of testing the previously blind. When still blind the patient should demonstrate light sense, in order to assure function of the retina, optic nerve and beyond, but resolution no better than light and dark. After successful surgery the patient must exhibit acuity sufficient to discriminate visually among the objects used for testing. Selected cases of blindness from occlusion of both eyes, such as may be caused by occlusive cataracts or corneal scarring, best satisfy these conditions. Post-op testing should begin as soon after surgery as possible - ideally when bandages are first removed. Crossmodal recognition has been reported in some cases long after surgery. But if it has not been tested immediately we cannot exclude the possibility of acquisition by experience and, as we have found (read below) acquisition seems to occur within days after return of normal conditions of seeing.

Recently the Prakash group, defined in reference ⁵, reported the outcome of tests performed in India and specifically designed to answer the classic Molyneux question.⁶ They managed to conduct critical tests on three successfully operated patients. Not only did they satisfy requirements specified above, but also the structure of the tests further assured compliance with the requirements.

Three two-alternative forced choice tests were administered. Initially one of two target stimuli, either a small object to be palpated but not seen or the object seen but not touched, was presented within arm's length for a short period of familiarization. Following immediately upon removal of the haptic target, either a pair of haptic or of visible objects, one of which was the target, the other novel, was presented and the patient instructed to choose which one of the two s(he) had previously experienced. Following removal of the visible target a pair of such objects, one of which was the target were presented for choice. The crossmodal choice tested crossmodal transfer. The two intramodal choices tested the discriminability of the stimuli in each modality as well as the patient's understanding of the task. As reported at VSS 2008 ⁶ all three patients performed at chance on the crossmodal choice but were clearly capable of the intramodal discriminations. Hence, the answer to our version of the Molyneux question is “no” within the confidence limits of the procedures. However, a few days after the patients were released the tests were repeated yielding the result of crossmodal transfer better than chance. We are left with the surprising finding of acquisition of the ability to perform the cross-modal task within a few days after surgery.

The rapidity of acquisition of crossmodal transfer implies that we must attribute the result to some process, which acts quickly so as to allow the newly sighted to perceive the identity between seen and felt impressions of objects. We conjecture that this same process could operate in the normally sighted infant from birth so as to enable the crossmodal identification reported in an extensive review to occur as early as two months of age ⁷ (but see below). In the following we try to establish the case that the sought-for process is the same as that which accounts for adaptation to perturbations of visual-haptic spatial mapping.

Visual-Haptic Mapping

What is involved in performing the crossmodal transfer ? Haptic and visual sensorineural channels are independent at early processing levels but at later levels coupling between their signals must be established to account for transfer when it occurs. It is the origin of this coupling (innate vs empirical) that has always underlain recurrent raising of the Molyneux question. Since, apart from time, spatial loci are the simplest perceptual properties common to both vision and haptics, it would seem that the coupling must be based on congruency of spatiotemporal signals. Congruency could in turn be determined from a map of haptic on visual spatiotemporal signals. The following argues for a process by which this mapping may be accomplished.

The performer of the crossmodal task first palpates the invisible object to gain its haptic representation (neuronal) in terms of a sequence of discrete movements with their tactual and proprioceptive accompaniments. The haptic representation must in turn elicit a visual representation for comparison with a visual target. We infer that in memory haptic representations are mapped against congruent visual representations that once activated may be compared with the visible samples to establish either identity or difference. The same must happen in opposite sequence when viewing an object without touching it and then recognizing it by touch. The seen and felt representations mapped together are then the presumed basis of crossmodal identification. According to our findings the congenitally blind lack such mappings. But they must rapidly develop after vision is achieved. Since it is the hand and its movements that create the haptic representation, one might as well restrict attention to the hand itself and consider the palpated representations of objects to be derived from a concatenation of the hand's positions and movements in space. Consideration is then reduced to that of how the hand itself achieves and retains the map of its seen and felt representations. This map should also account for the control of positioning and repositioning of the hand in body space as in reaching, for example, specification of the goal of a reach would be the mapped pair of representations of the hand at some remove - the goal of the reach -- from that of its initial position. One way of studying the mapping process is to perturb the habitual map and to observe the conditions under which recovery is achieved. These conditions are then candidates for accounting for the original mapping as it may occur in cases of recovered vision and perhaps in infancy.

Perturbing the Map

Normally the habitual seen-felt mapping is such that a visual stimulus can elicit an accurate reach (or other response) to the locus of the source of the stimulus. But it is easy to alter this orderly relation by any number of means: optical, mechanical, and electronic. Accordingly, for more than a century the use of procedures that do so has constituted a small industry of experiments known variously as sensory recombinations, rearrangements, perceptual compensation, adaptation of space perception etc, (see Welch ⁸ for an excellent but dated review). Rearrangement experiments have been performed in vision, hearing, and haptic sense modalities. The classic rearrangements are optical. Over a hundred years ago George Stratton studied the consequences of optically rotating his field of vision by 180 deg.^9,10 He also wore a mirror device designed to have him view his own body from above¹¹ . In time, at least some of the disturbing consequences, such as mislocalizations, were reduced if not eliminated and body parts were perceived as outside his own body. Stratton's findings were recently replicated using video systems to displace the subject's view of his own body ^12,13. Even earlier in time, ¹⁴ von Helmholtz discussed the wedge prism type of rearrangement that produces a lateral displacement of the image of the hand with erroneous localization followed in time by correction. This type of rearrangement was made popular beginning in the 1960s following which innumerable variations were performed ⁸ and continue to be performed. Mechanical analogues of the Helmholtz prism experiment have proliferated ^15-18, as have electronic analogues ^{12,13,19,20 21}. The essential feature of these types of perturbation is the production of a visible stimulus spatially displaced but closely coupled in time with haptic stimulation from a body part, often a limb.

Consider the classic Helmholtz experiment of seeing through refracting wedge prisms, which displace images on the retinae. It is as if the images of objects were displaced around the head without moving the objects. The observer, who may be initially unaware of the change, uses his habitual mapping of seen and felt versions of objects with the result that reaching for a visible target without sight of the hand (open loop), will be in error by approximately the prism-induced optical displacement. This displacement error has been widely termed a discrepancy in the literature ⁸ and regarded as the cause of consequent adaptation by error correction without much further explanation. But a further step in this experiment is revelatory of how compensation really occurs. The moment the observer sees his hand through the prism the spatial separation of seen and felt loci is either not sensed at all or is at least reduced from what might be predicted on the basis of physical separation ²². Despite the prism-induced geometric separation of haptic and visual signals, the immediate effect of viewing the hand through a wedge prism tends to eliminate the discrepancy that might be predicted on the basis of physical separation. Even very brief exposures to near-simultaneous seen and felt signals from the hand/arm remap their combination so as to restore correct localization. Moreover, a similar case can be made for a perturbation of vision that alters the shape of objects as in the experiments of Rock et al,²³. The prism-produced displacement supplies a parameter value, which remaps the entire range of representation of the felt hand in body-centered space ²⁴. Harris and others²⁵ have attributed these changes to an altered position sense of the limb leaving open the question of whether it is a cause or an effect. Since the haptic localization tends to approximate the visual, it is referred to as “captured” after Tastevin ¹⁸. When complete capture is achieved, haptic and visual representations tend to be congruent and reaching for a visible target or other mode of visual-haptic localization becomes accurate. In summary, this observation implies that spatial discrepancy, instead of being just an error signal, actually supplies the parameter value for remapping the relation between visual and haptic space thereby restoring spatial congruence between the perturbed sensory systems. The term near-simultaneous can be quantified from the effects of time delay on adaptation ^19,26.

Partial Capture

In most rearrangement experiments the capture-displacement reported is partial, a consequence, we believe, of the residual influence of the previous habitual mapping between vision and haptics. There appears to be an averaging process between the full capture remapping and the previous habitual mapping. This account is borne out by several observations. The aftereffect, following removal of the prism and return to habitual conditions of seeing, can be taken as evidence for persistence of the prism-induced remapping however short-lived it may be. Just as the habitual mapping persists during the remapping consequence of prism exposure, so the remapping now persists after prism removal and continues to be averaged with the habitual mapping. With continued exposure to this condition, the new mapping becomes stronger, the effective adaptation increases, and the aftereffects will more and more clearly outlast the generating condition on removal of the perturbation. When experimenters (and subjects) are sufficiently persistent, full and exact adaptation can be produced as in some of the experiments of Ivo Kohler, Hein, Held & Bossom, Mikaelian & Held ^27-30 and others. In a few cases dual localizations - one showing complete adaptation, the other zero -- have been reported^28,31. Finally, the adaptive shift never significantly exceeds the initial differential between visual and haptic loci. The discrepancy between visual and haptic representations determines the extent of the remapping, which may be reduced by the influence of prior mapping, but never significantly increased beyond full correction of the induced error.

Adaptation

Capture is the prototype of adaptation to rearrangement. Analysis of many of the various types of rearrangement reveals that after the initial sight of the rearranged scene, continuing exposure constitutes iteration of the capture conditions. Many prism experiments have repeated short-lived conjunctions of visual and haptic stimulations ^16,32 which are, in effect,capture opportunities. Others entail continuous movement of arm and hand. Since the hand movements, as observed by the performer, obligatorily entail a conjunction of near-simultaneous changes in vision and proprioception they also constitute a series of capture opportunities. Iteration of movements with their simultaneous paired signals constitutes synchronous signals. Active movement ensures synchroneity among the sensory accompaniments of movement: visual, tactual (haptic), and auditory. Desynchronization by passively imposed movement ³³, time delay ²¹, mechanical interference ³⁴ or other source of noise ¹³ reduces or eliminates adaptation. It has also been pointed out that active movement entails a more selective proprioceptive response than does passive movement with a probable decrease in noise.³⁵

The Answer

Now it may be obvious how the process that adapts the perturbation-induced discrepancy may account for the rapid acquisition of crossmodal mapping. For the blind there are no visual representations. On acquiring vision and returning to normal activities -- viewing the hand for example -- sources of visual stimulation should rapidly establish neural representations in the normal course of post-surgical activity. Here we must recognize that such representations may very well not be as well calibrated as the habitually available ones involved in prism adaptation. Nonetheless we press on. Initially there should be no map of combined visual-haptic space. Just as in prism perturbation, the spatial representations are likely to be discrepant and crossmodal identification absent. But the crossmodal interaction via the process we have explored above, especially capture, should quickly map common loci for objects detected by both senses. In the case of the former patient there can be no previous mapping. Hence the new mapping should be accurate and serve as a basis for the crossmodal matching capability.

The case for acquisition by the neonate is similar. However, recently Streri and Gentaz ³⁶ in ingenious studies find that newborns show crossmodal recognition of shape from hand to eyes and, consequently, claim that they have demonstrated amodal perception of shape and have answered “yes” to the Molyneux question. However, the authors mention a caveat to their conclusion whose interpretation they characterize as “very difficult”: that is, the crossmodal transfer is demonstrable only with the right hand although both hands were tested. Perhaps another caveat is in order. What the authors call a “newborn” is in fact 54 hours old on average. Consider the propensity of infants to assume the ATNR (asymmetrical tonic neck reflex) posture placing a hand in front of their eyes. Given the speed of capture and its hand/arm selectivity, the conditions are ripe for an interpretation of the infants' crossmodal transfer using the right hand in terms of the mapping process discussed above. Moveover, this interpretation would relieve the embarassment of having to claim that an amodal perception is specific to one hand. But irrespective of the newborn's native capacity, the adaptation process should at least provide an assurance of correct performance by both hands.