Visuo-tactile cross-modal associations in cortical somatosensory cells

Abstract

Recent studies show that cells in the somatosensory cortex are involved in the short-term retention of tactile information. In addition, some somatosensory cells appear to retain visual information that has been associated with the touch of an object. The presence of such cells suggests that nontactile stimuli associated with touch have access to cortical neuron networks engaged in the haptic sense. Thus, we inferred that somatosensory cells would respond to behaviorally associated visual and tactile stimuli. To test this assumption, single units were recorded from the anterior parietal cortex (Brodmann's areas 3a, 3b, 1, and 2) of monkeys performing a visuo-haptic delay task, which required the memorization of a visual cue for a tactile choice. Most cells responding to that cue responded also to the corresponding object presented for tactile choice. Significant correlations were observed in some cells between their differential reactions to tactile objects and their differential reactions to the associated visual cues. Some cells were recorded in both the cross-modal task and a haptic unimodal task, where the animal had to retain a tactile cue for a tactile choice. In most of these cells, correlations were observed between stimulus-related firing in corresponding cue periods of the two tasks. These findings suggest that cells in somatosensory cortex are the components of neuronal networks representing tactile information. Associated visual stimuli may activate such networks through visuo-haptic associations established by behavioral training.

Anatomical and physiological studies indicate that somatosensory information is processed both in parallel and sequentially through areas 3, 1, and 2 of anterior parietal cortex (1–6). Cells in these areas respond not only to the passive touch of the hand (7, 8) but also to the active grasping of objects (9–13). Recent work (14) indicates that cells in somatosensory cortex significantly change their firing frequency while the animal has to memorize information about a tactile stimulus for a later behavioral choice. In some cells, that change in firing is selective (stimulus dependent). Furthermore, in a cross-modal (visuo-haptic) delay task (15), some cells appear to participate in the short-term retention of associated nontactile (visual) information for a later tactile choice. This evidence suggests that somatosensory cells are part of widely distributed cortical networks that are active in the perception and short-term memory of haptic information. In haptic behavior, one such network and its constituent neurons may be activated by any of the stimuli behaviorally associated with a tactile discrimination, including those of other sensory modalities. Following this reasoning, we assumed that, during the performance by the animal of a cross-modal visuo-haptic task, somatosensory cells would show similar responses to pairs of visual and tactile stimuli that have been associated with each other by prior training. Electrophysiological evidence of cross-modal association has been obtained in some areas of the association cortex, such as V4 and inferotemporal cortex (16–19). In the anterior bank of the intraparietal sulcus, some tactually responsive cells have been reported to have visual receptive fields (20). Neurons in the lateral intraparietal area have been seen to respond to auditory stimuli in an auditory-saccade task (21, 22). The present study searched for cells in the somatosensory cortex that would show correlated changes of firing frequency in response to visual and tactile stimuli that identify the same physical object.

Methods

Three adult rhesus monkeys (Macaca mulatta) were the experimental subjects for this study. They had been used in studies of short-term memory (14, 15). Animal care and surgical procedures were approved by the Animal Research Committee at the University of California, Los Angeles. The animals were trained to perform a visuo-haptic memory task in a fully automated, computer-controlled apparatus. During the task, the animal was seated in a primate chair facing a panel with a rectangular screen at about eye level for visual display (30 × 50 mm) and a rectangular opening at about waist level for access to tactile test objects (Fig. 1). The distance between the eyes of the animal and the screen was about 20 cm. A pair of visual images (icons) was used. These were black and white patterns of parallel stripes (3.5 mm apart). The stripes were vertical in one icon and horizontal in the other. The opening for the tactile test objects was normally closed by a shutter. When the shutter was opened (downward sliding), the animal could reach out through the opening and manipulate the objects behind the panel (the objects were at all times out of sight). The test objects were two vertical cylindrical rods of identical dimensions (axis, 150 mm and diameter, 19 mm), but different direction of parallel ridges on their surface (ridges 6 mm apart). One rod had the ridges along the axis of the cylinder (vertical ridges) and the other around its circumference (horizontal ridges). When not in the act of reaching and grasping the objects, the performing hand of the animal rested on a rounded pedal (handrest) in the center of the lower edge of the opening. The other hand was at all times restricted from access to the test objects by a plate attached to the primate chair. A displacement-sensitive transducer was connected to the spring-suspended seat of the animal. Signals from this transducer were recorded and used for control of body movements.

Figure 1.

(Upper) V-H TASK. Schematic diagram of the visuo-haptic cross-modal task. (Upper Left) Monkey watching the visual cue (icon) in the center of the panel, the operating hand resting on the handrest. (The opening that gives manual access to the test objects is occluded.) (Upper Right) Events in a trial of the task labeled sequentially. Cell discharge is analyzed in three trial epochs (time spans indicated): visual cue, delay, and haptic choice. In this task, the animal rests his hand on the handrest continuously except for the choice period. In the sample trial displayed, the visual cue is the vertical icon, and the animal correctly matches the cue with a pull on the vertical rod. (Lower) H-H TASK. (Lower Left) A simplified drawing of the test apparatus for the haptic-haptic unimodal task. The monkey palpates the sample object. (Lower Right) Schematic diagram of the events in a trial. The animal touches the vertical rod during the sample period, and after the delay pulls it in the correct choice.

A task trial consisted of the following sequence of events—all registered by electronic switches and sensors. The trial began with the 3-s presentation of one of the icons (vertical or horizontal). The direction of its stripes symbolized the direction of the ridges in the rod to be chosen haptically later, after a 20-s delay (14 s in some tests). At the end of that delay, an auditory signal marked the accessibility of the two rods side by side. One rod was 5.5 cm to the right of the center of the opening, and the other 5.5 cm to the left. As soon as they became accessible, the animal reached out and palpated the rods. A pull of either rod led to closure of the opening and to fluid reward (2 ml) if the choice was correct. The icon and the position of its corresponding rod changed at random between trials. This prevented the animal from using spatial clues for the choices. Early in the study, instead of the icon, the monkey was presented with the sight of the sample object through a window at eye level—later substituted by the projection display screen.

Before testing the animals on the visuo-haptic task described above, the animals had been trained on a unimodal (haptic-haptic) task. This allowed comparisons of unit discharge between the two tasks. In that unimodal tactile task, a trial started with a click, which served as an alerting signal about 1.5 s before the touch of the tactile sample. The sample object was identical to one of the objects used for visuo-haptic choice. Here, instead of the visual cue, the animal had to palpate that sample object for the tactile match after the delay. In every other respect, the task was identical to the cross-modal task.

After the monkey had undergone behavioral training (performance criterion above 75% correct), two cylindrical pedestals for microelectrode recording were implanted bilaterally on the parietal cortex, leaving the dura intact. The pedestals were intended to be placed over hand representation areas of anterior parietal cortex (Brodmann's areas 3a, 3b, 1, and 2). The implantation was guided by cranial landmarks, our own experience with previous implants, and a stereotaxic map (courtesy of T. P. Pons, Wake Forest University, Winston-Salem, NC). In one of the animals, a pair of EOG (electrooculogram) electrodes was implanted in the periorbital bone for monitoring horizontal eye movements.

For a recording session, the animal was placed in the testing apparatus with its head fixed. Single-unit activity was recorded extracellularly with Elgiloy microelectrodes (impedance 1–2 megaohms). Spike records were selected for analysis on the basis of stability, uniformity, and clear isolation from background noise and the spikes from other units. In some units, the receptive field on the hand of the animal was tested. The unit spikes were converted into standard pulses and saved on computer disks, together with electrical markers for the trial events and microelectrode position. Both the unit spikes and the standard pulses generated by them were monitored on oscilloscope, and the average spike-frequency histograms of recorded units were displayed online in computer monitors. Spike data acquisition started 17 s before the cue and ended 7 s after the choice pull. The activity during the 17-s pretrial period was used as baseline record. During that baseline period and the subsequent visual cue and delay periods, the animal sat quietly in the chair and rested its hand on the handrest. If the animal moved the hand off the handrest, or its body movement exceeded a certain level during either of those periods, the trial was canceled. During the recording from any given unit, we attempted to obtain spike records through a minimum of 10 correct-response trials with each cue. The records from a few units with relatively fast and stable firing were also collected for the analysis, even though tested on fewer trials.

The firing of each unit was statistically analyzed, especially with regard to frequency changes in temporal relation to the trial events. In the visuo-haptic cross-modal task, these events were: (i) the visual cue-on; (ii) the cue-off; (iii) the hand leaving the handrest pedal for choice; (iv) the first touch of the object matching the sample; and (v) the pull at the choice of an object. The first two events (i and ii) defined the end of baseline, the cue period, and the beginning of the delay period. The event marker #4 was used for histogram and time-locking in statistical analysis of unit activity during the choice period. In the haptic-haptic task, the marked events were: (i) the click; (ii) the performing hand leaving the pedal for the sample object (pedal-off); (iii) the first contact with the sample; (iv) the last contact with the sample; (v) the hand return to the pedal (pedal-on); (vi) the hand leaving the pedal for choice; (vii) the first touch of the object matching the sample; and (viii) the pull of the chosen object. Of these event markers, the first five (i to v) defined successively the end of baseline, the sample period, and the delay period. Differences of average firing frequency between each trial period and intertrial baseline were submitted to a Student's t test, by using intertrial variance as the basis of the error term (P < 0.05). The differences of responsiveness of a unit between one visual cue, or one test object, and the other, were also assessed by calculating the differences of the differences of average firing with respect to baseline (t test). For examining correlations of unit firing between different trial epochs, the cue and choice periods were divided into 500-ms bins. The cue-and object-related differences of responsiveness were calculated, and the bin with the largest difference was used in correlation analysis. A Monte-Carlo simulation was also used to confirm the statistical significance of the results. Fictitious “cells”—with random temporal distribution of interspike intervals—were generated based on the frequency range of the cells observed; those fictitious “cells” were submitted to the analyses used on real data. After completion of the recording experiments, small electrolytic lesions were made in the cortex and underlying white matter at multiple locations. These microlesions were used as reference marks for localization of the recorded units. The estimated positions of all units were reconstructed and marked on enlarged photographic prints of the sections (80 μm) stained by the Nissl method.

Results

The learning of the tasks was accompanied by gradual increases in accuracy and efficiency of performance. Hand movements became precise, stereotypical, and economical, of almost constant duration from one trial to the next. In the fully trained animal, object sampling by touch (in the unimodal task) took about 1.5 s. During the choice period of both tasks, the hand of the animal almost always grasped first the object on the side of its operating hand. In most cases, if the object matched the sample at the first grasp, the animal pulled it; if not, the hand of the animal would go to the other side to feel and pull the correct object. The duration of the choice period was about 1–1.5 s (time from the first touch of the cue-corresponding object to the pull).

One hundred twenty-six single units constitute the database of this study. They were all recorded from anterior parietal cortex (most of them in hand-representation areas). Out of those 126 units, 25 were recorded in the visuo-haptic cross-modal task only. Fifty-four units were recorded in both visuo-haptic task and haptic-haptic task. Forty-seven units were recorded in the haptic-haptic unimodal task only.

Out of the 79 cells that were recorded in the visuo-haptic task, 39 showed change in their firing frequency while the visual cue was present, 27 with increased firing and 12 with decreased firing with respect to baseline, with or without stimulus selectivity. Thirty-three of the 39 visually responsive cells also showed alteration in their firing frequency during the haptic-choice period. In the majority of those cells (24/33), the tactile reaction to the objects was in the same direction (excitation or inhibition) as the reaction to the visual cue (Fig. 2). Twenty-eight of the 39 cells were also recorded in the haptic-haptic unimodal task. Eight of those cells failed to respond to the click, in other words, to the attention-eliciting signal in that task, even though they responded to the visual cue in the visuo-haptic task.

Figure 2.

Average frequency histograms (bin size, 500 ms) from two units (unit A and unit B); firing in all trials is averaged together, regardless of icon (trial number in parentheses), horizontal (Hori.) or vertical (Vert.) stripes. The locations of the two units are indicated by triangles in brain section diagrams. (A) This unit increases the firing rate when the animal is presented with the visual icons in the cue period, and the tactile objects in the tactile choice period. The two insets show changes in short time scale (bin size, 50 ms). One inset averages discharges around the onset of the visual cue, and the other around the first contact with an object at choice. (B) Increased firing with the presentation of the cue. Firing stays elevated throughout the delay. This unit also responds to object touch.

Sixteen cells (16/79, 20.3%) showed stimulus selectivity at the haptic choice: significant differential reaction (P < 0.05) to the two tactile objects during the choice period. A majority (13/16, significant proportion, P < 0.05 by sign test, and Monte-Carlo simulation) showed coherent reactions to the visual cues and tactile choices, even though preference for a cue was not statistically significant in every cell (in five cells it was, P < 0.05) (Fig. 3). Thus, a preferential cell response to the vertical ridges during the haptic choice period tended to be preceded, at the cue period, by preferential response to the vertical cue, and likewise for horizontal ridges and cue. (Of the 63 cells that were nondifferential in the choice period, 34 responded in the same direction in both the cue and choice periods and 29 in the opposite direction.)

Figure 3.

Rasters and average frequency histograms (bin size, 500 ms) from a unit showing differential activity in the two periods, cue (Left) and choice (Right). On the left, the time-locking event is the onset of the visual cue; on the right, the first contact of the matching object. The middle part of the delay is omitted from the histograms (dotted line). The cell favors the horizontal visual cue (white histogram, P < 0.05). Accordingly, it favors the horizontal ridges (white histogram) at the choice (P < 0.01).

Correlational analysis of stimulus-related differences between trial periods was performed in the 79 cells recorded in the cross-modal task. The 16 differential cells showed significant correlations (Fig. 4) between the cue and the choice periods (r1 = 0.6913, P < 0.01). No significant correlations were found in the Monte Carlo-generated trains. Nor were correlations found in the remaining 63, nondifferential, cells (r2 = −0.1409). The difference between r1 and r2 was highly significant (P < 0.01).

Figure 4.

Correlations of cell-firing between the haptic-choice period and the cue period in the visuo-haptic task. (D1) Difference between average firing frequencies elicited by the two objects at the tactile choice; and (D2) difference in average firing to the two visual stimuli at the cue period. (A) Correlation calculated in 16 cells, showing significant object-selective reaction at choice. (B) Correlation in 63 nonselective cells. The differential cells, but not the nondifferential cells, show significant correlation (P < 0.01).

Fifty-four cells, out of the 79 recorded during the cross-modal task, were also recorded in the unimodal task. Their activities in the two tasks were compared. In the cue period, 37 units (37/54, significant proportion, P < 0.01, by sign test, and Monte Carlo simulation) exhibited the equivalent stimulus preference (same direction of response preference) in the two tasks (Fig. 5), whereas 17 showed the opposite preference; in the delay period, 33 units showed coherent firing after the stimuli of the two tasks. In sum, in each of those two periods, a majority of cells preferred equivalent stimuli of the two modalities.

Figure 5.

Average frequency histograms (bin size, 1 s) from a cell recorded in both the haptic-haptic unimodal task and the visuo-haptic cross-modal task. In the haptic-haptic task, the white histogram represents the average firing of the cell in trials with the horizontal rod as the sample, and the patterned histogram the records of trials with the vertical rod. The locking event for the histograms is the beginning of the touch of the sample object. In the visuo-haptic task, the locking event is the visual cue-on. (White histogram for horizontal icon, and patterned histogram for vertical icon.) The cell, in both tasks, prefers the horizontal pattern of stripes or ridges.

Forty-seven cells were recorded solely in the unimodal haptic task. They were selected on the basis of their differential reaction in the tactile sample period (cue). Only 17 (17/47, 36.2%) showed a significant differential reaction in both sample and choice periods. In all 17 cells, both reactions were preferential for the same object.

Discussion

The present results confirm our inference that, in the animal trained to perform a cross-modal visuo-haptic task, cells of the somatosensory cortex would change their firing frequency when the animal is presented with either visual or tactile stimuli that have become associated in the learning of the task. As our data show, the discharge of some cells exhibits concordant selectivity to the associated stimuli of the two modalities—either between cue and choice in the cross-modal task, or between the cues of the two tasks, unimodal and bimodal. Because in those cells the activation is cross-modally correlated, it is reasonable to suppose that those cells are component elements of a network in somatosensory cortex representing the tactile objects, and that the visual stimuli have access to that network and modulate the activity of its cells by way of a cross-modal association that may take place in other cortical areas. Both kinds of stimuli, in either task, cross-modal or unimodal, may therefore activate the same haptic network. Thus, by association, the visual cue may activate the internal representation of a tactile stimulus. Somatosensory cells of that network may respond to the tactile information that the visual stimulus represents, but not necessarily to that stimulus itself.

The results of this study complement those of our previous study (15). There we showed that somatosensory cells engaged in short-term memory (delay activity) of the visual stimulus that the animal needed for a subsequent haptic choice. Present results go beyond the role of those cells in “working” visual memory. By revealing cross-modal parallels of cell response to behaviorally associated stimuli of different modality, present findings suggest that neurons in somatosensory networks are part of the cortical representation of the associations between stimuli.

Even at an early stage of a cortical sensory system (primary areas), the functional properties of cells are more subject to plastic change than generally recognized (23). This notion is well supported by studies of cortical plasticity in several sensory systems of the adult animal (24–30). Our findings indicate that cells in anterior parietal cortex, an early cortical stage of somatosensory processing, participate in high-level cognitive operations (discrimination, short-term memory, and decision making). These operations imply that a preexisting neuronal substrate, in the same cortex, has become part of the long-term memory of cross-modal associations. Thus, the same substrate may be used by the animal for the representation as well as the analysis of sensory information (31–34). A related assumption is that in performance of the tasks, the activity of somatosensory cells is modulated by “higher level” cortical areas, such as posterior parietal cortex (3). Recent studies show the formation in this cortex of experience-induced cross-modal associations through behavioral training (21, 22). Cells in the lateral intraparietal area respond to auditory stimulation after the animal has been trained in an auditory-saccade task. That area had been thought to be unresponsive to auditory stimulation. Another study (35) shows cross-modal (audio-visual) associations in cells of prefrontal cortex. Cross-modal associations have been described in other cortices of association—V4 and inferotemporal cortex (16–19).

The participation of somatosensory cortex in cross-modal associations has at least one general implication relevant to cortical neurobiology. That implication is that primary sensory cortices not only can process modality-specific sensory information but also can become part of widely distributed cortical networks that encode the procedural memory of cross-modal behavioral tasks. Furthermore, the results of our study strengthen the multimodal nature of cortical memory networks. The functional operation of these networks in those tasks is largely unknown, but it may depend, at least in part, on inputs mediated by the claustrum, a structure implicated in the formation of cortical cross-modal associations (36, 37).

The performance of the visuo-haptic delay task requires, essentially, the orderly activation of those postulated networks in the perception of the visual cue, its short-term retention, and the tactile choice. Those three cognitive processes are causally and successively linked to each other. Thus, the reaction of a somatosensory cell to the touch of the objects at the choice probably reflects the integration of tactile information with information on the visual cue, the latter having been retained through the delay. That integration of temporally separated items of information (cue and choice), which is essential in goal-directed behavioral sequences, is empirically supported here by evidence of significant firing correlations between cue and choice. On the other hand, the finding that in the unimodal task, some cells respond selectively to an object as the tactile cue but not to the same object for tactile choice suggests that in the choice period, additional factors intervene (e.g., decision making). In any case, the integration at the choice may involve inputs from other brain structures anatomically connected with somatic cortex, such as the posterior parietal cortex (38) and the primary motor cortex (39–41).

Because present results show unit reactions related to performance of a learned visuo-haptic task in primary sensory cortex, we have interpreted our results as evidence that this cortex participates in perception and short-term retention of visual stimuli associated with touch. Neurons in somatosensory networks may receive modulation from other associative cortical areas in which the cross-modal association may occur. This evidence appears novel, at least for a specific sensory cortex. Nevertheless, our findings need to be further substantiated by temporally longitudinal unit recordings from the same cortical area in the process of the learning of the task by the animal.